Community
Participate
Working Groups
Currently it is not possible to run/deploy regular Equinox bundles with Virgo from the Eclipse IDE. On the other hand, PDE does not work out of the box with EclipseRT bundle projects. I guess this is mostly because of the different structures, when the bundles live as projects in the IDE rather being packed as regular OSGi JAR files. It would be great if PDE/Euqinox bundles would be first class citizens in the Virgo world. This would make it much easier to develop server-side OSGi bundles for Virgo using Eclipse as IDE.
I would like to see support for this, too. It's truly a pain when you share bundles between rcp and virgo apps.
Thanks for raising this. Flagging as enhancement.
+1 for this. Please! :-) Thomas Kratz also wanted this: (see http://eclipsedriven.blogspot.com/2011/06/eclipse-virgo-ide-tooling-100m01.html ) " Dear Martin, Dear Spring, please let me deploy PDE bundles. "
IMHO, the first step in this direction would be to replaces the Virgo 'EclipseRT OSGi Bundle' facet with the Libra 'OSGi Bundle' facet. If you need help with this, let me know.
+1 for this.
+1 from me too.
+1
*** Bug 353382 has been marked as a duplicate of this bug. ***
I'm thinking that this should be the number one priority after taking care of the dependency issues and some key server issues. It will really open up Virgo to a much wider set of usages, including a lot of cool stuff outside of traditional web deployment scenarios. I think a really nice goal to have in mind would be to create a seamless experience where for example working with a Virgo Runtime target would have exactly (or nearly) the same usage patterns as working with a hosted Eclipse app.
OK, now that we've taken care of the more fundamental build and quality issues, it's time to look at this one more seriously. Rather than plow ahead with a solution that I think is best, we'd like to get feedback on what would work best for actual users. Basically the issue is "how do we best provide a seamless developer experience in developing, provisioning and managing PDE bundles within various Virgo Runtime targets"? Because of the way that Eclipse, PDE, P2 and various other tooling has evolved, there are an almost astonishing number of possibilities to do pretty much the same thing, all of which have plus and minutes. I'd like to try to put together one canonical story, though of course that doesn't prevent us from offering other possibilities. ______________ Here are a few user stories that I came up with, please feel free to add your own: 1. Workspace-driven. Allow users to arbitrarily select bundles (plugin projects) from workspace and deploy them into Virgo in an ad hoc way. Perhaps also support this at the feature level. 2. Builder and/or nature based approach. This enhances 1 as suggested in comment 4, where we support the ability to add a wtp facet to any bundle such that this bundle could then be deployed to a server by adding it directly using the Server Add and Remove tool or other server tooling. 2. Pure feature.xml model. User defines *one feature to rule them all" that specifies all features, included features, and isolated plugins that should be included within a deployment. RCP has of course the support of a .product construct, but I don't think we'd need that to define dependencies. Best practice no I think in RCP products is to include one and only one feature in a product, and use that to define all of the dependencies. The product file also allows you to define other pieces such as bundle start level, configurations, icons, etc. with the possible exception of the first are really appropriate to Virgo for those obvious reason that we aren't defining a stand-alone project. 3. Launcher support. This would allow developers to specify launch configuration so that full environments could be launched into an existing and/or perhaps automatically configured Virgo runtime. You could manage launch dependencies in the same way that you do so now when defining runtime configurations. 4. Target platform based approach. You could define a full set of features from various arbitrary locations that are then used to populate the Runtime. This would mostly leverage the existing Target Platform and P2 tooling that we all know and love. 5. Leverage the existing Server based tooling. Here you could open up a Server in the editor, and install bundles from the existing workspace. This is sort of like 1 and 2 but from a server centric point of view. 6. Support all of the above through Virgo plans/pars artifacts. As scary as it is to say, I think it's likely that I've actually missed other options. Of course we'll have to balance what we'd like with what is most doable given resources. WRT to dependencies I'm not even going to discuss approaches here that involve other technologies, such as Maven, Buckminster, etc.. not that they're a bad idea, but you'll have to build that yourself. :) Whatever solution we use should not require any additional tooling except for WTP -- that seems unavoidable and we have the dependency already anyway -- and perhaps Libra. As Libra has a focus on enabling this kind of stuff as well, we would like to coordinate whatever we do with those approaches so that we're leveraging appropriate technology and not duplicating effort. ______________ So those were the options. Here are my not completely thought out and likely to change opinions: 1 and 2: As much as they seem like low-hanging fruit, I'm a bit leery. While it scratches an immediate itch case 1 isn't really that maintainable. And really it's kind of pointless. Case 2 requires the inclusion of a specific nature for every project that you want to deploy as a bundle, which is I think one of the things we are trying to avoid. This doesn't exclude some sort of (potentially Libra based) nature option, and we do need some way to actually provide a hook for the WTS tooling, but it may be that some sort of feature project based solution is a better way to accomplish this. Perhaps people feel differently about this, but my bias is toward supporting existing plugin based developers as seamlessly as possible, without requiring them to make changes to their existing projects. 3. This is the one that I like best in terms of maintainability, flexibility and docking with existing run and building approaches. My quess is that it would also be relativly easy to leverage all of the existing WTP goodness using this approach. 4. Like 1 and 2, this one would be totally seamless for existing PDE developers and IIRC is the way that tools like GWT support PDE like development. It was the option I was initially imagining as an ideal approach. But it has the major disadvantage of being tied to a particular user's workspace and idiosyncrasies. It may also be more difficult to develop and integrate with existing tooling. It's also a potential source of confusion for users as the launchers are also used to manage the Virgo runtime itself. But it definitely has it's attractions. 5. While I can see that it would be attractive to have the sense of directly working with an existing server, and it would be helpful from a maintenance and testing POV, I don't think we should be defining and managing this stuff at the server level. I think from a developer perspective we should get away from the idea of maintaining a separate monolithic server somewhere and instead focus on creating repeatable atomic installations that can safely and seamlessly be deployed anywhere. But I don't have enough experience with web server side stuff to know if that is a realistic way to look at actual development scenarios. 6. The key question here is whether we use a pure tooling approach to deploying these artifacts or do we want to leverage existing artifacts, e.g. pars and plans? I think the answer here is the latter. It's important that we don't duplicate functionality that is already provided within Virgo itself, and especially that leverage the existing Virgo Server constructs as a lot of work and thought has gone into those, and we want to have seamless maintainability with Virgo installations that are used for say combined Web deployments and PDE-based deployments. I would favor using plans, not parsand taking a strong feature based approach. Plans give us the really nice benefit of allowing control over deployment order, and perhaps this could even be enriched down the line. One downside is that we would then have to have at least one project that contains these artifacts. We might want (sigh) anew artifact to associate a plan with a feature and eventually to allow other configuration. Or we could create a simple builder that produced a plan from a feature. I like that option, and it is by far the simplest to implement, though it means that you probably couldn't maintain the plan separately. So, with all of that discussion out of the way, here's what I would do today if I was locked in a room without internet access for a few days: Provide builder, nature and associated IDE integrations for feature projects that generated a plan file. You might also be able to manually turn a feature into a plan through a menu item. Included in the plan would be all of the plugins and included features. There wouldn't be any additional dependency management beyond that. OK, your turn..
> Provide builder, nature and associated IDE integrations for feature projects > that generated a plan file. You might also be able to manually turn a feature > into a plan through a menu item. Included in the plan would be all of the > plugins and included features. There wouldn't be any additional dependency > management beyond that. Realized I made that task sound a lot simpler than it is! Not only do we need to create the plan, we actually need to build the resources and deploy them to the server. :) This is where we'd like be able to leverage PDE and perhaps Libra tools the most. One option here would be to reuse the existing "Deployable Feature" export capability, but I'd really like to avoid that heavy-weight a solution. I think we're all hoping for the interactive Eclipse development flow. Ideally that would be "Save, builds and it's there" but at worst you'd want to be able to do a single Publish and have the export handled seamlessly and quickly; and we really couldn't use the PDE Export mechanism for that. OTOH, might we also need to support a more hardened/consistent (and perhaps supporting headless) option for deployment of artifacts?
(In reply to comment #11) > 4. Like 1 and 2, this one would be totally seamless for existing PDE developers > and IIRC is the way that tools like GWT support PDE like development. It was > the option I was initially imagining as an ideal approach. But it has the major > disadvantage of being tied to a particular user's workspace and idiosyncrasies. > It may also be more difficult to develop and integrate with existing tooling. > It's also a potential source of confusion for users as the launchers are also > used to manage the Virgo runtime itself. But it definitely has it's > attractions. Note that in Juno, PDE has added an API to work with target definitions. While your workspace still only can have one active target platform, you can have multiple target definitions. The target definitions can be stored in the workspace metadata and edited with a wizard (target platform preference page) or they can be stored in projects as xml files (*.target) and edited with an editor. I am sure there will be limits with how far this could be extended, but this option is more feasible in Juno+. API Tooling uses the target API to generate their baselines from a product install.
(In reply to comment #13) > (In reply to comment #11) > > 4. Like 1 and 2, this one would be totally seamless for existing PDE > developers > > and IIRC is the way that tools like GWT support PDE like development. It was > > the option I was initially imagining as an ideal approach. But it has the > major > > disadvantage of being tied to a particular user's workspace and > idiosyncrasies. > > It may also be more difficult to develop and integrate with existing tooling. > > It's also a potential source of confusion for users as the launchers are also > > used to manage the Virgo runtime itself. But it definitely has it's > > attractions. > > Note that in Juno, PDE has added an API to work with target definitions. While > your workspace still only can have one active target platform, you can have > multiple target definitions. The target definitions can be stored in the > workspace metadata and edited with a wizard (target platform preference page) or > they can be stored in projects as xml files (*.target) and edited with an > editor. > > I am sure there will be limits with how far this could be extended, but this > option is more feasible in Juno+. API Tooling uses the target API to generate > their baselines from a product install. Thanks Curtis, that info is extremely helpful. So if we used .target as the primary artifact, we could leverage all of that dependency management stuff. We could also allow people to define arbitrary installations that may or may not have any inclusions from the current workspace itself. The downside in development scenarios would be that it would require a one to one mapping between what the developer is working on and what is deployed to the server in the case where you have Also, as there is no provision for build artifacts, we would have to layer on top of that or again make use of pieces of the Launch capabilities to allow inclusion of selected workspace items. Then we'd need some way of persisting and maintaining that. I wonder how much of that Runtime support stuff is provided as API. Again, as problematic as it can sometimes be, the RCP model seems that it might be a good fit here, as you are providing a set of features on top of a target platform. (FWIW, Buckminster also uses Target Platforms extensively, both as a consumer and producer.) One really cool potential advantage of such approach that included target platforms could be that it could allow complete auto-provisioning of a Virgo app. That is, once you've developed a local application, imagine building that app to a P2 update site and then being able to take a single artifact, copy it over to a running virgo server, and have it contact all of the P2 sites itself to provision the complete application. (Buckminster uses something like that approach.) Glyn, how extensive is the p2 support on the Virgo server side? Any new functionality anticipated there that might be relevant?
IMHO there are three use cases in OSGi/Virgo development: 1. Developing a base application (similar to an RCP Product or an EAR) 2. Developing a bundle (or a plug-in, or a library) 3. Developing an app server / stack / runtime engine, with predefined dependencies In all cases the core app bundles themselves would be "plain" OSGi bundles, not Eclipse plug-ins / PDE projects. However, these bundles may depend on packages or use services that are provided by Eclipse plug-ins. The dependency plug-ins may be in the workspace or may be specified via target platform, e.g. from the Orbit p2 repository. Another possibility is that some bundles can be provided via OBR repository. IMHO at least a manual way to deploy plug-ins whether workspace plug-ins or plug-ins in target platform to a Virgo server instance (there can be several server instances, just like Tomcat etc.) should be provided. This is the most flexible and is useful during experimental/early developments, but it is not "scalable" for "quick launch" use case (ie. other developer checking out a project and want to run it with experience like deploying a plain WAR). With this functionality, Saving a file should trigger a build and redeploy of the affected bundle. No Virgo restart is necessary and this works without needing to use Debug (it's OSGi anyway). If this functionality is working, after that we can move on to more complex functionality like feature/plan support. But I think we should get the most basic stuff working first.
FWIW, I'm a huge fan of the Equinox "-dev" option. It allows to use the bundles as is from the workspace without any deployment step. The PDE launch configurations do all the magic (eg., generate the necessary bundles.info). So, all of this would just work if I can launch Virgo as an "Eclipse Application". :)
Miles, it would make sense if you take a look at the server adapters for Equinox, Felix and Knopflerfish contributed to the Libra project. These are the org.eclipse.libra.framework.* packages. These server adapter are based on a target definition and wrap all the PDE logic behind the WTP Server Framework. I agree with Gunnar that the Equinox "-dev" options is a great one.
in my view most critical bit would be working with two (possibly conflicting) "target platforms", i.e. PDE uses target platform definitions for dependency resolution and Virgo uses target runtimes (a la WTP) for bundle dependencies. Working with multiple target platforms in the same application would be riddled with problems for users. For example if one adds a PDE bundle to a Virgo server, the "publishing" job maybe required to resolve dependencies from two conflicting targets, which may not be solvable problem if there are conflicts. The best method I can imagine is to expose the Virgo Target Runtime bundles as a Target Platform so that Virgo Bundle Projects and PDE Bundle projects use the same set of bundles. We are experimenting with a similar idea for Libra launchers by creating a distinct target platform definition for each framework instance. This would remove the multiple bundle sources issue. However, working with Target Platform definitions the PDE way is not good from the user experience point of view. WTP target platform concept is much better way to handle the problem. A nice solution could be the ability add a target runtime into a PDE target platform definition and delegate the bundle resolution to Target Runtimes. PDE Projects would not changed (no new natures builders facets etc.) Libra may not be a bad place to build this support.
Hi Everyone, What amazing feedback! Some further clarifications. Everyone please weigh in w/ your + or - on issues you care about. (In reply to Hendy's comment #15) > In all cases the core app bundles themselves would be "plain" OSGi bundles, not > Eclipse plug-ins / PDE projects. Yes, it doesn't make sense for them to be bundle projects, but what about PDE feature projects, that is including the feature builder and nature? This seems like a very natural way of providing a clean path for PDE developers. > However, these bundles may depend on packages or use services that are provided > by Eclipse plug-ins. The dependency plug-ins may be in the workspace or may be > specified via target platform, e.g. from the Orbit p2 repository. Another > possibility is that some bundles can be provided via OBR repository. From what little I know on this, the OBR repository is not planned to be maintained. So this is a really good reason to begin to think about transitioning that all to Orbit. > (there can be several server instances, just like Tomcat etc.) Yes, this is of course how it is managed. > flexible and is useful during experimental/early developments, but it is not > "scalable" for "quick launch" use case (ie. other developer checking out a > project and want to run it with experience like deploying a plain WAR). We have just created a bug 374686 to show the Virgo Servers file system directly from w/in Eclipse. I think this is a great example of obvious low hanging fruit that would make day to day usage so much nicer; not having to go back and forth to native file system. One thought I had is that this could be extended so that you could actually DnD stuff from package/project explorer -- such as web artifacts -- into the target server directory. That made me think, why not provide the ability to drag Eclipse projects directly into the server? That could then (given user preferences) actually deploy all of the plugin dependencies for that PDE object as well. + or -? Parenthetically, I don't know about other Web Server projects, but in Virgo we have a lot of stuff that is buried in code that could just as well be scripted -- perhaps in (ugh, ant) or a nice little DSL-- for extension, etc.. It would be nice to have a generic model for this. With something like that, people could define their own deployment scenarios easily. > With this functionality, Saving a file should trigger a build and redeploy of > the affected bundle. No Virgo restart is necessary and this works without > needing to use Debug (it's OSGi anyway). Right! That's such a neat key feature of Virgo, and one of the things we want to make transparant. > If this functionality is working, after that we can move on to more complex > functionality like feature/plan support. But I think we should get the most > basic stuff working first. You make a pretty convincing argument. + or -? (In reply to comment #17) > Miles, it would make sense if you take a look at the server adapters for > Equinox, Felix and Knopflerfish contributed to the Libra project. These are the > org.eclipse.libra.framework.* packages. Will do, again w/ the caveat that I see a key requirement of any solution not requiring changes to existing projects. + or - ? (In reply to comment #18) > in my view most critical bit would be working with two (possibly conflicting) > "target platforms", i.e. PDE uses target platform definitions for dependency > resolution and Virgo uses target runtimes (a la WTP) for bundle dependencies. OK, we need to think about what the real key differences are here and whether they canbe bridged. > Working with multiple target platforms in the same application would be riddled > with problems for users. For example if one adds a PDE bundle to a Virgo server, > the "publishing" job maybe required to resolve dependencies from two > conflicting targets, which may not be solvable problem if there are conflicts. I don't think this is insurmountable issue, really. It is really no different than the current RCP situation right? In a Target Platform deployment secenario there would always be one and only one target platform for a given virgo server. That could be the development environment Target Platform, or another one. I'm not saying that RCP deployment isn't painful, but every PDE developer is quite familiar with the scenario of having something that builds perfectly well in your local workspace, but then when you go to export it, you spend hours (days) trying to get the product dependencies right so that PDE can actually build a working app. That just goes w/ the RCP / App deployment territory, and when you do RAP deployments, you get to have that fun again. Virgo will actually be a lot easier because we won't have to deal with all of the native bundle stuff, etc.. > The best method I can imagine is to expose the Virgo Target Runtime bundles as a > Target Platform so that Virgo Bundle Projects and PDE Bundle projects use the > same set of bundles. Interesting idea and I could see it's usefulness, but I think it's actually backward from a PDE developer's POV. Here, you might want to create a target Runtime that can incorporate a PDE Target Platform. > We are experimenting with a similar idea for Libra launchers by creating a > distinct target platform definition for each framework instance. This would What do you mean by "framework instance" in this context? > remove the multiple bundle sources issue. However, working with Target Platform > definitions the PDE way is not good from the user experience point of view. WTP > target platform concept is much better way to handle the problem. To be clear, this task takes the the PDE developer's POV, so for that reason, the PDE way is by definition the best user experience. :D We could have another task for "the best possible way to deploy web and PDE bundles in a generic way". > A nice solution could be the ability add a target runtime into a PDE target > platform definition and delegate the bundle resolution to Target Runtimes. PDE > Projects would not changed (no new natures builders facets etc.) I like that part. Again, I think we may be coming to this from somewhat different angles. I don't envision the typical PDE developer wanting to consume web target resources; rather I think of them as wanting to be able to consume PDE bundles from within web targets. + or -? > Libra may not be a bad place to build this support. Yes, agreed that Libra seems to be the place where a lot of this kind of stuff should happen. I'n going to send an email to the list suggesting we discuss how to carve all of this up.
One generic comment, as I realized that I might be creating an impression of not caring about what is important to a lot of users or developers. So first, please excuse my biases toward PDE approach in bug; part of that is probably just the old "when you've been using a hammer, everything looks like a nail" problem. I thought I should be clear about what what's motivating my emphasis about this bug, at it may not be obvious from those who have a more web-centric POV. Virgo is a great deployment environment for pure PDE developers who are creating apps that don't need / want all of the Eclipse Platform dependencies. Just as an example, I might want to create a service that takes in a bunch of EMF models and does some kind of batch changes to it. In these scenarios, there are *no* web dependencies at all. So I see this bug as about supporting *those users* in the absolutely most transparent, low impact way possible. Does that fit with anyone else's view? That may not be what other people had in mind with this bug, in which case I wasn't being flip when I suggested that perhaps we should have another broader bug for the "integrating PDE bundles with WTP deployment tools" bug.
(In reply to comment #19) > > However, these bundles may depend on packages or use services that are provided > > by Eclipse plug-ins. The dependency plug-ins may be in the workspace or may be > > specified via target platform, e.g. from the Orbit p2 repository. Another > > possibility is that some bundles can be provided via OBR repository. > > From what little I know on this, the OBR repository is not planned to be > maintained. So this is a really good reason to begin to think about > transitioning that all to Orbit. I think there's a bit of confusion here. Perhaps what was meant in comment 15 was the forthcoming specification from the OSGi Alliance which defines an interface to a bundle repository. Hendy: please could you clarify? (I got the impression that Miles might have been thinking of the SpringSource Enterprise Bundle Repository (EBR) which is not being very actively maintained. I am investigating community alternatives with some other vendors, hoping to bring a proposal forward before long. Orbit is *not* a suitable replacement for this as it only includes Eclipse IP approved bundles and a community bundle repository would include (much) more. But when we have a community bundle repository up and running, it would be great if the tooling could integrate with it.)
Re: comment #19 and #21 Yes I meant the OBR repository *specification*, not the Eclipse OBR repository or any specific physical location. Forgive me for ever mentioning OBR. I think it's outside the scope of this bug although perhaps someday we can think about OBR integration, however at this point it's pretty much premature. For those curious, OBR is a somewhat direct competitor to p2. It allows OSGi servers, when a bundle is deployed, to search and download matching dependencies automatically by matching Import-Package and its versions. However Virgo won't support OBR at least in the near future ( https://bugs.eclipse.org/bugs/show_bug.cgi?id=350839#c3 ) Re: Target Platform and PDE, I fully support this. PDE plug-ins (that "doubles" as a Virgo bundle) should work just like PDE plug-ins. They can be deployed to a Virgo instance, and will act like an OSGi bundle. I think it's the developer's job to make sure that the Virgo instance environment has the needed dependencies of the deployed bundle(s), in the same way that an RCP developer must make sure the plug-ins have all its dependencies satisfied. Virgo Tooling can provide aid if, for example, a user deploys a plug-in/bundle that has a dependency (Import-Package/Require-Bundle) that can not be satisfied by the target instance. But this should not be mandatory, ie. should be possible to run/launch a "broken" Virgo instance. (It's the OSGi spirit anyway, bundles/services can come and go anytime) Overall, I agree that the "development" target platform/definition (used for JDT completion, build etc.) should be separate from the Virgo instance(s) used for running & debugging.
(In reply to comment #22) > Re: comment #19 and #21 > > Yes I meant the OBR repository *specification*, not the Eclipse OBR repository > or any specific physical location. Thanks for the clarification. This spec is part of the forthcoming OSGi Enterprise R5 spec and has actually morphed into somewhat separate Repository and Resolver service spec chapters. At least a final draft should be available for EclipseCon next week. > > Forgive me for ever mentioning OBR. I think it's outside the scope of this bug > although perhaps someday we can think about OBR integration, however at this > point it's pretty much premature. Thanks for mentioning it. It's something we should keep in mind. If there is enough user demand for it, perhaps someone will step forward and provide an implementation for Virgo. ;-) > > For those curious, OBR is a somewhat direct competitor to p2. It allows OSGi > servers, when a bundle is deployed, to search and download matching > dependencies automatically by matching Import-Package and its versions. However > Virgo won't support OBR at least in the near future ( > https://bugs.eclipse.org/bugs/show_bug.cgi?id=350839#c3 ) Correct.
I've cast my non-binding +1 for this bug a few months ago but since that time I've learned that PDE is not the only and by far not the easiest way to develop OSGi bundles in Eclipse. My current toolset of of choice is: * Bndtools http://bndtools.org/ * m2e http://eclipse.org/m2e/ with Eclipse Tycho connector * Maven Bundle Plugin http://felix.apache.org/site/apache-felix-maven-bundle-plugin-bnd.html * PAX Exam http://team.ops4j.org/wiki/display/paxexam/Pax+Exam Some benefits of this setup: - tedious parts of manifests macro-generated by BND - compatible builds with Eclipse and commandline Maven - dependency resolution from workspace projects and local and remote OBR repositories - framework startup time < 1 second (no kidding) - instant save / compile / bundle refresh cycle All of this is working quite nicely today, and it's only going to get better. I need to experiment a bit more with Tycho but it seems that one can generate P2 repository with features built from bundles created by BND / maven-bundle-plugin. Then one could provision Virgo Nano from such P2 repository in production environment to take benefit of it's serviceability features. End-to-end awesomeness :)
(In reply to comment #24) > I've cast my non-binding +1 for this bug a few months ago but since that time > I've learned that PDE is not the only and by far not the easiest way to develop > OSGi bundles in Eclipse. > > My current toolset of of choice is: > * Bndtools http://bndtools.org/ > * m2e http://eclipse.org/m2e/ with Eclipse Tycho connector > * Maven Bundle Plugin > http://felix.apache.org/site/apache-felix-maven-bundle-plugin-bnd.html > * PAX Exam http://team.ops4j.org/wiki/display/paxexam/Pax+Exam > > Some benefits of this setup: > - tedious parts of manifests macro-generated by BND > - compatible builds with Eclipse and commandline Maven > - dependency resolution from workspace projects and local and remote OBR > repositories > - framework startup time < 1 second (no kidding) > - instant save / compile / bundle refresh cycle > All of this is working quite nicely today, and it's only going to get better. > > I need to experiment a bit more with Tycho but it seems that one can generate > P2 repository with features built from bundles created by BND / > maven-bundle-plugin. Then one could provision Virgo Nano from such P2 > repository in production environment to take benefit of it's serviceability > features. End-to-end awesomeness :) I agree that bndtools and PAX Exam are excellent tools and I would love to see explicit Virgo support appear in these tools. I would also love to see the Virgo tooling give those tools some strong competition and I think the Virgo tooling is definitely ahead in terms of explicit Virgo support and some of the advanced features. Please note that Virgo bundlor is approximately equivalents to bnd - you simply maintain a template and bundlor does bytecode analysis to generate the bundle manifest. There is also a maven bundlor plugin.
(In reply to comment #25) > Please note that Virgo bundlor is approximately equivalents to bnd - you simply > maintain a template and bundlor does bytecode analysis to generate the bundle > manifest. There is also a maven bundlor plugin. My experience of Bundlor has been universally bad. It generally seems to put in exact version ranges (like [1.2.3.2012,1.2.3.2012] which essentially prevents the runtime framework from being able to use upgraded versions of the bundles when installed. (The same problem is also visible in some of the EBR springsource components). In fact, most of my problems with Virgo and OSGi development have been due to a combination of bundlor and the user region re-writing bundle imports (especially package imports) leading to applications which otherwise would be compatible together leading to not working when installed in the user region. I gave up on both bundlor and Virgo's user region some time ago, and get by by installing items into the kernel region only and life has generally become much better.
(In reply to comment #26) > (In reply to comment #25) > > Please note that Virgo bundlor is approximately equivalents to bnd - you simply > > maintain a template and bundlor does bytecode analysis to generate the bundle > > manifest. There is also a maven bundlor plugin. > > My experience of Bundlor has been universally bad. It generally seems to put in > exact version ranges (like [1.2.3.2012,1.2.3.2012] which essentially prevents > the runtime framework from being able to use upgraded versions of the bundles > when installed. (The same problem is also visible in some of the EBR > springsource components). That's the conservative default, but a different policy can easily be specified in the template as in the following example from Virgo's kernel core bundle: Import-Template: org.eclipse.virgo.util.*;version="${org.eclipse.virgo.util:[=.=.=, =.+1)}" When the variable org.eclipse.virgo.util is set to 3.0.0.RELEASE, this produces the version range [3.0.0.RELEASE, 3.1). > > In fact, most of my problems with Virgo and OSGi development have been due to a > combination of bundlor and the user region re-writing bundle imports > (especially package imports) leading to applications which otherwise would be > compatible together leading to not working when installed in the user region. Imports are only rewritten when bundles are included in a PAR or a scoped plan. So they can be avoided completely by using unscoped plans. However, the rewriting really shouldn't cause the kinds of problems you describe, so if you still have specific examples to hand, it would be very helpful if you could raise a bug. > > I gave up on both bundlor and Virgo's user region some time ago, and get by by > installing items into the kernel region only and life has generally become much > better. Sorry to hear that. Have you looked at Virgo nano as it seems to fit your current usage very nicely? What tooling have you been using with the kernel as the Virgo IDE tooling is really designed for deploying to the user region? If you have used bndtools, for example, with Virgo it would be great if you could share your experience.
(In reply to comment #21) > (I got the impression that Miles might have been thinking of the SpringSource > Enterprise Bundle Repository (EBR) which is not being very actively maintained. Your impression is exactly correct. > repository would include (much) more. But when we have a community bundle > repository up and running, it would be great if the tooling could integrate > with it.) Yes, and that should be easy. We have not removed support for EBR, so all of that works now, and hopefully will not much more than a matter of redirecting the URL. (Which shouldn't be hard-coded as I believe it is now anyway.)
(In reply to comment #24) > I've cast my non-binding +1 for this bug a few months ago but since that time > I've learned that PDE is not the only and by far not the easiest way to develop > OSGi bundles in Eclipse. OK, again, when the Eclipse environment and community chooses a different build system than PDE we'll look at supporting that. :D Editorially, I agree that PDE is far from ideal and requires far too much care and feeding and artefact maintenance, the current trend toward basing builds on self-evolving, ad hoc and marginal quality (m2e) build systems is frightening form the perspective of building software community where the most important thing is not that we have the absolutely best build system imaginable but that everyone is able to build from the same set of common artifacts without learning new tooling.
(In reply to comment #27) > > My experience of Bundlor has been universally bad. It generally seems to put in > > exact version ranges (like [1.2.3.2012,1.2.3.2012] which essentially prevents > > the runtime framework from being able to use upgraded versions of the bundles > > when installed. (The same problem is also visible in some of the EBR > > springsource components). > > That's the conservative default, but a different policy can easily be specified > in the template as in the following example from Virgo's kernel core bundle: And note btw that that is no different than the PDE Manifest editor, which does exactly the same thing with exactly the same consequences. At least with bundlor you can change the default. But if this is the major negative of bundlor, Virgo and/or Libra could look at relaxing that; raise a bug if that's what you'd like to see.
(In reply to comment #29) > the current trend toward basing builds on > self-evolving, ad hoc and marginal quality (m2e) build systems is frightening > form the perspective of building software community where the most important > thing is not that we have the absolutely best build system imaginable but that > everyone is able to build from the same set of common artifacts without > learning new tooling. Well, this sounds quite a lot like Maven promise "builds repeatable by anyone, anywhere, anytime". In my own personal experience Maven lives up to that promise, as long as you stick with the best practices. On the other hand, if you can't apply the best practices because your are converting a legacy application with a host of "special case" requirements, on a tight schedule and limited budget... things are going to get ugly. Regarding m2e, it was awfully quirky before version 1.0 but since then it is rock solid. The flip side is that support for non-standard Maven plugins that "somewhat worked" before stopped working completely with the option to patch it back it on plugin-by-plugin basis through well defined extension mechanism. This angered and frustrated many users, who don't have time or expertise to work on the required extensions, or simply did not understand the whole problem. But I digress. Irregardless of my personal preference for Maven and related tools, I appreciate that as long as PDE is the standard tool for OSGi development in Eclipse ecosystem, Virgo tooling should make integration with it a priority over other options available.
(In reply to comment #31) > Well, this sounds quite a lot like Maven promise "builds repeatable by anyone, > anywhere, anytime". In my own personal experience Maven lives up to that > promise, as long as you stick with the best practices. On the other hand, if > you can't apply the best practices because your are converting a legacy > application with a host of "special case" requirements, on a tight schedule and > limited budget... things are going to get ugly. Yes, and again speaking editorially and OT, this is simply the fundamental tradeoff in any complex system between leverage and configurability. I think Maven's approach is perfectly sound, but unless everyone is doing it things fall apart. Personally, I think that Buckminster provides much more control over the build process as a whole, while still providing the high-level constructs that make it relatively easy to work with. But it seems that Buckminster hasn't become the blessed approach for managing large scale builds. > Regarding m2e, it was awfully quirky before version 1.0 but since then it is > rock solid. I think it's still got some maintenance issues; silly things like the fact that it is still using Usage Data Collector, but they do affect overall usability and perception. But my major concern w/ Maven per se is the introduction of new non-standard artifacts into builds. I can understand *why* this has happened, it's because there aren't appropriate first-class artifacts for important aspects of builds defined by PDE and so people have had to come up with solutions to fill the gaps. But there has to be a better solution than hand-maintaining POMs / obscure xml. The flip side is that support for non-standard Maven plugins that > "somewhat worked" before stopped working completely with the option to patch it > back it on plugin-by-plugin basis through well defined extension mechanism. > This angered and frustrated many users, who don't have time or expertise to > work on the required extensions, or simply did not understand the whole > problem. I think the dominant example is in green-pages, when M2E complains: "Description Resource Path Location Type maven-dependency-plugin (goals "copy-dependencies", "unpack") is not supported by m2e. pom.xml /greenpages-solution line 69 Maven Project Build Lifecycle Mapping Problem" And it simply isn't clear what to do with that. OK, all totally off-topic now and not the official view point of Virgo Tooling, its staff, or sponsors. ;D
(In reply to comment #20) > One generic comment, as I realized that I might be creating an impression of > not caring about what is important to a lot of users or developers. So first, > please excuse my biases toward PDE approach in bug; part of that is probably > just the old "when you've been using a hammer, everything looks like a nail" > problem. > > I thought I should be clear about what what's motivating my emphasis about this > bug, at it may not be obvious from those who have a more web-centric POV. Virgo > is a great deployment environment for pure PDE developers who are creating apps > that don't need / want all of the Eclipse Platform dependencies. Just as an > example, I might want to create a service that takes in a bunch of EMF models > and does some kind of batch changes to it. In these scenarios, there are *no* > web dependencies at all. So I see this bug as about supporting *those users* in > the absolutely most transparent, low impact way possible. Does that fit with > anyone else's view? > > That may not be what other people had in mind with this bug, in which case I > wasn't being flip when I suggested that perhaps we should have another broader > bug for the "integrating PDE bundles with WTP deployment tools" bug. Miles, your POV is absolutely valid. But, of course, it's not the only one among the interested people in this bug. Some of the adopters of the Libra project have the web-centric POV - providing web app developers with capabilities for using OSGi. These developers most likely have not any previous experience with PDE. While they understand OSGi fundamentals like the Manifest and Declarative Services, they get easily confused with PDE artifacts like target platforms, build.properties, etc. Nevertheless, the improvements targeted in this bug will be an important basis for future improvements for those web app developers.
(In reply to comment #31) > Well, this sounds quite a lot like Maven promise "builds repeatable by anyone, > anywhere, anytime". In my own personal experience Maven lives up to that > promise, as long as you stick with the best practices. Hi Rafal This is off topic, but it's important and it at least relates to Virgo, so please bear with me. When we tried using Maven back near the beginning of Virgo (SpringSource dm Server actually), we couldn't get repeatable builds in spite of having plenty of Java expertise on the team. Perhaps the issue was that the best practices for Maven were either not clearly defined and documented or that we somehow missed them. So do you happen to have a link to the best practices? If you prefer not to pollute this bug, please email gnormington@vmware.com. Thanks! Regards, Glyn
(In reply to comment #34) > When we tried using Maven back near the beginning of Virgo (SpringSource dm > Server actually), we couldn't get repeatable builds in spite of having plenty > of Java expertise on the team. I suspect this is a matter of not working the way Maven expects people to work, rather than people having Java experience. You've mentioned this a few times, and it's not clear to me how you managed to come up with a 'non-repeatable build' in Maven. If you are trying to use Maven to build against JARs checked into a file system, You're Doing It Wrong(TM). Even Gradle, widely seen as the successor to Ant/Ivy, has seen the light and discourages referring to assets by location in a file system and instead uses standard Maven GAV co-ordinates to build the code. > Perhaps the issue was that the best practices > for Maven were either not clearly defined and documented or that we somehow > missed them. > > So do you happen to have a link to the best practices? The getting started document shows you how to get started; nothing about it documents non-best practices. http://maven.apache.org/guides/getting-started/index.html The only thing that I could possibly suggest would be to set up a continuous integration server that cleans the local maven repository (typically ~/.m2/repository) so that any build products that are 'install'ed to the local file system won't affect subsequent builds on the same hosts. Also, it is generally the case that a repository solution (such as Nexus) is used and at the end of the build the artefacts are published into the repository for others to consume. Perhaps the issue is you are thinking something is 'non-repeatable' without defining why you think that? Alex
(In reply to comment #35) > Perhaps the issue is you are thinking something is 'non-repeatable' without > defining why you think that? This was about 4 years ago, but one of the issues I remember was that the built artefacts seemed to vary depending on whether the building machine (which was a laptop) was connected to the internet. At the time we were doing a lot of travelling and developing concurrently, so we wanted repeatability when disconnected from the internet. We presumably did something which is not considered best practice, but I don't know what. If we ever try again, which is admittedly unlikely, it would be good to know what best practice is ahead of time.
(In reply to comment #34) > When we tried using Maven back near the beginning of Virgo (SpringSource dm > Server actually), we couldn't get repeatable builds in spite of having plenty > of Java expertise on the team. Perhaps the issue was that the best practices > for Maven were either not clearly defined and documented or that we somehow > missed them. > > So do you happen to have a link to the best practices? I've posted my reply in Virgo forum: http://www.eclipse.org/forums/index.php/m/827132/#msg_827132
OK, as if we needed one, there is one more wrinkle to add to the mix. There is this exciting possibility of provisioning Virgo directly from p2.* In this case, we could imagine the Tooling as simply being a front-end for that provisioning process. Then for deployment, we'd use the PDE tools to create a local P2 site that would provision the workspace build itself. You would then: 1. Build the workspace. 2. Export/update workspace P2 Artifacts 3. Trigger P2 update from Virgo. In this case we'd either 1) wait for Virgo Runtime to provide these features and then support them in tooling or 2) Develop some basic functionality on the tooling side for later possible use in Runtime. Compared to the other simple options, this is probably a non-strarter for now, *unless* there has already been work done on the runtime side. Boris, Glyn, comments? *See: bug 343543 bug 309778 http://bkapukaranov.wordpress.com/2011/07/12/rt-meets-p2/
Hey Miles, Indeed every Virgo flavor can be provisioned directly from p2 via the available update site(http://download.eclipse.org/virgo/updatesite/3.5.0.M03). As the tooling lives in Eclipse along with the p2 infrastructure it can leverage that and access/trigger the p2.director application. This should make it fairly easy to provision a Virgo server somewhere on the filesystem. There are at least two(maybe more) scenarios of how the tooling-server interaction can work. 1. Use the Virgo update site and the PDE mechanics thus creating effectively a runnable target platform that is equal to a Virgo server and run bundles on top of it using the already existing PDE stuff. I've seen this in action(not with Virgo) and personally don't like it. Firstly because bug 351417 means that the created target platform cannot be equal to an actual server - the configure phase is very important. Another downfall is with this approach the server is provisioned in the Eclipse workspace which is bad for users - they can't manage it easily. Secondly this approach won't work with Virgo's regions because Eclipse and respectively PDE just doesn't support the concept. 2. Use the Virgo update site to provision a Virgo instance somewhere on the filesystem. From then on it's business as usual - deployment via mbeans or copying in pickup or whatever. This is my preferred approach as it will work consistenlty for all Virgo flavors and also ensures that no matter if it's managed by the IDE or standalone the runtime users will play with be exactly the same which I believe is extremely valuable. That's for the provisioning and deployment part. What happens when users develop new bundles? PDE dictates that all PDE bundles are built against a target platform that includes all workspace PDE projects + (by default)the eclipse/plugins directory. For Virgo we need something similar. Here again we need to take into account that just using the Virgo update site to create a target platform won't work as it only exposes kernel region stuff directly while users will often develop only against the user region. I'll divide this into two cases: 1. Nano/Nano Full - Here it's easy to create a target platform - either take /plugins and /lib and create a target platform around them or use the update site - Nano has a single region so the update site content is the same as the visible runtime content. 2. VK/VTS/VJS - taking the update site as a target platform won't work. Also taking the repositories and building the target platform on top of them won't be sufficient as there are shared packages/services from the kernel to the user region so it's a bit more complex. I imagine it would be best if the tooling could process the Virgo regions configuration and build the target platform with respect to it, probably including there stuff from /repository and /plugins, thus giving users equal build and runtime environment. This would be quite difficult to achieve so a first step might be to build only agains the stuff in /repository - this will provide a bit limited but safe experience to the users, safeguarding them from accessing something that is not visible. Best Regards, Bobby
(In reply to comment #39) > 1. Use the Virgo update site and the PDE mechanics thus creating effectively a > runnable target platform that is equal to a Virgo server and run bundles on top > of it using the already existing PDE stuff. I've seen this in action(not with > Virgo) and **personally don't like it**. OK, I stopped there. You know far more about these issues than I. > 2. Use the Virgo update site to provision a Virgo instance somewhere on the > filesystem. From then on it's business as usual - deployment via means or To be clear, I don't care about the initial provisioning step. Here I'm assuming that we already have a fully functioning kernel / Tomcat install. > That's for the provisioning and deployment part. What happens when users > develop new bundles? > PDE dictates that all PDE bundles are built against a target platform that > includes all workspace PDE projects + (by default)the eclipse/plugins > directory. And actually, best practice is *not* to include the eclipse/plugins directory. In fact, I regard having that in the TP as damage (something that ironically we currently do need to do in Virgo). > 2. VK/VTS/VJS - taking the update site as a target platform won't work. Also > taking the repositories and building the target platform on top of them won't > be sufficient as there are shared packages/services from the kernel to the user > region so it's a bit more complex. I imagine it would be best if the tooling > could process the Virgo regions configuration and build the target platform > with respect to it, probably including there stuff from /repository and > /plugins, thus giving users equal build and runtime environment. This would be > quite difficult to achieve so a first step might be to build only agains the > stuff in /repository - this will provide a bit limited but safe experience to > the users, safeguarding them from accessing something that is not visible. And if I'm reading what you're saying correctly, I think this is actually the *only* thing I'd like to support. In a typical base PDE setup, what you'd want is for people to be able to simply: 1. Define a complete target platform that would include all needed features / plugins, regardless of deployment platform. 2. Add in workspace plugins / features as appropriate. So yes, both Tooling and Runtime would share the TP for most development scenarios, though you'd need to support defining different TPs for target runtimes. But they could certainly be provisioned in separate steps, and in fact should be, just as is done for an RCP app. So all this means is that we need to be able to have the runtime take care of fully provisioning the TP is valid on the runtime side. We don't actually care *how* that happens, e.g. where the various artifacts end up. In this scenario, we might go the extra step of allowing the runtime to *pull* its dependencies from a Tooling maintained P2 site, or we might simply have the Tooling *push* the extra workspace bits to the Runtime, e.g. pickup. I think the second option would be far simpler and more performant (and necessary for debugging support??), but the first would be a much cleaner option. I guess this is analogous to the distinction between running a live hosted runtime and a built RCP app. So I think we'd look at supporting the second option initially. And that would be complimentary with the simple publish plugin approach that we've discussed. If we could get this working somewhat easily, I think we'd have a great solution that would expand capabilities quite a bit beyond the PDE scenario to generic provisioning support and configuration.
Hi, here is a proposal: leverage Equinox support for "external" classpath entries in the MANIFEST.MF. Equinox allows declaring external entries in a bundle classpath. This is obtained by prefixing the entry with "external:" and using a URL to point the classpath entry, for example: Bundle-ClassPath: external:file:///home/user/workspace/PDEPluginProject/bin, external:file:///tmp/some.jar By using such a feature, Virgo tools could allow developers to add to a server instance both Virgo OSGi Bundle projects and PDE Plug-in projects. Then, when publishing takes place, instead of copying the bundles into VIRGO_HOME/stage folder, the tools could create a bundle folder for each project in VIRGO_HOME/stage and generate on the fly a MANIFEST.MF file for each project, which is obtained from the original MANIFEST.MF file in the workspace, by replacing the original classpath entries with derived "external" entries. In this way (1) publishing would be faster, and (2) it should be possible to easily deploy both the PDE plug-ins and the OSGi bundle projects to a Virgo Server instance within Virgo Tools. It remains the responsibility of the developer who wants to develop for Virgo using PDE to setup the PDE target platform that points to the Virgo bundle repositories.
To be able to use PDE bundles with Virgo I wrote a simple eclipse plug-in that contributes a nature and a builder. Now I can create PDE plug-in projects and the I add them the Virgo nature as well as my own nature and builder via a popup menu action. I don't enable the Virgo classpath container anymore, as PDE will provide its "Plugin dependencies" container. Then, when I build the project my builder copies META-INF folder from project root to /bin and similarly it parses MANIFEST-MF to copy to bin all the libraries listed in Bundle-ClassPath. So far it's working fine. There are a couple of little bugs to sort out but my colleagues (who were already used to working with PDE) are now happily developing for Virgo using PDE tools. If I manage to sort out the remaining issues I'll probably publish the sources to github and post a comment here with the link.
GianMaria, Interesting approach. It should be possible to do this without the builders as well, by pointing the external entries to the Eclipse working build. Overall though, this sounds like a solution for development time, not final deployment..? Miles
Hi Miles, sorry for my late reply, I was on vacation in the last two weeks. I am not sure I understood your comment/suggestion. So far, the approach I described in my last comment (#42) is working fine for both development and deployment. At development time, the fact that the PDE Plug-in project also feature the Virgo OSGi Bundle nature allows adding them to a Virgo Server instances in Virgo Tools, and thanks to my builder which copies META-INF/ and nested JARs to /bin the server starts properly. To deploy binaries I just have to export projects as Deployable plug-ins and fragments using the PDE wizard and the resulting bundles will work perfectly in Virgo. In fact, our headless build system is based on Eclipse PDE Build and the resulting binaries can be deployed to Virgo and work perfectly (we manually maintain a Virgo plan file). I really don't understand what you meant when you said "do this without the builders as well, by pointing the external entries to the Eclipse working build." If you refer to the possibility of using the Eclipse virtual file system to create a link to the META-INF folder in /src so that the eclipse builder would copy it to /bin, this is an approach I tried, but I eventually dropped it because the link definition existed only in my .metadata/ folder and could not be easily shared using standard Eclipse team support.
We had to focus on the core Virgo issues so haven't been able to get to this one. Sorry. :( Contributions are of course welcome and I'm sure this will be on the top of the list for further work.
Hi all, three months after my original comment #42, I was finally able to clean up a bit the code and write the documentation for my plug-in that enables using PDE projects with Virgo. Sources, binaries and documentation available here: https://github.com/giamma/pde2virgo/
(In reply to comment #46) > Hi all, > > three months after my original comment #42, I was finally able to clean up a > bit the code and write the documentation for my plug-in that enables using > PDE projects with Virgo. > > Sources, binaries and documentation available here: > > https://github.com/giamma/pde2virgo/ Fantastic! Would you like to write a brief blog about this which we could use to publicise your work (and link to from the Virgo home page)? Meanwhile, I'll tweet and add a link to the tooling section of the Virgo wiki.
Hi Glyn, thank you very much for such a warm welcome! I'll be happy to write a blog post, I hope to be able to do that over the week end.
(In reply to comment #46) > Hi all, > > three months after my original comment #42, I was finally able to clean up a > bit the code and write the documentation for my plug-in that enables using > PDE projects with Virgo. > > Sources, binaries and documentation available here: > > https://github.com/giamma/pde2virgo/ Excellent! I look forward to taking a look as soon as I can (which won't be as soon as I'd like. :))
The master branch and 1.5 snapshot builds for the tooling now include support for PDE consisting in the following: - the runtime wizard allows the user to specify that he/she wants to develop for Virgo using PDE. If such selection is made, the wizard will read the Virgo repository properties file and create a PDE target platform definition named after the Virgo Runtime instance and set it active. The user can also edit the list of folders before finishing the wizard, and changes will be applied to the repository properties file as well. - a new wizard exists in the Virgo category for creating PDE plug-in projects that are enhanced with a custom nature and builder. The wizard also allows the user to specify a Web context path, in which case the created PDE plug-in project becomes a WAB (the wizard extends and inherits most behaviour from a public PDE abstract wizard). - the Server Editor includes a new section in the first page to quickly refresh a PDE target platform definition (useful when new bundles are added) or to recall the editing screen. In addition: - A decorator is contributed that adds a tiny rt sphere decoration to the top left of PDE plug-in projects created for Virgo - A similar decorator adds the same decoration to server instances in the Server View if a PDE target platform definition exists that is named equally to the Server runtime This feature has been tested with Virgo 3.6.x and current 3.7.
