Community
Participate
Working Groups
p2 provides a complete framework for publishing root files with a feature but it turns out it does not provide a default implementation of the org.eclipse.equinox.p2.publisher.actions.IFeatureRootAdvice This puts each tool that use the publisher responsible for providing such implementation: - here is buckminster's: http://dev.eclipse.org/viewsvn/index.cgi/trunk/org.eclipse.buckminster.pde/src/java/org/eclipse/buckminster/pde/tasks/FeatureRootAdvice.java?root=Tools_BUCKMINSTER&view=markup - there is another one very similar in PDEBuild. - here is tycho's bug about this: https://issues.sonatype.org/browse/TYCHO-465 Jeff and I discussed this briefly and in fact there might have been such implementation at some point: it seems it disappeared.
The removal of the original implementation was bug 278086. The key point is that the build.properties files and the root.* properties themselves are, as mentioned in the tycho bug, source artifacts that are "proprietary pde.build". I don't really know what a p2 publisher implementation of this interface would look like without tying it to a specific build technology which is better left implemented by that builder. As a side note, these properties are originally implemented by running generated ant scripts and as such support all the normal ant wildcards ("**/*.exe" etc). PDE/Build is using ant classes to expand these. This is missing from all the other implementations I've seen (along with support for "absolute:" as well as the "root.[<config>].folder.<path>" properties).
As you point out, the root files mechanism is subtle and reasonably complex to implement. Perhaps we can find a way to factor out the PDE implementation such that it can be used by other build systems. Short of the use of Ant for filesets, there does not seem to be much PDE-specific about PDE's root file support. Doing this would help these other communities "get it right" as it were. As for publishing, it has always bothered me that features (and products) cannot just be "published", they need to be "built". There certainly are some technical hurdles but from an end user perspective, the people who create features and products are often quite different from the folks who create the bundles etc. For them, getting and setting up PDE build is a daunting task. It has high apparent complexity. It should be dead easy for a system integrator to walk up to a repo and create features and products, get them into the repo and start deploying. It may be that only a restricted set of a feature's full glory would be supported by this mechanism. How many people use complex Ant patterns in their root file specs? How many have code in their features? Pretty much 100% of the cases we see in the wild don't do anything like this. It also may well be that features are just the wrong mechanism for doing root files. It does feel like an odd combination. For example, a built feature (JAR) has no root files or build.properties. We often talk about built and un-built features in the same way when really they are quite different. I think this was one of the causes of the apparent problems in the original feature publishing action. In any event, the end users are suffering. IMHO we either need to make it butt-simple for people to get, configure and run PDE Build to go from source feature/product to stuff in the repo or provide some publishing actions that hook it together so they just run the publisher.
The question that should be asked is what prevents others from using the PDE implementation. After all, the whole concept was invented by PDE. The reason Buckminster has its own implementation is a historical bug workaround. We can probably revert to using the PDE version. In any case, I don't think this type of functionality belongs in p2. Instead, I think p2 should be kept mean and lean and free from build concerns. My preference would be to move in the opposite direction and say that everything that has anything to do with features, bundles, etc. is in the PDE domain. p2 is a generic provisioning system and it should not contain tooling for any specific build flavor. In fact, I don't think it should contain build related stuff at all. A low-level, well defined API to allow pre-built artifacts to be published is motivated in p2 but that API should be restricted to publish complete IU's and pre-built artifacts. Advices, p2.inf and build.properties files, management of root files, all of that belongs in the build domain and the current fuzzy separation of concern needs an overhaul.
Publishing necessarily straddles the line between pure "build" and pure "provisioning". It is the act of taking some bytes, creating metadata that describes the bytes in terms that p2 understands and putting the bytes in a repo in a way that p2 understands. It is quite rare that a given artifact or artifact set (bundles and features excepted) is completely self-describing such that all information needed by p2 is inherently available. As such, the publisher needs additional information telling it what to do. That is, advice and p2.inf etc The usecases being pursued here involve the assembly of pre-existing components stored in p2 repos to form products. To do this system integrators need to create features and .product files, get the corresponding artifacts (e.g., root files) and metadata into a p2 repo and then deploy. The workflows for doing this currently are complicated by any number of "build" concerns that are not relevant to the system integrator role. Comment 0 is really saying, "if we could use the publisher to publish root files then our workflow would be a lot easier". Their use-case is Maven/Tycho based. Mine does not involve a build system at all. While I fully support that idea, the real goal IMHO is the simplified workflow. If someone could give say a handful of simple info (feature, repos, ...) and run something to get the job done, this would be an 80% solution. Whether this publishing functionality lives in p2 is not really the point. The point is that it is currently hard to use and requires having PDE, Resources, JDT, update configurator, ... around. Summary: We agree that if the PDE implementation of IRootFileAdvice were easily usable by others (i.e., not tied to PDE), it would be "a good thing" and would enable better workflows, lets duplicate code and more consistency in output.
Jeff, (In reply to comment #4) > > Comment 0 is really saying, "if we could use the publisher to publish root > files then our workflow would be a lot easier". Their use-case is Maven/Tycho > based. Mine does not involve a build system at all. While I fully support > that idea, the real goal IMHO is the simplified workflow. If someone could give > say a handful of simple info (feature, repos, ...) and run something to get the > job done, this would be an 80% solution. Whether this publishing functionality > lives in p2 is not really the point. The point is that it is currently hard to > use and requires having PDE, Resources, JDT, update configurator, ... around. > > Summary: We agree that if the PDE implementation of IRootFileAdvice were easily > usable by others (i.e., not tied to PDE), it would be "a good thing" and would > enable better workflows, lets duplicate code and more consistency in output. That is exactly what I was looking for when I worked on Tycho for the jetty build: a small packaging of the publisher that can be invoked from the command-line. To be able to access all the functionality needed, I ended-up extending the publisher application and exposing various other operations. A couple of pointers to the code shows how the community would do a much better job out of it than a humble build-tool developer under pressure to demo the benefits of adopting p2 ;) The new Actions, Advice, Applications: http://github.com/hmalphettes/sonatype-tycho/tree/p2/tycho-p2/org.sonatype.tycho.p2.publisher/src/main/java/org/sonatype/tycho/p2/publisher/ The product file: http://github.com/hmalphettes/sonatype-tycho/blob/p2/tycho-p2/tycho-p2-runtime/tycho-p2-runtime.product
(In reply to comment #4) > ... As such, the publisher needs additional information telling it what to do. That is, advice > and p2.inf etc > I disagree. That's like telling a jar packager how to assemble the content of a manifest. What it needs is a manifest that has been compiled by a build system, i.e. the p2 publisher should not construct the IU during publishing. It should be handed one that has been created by the build system. The p2.inf file is just one way of influencing how this artifact (the IU) is built and as such, it clearly belongs in a build system. The API I'd like to see knows how to publish IU's and artifacts to their respective repositories. An optional extension to that API also knows how to manage composite repositories. That's about it. Why do we go beyond this functionality in the p2 domain? I'm all for a simplified workflow but I'm advocating a layered approach. To me .product and features is very far from a p2 I'd like to use for Ruby, PHP, or other types of systems that also has a well defined component structure. So why do I need to be burden with it? If PDE, Tycho, and Buckminster want's a layer where they share common build tasks, that's all great, but it doesn't belong in p2.
(In reply to comment #6) > I disagree. That's like telling a jar packager how to assemble the content of a > manifest. What it needs is a manifest that has been compiled by a build system, > i.e. the p2 publisher should not construct the IU during publishing. It should > be handed one that has been created by the build system. I think we have a conceptual or assumption differences here. If the IUs are already constructed then there is no need for a publisher, just call the metadata repo api to add the ius. Whether you call it build or p2 or ... it does not change the need for a thing that can take various artifacts and construct IUs. The p2 team supplies an extensible publisher mechanism but nothing is forcing people to use it. > The p2.inf file is > just one way of influencing how this artifact (the IU) is built and as such, it > clearly belongs in a build system. As far as I recall, the p2.inf file is input to the publisher. The PDE build itself does not create IUs or particularly use the p2.inf files. It calls the publisher. > The API I'd like to see knows how to publish IU's and artifacts to their > respective repositories. An optional extension to that API also knows how to > manage composite repositories. That's about it. Why do we go beyond this > functionality in the p2 domain? Sounds interesting but does not fit with the (well , my) definition of "publisher". That feels more in the world of mirroring and repo API. In any event, this bug is about the creation of IUs not their management. > I'm all for a simplified workflow but I'm advocating a layered approach. To me > .product and features is very far from a p2 I'd like to use for Ruby, PHP, or > other types of systems that also has a well defined component structure. So why > do I need to be burden with it? If PDE, Tycho, and Buckminster want's a layer > where they share common build tasks, that's all great, but it doesn't belong in > p2. FWIW, the publisher itself is completely artifact/domain agnostic. The publisher bundle *happens* to supply a number of OSGi/Eclipse-specific publishing actions. These could (and perhaps should) be factored out into another bundle. If they were, we could cage match for who gets/has to own them but that is beside the point. The point is that the current implementation of the requested function (getting root files into repos) is part of PDE Build and not runnable independently. As a result, there is no simple means of publishing features in all their glory. The same is true of products for that matter. System integrators and writers of alternative build mechanisms need an easy to use/run mechanism.
Good points. And yes, we have a very different view of what publishing is. To me, publishing is about making things available whereas building is about constructing those things. The construction of an IU is thus a build action, not publishing. Buckminster has it's own artifacts and it's own way of creating an IU. b3 has too. Maven has meta-data of their own and creates both manifests and IU's based on that. Many other build systems have similar artifacts. The p2.inf is sitting somewhere in the middle and we could do well without it. It just complicates an otherwise clean workflow. I can even argue that it was created out of necessity to overcome some limitations in the feature, product, and plug-in declarations and as such, it should really be part of PDE. It doesn't have any motivation beyond that. There's also the case of the old legacy update sites. At present, p2 is capable of interpreting those. I understand why it has to be that way but I think the logic for it belongs in PDE, not in p2. Reason being that it's highly PDE specific. Today, p2 actually 'builds' the p2 repository from categories, features, and plug-ins on the fly. If p2 was a clean and truly agnostic provisioning system, none of that would be in the p2 code base. I'm very aware that every piece of code that is in p2 is there for a good reason and it is really needed. I just think it's in the wrong place. The sooner a clear separation of concern is established, the better and there's really no need to have the current fuzzy line between the fully agnostic p2 provisioning system and the various build systems that use it.
Its interesting as I see the Publisher supporting an even cleaner separation between things. For example, there is no need for a "builder" to know anything about p2 or IUs. Folks that are building Ruby or C or PHP don't want to know about that stuff. Indeed build time and publishing time may be months apart. The Publisher is there to support system integrators that want to make "offerings" that include these built pieces who need to compose them in some form that can be deployed and run. If they are using p2 then p2 metadata needs to be produced. Call it a p2 metadata builder if you like and move it to some other place. Doesn't change the need or the design. NONE of this is to say that the Publisher itself should know about Ruby, C, PHP, Java, ... That is why it was designed to be extensible. Got a Ruby gem to publish? Write a GemAction, configure it in the Publisher and voila. Got a OSGi bundle or Eclipse Feature? Write a publishing action. The fact that currently these OSGi/Eclipse actions happen to be supplied by the p2 team in the p2.publisher bundle is an artifact (no pun intended) of history. I agree that they could be factored out into a separate bundle. The namespace of that bundle is a separate discussion. As long as it does not have bogus dependencies I think most people can understand that a sophisticated system like p2 will have different parts like a core, ui, and surrounding tools. As for p2.inf etc. yeah, its ugly and is doomed to be hacked and made uglier. In a perfect world all artifacts to be put in p2 repos would be completely self-describing for all user scenarios. Unfortunately, that is not the case. p2.inf enables folks to fill the gaps in the information inherently available in the artifacts being published. It is one way of providing advice and decoupling the various roles at play here. So back to the original bug, Andrew, do you see a way of factoring out PDE Build's feature-related publishing actions/advice such that someone running just the publisher can use it? Or is there something fundamental to how it works that binds it tightly to PDE Build? Can we simplify the root files mechanism if that helps?
I think we agree on most parts. I have nothing against an extensible and agnostic framework that supports meta-data construction. My objections start where the agnosticism suffers. Today more then 70% of the publisher code belongs in PDE. Remove everything that deals with features, products, branding, and root files and you don't have much left. (In reply to comment #9) > As for p2.inf etc. yeah, its ugly and is doomed to be hacked and made uglier. The thing that really bugs me about the p2.inf file is that it now must be regarded as API and as such, will remain forever. There could be one way to deal with that and that's if we acknowledge the reasons for it's invention and move it to the PDE domain where it makes sense when used together with the current PDE artifacts. In p2, I'd like to see a way to persist an IU. Not as a repository. Just the IU. Such an artifact would make it possible to create a p2 component ready for publishing. Today everything is baked together and called 'publishing' which makes it difficult to accomplish many use-cases.
(In reply to comment #10) > Today more then 70% of the publisher code > belongs in PDE. Remove everything that deals with features, products, branding, > and root files and you don't have much left. Sure. We agree that the current p2.publisher bundle could be refactored. We still need a lightweight way of publishing root files. > The thing that really bugs me about the p2.inf file is that it now must be > regarded as API and as such, will remain forever. There could be one way to > deal with that and that's if we acknowledge the reasons for it's invention and > move it to the PDE domain where it makes sense when used together with the > current PDE artifacts. It being sucky and API is unfortunate. There is however very little in a p2.inf that is Eclipse or OSGi artifact specific. Its all about capabilities and requirements, touchpoint instructions etc. I could see using it when publishing Ruby Gems just as well. > In p2, I'd like to see a way to persist an IU. Not as a repository. Just the > IU. Such an artifact would make it possible to create a p2 component ready for > publishing. Today everything is baked together and called 'publishing' which > makes it difficult to accomplish many use-cases. Sure it would be great if all interesting deployable things came with p2 metadata. Then we would just have an IUPublishingAction that read the artifact and put it in a repo. Sweet. As we have seen with OSGi bundle manifests however, getting unrelated parties to create artifacts that are of no interest or use to them is, errr, challenging. For the foreseeable future, much like Orbit, we are going to have to create the required metadata for these artifacts.
(In reply to comment #11) > It being sucky and API is unfortunate. There is however very little in a > p2.inf that is Eclipse or OSGi artifact specific. Its all about capabilities > and requirements, touchpoint instructions etc. I could see using it when > publishing Ruby Gems just as well. > My point was that we still have an opportunity to keep the p2 API clean by considering this a PDE thing. That in itself has some value IMO. Although generic, I doubt anyone uses it for anything else today. > Sure it would be great if all interesting deployable things came with p2 > metadata. Then we would just have an IUPublishingAction that read the artifact > and put it in a repo. Sweet. Exactly :-) > As we have seen with OSGi bundle manifests > however, getting unrelated parties to create artifacts that are of no interest > or use to them is, errr, challenging. For the foreseeable future, much like > Orbit, we are going to have to create the required metadata for these > artifacts. Of course, but as long as no other options exists it's going to stay that way forever. What I'm asking is that the process of creating the meta-data is split up so that there is a choice. Do I want to create my repositories from bits and pieces at this point, or just the IU? Perhaps my system is based on IU's from the very start? Perhaps I actually generate the Manifest from the IU instead of the opposite? Or maybe I have a really smart authoring tool for IUs (a feature editor replacement)? The current approach, although generic and agnostic, is hard to grock and for the cases I just mentioned, not very useful. If divided into two parts, meta-data construction and publishing, it could be.
(In reply to comment #4) > Summary: We agree that if the PDE implementation of IRootFileAdvice were easily > usable by others (i.e., not tied to PDE), it would be "a good thing" and would > enable better workflows, less duplicate code and more consistency in output. We tried to move the IFeatureRootAdvice from the pde.build ant coding to the pde.publishing bundle that is created within the activities around #331974 for consolidating publisher related stuff from PDE and P2 implementations. It turned out that this is no simple movement to the common bundle as the implements relies heavily on Ant functionality to collect the list of files and (therefore magically) support these patterns. Moving the Ant parts with the implementation would end up with class loader issues when used in bundle code and in ant coding simultaneously. With the obvious solution not working out, the next best approach is to start a new implementation for IFeatureRootAdvide that is not bound to additional technologies (like Ant file sets). This implementation should start with supporting the fundamental root file mechanism and leaving all the fancy parts out. When this implementation has eventually evolved far enough, we can even delete the pde.build implementation to consolidate between p2 and pde. For Tycho there is an implementation that is capable of publishing root files with basic functionality. It is not yet far enough to replace the current implementation in pde.build, but it could be used as a starting contribution. Currently support for permissions (which for sure is mandatory) is added, once this is done, we would propose the code here.
(In reply to comment #13) Right. If you look far enough back in the history of the publisher bundle you will find an implementation of IRootFileAdvice that, while buggy and incomplete, does provide at least one possible starting point for an implementation. I completely agree that most of the uses of root files can be handled with simple interpretations of the Ant patterns. That is, we should support just an easy to implement subset of Ant patterns. That was the approach taken in the original publisher implementation.
Sorry for the late comment, I'm trying to catch up on a few hundred unread bug emails. When adding this functionality back to the publisher, please see bug 248578. The code was originally removed because it broke dropins by generating bad metadata for org.eclipse.equinox.executable.
(In reply to comment #15) > When adding this functionality back to the publisher, please see bug 248578. > The code was originally removed because it broke dropins by generating bad > metadata for org.eclipse.equinox.executable. The way I see it, the publisher bundles should provide an implementation of IFeatureRootAdvice, but that implementation should not be active by default. So the dropins code would not be affected. In case people are interested, the implementation in Tycho that shall be contributed to p2 is pretty far already - only the support for patterns is missing (see https://issues.sonatype.org/browse/TYCHO-604 ). We aim for contributing this for 3.8 M1. (Should I assign this to me? I'm not p2/pde.build committer though.)
This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. If you have further information on the current state of the bug, please add it. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. -- The automated Eclipse Genie.
