Bugzilla – Full Text Bug Listing
| Summary: | Provide a convenient way to ship source bundles | ||||||
|---|---|---|---|---|---|---|---|
| Product: | z_Archived | Reporter: | Mickael Istria <mistria> | ||||
| Component: | Tycho | Assignee: | Igor Fedorenko <igor> | ||||
| Status: | RESOLVED FIXED | QA Contact: | |||||
| Severity: | enhancement | ||||||
| Priority: | P3 | CC: | cornelius.dirmeier, gregory.amerson, gunnar, hmalphettes, igor, jan.von.loewenstein, nyssen, t-oberlies | ||||
| Version: | unspecified | ||||||
| Target Milestone: | --- | ||||||
| Hardware: | All | ||||||
| OS: | All | ||||||
| See Also: | https://bugs.eclipse.org/bugs/show_bug.cgi?id=329162 | ||||||
| Whiteboard: | |||||||
| Bug Depends on: | 220254, 347483 | ||||||
| Bug Blocks: | |||||||
| Attachments: |
|
||||||
|
Description
Mickael Istria
This is related to this report: https://issues.sonatype.org/browse/TYCHO-192 @Mickael why do you need source features? Can you provide a user story and/or relevant documentation that gives us some context? As I am moving GMF Tooling build to Tycho, I have to provide the same features as the current Athena builder produces. Among these features, there is the source feature, which is composed of the source bundles for the main feature, which is generated from the main feature. What happens (if I understand well) is that for each of the bundles listed in feature.xml, the builder generates the source bundle, and then generates a source feature containing them. The aim of source features is to have a distribution containing directly sources of plugins. It is necessary for contributors who want to start tracking an issue, or for plug-in/RCP developers when debugging their own code. The user story is: I have my feature featureA containing bundleA1, bundleA2, and I want to provide the source bundles for all my bundles in featureA. Athena is able to generate a featureA.source, containing bundleA1.source and bundleA2.source directly from featureA. With Tycho, it requires more effort: featureA.source has to be hand written and maintained. The best behavior would be to have a Tycho plugin create featureA.source from featureA, the same way tycho-source-plugin provides a bundleA1.source from bundleA1. Basically, the plugin would read the feature.xml, and append ".source" where necessary and generate the feature. Feel free to ask for more details or example if necessary. To make sure I understand, featureA.source will be installed by developers who build on top of bundles included in featureA and what to be able to step through their source code in the debugger or just look at the code without importing bundleA1 and bundleA2 from their source code repository. Did I get this right? Yes, that's exactly the use case. At least, that's the way we use them in our team. +1. For jetty, at the moment we end up maintaining by hand the source features. Note that PDE has added in 3.7 the option to resolve the sources without having to explicitly have a set of source features: http://mcaffer.com/2010/12/would-you-like-source-with-that/ https://bugs.eclipse.org/bugs/show_bug.cgi?id=329162 We could mirror this behavior when a p2 repository is published: for each bundle published in the repo, try to locate the source bundle and publish it too. Sorry to add a little bit to the last thought: The trend that Jeff showed is that the publishing of sources should be as transparent as possible. For example equinox does not publish a source feature for org.eclipse.equinox.core.feature and org.eclipse.equinox.p2.core.feature although the source bundles for those features are available in the same repository. eclipse.org/rtp publishes those 2 features. But we have not found a good way to publish the source bundles for them. Transparently publishing the source bundles that can be located for each bundle published in a repo would be the enhancement. Internally, we would use a "p2-mirror-with-sources". I have ported the "with-sources" to p2director so if you like the idea I can try to help with this. Let me know if I am hi-jacking this bug and I'll file a different one. Thanks! @Hugues: I don't think it is hijacking at all, that's the same use case; and having an even more transparent implementation for it is always a good idea. Indeed, there is no real need for a source feature. The real need is to be able to see the sources for a bundle when someone want them, without effort for the user. For what I understood of bug #329162, it seems to be something that has been modified internally in PDE build, not sure anything in this implementation can be reused for Tycho. I am not easy enough with internal Tycho implementation to get your point about the "p2-mirror-with-sources", but I trust you ;) (In reply to comment #8) > @Hugues: I don't think it is hijacking at all, that's the same use case; and > having an even more transparent implementation for it is always a good idea. OK :) > > Indeed, there is no real need for a source feature. The real need is to be able > to see the sources for a bundle when someone want them, without effort for the > user. > > For what I understood of bug #329162, it seems to be something that has been > modified internally in PDE build, not sure anything in this implementation can > be reused for Tycho. PDE calls the P2 APIs directly. p2director is the command line application that you point at a set of p2 repository and can install/update features and products. I have extended p2director on github to support an -addSources argument. It is implemented by porting the code that Jeff did for PDE into p2director. In my pending tasks, I am supposed to redo this work as a patch to p2director. > > I am not easy enough with internal Tycho implementation to get your point about > the "p2-mirror-with-sources", but I trust you ;) Trust Tobias, Jan Sievers, Igor and the other tycho committers :) Internally, tycho creates a small p2 repository for each built feature. When tycho creates the repository, it calls the p2mirror app to mirror each one of those repository. In effect aggregating them into the published repository. The day I get to file as a patch to p2director for the -addSources support, I'll try to do it for p2mirror as well. Tobias is the person to convince here as he is an equinox incubator committer. (In reply to comment #8) > Indeed, there is no real need for a source feature. The real need is to be able > to see the sources for a bundle when someone want them, without effort for the > user. I changed the title from "Provide a tycho-source-feature plugin" to "Provide a convenient way to ship source bundles" describe the more general requirement. > For what I understood of bug #329162, it seems to be something that has been > modified internally in PDE build, not sure anything in this implementation can > be reused for Tycho. Tycho provides a plugin to generate a source bundle when building a binary bundle. Unfortunately, there is no documentation for it (bug 344718). (In reply to comment #9) > The day I get to file as a patch to p2director for the -addSources support, > I'll try to do it for p2mirror as well. This is exactly what we would need. Please add the "mirror source" p2 bug as dependency of this bug. I'm also p2 committer, so we should be able to get this into p2 and Tycho. OK, I field the enhancement to p2mirror here: bug 345512. Thanks Hugues for driving the discussion. Through the discussions in bug 345512, we figured out that we have three options: 1) Create a custom mirror tool for Tycho that includes the patch proposed in bug 345512. 2) Publish a filtered requirement from binary bundles to sources (as proposed in bug 347483) and activate the filter (on request) when mirroring the content for eclipse-repository. 3) Publish the "source of" information (as proposed in bug 220254) and do the necessary changes in p2 and Tycho so that this information can be used to include sources. I don't like option 1 because it is the most expensive one in terms of maintenance for Tycho. Also, I think that the issue we are seeing here is quite general, so I would prefer to see a solution implemented in p2. Option 2 is fairly simple to implement, solves this issue, and would even give immediate benefit for other p2 tools. For example it would also allow to create installations with the p2 director that include source bundles. This leads to the main downside of option 3: Solving the problem of getting sources bundles into a p2 repository is fairly straighforward once bug 220254 is implemented. But what then: Unless other p2 tools (like the p2 director) get explicit support for the "source of" information, the source bundles in the p2 repository will be useless. So to solve this issue, we need to continue the discussions in bug 347483 and bug 220254 and implement one of the proposals. Thanks for the kudo's Tobias. Here are my 2 cents, maybe slightly out of the scope of tycho: I favor #1 and I maintain an extended director+publisher+mirror application that I invoke on the command-line for some custom build promotion scripts.[1] There are a whole lot of features that don't fit in p2 core scope and that are required to support tycho builds and OSGi provisioning scenarios. So far these features end-up coded in ant-tasks and in PDE and then others need to redo these things. Maybe we could come up with a separate incubator project in equinox or p2 to develop such APIs and command-line tool? Developing this tool requires opening up a little bit some of the p2 code. I agree that your solution is shooting for a higher standard by the way: support the semantics of sources abstracted away from OSGi and the java scenarios. [1]https://github.com/intalio/org.eclipse.equinox.p2.director.extended I have found out that my proposal in bug 347483 doesn't work with target files with includeMode="slicer". Therefore we only have two options left: - Publish the "source of" information (as proposed in bug 220254) and do the necessary changes in p2 and Tycho so that this information can be used to include sources. - Generate source features in Tycho. (In reply to comment #14) > I have found out that my proposal in bug 347483 doesn't work with target files > with includeMode="slicer". Therefore we only have two options left: > - Publish the "source of" information (as proposed in bug 220254) and do the > necessary changes in p2 and Tycho so that this information can be used to > include sources. > - Generate source features in Tycho. From a migration perspective (I am currently investigating how to migrate GEF to a new Tycho/Hudson based build infrastructure), the second option would really be nice, as otherwise a lot of changes to the existing project structures would have to be done, where generate.feature support of PDE is currently used (I suppose this will be a lot of projects). It would be nice to also support the customization options (sourceTemplatePlugin and sourceTemplateFeature), which are supported by PDE (see http://help.eclipse.org/indigo/index.jsp?topic=%2Forg.eclipse.pde.doc.user%2Ftasks%2Fpde_source_generation.htm for details). (In reply to comment #15) > From a migration perspective (I am currently investigating how to migrate GEF > to a new Tycho/Hudson based build infrastructure), the second option would > really be nice, as otherwise a lot of changes to the existing project > structures would have to be done, where generate.feature support of PDE is Other than adding single sources feature project, what else do you need to change? (In reply to comment #16) > (In reply to comment #15) > > From a migration perspective (I am currently investigating how to migrate GEF > > to a new Tycho/Hudson based build infrastructure), the second option would > > really be nice, as otherwise a lot of changes to the existing project > > structures would have to be done, where generate.feature support of PDE is > > Other than adding single sources feature project, what else do you need to > change? The problem is that I have to do these changes, what makes migration harder. If Tycho would support source feature generation, the structure of the project could remain intact (so that pde can still be used, while building up a new Tycho-based build). While this is thus not nice, for me the biggest problems seems to be that there is no equivalent to PDE's sourceTemplatePlugin customization, when generating source bundles (or am I missing something here). If Tycho should generate source features, they should include the productive feature in order to ensure that binary and source versions match (cf. bug 363205). (In reply to comment #18) > If Tycho should generate source features, they should include the productive > feature in order to ensure that binary and source versions match (cf. bug > 363205). Can you elaborate on this? I am not sure I understand what "productive feature" is. Sorry, I meant "binary feature". We had the case where our target platform contained two versions of the features from Eclipse Platform. Due to a spurious behaviour of the p2 resolver (something like bug 356437), this resulted in a product with binary features in version A and source features in version B. This was possible because we did not include the "SDK" features, which include both binary features and source features and therefore ensure that the versions match. Since no-one really needs the source features separately, I would propose that Tycho should produce source features which are also SDK features at the same time: The source features shall include the source bundles and the binary feature. This would ensure that you can only install matching sources and binaries. (In reply to comment #20) > Since no-one really needs the source features separately, I would propose that > Tycho should produce source features which are also SDK features at the same > time: The source features shall include the source bundles and the binary > feature. This would ensure that you can only install matching sources and > binaries. +1 I like that proposal. It makes things simpler than the current setup by introducing a convention. Will it be possible to influence the name of the generated SDK feature (eg. "SDK" vs. "Target Components")? (In reply to comment #20) > Sorry, I meant "binary feature". > > We had the case where our target platform contained two versions of the > features from Eclipse Platform. Due to a spurious behaviour of the p2 resolver > (something like bug 356437), this resulted in a product with binary features in > version A and source features in version B. This was possible because we did > not include the "SDK" features, which include both binary features and source > features and therefore ensure that the versions match. > > Since no-one really needs the source features separately, I would propose that > Tycho should produce source features which are also SDK features at the same > time: The source features shall include the source bundles and the binary > feature. This would ensure that you can only install matching sources and > binaries. Ah, okay, this makes sense. Btw, I started to prototype source feature generation couple of weeks ago [1] and may be able to finish this before 0.14 but not 100% sure yet. [1] http://git.eclipse.org/c/tycho/org.eclipse.tycho.extras.git/tree/tycho-source-feature-plugin (In reply to comment #20) > Since no-one really needs the source features separately, I would propose that > Tycho should produce source features which are also SDK features at the same > time: The source features shall include the source bundles and the binary > feature. This would ensure that you can only install matching sources and > binaries. Sounds very good for SDK producers and end-users! I've finished tycho-extras:tycho-source-feature-plugin and it should now be possible to configure Tycho to automatically generate sources features during the build. See [1] and [2] for specific changes.
The configuration is somewhat verbose, unfortunately (see below), but this is currently the only way to make Tycho consider 'final' p2 metadata of attached artifacts. More concise configuration would either need to make sources feature generation part of default lifecycle for eclipse-feature projects or changes to maven to allow control over plugin execution order.
<build>
<plugins>
<plugin>
<groupId>org.eclipse.tycho.extras</groupId>
<artifactId>tycho-source-feature-plugin</artifactId>
<version>${tycho-extras-version}</version>
<executions>
<execution>
<id>source-feature</id>
<phase>package</phase>
<goals>
<goal>source-feature</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.eclipse.tycho</groupId>
<artifactId>tycho-p2-plugin</artifactId>
<version>${tycho-version}</version>
<executions>
<execution>
<id>attached-p2-metadata</id>
<phase>package</phase>
<goals>
<goal>p2-metadata</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
As was suggested in comment 20, generated sources feature includes corresponding main feature. It also includes sources features of all features included in the main feature (see basic integration test to understand what I mean).
tycho-source-feature-plugin validates if all included sources bundles/features to be available and fails the build if any is missing.
It is possible to exclude specific sources bundles/features using configuration similar to
<configuration>
<excludes>
<plugin id="sourcefeature.bundle.nosource"/>
<feature id="sourcefeature.feature.nosource"/>
</excludes>
</configuration>
[1] http://git.eclipse.org/c/tycho/org.eclipse.tycho.extras.git/commit/?id=8831506f29b3ebf78d294f80df287ba8eb536c56
[2] http://git.eclipse.org/c/tycho/org.eclipse.tycho.git/commit/?id=ea8d01e910ed5e7ed66166f2bfc0401df24d81ab
Wouldn't it be more convenient to make this Mojo part of the org.eclipse.tycho:tycho-source-plugin ? Current implementation makes plugin "active" only if there is a "sourceTemplateFeature" folder available. It is not trivial, the plugin should work too in case there isn't such a folder. Created attachment 210586 [details]
Git patch to make sourceTemplateFeature optional
(In reply to comment #26) > Current implementation makes plugin "active" only if there is a > "sourceTemplateFeature" folder available. > It is not trivial, the plugin should work too in case there isn't such a > folder. There has to be some sort of trigger to enable/disable source feature generation for individual projects. (In reply to comment #28) > (In reply to comment #26) > > Current implementation makes plugin "active" only if there is a > > "sourceTemplateFeature" folder available. > > It is not trivial, the plugin should work too in case there isn't such a > > folder. > > There has to be some sort of trigger to enable/disable source feature > generation for individual projects. Ok, But it seems to be not possible to factorize this configuration in a parent pom with packaging pom: Failed to execute goal org.eclipse.tycho:tycho-p2-plugin:0.14.0-SNAPSHOT:p2-metadata (attached-p2-metadata) on project features: Execution attached-p2-metadata of goal org.eclipse.tycho:tycho-p2-plugin:0.14.0-SNAPSHOT:p2-metadata failed. IllegalStateException -> [Help 1] So the only way I've found to use this plugin is to put it directly in eclipse-features poms. So there is no need for a trigger if this action (source-feature+p2-metadata) can only be set on individual projects. |