Community
Participate
Working Groups
Created attachment 182115 [details] patch to generate source bundles It is often interesting to have the build output source. PDE UI supports this by generating source feature declarations with all the right markup. PDE Build however does not have an option to "get source with that". Attached here is a patch that provides two PDE Build switches: generateUberSourceFeature and includeAllSource. (naming is preliminary). generateUberSourceFeature (boolean): if true it tells PDE build to create an uber source feature that references source bundles for the things being built. There is no attempt to create a source feature structure that matches the runtime feature structure. The goal here is to simply get the source bundles generated and into the resultant repository. This coupled with the new PDE UI "would you like source with that" capability in target management (see bug 328929), allows people to easily create SDKs for their features and products. includeAllSource (boolean): refines what the uber source feature contains. if false, only source for things actually being compiled is generated and published. If this flag is true, all possible known source is included. This latter setting allows for the creation of complete SDKs from existing SDKs. This patch (against head) is a prototype to show the idea and gauge interest. It has been tested in a couple of PDE UI export work flows without using PDE UI's source generation facilities and works as described. Topics: - The output of such a build includes an additional feature (the uber source feature). This feature is not really intended for explicit use rather it is an unwanted artifact of the process. Need to figure out how to stop its generation/inclusion - Flag naming. "ubersourcefeature" is very implementation-y. Need a better name that speaks to users. - The implementation relies on a hack/convention whereby source bundles end in .source. This is indeed the case for our stuff but is kinda hacky. It would be better to have a more explicit way of telling source bundles from others. The manifest has this information for built bundles and for things that we are building we should know that it is an auto-generated source bundle. We just need to track this info. - PDE UI integration. Once this behaviour is nailed down we should look at how PDE UI integrates with this. Currently they have their own mechanism for exporting with source. Perhaps that can be simplified in favor of support in PDE Build? - As an enhancement, we should also be able to enable excluding things from the uber source feature. Perhaps using regex or ant patterns or something trivially simple. A workaround for folks that really care is to ensure that source they don't want included is not present in the target/base. But that forces them into more management of complex data sets when a simple set of patterns would do the trick. - This patch introduces two booleans but not all four combinations are valid. It may be better (less error prone) for users if there was just one setting (eg., "autoSourceGenerationMode") with value like NONE, ALL, COMPILED - Given the goal of this enhancement (to get the source bundles in the output repo) it may be good to tie its enablement to whether or not we are creating a repo (whatever that means).
Any comments on this patch?
Created attachment 185672 [details] alternate patch (In reply to comment #0) > - The implementation relies on a hack/convention whereby source bundles end in > .source. This is indeed the case for our stuff but is kinda hacky. It would > be better to have a more explicit way of telling source bundles from others. > The manifest has this information for built bundles and for things that we are > building we should know that it is an auto-generated source bundle. We just > need to track this info. This is already tracked to some degree by the Eclipse-SourceBundle manifest header. The utility method Utils.isSourceBundle can be used here. However, I believe this only identifies the "new" individual source bundles, I don't think the old source plug-ins which contain src.zip files for multiple bundles will have this header. I would be ok with ignoring the old shape here. > - As an enhancement, we should also be able to enable excluding things from the > uber source feature. Perhaps using regex or ant patterns or something > trivially simple. A workaround for folks that really care is to ensure that > source they don't want included is not present in the target/base. But that > forces them into more management of complex data sets when a simple set of > patterns would do the trick. There exists a mechanism to specify extra features & plugins to include in a generated source feature. This can also be used to exclude things from the source feature. This is done by decorating the generate.plugin@ or generate.feature@ properties with extra stuff. See SourceGenerator.associateExtraEntries. We should re-use this somehow. I've attached a new patch that using a different strategy in creating the source, it does not require any changes to BuildDirector or FeatureGenerator. We create an in-memory feature (no need to persist to disk) feature containing all the plugins for which we want source. Then pass this feature to SourceGenerator#generateSourceFeature which handles generating all the corresponding source. Finally call generator.generate on the new source feature. Things we need: -Specify the id & version for the new source feature. I think this would be useful for people, they can go to a repo and get all source. - A way to specify the "extraEntries" string which allows for adding additional features/plugins (ie docs) or excluding things. - Consider allowing specifying another feature to use as a template for other thigns like license, label, etc on the generated source feature. - Changes to the ant scripts & generate task to pass all this stuff in,
Thanks Andrew for looking at this and the new patch. I like the new version. Cleaner and simpler. > have this header. I would be ok with ignoring the old shape here. Agreed > There exists a mechanism to specify extra features & plugins to include in a > generated source feature. This can also be used to exclude things from the > source feature. This is done by decorating the generate.plugin@ or > generate.feature@ properties with extra stuff. See > SourceGenerator.associateExtraEntries. We should re-use this somehow. Right. For now however, I'm fine with not being fancy. > I've attached a new patch that using a different strate > Things we need: > -Specify the id & version for the new source feature. I think this would be > useful for people, they can go to a repo and get all source. The initial point of this direction was to make the source feature obsolete and irrelevant. I do agree however that it may be interesting to have an IU available that references the auto-generated bundles. This would, for example, allow for easier mirroring. Given the enhancements to the PDE target platform source facilities, normal users should never have to see or care about this source feature. They get source simply by saying "I want source with that". Note also that if you do a build and already have source bundles around, the output uber source feature would not include these so really cannot be relied on for managing source bundles. To get PDE build "out of the middle", I agree that we should allow id and version specification but would like to not go much further than that. > - A way to specify the "extraEntries" string which allows for adding additional > features/plugins (ie docs) or excluding things. Interesting possible enhancement but not essential IMHO. > - Consider allowing specifying another feature to use as a template for other > thigns like license, label, etc on the generated source feature. Related to the above, the source feature is a by-product of this build process. An IU that allows for easy mirroring etc is useful but normal workflows should not expose/require this IU. I'd prefer to see these capabilities as possible enhancements. > - Changes to the ant scripts & generate task to pass all this stuff in, Yup. will attach an updated patch for this later today.
Created attachment 186439 [details] updated patch that includes script and task updates This patch is based on Andrew's latest and adds in the Ant task changes as well as genericTargets.xml changes. I did a few cosmetic changes to make the setup more consumable and expose fewer implementation details. Namely - we no longer talk about generating uber source features and including all. Instead there is a "source bundle mode". "all" and "built" are the two main settings corresponding to includeAll = true and false respectively. - added properties for the feature name and version (sourceBundlesFeatureId, sourceBundlesFeatureVersion) Note that I was not able to test this out as I could not get it to export. There is something really goofy going on with all the dependencies in the Ant task world. Branding depends on repo2runnable that depends on jarprocessor, ... and somewhere a long the line something is not being found. Basically I can;t get my workspace setup to compile any task that has a dependency on another task. I'd be happy to test this out if I could get it to build but...
Seems the export problem is due to Bug 333981. I'm attempting to get the pde build product exported and tested but there are other issues with the setup that make it not exportable.
Created attachment 186551 [details] Updated patch with minor tweaks this is an updated patch with few robustness tweaks. I managed to get the pde builder product exporting and running with these changes and can confirm that <none>, all and built options work for the sourceBundleMode property. The desired source bundles are output into the repo and NOT to the product install (when building a product). Note I also updated the headless build build.properties to have some template entries for the new properties.
It felt to me like there was a large hole here if you wanted anything other than a set of source bundles in a p2 repo. For example, if you wanted to categorize them or anything like that. There was very little control over the containing feature that holds all of these source bundles. What I did was add another property "sourceBundleTemplateFeature", which specifies a feature that we will base the resulting source feature on. (ie the normal relationship from feature to generated source feature). Things like descriptions & licenses get copied from here, as well, a "sourceTemplateFeature" sub folder can be used to provide extra files that should be part of the generated source feature (including a p2.inf file, which can then be used to augment the p2 metadata). If this property is not specified, then we remain as before with a default name. So there are a couple of ways to set the properties: 1) set only "sourceBundleMode". End up with a source container feature named "org.eclipse.pde.build.uber.feature.source" and all the source bundles are in the repo. 2) Set sourceBundleMode + sourceBundleFeatureId (+ sourceBundleFeatureVersion), this is the same as (1) except you control the id & version of the resulting source container 3) sourceBundleMode + sourceBundleTemplateFeature, allows pretty complete control over the resulting source container I have released the changes with the addition of one unit tests based on (3)
awesome. I'll try this out. I agree that the improvements you suggested are useful.
