Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 406825

Summary: consolidate master-x features into one.
Product: [Eclipse Project] Platform Reporter: David Williams <david_williams>
Component: RelengAssignee: David Williams <david_williams>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: P3 CC: mat.booth, tjwatson
Version: 4.3   
Target Milestone: 4.4   
Hardware: PC   
OS: Linux   
Whiteboard:
Bug Depends on: 420725, 426634, 426645    
Bug Blocks: 432776    

Description David Williams CLA 2013-04-29 13:18:18 EDT
The "master-xyz" features were required (or, useful) by PDE build, since it is "feature oriented" and built "top down" from the feature definitions. That's no longer true for Tycho/Maven based builds and these master features are "fragile", especially since we try to remove them from our final repository. 

In theory, we could just not remove them, but then at best that leaves "junk" in our repository, and at worst, we've had cases in the past where the repo, as a whole, was not self consistent and would fail certain b3 aggregation operations or "complete mirroring" operations. 

Plus, as is, I think most of what is in them, is no longer needed to be in a master feature, at all. 

We've easily gotten rid of two of them master-jetty (bug 406746) and master-ecj (bug 406756) so suspect the remaining two should not be too much work. 

I plan to do this "incrementally" to be safest, but would then experiment with removing unnecessary parts. 

One step will be to remove "the real" features we want in the repository "up" to the category.xml file. That is where they belong, and the master features were used in the initial prototype (I think) as a quick short cut (maybe too, so that PDE build would continue to work, as is ... but ... we are past that point. 

The other step will be to move all the "stray" bundles listed in remaining master- features "up" to the main master feature. so they are all in one place. 

As far as I know, there is only one thing that _must_ be in that master feature, and that is the  org.eclipse.jdt.core.compiler.batch plugin. There, the reason is we want to provide that plugin in our repository, but it is not contained in any other feature, and only features can be listed in content.xml file in building a repository. In this sense, the "master feature" becomes a sort of "misc. feature", instead of "master". 

Many, if not all, the other "misc plugins" listed in master-equinox* features used to be required to "pull things" from repos before they were needed by the build ... but, that is not the way Tycho/Maven work, and things are pulled automatically when needed, pretty much. 

As it is now, these "master-" features are only used to create the final repository.
Comment 1 David Williams CLA 2013-04-30 09:22:45 EDT
As a temporary work around, I'm going to "expand" what we remove ... to remove the associated "jars", as well as the "feature groups". Seems like used to not, but PDE probably didn't produce them. Now there is a jar in the repo, and I've seen several cases where the build fails due to the way the new comparator works. 
So, need to avoid having "matching versions/qualifiers" in repo. 

(There must be an easier way :/
Comment 2 David Williams CLA 2013-05-01 08:55:55 EDT
For M7, I plan to remove this module from from the root pom in platform.releng

 <module>features/master-equinox-p2</module>

We do not use it anywhere. It used to be in master-equinox, but we removed from there long ago: 

 <!-- see bug 368488 and bug 377314           
 <includes
         id="master-equinox-p2"
         version="0.0.0"/>        
 -->   

Second, for the master features we no longer use, I plan to delete file the files in the folder, except leave a readme pointing to the removal bug. And, then plan to remove the actual folder itself, to avoid clutter. 

Not sure I can finish "the rest" of the consolidation for M7, but still investigating what needs to be done.
Comment 3 David Williams CLA 2013-05-01 23:47:53 EDT
If there is a safe/easy way to make remaining changes, would still like to do for Kepler. But, not a blocker if remaining work needs to be deferred -- we need an "embarrassing" severity for this one :)
Comment 4 David Williams CLA 2013-05-13 09:46:31 EDT
Remaining work to modularize will need to be deferred to next release.
Comment 5 David Williams CLA 2014-01-26 01:31:48 EST
bug 420725 will get rid of 
org.eclipse.equinox.starterkit.product.feature
so adding that bug to 'depends' list (though, is kind of indirect coincidence).
Comment 6 David Williams CLA 2014-04-01 15:52:17 EDT
By chance, while working on bug 431540, it became apparent why we still had an "equinox-master" feature, and the cleanup done in bug 431540 makes it feasible to remove equinox-master. The primary reason we still "need" it, is that it was being used as part of some "post build" scripts to "publish" equinox ... which are not necessary, and especially not necessary after fixing bug 431540. 

The only three features left in "equinox master" are 

org.eclipse.equinox.sdk
org.eclipse.osgi.compatibility.plugins.feature
org.eclipse.equinox.weaving.sdk

And the weaving.sdk is itself included in equinox sdk so that could be removed immediately. In conversation with Tom, 
org.eclipse.osgi.compatibility.plugins.feature
should not be part of sdk, nor Equinox download page, but we still make it available in are main project repository, in case anyone needs it for some really really old plugin that they do not want to migrate. 

So, the next step it to move 
org.eclipse.equinox.sdk
org.eclipse.osgi.compatibility.plugins.feature
into the one "master-feature", and finishing cleaning up part of the post-build publishing scripts so they no longer rely on equinox master feature, and then it can be removed. (from repository, and pom). 

After that, it should be feasible to move all the master-feature items directly into the "category.xml file" and no longer have any "master" features (which, were left over from PDE builds, which used them, top down, to "drive the build".
Comment 7 David Williams CLA 2014-04-10 16:19:41 EDT
(In reply to David Williams from comment #6)

I've removed the master-equinox project from the build, moving two of the remaining features into "master/feature.xml". [As explained above the "master feature" listed 
org.eclipse.equinox.sdk
org.eclipse.osgi.compatibility.plugins.feature
org.eclipse.equinox.weaving.sdk
but "weaving" is "included" in SDK, so no need to include it here. 
So, to master, moved only 
org.eclipse.equinox.sdk
org.eclipse.osgi.compatibility.plugins.feature


It also required some changes in post-build publishing ... that was easy enough. 

I think mentioned in "doc excludes" file too plus is in some "post build" scripts to remove "equinox-master" IU and feature from repository (I'll do a workspace wide search) but just wanted to document here it has been removed. 

To verify, should check nightly build tonight (4/10) to to make sure "download page" has what it needs, and that repository has same number of plugins and features as before. 

To document it here, the last tagged version that would have had it is 
I20140408-1645
Comment 8 David Williams CLA 2014-04-11 09:46:54 EDT
(In reply to David Williams from comment #7)
> 
> To verify, should check nightly build tonight (4/10) to to make sure
> "download page" has what it needs, and that repository has same number of
> plugins and features as before. 

I confirmed the nightly equinox page "looks normal" and that the nightly N build repository has same number of files as the last few nightlies (956 grand total -- I just everything under composite child). 

In theory, should be easy to get rid of "master" now. 
But, I'll try some test runs on local machine, and see what hidden assumptions we might have on it.
Comment 9 David Williams CLA 2014-04-17 08:24:15 EDT
Fixed with 

http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/commit/?id=e3751afbfec7b286881bdcfb3b8afadb7c323fb9

http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/commit/?id=737bee7c5b6f1c2c7a3cf42b3ceee2281f8f3b40

which combines fix for 
Bug 432776 - move and simplify eclipse.platform.repository

There is one visible difference I've noticed ... seems there are 8 more "*jar.pack.gz" files being pulled in from Orbit (or other?) repos. Not sure why this would be, but seems like it could not cause harm -- since the jar forms were being include previously ... and might "turn out the same" in a fully promoted I build? 

./plugins/org.aspectj.runtime_1.7.3.20130403143700.jar.pack.gz
./plugins/org.easymock_2.4.0.v20090202-0900.jar.pack.gz
./plugins/org.hamcrest.integration_1.1.0.v201303031500.jar.pack.gz
./plugins/org.hamcrest.library_1.1.0.v20090501071000.jar.pack.gz
./plugins/org.hamcrest.text_1.1.0.v20090501071000.jar.pack.gz
./plugins/org.hamcrest_1.1.0.v20090501071000.jar.pack.gz
./plugins/org.mockito_1.8.4.v201303031500.jar.pack.gz
./plugins/org.objenesis_1.0.0.v201105211943.jar.pack.gz

= = = =  

Leaving open as a bit more clean up to do, such as remove "master" from some of the clean up scripts.
Comment 10 David Williams CLA 2014-04-17 08:48:09 EDT
removed a few other places 'master' feature was mentioned, one was whole while that has never been used, as far as I know. As few other "commented out" sections that are no longer helpful. 

http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/commit/?id=57bb299273c5e39d59141dd36ac6933fc0bda61f

And believe it or not ... there goes the last "pure pde" vestige of the old PDE build!