Bugzilla – Full Text Bug Listing
| Summary: | Mirroring http://download.eclipse.org/releases/indigo fails to retrieve all of the SR1 artifacts | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Technology] CBI | Reporter: | Terry Parker <tparker> | ||||
| Component: | CBI p2 Repository Aggregator | Assignee: | Project Inbox <b3.aggregator-inbox> | ||||
| Status: | RESOLVED FIXED | QA Contact: | |||||
| Severity: | normal | ||||||
| Priority: | P3 | CC: | david_williams, john.arthorne, thomas | ||||
| Version: | unspecified | ||||||
| Target Milestone: | --- | ||||||
| Hardware: | PC | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Attachments: |
|
||||||
|
Description
Terry Parker
Ping. What is the correct configuration file to mirror the entire http://download.eclipse.org/releases/indigo/ repository? The entire repository? It contains several versions of most plugins since it's a composite containint all releases (currently the June and September releases). In order to mirror everything, you will need a ValidationSet that corresponds to each child in the composite. An alternative is to mirror only the last child (i.e. the last release) found in the composite. Please note that the aggregator is not a pure mirroring tool. It actually verifies that everything can be installed together into a combination of environments using the p2 planner. Each individual release of Indigo is created that way, but the combination of releases will not pass validation. Hence the need for several validation sets. I'm closing this bugzilla since what you experience is the intended behavior. (In reply to comment #2) > The entire repository? It contains several versions of most plugins since it's > a composite containint all releases (currently the June and September > releases). In order to mirror everything, you will need a ValidationSet that > corresponds to each child in the composite. > > An alternative is to mirror only the last child (i.e. the last release) found > in the composite. > > Please note that the aggregator is not a pure mirroring tool. It actually > verifies that everything can be installed together into a combination of > environments using the p2 planner. Each individual release of Indigo is created > that way, but the combination of releases will not pass validation. Hence the > need for several validation sets. > > I'm closing this bugzilla since what you experience is the intended behavior. Mirroring the last child and then using the aggregator to resolve all the metadata is probably what I need. It is unclear to me what the aggregator should do when the ValidationSet is left empty. Current behavior is to mirror all of the features, but at random versions if multiple versions are available. Two more consistent alternatives are: 1) Mirror nothing (always require an explicit ValidationSet) 2) Mirror all available features at the most current version The aggregator doesn't do random selections. The approach chosen when nothing specified is #2. If the desired action is to mirror nothing, just remove the repository mapping ;-). I think the ability to mirror all features is an important one. (In reply to comment #4) > The aggregator doesn't do random selections. The approach chosen when nothing > specified is #2. > > If the desired action is to mirror nothing, just remove the repository mapping > ;-). I think the ability to mirror all features is an important one. #2 isn't what I see, as stated in my original bug report. It would be great if it worked that way, but it doesn't. In that case I misread your original report. Unless otherwise specified it should mirror all features found, but only the latest version. If you are seeing another behavior, then I must ask if you had any errors in the log during mirroring or if the mirroring didn't complete normally? (In reply to comment #6) > In that case I misread your original report. Unless otherwise specified it > should mirror all features found, but only the latest version. If you are > seeing another behavior, then I must ask if you had any errors in the log > during mirroring or if the mirroring didn't complete normally? I cleared the log and the aggregator output directory, and simplified my aggregator file to be the following: <?xml version="1.0" encoding="ASCII"?> <aggregator:Aggregation xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:aggregator="http://www.eclipse.org/b3/2011/aggregator/1.1.0" label="Indigo" buildRoot="/usr/local/google/users/eclipse_mirror/bucky"> <validationSets label="main"> <contributions label="Indigo 3.7"> <repositories location="http://download.eclipse.org/releases/indigo/" description="Eclipse 3.7 release"/> </contributions> </validationSets> <configurations operatingSystem="linux" windowSystem="gtk"/> </aggregator:Aggregation> No errors in the error log, and everything completed normally. $ grep Error .log | wc -l 0 $ grep "org.eclipse.ui.ide," .log !MESSAGE - mirroring artifact osgi.bundle,org.eclipse.ui.ide,3.7.0.v20110809-1737 !MESSAGE - mirroring artifact osgi.bundle,org.eclipse.ui.ide,3.7.0.I20110519-0100 $ grep "org.eclipse.jdt," .log !MESSAGE - mirroring artifact osgi.bundle,org.eclipse.jdt,3.7.0.v201106131736 !MESSAGE - mirroring artifact org.eclipse.update.feature,org.eclipse.jdt,3.7.0.v20110520-0800-7z8gFchFMTdFYKuLqBLqRja9B15B I'll attach the resulting content.jar file. Created attachment 207067 [details]
p2 metadata data from mirroring Indigo SR1
It's unclear to me what IU's are are missing. Can you please provide an id to look for? Based on your grep and earlier comments I checked and found that these IU's are indeed present in the attached metadata: org.eclipse.sdk.ide org.eclipse.sdk.feature.group org.eclipse.jdt.feature.group Please note that the last one doesn't match your grep since you have a ',' at the end. Same is true for bundles like 'org.eclipse.ui.ide.application', 'org.eclipse.ui.ide.source', etc. They are all present in the metadata. (In reply to comment #9) > It's unclear to me what IU's are are missing. Can you please provide an id to > look for? Based on your grep and earlier comments I checked and found that > these IU's are indeed present in the attached metadata: > > org.eclipse.sdk.ide > org.eclipse.sdk.feature.group > org.eclipse.jdt.feature.group > > Please note that the last one doesn't match your grep since you have a ',' at > the end. Same is true for bundles like 'org.eclipse.ui.ide.application', > 'org.eclipse.ui.ide.source', etc. They are all present in the metadata. The IUs _are_ present but the mirroring chose the SR0 version, not the SR1 version. To get the SR1 versions mirrored, I need to explicitly call them out. Without naming explicit versions, the version choice the aggregator makes is seemingly random: <unit id='org.eclipse.jdt.feature.group' version='3.7.0.v20110520-0800-7z8gFchFMTdFYKuLqBLqRja9B15B' singleton='false'> <unit id='org.eclipse.platform.feature.group' version='3.7.1.r37x_v20110729-9gF7UHOxFtniV7mI3T556iZN9AU8bEZ1lHMcVK' singleton='false'> The p2 planner is handed all the latest features as roots. It creates a transitive scope and comes up with a plan to install it all. If an older feature was chosen, then that's because some other constraint prevents the newer one from being chosen. I made some detective work and found the following 'Category' in SR0: id='Programming Languages' version='0.0.0.9L7M7IcLTAY6EAxdnsawDz-TMd7L' which has the following requirement: name='org.eclipse.jdt.feature.group' range='3.7.0.v20110520-0800-7z8gFchFMTdFYKuLqBLqRja9B15B' Incidentally, the SR1 category looks like this: id='Programming Languages' version='0.0.0.8A7A7DcLTAY6CQpxwt7AqHAlKAAh' Pay special attention to the version qualifier. Do you see that the SR0 version is higher? The SR1 category then also has this requirement: name='org.eclipse.jdt.feature.group' range='3.7.1.r371_v20110810-0800-7z8gFcoFMLfTabvKsR5Qm9rBGEBK' As I recommended earlier, instead of trying to mirror the composite that contains both the SR0 and the SR1, please try to mirror SR1 only. I.e. don't use: http://download.eclipse.org/releases/indigo instead use: http://download.eclipse.org/releases/indigo/201109230900 (In reply to comment #11) > The p2 planner is handed all the latest features as roots. It creates a > transitive scope and comes up with a plan to install it all. If an older > feature was chosen, then that's because some other constraint prevents the > newer one from being chosen. > > I made some detective work and found the following 'Category' in SR0: > > id='Programming Languages' > version='0.0.0.9L7M7IcLTAY6EAxdnsawDz-TMd7L' > > which has the following requirement: > > name='org.eclipse.jdt.feature.group' > range='3.7.0.v20110520-0800-7z8gFchFMTdFYKuLqBLqRja9B15B' > > Incidentally, the SR1 category looks like this: > > id='Programming Languages' > version='0.0.0.8A7A7DcLTAY6CQpxwt7AqHAlKAAh' > > Pay special attention to the version qualifier. Do you see that the SR0 version > is higher? The SR1 category then also has this requirement: > > name='org.eclipse.jdt.feature.group' > range='3.7.1.r371_v20110810-0800-7z8gFcoFMLfTabvKsR5Qm9rBGEBK' > > As I recommended earlier, instead of trying to mirror the composite that > contains both the SR0 and the SR1, please try to mirror SR1 only. I.e. don't > use: > > http://download.eclipse.org/releases/indigo > > instead use: > > http://download.eclipse.org/releases/indigo/201109230900 Thomas, Thanks for your detective work, much appreciated! Inconsistent metadata is...no fun. I'll use the SR1 site and get exactly what I need. I did a search on that exact URL and on "Eclipse Indigo SR1" and only turn up the SR1 site coincidentally in bug reports. Is there a document anywhere that describes the Eclipse update site/mirroring structure? (In reply to comment #12) > I did a search on that exact URL and on "Eclipse Indigo SR1" and only turn up > the SR1 site coincidentally in bug reports. Is there a document anywhere that > describes the Eclipse update site/mirroring structure? There might be. I'll have to pass that question to David Williams. Personally, I usually download the http://download.eclipse.org/releases/indigo/compositeContent.jar and examine its children in order to get the complete structure.
>
> There might be. I'll have to pass that question to David Williams. Personally,
> I usually download the
> http://download.eclipse.org/releases/indigo/compositeContent.jar and examine
> its children in order to get the complete structure.
There's no documentation for the "sub" repositories since that's considered "internal" and not intended for direct use. So Thomas's method of getting the compositeContent.jar and inspecting it is the way to go.
The "sub" repository wouldn't be guaranteed to be as persistent, necessarily, as the top level one, (which is why it is "internal" and not published to be used) but for your case of wanting to snag a mirror, it should never be so transient as to prevent that.
> Pay special attention to the version qualifier. Do you see that the SR0 version
> is higher? The SR1 category then also has this requirement:
I thought this wasn't supposed to matter ... that categories were always simply merged?
If the "merging" is only done in the p2 UI, then sounds like a better publishing solution is required to they are monotonic? Or Would a possible solution be to have the categories themselves as a separate repo and essentially only provide one set of categories for the repo, and ones from previous releases would be "dropped"?
(In reply to comment #15) > If the "merging" is only done in the p2 UI, then sounds like a better > publishing solution is required to they are monotonic? Or Would a possible > solution be to have the categories themselves as a separate repo and > essentially only provide one set of categories for the repo, and ones from > previous releases would be "dropped"? Personally, I've always thought that the whole idea of using installable units to describe categories (which aren't installable at all) is wrong. They do indeed break the general contract for 'planning', especially since they have distinct version requirements on the features that they include. Categorization is IMO orthogonal to installation requirements and can be done in many different ways depending on the users need. The current solution was motivated by the fact no more work was needed. Looking at all confusion that this has caused and all code that since has been written to deal with exceptions caused by theses non-IU IUs, I sincerely doubt that. But that's another topic. That said, I believe that the aggregator actually does merge categories, just like the p2 IU does. I distinctly remember the annoyance of having to put specialized code like that in place. This means that the category versions isn't the likely cause for the problem that Terry encounters. It must be some other meta-data inconsistency or perhaps something that has its roots in bug 362692 or bug 363965. [Bookkeeping change only. Moving bugs to the new "home" of aggregator, CBI. No change to assignee for resolved and verified bugs.] |