Community
Participate
Working Groups
+++ This bug was initially created as a clone of Bug #370405 +++ I'm starting this as a clone of bug 370405 since somewhat related to that bug, but is technically a separate issue. bug 370405 was fixed by removing the 3.7 pointer, but then it was discovered that 4.2 M5 did not contain all required artifacts, such as "p2 discovery" and possibly others. Those missing artifacts are in the 3.8milestones M5 repository so we can "get" them from there, for now, but (apparently) this causes "all" of 3.8 to be be pulled into the staging repo, which is not acceptable long term. It is still not known if that is a bug, or a feature :) of the aggregator (or, my scripts, settings, etc.) but ... while that's being investigated in parallel, I thought best to open this bug so the original "3.7 related" issue can be closed.
My latest "experiment" (after removing Equinox and Eclipse as "trusted repositories") was to try and name just a few bundles in the equinox.b3aggrcon file, in an attempt to get just the (presumably few) bundles we need from that 3.8 M5 repo, but, still, all (or nearly all) the 3.8 repo seems to be being pulled in. This could be a bug in aggregator (or scripts) or, could be a "feature" of aggregator in the way it treats those special, synthetic "root" features? Such as org.eclipse.rcp_root 4.2.0.v20120118-1921-7IAN8ZBrHPKY8yZnllz-UbN88SKo 3.8.0.v20120105-1545-92BkG2FFw3Ez0F0S383nTygo Just a wild guess. The essential part of the equinox contribution file is <repositories location="http://download.eclipse.org/eclipse/updates/3.8milestones/S-3.8M5-201201251800" description="Eclipse and Equinox"> <bundles name="org.eclipse.equinox.p2.discovery"/> <bundles name="org.eclipse.equinox.p2.discovery.compatibility"/> <bundles name="org.eclipse.equinox.p2.ui.discovery"/> .... with "features" disabled .... </repositories> But, if looking at the aggregation log, such as https://hudson.eclipse.org/hudson/view/Repository%20Aggregation/job/juno.runAggregator/258/console You can see a long list of artifacts being mirrored from that 3.8M5 repo, when (according to how I thought aggregator worked, should just be those 3 specified).
Adding Thomas to CC list ... Thomas, any thoughts on why all of 3.8M5 repo is being pulled in? When I'm trying to get just a few things from there? This "aggregator issues" will hopefully be temporary, and eventually the 4.2M6? repo be "complete" ... I'm just looking for a temporary solution, to better gauge (early) if anyone is "accidentally" depending on anything from intentionally 3.8-only stream.
Just to give an update: after much "trial and error" I think the aggregator is working like it is supposed to, and it is some other project or two that is explicitly pulling in the "3.8 platform" into the aggregated repository. I've narrowed it down to 5 or 6 and will continue my trials and errors. (Its hard finding by searching through content.xml file, just because it is so large and complicated ... at least, for me). But, this shows the importance of having a correct, complete 4.2-only repo to work with. It will make it more obvious if/when others are "making a mistake". What is the outlook for getting a proper 4.2-only repo from the Eclipse (and Equinox) teams? I'd suggest an extra early warm-up contribution toward M6 once its ready. If, for some reason, it is going to be a while before such a repository is built, one possible option is that I can make such a repository from M5 contribution, using the b3 aggregator, and then "contribute" that as the platform's early M6 warm-up. Naturally, that wouldn't be the best solution, since a) extra work :) and b) might contain other "inaccuracies"?
I have narrowed it it down to the objectteams contribution that is pulling in the 3.8 platform. Not sure if it is direct or by the special coupling it has with specific version of JDT. Doing a run now on build.eclipse.org, with object team contribtion disabled, to confirm.
I've confirmed removing "objectteams" contribution prevents the 3.8 platform from being pulled in. The list of mulitple bundles at different version numbers, http://build.eclipse.org/juno/simrel/reports/nonUniqueVersions.txt shows more the usual list of suspects .. not ui.workbench, platform, etc. FYI: I opened bug 370677 for objectteams contribution, if anyone wants to follow that specific issue.
One more note, the 4.2 M5 "archived repo" such as at http://www.eclipse.org/downloads/download.php?file=/eclipse/downloads/drops4/S-4.2M5-201201271145/eclipse-4.2-4.2M5-repository.zip does appear to contain 4.2 only artifacts (as would be expected) but does seem to be missing the p2 discovery bundles. Currently, for pre-M6 staging, I'm picking up these 3 p2 discovery bundles (only) from the 3.8 M5 repo: <bundles name="org.eclipse.equinox.p2.discovery" /> <bundles name="org.eclipse.equinox.p2.discovery.compatibility" /> <bundles name="org.eclipse.equinox.p2.ui.discovery" />
(In reply to comment #2) > Adding Thomas to CC list ... Thomas, any thoughts on why all of 3.8M5 repo is > being pulled in? When I'm trying to get just a few things from there? Reading the rest of the comments I guess that perhaps your trial and error has answered those questions? If not, let me know what I can do to help.
(In reply to comment #7) > (In reply to comment #2) > > Adding Thomas to CC list ... Thomas, any thoughts on why all of 3.8M5 repo is > > being pulled in? When I'm trying to get just a few things from there? > > Reading the rest of the comments I guess that perhaps your trial and error has > answered those questions? If not, let me know what I can do to help. Maybe just one "educate me some more" thing. I tried to use "exclusion rules" to "hide" the 3.8 things, such as <mapRules xsi:type="aggregator:ExclusionRule" name="org.eclipse.platform.feature.group" versionRange="3.8.0.v20120103-2134-9gF7hHf2G25BwvMiyqMMfhz04JllcDTwcl9Cqy3"/> <mapRules xsi:type="aggregator:ExclusionRule" name="org.eclipse.rcp.feature.group" versionRange="3.8.0.v20120105-1545-92BkG2FFw3Ez0F0S383nTygo"/> to avoid 3.8 level things from "being considered" in resolving. I was hoping that would result in a nice message such as "objecteams required 3.8 but it can not be found" but instead, just got vague error messages such as the following (and that's even with objectteams dissabled!). So ... what's the secret for using exclusion rules? Any tips? Are they only suitable for true "leaf" components? Thanks. = = = = = = !ENTRY org.eclipse.b3.util 4 -364274674 2012-02-06 02:27:33.308 !MESSAGE Build failed! Exception was org.eclipse.core.runtime.CoreException: Cannot complete the install because some dependencies are not satisfiable !STACK 1 org.eclipse.core.runtime.CoreException: Cannot complete the install because some dependencies are not satisfiable at org.eclipse.b3.aggregator.engine.ValidationSetVerifier.run(ValidationSetVerifier.java:677) at org.eclipse.b3.aggregator.engine.Builder.runRepositoryVerifier(Builder.java:1700) at org.eclipse.b3.aggregator.engine.Builder.run(Builder.java:1631) at org.eclipse.b3.aggregator.presentation.AggregatorActionBarContributor$BuildAggregationAction$1.run(AggregatorActionBarContributor.java:292) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54) !SUBENTRY 1 org.eclipse.equinox.p2.director 0 1 2012-02-06 02:27:33.309 !MESSAGE Cannot complete the install because some dependencies are not satisfiable
This bug will raise its ugly head again for M6, as far as I can tell. The "root" problem is that the 4.2 repo <still> does not contain some commonly requiried equinox bundles: <bundles name="org.eclipse.equinox.p2.discovery" /> <bundles name="org.eclipse.equinox.p2.discovery.compatibility" /> <bundles name="org.eclipse.equinox.p2.ui.discovery" /> To "solve" this problem, the eclipse.b3aggrcon file contributes the "normal" 4.2 milestone repository, but to "make up" for the missing equinox bundles, but equinox.b3aggrcon file contributes the 3.8 milestone repository ... for just those bundles. BUT, the complication, is that "Eclipse" and "Equinox" are considered "trusted repos", so they are not literally copied to the juno milestone repository, but instead, a "compositeArtifacts" file used to combine everything ... thus, other contributions to Juno can "find" things from either repo, as "objectteam" does. for Juno M5, the composite repo looks like this: <children size='3'> <child location='http://download.eclipse.org/eclipse/updates/3.7milestones/S-3.7RC5-201106131736'/> <child location='http://download.eclipse.org/eclipse/updates/4.2milestones/S-4.2M5-201201271145'/> <child location='aggregate'/> </children> And, honestly, not sure why it says "3.7" there ... the b3aggrcon file does say <repositories location="http://download.eclipse.org/eclipse/updates/3.8milestones/S-3.8M5-201201251800" description="Eclipse and Equinox"> <bundles name="org.eclipse.equinox.p2.discovery"/> <bundles name="org.eclipse.equinox.p2.discovery.compatibility"/> <bundles name="org.eclipse.equinox.p2.ui.discovery"/> So, I may have corrected the b3aggrcon file after M5? At any rate, Eclipse and Equinox need to "point to" only one 4.2 repository, that has everything in it. Or else, if desired, I could change the aggregator to no longer consider Eclipse and Equinox as "trusted repos", so no composite artifacts, only what is needed (by Eclipse and Equinox) is "copied over" to Juno, Juno M6 would be a "simple" repo, and objectteam would then _have_ to adjust to "4.2 only" artifacts. The reason we considered them "trusted repos" to begin with is to "save space" ... and the fact that they could be considered "perfect", and "constant" and "reliable". It would be "safer" to not have "trusted repos" and copy everything to one location ... at the cost of a fair amount of increase in size of the common repo. The platform would add roughly .5 gig to the 1 gig common repo. (50% larger). Think it could be fixed? (have all in one repo) or should we plan on having Juno a simple repo (at least for M6, and maybe it'd be a good idea to always have as a simple repo? ... take the safer approach?)
(In reply to comment #9) > The "root" problem is that the 4.2 repo <still> does not contain some commonly > requiried equinox bundles: > > <bundles name="org.eclipse.equinox.p2.discovery" /> > <bundles name="org.eclipse.equinox.p2.discovery.compatibility" /> > <bundles name="org.eclipse.equinox.p2.ui.discovery" /> > To say this better, the real "root problem" is that there are missing features and a "product" from the 4.2 repo ... and these have been missing for all of Juno, and are believed to be "fixed automatically" once 4.2 is "built from scratch": <features name="org.eclipse.equinox.sdk.feature.group" <features name="org.eclipse.equinox.p2.discovery.feature.feature.group" <features name="org.eclipse.rcp.source.feature.group" <products name="org.eclipse.rcp.sdk.id" So, the question(s) are, can those be provided for M6? If not, I think only alternative is to change Juno to a "simple repo" and I think safer to use that for Juno and subsequent service releases, despite the "duplication". I should also say, I spent a week or so experimenting with the simple repo format, and making required changes to scripts and the "simrel repo reports/test" bundle to work with simple repos or the composite artifacts format ... and seemed to work (I did not check reports in detail, there may be some spurious "warnings" reported that would not have been previously) ... but, then "changed back" recently, assuming this would be fixed by M6. So, in other words, could change to simple repo strategy relatively easily (since I've already done a lot of work to make sure we could :) Its a little more than than setting a flag, but 3 or 5 files need minor editing to do one or the other.
From some chat's with Kim, is sounds feasible to have a mirror script (likely needed only for M6) to mirror these features/products from 3.8 repo to 4.2 repo: <features name="org.eclipse.equinox.sdk.feature.group" <features name="org.eclipse.equinox.p2.discovery.feature.feature.group" <features name="org.eclipse.rcp.source.feature.group" <products name="org.eclipse.rcp.sdk.id" I was recalling in earlier milestones there was a problem indicated in aggregator that something didn't exist in 3.8 or something was incompatible ... but, that may have been me misunderstanding something ... or something that has since been fixed, since I changed the b3aggrcon file to "pull" those features (from 3.8) and didn't see any problems (when testing locally, so running a "warm up" build now, still using 3.8 and 4.2 repo, but with those features enabled instead of them disabled and using bundles only. Assuming that works, the "simple mirroring script" might work (to make sure use only one repo in aggregation, and to make sure the 4.2 eclipse repo matches what is in Juno repo). [Might still not be a bad idea to not use "trusted repo", and provide Juno as a simple repo, instead of composited artifacts ... but, for simple "safety" since so many other transitions going on ... but, that'd be a (slightly) different issue].
Yes, David, I'm testing that right now.
So the mirrored repo looks fine. We are running another 4.2 build toward M6, once this is available we can try mirroring the 3.8 equinox stuff to that repo and then seeing how it works in the Juno build. <project default="main"> <target name="main"> <property name="source" value="/home/users/kmoir/repo/3.8/I20120314-1800"/> <property name="destination" value="/home/users/kmoir/repo/4.2/I20120314-2200"/> <p2.mirror source="file://${source}"> <destination kind="metadata" location="file://${destination}" name="my repo" format="file://${source}" /> <destination kind="artifact" location="file://${destination}" name="my repo" format="file://${source}" /> <iu id="org.eclipse.equinox.sdk.feature.group" version="" /> <iu id="org.eclipse.equinox.p2.discovery.feature.feature.group" version="" /> </p2.mirror> </target> </project> Some notes: The org.eclipse.rcp.source (4.2.0.qualifier) and org.eclipse.rcp.sdk.id (4.1.0 qualifier - perhaps product id should be incremented) are from the 4.2 repo. The 4.x stream org.eclipse.sdk feature includes the org.eclipse.rcp feature which includes the org.eclipse.e4.rcp feature.
Here's what I'm currently seeing, with the M6 candidate 1) the 4.2 repo does seem to "have what it needs" in terms of p2 discovery, etc., so I could use just one repo for eclipse and equinox and the composisteArtifacts.jar/xml file correctly has just two entries: <child location='http://download.eclipse.org/eclipse/updates/4.2milestones/S-4.2M6-201203151300'/> <child location='aggregate'/> But, as can be seen in the "non unique versions" report, at http://build.eclipse.org/juno/simrel/reporeports/reports/nonUniqueVersions.txt It appears there is still a lot of "duplicate" versions, coming from the platform, such as org.eclipse.platform.feature.group 3.8.0.v20120103-2134-9gF7iHf2G25BwwUajkz-jd3GgNBuE8PTgeGggBZ 4.2.0.v20120221-1643-9JF7AH9EFwCXKafDVIaXhomQr7ML0EZWz-1ITJmGqfM7b I checked the eclipse M6 repo, and indeed, they are both included there. Is this a problem? Not sure how much of one. I think it is only in the sense of not giving a good test, to see if anyone is still depending on that 3.8 version, instead of (only) 4.2. So, I suggest we count Eclipse M6 repo as good enough, but I'll still switch aggregator to not use composite repo, so it will be clearer then, if anyone is indeed depending on 3.8 platform. As is, the nonUniqueVersions report is, conceptually, just blindly "picking up" everything in 4.2 repo, even if no one is actually using it.
Hmm, perhaps I should run a remove iu operation to remove those 3.8 stream features...this is not really what I intended. I didn't want the 3.8 stream features there but it wouldn't mirror with some more restrictive mirroring options.
(In reply to comment #15) > Hmm, perhaps I should run a remove iu operation to remove those 3.8 stream > features...this is not really what I intended. I didn't want the 3.8 stream > features there but it wouldn't mirror with some more restrictive mirroring > options. I've converted to use "simple" repo, not having the "trusted repo" be a compositeArtifact.jar/xml file. But no joy. Same duplication. http://build.eclipse.org/juno/simrel/reporeports/reports/nonUniqueVersions.txt My concern is that one of the features you are mirroring, org.eclipse.equinox.sdk.feature.group org.eclipse.equinox.p2.discovery.feature.feature.group is what is "pulling in" the 3.8 platform. So, I think we were better off before, "hacking in" the three p2 discovery bundles we needed. My guess is that the remove iu operation probably would not work, since, if it is true one of those mirrored features is pulling in 3.8, then I suspect the remove iu operation would not allow some dependency to be removed, and leave the repo in an inconsistent state. Is it possible to "revert" the milestone to the original 4.2 candidate I-build repo, and I'll just put back in the "hack" that pulls in the three discovery bundles that were required by others before, during M5?
I just put it back :-)
Well, things are better, according to http://build.eclipse.org/juno/simrel/reporeports/reports/nonUniqueVersions.txt but ... still some surprising "mix" of 4.2 and 3.8. org.eclipse.platform_root 3.8.0.v20120103-2134-9gF7iHf2G25BwwUajkz-jd3GgNBuE8PTgeGggBZ 4.2.0.v20120221-1643-9JF7AH9EFwCXKafDVIaXhomQr7ML0EZWz-1ITJmGqfM7b org.eclipse.platform.feature.group 3.8.0.v20120103-2134-9gF7iHf2G25BwwUajkz-jd3GgNBuE8PTgeGggBZ 4.2.0.v20120221-1643-9JF7AH9EFwCXKafDVIaXhomQr7ML0EZWz-1ITJmGqfM7b org.eclipse.rcp.feature.jar 4.2.0.v20120118-1921-7IAO8ZBrHPUY91YehqxSmDN2WLrW 3.8.0.v20120105-1545-92BlG0gFw3Ez0FtOq0z-nU4to And, from "manually" looking at the content.jar/xml files, these do exist in the Eclipse 4.2 repo, not just the Juno aggregation. So ... I am sort of at a loss as to how this came about. Or, what to do about it. I see the M5 milestone had similar mix (from manually peeking at the content.jar/xml file in 4.2milestones/S-4.2M5-201201271145. I will try one more thing via the aggregator. I currently have not specified version ranges, on purpose, thinking that meant "get the highest version" -- and pretty sure that's true. But, who knows, maybe a bug or something odd makes it act like "get all versions" in this case ... so I'll experiment with that to see if (at least) the Juno repo can be fine tuned. But am not optimistic.
(In reply to comment #18) > I will try one more thing via the aggregator. I currently have not specified > version ranges, on purpose, thinking that meant "get the highest version" -- > and pretty sure that's true. But, who knows, maybe a bug or something odd makes > it act like "get all versions" in this case ... That all depends on what the requirements are. If multiple versions must be pulled in in order to satisfy them, and if there are no conflicting 'max' limits, then the planner will create a plan where multiple versions exists. As a test, try setting a range that explicitly selects one of the versions. That will probably tell you where the requirement for the other version comes from since it will make it impossible for the planner to fulfill both.
(In reply to comment #19) > (In reply to comment #18) > > As a test, try setting a range that explicitly selects one of the versions. > That will probably tell you where the requirement for the other version comes > from since it will make it impossible for the planner to fulfill both. I tried adding, in the eclipse contribution, a feature specifically for org.eclipse.platform.feature.group, since that's one still showing up as also getting pulled in at 3.8 level. I specified "minimum" value for it ... and even tried "excluding" 3.8 versions ... but still gets sucked in, as far as I can tell. <features name="org.eclipse.platform.feature.group" versionRange="4.2.0.v20120221-1643-9JF7AH9EFwCXKafDVIaXhomQr7ML0EZWz-1ITJmGqfM7b"/> <mapRules xsi:type="aggregator:ExclusionRule" name="org.eclipse.platform.feature.group" versionRange="[3.8.0.v20120103-2134-9gF7iHf2G25BwwUajkz-jd3GgNBuE8PTgeGggBZ]"/> <mapRules xsi:type="aggregator:ExclusionRule" name="org.eclipse.platform" versionRange="[3.8.0.v201203141800]"/> I convinced myself this isn't merely an eclipse or aggregator problem by creating a new project, with only eclipse and equinox making contributions, and in that case, only 4.2 was pulled in, as would be expected. So, not sure how the planner "succeeds" in full-filling someone else's requirements, with those constraints in place. I'm sure there's an explanation, but I'm not seeing it.
FYI, as before, if I "exclude" the objectteam contribution, then 3.8 platform does not be "pulled in" to Juno repository, so have reopened bug 370677.
When you did the test with the minimum version, did you use any trusted repositories? Or multiple validation sets? I'm asking because that's the only ways I know to either bypass the planner or invoke it more than once.
(In reply to comment #22) > When you did the test with the minimum version, did you use any trusted > repositories? Or multiple validation sets? I'm asking because that's the only > ways I know to either bypass the planner or invoke it more than once. No, no trusted repos. Only one validation set. I am about to make more comments in bug 370644 -- since not so related to this one, which should be about the correctness of the Eclipse 4.2 repo -- but no "answers" exactly, just narrowing it down a bit.
(In reply to comment #21) > FYI, as before, if I "exclude" the objectteam contribution, then 3.8 platform > does not be "pulled in" to Juno repository, so have reopened bug 370677. x-ref: I've moved bug 370677 to b3 because I could narrow the undesired behavior down to being caused by a 'patch="true"' directive in a feature.xml. I believe this is a tool bug, nothing we can fix in the Object Teams repo!
May I ask permission to re-enable the Object Teams contribution for Juno M6? After those experiments in bug 370677 I don't see, how pulling in platform_3.8 could be our fault, nor how we could prevent it. Object Teams has no dependency, direct or indirect, on platform_3.8. The problem could likely be avoided by either: - using SDK and Equinox repos that *don't* contain the undesired platform_3.8 or - fixing b3 aggregator so that 'patch="true"' does not affect the aggregation in this way. Maybe we can postpone the goal to be free from 3.8 artifacts until M7?
One interesting observation is that the eclipse/updates/4.2xxx repositories all contain both the 3.8 and the 4.2 platforms. I'm sure there's a reason for that. I also think that if it does, then what's the harm in the SimRel doing that too?
(In reply to comment #25) > May I ask permission to re-enable the Object Teams contribution for Juno M6? > No ... because you do not even need to ask permission! :) Feel free to re-enable it when ever you'd like. I removed it merely to "prove" that there is something about that contribution that pulls it in ... or interacts with p2 or aggregator to cause it to be pulled in. I'd still like to think there was a way to to have a fix and have a "purer" M6 ... but ... doesn't sounds like its anything obvious.
(In reply to comment #27) > (In reply to comment #25) > > May I ask permission to re-enable the Object Teams contribution for Juno M6? > > I have reenabled the object teams feature (just now).
There's no plan to fix the old milestone repos, and this issue will be fixed by building both 3.8 and 4.2 "from scratch".
