Bugzilla – Full Text Bug Listing
| Summary: | can not "aggregate" Orbit | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Tools] Orbit | Reporter: | David Williams <david_williams> | ||||||||
| Component: | bundles | Assignee: | Orbit Bundles <orbit.bundles-inbox> | ||||||||
| Status: | NEW --- | QA Contact: | |||||||||
| Severity: | minor | ||||||||||
| Priority: | P3 | CC: | gunnar, thomas | ||||||||
| Version: | unspecified | ||||||||||
| Target Milestone: | --- | ||||||||||
| Hardware: | PC | ||||||||||
| OS: | Linux | ||||||||||
| Whiteboard: | |||||||||||
| Attachments: |
|
||||||||||
|
Description
David Williams
The filter is in place because the bundle com.google.gwt.user must not be used at runtime. It's only of use at compile time for compiling GWT Java code into JavaScript. At runtime, the bundle com.google.gwt.servlet should be used. For 2.1.0 the filter has been replaced with an invalid p2 requirement (via p2.inf). I see two options: * update 2.0.3 t also use the p2.inf file * pull 2.0.3 from the repo Let's remove it, at least from "active builds" for Juno ... since there is a 2.1.0 version, and a 2.0.4 version, I think that fits in our "retention policy". I checked and doesn't appear to be in any of the "common repos" so doubt "in active use" ... and actually, none of the versions appear in the common repos (which doesn't mean someone doesn't use it, just FYI). One last question, to educate me, how was this filter added? [I'm assuming, hoping, 2.0.4 will not have any filter and not have the p2.inf file?] I'm just wondering if 2.0.3 is removed, if we'll have trouble with 2.0.4? FYI, I've commented out this bundle, at 2.0.3 level, from feature.xml. So, later today I can tell if then we can "aggregate" or if we'll just run into something else. (And, would like to know, still, how the filter is added to begin with). Thanks (In reply to comment #3) > One last question, to educate me, how was this filter added? [I'm assuming, > hoping, 2.0.4 will not have any filter and not have the p2.inf file?] The OS filter must be added to the MANIFEST.MF file. The PDE editor provides an input field for that on the "Overvie" tab. (In reply to comment #5) > (In reply to comment #3) > > One last question, to educate me, how was this filter added? [I'm assuming, > > hoping, 2.0.4 will not have any filter and not have the p2.inf file?] > > The OS filter must be added to the MANIFEST.MF file. The PDE editor provides an > input field for that on the "Overvie" tab. Ok, thanks. I see now the reason I couldn't find it is that the "head" version of the 203 branch does not have it, even though the tagged version, v20100419-0751, does have it. Any reason for that? FWI, without "203" in the build, it gets further, but still ends up with similar problem for the 210 version ... as you've said, it is meant to be "restricted". I'm wondering ... do we need a "runtime" category? As we do for the releases themselves? Oh, meant to paste in whole message, that I get locally:
Cannot complete the install because one or more required items could not be found.
Missing requirement: Google Web Toolkit User Library (with source) 2.1.0.v20101206-1430 (com.google.gwt.user 2.1.0.v20101206-1430) requires 'com.google.gwt.user com.google.gwt.user.browser.runtime [1.0.0,2.0.0)' but it could not be found
InstallableUnit(com.google.gwt.user.browser.runtime [1.0.0,2.0.0)) is required by:
ValidationSet(main)
Contribution(Oribt Build Machine)
MappedRepository(http://build.eclipse.org/orbit/committers/orbit-I/20111128154457/I20111128154457/repository/)
InstallableUnit(org.eclipse.orbit.build.feature.set1.feature.group 1.0.0.v201111281525-ExFEfFDIJHSV8c8esdJ9FlBNSg9K)
InstallableUnit(com.google.gwt.user 2.1.0.v20101206-1430)
Adding Thomas Hallgren in case he'd have some advise on how to make Orbit repo "aggregator friendly". Thomas, so summarize, our one feature in Orbit is never intended to be used to install anything, but bundles "pulled" into other features. Can/how someone set up an aggregator to "mirror" the repo? In this case, the c.g.gwt.user bundle has this special "constraint" in its p2.inf file: # this file tries to prevent installation into runtimes requires.0.namespace = com.google.gwt.user requires.0.name = com.google.gwt.user.browser.runtime requires.0.range = [1.0.0,2.0.0) Would we _have_ to have a separate feature for it, to be used with aggregator (and, not crazy about that idea, doubt we would) ... hoping there's some easy solution, similar to how we do "runtime" features in main releases. ... I forget the "special" constraint they use? Created attachment 207615 [details]
example aggregator file
In case it helps, the attachment is example of the type of aggregator file I've been using. It points to a specific (temporary) repo on build machine, and needs a "platform" repo on local machine (since some of our bundles required "system.bundle") or other things "in the platform runtime").
FWIW, the "secret sauce" used in release train to prevent "IDE install" issues is: <required namespace='A.PDE.Target.Platform' name='Cannot be installed into the IDE' range='0.0.0'/> That might only matter if on a feature? And, not sure, Gunnar, did you pick your "key namespace" for a reason? Or just one that didn't exist ... in other words, could you change to the "A.PDE.Target.Platform' version if it helped, or are there dependencies on you using your current values? (In reply to comment #11) > ... > <required namespace='A.PDE.Target.Platform' name='Cannot be installed into the > IDE' range='0.0.0'/> > > That might only matter if on a feature? > For what its worth, I "hacked in" this "A.PDE.Target.Platform" into a local content.jar/xml file I had, and the aggregator did in fact "work" ... at least, it got farther then it did, getting past that "not found" problem. Next problem was that glassfish requires jdt compiler, so needed to add JDT repo also my list of "pre-req" repos ... no biggie. But, then another, worse problem ... only one version of singletons's allow!? (full message below). We "get around" this in Orbit builds, I think, by specifying resolution.devMode=true I tried adding -Dresolution.devMode=true to my eclipse that's running the aggregator ... but, same problem. I am a little surprised to see this, since, for example, we sometimes have "more than one version" in release stack ... but ... I'm guessing ... never in same feature. Suggestions welcome. = = = = = Full error message: Cannot complete the install because of a conflicting dependency. Only one of the following can be installed at once: International Components for Unicode for Java (ICU4J) 4.0.1.v20100112 (com.ibm.icu 4.0.1.v20100112) International Components for Unicode for Java (ICU4J) 4.2.1.v20100412 (com.ibm.icu 4.2.1.v20100412) International Components for Unicode for Java (ICU4J) 4.4.2.v20110823 (com.ibm.icu 4.4.2.v20110823) bundle(com.ibm.icu 4.2.1.v20100412) is required by: ValidationSet(Orbit Build) Contribution(Orbit Build) MappedRepository(file:///home/shared/orbit/committers/orbit-I/20111128152919/I20111128152919/repository) InstallableUnit(org.eclipse.orbit.build.feature.set1.feature.group 1.0.0.v201111281525-ExFEfFDIJHSV8c8esdJ9GfBPSg9K) bundle(com.ibm.icu 4.4.2.v20110823) is required by: ValidationSet(Orbit Build) Contribution(Orbit Build) MappedRepository(file:///home/shared/orbit/committers/orbit-I/20111128152919/I20111128152919/repository) InstallableUnit(org.eclipse.orbit.build.feature.set1.feature.group 1.0.0.v201111281525-ExFEfFDIJHSV8c8esdJ9GfBPSg9K) (In reply to comment #12) > For what its worth, I "hacked in" this "A.PDE.Target.Platform" into a local > content.jar/xml file I had, and the aggregator did in fact "work" ... at least, > it got farther then it did, getting past that "not found" problem. We should likely roll this out then! > And, not sure, Gunnar, did you pick your "key namespace" for a reason? Or just > one that didn't exist ... in other words, could you change to the > "A.PDE.Target.Platform' version if it helped, or are there dependencies on you > using your current values? I picked just one that didn't exist. Frankly, I didn't think of "A.PDE.Target.Platform" back then. I've opened two feature enhancement requests on aggregator as a result of this "investigation". Bug 365004 - "runtime only" namespace should be property and list Bug 365006 - aggregator should be more forgiving if osgi.resolverMode=development specified I think that as things stands right now we should ensure that all features and bundles that are not intended to be installed into the IDE should either use the *same workaround* to prevent it, or use a *proper solution*. The workaround is to add the infamous requirement to "A.PDE.Target.Platform". So please change any other requirements that has the same purpose to this. OR Use a proper solution which could be to use a negative requirement to something that will only be found in the IDE, such as the IDE product itself. When the proper solution is used, then the aggregator must use two validation sets for Juno. One where the IDE product is present and one that contains all features not intended for the IDE. It might be convenient to have a third, abstract validation set, that would the two would extend from. Gunnar, for M4 I'd like to get this at least partially fixed, by changing the p2 meta data to following ... so will check in this change soon. # this file generates p2 metadata to help prevent installation into IDEs. It is intended for runtimes only. requires.0.namespace = A.PDE.Target.Platform requires.0.name = Cannot be installed into the IDE requires.0.range = 0.0.0 Depending on how bug 365004 evolves, there might be a different/better solution later, but doubt that'll happen in time for M4 (Or, that is, for Orbit to react to M4 changes ... since we go first ... this way we'll at least be consistent). Thanks David! That's fine with me and consistent with the workarounds elsewhere. ;) (In reply to comment #17) > Thanks David! That's fine with me and consistent with the workarounds > elsewhere. ;) For what its worth, I tagged with v201111291941 and have submitted for a build. 2 steps forward, one step back ... With the latest fixes AND the latest additions ... http://download.eclipse.org/tools/orbit/downloads/drops/I20111129194920/repository/ ... the aggregator now "fails" with this message due to "platform constraints" I'm assuming we'd need a "platform feature" for each special case ... so ... makes it all the more unlikely Orbit will ever be "aggregator friendly" .... unless someone has some insights I'm not thinking of. = = = = Cannot complete the install because one or more required items could not be found. Missing requirement: Orbit Build Feature 1.0.0.v20111129-0920-F71FEfFDgJM8oIrQuz03JLwW8NSg (org.eclipse.orbit.build.feature.set1.feature.group 1.0.0.v20111129-0920-F71FEfFDgJM8oIrQuz03JLwW8NSg) requires 'org.apache.commons.daemon.linux.x86 [1.0.5.v20111129-0900]' but it could not be found InstallableUnit(org.apache.commons.daemon.linux.x86 [1.0.5.v20111129-0900,1.0.5.v20111129-0900]) is required by: ValidationSet(Orbit Build) Contribution(Orbit Build) MappedRepository(http://build.eclipse.org/orbit/committers/orbit-I/20111129194920/I20111129194920/repository/) InstallableUnit(org.eclipse.orbit.build.feature.set1.feature.group 1.0.0.v20111129-0920-F71FEfFDgJM8oIrQuz03JLwW8NSg) Created attachment 207872 [details] better aggregator file This aggregator is better defined, and based on Orbit "M4" ... though still assumes platform repos are local. I've found that instead of mirrroring based on features, we can mirror based on categories ... and, we do have one category that says "all bundles". Something is "off" (probably only taking "most recent versions?) since the result is is 704 bundles (jars and packgz files) whereas the full repo has 1062 bundles (jars and pack.gz files). But, even this "kludge" discovered a few new problems that could maybe be fixed in Orbit, see bug 365340, so I think the effort is worth while. To confirm, yes, mirroring the "all bundles" category does just pick the most recent version. Not sure if there is a way to get it to "pick all" ... I couldn't see if if so. But, I also tested the "produce maven repo" and seemed to work, producing the characteristic "maven-p2" side-by-side repo that looks like artifacts.jar ch com commonj content.jar edu java_cup javax lpg me net org osgi p2.packed Of I've not tested if these repo "works" ... just saying the aggregator produces it. Pretty cool! The aggregator uses the p2 planner to come up with a plan that is consistent. In essence, this is the same as asking the planner to install the selected material. This is why a lot of duplicates are discarded. The aggregator is capable of aggregating several such plans (Validation Sets) but that in turn implies that it has some roots to work with. Just passing it the whole orbit repository will not suffice. It would then need some root features that describes what bundles to include in each set. (In reply to comment #22) > The aggregator uses the p2 planner to come up with a plan that is consistent. > In essence, this is the same as asking the planner to install the selected > material. This is why a lot of duplicates are discarded. > > The aggregator is capable of aggregating several such plans (Validation Sets) > but that in turn implies that it has some roots to work with. Just passing it > the whole orbit repository will not suffice. It would then need some root > features that describes what bundles to include in each set. Does it literally need "root features" or could it be "list of bundles (with versions)"? If the former, that'd mean we'd need to change the way we we do things in Orbit pretty drastically. If a "list of bundles" suffices, there might be some hope of automating the generation of the lists, for subsequent use in b3 aggregator. The reason for going through all that effort would have to be thought through ... so far, I see two, maybe three advantages ... a) better check on pack.gz files, b) possible generation of side-by-side maven repository, and c) possibly ending up with a repository with no "features" in it, at all. Currently our "build features" do end up in our Orbit p2 repository, which is sort of incorrect ... since those features can't be "installed" into anything. Thanks as usual for the discussion. Created attachment 207877 [details]
aggregator file, showing bundles in multiple validation sets
To answer my own question, does not need to be features ... could define validation sets with just bundles ... having "old" versions in different validation sets. Then, each validation set "validates" and they all end up being aggregated to the same repo. In the attached example, I just used two bundles as example test cases:
<bundles name="com.ibm.icu" versionRange="[4.4.2,4.4.3)"/>
<bundles name="com.ibm.icu" versionRange="[4.0.1,4.0.2)"/>
So ... if anyone every thought it worthwhile, we probably could generate appropriate ValidationSets automatically during those same parts of our builds that produce team project sets, and p2 map files ... just a little "multipass loop" to generate validation sets until all versions accounted for. There would be 3 or 4 validation sets, I suspect. (An interesting side effect might be that it'd highlight if we were generating TOO many old versions that should be "retired" from active builds.
Might take a day of work, so not trivial, but, not impossible, if it was ever decided to be worth while.
I'm not sure how the "categories" would be handled/treated. We currently auto-generate those too during our build. Might need to put those in a separate repo ... or, "redefine" them in the aggregator file we auto-generate? Sounds like a lot of work :)
Oh, even more work ... "source bundles" might be another complication. Sometimes we have them, sometimes not ... so, we'd have to make sure "matching" source ended up in same validation set as its "host" bundle. Not that hard, since I suspect we always use ".source" to identify them ... but ... adds a little complication to it.
Sounds more like 3 day of work now :)
Doing a mass "reset to default assignee" of 21 bugs to help make clear it will (very likely) not be me working on things I had previously planned to work on. I hope this will help prevent the bugs from "getting lost" in other people's queries. Feel free to "take" a bug if appropriate. |