Bugzilla – Full Text Bug Listing
| Summary: | regenerating /releases/galileo to remove one project | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Community | Reporter: | David Williams <david_williams> | ||||||||||
| Component: | Cross-Project | Assignee: | David Williams <david_williams> | ||||||||||
| Status: | RESOLVED FIXED | QA Contact: | |||||||||||
| Severity: | normal | ||||||||||||
| Priority: | P3 | CC: | john.arthorne, kim.moir, mik.kersten, mknauer, pwebster, ruediger.herrmann, steffen.pingel, tjwatson, wayne.beaton | ||||||||||
| Version: | unspecified | ||||||||||||
| Target Milestone: | --- | ||||||||||||
| Hardware: | PC | ||||||||||||
| OS: | Windows XP | ||||||||||||
| Whiteboard: | |||||||||||||
| Attachments: |
|
||||||||||||
|
Description
David Williams
If I understand the notes correctly, the following is the minimum list that should be removed right away: net.sf.cglib_2.1.3.v200906161300.jar org.apache.axiom_1.2.5.v200906161300.jar org.apache.cxf.cxf-bundle_2.1.4.v200906161300.jar org.apache.cxf.cxf-rt-bindings-jbi_2.1.4.v200906161300.jar org.apache.cxf.cxf-rt-transports-jbi_2.1.4.v200906161300.jar org.apache.servicemix.cxf.binding.nmr_4.0.0.v200906161300.jar org.apache.servicemix.cxf.transport.nmr_4.0.0.v200906161300.jar org.apache.servicemix.cxf.transport.osgi_4.0.0.v200906161300.jar org.apache.servicemix.document_1.0.0.v200906161300.jar org.apache.xbean.xbean.classloader_3.5.0.v200906161300.jar org.apache.xbean.xbean.spring_3.5.0.v200906161300.jar org.codehaus.stax2_3.2.7.v200906161300.jar org.jvnet.staxex_1.0.0.v200906161300.jar org.objectweb.howl_1.0.1.1_v200906161300.jar org.springframework.aop_2.5.6.v200906161300.jar org.springframework.beans_2.5.6.v200906161300.jar org.springframework.context_2.5.6.v200906161300.jar org.springframework.core_2.5.6.v200906161300.jar org.springframework.osgi.core_1.2.0.rc1_v200906161300.jar org.springframework.osgi.extender_1.2.0.rc1_v200906161300.jar org.springframework.osgi.io_1.2.0.rc1_v200906161300.jar servicemixcommon_2009.1.0.v200906161300.jar servicemixhttp_2009.1.0.v200906161300.jar servicemixsoap2_2009.1.0.v200906161300.jar servicemixsoap_2009.1.0.v200906161300.jar servicemixutils_1.1.0.v200906161300.jar One complication, just to document it, is that once /releases/galileo is touched, the number of available mirrors will drop to zero until the change propagates. (At least, that's my understanding). Kind of a shame removing files would invalid mirrors (since, presumably, would not effect things that are still there, that people would want to install ... but, I think this is the way it has to be as long as we "key" mirror sites based on directory names (rather than file names, such as artifacts.jar/xml). Also, there's a step webmasters have to do to package up and distribute the discovery site to Amazon Web Services, from what I understand (i.e. not a normal mirror that just rsync's the contents). I'm waiting to hear if, due to these complications, there's a better or worse time to make modifications to /releases/galileo. As mentioned on the planning mailing list, it's not quite as simple as using the released tagged version of Galileo .build file (which were tagged with RCa). The problem is that some URLs have changed, having been in a "temporary" site, but now in a permanent site. Worse, 3 projects have changed their feature IDs in their current .build files, and I tried to use the old feature IDs, but they no longer could be found. Thomas Hallgren had a good suggestion; branch the RCa version, and change all the site URLs to /releases/galileo so we'd basically "recreate" the site, using itself as source, but minus the Swordfish contribution of course. This is working fairly well, with the following interesting exceptions: 1) turns out, TPTP build file mentioned some bundles for IPF architecture, and these never were mirrored to Galileo to begin with, so caused an error when trying to re-create from Galileo. Easy enough to remove them, and, I guess, it could be argued they couldn't have been here, if not used (but, I can see the site that would want to be prepared, in case we ever did start mirroring IPF fragments. 2) There were two projects, WTP and SVN, that took advantage of the feature in the builder that will assemble (pull) capability plugins from a capability feature, but the feature itself is not mirrored (only the capability plugins). This could be fixed, in this case, by allowing these two projects to point back to their "home" site to still get the capability features from there. Once I regenerate using this approach, I'll do a diff with released version and see if it matches (except for the removed features and plugins). Created attachment 140548 [details]
output from diff
I've re-created a 'staging' version of Galileo. I diffed it with /releases/galileo and for the most part made sense (i.e. showed swordfish features and bundles gone) but there are some disconcerting issues. Its hard to tell if here is/was issue with original /releases/galileo or the current process.
For example,
org.apache.commons.logging_1.0.4.v200904062259.jar.pack.gz "differs" but since exact same version, they should be identical.
com.ibm.icu.base_4.0.1.v20090415.jar is no longer pulled in ... but, odd that it would have been in the first place (but, even if odd, not sure if it will effect anyone).
Another oddity is that there are several cases similar to:
original has: javax.servlet_2.5.0.v200806031605.jar
staging has: javax.servlet_2.5.0.v200806031605.jar.pack.gz
If we were going to mirror it at all, the staging site is correct (using pack.gz) but I'm surprised we mirror it at all either time ... seems it should be part of the "trusted" contribution of Eclipse,Equinox. So, I don't understand where the pack.gz version came from.
So ... some study is required.
I should have mentioned, for documentation if nothing else, I'm working with the build model in a branch named 'galileo_unrelease_swordfish'. That's where I've changed the URLs to point to /releases/galileo, etc. Created attachment 140559 [details]
diff of unzipped logging jars
For the record, in those cases where diff reports some "pack.gz" files differ, it appears only related to the signing of the versions that end up getting pulled in. I say "only" since they both verify fine (valid signatures) but doing a "diff" on the unpacked, unzipped jars reports some odd, but minor differences in signature related lines. I'm not sure what causes the differences ... probably something like re-signing with a different version of a VM or similar ... but, since they 'verify' ok, I think those differences do not matter.
Created attachment 140564 [details]
writeup of other comparisons of "diffs"
This attachment contains my notes while investigating 5 other cases of "things being different". While there are mysteries, I don't see anything that invalidates the general approach nor the current /releases/staging repo.
While 'staging' is not a perfect reproduction of what we had before, it should work as it did before, from what I can tell.
Unless others have some comments or suggestions to improve the 'staging' repo, I think the next sanity check should be for someone to first install everything (except swordfish) from /releases/galileo, starting of course, with a fresh 'platform' and then second, starting fresh again, install everything from /releases/staging. And then diff the two installs. The features and plugins directories should be identical.
Any volunteers?
If this sanity check passes, I'd say it'd be reasonable to ask projects to do some amount of re-testing ... say over next week or two? (Holidays don't make it easy).
Any other ideas?
I want to document too, that we could try re-spining against current project repos (not /releases/galileo) but 3 projects have already changed their feature versions, so I'm pretty sure this would lead to more differences, even if it cleared up some other differences. My guess is these incubating projects might be used in the Modeling package, which would then cause more churn there. Specifically: Currently in .build files; <features id="org.eclipse.emf.ecoretools.sdk" version="0.9.0.v200906221231" repo="//@repositories.0"> <features id="org.eclipse.emf.mint.sdk" version="0.8.0.v200906161513" repo="//@repositories.0"> <features id="org.eclipse.uml2tools.sdk" version="0.9.0.v200906190654" repo="//@repositories.0"> But, in Galileo repo: <features id="org.eclipse.emf.ecoretools.sdk" version="0.9.0.v200906031210" repo="//@repositories.0"> <features id="org.eclipse.emf.mint.sdk" version="0.8.0.v200906110922" repo="//@repositories.0"> <features id="org.eclipse.uml2tools.sdk" version="0.9.0.v200906031456" repo="//@repositories.0"> But, just documenting here in case others want to explore or discuss. I have deleted those files listed in comment #1 from /releases/galileo Once that deletion propagates to all mirrors (and AWS) we'll be "in compliance" and presumably Swordfish (and hopefully only Swordfish) will give users some odd, frustrating errors if they try to install it. I can't do much to sanity check the /releases/staging directory until bug 282080 is fixed. I've done the sanity check mentioned in #7 by installing everything from /releases/galileo (except swordfish) and then (separately) installing everything from /releases/staging. There was no difference in the resulting features and plugins installed in those two instances. Note: I did not quite install everything. There is a RAP SDK that is "PDE Target Only" and I did not install or verify that one. I'll leave to others. I suggest a one week period for others to test (as they like and as they are able) and if no complaints by that one week deadline, we'll move /releases/staging to /releases/galileo on evening of 7/8. Thanks David. On the "where's ICU mystery", I did discover that one of the pack.gz files on the RAP site is invalid, see bug 282318, but that would not exactly explain the results I'm seeing (since didn't include that site, in my original run against the /releases/galileo site). Still not sure exactly how ICU is not being mirrored now ... I do suspect its the RAP SDK that "includes" it ... as I have everything else installed, and only the RCP-feature has it "included". And, it should not literally try to "get" it from the eclipse trusted site, if I understand the builder. (In reply to comment #12) Just another weired observation - I was looking for the ICU files in the Eclipse Platform repository (see path below) and it turns out that all the .pack.gz files are empty@: mknauer@build:~/downloads/eclipse/updates/3.5/R-3.5-200906111540/plugins> ls -l *icu* -rw-r--r-- 1 sdimitro eclipseadmin 5787480 2009-06-22 09:24 com.ibm.icu_4.0.1.v20090415.jar -rw-r--r-- 1 sdimitro eclipseadmin 0 2009-06-22 09:24 com.ibm.icu_4.0.1.v20090415.jar.pack.gz -rw-r--r-- 1 sdimitro eclipseadmin 69536 2009-06-22 09:24 com.ibm.icu.base_4.0.1.v20090415.jar -rw-r--r-- 1 sdimitro eclipseadmin 0 2009-06-22 09:24 com.ibm.icu.base_4.0.1.v20090415.jar.pack.gz -rw-r--r-- 1 sdimitro eclipseadmin 113440 2009-06-22 09:24 com.ibm.icu.base.source_4.0.1.v20090415.jar -rw-r--r-- 1 sdimitro eclipseadmin 0 2009-06-22 09:24 com.ibm.icu.base.source_4.0.1.v20090415.jar.pack.gz -rw-r--r-- 1 sdimitro eclipseadmin 1582720 2009-06-22 09:24 com.ibm.icu.source_4.0.1.v20090415.jar -rw-r--r-- 1 sdimitro eclipseadmin 0 2009-06-22 09:24 com.ibm.icu.source_4.0.1.v20090415.jar.pack.gz (In reply to comment #13) > (In reply to comment #12) > Just another weired observation - I was looking for the ICU files in the > Eclipse Platform repository (see path below) and it turns out that all the > .pack.gz files are empty@: > Yes, that bug is being tracked in bug 272140. Created attachment 141042 [details] correct attachement for commnent #7 I see I attached the wrong "notes" to comment #7 (it was just a list of diffs), so here is the summary, raw notes from the differences I saw. For what it's worth, I did regenerate the staging site, since Kim and Thomas and John fixed the platform's site and surgically removed the invalid entries for the icu packed files. This allowed a "green" build. and those three icu files were the only differences from previous staging site. (And, again, technically, they don't really have to be there ... but, the point was to try and reproduce the site exactly as it was ... minus the one project. Another complication was to tweak several .build files so as to not re-build the capabilities plugins. Thomas explained how I could "spoof" the build to simply include what was built before, instead of rebuild it. I tagged the final edited .build files with RC5b. (RC5a was the original Galileo release). The final Hudson log is https://build.eclipse.org/hudson/view/Galileo/job/galileo.runBuckyBuild/626/ Comment on attachment 140564 [details]
writeup of other comparisons of "diffs"
I attached wrong file.
I'm done. Swordfish removed. Site seems to work ok, though will be a while before mirrors re-populate. |