Bugzilla – Full Text Bug Listing
| Summary: | Mirroring helios release repository fails with MD5 errors on some jars | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Technology] CBI | Reporter: | Micah Hainline <micah_hainline> | ||||
| Component: | CBI p2 Repository Aggregator | Assignee: | Project Inbox <b3.aggregator-inbox> | ||||
| Status: | RESOLVED FIXED | QA Contact: | |||||
| Severity: | normal | ||||||
| Priority: | P3 | CC: | aniefer, david_williams, irbull, Kenn.Hussey, pascal, rsternberg, thomas, wolfgang.haug | ||||
| Version: | unspecified | ||||||
| Target Milestone: | --- | ||||||
| Hardware: | PC | ||||||
| OS: | All | ||||||
| Whiteboard: | |||||||
| Attachments: |
|
||||||
|
Description
Micah Hainline
A workaround should be to set the java system property "eclipse.p2.MD5Check=false". Beyond that, this seems to be an issue with the repository itself, likely that the jars were modified (perhaps signing/packing) without updating the artifacts.xml. p2 does provide an ant task "p2.process.artifacts" (http://help.eclipse.org/galileo/index.jsp?topic=/org.eclipse.platform.doc.isv/guide/p2_repositorytasks.htm) that can fix such repos. I'll see if I can reproduce. Sure seems odd that it'd be a problem with /releases/helios repository, since it is produced using p2 mirroring-like API and has several integrity checks built into its production. Micah, I assume your M20100120-0800 is "Galileo SR2"? (In reply to comment #2) > Micah, I assume your M20100120-0800 is "Galileo SR2"? In this case I'm using the stand-alone p2 director downloaded from buckminster. I pulled the build id from the config.ini. http://www.eclipse.org/downloads/download.php?file=/tools/buckminster/products/director_latest.zip http://www.eclipse.org/buckminster/downloads.html > Sure seems odd that it'd be a problem with > /releases/helios repository, since it is produced using p2 mirroring-like API > and has several integrity checks built into its production. I did just finish mirroring helios using eclipse.p2.MD5Check=false and was successful. I then was able to finish my build using my new mirror and everything went perfectly. Thanks for the workaround Andrew! Thomas, I was able to reproduce this, using p2.mirror ant task, from both a Galileo SR2 (eclipse sdk) and 3.6M7. I "hit" it on different bundles than reported here (o.e.m2m.qvt.oml.doc), and not sure how/if I can set "verbose" or "ignoreErrors" on ant task, so it stopped on first error. [Also, got funny issues from, apparently, bad mirrors, but set eclipse.p2.mirrors=false and then got these check sum errors)]. My question is, how can this happen? Does the builder just "copy" checksums provided by projects? Or does it recompute? The later being a big bug on "us", the former is more an issue of how to detect and report problem back to contributing project. Does this "interact" with our current practice of unpacking pack.gz files? Advice? Thanks, And, thanks for reporting, Micah. I did some investigation and I think I know what's going on. The jars in question exists as both normal jars and as pack200 jars. My first observation was that the jar isn't normalized (packed/unpacked) prior to being packed. This means that the unpacked version will differ slightly from the original. My second observation was that the file isn't signed. If it had been signed, the aggregator verifier would have complained about "file has been tampered" when unpacking the pack200 file. My third observation indicates a bug in the aggregator. When both a normal jar and a packed jar exists, and when we use the 'unpack as sibling' strategy, then we only copy the pack200 file (which is OK) and then we simply copy both aritfact descriptors. And there we have it. The artifact descriptor contains the MD5 sum. This, in combination with an unsigned and never normalized pack200 file, gives us this error. The aggregator could be improved to detect this problem by doing an MD5 check on the unpacked file and verify that the checksum is indeed the same as the original jar from the source repository. If it isn't, then it's apparent that the packed file wasn't normalized and perhaps issue a warning. It should definitely update the descriptor with the new MD5 checksum. (In reply to comment #5) > The aggregator could be improved to detect this problem by doing an MD5 check > on the unpacked file and verify that the checksum is indeed the same as the > original jar from the source repository. If it isn't, then it's apparent that > the packed file wasn't normalized and perhaps issue a warning. It should > definitely update the descriptor with the new MD5 checksum. David, can you confirm that jars that produce this error aren't signed? In general an incorrect md5 sum doesn't necessarily imply an unsigned jar, since the signing looks at the contents and not the jar container itself. I seem to recal that the different jarsigner and pack tools actually left the jar in slightly different zip formats wrt crcs and file sizes and such. The JarProcessor code used by the foundation to sign does try to "normalize" a jar to avoid these differences since it works recursively on nested jars. > > David, can you confirm that jars that produce this error aren't signed? > Not quickly nor easily nor reliably with respect to exact current. I suspect the first step should be to fix builder/aggregator to use the right CRCs or jars since that's a known problem, then add a "unit test" to make sure the repository can be mirrored. I've opened bug 312976 to track that fix. One thing that concerns me about the original report in comment 0 is that is says some jars were missing, such as "org.apache.commons.discovery, org.apache.log4j, and org.eclipse.wst.internet.monitor.core". I'm nearly certain those would have been correctly conditioned and signed, At least, as delivered by Orbit or WTP ... though of course, some other project might be contributing them also, and "messing them up" somehow. That'll be tough to track down. So ... not sure what to do with this bug. For now could say its a dup of bug 312976 and we'll try to track down the "project level" problems once that is fixed? (In reply to comment #7) > One thing that concerns me about the original report in comment 0 is that is > says some jars were missing, such as "org.apache.commons.discovery, > org.apache.log4j, and org.eclipse.wst.internet.monitor.core". That's probably because the entire mirror process was killed by the bad MD5 check. I'm sure those particular plugins are fine, and would have been downloaded without problem if not for the failure of one of the previous plugins. (In reply to comment #6) > David, can you confirm that jars that produce this error aren't signed? > Sorry if I was unclear. I did verify that. > In general an incorrect md5 sum doesn't necessarily imply an unsigned jar, No, and that's not the issue. The problem is that we don't copy both the pack200 and the normal jar. We only copy the pack200 file. Then we unpack it and trust that the MD5 checksum of the unpacked jar is in accordance with the MD5 checksum of the normal jar in the source. In this case it isn't since the pack200 file isn't normalized. The signing isn't really relevant. I'm just pointing out that if the file had been signed, it would have been apparent at an earlier stage that it wasn't normalized. The verifier would have puked on it. Created attachment 168655 [details]
mrror log
Kind of a side, user question, but I corrected the checksums on the M7 repo, currently only in ~/downloads/releases/helios/201005070900a.
But I still get some mirror errors (unreleated to checksums). I suspect these error messages (see mirror log) are all related to "runtime target" things. What is the correct way to "mirror everything"? My mirror tasks currently is as follows
<project
default="p2Mirror"
basedir=".">
<property name="todir" value="/shared/helios/temp" />
<target
name="p2Mirror">
<p2.mirror ignoreErrors="true" log="${todir}/mirrorlog.txt">
<repository
location="file:/${todir}" name="A temp test repo" />
<source>
<repository
location="file://home/data/httpd/download.eclipse.org/releases/helios/201005070900a/" />
</source>
<slicingOptions platformFilter="true" />
</p2.mirror>
</target>
</project>
To cross reference, I opened bug 313053 and bug 313055 on projects showing checksum errors. If you set platformFilter to true, you will force the slicer to consider all filters. That becomes problematic since the Helios repository lacks a lot of platform combinations (bug 312011) needed by the rcp feature (and others). Apparently some requirements are wrong. The string "A.PDE.Target.Feature" should actually be "A.PDE.Target.Platform". I'm puzzled by the fact that the former made it through the aggregator. I need to investigate that further. In any case, the people in charge of RAP should be aware that their requirement is incorrect. (In reply to comment #12) > Apparently some requirements are wrong. The string "A.PDE.Target.Feature" should > actually be "A.PDE.Target.Platform". I'm puzzled by the fact that the former > made it through the aggregator. I need to investigate that further. > > In any case, the people in charge of RAP should be aware that their requirement > is incorrect. Are you sure that there is a problem with RAP? Our p2.inf looks ok: requires.1.namespace=A.PDE.Target.Platform requires.1.name=Cannot be installed into the IDE requires.1.range=[0.0.0,0.0.0] I had an email discussion with Kenn where he mentioned something about late changes to p2.inf files. Perhaps they didn't make it into M7? Judging from the log, the 'org.eclipse.emf.rap.common.ui.feature.group' looks incorrect but 'org.eclipse.emf.rap.edit.ui.feature.group' is OK. Is the p2.inf file the same in all features? Also, I would suggest that you change: requires.1.range=[0.0.0,0.0.0] into: requires.1.range=0.0.0 The latter means >= 0.0.0 which is the same as the empty range (I don't think that's what's causing the problems though). (In reply to comment #12) > If you set platformFilter to true, you will force the slicer to consider all > filters. That becomes problematic since the Helios repository lacks a lot of > platform combinations (bug 312011) needed by the rcp feature (and others). > Thanks for this clarification on what platformFilter=true means. I added it, based on some blog I read, wondering if it would make a difference in this case. I seemed to have no effect. But sounds like you are saying everything _should_ mirror with the task I have (minus the platform filter) ... so, we can certainly try it for RC1. I think it has the same problems as M7. From some local tests I did. (In reply to comment #14) > I had an email discussion with Kenn where he mentioned something about late > changes to p2.inf files. Perhaps they didn't make it into M7? These were changes in EMF, actually, resulting in an M7a build. As far as I know the changes did make it into Helios, as they were done on May 4 > Judging from the log, the 'org.eclipse.emf.rap.common.ui.feature.group' looks > incorrect but 'org.eclipse.emf.rap.edit.ui.feature.group' is OK. Is the p2.inf > file the same in all features? Yes, it's the same in all (EMF RAP) features. Not sure how/why the wrong one is being picked up for common.ui... > > Also, I would suggest that you change: > > requires.1.range=[0.0.0,0.0.0] > > into: > > requires.1.range=0.0.0 This was also done (on May 4) for EMF RAP features... FYI, repos created with new b3.aggreator do not seem to have this checksum problem, while buckyBuild ones do. (Same input) Thomas, Is that what you'd expect? Or just coincidence? Of course, the jars also differ ... so, not sure what is correct where. (In reply to comment #14) > Also, I would suggest that you change: > > requires.1.range=[0.0.0,0.0.0] > > into: > > requires.1.range=0.0.0 Applied this change to the RAP runtime as well. Will be changed in RC1. This bugzilla is all over the place :-) I will close it as fixed now since most of the problems seems to have been addressed and I'm not sure how to apply the remaining issues to concrete problems in the current b3 aggregator. Please add new bugzillas if you feel that something was left out. [Bookkeeping change only. Moving bugs to the new "home" of aggregator, CBI. No change to assignee for resolved and verified bugs.] |