Community
Participate
Working Groups
We don't provide unpacked jars on the Buckminster update site. This means that we discover when people have problems unpacking their jars earlier than most other projects. In most cases this is never noticed since p2 silently switches to use the the unpacked jar when unpacking fails (this "feature" could be discussed further, but it's not the subject of this bugzilla). None of our nested jars can be unpacked when using Java 1.7.0. They are conditioned (repacked) and they unpack OK with Java 1.6. With Java 1.7, the p2 install process fails with the following message: Problems downloading artifact: osgi.bundle,org.eclipse.buckminster.ant,1.2.360.v20110828-1131. File has invalid content:/tmp/signatureFile6759261657287677858.jar Invalid content:bmant.jar The file "bmant.jar" in the jar "/tmp/signatureFile6759261657287677858.jar" has been tampered! Invalid content:ant_tasks/ant_tasks.jar The file "ant_tasks/ant_tasks.jar" in the jar "/tmp/signatureFile6759261657287677858.jar" has been tampered! Steps to Reproduce: 1. Install a clean Eclipse 3.7.1 2. Start this eclipse with Java 1.7.0 3. Open the "Install New Software" dialog 4. Enter 'http://download.eclipse.org/tools/buckminster/updates-3.7' 5. Install the 'Buckminster - Core' feature The error above occurs after some time when the download has completed and unpacking starts. When trying the exact same steps again, this time using Java 1.6, everything works as expected. My conclusion is that even if you do recondition a jar prior to packing, that is not enough. A Java 1.7 unpack will still create a slightly different result than a Java 1.6 unpack.
When testing this, please use http://download.eclipse.org/tools/buckminster/updates-3.7.save as the update site instead. The reason is that I will rebuild the original site to provide unpacked jars as a work around until this issue has been resolved.
*** Bug 365490 has been marked as a duplicate of this bug. ***
(In reply to comment #2) > *** Bug 365490 has been marked as a duplicate of this bug. *** The dup'ed bug provides 4 more "use cases" that demonstrate this problem: org.apache.ant,1.8.2.v20110505-1300 org.junit,3.8.2.v3_8_2_v20100427-1100 org.jmock,1.2.0.v201108251452 org.junit,4.8.2.v4_8_2_v20110321-1705 Note that in Orbit, we do have other cases of "nested jars" ... I'm not quite sure that the difference is that allows some to work, and some not to work. For example, javax.xml.soap 1.2.0 org.apache.axis 1.4 I those two cases we do specify jarprocessor.exclude.children=true in the eclipse.inf file ... that is, exclude from packing and signing ... in the 4 former failing cases may we specify only jarprocessor.exclude.children.sign=true (I say "may" since ant was the only one I looked at closely). In Orbit, we use Java 5 do to the pack conditioning and eventual pack. As mentioned in the dupped bug, everything works fine if we use Java 6 while unpacking ... but fail if using Java 7.
I'm running into the same problem. Is there a plan to solve that for Java7? The time to use Java6 as fallback will be limited...
Here is my analysis so far: When we fetch packed jars we process (unpack) them. This essentially means we recreate new unpacked jars. It would appear that we compare MD5s using these processed jars, and the process (recreated) jars are slightly different when produced on Java 1.7 vs. Java 1.6 (or earlier). I traced through the JarOutput stream tonight, and compared Java 1.6 with Java 1.5, and yes, there is a slight difference. When a JarEntry is being written it follows this pattern: LOC Signature (4 bytes) Version (2 bytes) Flag (2 bytes) On Java 7, the flag is written as 0x88, whereas on Java 6, it's written as 0x80. (Notice the final 8 vs 0). It is this pesky little byte that's throwing our MD5 hashes off. I'll keep digging into what the "general purpose flag" means and why it's different.
The extra "8" in the Jars created with Java 1.7 comes from UTF-8 encoding. No encoding was set on ZipOutputStreams, and now it defaults to UTF-8. Unfortunately, I can't see an easy way to override this value, since it's actually set on the constructor for ZipOutputStream, but JarOutputStream doesn't expose this.
Ian, as a workaround we could configure the bundle with an eclipse.inf so nested jars are not packed jarprocessor.exclude.children.pack http://wiki.eclipse.org/JarProcessor_Options or all nested jars by adding this option to the pack.properties in the builder
(In reply to comment #7) > Ian, as a workaround we could configure the bundle with an eclipse.inf so > nested jars are not packed > > jarprocessor.exclude.children.pack > > http://wiki.eclipse.org/JarProcessor_Options > > or all nested jars by adding this option to the pack.properties in the builder If I understand this correctly, this would have to be done for all bundles. Are not the jar'ed bundles conditioned as well? It seems this should fail for all conditioned jars that are signed. Or ... I think I am missing something. Is it because the outer bundle is signed and contains a signed entry for the inner jar, and once we unpack that inner jar it does not match the outer signed entries MD5 check? It also seems all existing repos are broken if they contained nested conditioned jars and they are installed using Java 7.
(In reply to comment #8) > If I understand this correctly, this would have to be done for all bundles. > Are not the jar'ed bundles conditioned as well? It seems this should fail for > all conditioned jars that are signed. Or ... I think I am missing something. In short, I don't know. First I should apologize. I re-read what I wrote last night and it was confusing (and I think I said 1.6 when I should have said 1.7). But the problem is pretty straight-forward, when we re-create a Jar using 1.7 the bytes are not identical to the same Jar created with an earlier JRE because of the existence of the UTF-8 flag in the output. Unfortunately I'm not sure when we re-create Jars (we do this when processing pack.gz but I don't know about other cases), and I don't know what MD5 we use in our artifacts.jar. I made slight leap in assuming that the different bytes are the cause of this bug. I found the problem while tracking down a p2 test failure. There could be more than one problem here. I will follow the steps Thomas outlined and see if the UTF-8 flag is causing this MD5 problem. Regardless, there is clearly a difference in how packed jars are handled on newer JREs.
Looking at the ZipInputStream and ZipOutputStream from Java 7 I am not sure how we will solve this issue. AFAICT there is no way to figure out what the general purpose flags were for the entries read from the original jar file so that we can use them when writing the unpacked jar file. Even if we could figure that out, there is no way to force the flags to be used on the ZipOutputStream. ZipOutputStream.putNextEntry completely overrides the general purpose flag of the ZipEntry's package private field (which we cannot set anyway!) and then sets the flag for EFS if the charset supports UTF8. In short: I see no way, using the Java 7 jar/zip APIs, that you can take a ZipInputStream and write a ZipOutoutStream that is byte for byte (flag for flag) identical to the original.
(In reply to comment #10) > In short: I see no way, using the Java 7 jar/zip APIs, that you can take a > ZipInputStream and write a ZipOutoutStream that is byte for byte (flag for > flag) identical to the original. So ... a VM/JDK bug? (change in behavior?) I was going to say that a Java 7 'unpack200' (and jarsigner -verify) worked just fine ... because I've tried and there were no errors, per se, but, looking at it closer, that doesn't appear to be exactly right. Using the Java 7 jarsigner -verify results in an extra message the Java 5 jarsigner -verify doesn't have: This jar contains entries whose certificate chain is not validated. But, this message appears for jars and unpacked pack.gz files equally ... so, not the same as this bug ... might be an "enhancement" or some sort? ... I'm just saying, things do seem odd in behavior change in Java 5 and Java 7. I've tried searching http://bugs.sun.com/bugdatabase/ without much luck.
(In reply to comment #10) > In short: I see no way, using the Java 7 jar/zip APIs, that you can take a > ZipInputStream and write a ZipOutoutStream that is byte for byte (flag for > flag) identical to the original. And ... a question to show my possible ignorance, but I don't think "signature verification" is done on the jar as a whole ... just if the signature recored in manifest.mf file matches currently computed value of the file (which, in the case of nested jars (as a whole), should be done before any "rewrite", correct?)
(In reply to comment #12) > (In reply to comment #10) > > > In short: I see no way, using the Java 7 jar/zip APIs, that you can take a > > ZipInputStream and write a ZipOutoutStream that is byte for byte (flag for > > flag) identical to the original. > > And ... a question to show my possible ignorance, but I don't think "signature > verification" is done on the jar as a whole ... just if the signature recored > in manifest.mf file matches currently computed value of the file (which, in the > case of nested jars (as a whole), should be done before any "rewrite", > correct?) Here is the issue as I understand it (and I admit my understanding may be wrong): In the original error from comment 0 we have this: (In reply to comment #0) > Problems downloading artifact: > osgi.bundle,org.eclipse.buckminster.ant,1.2.360.v20110828-1131. > File has invalid content:/tmp/signatureFile6759261657287677858.jar > Invalid content:bmant.jar > The file "bmant.jar" in the jar "/tmp/signatureFile6759261657287677858.jar" has > been tampered! > Invalid content:ant_tasks/ant_tasks.jar > The file "ant_tasks/ant_tasks.jar" in the jar > "/tmp/signatureFile6759261657287677858.jar" has been tampered! In this case we have a bundle org.eclipse.buckminster.ant with a nested jar (bmant.jar). Before signing the bundle osgi.bundle,org.eclipse.buckminster.ant the bmant.jar was conditioned (packed/unpacked). This ensured that the jar entry for bmant.jar could be signed correctly and have the correct checksum for the unpacked content of bmant.jar in the signature included in the outer bundle org.eclipse.buckminster.ant The problem is when we try to reconstruct (unpack) the bmant.jar on Java 7 we get some extra flags set for each entry of the bmant.jar. This changes the overall checksum of the bmant.jar compared to when the outer bundle org.eclipse.buckminster.ant was signed. I believe there are two checks done by p2 when downloading and provisioning an artifact. First it downloads the overall artifact and checks that its checksum matches what the repo says the overall artifact checksum should be. Then for signed bundles, p2 does an extra check to make sure the signed bundle content has not been tampered with since it was signed. This second check is what I think is failing because the overall checksum of the nested jar bmant.jar (when unpacked with Java 7) is different than the checksum of bmant.jar when the bundle org.eclipse.buckminster.ant was signed. So to answer your question: > just if the signature recored > in manifest.mf file matches currently computed value of the file (which, in the > case of nested jars (as a whole), should be done before any "rewrite", > correct?) This has to be done after unpacking bmant.jar since that jar was unpacked when we signed the overall bundle org.eclipse.buckminster.ant
(In reply to comment #7) > Ian, as a workaround we could configure the bundle with an eclipse.inf so > nested jars are not packed > > jarprocessor.exclude.children.pack > > http://wiki.eclipse.org/JarProcessor_Options > > or all nested jars by adding this option to the pack.properties in the builder In light of this bug, it seems this option should be enabled by default (to exclude children jars). I would like confirmation back from Ian (or others on p2) that this really only effects nested jars of signed bundles.
Please see http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5030283. The only hack around this would be to provide your own Charset class and use reflection to call the new Java 7 APIs that also take the charset.
(In reply to comment #15) > Please see http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5030283. > > The only hack around this would be to provide your own Charset class and use > reflection to call the new Java 7 APIs that also take the charset. This would allow us to write the jar in the "old-style" format, but we still must be able to determine if we must write the jar out in the "new-style" as well. We cannot just do it the same "old-style" way all the time since the provided jar may be in the "new-style".
For everyone's info: ECF consumers are now running into this problem as well...see bug 376306.
(In reply to comment #17) > For everyone's info: ECF consumers are now running into this problem as > well...see bug 376306. Does this only affect users who use packed200 jars? Do you keep the old (unpacked) jars around too?
(In reply to comment #18) > (In reply to comment #17) > > For everyone's info: ECF consumers are now running into this problem as > > well...see bug 376306. > > Does this only affect users who use packed200 jars? Do you keep the old > (unpacked) jars around too? I believe it only effects users who use packed200 jars. We do keep the old jar around...in our problematic case, it's a library (xpp) required by the Smack xmpp api here: http://git.eclipse.org/c/ecf/org.eclipse.ecf.git/tree/protocols/bundles/org.jivesoftware.smack/jars
(In reply to comment #16) > (In reply to comment #15) > > Please see http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5030283. > > > > The only hack around this would be to provide your own Charset class and use > > reflection to call the new Java 7 APIs that also take the charset. > > This would allow us to write the jar in the "old-style" format, but we still > must be able to determine if we must write the jar out in the "new-style" as > well. We cannot just do it the same "old-style" way all the time since the > provided jar may be in the "new-style". This bug has a target milestone of Juno M7 but from this comment it isn't apparent that there is a clear way to fix it. Is there a viable workaround for the 3.8 release?
(In reply to comment #20) > This bug has a target milestone of Juno M7 but from this comment it isn't > apparent that there is a clear way to fix it. Is there a viable workaround for > the 3.8 release? I've been doing more investigation on this. The workaround is not to pack inner jars or also produce canonical (non-packed bundles). This only affects packed bundles, which in turn have packed inner jars, when the original (outer jar) is signed. I wanted to see if the platform/sdk were ok. However, it doesn't appear we are packing any of bundles (see bug 377974). Also, as a side, I found out that we actually try the packed jar 200 times before giving up (and finally downloading the canonical form). See bug 377976.
(In reply to comment #21) > (In reply to comment #20) > > This bug has a target milestone of Juno M7 but from this comment it isn't > > apparent that there is a clear way to fix it. Is there a viable workaround for > > the 3.8 release? > > I've been doing more investigation on this. The workaround is not to pack > inner jars or also produce canonical (non-packed bundles). This only affects > packed bundles, which in turn have packed inner jars, when the original (outer > jar) is signed. > > I wanted to see if the platform/sdk were ok. However, it doesn't appear we are > packing any of bundles (see bug 377974). > > Also, as a side, I found out that we actually try the packed jar 200 times > before giving up (and finally downloading the canonical form). See bug 377976. Internally we mirror the SR2 release using the b3 aggregator, and when attempting to update Eclipse 3.7.1 to 3.7.2 off of the mirrored repository, we get the following failures: Problems downloading artifact: osgi.bundle,org.eclipse.ant.core,3.2.302.v20120110-1739. File has invalid content:/tmp/signatureFile3960810583362569482.jar Invalid content:lib/antsupportlib.jar Problems downloading artifact: osgi.bundle,org.eclipse.ant.launching,1.0.101.v20120110-1739. File has invalid content:/tmp/signatureFile7107106176313886288.jar Invalid content:lib/loggers.jar Invalid content:lib/antdebug.jar Invalid content:lib/remote.jar Problems downloading artifact: osgi.bundle,org.eclipse.ant.ui,3.5.101.v20120110-1739. File has invalid content:/tmp/signatureFile3294722720667846101.jar Invalid content:lib/remoteAnt.jar Invalid content:lib/antrunner.jar Problems downloading artifact: osgi.bundle,org.eclipse.core.resources,3.7.101.v20120125-1505. File has invalid content:/tmp/signatureFile7051583781585861877.jar Invalid content:ant_tasks/resources-ant.jar Problems downloading artifact: osgi.bundle,org.eclipse.equinox.p2.director.app,1.0.300.v20111126-0153. File has invalid content:/tmp/signatureFile4398375309217570016.jar Invalid content:ant_tasks/director-ant.jar Problems downloading artifact: osgi.bundle,org.eclipse.equinox.p2.jarprocessor,1.0.200.v20110815-1438. File has invalid content:/tmp/signatureFile6614637867881709557.jar Invalid content:lib/jarprocessor-ant.jar Problems downloading artifact: osgi.bundle,org.eclipse.equinox.p2.metadata.repository,1.2.0.v20110815-1419. File has invalid content:/tmp/signatureFile8445114556116565336.jar Invalid content:ant_tasks/metadataRepository-ant.jar Problems downloading artifact: osgi.bundle,org.eclipse.equinox.p2.publisher.eclipse,1.0.0.v20110815-1438. File has invalid content:/tmp/signatureFile1888418916153552854.jar Invalid content:pdepublishing.jar Invalid content:ant_tasks/pdepublishing-ant.jar Problems downloading artifact: osgi.bundle,org.eclipse.equinox.p2.repository.tools,2.0.100.v20110815-1438. File has invalid content:/tmp/signatureFile1823150235942128588.jar Invalid content:lib/repository-tools-ant.jar Problems downloading artifact: osgi.bundle,org.eclipse.help.base,3.6.2.v201202080800. File has invalid content:/tmp/signatureFile8482388247338961863.jar Invalid content:ant_tasks/helpbase-ant.jar Problems downloading artifact: osgi.bundle,org.eclipse.jdt.launching,3.6.1.v20111006_r372. File has invalid content:/tmp/signatureFile5393585445397322200.jar Invalid content:lib/launchingsupport.jar Problems downloading artifact: osgi.bundle,org.eclipse.platform,3.7.2.v201202080800. File has invalid content:/tmp/signatureFile4305113726334015614.jar Invalid content:platform.jar I don't get these when updating using the http site directly. I created our internal mirror with the following aggregator file: <?xml version="1.0" encoding="ASCII"?> <aggregator:Aggregation xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:aggregator="http://www.eclipse.org/b3/2011/aggregator/1.1.0" label="Indigo" packedStrategy="COPY" buildRoot="/tmp/eclipse_mirror"> <validationSets label="main"> <contributions label="Indigo 3.7"> <!-- To find individual SRN sites, download the http://download.eclipse.org/releases/indigo/compositeContent.jar and http://download.eclipse.org/webtools/repository/indigo/compositeContent.jar files and inspec them. --> <repositories location="http://download.eclipse.org/releases/indigo/201202240900" description="Eclipse 3.7 release"/> <repositories location="http://download.eclipse.org/webtools/downloads/drops/R3.3.2/R-3.3.2-20120210195245/repository/" description="Eclipse 3.7 Web Tools"/> <repositories location="http://download.eclipse.org/webtools/patches/drops/R3.3.2/P-3.3.2-20120305171618/repository/" description="Eclipse 3.7 Web Tools Patches"/> </contributions> </validationSets> <configurations operatingSystem="linux" windowSystem="gtk"/> </aggregator:Aggregation> If I change "packedStrategy" to "UNPACK", the resulting mirror does not run into this bug.
(In reply to comment #22) Things will still fail, but it now (as of 20 minutes ago) only fail once and then download the canonical form. > If I change "packedStrategy" to "UNPACK", the resulting mirror does not run > into this bug. Is this a workaround for you? The problem is that we can't find a general solution (that allows packing to happen on either 1.6 or 1.7 and unpacking to happen on either 1.6 or 1.7) without shipping our own jar writer. There was a breaking change in the binary format for the Jar writer between 1.6 and 1.7. Since we don't have the resources to implement our own jar writer (and even if we did, it would't help those who are working on Eclipse 3.7) all we can suggest is: 1. Don't pack inner jars 2. Ensure that the canonical form (non packed form) is kept around. Let me know if these work arounds work for you Terry.
(In reply to comment #23) > (In reply to comment #22) > Things will still fail, but it now (as of 20 minutes ago) only fail once and > then download the canonical form. > > > If I change "packedStrategy" to "UNPACK", the resulting mirror does not run > > into this bug. > > Is this a workaround for you? It works for my immediate needs. The 3.7 SR2 repository was packed with 1.6 and I can control the JDK that the b3 aggregator uses when I do the mirroring (whereas I can't control what JDK users are running when they try to update against my mirrored repository.) If I were trying to mirror multiple repositories that used 1.6 and 1.7 packing, I would have to carefully orchestrate the mirroring to use the correct JDK for each repository. > > The problem is that we can't find a general solution (that allows packing to > happen on either 1.6 or 1.7 and unpacking to happen on either 1.6 or 1.7) > without shipping our own jar writer. There was a breaking change in the binary > format for the Jar writer between 1.6 and 1.7. > > Since we don't have the resources to implement our own jar writer (and even if > we did, it would't help those who are working on Eclipse 3.7) all we can > suggest is: > > 1. Don't pack inner jars > 2. Ensure that the canonical form (non packed form) is kept around. > > Let me know if these work arounds work for you Terry. In my mirroring I am essentially getting the latter by restoring the unpacked form using the correct JDK. These workarounds don't help for existing repositories that don't employ them, but will work fine for any newly created ones that do. If one or the other is put in place for the 3.8.0 release then we shouldn't have issues going forward.
(In reply to comment #23) <stuff deleted> > Since we don't have the resources to implement our own jar writer (and even if > we did, it would't help those who are working on Eclipse 3.7) all we can > suggest is: > > 1. Don't pack inner jars > 2. Ensure that the canonical form (non packed form) is kept around. Question: As per bug 376306, we (ECF) have a third party jar that's checked in for our org.jivesoftware.smack bundle...in git here: http://git.eclipse.org/c/ecf/org.eclipse.ecf.git/tree/protocols/bundles/org.jivesoftware.smack/jars How can we best avoid this problem in future/Juno releases of ECF? We're not explicitly packing the inner jar...at least to my knowledge. Thanks for any info. Scott
(In reply to comment #25) > Question: As per bug 376306, we (ECF) have a third party jar that's checked in > for our org.jivesoftware.smack bundle...in git here: > > http://git.eclipse.org/c/ecf/org.eclipse.ecf.git/tree/protocols/bundles/org.jivesoftware.smack/jars > > How can we best avoid this problem in future/Juno releases of ECF? We're not > explicitly packing the inner jar...at least to my knowledge. First and foremost, we should all ensure that we are keeping our canonical (non packed.gz files) around. I noticed that the ECF site doesn't include these. This is the failsafe. As for the inner jars... crap... (bang head against the desk). I'm not really sure what to suggest, and I don't really know how to verify that your inner jar isn't packed. During the jar processor, I noticed that the following jar was being processed: /var/folders/jm/k2r4m3d960gdxdmtk38t6trm0000gn/T/work4196607119727135521/temp.p2.optimizers.incoming1213323703561656481.jar/jars/xpp.jar.pack.gz This would indicate that the xpp.jar (the inner jar) was packed. But I can't find any way to control this. I'll see if I can track down Andrew N. He wrote the jar processor years ago, so maybe he can shed some light on this issue. Sorry I can't be of more help Scott.
(In reply to comment #26) > (In reply to comment #25) > > This would indicate that the xpp.jar (the inner jar) was packed. But I can't > find any way to control this. Can you add an eclipse.inf file to the META-INF directory, and specify jarprocessor.exclude.children See http://wiki.eclipse.org/JarProcessor_Options
(In reply to comment #27) > Can you add an eclipse.inf file to the META-INF directory, and specify > > jarprocessor.exclude.children > > See > http://wiki.eclipse.org/JarProcessor_Options Thanks David, if that results in the jar being marked as not conditioned then it might work (fingers crossed). I also noticed that the platform nested jars are still being conditioned (seee bug 378200). If either ECF or the platform disables the packing of inner jars, I'd be happy to test it out.
(In reply to comment #28) <stuff deleted> > Thanks David, if that results in the jar being marked as not conditioned then > it might work (fingers crossed). I also noticed that the platform nested jars > are still being conditioned (seee bug 378200). If either ECF or the platform > disables the packing of inner jars, I'd be happy to test it out. Thanks David and Ian. I've added an eclipse.inf with the given option here: http://git.eclipse.org/c/ecf/org.eclipse.ecf.git/tree/protocols/bundles/org.jivesoftware.smack/META-INF Our builder is not available at this moment, but hopefully it will be back up soon, and I will then run a build and test whether this works ok. I'll let all know when this test is done.
(In reply to comment #29) <stuff deleted> > > Our builder is not available at this moment, but hopefully it will be back up > soon, and I will then run a build and test whether this works ok. I'll let all > know when this test is done. Oour builder is back online, and I've made the addition described in comment 27 (adding single line to eclipse.inf in enclosing plugin...for in our case...the org.jivesoftware.smack enclosing plugin). The resulting org.jivesoftware.smack plugin is here: https://build.ecf-project.org/jenkins/job/R-HEAD-sdk.feature/70/artifact/site.p2/plugins/org.jivesoftware.smack_3.1.100.v20120503-1753.jar.pack.gz I observe from unpacking this file that the embedded jar (i.e. lib/xpp.jar...that was previously packed into lib/xpp.jar.pack.gz according to comment 26...is now *not packed*...i.e. the embedded jar in the smack jar above is simply lib/xpp.jar. So I think the eclipse.inf directive is working for our case (preventing the embedded xpp.jar file from being packed). I don't currently run win 7, so I'll have to find someone who does to do a final test, but I think the eclipse.inf directive is indeed working. Thanks much David and Ian.
(In reply to comment #30) Ugh. Unfortunately, it seems that java 7 users are still experiencing this problem in our embedded jar use case. Please see https://bugs.eclipse.org/bugs/show_bug.cgi?id=376306#c9 Perhaps this is as described by Thomas in the original Description for this bug. Any other workaround thoughts? It seems that this is quite likely to bite a number of projects...once Oracle moves to java 7 as default (not sure when that is).
At this point, the only way to not get burnt by this is to keep the unpacked jars available in the repo.
(In reply to comment #30) > https://build.ecf-project.org/jenkins/job/R-HEAD-sdk.feature/70/artifact/site.p2/plugins/org.jivesoftware.smack_3.1.100.v20120503-1753.jar.pack.gz > I can't seem to use that repo as it's using a self-signed cert. I'm using [1] instead (from the other bug report, I think this is the newest stuff). [1] http://download.eclipse.org/rt/ecf/3.5Test/site.p2 > I observe from unpacking this file that the embedded jar (i.e. > lib/xpp.jar...that was previously packed into lib/xpp.jar.pack.gz according to > comment 26...is now *not packed*...i.e. the embedded jar in the smack jar above > is simply lib/xpp.jar. So I think the eclipse.inf directive is working for our > case (preventing the embedded xpp.jar file from being packed). With a debugger attached (instead the jar processor), I'm still seeing the inner jar as temp.p2.optimizers.incoming3236402198203485243.jar/jars/xpp.jar.pack.gz (notice the pack.gz). So it appears that the eclipse jar processor still thinks it's packed. How did you unpack the jar Scott?
(In reply to comment #33) <stuff deleted> > (notice the pack.gz). So it appears that the eclipse jar processor still > thinks it's packed. > > How did you unpack the jar Scott? I used java 6 pack200 to unpack the plugin/outer jar: https://build.ecf-project.org/jenkins/job/R-HEAD-sdk.feature/70/artifact/site.p2/plugins/org.jivesoftware.smack_3.1.100.v20120503-1753.jar.pack.gz I didn't realized that unpacking this outer jar would unpack the xpp.jar...if that's what's happening. Thanks.
Unfortunately we just don't have an answer for this. The checksums of jars are different when created / computed with Java 6 vs. Java 7. Unless we start to ship different signatures for different JREs, I don't really see a path forward. As I've mentioned, I'm surprised this hasn't hit us before since we are putting a lot of faith in a binary format that I'm not sure has a concrete specification. I'm sure that the format will change again in the future (with a different vendor or a different Java version). Please make sure that you keep your canonical (non-packed) jars around. At least p2 can fall back to these.
How might I see this manifested in practice? (other than the original, common 0 use case). I ask because I was trying some "install everything" stress tests on .../releases/juno I thought things (e.g times, number of re-directs back to download.eclipse.org) would be much worse when doing the tests with Java 7, than with Java 6. Oddly, the time was about the same, about 20 minutes (just, "watching the clock" measurements) ... and that is everything except "runtime targets" category. [Keep in mind, if you try this yourself, that .../releases/staging is never mirrored, so is not a good test case for this.] I'll paste some data below, not for time or "unpack errors" (that is, what I'm asking "how to see") .... but you will note there were not many attempts to go back to "download.eclipse.org". None when IBM's JRE was used, and only 28 (out of 4000 or so) ... so ... have I gotten the wrong impression on how this bug manifests itself? Does it not result in more "give up and try original request source" attempts? Or, do the re-try attempts to go back to the mirrors when getting the canonical form? I don't see any obvious "exceptions" in the log (well, a few, but only when mirrors "failed to respond"). Note, the .options file I used, and linux grep command I used to get the "histograms" are those documented at http://wiki.eclipse.org/Equinox/p2/p2.mirrorsURL#When_not_to_use_the_p2.mirrorsURL_property.3F = = = JRE 1.7.0 IBM J9 2.6 Linux amd64-64 20110810_88604 (JIT enabled, AOT enabled) J9VM - R26_Java726_GA_20110810_1208_B88592 JIT - r11_20110810_20466 GC - R26_Java726_GA_20110810_1208_B88592 J9CL - 20110810_88604 1854 http://mirrors.med.harvard.edu 1349 http://www.gtlib.gatech.edu 771 http://ftp.osuosl.org 126 http://ftp.ussg.iu.edu 79 http://mirror.cc.columbia.edu 55 http://carroll.aset.psu.edu 29 ftp://eclipse.mirrors.tds.net 24 http://eclipse.c3sl.ufpr.br 16 http://ftp.cs.pu.edu.tw 13 http://mirrors.ibiblio.org 13 ftp://ftp.ussg.iu.edu 11 http://eclipse.stu.edu.tw 10 http://mirrors.xmission.com = = = = java version "1.7.0" Java(TM) SE Runtime Environment (build 1.7.0-b147) Java HotSpot(TM) 64-Bit Server VM (build 21.0-b17, mixed mode) 2241 http://ftp.osuosl.org 1033 http://carroll.aset.psu.edu 674 http://mirrors.med.harvard.edu 230 http://mirror.cc.columbia.edu 76 http://mirror.cc.vt.edu 28 http://mirrors.ibiblio.org 28 http://download.eclipse.org 22 http://mirrors.xmission.com 6 http://eclipse.stu.edu.tw 5 http://eclipse.c3sl.ufpr.br 3 http://www.gtlib.gatech.edu = = = = = So ... am I seeing/doing something wrong ... or, is this not a big problem "in practice"? [Other than perhaps the original comment 0 use case]. Sorry I didn't capture the Java 6 data (nor, exact times) but ... I was expecting to see a huge difference from some of what I've heard ... so ... thought I'd ask here what I should be doing/seeing to adequately test this in the context of "Simultaneous Release"? [If it matters, I was using the final platform RC4 candidates as the "starting point, to install things into .... not, for example, updating an Indigo version, which does have another, larger problem with Java 7 and p2 repos (see bug352049). ] Thanks for any hints or tips ... I'm mostly just curious ... almost wondering if the Java 7 VM's "fixed" something?! ... Or, more likely, just not that big of a problem in Simultaneous Release context?
It is unrealistic to assume p2 will be always able to reconstruct byte-to-byte identical jar after doing pack200-unpack200 roundtrip. The only reliable/guaranteed way to keep signatures of nested jars intact during this is to NOT pack200 them.
FWIW, bug 396553 is an example of a nested packed jar causing problems, for Java 6 or Java 7 ... though it was packed inside a regular jar which is likely unusual (we decided to not pack, for now, the parent jar since it was having pack200 errors). It illustrates the need to change the default to not pack nested jars. (hint hint nudge nudge :)
Created attachment 225154 [details] I think this patch will work to change the default for nested jars The changes come down to one method, "shouldPack", but is kind of a big change, partially due to the way the original was written (with multiple return points, instead of one, at the end) and lots of implicit cases (since default was always "true" in the past). I tried to explicitly cover all cases and added a "verbose" println to each case so its easier to see what's being assumed/set for each case. I'll attach a "test run" I did against our platform's "master.zip", just calling "repack" on it ... but think the logic is same for the subsequent "pack step".
Created attachment 225155 [details] testoutput from workspace I tested this patch by running org.eclipse.equinox.p2.jarprocessor with the arguments: -verbose -processAll -repack /home/davidw/temp/eclipse-master-M20130102-1200.zip The eclipse-master-M20130102-1200.zip I snagged from the "signing directory" during a build, before it had been processed. I'd offer to share it, but it is 250 Megabytes, so hope others who want to test have some smaller samples. :) The output, though, from my spot checks, indicates the right thing is being done, "marked" jars processed as before, unmarked "main" jars still processes, but nested jars now excluded by default. Feedback welcome.
The PDE/Build test suite does make some use of the p2.jarprocessor, making sure they all pass with these changes would be another good test case.
Just to cross-reference, this difference in behavior of zipped input/output streams (between Java 6 and Java 7) is at heart the same issue as documented in bug 408944.
I have the same Problem with invalid Content in plugins and Features. I have a target platform with a huge bunch of plugins and have created a p2 repository from it to use it in a maven/tycho build. I get the error with the signature for all plugins/bundles which where unpacked in the target platform and have been packed by the FeatureAndBundlesPublisher application. I used the following command: %ECLIPSE_SDK%\eclipse.exe -debug -consolelog -nosplash -verbose -application org.eclipse.equinox.p2.publisher.FeaturesAndBundlesPublisher -metadataRepository file:/D:/JavaTargets/try/repositories/%TARGET_PLATFORM% -artifactRepository file:/D:/JavaTargets/try/repositories/%TARGET_PLATFORM% -source D:\JavaTargets\try\%TARGET_PLATFORM% -configs -compress -publishArtifacts I tried with Java6 and Java7 but found no way to make it work. I also tried without -compress but this does not work either.
Any updates on this issue? We use a Tycho build together with a target platform based on p2 repositories. We get the same kind of error, therefore our development department is prevented from upgrading to Java 7 ...
I found out that if I delete the files eclipse.inf, ECLIPSE_.RSA and ECLIPSE_.SF in the target platform in all the extracted Features and plugins before i create the p2 repository then the maven/tycho build works fine
(In reply to Alexander Weickmann from comment #44) > Any updates on this issue? We use a Tycho build together with a target > platform based on p2 repositories. We get the same kind of error, therefore > our development department is prevented from upgrading to Java 7 ... This "bug" won't be fixed in the sense that things will "suddenly work" with Java 7. It is due to a change in Java 7 "zip" behavior (for good reasons) and the "fix" here will be just to change the default to not pack "inner jars". Here's what this means to you: a) any "old" repositories you use, will always have the issue. (but there are workarounds for those, described below). b) if there are "just a few" problematic cases in current development, you might open bugs on those projects and ask them to not pack their inner jars (they can use following directive in the eclipse.inf file: jarprocessor.exclude.children.pack=true c) This really shouldn't prevent your development department from upgrading to Java 7 ... though there might be "more setup and install work" for you to do. I'm a little uncertain of your exact use-case, but 1) if you are using "old" repositories, produced with Java 6 (or 5) pack200, then you can easily (still) use Java 7 and use the org.eclipse.update.jarprocessor.pack200 option to "point to" a Java 6 (or Java 5) version of unpack200 ... or perhaps specify none? ... See http://wiki.eclipse.org/JarProcessor_Options pack200 and unpack200 are "standalone executables" so you don't need the whole Java 5 or Java 6 VM installed. 2) you (and projects at Eclipse.org) should always have the "unpacked jar" version available, so even if the pack.gz version fails, then the unpacked jar should still work as fall-back. (At least, that's true of p2 ... if not true of Tycho, then that's their bug). Let me know if you've tried these options and they haven't worked ... perhaps I'm misunderstanding your exact use case or situation. I mostly just wanted to clarity this bug won't be "fixed" for old repositories, the conclusion this far is just to change the default and worry about future repositories ... which, we do still need to do soon!
Hey David. Thanks for the information you provided. It will be invaluable to us. We will try it after the next major release of our product is done.
(In reply to comment #46) > 2) you (and projects at Eclipse.org) should always have the "unpacked jar" > version available, so even if the pack.gz version fails, then the unpacked jar > should still work as fall-back. (At least, that's true of p2 ... if not true of > Tycho, then that's their bug). There was a bug (bug 412945), but only 0.19.0-SNAPSHOT was affected. All released Tycho versions use the non-packed artifact as fallback.
I'm seeing this issue when using the b3 aggregator: I understand that Java 7 has a problem, but when using unpack200 from Java 6 it should work(?), so I configured b3 to run on a JVM 7 (needing Java 7 for some other reason), but use -Dorg.eclipse.update.jarprocessor.pack200 to point to the bin/ directory of Java 6. I checked using strace that the option is correctly picked up, i.e., unpack200 is indeed found in the Java 6 installation. Still aggregation fails and I'm seeing this in my log: !ENTRY org.eclipse.b3.util 1 -364274674 2013-11-10 00:23:30.234 !MESSAGE - mirroring artifact osgi.bundle,org.jboss.netty,3.2.5.Final-20130531-2315 !ENTRY org.eclipse.b3.util 4 -364274674 2013-11-10 00:23:31.485 !MESSAGE Unable to unpack artifact osgi.bundle,org.jboss.netty,3.2.5.Final-20130531-2315 in repository file:/.../aggregation/build/final/aggregate: Invalid content:netty-3.2.5.Final.jar !STACK 1 org.eclipse.core.runtime.CoreException: Unable to unpack artifact osgi.bundle,org.jboss.netty,3.2.5.Final-20130531-2315 in repository file:/.../aggregation/build/final/aggregate: Invalid content:netty-3.2.5.Final.jar at org.eclipse.b3.util.ExceptionUtils.fromMessage(ExceptionUtils.java:53) at org.eclipse.b3.aggregator.engine.MirrorGenerator.unpackToSibling(MirrorGenerator.java:435) at org.eclipse.b3.aggregator.engine.MirrorGenerator.mirror(MirrorGenerator.java:334) at org.eclipse.b3.aggregator.engine.MirrorGenerator.run(MirrorGenerator.java:508) at org.eclipse.b3.aggregator.engine.Builder.runMirroring(Builder.java:1706) at org.eclipse.b3.aggregator.engine.Builder.run(Builder.java:1656) at org.eclipse.b3.aggregator.engine.Builder.run(Builder.java:1686) at org.eclipse.b3.cli.AbstractCommand.run(AbstractCommand.java:133) at org.eclipse.b3.cli.Headless.run(Headless.java:161) at org.eclipse.b3.cli.Headless.start(Headless.java:197) at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196) at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110) at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79) at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:354) at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:181) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:636) at org.eclipse.equinox.launcher.Main.basicRun(Main.java:591) at org.eclipse.equinox.launcher.Main.run(Main.java:1450) at org.eclipse.equinox.launcher.Main.main(Main.java:1426) Caused by: org.eclipse.osgi.signedcontent.InvalidContentException: The file "netty-3.2.5.Final.jar" in the jar "/tmp/signatureFile2535006551054230169.jar" has been tampered! at org.eclipse.osgi.internal.signedcontent.DigestedInputStream.verifyDigests(DigestedInputStream.java:102) at org.eclipse.osgi.internal.signedcontent.DigestedInputStream.read(DigestedInputStream.java:129) at org.eclipse.osgi.internal.baseadaptor.AdaptorUtil.getBytes(AdaptorUtil.java:247) at org.eclipse.osgi.baseadaptor.bundlefile.BundleEntry.getBytes(BundleEntry.java:96) at org.eclipse.osgi.internal.signedcontent.SignedContentImpl$SignedContentEntryImpl.verify(SignedContentImpl.java:165) at org.eclipse.osgi.internal.signedcontent.SignedContentFile.getSignedEntries(SignedContentFile.java:51) at org.eclipse.equinox.internal.p2.artifact.repository.SignatureVerifier.verifyContent(SignatureVerifier.java:82) at org.eclipse.equinox.internal.p2.artifact.repository.SignatureVerifier.verify(SignatureVerifier.java:59) at org.eclipse.equinox.internal.p2.artifact.repository.SignatureVerifier.close(SignatureVerifier.java:100) at org.eclipse.equinox.internal.provisional.p2.artifact.repository.processing.ProcessingStep.close(ProcessingStep.java:85) at org.eclipse.equinox.internal.p2.artifact.processing.AbstractBufferingStep.close(AbstractBufferingStep.java:73) at org.eclipse.equinox.internal.p2.artifact.repository.simple.SimpleArtifactRepository.reportStatus(SimpleArtifactRepository.java:1183) at org.eclipse.equinox.internal.p2.artifact.repository.simple.SimpleArtifactRepository.downloadArtifact(SimpleArtifactRepository.java:591) at org.eclipse.equinox.internal.p2.artifact.repository.simple.SimpleArtifactRepository.getArtifact(SimpleArtifactRepository.java:723) at org.eclipse.b3.aggregator.engine.MirrorGenerator$CanonicalizeRequest.perform(MirrorGenerator.java:92) at org.eclipse.b3.aggregator.engine.MirrorGenerator.unpackToSibling(MirrorGenerator.java:423) ... 21 more Never mind if b3 line numbers don't match your sources, that's what I was working on... Is this expected, or should using Java 7 actually be fine with the given configuration option?
(In reply to Stephan Herrmann from comment #49) > I'm seeing this issue when using the b3 aggregator: > < ... > > Is this expected, or should using Java 7 actually be fine with the given > configuration option? I've just skim read your post ... but the "problem" occurs when a nested jar is packed with Java 7. After that, no version of Java will be able to unpack the nested jar. Teams with nested jars should take steps to not pack nested jars. The method of accomplishing that, depends a little on which build technology is used. But, that's the short term solution.
(In reply to David Williams from comment #50) > (In reply to Stephan Herrmann from comment #49) > > I'm seeing this issue when using the b3 aggregator: > > > < ... > > > > Is this expected, or should using Java 7 actually be fine with the given > > configuration option? > > I've just skim read your post ... but the "problem" occurs when a nested jar > is packed with Java 7. After that, no version of Java will be able to unpack > the nested jar. > > Teams with nested jars should take steps to not pack nested jars. The method > of accomplishing that, depends a little on which build technology is used. > But, that's the short term solution. Whoops, meant to paste this "repo report" which for Sim. Release, shows which have nested jars. http://build.eclipse.org/simrel/luna/reporeports/reports/verifydiroutput/nestedjars.txt The "fall back", though, should be that p2 "gives up" on the packed version and simply gets the jar file. I'm pretty sure that works well for pure p2. But, not sure all build technologies use pure p2.
(In reply to David Williams from comment #50) I'm hesitant to comment here, before Java 8 GA stuff is out of the door ... :) > (In reply to Stephan Herrmann from comment #49) > > I'm seeing this issue when using the b3 aggregator: > > > < ... > > > > Is this expected, or should using Java 7 actually be fine with the given > > configuration option? > > I've just skim read your post ... but the "problem" occurs when a nested jar > is packed with Java 7. After that, no version of Java will be able to unpack > the nested jar. Am I to read this as: osgi.bundle,org.jboss.netty,3.2.5.Final-20130531-2315 is unusable?
(In reply to Stephan Herrmann from comment #52) > (In reply to David Williams from comment #50) > > I'm hesitant to comment here, before Java 8 GA stuff is out of the door ... > :) Can't wait to hear that news! ... (see bug 428085 and bug 427873) > > > (In reply to Stephan Herrmann from comment #49) > > > I'm seeing this issue when using the b3 aggregator: > > > > > < ... > > > > > > Is this expected, or should using Java 7 actually be fine with the given > > > configuration option? > > > > I've just skim read your post ... but the "problem" occurs when a nested jar > > is packed with Java 7. After that, no version of Java will be able to unpack > > the nested jar. > > Am I to read this as: osgi.bundle,org.jboss.netty,3.2.5.Final-20130531-2315 > is unusable? Well, the '*.jar' version should always be usable. And ... I think I skim read too fast. Many pack.gz files with nested packed jars could be unpacked with Java 6 ... especially if they were packed with Java 5 or 6 ... I am less certain (now) about whether or not nested jars created with Java 7 could be unpacked with Java 7. But, again, the *.jar version should always be usable. The b3 aggregator has high(er) standards when it comes to reporting errors -- as a feature :) But, does seem like using -Dorg.eclipse.update.jarprocessor.pack200 should "work" ... and its a vague memory, but I seem to recall I've had trouble getting that to "point to" the "right" place, at times.
Using Mars.1 on Java 1.7 to resolve a target platform containing a Galileo p2 site (of the Eclipse platform), the provided workaround does not work. Still getting "file has invalid content" error messages in target definition editor. Option: -Dorg.eclipse.update.jarprocessor.pack200=C:/java/jdk1.6.0_43/bin Is the path to the Java 6 bin folde sufficient, or am I missing something here? Thanks! (In reply to David Williams from comment #53) > (In reply to Stephan Herrmann from comment #52) > > (In reply to David Williams from comment #50) > > > > I'm hesitant to comment here, before Java 8 GA stuff is out of the door ... > > :) > > Can't wait to hear that news! ... (see bug 428085 and bug 427873) > > > > > > (In reply to Stephan Herrmann from comment #49) > > > > I'm seeing this issue when using the b3 aggregator: > > > > > > > < ... > > > > > > > > Is this expected, or should using Java 7 actually be fine with the given > > > > configuration option? > > > > > > I've just skim read your post ... but the "problem" occurs when a nested jar > > > is packed with Java 7. After that, no version of Java will be able to unpack > > > the nested jar. > > > > Am I to read this as: osgi.bundle,org.jboss.netty,3.2.5.Final-20130531-2315 > > is unusable? > > Well, the '*.jar' version should always be usable. And ... I think I skim > read too fast. Many pack.gz files with nested packed jars could be unpacked > with Java 6 ... especially if they were packed with Java 5 or 6 ... I am > less certain (now) about whether or not nested jars created with Java 7 > could be unpacked with Java 7. > > But, again, the *.jar version should always be usable. The b3 aggregator has > high(er) standards when it comes to reporting errors -- as a feature :) > > But, does seem like using -Dorg.eclipse.update.jarprocessor.pack200 should > "work" ... and its a vague memory, but I seem to recall I've had trouble > getting that to "point to" the "right" place, at times.
Eclipse requires Java 1.8 now. Closing. Please open new bug if you still face some other unpacking issues.
