Bugzilla – Full Text Bug Listing
| Summary: | NPE when installing a signed, packed bundle that contains a nested jar on Windows | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Eclipse Project] Platform | Reporter: | Martin Oberhuber <mober.at+eclipse> | ||||||||||||||
| Component: | Update (deprecated - use Eclipse>Equinox>p2) | Assignee: | Platform-Update-Inbox <platform-update-inbox> | ||||||||||||||
| Status: | VERIFIED FIXED | QA Contact: | |||||||||||||||
| Severity: | blocker | ||||||||||||||||
| Priority: | P3 | CC: | ahunter.eclipse, aniefer, daniel_megert, denis.roy, hkyleung, jeffmcaffer, john.arthorne, Kevin_Haaland, kim.moir, nboldt | ||||||||||||||
| Version: | 3.2.1 | ||||||||||||||||
| Target Milestone: | 3.3 M6 | ||||||||||||||||
| Hardware: | PC | ||||||||||||||||
| OS: | Windows XP | ||||||||||||||||
| URL: | http://download.eclipse.org/dsdp/tm/signedUpdates | ||||||||||||||||
| Whiteboard: | |||||||||||||||||
| Bug Depends on: | 168583 | ||||||||||||||||
| Bug Blocks: | 149717, 175284, 188309 | ||||||||||||||||
| Attachments: |
|
||||||||||||||||
|
Description
Martin Oberhuber
Created attachment 53295 [details] .log file showing the bug Reproduced this issue today: * Start fresh installed eclipse-SDK-3.2.1 on WinXP * Software Updates > Find and Install > New Remote Site http://download.eclipse.org/dsdp/tm/signedUpdates * Select RSE Runtime / Core and ssh --> remotecdt is not required, as described earlier * Install All: when prompted by dialog, let Eclipse restart * Software Updates > Find and Install > same site as before * Select RSE Runtime / FTP * UM downloads files, "signed feature" dialog is shown * Press "Install All" button --> Installation aborts with an "Internal Error" --> .log file attached What's interesting is that I could NOT reproduce the issue on eclipse-3.2.1-linux-gtk, when I went to the same update site, selected ALL features there and installed all of them at once. It appears to be necessary that 2 features are installed first, Eclipse is restarted then, and then the other feature is installed. I was finally able to track this down: * The problem is only with the signed and packed jar, so it only happens with an 1.5 JVM, because only with 1.5 JVM will load the .jar.pack.gz file (1.4 will load the signed but not packed .jar file). That's the reason why I could not reproduce it on linux-gtk in comment 2, because I was running that under an 1.4 jvm. * The NPE happens in Update Manager because the signed, packed, nested jar is considered corrupt, but in CertVerifier.alreadyValidated() the status IVerificationResult.CORRUPT is not considered. So, it tries to access the root certificate which is not set (==null) in case the jar is considered corrupt. So, summing up -- I was providing the UM with a corrupt jar (see bug 135044 comment 56), but the UM should handle the corrupt jar more gracefully -- it should tell me which entry of the jar is considered corrupt, instead of bailing out with a NullPointerException. The fix should be easy by considering IVerificationResult.CORRUPT in CertVerifier.alreadyValidated(). Created attachment 54102 [details] Original unpacked unsigned jarfile Observed with Eclipse 3.3M3: It turns out that after a repack, sign, pack, unpack sequence with the latest jarprocessor from bug 135044 (as per today), jarsigner -verify considers the file OK on build.eclipse.org, but the Update Manager still runs into the NPE. Looks like the UM is not checking integrity of the signatures properly. I'm going to attach the repacked, signed, packed jar with the next comment. Created attachment 54104 [details] Repacked, signed, packed jarfile Here is the repacked, signed, packed jarfile causing the problems. I packed like this: #Use Java5 on build.eclipse.org - need JRE for pack200 export PATH=/shared/common/ibm-java2-ppc64-50/bin:$PATH export PATH=/shared/common/ibm-java2-ppc64-50/jre/bin:$PATH SITE=/home/data/httpd/download-staging.priv/dsdp/tm/test java -Dorg.eclipse.update.jarprocessor.pack200=$mydir \ $HOME/ws/jarprocessor/jarprocessor.jar \ outputDir $SITE -processAll -repack $SITE sign $SITE nomail java -Dorg.eclipse.update.jarprocessor.pack200=$mydir \ $HOME/ws/jarprocessor/jarprocessor.jar \ -outputDir $SITE -processAll -pack $SITE The file "pack200" in the local directory is a shellscript which enforces -E4 switch for packing in order to work around the Java1.5 limitation: #!/bin/sh # Workaround for downgrading effort of pack200 to avoid VM bug # See https://bugs.eclipse.org/bugs/show_bug.cgi?id=154069 pack200 -E4 $* After this, I verified the file like this: mkdir xx cd xx java $HOME/ws/jarprocessor/jarprocessor.jar -outputDir . -unpack ../*.jar jarsigner -verify *.jar jar xfv *.jar lib cd lib jarsigner -verify *.jar And it verifies OK, but UM tells me it were corrupt. Created attachment 54105 [details]
Associated feature for reproducing the problem in UM
Here is the feature.jar.pack.gz associated with the plugin.jar.pack.gz causing the problem. This can be used for reproducing the issue in UM.
I reproduced the issue with both Eclipse-3.3M3 and Eclipse-3.2.1 on a Windows XP computer running Sun Java 1.5.0_07
I have just been skimming this bug but comment 5 mentioned a packed feature. Why are we packing features? Pack200 only does interesting things for code jars. Features, in general do not have any code and if they do, it is very little. Are the Callisto features packed? Does this affect the creation of update site digests? Sorry for the confusion -- this is talking about a signed, packed bundle that contains a nested jar. Updated the summary accordingly. Made another test: with the latest and greatest jarprocessor, I added META-INF/eclipse.inf specifying jarprocessor.exclude.children.sign=true I had expected that this would force the sign script to sign the nested jarfile in its binary form. But after a repack, sign, pack, unpack, operation, jarsigner -verify claims that the nested jar were corrupt. Is this a bug in the sign script after all? Workaround: I finally got my jars to install as signed bundle with the following entries in META-INF/eclipse.inf: #do not sign individual entries of nested jars; sign the archive as a whole #instead; also do not pack until bug [163421] is resolved jarprocessor.exclude.children.pack = true jarprocessor.exclude.children.sign = true The disadvantage of this is, that the nested jar is not pack200'ed so my bundles are now 160K instead of 50K to download. My feeling is that the problem occurs because of the following: * Outer and inner nested jar are repacked (conditioned) * Outer and inner jar are signed * Because of signing the inner jar, many new strings are in the inner Jar's META-INF/Manifest.mf file, so the Stringtable changes; this leads pack200 to pack differently * When the signed inner jar is now packed and expanded, this step is not lossless any more, because the signed (modified) jar is not conditioned any more. So, the signature verification of the binary nested jar fails. In order to fix this, I think the "sign" script would need to have an additional option like "-repackAll" that does the following: 1. condition nested jar 2. sign nested jar 3. condition nested jar again 4. now sign the outer jar, including the binary nested jar It appears that currently step (3) is missing and because of that the verification fails. I would blame the corruption on bug 168583. However, Update should not throw an NPE when it gets a corrupt jar. It looks like a new manifestation of the issue is in bug 175284, visible on the Eclipse Platform M5 milestone update site. Martin, what leads you to conclude this bug is related to bug 175284? The exception stacks look quite different... We believed this NPE was a side-effect of a bug in Jarprocessor that has since been fixed (bug 168583). Can you confirm whether or not you have seen this NPE since bug 168583 was fixed? (In reply to comment #11) I originally concluded that bug 175284 is related to this one, because from a user's perspective it's exactly the same symptoms: packed and signed, nested jarfile, fails to install on Java5 or later. When originally working on this issue and filing this bug, I had actually seen various other backtraces occasionally, including the one captured in bug 175284. The kind of backtrace one gets seems to depend on whether there are other features being installed together with the problematic one. So, the NullPointerException had seemed to be the most obvious one. I also noticed that while bug 168583 had been closed, this one had never been closed, so I had concluded that nobody had understood or tested the real implications or verified that this one would indeed go away with the other fix. To answer your second question, I saw exactly the same NPE today when trying to get JDT 3.3M5eh via update manager into an eclipse-platform-3.3M5eh installation, so apparently this is not fixed in 3.3M5eh and/or the signing scripts used by the Platform and/or Platform basebuilder: * On Linux RHEL 4 / Sun Java 1.6.0, install eclipse-platform 3.3M5eh (build I20070222-0951) * Help > Software Updates > Find and Install > Search for additional features > New Remote site: http://download.eclipse.org/eclipse/updates/3.3milestones * Select "3.3M5eh > Java Development Tools * Finish, accept license, install into a new extension location, Install --> After installation started, I got this, which is exactly the same backtrace: !ENTRY org.eclipse.update.core 4 0 2007-02-27 12:29:42.799 !MESSAGE Digest could not be parsed:Unable to retrieve remote reference "http://download.eclipse.org/updates/3.3milestones/digest.zip". [Server returned HTTP response code: "404 Not Found" for URL: http://download.eclipse.org/updates/3.3milestones/digest.zip.] !ENTRY org.eclipse.update.core 4 0 2007-02-27 12:32:58.546 !MESSAGE Unable to complete action for feature "Eclipse Java Development Tools" due to errors. !STACK 1 org.eclipse.core.runtime.CoreException: Verification of feature unsuccessful. Installation cancelled. [java.lang.NullPointerException] at org.eclipse.update.core.Utilities.newCoreException(Utilities.java:223) at org.eclipse.update.core.Utilities.newCoreException(Utilities.java:254) at org.eclipse.update.core.Feature.verifyReferences(Feature.java:981) at org.eclipse.update.core.Feature.install(Feature.java:377) at org.eclipse.update.internal.core.SiteFile.install(SiteFile.java:96) at org.eclipse.update.internal.core.ConfiguredSite.install(ConfiguredSite.java:155) at org.eclipse.update.internal.core.ConfiguredSite.install(ConfiguredSite.java:119) at org.eclipse.update.internal.operations.InstallOperation.execute(InstallOperation.java:92) at org.eclipse.update.internal.operations.BatchInstallOperation.execute(BatchInstallOperation.java:84) at org.eclipse.update.internal.ui.wizards.InstallWizard2.install(InstallWizard2.java:375) at org.eclipse.update.internal.ui.wizards.InstallWizard2.access$1(InstallWizard2.java:372) at org.eclipse.update.internal.ui.wizards.InstallWizard2$1.run(InstallWizard2.java:485) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:58) org.eclipse.core.runtime.CoreException[0]: java.lang.NullPointerException at org.eclipse.update.internal.verifier.CertVerifier.alreadyValidated(CertVerifier.java:221) at org.eclipse.update.internal.verifier.CertVerifier.verify(CertVerifier.java:152) at org.eclipse.update.internal.verifier.CertVerifier.verify(CertVerifier.java:133) at org.eclipse.update.core.Feature.verifyReferences(Feature.java:967) at org.eclipse.update.core.Feature.install(Feature.java:377) at org.eclipse.update.internal.core.SiteFile.install(SiteFile.java:96) at org.eclipse.update.internal.core.ConfiguredSite.install(ConfiguredSite.java:155) at org.eclipse.update.internal.core.ConfiguredSite.install(ConfiguredSite.java:119) at org.eclipse.update.internal.operations.InstallOperation.execute(InstallOperation.java:92) at org.eclipse.update.internal.operations.BatchInstallOperation.execute(BatchInstallOperation.java:84) at org.eclipse.update.internal.ui.wizards.InstallWizard2.install(InstallWizard2.java:375) at org.eclipse.update.internal.ui.wizards.InstallWizard2.access$1(InstallWizard2.java:372) at org.eclipse.update.internal.ui.wizards.InstallWizard2$1.run(InstallWizard2.java:485) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:58) (In reply to comment #12) > > New Remote site: http://download.eclipse.org/eclipse/updates/3.3milestones > !MESSAGE Digest could not be parsed:Unable to retrieve remote reference > "http://download.eclipse.org/updates/3.3milestones/digest.zip". [Server > returned HTTP response code: "404 Not Found" for URL: > http://download.eclipse.org/updates/3.3milestones/digest.zip.] Looks like a typo (missing /eclipse/): http://download.eclipse.org/eclipse/updates/3.3milestones http://download.eclipse.org/updates/3.3milestones/digest.zip Thanks Nick, the error has been fixed and is replicating to eclipse.org. This did used to work but perhaps the apache alias has been changed. Verified that the bug is still in M6 candidate I20070320-0010. Setting severity blocker since this blocks all Europa projects from signing; but starting with signed bundles early is important in order to find other potential issues due to signing. Tested on Windows XP, Sun Java 1.5.0_11: - Add update site "http://download.eclipse.org/dsdp/tm/signedUpdates" - Select RSE Extender SDK - Install all It fails with a java.lang.SecurityException: SHA1 digest error for META-INF/eclipse.inf when trying to install org.eclipse.rse.core.source_2.0.0.v2007022-7J-_7_4kMNfMqf_.jar.pack.gz --> Backtrace attached. I'm pretty sure that the bug is in the update manager client, because I tried to manually download the jar.pack.gz file in question, an process it on commandline. Jarsigner -verify reports OK on both Windows and Linux: I:\rsetest\x>unpack200 org.eclipse.rse.core.source_2.0.0.v2007022-7J-_7_4kMNfMqf _.jar.pack.gz org.eclipse.rse.core.source_2.0.0.v2007022-7J-_7_4kMNfMqf_.jar I:\rsetest\x>jarsigner -verify org.eclipse.rse.core.source_2.0.0.v2007022-7J-_7_ 4kMNfMqf_.jar jar verified. Created attachment 61531 [details] .log with exceptions on I20070320-0010 (In reply to comment #15) attaching .log with exceptions on I20070320-0010 (In reply to comment #14) > Thanks Nick, the error has been fixed and is replicating to eclipse.org. > > This did used to work but perhaps the apache alias has been changed. Platform has owned the update.eclipse.org host traditionally, so perhaps you meant: http://update.eclipse.org/updates/3.3milestones/digest.zip Either way, the above is the same as http://download.eclipse.org/updates/3.3milestones/digest.zip The Apache alias hasn't changed. Sorry for the noise. I meant http://update.eclipse.org/updates/3.3milestones/digest.zip is the same as http://download.eclipse.org/eclipse/updates/3.3milestones/digest.zip Kim, if you don't need update.eclipse.org anymore, let me know and we'll repurpose it :-) (In reply to comment #15) > Verified that the bug is still in M6 candidate I20070320-0010. Interestingly, exactly the same jar.pack.gz that is problematic with I20070320-0010 on Windows installs OK with I20073020-0010 on Linux/Sun 1.6.0 IMHO this is an indication that the problem is somewhere with CRLF processing, similar to bug 168583 but slightly different. I have identified a small bug in the jarprocessor, which somehow escaped the fix for bug 168583. It results in a single byte's difference when unpacking something packed on a different platform (thank you martin for comment #19). I'm just testing now to see if this is the only issue. Created attachment 61563 [details] patch Attached is a single line patch. Without this patch, the pack.gz file from comment #4 failed to verify when unpacked on windows. With this patch, the verify succeeded. As well, with this patch, I was able to update dsdp as per comment #15. We should try and get this commited for the next M6 build What a difference a new line makes :-). Versioned and released. Please verify in the 4pm build. Verified with I20070321-1800 On Windows XP SP1, Sun 1.5.0_11 and Linux RHEL4, Sun 1.6.0 with the package from http://download.eclipse.org/dsdp/tm/signedUpdates that used to fail before. Thanks for getting this fix into M6! |