Community
Participate
Working Groups
Andrew has written an improved signing/packing application. It takes the following input: - A zip file name - An output directory - A list of commands to run (pack, unpack) - The name of a signing script It then recursively walks over the input zip file, and extracts all nested JARs/ZIPs to arbitrary depth. For each JAR encountered, it runs the Java pack200 compression algorithm, and then invokes the provided signing script. It has further options for supplying a list of exclusions for both the signing and packing steps, but I won't get into the details because the exclusion list is supplied in a file within the input zip, and is provided by the feature being signed/packed. The following changes need to be made to the signing scripts on download.eclipse.org: 1) Create a new script that will invoke this new packing/signing tool rather than the current /usr/bin/sign.sh. /usr/bin/sign.sh will instead be invoked by the new packing tool for each JAR that it encounters. 2) Have this new script operate on the signing queue that is fed by /usr/bin/sign script. This change will address the following: 1) Allow nested JARs to be signed 2) Allow JAR compression on the Callisto update site (Bug 127375 -> huge bandwidth savings) 3) Allow projects to specify exclusion rules if there are any JARs that must not be either signed or packed for technical or legal reasons.
BANDWIDTH SAVINGS ALERT Um, where do I get my grubby hands on this application?
Seriously, where is the bug that: a) requests the creation of this tool b) requests community feedback on its functionality c) requests community feedback on its features d) requests IT feedback on security/performance requirements for running software on our servers? D.
What's the easiest way for you to consume it? It will likely require further tweaks over the next month, but we'd like to at least test it in the eclipse.org environment soon. For now, should we just stick it on a web page and give you the link? The application is actually buried in a plugin in the Eclipse SDK - but it can be run standalone by just taking the plugin JAR file.
The main bug reports are bug 78208 for the signing, and bug 127375 for the enhanced update facility. The tool was needed to satisfy the requests in those two bug reports. I'm not sure what you need for IT feedback on security/performance requirements. Can you elaborate on the kind of information you're looking for?
The first version of this tool has been attached to bug 127375 as a patch to org.eclipse.update.core. This version does packing/unpacking/signing of jars. It does not currently recurse into nested jars because of issues getting signed jars containing nested jars to verify after packing & unpacking.
Thanks for the links. > I'm not sure what you need for IT feedback on security/performance > requirements. Can you elaborate on the kind of information you're looking for? - What license does this software use? I need to make sure I can legally run it. - Who maintains it? Where do I get fixes and patches? - What is the security record of this application? - How has it been tested? Are there any known memory leaks? - Does this open TCP ports? Are there any backdoors? - You say it extracts files .. Where? Does it write temporary files, and if so, where? - What are the system requirements? Is it a binary file? If so, does it run on PPC-64? Itanium2? - Where do I find the documentation on how to use it? - Is it SMP- and multiuser-safe? - Comment 5 sais it does the signing. Comment 0 sais it uses our signing script. Which one is it? Of course the answer to many of those questions may seem obvious, but if this runs on eclipse.org servers, it can potentially affect the performance of our servers. Unzipping, signing, and packing is both disk and CPU intensive so running this in a busy environment is not necessarily trivial. Also, I don't have a history of running beta code on production servers. Feel free to put it on a page and attach the link, or to upload it to your home directory on dev.eclipse.org. I'll sandbox it and post results. D.
These are all very reasonable questions. I'll write up some documentation.
Note that you don't have to run this on a production server, just a server that has access to the update site files.
Created attachment 37873 [details] Jar processor application with source and readme
Created attachment 37872 [details] Jar processor application with source and readme
I think the best way to think of this application is a part of the PDE build process. It is run on an eclipse.org machine because it needs access to the foundation's signing certificates, but otherwise it is owned, maintained and run by the release engineering of the eclipse projects that want to use it. I.e., nobody is expecting the webmaster to own and maintain this. Here are some answers to specific questions: Q: What license does this software use? I need to make sure I can legally run it. A: It is entirely EPL code written by Eclipse committers. By attaching it to this bug it is subject to the Eclipse licensing terms. Q: Who maintains it? Where do I get fixes and patches? A: It will be maintained by the Eclipse committers who are using it. Currently, committers on the platform core and release engineering teams (although future patches from other committers would be welcome ;) Q: What is the security record of this application? A: This is a tough one, since it was mostly written this week. I will say that it must only ever be run by a trusted party, because it will sign arbitrary JARs with the eclipse foundation certificates. Q: How has it been tested? Are there any known memory leaks? A: No known memory leaks. It is not a long-running application - it executes on a zip, and then exits without holding memory after exiting. It has been tested by Andrew on a local machine, and I would like to do further testing this week by using it to sign and pack a copy of 3.2 M6 Eclipse SDK. Q: Does this open TCP ports? Are there any backdoors? A: The application itself does not open network connections. It does call out to the Java "jarsigner" application, which may be need to make network calls using the IETF timestamp protocol (RFC 3161) to obtain a trusted timestamp. It does not do this currently. The application has no backdoors per se. However, it does make two external calls - one to an arbitrary jarsigner script provided as a command line argument, and the other to the Java pack200 command (the location of which can also be specified by the caller). The caller has to be trusted that they won't use it to invoke arbitrary applications rather than the signing application. Q: You say it extracts files .. Where? Does it write temporary files, and if so, where? A: The output directory can be specified as a command line argument (-outputDir). Otherwise, it writes to the current working directory. It extracts files to the output directory, signs and packs them, rebuilds the final output file, and then deletes any temporary files. Q: What are the system requirements? Is it a binary file? If so, does it run on PPC-64? Itanium2? A: It's a Java application. Write once, run anywhere. Q: Where do I find the documentation on how to use it? A: There is a readme file in the JAR. I am also assembling a web page that gives details about it and provides people a way to get a copy of it (I'll post a link here when that is ready). Q: Is it SMP- and multiuser-safe? A: Yes. Q: Comment 5 sais it does the signing. Comment 0 sais it uses our signing script. Which one is it? A: It does signing by calling your signing script ;)
Next step: I'd like to test this manually by running it to sign/pack a copy of the Eclipse 3.2 M6. I can run this from my home directory without additional permissions, but a copy of Java 1.5 will be needed to run it with.
(In reply to comment #11) > ... I will say that > it must only ever be run by a trusted party, because it will sign arbitrary > JARs with the eclipse foundation certificates. I'm sure you all know how to "protect" this properly, and probably have this all worked out .... but, if this helps .... I have seen some "signing processes" have two levels of protection, one that could be used for "nightly" or "weekly" builds would be signed without requiring a passphrase, and appropriately marked "developement build certificate" or similar. This allows it to run in a purely automated way. But, then, the milestone and official releases would actually require a passphrase to do the signing, so these would be just a tad bit more secure, since some trusted person would actually have to be sitting there ready to enter the passphrase when prompted. Just thought I'd mention it. I myself do not think this is *required* ... I'd be happy with any signing at all! But .... in case it helps improve "ease of use" while maintaining appropriate levels of security.
John, thanks for your answers. Because you know better how to test this (and you probably know more Java than I) you can install a 1.5 JDK in your home directory as well and do your own testing without waiting for me? If not, I'll try to get 1.5 installed today. I suggest you use build.eclipse.org (ssh using your committer account) as the test platform, as that's the machine that we'll use to sign. D.
As per bug 134264, the IBM java sdk 1.5 has been installed in /shared/common/ibm-java2-ppc64-50/ Planning to work on updating the signing scripts next week. D.
Created attachment 37948 [details] Improved JAR signer The previous revision didn't properly handle recursive packaging.
I successfully ran this last night on an Eclipse SDK nightly build, and the result looks good. I used a fake signing script that uses a test certificate because I don't have permissions to execute Denis' signing script (I shouldn't because it cat's the certificate file that nobody should be able to see). The good news for Denis is that the signing "scripts" are now quite simple. Here is the script I used to launch the process (this would be kicked off by the cron job): #!/bin/sh /shared/common/ibm-java2-ppc64-50/jre/bin/java jarprocessor.jar -outputDir out -repack -sign signing-script.sh $1 Where signing-script.sh calls jarsigner on a single jar provided as a parameter. My signing script implementation was as follows (more complex for denis because he needs to pipe the certificates from the private location: #!/bin/sh /shared/common/ibm-java2-ppc64-50/bin/jarsigner -keypass password -storepass password $1 certificate-name
Created attachment 38429 [details] latest version The code for this jarprocessor has been applied to org.eclipse.update.core, see bug 127375. Attached is the latest version, differences from the previous version are largely cosmetic.
> parameter. My signing script implementation was as follows (more complex for > denis because he needs to pipe the certificates from the private location: > > #!/bin/sh > > /shared/common/ibm-java2-ppc64-50/bin/jarsigner -keypass password -storepass > password $1 certificate-name My signing script allowed for a .zip or .gz extension, unzipped said file, signed all the JARs inside and rezipped after signing was done. Do I still need to provide this functionality, or is this provided by jarprocessor.jar? ie., John, does this work for you? #!/bin/sh /shared/common/ibm-java2-ppc64-50/jre/bin/java jarprocessor.jar -outputDir out -repack -sign signing-script.sh eclipse-SDK.zip #!/bin/sh /shared/common/ibm-java2-ppc64-50/bin/jarsigner -keypass password -storepass password eclipse-SDK.zip certificate-name D.
jarprocessor handles the recursive unzipping of zip files, and then it calls an external signing script for each JAR file it encounteres. So, all the signing script needs to handle is a single JAR file rather than a zip. I.e., it should essentially just supply the certificate name and passwords, and call jarsigner on the input JAR provided as a parameter.
The jarprocessor will work on a zip file, but not on .gz files. Also, the zip files are not handled recursively, we will process all the jars in the zip (and recursively process nested jars in those jars) but we don't recurse on nested zips in the zip.
Ah. So this is the workflow: 1. Trusted Committer puts [.jar|.zip|.tar.gz] file in the downloads staging area 2. Trusted Committer issues this command on any of the eclipse servers: /usr/bin/sign [.jar|.zip|.tar.gz] file to add to the queue 3. genie processes the queue on the signing machine 4. genie launches jarprocessor.jar on each file in the queue, regarless of the extension 5. jarprocessor.jar calls the signing script for each JAR it encounters 6. signing script invokes jarsigner, reading the keystore and store password 7. jarprocessor.jar does its majic and outputs the signed/packed file wherever specified. 8. genie notifies the Trusted Committer that signing is done, according to bug 132046 Make sense?
(In reply to comment #21) > but we don't recurse on nested > zips in the zip. The current signer doesn't do this either, and I don't believe it was a requirement. Thanks
Yes, that workflow is accurate. Minor exceptions: The jarsigner doesn't handle tar/gzip, as mentioned by Andrew. I'm not sure if tar/gzip support is needed, since zip is available on all platforms. jarsigner allows an output directory to be specified. Ideally the trusted committer could supply the output directory parameter in step 2, and it would get passed through to jarsigner. That would remove the need for any notification that the signing is complete.
I modified the queue processing to invoke jarprocessor.jar. I copied org.eclipse.ui.win32_3.1.0.jar to the staging area and added it to the queue. It seems to have been signed successfully: running jarsigner -verify -verbose reports: jarsigner -verify -verbose org.eclipse.ui.win32_3.1.0.jar 1703 Apr 20, 2006 2:54:28 PM META-INF/MANIFEST.MF 1464 Apr 20, 2006 2:54:28 PM META-INF/ECLIPSE.SF 3487 Apr 20, 2006 2:54:28 PM META-INF/ECLIPSE.RSA [snip] I copied eclipse-SDK-3.2RC1-win32.zip to the staging area and added it to the queue. jarprocessor.jar worked for a while, and I saw pack200 in the process list also, and the original file seems to have changed. However, after unzipping the "signed" SDK, running jarsigner -verify -verbose on random jars in the plugins directory shows no signing occurred. No error message was output to the console. D.
Are you using the jarsigner from comment #18? The first version didn't properly handle recursive signing. I'll give this a shot tomorrow and see what I can find. If it succeeds, signing 3.2 RC1 should take about 45 minutes (we ran several tests on this last week). Andrew: we had talked about a -verbose flag to jarsigner for producing more useful output. Have you done anything like this? We need to minimally know when the signing step has failed, because I recall there was no indication of this when our early signing attempts failed due to script permission problems.
I haven't done a -verbose mode yet, but it shouldn't be too hard and I can do it tomorrow. Since the signing worked on the individual .jar, I would expect it to work for jars out of the .zip file as well. One thing to note is that while working, the output jars would be placed in a eclipse-SDK-3.2RC1-win32.zip.temp zip file in the output directory. This file is renamed to eclipse-SDK-3.2RC1-win32.zip at the end. So if you interupted the processor before it ran to completion, are you looking at the correct output?
(In reply to comment #26) > Are you using the jarsigner from comment #18? The first version didn't > properly handle recursive signing. > Yes.
Created attachment 39183 [details] jarprocessor with verbose mode This jarprocessor.jar contains a -verbose mode and corresponds to the patch https://bugs.eclipse.org/bugs/attachment.cgi?id=39181 on bug 127375 which has not yet been released to head.
Denis, I can't figure out what may be wrong for you. I'm not able to run your signing script because I don't have write permissing in the staging directory where the input file must be placed (/home/data/httpd/download-staging.priv/). However, I ran jarprocessor in my own home directory again on build.eclipse.org and it worked fine. You can see the scripts I used in my home directory (sign-pack.sh to kick the whole thing off, and my-sign.sh for calling jarsigner). The only thing I can think of is permission problems: write access to the output directory, execute permission on all the scripts, etc. If you try running with Andrew's latest attachment and the -verbose flag, it should give more detail about the progress it is making.
Created attachment 52078 [details] Latest jar signer with source Webmaster, can you upload this latest version of the jarsigner application? This fixes a few bugs in the original attachment, adds a -verbose mode, and adds the ability for a nested JAR to opt out of signing and/or packing (bug 146066). Also, the "processAll" command line argument needs to be added when calling jarprocessor. This is because the default usage of jarprocessor is by Eclipse Update. When packing jars for use in Eclipse Update, we must only pack jars that have already been conditioned by the pack utility. The processAll argument specifies that all JARs should be processed regardless of whether they have been previously conditioned by pack200. We'll likely need another iteration or two of jarprocessor before we are done, but I'd like to get this one uploaded so the releng team can start running test builds (although springing it on them during milestone week could also be fun :). The version of jarprocessor currently on build.eclipse.org cannot handled nested zips of jars so we cannot run test builds with it. Source code is attached in the jar. As always, feel free to ping me with any questions or follow up here.
Created attachment 52169 [details] new Latest This latest jarprocessor.jar (with source) corresponds to the patch on bug 159028. (attachment https://bugs.eclipse.org/bugs/attachment.cgi?id=52167&action=view)
Denis, assuming you haven't already updated it, can you take the latest jarprocessor.jar from Andrew's comment #32? Sorry for the churn, but the goal of this recent work is to allow us to specify all signing and packing requirements within the jar to be signed, thus avoiding passing parameters around and avoiding work on your end every time we have a change in signing/packing requirements.
(In reply to comment #33) > Denis, assuming you haven't already updated it, can you take the latest > jarprocessor.jar from Andrew's comment #32? Sorry for the churn, but the goal > of this recent work is to allow us to specify all signing and packing > requirements within the jar to be signed, thus avoiding passing parameters > around and avoiding work on your end every time we have a change in > signing/packing requirements. By "we" you mean "every committer on every project who will need to sign jars", right? Command-line parameters are a great way to make the system more flexible for all. I installed the newest jarprocessor.jar file, and changed the command line as such: java jarprocessor.jar -outputDir $DIR -repack -processAll -sign sign.sh $FILE where sign.sh calls jarsigner BTW, now that we have an official certificate and all, are we ready to allow all the projects to begin signing their plugins? I understand that Platform is not quite ready to ship a signed build, but the other projects could at least sign their stuff now, right? D.
>By "we" you mean "every committer on every project who will need to sign jars", >right? Command-line parameters are a great way to make the system more >flexible for all. Well, I can't say for certain that we are handling every possible requirement for all projects yet, but that's our hope. The latest changes allow someone to add a configuration file to the zip which will configure how the signing/packing is done for their jars. Usually command line arguments are great, but in this case there is a level of indirection created by the signing queue that makes passing parameters difficult. By embedding the parameters in a configuration file in the zip it should be easier to manage. > I installed the newest jarprocessor.jar file, and changed the command line Thanks! We'll start running tests on this right away. >BTW, now that we have an official certificate and all, are we ready to allow >all the projects to begin signing their plugins? I understand that Platform is >not quite ready to ship a signed build, but the other projects could at least >sign their stuff now, right? Absolutely. I was going to wait until we had run a few test builds before broadcasting it, but this signing infrastructure is fair game for all projects. Feel free to put the word out to other projects. All you need to manage on your end is adding representatives from each project to the "signers" ACL.
Once the test builds are successful, I'll add a short note to the platform releng wiki on how we integrated signing into our build scripts. This may help other teams avoid any gotchas that we encountered in our testing.
> every time we have a change in > signing/packing requirements I guess I just wanted to be super clear that we'll be using jarprocessor.jar for all the projects, not just Eclipse, so care should be taken before modifying it because "we" (Eclipse project) have a change in requirements. Of course you knew this, I just wanted to be explicit about it. > By embedding the parameters in > a configuration file in the zip it should be easier to manage. All the projects need to know how to embed their parameters in ZIP files (I currently don't know how). I believe we should document the jarprocessor functionality on a wiki page, unless it's documented somewhere else.
There is currently some (slightly old) documentation on the jar processor in the Pack200 wiki page: http://wiki.eclipse.org/index.php/Pack200#Jar_Processor. I will split it out into its own page and update it to reflect the latest changes. As well, I will link the updated version from the Jar Signing page.
>I guess I just wanted to be super clear that we'll be using jarprocessor.jar >for all the projects, not just Eclipse, so care should be taken before >modifying it because "we" (Eclipse project) have a change in requirements. Understood. We'll consider jarprocessor API from this point. Adding new features isn't a problem, but we must not break anyone who is using it (unless everyone agrees to it). This makes me think we'll likely need a parallel signing script (or an isolated machine) at some point for running tests of new versions of jarprocessor without fear of breaking anyone else.
Denis Could you please add -verbose to the signing parameters in the /usr/bin/sign script Also, could you add another parameter that will allow us to pipe the output somewhere of the script somewhere to look at it. If you want to make a sign-debug script to allow you retain the original script, that would be fine too. These are the problems that I am seeing The resulting zip is not stored in the correct directory but rather the existing zip is updated. The resulting zip is not signed, but it is packed. It's hard to troubleshoot this from my end because I don't have any error messages etc. Here's the command I use my_staging_dir> /usr/bin/sign path_to_zip/eclipse-master-N20061002-2110.kim.zip mail path_to_output_dir
One possibility is that the output directory argument isn't being passed along correctly by the signing queue. We ran tests with this same version of jarprocessor on build.eclipse.org before attaching it here, and those tests ran fine. I'm wondering if the output directory argument is being lost and that's why Kim is seeing the input JAR being overwritten and incorrectly signed when invoking /usr/bin/sign.
There's a problem with the output directory being stored in the queue. It's stored as the current directory. I'll fix this.
I've fixed the problem with the outputDir. I've also added a restriction that outputDir needs to be in the staging area, as our signing genie may not have write access to the outputDir.
Denis, Thanks for fixing this. The file now goes in the correct output directory :-) However, I'm still seeing the issue where the resultant file is conditioned, but not signed. I've sent you a private email with the location of a test zip containing a small subset of the eclipse plugins that need to be signed. Could run the script with this file and see from your end if there are any errors that indicate why it is not being signed. Or perhaps you could enable the logging the output of the script from your end so I could try to troubleshoot why the file isn't being signed?
Other issues I have noticed: If you run the script against a single jar (not zipped) it is signed and conditioned. If you run the script against a single zip file consisting of a single jar, it is only conditioned, not signed. I don't this problem running the signing portion of John's test script which is something like this jarsigner -keypass password -storepass password -verbose $1 test >> sign.log It seems like your signing process fails if the input file is a zip. jarsigner accepts a "-verbose" argument. For debugging it would be good if you added -verbose and then piped the output to a file. This would allow us to determine if there was a problem with the signing process. Today we can't tell if the file was properly signed or if there was an error other than verifying the resultant file. Are you able to reproduce this behaviour with the test file I provided?
Previously, my signing script would handle recursiveness through ZIP and .tar.gz files by unzipping them, then re-running the script on all JAR files inside. I am told that jarprocessor.jar now handles the recursiveness, but I am unsure of how this works. If we assume the base command is as per comment 17: /shared/common/ibm-java2-ppc64-50/jre/bin/java jarprocessor.jar -outputDir out -repack -sign signing-script.sh $1 If $1 is a .jar file, jarprocessor sends the .jar file to my signing script for signing. That is known to works. If $1 is a zip file, how does jarprocessor call the signing script? Once for each individual jar inside? This is what I assumed, as all the code to unzip and recursively sign JARS inside my signing script has been disabled. How does jarprocessor handle a ZIP file? I'll look into doing somethign with -verbose.
If $1 is a zip file, the jarprocessor should extract the jars to a temporary location under the outputDir, and then call the signing script on that extracted jar. Could there be a permissions problem here? Will these dynamically extracted files have the correct permissions needed for the signing script? The difference between the pack and the sign is that the pack only needs to read this extracted file and it will write its output to a new file. The sign however writes back to the same file.
It's the same user that runs jarprocessor and the signing. The files created by jarprocessor are owned rwx by the signer, so it's not a permissions issue. I launched the signing of the zip file that Kim gave me, and killed jarprocessor in the middle so I can "see" the temp files it created. I then su'd as the signer account and manually ran the signing script on one of the temporary jars extracted by jarprocessor. Signing was successful. I will try to capture the -verbose output somehow, because at this point, I have no clue.
This is almost funny. The directory created by jarprocessor ended with ".zip", and this would confuse the signing script (originally designed to handle .zips) because the filename had a portion ending with .zip and a portion ending with .jar I have fixed this, and ran the signer again. After signing was completed (took significantly longer), I unzipped the "signed" zip that was in outputDir and verified a jar: jarsigner -verify -verbose org.eclipse.core.boot_3.1.100.v20060603.jar 580 Oct 26, 2006 2:50:08 PM META-INF/MANIFEST.MF 452 Oct 26, 2006 2:50:08 PM META-INF/ECLIPSE.SF 3487 Oct 26, 2006 2:50:08 PM META-INF/ECLIPSE.RSA 0 Sep 22, 2006 12:42:44 AM META-INF/ sm 57 Oct 26, 2006 2:50:04 PM META-INF/eclipse.inf sm 1432 Sep 22, 2006 12:42:44 AM about.html sm 595 Sep 22, 2006 12:42:44 AM plugin.properties sm 147 Sep 22, 2006 12:42:44 AM readme.txt s = signature was verified m = entry is listed in manifest k = at least one certificate was found in keystore i = at least one certificate was found in identity scope jar verified. I have enabled logging to /tmp/jarsigner. This file is deleted before any queue handling occurs, so you can monitor output of jarprocessor, sign.sh and jarsigner. Let me know how this works out.
Awesome! Thanks Denis - we'll try a build on our end soon and see how it goes.
One current limitation is that the group owner of outputDir must have write permissions to allow the queue handler to write. I'm seeing this: java.io.FileNotFoundException: download-staging.priv/eclipse/N20061026-0010-out/ (Permission denied) I've set g+w that for the above directory, but I'll need to modify sigh.sh to make sure it is group writable.
(In reply to comment #51) This has been fixed. I also made the output files (the signed files) group writable also, so that teams can easily move/delete them.
Thanks Denis for all your help, I have released the changes and am running a test build. We hope to have signed builds in place for next week, after M3 is released.
I have create the wiki page http://wiki.eclipse.org/index.php/JarProcessor_Options with docs on the properties available. It is also linked from the JarProcessor section here: http://wiki.eclipse.org/index.php/Pack200#Jar_Processor
Denis: I think this bug can be marked fix. We are now running signed builds and it looks like it is working well. We can open new bug reports if any further issues come up. Thanks for all your help.
The one thing that's currently not clear for me, is how Denis' "sign" script can interact with pack200 for an update site. My problem is: I create the jars for my update site, then sign them, but when I run pack200 on them the signed jars get corrupted since the packer adds (or modifies) the Eclipse.inf files. It's problematic for single-jar plugins, it's even more problematic for nested jar's (because the nested jar gets packed recursively, thus destroying the signature). For me it feels like signing and pack200 must be integrated in order to resolve this, i.e. have a single sign-and-pack application. Just having signed ZIPs come out of the build is not the solution, until you also have a signed and packed update site. Or is there any solution to this that I have overlooked?
Martin: You're absolutely right that signing and pack200 need to be integrated. Pack200 is "lossy" in that the bytes returned after unpacking are not always identical to the bytes input to the packing (code is functionality identical, but there is some rearranging of things like the constant pool in class files). Thus packing after signing will corrupt the signatures and verification will fail. On the other hand, signing has to occur on unpacked content, so it seems like a catch-22. The solution is a three step approach: 1) Call pack200 with -repack argument. This conditions the JAR content so that subsequent pack pack/unpacks will produce identical bytes. 2) Sign JAR 3) Call pack200 again on the signed content to produce the pack.gz compressed content. The JarSigner tool called by Denis' script performs 1) and 2). You then need to perform step 3) afterwards. The easiest way to do this on recursive zips of jars is to call JarSigner again. See this page for more details: http://wiki.eclipse.org/index.php/Platform-releng-signedbuild JarSigner is capable of performing steps 1, 2, and 3 all together, but it requires passing the additional -pack argument. There is currently no way to pass this extra arguments through the indirection of Denis' signing scripts. However, there is absolutely no performance gain in combining step 3) with the first two steps, and it may be more efficient to perform step 3) on a different machine to avoid copying the packed content (depending on the details of your end to end build process).
I should add that this wiki page describes the relationship between packing and signing and describes the JarProcessor tool: http://wiki.eclipse.org/index.php/Pack200 And this page on signing has recently been updated with additional links, and information on potential runtime effects of signing: http://wiki.eclipse.org/index.php/JAR_Signing
Thanks for your help. In fact I did try 1.) Run jarprocessor -processAll -repack on the update site to condition the jars 2.) sign the jars with Denis' script 3.) Run jarprocessor -processAll -pack The problem I'm having is, that this doesn't seem to work for nested jars. To be more concrete, it's failing on http://download.eclipse.org/dsdp/tm/signedUpdates/org.apache.oro_*.jar.pack.gz -- see also bug 163421. For some reason, the UM is considering the repacked, signed, packed nested jar corrupt. I must admit that I did not use the latest and greatest jarprocessor, but the one from Eclipse 3.2.1 -- should that make any difference?
Martin can you try it with the JarProcessor that is attached here? (It was built from bug 159028, whose changes have not yet been released to CVS). The signing/packing/verification of nested jars has always been delicate, and I hope the problems you are seeing are from mixing versions of the jar processor. If it is still a problem, please raise a new bug with instructions on where to get an unprocessed version of your jar for testing and whatever options you are using.
Ha! It turns out that with the latest jarprocessor, when I unpack my fully processed jarfile on build.eclipse.org, jarsigner -verify claims that the file is OK. Still, the Update Manager runs into a NullPointerException and cannot install the very same file. All details are in bug 163421, including attachments of the original jar, the processed jar.pack.gz as well as the associated feature to reproduce the bug on Eclipse 3.3M3 or 3.2.1, Windows XP, Sun Java 1.5.0_07
Am I still good to close this as fixed?
When I understand right, the "new Latest" jarsigner is still not checked in to CVS but only attached here. For this reason, I'd wait with closing this until the jarsigner is officially available. Also, I'd love to see bug 163421 understood and hopefully fixed before closing this one, since fixing 163421 may have impact on the signer script.
Marking as dependent on the bug to release the jarprocesser changes. I'll close this when the changes are released.
Marking as fixed. The latest signing/packing application that is on eclipse.org has also been released into the Eclipse platform as part of the update component.
