Bugzilla – Full Text Bug Listing
| Summary: | Add signing capability to the CBI build for the eclipse platform | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Technology] CBI | Reporter: | Andrea Ross <andrea.ross> | ||||
| Component: | prototype | Assignee: | CBI Dummy user <cbi.prototype-inbox> | ||||
| Status: | RESOLVED FIXED | QA Contact: | |||||
| Severity: | normal | ||||||
| Priority: | P2 | CC: | curtis.windatt.public, daniel_megert, david_williams, igor, jan.sievers, john.arthorne, kim.moir, konstantin, pwebster | ||||
| Version: | unspecified | ||||||
| Target Milestone: | --- | ||||||
| Hardware: | PC | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Bug Depends on: | 387557, 390191, 390192, 390193 | ||||||
| Bug Blocks: | 372792 | ||||||
| Attachments: |
|
||||||
|
Description
Andrea Ross
Here is some more detailed information about implementing eclipse-style artifact signing with Tycho === The goals 1. MUST HAVE. builds run at eclipse infrastructure should produce artifacts signed with eclipse signing certificate and current signing procedures 2. MUST HAVE. it should be possbile to run platform build outside of eclipse infrastructure, without signing 3. NICE TO HAVE. it should be possible to run platform build outside of eclipse infrastructure using non-eclipse signing certificates and signing procedures === Tycho approach to artifact signing Normally, artifacts signing is expected to happen during each project's build and signed artifacts put in place of the original. This guarantees that all consumers of project output will use the signed signed copy of the artifacts. This is particularly important for the monolith platform build, where project output is included in at least two deliverables, the platform SDK archives and platform P2 repository. So overall flow for individual bundle and feature project will look like this - compile classes, package bundle jar, sources jar, etc - pack200 conditioning of all packaged jars - sign the jars In addition, for eclipse repository projects - generate pack200 files === Required implementation changes * pack200 conditioning and generation of pack200 files. This is generally useful and can be implemented either in Tycho (preferred) or in Dash. * signing of Maven project artifacts using Eclipse signing procedures. This is specific to Eclipse, so I suggest this as new goal of eclipse-signing-maven-plugin === Eclipse signing performance problems Eclipse signing procedures has significant per-signing request overhead, about one minute to sign a single jar, according to my observations. There are over 400 individual modules in the monolith platform build. This means that signing will add at least 400 minutes to overall build time. Additional investigation is required to understand why Eclipse signing procedures take such long per-request time and if it is possible to significantly reduce the signing time. Upping the priority on this ticket. Just a note on the current PDE Build based process: 1) We build all of the bundles into a p2 repo. 2) then we sign the repo (incur the overhead of the eclipse signing infrastructure once). 3) then we mirror the p2 repo through the comparator. 4) this conditioned repo is then used to create the platform SDK archives. Step 3 guarantees that if the last build contains bundleX_3.0.0.v20110523 and this build contains the same bundle (as the source hasn't changed) we use bundleX from the last build's p2 repo. PW FYI, see bug 334032 for (another) bug about slow signing turn-around when trying to sign jar by jar. > Eclipse signing procedures has significant per-signing request overhead, about
> one minute to sign a single jar, according to my observations. There are over
> 400 individual modules in the monolith platform build. This means that signing
> will add at least 400 minutes to overall build time.
Please allow me to explain the current design.
Files are placed in a queue by the 'sign' command on build.e.o and the Hudson slaves.
On build.e.o, the queue is processed by cron job every minute. I've recently changed this so that the queue is serviced every 20 seconds.
The reasons for the existence of a queue are:
1. Parallelism: multiple queues can be serviced simultaneously on build.e.o
2. Scalability: as deman increases, signing can be farmed out to many more machines
3. Throttling: the queue handlers actively examine current server load and can wait if too many signers are running concurrently. This prevents the server from becoming unresponsive from too many signers
I'm open to ideas on how to make this better. If the typical use-case is to submit 400 jars to sign, I can modify the current handler to break down the queue into smaller units so up to 16 handlers can run at once.
400+ jars need to be signed in sequence, one at a time. Current signing queue implementation takes about ~40 per jar (according to my measurements couple of months ago), 40 times 400 equals 16K seconds or ~4.5 hours just to sign everything. So you're submitting a file to sign, then you wait for it to be signed, then you submit another one? Wow. That seems immensely inefficient. Just zip up your jars and submit the zip. jarprocessor.jar will recurse into it and sign them all sequentially. That way it only takes about 1.5 second per jar to sign. Otherwise, I just submitted 570 individual jars to the queue, and I have three jarsigner processes running. > Otherwise, I just submitted 570 individual jars to the queue, and I have three
> jarsigner processes running.
Correction, 2 signers were running.
#1 processed 250 jars in 27 minutes
#2 processed 320 jars in 36 minutes
So 570 jars were signed in 36 minutes total, and the server was only using 20% of its CPU cycles.
So from here, I'd like to better understand what the requirement is.
(In reply to comment #7) > So you're submitting a file to sign, then you wait for it to be signed, then > you submit another one? > > Wow. That seems immensely inefficient. > Tycho fully builds each project, or "reactor module" as they call it, before proceeding to build the next project. Although it is possible to restructure the build to sign all project jars in one shot, this would make CBI build rather cumbersome to use and I would like to avoid doing this if at all possible. (In reply to comment #8) > > Otherwise, I just submitted 570 individual jars to the queue, and I have three > > jarsigner processes running. > > > Correction, 2 signers were running. > > #1 processed 250 jars in 27 minutes > #2 processed 320 jars in 36 minutes > > So 570 jars were signed in 36 minutes total, and the server was only using 20% > of its CPU cycles. > > So from here, I'd like to better understand what the requirement is. Tycho build looks something like this Project-001 compile package sign Project-002 compile package sign ... Project-450 compile package sign This is executed by a single thread. The requirement is to make each "sign" step take <10 seconds or even less than that (this is really up to the Platform developers to decide how fast is "fast enough", but personally I think spending over an hour signing is almost too long). > Tycho build looks something like this
>
> Project-001
> compile
> package
> sign
> Project-002
> compile
> package
> sign
Ah, I see. Thanks for that.
Simplest way for me to do this right now (and still retain black-box/privilege separation for signing) is to use xinetd. Instead of putting a file in a queue, a call to "sign file.jar" would hit a local TCP port which would be serviced immediately by xinetd. You could then wait for the signing response back from the TCP connection. I don't expect signing a single jar to take more than 1.5 second, and that includes fetching the secure timestamp from verisign.
I'll experiment with this tomorrow and we can try it out.
Immediate signing is now in place. Old signing is unaffected. Instead of calling sign jarfile.jar mail (or nomail) Use: sign jarfile.jar now Optionally, skip the repack process and save a second or two per jar: sign jarfile.jar now skiprepack (In reply to comment #12) > Immediate signing is now in place. Old signing is unaffected. > > Instead of calling > > sign jarfile.jar mail (or nomail) > > Use: > > sign jarfile.jar now Very cool, Denis, can we use that for our platform zip files too? :) > Optionally, skip the repack process and save a second or two per jar: > > sign jarfile.jar now skiprepack That assumes it is already "repacked", right? Otherwise, it will break once ran though pack200 after signed. > Very cool, Denis, can we use that for our platform zip files too? :) Sure. The intention was to allow quick signing of individual jars, but it uses the signing infra in place, so it will work for zips too. I think it's worth noting that the 'sign' command won't return until signing is complete... And there's no feedback. Perhaps I should output some feedback to the console... > > sign jarfile.jar now skiprepack > > That assumes it is already "repacked", right? Otherwise, it will break once ran > though pack200 after signed. Um, I don't know? I have an experimental web service for signing. For obvious reasons, it can only be reached from within the Eclipse.org firewall. Given a file (anywhere, doesn't need to be in the staging area): droy@build:$> ls -l total 392 -rw-rw-r--+ 1 droy technology.phoenix 389579 2012-04-11 16:53 org.eclipse.pde.core_3.0.0.jar Uploading the file as a mime-encoded attachment using POST (just like a browser) will return the file signed. Again, this method is highly experimental, and definitely only used for JAR files. droy@build:$> curl -F data="@org.eclipse.pde.core_3.0.0.jar" -o org.eclipse.pde.core_3.0.0-signed.jar http://build.eclipse.org:31338/ -F data="@filename.jat" specifies the local file to upload. -o filename-signed.jar is the target filename for the signed jar This example is using curl. Similar calls using wget or Java HTTP api should produce similar results. The "official" URI to sign is: http://build.eclipse.org:31338/sign I felt adding /sign in there added to the significance of the URI when looking at it in a script. I've also added docs: http://wiki.eclipse.org/IT_Infrastructure_Doc#Sign_my_plugins.2FZIP_files.3F I'll announce the new signing options on cross-project shortly. Please test it out and repoen if you see something missing. *** Bug 334032 has been marked as a duplicate of this bug. *** (In reply to comment #14) > > > > sign jarfile.jar now skiprepack > > > > That assumes it is already "repacked", right? Otherwise, it will break once ran > > though pack200 after signed. > > Um, I don't know? I should not have asked as a question. I mentioned a "warning" in bug 275516 comment 5, which I will now add to the signing wiki page. I saw that; thanks. I am having troubles using the new signing webservice. Although it works in most cases it silently fails for some jars. Here is what I see $ curl -F data="@org.eclipse.m2e.maven.runtime-1.1.0-SNAPSHOT.jar" -o xxx.jar http://build.eclipse.org:31338/ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 4174k 0 0 100 4174k 0 10.5M --:--:-- --:--:-- --:--:-- 10.5M The output file is not created and I do not believe HTTP response contains any reason of the failure. I left the sample jar in ~ifedorenk/public_html/org.eclipse.m2e.maven.runtime-1.1.0-SNAPSHOT.jar . The http signer service was inheriting the global file upload size limit of 2MB. I've increased that to 25MB and the file was signed successfully. Please give it a try now. This got me further, but I now get same/similar failure on /opt/public/cbi-ng/R4_platform-aggregator/eclipse.jdt.ui/org.eclipse.jdt.ui/target/org.eclipse.jdt.ui-3.8.0-SNAPSHOT.jar which is slightly under 10M. I also think signer should return HTTP 4xx code with a message explaining why the file was rejected. > This got me further, but I now get same/similar failure on > /opt/public/cbi-ng/R4_platform-aggregator/eclipse.jdt.ui/org.eclipse.jdt.ui/target/org.eclipse.jdt.ui-3.8.0-SNAPSHOT.jar > which is slightly under 10M. The POST size was limited to 8M, so I've increased that to 25M also. > I also think signer should return HTTP 4xx code with a message explaining why > the file was rejected. PHP terminates the connection before the script can actually be executed, so I can't control the response code. In any case, the content-length header has a '0' value, so that would be an even better indicator that something went wrong. I do not believe 25M is enough, there are jars that are close to 50M in juno release repository already. $ pwd; ls -lS | head /home/data/httpd/download.eclipse.org/releases/juno/201204110900/plugins total 1769152 -rw-rw-r-- 1 david_williams callistoadmin 48520272 2012-05-10 01:03 org.eclipse.scout.rt.rap.target_3.8.0.201205091247.jar -rw-rw-r-- 1 david_williams callistoadmin 35428033 2012-05-10 01:03 org.eclipse.platform.doc.isv_4.2.0.v20120502-1351.jar -rw-rw-r-- 1 david_williams callistoadmin 19910207 2012-05-10 01:03 org.eclipse.platform.doc.isv_4.2.0.v20120502-1351.jar.pack.gz -rw-rw-r-- 1 david_williams callistoadmin 17370032 2012-05-10 01:03 org.eclipse.modisco.doc_0.10.0.v201205091435.jar -rw-rw-r-- 1 david_williams callistoadmin 14458581 2012-05-10 01:02 org.eclipse.cdt.doc.user_5.2.0.201205092110.jar -rw-rw-r-- 1 david_williams callistoadmin 14087675 2012-05-10 01:02 org.eclipse.cdt.doc.user_5.2.0.201205092110.jar.pack.gz -rw-rw-r-- 1 david_williams callistoadmin 10328912 2012-05-10 01:02 org.eclipse.jdt.ui_3.8.0.v20120502-1904.jar -rw-rw-r-- 1 david_williams callistoadmin 8235499 2012-05-10 01:02 org.eclipse.birt.report.data.oda.excel_4.2.0.v20120508.jar -rw-rw-r-- 1 david_williams callistoadmin 8121857 2012-05-10 01:03 org.eclipse.wst.jsdt.ui_1.1.201.v201204101526.jar Ok, I cranked everything up to 1G If it explodes, it explodes. I think we can declare victory on this on. Testing showed it working well. Great job Denis & Igor! (In reply to comment #26) > I think we can declare victory on this on. Testing showed it working well. > Great job Denis & Igor! I think this is premature. For CBI/Tycho signing capabilities to be truly useful, eclipse-signing-plugin needs to be available from a well-maintained and supported maven repository. That's a good & fair point. I'll open a new bug to track that as it is something we need in a generic sense. I did some real ;-) testing of cbi signing and pack200 support using m2e as guinea pig and ran into bug 387557. This means that signed and pack200'ed artifacts produced by cbi cannot be installed by current p2. From an eclipse Platform build point of view, this is close to done. We just have 2 places where we currently have to delete the eclipse.inf: eclipse.jdt.core/org.eclipse.jdt.core/META-INF/eclipse.inf eclipse.pde.build/org.eclipse.pde.build/META-INF/eclipse.inf Both have jarprocessor.exclude.children=true in them. PDE build needs that specified, right David. But as it is this causes trouble with tycho. One suggested alternative was to exclude these bundles from packing as well (although it won't effect signing). jarprocessor.exclude.pack=true Thoughts? PW (In reply to comment #30) > From an eclipse Platform build point of view, this is close to done. > > We just have 2 places where we currently have to delete the eclipse.inf: > > eclipse.jdt.core/org.eclipse.jdt.core/META-INF/eclipse.inf > eclipse.pde.build/org.eclipse.pde.build/META-INF/eclipse.inf > > Both have jarprocessor.exclude.children=true in them. PDE build needs that > specified, right David. But as it is this causes trouble with tycho. > > One suggested alternative was to exclude these bundles from packing as well > (although it won't effect signing). > > jarprocessor.exclude.pack=true > > Thoughts? > > PW I was thinking jarprocessor.exclude.children=true jarprocessor.exclude.pack=true which should produce consistent results both in Tycho and PDE/Build.
>
> I was thinking
>
> jarprocessor.exclude.children=true
> jarprocessor.exclude.pack=true
>
> which should produce consistent results both in Tycho and PDE/Build.
You mean consistent from here on, right? But not with previous releases, right?
In Juno SR0 the packed jdt.core jar was a lot smaller than the non-packed version.
4575K org.eclipse.jdt.core_3.8.1.v20120531-0637.jar
1738K org.eclipse.jdt.core_3.8.1.v20120531-0637.jar.pack.gz
So ... I'm not that concerned about one case, especially if temporary solution, but hate to think we'd lose that ability everywhere, forever. Over all of Eclipse Foundation projects it might start to make a substantial difference.
(Though many people wouldn't mind giving up packing altogether (ha ha)).
But mind if I ask for some clarification? Sorry I've not followed this closer.
I thought I remember reading that
jarprocessor.exclude.children.pack
would "be ok" and that
jarprocessor.exclude.children.sign
would "be ok".
Did I get that wrong?
If that is correct, then wondering how
jarprocessor.exclude.children (which is just short hand for both)
would make it not ok.
Thanks,
(In reply to comment #32) > > > > I was thinking > > > > jarprocessor.exclude.children=true > > jarprocessor.exclude.pack=true > > > > which should produce consistent results both in Tycho and PDE/Build. > > You mean consistent from here on, right? But not with previous releases, > right? > > In Juno SR0 the packed jdt.core jar was a lot smaller than the non-packed > version. > > 4575K org.eclipse.jdt.core_3.8.1.v20120531-0637.jar > 1738K org.eclipse.jdt.core_3.8.1.v20120531-0637.jar.pack.gz > > So ... I'm not that concerned about one case, especially if temporary > solution, but hate to think we'd lose that ability everywhere, forever. Over > all of Eclipse Foundation projects it might start to make a substantial > difference. > (Though many people wouldn't mind giving up packing altogether (ha ha)). > > But mind if I ask for some clarification? Sorry I've not followed this > closer. > I thought I remember reading that > jarprocessor.exclude.children.pack > would "be ok" and that > jarprocessor.exclude.children.sign > would "be ok". > Did I get that wrong? > If that is correct, then wondering how > jarprocessor.exclude.children (which is just short hand for both) > would make it not ok. > > Thanks, no eclipse.inf jarprocessor.exclude.pack=true jarprocessor.exclude.sign=true jarprocessor.exclude.pack=true && jarprocessor.exclude.sign=true OK jarprocessor.exclude.pack=false && jarprocessor.exclude.sign=false Not OK. In all OK cases Tycho and PDE/Build will produce different results for bundles with nested jars unless jarprocessor.exclude.children=true. Here is more detailed background information We have to make three tools place nicely together -- Tycho, PDE/Build and P2. Due to a bug in P2 Juno and earlier, Tycho is not able to produce packed and signed bundles that have META-INF/eclipse.inf file. If compatibility with "older" versions of P2 was not a requirement, both Tycho and PDE/Build could produce packed&signed bundles. I don't think we can ignore Juno P2 however. There is also incompatibility between Tycho and PDE/Build, where the former never packs or signs nested jars and the latter packs and signs nested jars by default. If compatibility with PDE/Build was not a concern, it would be possible to remove eclipse.inf from the affected projects and Tycho would produce packed&signed bundles similar to what was shipped with Juno SR0. I believe this leaves us with the following alternatives, in no particular order * backport p2 bug 387557 to Juno and give up on compatibility with earlier p2 versions * drop requirement to be able to build with both PDE/Build and Tycho * use exclude.children=true/exclude.pack=true for the two affected projects, which will result in diminishingly small increase in download size * "teach" tycho how to produce eclipse.inf in the undocumented/proprietary format supported by p2 (this will have to be pure cbi solution as I do not believe this code is generally useful in Tycho). Personally, I think exclude.children=true/exclude.pack=true provides the best tradeoff given the alternatives. Also, I want to stress once again that the problem only affects packed and signed bundles with nested jars and eclipse.inf. All other bundles are not affected (In reply to comment #33) > Personally, I think exclude.children=true/exclude.pack=true provides the > best tradeoff given the alternatives. > I'm leaning towards this myself, as it would work in both PDE build and tycho, but would like to hear any opinions from the teams effected first. It's really a releng/build time parameter, so it should not effect behaviour. PW Created attachment 221279 [details]
List of eclipse.inf file with "exclude children"
I went through all the Juno (SR0) jars on the common repo site and found only 20 that used
jarprocessor.exclude.children = true
So assuming a world that everyone was using this system ... there would be 20 additional jars that could not be "packed". That's out of about 4450. I found this re-assuring that it would not effect overall size or bandwidth in the big picture of things. And reassuring not too many projects would have to "convert" too many eclipse.inf files to move to this new system.
For jdt.core the eclipse.inf file was originally added to workaround a problem with invalid signatures on nested packed JARs (193570). So, one could say this might not even be needed anymore (this is worth a try). However, there's currently a more severe problem coming along the road regarding pack200 and nested JARs when running Java 7, see bug 361628 and bug 378200. (In reply to comment #32) > In Juno SR0 the packed jdt.core jar was a lot smaller than the non-packed > version. > > 4575K org.eclipse.jdt.core_3.8.1.v20120531-0637.jar > 1738K org.eclipse.jdt.core_3.8.1.v20120531-0637.jar.pack.gz > David, I am not sure what these numbers should poof. The 4575K JAR is compressed using pack200. So, the interesting number would be the size of that JAR but not with pack200. Could you provide that number to judge whether we could drop pack200? (In reply to comment #36) > David, I am not sure what these numbers should poof. The 4575K JAR is > compressed using pack200. So, the interesting number would be the size of > that JAR but not with pack200. Could you provide that number to judge > whether we could drop pack200? The nightly shouldn't have any packing or normalizing applied to it, right David? 4696218 org.eclipse.jdt.core_3.9.0.N20120913-2000.jar. About 4586K (less?) PW (In reply to comment #37) > (In reply to comment #36) > > David, I am not sure what these numbers should poof. The 4575K JAR is > > compressed using pack200. So, the interesting number would be the size of > > that JAR but not with pack200. Could you provide that number to judge > > whether we could drop pack200? > > The nightly shouldn't have any packing or normalizing applied to it, right > David? Correct. I just compared the JARs and the difference is pretty small (~10k). I'd still like to try removing the inf file entirely and see how it goes. @Paul > The nightly shouldn't have any packing or normalizing applied to it, right > David? Sometimes I have had my doubts But, I just confirmed not conditioned (normalized). @Dani > > 4575K org.eclipse.jdt.core_3.8.1.v20120531-0637.jar > > 1738K org.eclipse.jdt.core_3.8.1.v20120531-0637.jar.pack.gz > > > > David, I am not sure what these numbers should poof. The 4575K JAR is > compressed using pack200. So, the interesting number would be the size of > that JAR but not with pack200. Could you provide that number to judge > whether we could drop pack200? This is probably a mute point, since I think we all agree with Igor's proposal, but I feel obligated to answer. The 4575K JAR is not _compressed_ using pack200 ... it is merely "conditioned" using pack200 for later compression (and the reason for that conditioning is so that it can be packed more efficiently, and the packed (compressed) version survive signing verification once uncompressed. So, the proposal means there would no longer (ever) be the smaller "pack.gz" version so anyone using p2 say to create a package starting with just director or raw platform would "suffer" through the extra 3 Megabyte download. And my concern was never about just one jar ... but more the effect on the whole eco-system or simultaneious release, once we convince everyone to move to tycho/CBI (well ... almost everyone :) But my later analysis was that there are not very many of these cases to worry about; just 20 or so out of 5000 and several hundred are already not compressed, for various reasons. So, no worries from me. Sounds like a good proposal. Thanks Igor for finding an easy, compatible, forward moving solution. And as we've said before ... nested jars aren't good anyway :) Now if we can just get that Orbit project to comply :) [As a general rule, its been pretty well established that conditioning a Java jar file does not increase its size much ... but, can sometimes negatively increase the size of a jar file that contains no Java code ... ironically ... which is why some people say we should not pack200 doc bundles, feature.jars, etc. ... but, again, not sure of the practical difference?] Ah ... feels good to have the time to be wordy again :) (In reply to comment #36) > For jdt.core the eclipse.inf file was originally added to workaround a > problem with invalid signatures on nested packed JARs (193570). So, one > could say this might not even be needed anymore (this is worth a try). > However, there's currently a more severe problem coming along the road > regarding pack200 and nested JARs when running Java 7, see bug 361628 and > bug 378200. > FYI, both these do not affect tycho-built projects because Tycho never packs nested jars. ;-) (In reply to comment #39) > [As a general rule, its been pretty well established that conditioning a > Java jar file does not increase its size much ... but, can sometimes > negatively increase the size of a jar file that contains no Java code ... > ironically ... which is why some people say we should not pack200 doc > bundles, feature.jars, etc. ... but, again, not sure of the practical > difference?] FYI, Tycho only packs main bundle jars, it does not pack sources bundles or feature jars. Tycho signs everything, however. As far as I know, we have a "workable solution" to the signing and packing we need to do. If I've missed something as to why this was left re-opened, let me know. (I still think Tycho should support all forms of "eclipse.inf", or, at least better document what is supports, and what is doesn't ... but ... doesn't seem worth leaving _this_ bug open for that. |