Bugzilla – Full Text Bug Listing
| Summary: | many pack.gz files for indigo are not packed | ||
|---|---|---|---|
| Product: | Community | Reporter: | David Williams <david_williams> |
| Component: | Cross-Project | Assignee: | Jesse McConnell <jesse.mcconnell> |
| Status: | CLOSED FIXED | QA Contact: | |
| Severity: | major | ||
| Priority: | P3 | CC: | gorkem.ercan, igor, jesse.mcconnell, john.arthorne, kaloyan, mknauer, rsternberg, thomas, webmaster |
| Version: | unspecified | ||
| Target Milestone: | --- | ||
| Hardware: | PC | ||
| OS: | Linux | ||
| Whiteboard: | |||
|
Description
David Williams
ok, so the way to test/debug this is that jar files and pack.gz files should be wildly different sizes, right? is there a webmaster on here so I can get shell access to build.eclipse.org so I can resolve this issue? Another way to test is to check that the packed file is "gzip compressed data" by the file command. If I've understood it correctly, the culprits are reported as "Zip archive data". That should never be the case. I see a note in here that the 3.5 and 3.6 jarprocessor changed functionality so it wouldn't take the -f pack.properties file option anymore.. who manages that jar processor and could we get an idea how it should be used to factor in the pack.properties? this might be the root of a number of issues, we actually needed it for excludes as well (In reply to comment #4) > I see a note in here that the 3.5 and 3.6 jarprocessor changed functionality so > it wouldn't take the -f pack.properties file option anymore.. > > who manages that jar processor and could we get an idea how it should be used > to factor in the pack.properties? this might be the root of a number of > issues, we actually needed it for excludes as well I think John owns Jar Processor (or, knows would does) so added him to CC. I've never used the -f parameter, but use a "zip file" and put the pack.properties at the "root" of the zip file. Do you pass in jars one at a time? Or, as a big zip file? If the later, it'd be feasible to add the pack.properties ... not sure about the former case (never used that case). And, just in case, I assume you've seen http://wiki.eclipse.org/JarProcessor_Options and http://wiki.eclipse.org/JarProcessor_Options/Examples (I'd forgotten I'd written that later one, till I looked just now :) Also, wanted to emphasize, while I consider this a "bad" bug I do not consider it "blocking" since it was, apparently, this same way in the 'release', so I do not expect anyone to "jump through hoops" just for this one thing (especially if it risks breaking something else). We are at the point in Indigo SR1 that we should really just be addressing blocking bugs or regressions. Thanks for attending to the issue. ok, I will need access to build machine as running it locally does produce packed jars.. -rw-r--r-- 1 jesse jesse 50302 2011-09-07 13:01 org.eclipse.jetty.webapp.source_7.5.0.RC0.jar.pack.gz -rw-r--r-- 1 jesse jesse 109881 2011-09-07 13:01 org.eclipse.jetty.websocket_7.5.0.RC0.jar -rw-r--r-- 1 jesse jesse 31493 2011-09-07 13:01 org.eclipse.jetty.websocket_7.5.0.RC0.jar.pack.gz -rw-r--r-- 1 jesse jesse 54679 2011-09-07 13:01 org.eclipse.jetty.websocket.source_7.5.0.RC0.jar -rw-r--r-- 1 jesse jesse 36996 2011-09-07 13:01 org.eclipse.jetty.websocket.source_7.5.0.RC0.jar.pack.gz -rw-r--r-- 1 jesse jesse 33503 2011-09-07 13:01 org.eclipse.jetty.xml_7.5.0.RC0.jar -rw-r--r-- 1 jesse jesse 16605 2011-09-07 13:01 org.eclipse.jetty.xml_7.5.0.RC0.jar.pack.gz I'll do the same thing on the build machine and see how that works... but looking at the latest build we have on hudson, these look shrunk as well https://hudson.eclipse.org/hudson/view/Jetty-RT/job/jetty-rt-bundles/ws/jetty.bundles.repo/target/site/plugins/ The original jetty jars that were the same size were from 7.4.2...so it is possible that this issue has already been resolved and is fixed in the latest version of the eclipse-signing-maven-plugin coordinates to use are: org.eclipse.dash.maven:eclipse-signing-maven-plugin:1.0.2 David, your report about libra jars is strange. These files are contributed by the p2 repository available in the following folder of the build server: /home/data/httpd/download.eclipse.org/libra/releases/R-0.1.0-201106071417/plugins You can see that there is a significant difference between the .jar and .jar.pack.gz files. Is it possible that the problem is in the indigo aggregation build? BTW, I will switch the build to eclipse-signing-maven-plugin:1.0.2 - just in case. (In reply to comment #9) > > You can see that there is a significant difference between the .jar and > .jar.pack.gz files. > > Is it possible that the problem is in the indigo aggregation build? > Interesting. I do see that. And there certainly might be a problem in the aggregator. Or, it might be "masking" some other problem. Knowing that the "original" file system files appear correct, I investigated the metadata of libra a bit more (and, don't mean to "pick on" libra ... just a good example to follow). I focused on the current contribution to maintenance, which I believe is in /home/data/httpd/download.eclipse.org/libra/maintenance/M-0.1.1-201107280918 I first thought maybe the artifact "mappings" were wrong in artifacts.jar/xml, but they appear relatively correct. But the order is different than what I see in other artifacts.jar/xml files. I wonder if that matters? Libra's artifacts.jar/xml has: <mappings size="5"> <rule filter="(& (classifier=osgi.bundle))" output="${repoUrl}/plugins/${id}_${version}.jar"/> <rule filter="(& (classifier=binary))" output="${repoUrl}/binary/${id}_${version}"/> <rule filter="(& (classifier=org.eclipse.update.feature))" output="${repoUrl}/features/${id}_${version}.jar"/> <rule filter="(& (classifier=osgi.bundle) (format=packed))" output="${repoUrl}/plugins/${id}_${version}.jar.pack.gz"/> <rule filter="(& (classifier=org.eclipse.update.feature) (format=packed))" output="${repoUrl}/features/${id}_${version}.jar.pack.gz"/> </mappings> Where as the mappings from the released indigo repo have the "packed" rules earlier in the list: <mappings size='5'> <rule filter='(& (classifier=osgi.bundle) (format=packed))' output='${repoUrl}/plugins/${id}_${version}.jar.pack.gz'/> <rule filter='(& (classifier=osgi.bundle))' output='${repoUrl}/plugins/${id}_${version}.jar'/> <rule filter='(& (classifier=binary))' output='${repoUrl}/binary/${id}_${version}'/> <rule filter='(& (classifier=org.eclipse.update.feature) (format=packed))' output='${repoUrl}/features/${id}_${version}.jar.pack.gz'/> <rule filter='(& (classifier=org.eclipse.update.feature))' output='${repoUrl}/features/${id}_${version}.jar'/> </mappings> I am mostly guessing, if this is the issue, but seemed important, since this reference http://wiki.eclipse.org/Equinox_p2_Getting_Started_for_Releng#Artifacts.xml_mapping_rule_change says order is significant and "...this is a filter where the first entry to match wins (rules have to be specified from the most specific to the least)". This does not seem to completely explain what we are seeing, but I thought it might be related, so thought I'd post what I know (and am hoping a p2/aggregator expert might know more definitively ... hint hint :). I think you're right on target. The first rule with (classifier=osgi.bundle) will always win over the rule with (&(classifier=osgi.bundle)(format=packed)) so the latter will never get consulted. This means that when the repository is asked for the packed version, it will happily serve up an InputStream that reads from the jar file. That would explain what's going on here, that the bytes of the packed file are identical with the jar. Please note that this is not a problem with the aggregator ;-) and there's way the aggregator can compensate for it. The mappings has to be in the right order. which circles back to something the maven plugin ought to be checking for or validating I suspect I'll check that out in a bit and report back jesse Just to cross reference, there was a previous bug, bug 347591, that documented some issues and changes involving the mapping rules and publishPackFileAsSiblings attribute ... if it helps tie in history. Thanks. I found it, we are appending it to the mappings list and it looks like we need to be prepending it I'll get this fixed and deploy a snapshot of the plugin that can be tried out and validated...will report back when that is in place. Does this past muster?
<mappings size="5"><rule filter="(& (classifier=org.eclipse.update.feature) (format=packed))" output="${repoUrl}/features/${id}_${version}.jar.pack.gz"/>
<rule filter="(& (classifier=osgi.bundle))" output="${repoUrl}/plugins/${id}_${version}.jar"/>
<rule filter="(& (classifier=binary))" output="${repoUrl}/binary/${id}_${version}"/>
<rule filter="(& (classifier=org.eclipse.update.feature))" output="${repoUrl}/features/${id}_${version}.jar"/>
<rule filter="(& (classifier=osgi.bundle) (format=packed))" output="${repoUrl}/plugins/${id}_${version}.jar.pack.gz"/></mappings>
it is from the latest jetty bundles build
https://hudson.eclipse.org/hudson/view/Jetty-RT/job/jetty-rt-bundles/ws/jetty.bundles.repo/target/site/Jetty-bundles-repository-7.5.0.v20110901.zip
once someone can validate this for me I'll begin releasing the new plugin version
actually, only the feature made it first...that needs to be osgi.bundle format=packed up there as well I think? i hate xml like this, just for the record :) (In reply to comment #16) > > i hate xml like this, just for the record :) I'm not sure what you are doing in the "signing plugin" ... sounds like you are creating a repo, in addition to signing? Or, are you just copying and modifying a repo? In either case, seems like there should be a p2 API which would be a better long term solution.
first we call the pack process, then we sign and then the pack process again...on a bare repo this leaves us with a bad artifact.xml or artifact.jar
so we have a mojo that fixes the checksums and adds this format=packed metadata goop
The only p2 thing we have at the moment to use the jarprocessor which you would think would take care of all this junk :/
<mappings size="5">
<rule filter="(& (classifier=org.eclipse.update.feature) (format=packed))" output="${repoUrl}/features/${id}_${version}.jar.pack.gz"/>
<rule filter="(& (classifier=osgi.bundle) (format=packed))" output="${repoUrl}/plugins/${id}_${version}.jar.pack.gz"/>
<rule filter="(& (classifier=osgi.bundle))" output="${repoUrl}/plugins/${id}_${version}.jar"/>
<rule filter="(& (classifier=binary))" output="${repoUrl}/binary/${id}_${version}"/>
<rule filter="(& (classifier=org.eclipse.update.feature))" output="${repoUrl}/features/${id}_${version}.jar"/>
</mappings>
I have deployed a 1.0.3 release of the signing plugin that fixes this issue.
(In reply to comment #18) > we have a mojo that fixes the checksums Do you use p2.process.artifacts? That's what I use (after preconditioning and signing). It does (the final) pack and fixes the checksums (and, I assume fixes up the mappings ... I've never really checked "before and after" on the mappings ... but p2ish approach would be better. (And, I think "jar processor" should not get into p2 business :) Oh, and meant to say ... THANKS! no idea what what is, though it kinda looks like the ant target in athena I modeled the process the signing plugin uses maybe? if there is some nice tooling for making this easier I am good to use it, just get it into a repository I can pull it from and I'll give it a go sometime last time I asked around I just got pointed at athena so leveraged that cheers, jesse I switched the Libra build to use version 1.0.3 of the eclipse-signing-maven-plugin. If anyone wants to check the result, it is available at /home/data/httpd/download.eclipse.org/libra/maintenance/snapshot (In reply to comment #21) > no idea what what is, though it kinda looks like the ant target in athena I > modeled the process the signing plugin uses maybe? > Sort of. it is a p2 provided ant task. I opened bug 357130 as enhancement request to use p2 APIs (with a link to the help documents :) Thanks, again! |