Bugzilla – Full Text Bug Listing
Created attachment 251879 [details]
log files excerpts showing unpack errors: Java 6 vs. Java 7
Since we have a "future build" set up, but not in use yet, I changed it to use "Java 7 unpack200" to compare it's results with "Java 6 unpack200".
Although the messages are different, the "invalid pack.gz" files appears the same, or nearly so.
AND, there is no sign of the problems with nested packed jars I thought there would be, with Java 7.
BUT, some of these results (especially lack of nested packed jars problem) *might* be due to the fact that on our "future build" (at the stage it is at) bundles retrieved from previous runs are cached.
So ... I am going to manually "clean workspace" and let it run again ... and see how the results are different.
Using a fresh workspace, and still using Java 7, resulted in "no change".
I am surprised the "nested jars" did not cause an issue ... but, assume perhaps that's specific to a literal runtime or ... perhaps they are packed jars inside of a something else that Equinox normally doesn't open?
At any rate, same/similar errors, so believe these are simply "real" cases that are not handled by pack200. (It's not that uncommon ... out of the 5000 bundles we have, to have 4 bundles with some odd byte code that is "not handled".
[exec] Unable to unpack artifact osgi.bundle,org.eclipse.m2m.qvt.oml,3.5.0.v20150324-1623 in repository file:/home/hudson/genie.simrel/.hudson/jobs/simrel.mars.runaggregator.BUILD/workspace/aggregation/final/aggregate: Invalid content:org/eclipse/m2m/internal/qvt/oml/stdlib/model/StdlibFactory.class
[exec] Error during mirroring: org.eclipse.core.runtime.CoreException: Unable to unpack artifact osgi.bundle,org.eclipse.m2m.qvt.oml,3.5.0.v20150324-1623 in repository file:/home/hudson/genie.simrel/.hudson/jobs/simrel.mars.runaggregator.BUILD/workspace/aggregation/final/aggregate: Invalid content:org/eclipse/m2m/internal/qvt/oml/stdlib/model/StdlibFactory.class
[exec] Caused by: org.eclipse.osgi.signedcontent.InvalidContentException: The file "org/eclipse/m2m/internal/qvt/oml/stdlib/model/StdlibFactory.class" in the jar "/home/hudson/genie.simrel/.hudson/jobs/simrel.mars.runaggregator.BUILD/workspace/tmp/signatureFile4192107500851298291.jar" has been tampered!
[exec] Unable to unpack artifact osgi.bundle,org.eclipse.jubula.rc.javafx,3.0.0.201503231444 in repository file:/home/hudson/genie.simrel/.hudson/jobs/simrel.mars.runaggregator.BUILD/workspace/aggregation/final/aggregate: Unpacking fails because intermediate file is empty: /home/hudson/genie.simrel/.hudson/jobs/simrel.mars.runaggregator.BUILD/workspace/tmp/work954267804476060879/p2.optimizers.incoming8863718766434132343.jar
[exec] Mirroring artifacts from http://download.eclipse.org/technology/swtbot/releases/2.2.0
[exec] Error during mirroring: org.eclipse.core.runtime.CoreException: Unable to unpack artifact osgi.bundle,org.eclipse.jubula.rc.javafx,3.0.0.201503231444 in repository file:/home/hudson/genie.simrel/.hudson/jobs/simrel.mars.runaggregator.BUILD/workspace/aggregation/final/aggregate: Unpacking fails because intermediate file is empty: /home/hudson/genie.simrel/.hudson/jobs/simrel.mars.runaggregator.BUILD/workspace/tmp/work954267804476060879/p2.optimizers.incoming8863718766434132343.jar
We've just recently altered the pack200 usage for the affected o.e.j.rc.javafx bundle (due to bug 459639 comment 1) which actually has a BREE of 1.8. Shall we now - for M6 - disable the packing of that specific bundle and provide an updated version of our M6 repository? (In reply to Markus Tiede from comment #3) > We've just recently altered the pack200 usage for the affected > o.e.j.rc.javafx bundle (due to bug 459639 comment 1) which actually has a > BREE of 1.8. > > Shall we now - for M6 - disable the packing of that specific bundle and > provide an updated version of our M6 repository? Yes please. (For what it's worth, I tried unpacking with Java 8, and still an issue, so, it's not so much a matter of which version of Java is used, it is just that some bundles (byte codes) can not be correctly packed.) I updated our repository and restarted the aggregation build. See also bug 463510, where I propose we change the Eclipse Foundation's central "jar signing service" to use Java 7. (In reply to David Williams from comment #6) > See also bug 463510, where I propose we change the Eclipse Foundation's > central "jar signing service" to use Java 7. See also bug 463131 where Ed Willink tracked down some of the "source" of the problem. Assuming I'm interpreting correctly, using "higher" VMs to try and pack200 "lower" VM byte codes increases probability of corruption. The three main builders at eclipse, PDE, Tycho, and Buckminster all recognize the jarprocessor.exclude.pack=true directive in an eclipse.inf file (where the bundle will still be signed, but not compressed with pack200). I think when projects have problems specific bundles, that is one of the better courses of action (though, understandable to turn it off completely, if projects are "small" and not that many bundles). While "dealing with" these errors seems like a hassle, (well, is a hassle) it is a matter of us finding the problems early, rather than us producing a "poor quality" repository, and cause more hassles for users and adopters. [Thanks Ed] I have made the change to aggregation jobs to use Java 7. (And, remember, the aggregator does not actually do any packing or signing, it simply tests that the bundles can be unpacked with the given VM). So, marking as fixed. I meant to cross-reference to another change: in Orbit we will also move to Java 7 to do the builds and (try) to pack200 (bug 463616), and judging from the experience of moving from Java 5 to Java 6 (bug 462276) I anticipate several more bundles will have to be marked as "don't pack". (In reply to David Williams from comment #7) > Assuming I'm interpreting correctly, using "higher" VMs to try > and pack200 "lower" VM byte codes increases probability of corruption. Not exactly. My best theory is that it using a higher VM to pack a reference from a lower to higher VM. So what was ok in M5, while the platform shipped Java 5/6 classes to projects using Java 5/6, became a problem when in M6 the platform started shipping Java 7 classes. Consequently a 5/6 to 7 reference now needs packing, and this seems to be unsound with Buckminster 4.2. (In reply to David Williams from comment #9) > I anticipate several more bundles will have to be marked as "don't pack". Much easier to move to Java 7 classes. If the platform is shipping Java 7 classes, is there any point projects shipping 5/6? (In reply to Ed Willink from comment #10) > Much easier to move to Java 7 classes. If the platform is shipping Java 7 > classes, is there any point projects shipping 5/6? Certainly is in Orbit! Some of those are used by people who don't even use the Platform ... well ... not very much of it, anyway ... not to mention, we do not recompile from source, so what we have is what ever byte codes the third party provided. Just as well move to 8, as central batch signer. In bug 463510 I show that the version of pack200 and unpack200 are actually identical in Java 7 and Java 8 (at the moment). New Gerrit change created: https://git.eclipse.org/r/45782 New Gerrit change created: https://git.eclipse.org/r/45783 Gerrit change https://git.eclipse.org/r/45783 was merged to [master]. Commit: http://git.eclipse.org/c/simrel/org.eclipse.simrel.tests.git/commit/?id=b430533d171e1798ad30e69fe1713ac56cedb2ad fixed with commit above I have reverted the Sim. Release to use Java 6 to unpack jars, and to verify signatures. Not sure of a better choice as long as bug 463510 remains at Java 6. (In reply to David Williams from comment #17) > I have reverted the Sim. Release to use Java 6 to unpack jars, and to verify > signatures. If this is motivated solely by: The following errors occured when building Mars: org.eclipse.core.runtime.CoreException: Unable to unpack artifact osgi.bundle,org.eclipse.epsilon.flock.engine,1.3.0.201503231233 in repository file:/home/hudson/genie.simrel/.hudson/jobs/simrel.mars.runaggregator.BUILD/workspace/aggregation/final/aggregate: Invalid content:org/eclipse/epsilon/flock/model/domain/typemappings/PackageDeletion.class Caused by: org.eclipse.osgi.signedcontent.InvalidContentException: The file "org/eclipse/epsilon/flock/model/domain/typemappings/PackageDeletion.class" in the jar "/home/hudson/genie.simrel/.hudson/jobs/simrel.mars.runaggregator.BUILD/workspace/tmp/signatureFile3610007001751846141.jar" has been tampered! org.eclipse.core.runtime.CoreException: Unable to unpack artifact osgi.bundle,org.eclipse.epsilon.eol.engine,1.3.0.201503231233 in repository file:/home/hudson/genie.simrel/.hudson/jobs/simrel.mars.runaggregator.BUILD/workspace/aggregation/final/aggregate: Invalid content:org/eclipse/epsilon/eol/parse/Eol_EolParserRules.class Caused by: org.eclipse.osgi.signedcontent.InvalidContentException: The file "org/eclipse/epsilon/eol/parse/Eol_EolParserRules.class" in the jar "/home/hudson/genie.simrel/.hudson/jobs/simrel.mars.runaggregator.BUILD/workspace/tmp/signatureFile2227393089440708700.jar" has been tampered! Check the log file for more information: https://hudson.eclipse.org/simrel/job/simrel.mars.runaggregator.BUILD/72/console I am working around. QVTd temporarily, perhaps till Mars+1 M6, needs to use Epsilon that is a graduated non-SimRel project. 7 plugins are therefore redistributed and it is two of these that have new pack200 problems. Having discovered that it is only nested-and-packed jars that are a no-no, I can instead redistribute as private internal libraries. They will therefore be packed by QVTd and should be ok. (In reply to David Williams from comment #17) > I have reverted the Sim. Release to use Java 6 to unpack jars, and to verify > signatures. I have kicked off another build and the Epsilon plugins still fail, so I don't think that this reversion is a solution. I think that the problem is that the Epsilon plugins were re-signed during the Java 7 QVTd build. So the aggregator hates them whether using java 6 or 7 or 8. Given that we cannot revert the new certificate, I think we will just push the problem around until the signer stops re-signing with a different certifucate. (In reply to Ed Willink from comment #18) > (In reply to David Williams from comment #17) > > I have reverted the Sim. Release to use Java 6 to unpack jars, and to verify > > signatures. > > If this is motivated solely by: > > The following errors occured when building Mars: > No. I made the change slightly before seeing that error. The change was just to try and minimize issues that people have in doing aggregation builds. To at least have a known target to shot for. I suppose in a perfect world, we would "test" to make sure all commonly used VMs would unpack what ever VM a bundle was packed with. As far as I know, if a bundle can be unpacked by one VM, it could be unpacked by a higher one (with the exception of Java 7 and nested packed jars) but I think the reverse is less likely to be true -- that is, something packed with Java 8 is probably less likely to be able to be unpacked with Java 6 (or 7). I re-enabled "Java 8 unpack200" on aggregator jobs. If teams can not react on Wednesday, I'll consider alternative. Probably either try using Java 6 unpack200 again (but, not perfectly confident that will "solve" all problems, but it might. Also tempted to just not provide pack.gz files for RC1. It doesn't make sense to "hit the check mark" and "miss the user". (In reply to David Williams from comment #21) > I re-enabled "Java 8 unpack200" on aggregator jobs. > > If teams can not react on Wednesday, I'll consider alternative. Probably > either try using Java 6 unpack200 again (but, not perfectly confident that > will "solve" all problems, but it might. Also tempted to just not provide > pack.gz files for RC1. It doesn't make sense to "hit the check mark" and > "miss the user". David, I'm not sanguine that ECF can react and solve the jive xpp.jar problem on Wednesday. We've fussed with it before with little success, and I don't have hours budgeted/available for something unexpected like this. New Gerrit change created: https://git.eclipse.org/r/47316 Gerrit change https://git.eclipse.org/r/47316 was merged to [master]. Commit: http://git.eclipse.org/c/ecf/org.eclipse.ecf.git/commit/?id=f2276f5750e8dd0128cba55616a3b0644f2ee7bd (In reply to Eclipse Genie from comment #24) David I made that change you described. Can you verify that this would do the trick? [1] http://git.eclipse.org/c/ecf/org.eclipse.ecf.git/commit/?id=f2276f5750e8dd0128cba55616a3b0644f2ee7bd New Gerrit change created: https://git.eclipse.org/r/47374 Gerrit change https://git.eclipse.org/r/47374 was merged to [master]. Commit: http://git.eclipse.org/c/simrel/org.eclipse.simrel.build.git/commit/?id=33ed9107d315dff3a6085b66deaae6d5af0b7fa1 Thanks all for the efforts made to cleanup the 'pack.gz' files. There was one person/project that said they could not get to it today (if they hadn't by now) ... and they might be able to from wireless in their car while on vacation (while someone else was driving! :) ... but, hardly seems worth that. I will first try 'copy mode' in aggregator ... never used it before, not sure it works ... and if it doesn't ... will try one or two other things to get a green build for EPP packages. Thanks again ... your efforts now will pay off later. (In reply to David Williams from comment #28) > Thanks again ... your efforts now will pay off later. So our change to tackle the embedded jar failed. Do you think if bundleshape=dir would change something? (In reply to Wim Jongman from comment #29) > (In reply to David Williams from comment #28) > > Thanks again ... your efforts now will pay off later. > > So our change to tackle the embedded jar failed. Do you think if > bundleshape=dir would change something? No, I don't think that would help. What build system do you use? Where do you get that org.jivesoftware.smack.source bundle from? Your own repo I assume? Do you "rebuild it"? Or does it come pre-built? I ask all these because I see it has a "packed nested jar". (Is that the same problem you were trying to solve?) Depending on answers to these questions I can probably help with more specific advice. /home/hudson/genie.simrel/.hudson/jobs/simrel.mars.runaggregator.CLEAN_BUILD/workspace/aggregation/final/plugins/org.jivesoftware.smack.source_3.3.0.v20150506-1752.jar.pack.gz contains nested packed jars testing: jars/xpp.jar.pack.gz OK New Gerrit change created: https://git.eclipse.org/r/47451 Gerrit change https://git.eclipse.org/r/47451 was merged to [master]. Commit: http://git.eclipse.org/c/ecf/org.eclipse.ecf.git/commit/?id=d1daeccc92f7366acd82e8e9b9f9752f989bada4 (In reply to Eclipse Genie from comment #32) > Gerrit change https://git.eclipse.org/r/47451 was merged to [master]. > Commit: > http://git.eclipse.org/c/ecf/org.eclipse.ecf.git/commit/ > ?id=d1daeccc92f7366acd82e8e9b9f9752f989bada4 Your original jarprocessor.exclude.children=true Should have worked (I think you probably use Buckminster, to build?) And, it might be simply that "the latest" version did not get included in the aggregated repository? (In reply to David Williams from comment #33) > (In reply to Eclipse Genie from comment #32) > > Gerrit change https://git.eclipse.org/r/47451 was merged to [master]. > > Commit: > > http://git.eclipse.org/c/ecf/org.eclipse.ecf.git/commit/ > > ?id=d1daeccc92f7366acd82e8e9b9f9752f989bada4 > > Your original > jarprocessor.exclude.children=true > Should have worked (I think you probably use Buckminster, to build?) > > And, it might be simply that "the latest" version did not get included in > the aggregated repository? And, hmmmm ... not sure ... just guessing ... the old jarprocessor used by eclipse.org might not have had some fixes that are in the new jarprocessor. In other words, this my "interact" with bug 463510. I am not suggesting you "move to Java 8" or anything, at this point in time, (bug 463510 is sort of complicated to take advantage of, at the moment, for Buckminster users) ... but more to say your recent fix to "not pack" might help after all. (In reply to David Williams from comment #34) > (In reply to David Williams from comment #33) > > (In reply to Eclipse Genie from comment #32) > > > Gerrit change https://git.eclipse.org/r/47451 was merged to [master]. > > > Commit: > > > http://git.eclipse.org/c/ecf/org.eclipse.ecf.git/commit/ > > > ?id=d1daeccc92f7366acd82e8e9b9f9752f989bada4 > > > > Your original > > jarprocessor.exclude.children=true > > Should have worked (I think you probably use Buckminster, to build?) > > > > And, it might be simply that "the latest" version did not get included in > > the aggregated repository? > > And, hmmmm ... not sure ... just guessing ... the old jarprocessor used by > eclipse.org might not have had some fixes that are in the new jarprocessor. > In other words, this my "interact" with bug 463510. I am not suggesting you > "move to Java 8" or anything, at this point in time, In a different resources world we wouldn't mind moving to Java 8 immediately, but we can't do it prior to Mars for two reasons: 1) we would need to update Buckminster and other parts of our builder. This is likely to be a significant releng effort by itself. 2) we have some significant and potentially destabilizing releng changes to make because of our support for async remote services on both J8 and <J8. I can't commit to make such large and risky changes to our releng in the immediate term (prior to Mars, which for us is basically next week). New Gerrit change created: https://git.eclipse.org/r/47470 Gerrit change https://git.eclipse.org/r/47470 was merged to [master]. Commit: http://git.eclipse.org/c/simrel/org.eclipse.simrel.build.git/commit/?id=074e6d0d84586a95d29a1d54446b446af256439c For ECF, I repacked to xpp.jar and the build is now green. I hope you are still using the unpack200 with Java 8? (In reply to Wim Jongman from comment #38) > For ECF, I repacked to xpp.jar and the build is now green. I hope you are > still using the unpack200 with Java 8? Sorry, I'd turned it 'off' to get M7 out the door, but will turn it back on now. New Gerrit change created: https://git.eclipse.org/r/47603 Gerrit change https://git.eclipse.org/r/47603 was merged to [master]. Commit: http://git.eclipse.org/c/ecf/org.eclipse.ecf.git/commit/?id=8d06802208c2f90d291b3f49af7d51b111187861 New Gerrit change created: https://git.eclipse.org/r/47620 Gerrit change https://git.eclipse.org/r/47620 was merged to [master]. Commit: http://git.eclipse.org/c/simrel/org.eclipse.simrel.build.git/commit/?id=1dca04dd72f8ce5fe36df3acd0debd645f721dac I think this was done when Mars first shipped. If I have lost track of something that needs to be done in aggregation build, please re-open. |
Every time someone changes Java versions to build with, there is often a different set of Java bundles that no longer "pack" or "unpack" correctly. The b3 aggregator has been using Java 6 to do "unpack200" for a year or two, since that used to be "lowest common denominator". But, I believe it no longer is, since a great deal of the Eclipse SDK, even, requires Java 7. And, above the platform, suspect everyone builds with Java 7? Therefore, after M6 is done, I plan to change b3 aggregator to use Java 7 to do the unpack200. "Well why haven't you done that already", I can hear you ask. The reason is that then the few remaining cases where some of you have "nested packed jars" will no longer "unpack" under Java 7, and then that will break the b3 aggregation build. I believe most of these are in "window builder" (latest "repo report" below) and will require they re-build their bundles without packing the nested jars. The bundle as a whole can still be packed, just not any jars inside of it. This will not solve all the pack/unpack problems that some projects have been experiencing ... it might not even solve any! But, will at least remove one variable from the equation of why there has been so many issues here at the end of M6. Thanks, = = = = = /home/data/httpd/download.eclipse.org/releases/staging/plugins/org.eclipse.wb.swing.FormLayout.lib_1.8.0.r45x201412090035.jar.pack.gz contains nested packed jars testing: jgoodies-common-1.8.0-sources.jar.pack.gz OK testing: jgoodies-common-1.8.0.jar.pack.gz OK testing: jgoodies-forms-1.8.0-sources.jar.pack.gz OK testing: jgoodies-forms-1.8.0.jar.pack.gz OK /home/data/httpd/download.eclipse.org/releases/staging/plugins/org.eclipse.wb.rcp.swing2swt_1.8.0.r45x201412090039.jar.pack.gz contains nested packed jars testing: swing2swt.jar.pack.gz OK /home/data/httpd/download.eclipse.org/releases/staging/plugins/org.eclipse.wb.core.lib_1.8.0.r45x201412082350.jar.pack.gz contains nested packed jars testing: lib/MVEL-fork.jar.pack.gz OK testing: lib/asm-all-3.3.1.jar.pack.gz OK testing: lib/commons-beanutils-1.8.0.jar.pack.gz OK testing: lib/commons-collections-3.2.1.jar.pack.gz OK testing: lib/commons-digester-2.0.jar.pack.gz OK testing: lib/commons-io-2.0.1.jar.pack.gz OK testing: lib/commons-lang-2.5.jar.pack.gz OK testing: lib/commons-logging-1.1.1.jar.pack.gz OK testing: lib/commons-primitives-1.0.jar.pack.gz OK testing: lib/guava-13.0.1.jar.pack.gz OK testing: lib/nebula-cdatetime.jar.pack.gz OK testing: lib/nebula-cwt.jar.pack.gz OK /home/data/httpd/download.eclipse.org/releases/staging/plugins/org.eclipse.emf.doc_2.7.0.v20150123-0357.jar.pack.gz contains nested packed jars testing: tutorials/jet2/jetc-task.jar.pack.gz OK /home/data/httpd/download.eclipse.org/releases/staging/plugins/org.eclipse.wb.swing.MigLayout.lib_1.8.0.r45x201412090035.jar.pack.gz contains nested packed jars testing: ideutil.jar.pack.gz OK testing: miglayout15-swing.jar.pack.gz OK /home/data/httpd/download.eclipse.org/releases/staging/plugins/org.jivesoftware.smack.source_3.3.0.v20141221-2352.jar.pack.gz contains nested packed jars testing: jars/xpp.jar.pack.gz OK /home/data/httpd/download.eclipse.org/releases/staging/plugins/org.eclipse.m2m.atl.core.ant_3.6.0.v201412161312.jar.pack.gz contains nested packed jars testing: lib/atlAntTasks.jar.pack.gz OK /home/data/httpd/download.eclipse.org/releases/staging/plugins/org.eclipse.wb.runtime.lib_1.8.0.r45x201412082350.jar.pack.gz contains nested packed jars testing: lib/cglib-nodep-2.2.jar.pack.gz OK