Bugzilla – Full Text Bug Listing
| Summary: | [CBI] JDT core needs a way to produce ecj jar during tycho builds | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Eclipse Project] JDT | Reporter: | Krzysztof Daniel <krzysztof.daniel> | ||||||||||||||
| Component: | Core | Assignee: | Jay Arthanareeswaran <jarthana> | ||||||||||||||
| Status: | VERIFIED FIXED | QA Contact: | |||||||||||||||
| Severity: | major | ||||||||||||||||
| Priority: | P3 | CC: | akurtakov, aross, daniel_megert, david_williams, jarthana, krzysztof.daniel, manoj.palat, markus.kell.r, pwebster, robin, shankhba, srikanth_sankaran, stephan.herrmann, stephen.francisco, t0mus, thanh.ha | ||||||||||||||
| Version: | 4.2.1 | ||||||||||||||||
| Target Milestone: | 3.8.2+ | ||||||||||||||||
| Hardware: | All | ||||||||||||||||
| OS: | All | ||||||||||||||||
| Whiteboard: | To be verified for 4.3 M7 | ||||||||||||||||
| Bug Depends on: | |||||||||||||||||
| Bug Blocks: | 372792, 396082, 402693, 403350 | ||||||||||||||||
| Attachments: |
|
||||||||||||||||
|
Description
Krzysztof Daniel
Created attachment 218749 [details]
Patch proposition
I am -1 on using ant-run-plugin. CBI build will replace PDE/Build, so I see no reason to reuse customBuildCallbacks. Please list specific discrepancies between JDT bundles produced by existing PDE/Build and Tycho and lets see how to address them. I'm not calling customBuildCallbacks. The two plugins listed in the patch (compiler.tool and apt.core) update the batch compiler just after their compilation (add some classes). So the compilation plan looks somewhat like: (1) org.eclipse.jdt.core - compile - build batch compiler - build bundle (2) org.eclipse.jdt.apt.core & org.eclipse.jdt.compiler.tool - compile - update batch compiler - build bundle I have no idea how to achieve that result without maven-ant-run. You did not answer my question -- what are the differences between artifacts produced by PDE/Build and CBi? What files are missing, extra or different in CBI-produced artifacts? Changes are much smaller than I had expected: 1. Different name of the artifact: ech-global.jar vs org.eclipse.jdt.core-3.8.1-SNAPSHOT-batch-compiler.jar 2. Missing source of the ecj-global.jar (file ecjsrc-global.jar) In the batch-compiler, 3. missing META-INF/services/javax.tools.JavaCompiler - contributed by org.eclipse.jdt.compiler.apt 4. missing about.html 5. missing /org/eclipse/jdt/core/JDTCompilerAdapter* (2 classes) 6. missing org/eclipse/jdt/internal/antadapter/* (2 classes) It really concerns me why there is no more differences, if, according to build callbacks, more classes should be injected. (see http://git.eclipse.org/c/jdt/eclipse.jdt.core.git/tree/org.eclipse.jdt.compiler.apt/customBuildCallbacks.xml?id=306045e2789dbdf9b489ec4e706fbb1234edd021#n94 and http://git.eclipse.org/c/jdt/eclipse.jdt.core.git/tree/org.eclipse.jdt.compiler.tool/customBuildCallbacks.xml?id=306045e2789dbdf9b489ec4e706fbb1234edd021#n94) (In reply to comment #5) > Changes are much smaller than I had expected: > > 1. Different name of the artifact: ech-global.jar vs > org.eclipse.jdt.core-3.8.1-SNAPSHOT-batch-compiler.jar names of files under target/ directory are irrelevant and should be treated as build system internal. > 2. Missing source of the ecj-global.jar (file ecjsrc-global.jar) is this used or published anywhere? > > In the batch-compiler, > 3. missing META-INF/services/javax.tools.JavaCompiler - contributed by > org.eclipse.jdt.compiler.apt > 4. missing about.html > 5. missing /org/eclipse/jdt/core/JDTCompilerAdapter* (2 classes) > 6. missing org/eclipse/jdt/internal/antadapter/* (2 classes) > where do these classes come from? eclipse.jdt.core/org.eclipse.jdt.core/antadapter/* eclipse.jdt.core/org.eclipse.jdt.compiler.tool/src/META-INF/services no idea what is the origin of about.html. Tomek or Daniel, could you write a couple of words about the relation between org.eclipse.jdt.core, org.eclipse.jdt.apt.core & org.eclipse.jdt.compiler.tool ? The latter two seem to update the ecj jar located in the first bundle via customBuildCallbacks. How it is supposed to work? (In reply to comment #8) > Tomek or Daniel, > could you write a couple of words about the relation between A good opportunity to update the 'Plugins' section on http://www.eclipse.org/jdt/core/dev.php (In reply to comment #9) > (In reply to comment #8) > > Tomek or Daniel, > > could you write a couple of words about the relation between > > A good opportunity to update the 'Plugins' section on > http://www.eclipse.org/jdt/core/dev.php Can we get some detailed answer to Krzysztof's question? Unless there is coordinated effort from build system guys and project devs CBI would not move ahead on sufficient speed. Jay, please provide the answers. Thanks. (In reply to comment #7) > no idea what is the origin of about.html. It's from: eclipse.jdt.core/org.eclipse.jdt.core/scripts/about.html Here is what I understood from the export-ecj.xml: The following projects are involved in creating the ecj*.jar: org.eclipse.jdt.core org.eclipse.jdt.compiler.tool org.eclipse.jdt.compiler.apt All the above three projects are compiled and resources and compiled and we take select resources and classes from them to create the ecj.jar. So, if we don't want to use ant and customCallbacks.xml, I guess we need to first compile the aforementioned projects, keep a copy of the classes and resources we need for ecj, then build ecj out of them. Of course, I don't know anything about maven. So, there could be a better way. If those 3 projects are compiled already, can't we add an assembly step with the last module to create the other jar? PW Created attachment 225635 [details]
jdt.core.patch v2
Updated Krzysztof's patch to work with the latest master branch.
Created attachment 225636 [details]
jdt.core.patch R3_8_maintenance
Adding same patch that applies cleanly to R3_8_maintenance
I tested the patch and the classes from:
- org.eclipse.jdt.compiler.apt
- org.eclipse.jdt.compiler.tool
are now included in the org.eclipse.jdt.core-3.9.0-SNAPSHOT-batch-compiler.jar that is produced by org.eclipse.jdt.core.
jdt.core.patch v2 should be applied to the master branch
jdt.core.patch R3_8_maintenance should be applied to the 3.8 branch
Is it possible to run build JDT/Core alone and test the output? I tried and got this error: [FATAL] Non-resolvable parent POM: Could not find artifact org.eclipse:eclipse-parent:pom:1.0.0-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 18, column 11 If it's not possible to do it without a complete eclipse build, I will go ahead and release the changes. After releasing, I guess I can look at the CBI build somewhere and confirm the fix is working, can't I? (In reply to comment #18) > Is it possible to run build JDT/Core alone and test the output? I tried and > got this error: > > [FATAL] Non-resolvable parent POM: Could not find artifact > org.eclipse:eclipse-parent:pom:1.0.0-SNAPSHOT and 'parent.relativePath' > points at wrong local POM @ line 18, column 11 > > If it's not possible to do it without a complete eclipse build, I will go > ahead and release the changes. After releasing, I guess I can look at the > CBI build somewhere and confirm the fix is working, can't I? Unfortunately it's not possible to build the repos separately with CBI as it is today, you will need to run a full platform build following the instructions at [1]. Alternatively I believe David or Paul run a full build somewhere on build.eclipse.org occasionally. [1] http://wiki.eclipse.org/Platform-releng/Platform_Build
>
> Alternatively I believe David or Paul run a full build somewhere on
> build.eclipse.org occasionally.
>
Yes, if this is a pom only change, feel free to add to R3_8_maintenance and master ... master is not quite building yet, but we can do a 422 test build using cbi to confirm.
And, I should clarify, touching the pom will "change the version qualifier", even for Juno SR2 ... so, not sure if there are "special rules" about that, or if we need to discuss at Wednesday's planning/architecture call?
(In reply to comment #20) > > > > Alternatively I believe David or Paul run a full build somewhere on > > build.eclipse.org occasionally. > > > > Yes, if this is a pom only change, feel free to add to R3_8_maintenance and > master ... master is not quite building yet, but we can do a 422 test build > using cbi to confirm. > > And, I should clarify, touching the pom will "change the version qualifier", > even for Juno SR2 ... so, not sure if there are "special rules" about that, > or if we need to discuss at Wednesday's planning/architecture call? I think we are close to getting master/I-builds to run (as test builds) so if you wanted to release to master (Kepler) only, we should be able to test that soon. (In reply to comment #21) > I think we are close to getting master/I-builds to run (as test builds) so > if you wanted to release to master (Kepler) only, we should be able to test > that soon. I have released the patch both in master and R3_8_maintenance. Of course, I have increased the qualifier on SR2. The master branch didn't need that since it had already been bumped up earlier. David, it would be great if you can let me know whenever we have the builds (both master and R3_8x) so I can test the ecj.jar. > > David, it would be great if you can let me know whenever we have the builds > (both master and R3_8x) so I can test the ecj.jar. I still haven't gotten master to build, but tried a test 422 build (which should include your 38 version) ... but ... no joy. http://build.eclipse.org/eclipse/builds/422M/R4_2_maintenance/dirs/M20130116-0035/M20130116-0035/#JDTCORE I'll attach a snippet of log that mentions "ecj" and you can tell me if it makes sense. Created attachment 225697 [details]
snippet of log
I've not studied this, but a) seems to be "doing something", but b) I can't tell what it does with the results. (And, I have "searched the whole disk" where we do the build, and don't see it just left somewhere ... from what I can tell).
The log snippet probably contains lots more than you need, but if you search for "ecj.id" I think that's the main section?
(In reply to comment #24) > The log snippet probably contains lots more than you need, but if you search > for "ecj.id" I think that's the main section? I can trace it up to the point where we create the zip at: /opt/public/eclipse/builds/422M/R4_2_maintenance/gitCache/eclipse.platform.releng.aggregator/eclipse.jdt.core/org.eclipse.jdt.core/target/org.eclipse.jdt.core-3.8.3-SNAPSHOT-batch-compiler.jar Can't figure out what's going on after that. (In reply to comment #25) > (In reply to comment #24) > > The log snippet probably contains lots more than you need, but if you search > > for "ecj.id" I think that's the main section? > > I can trace it up to the point where we create the zip at: > > /opt/public/eclipse/builds/422M/R4_2_maintenance/gitCache/eclipse.platform. > releng.aggregator/eclipse.jdt.core/org.eclipse.jdt.core/target/org.eclipse. > jdt.core-3.8.3-SNAPSHOT-batch-compiler.jar > > Can't figure out what's going on after that. Not sure if this helps but here is what it's trying to do when building org.eclipse.jdt.compiler.tool: [DEBUG] (f) target = <target><path id="ecj.id"><fileset dir="/opt/public/eclipse/builds/422M/R4_2_maintenance/gitCache/eclipse.platform.releng.aggregator/eclipse.jdt.core/org.eclipse.jdt.compiler.tool/../org.eclipse.jdt.core/target/"><include name="org.eclipse.jdt.core-*-SNAPSHOT-batch-compiler.jar"/> </fileset> </path> <property name="ecj.file" refid="ecj.id"/> <echo message="UPDATE ${ecj.file}"/> <property name="unjarDestBin" value="/opt/public/eclipse/builds/422M/R4_2_maintenance/gitCache/eclipse.platform.releng.aggregator/eclipse.jdt.core/org.eclipse.jdt.compiler.tool/tempbin"/> <mkdir dir="${unjarDestBin}"/> <unjar dest="${unjarDestBin}" src="${ecj.file}"/> <zip update="true" destfile="${ecj.file}"><fileset dir="${unjarDestBin}"/> <fileset dir="/opt/public/eclipse/builds/422M/R4_2_maintenance/gitCache/eclipse.platform.releng.aggregator/eclipse.jdt.core/org.eclipse.jdt.compiler.tool/target/classes"><include name="**/*"/> </fileset> <fileset dir="/opt/public/eclipse/builds/422M/R4_2_maintenance/gitCache/eclipse.platform.releng.aggregator/eclipse.jdt.core/org.eclipse.jdt.compiler.tool"><include name="META-INF/services/**"/> </fileset> </zip> <delete dir="${unjarDestBin}" failonerror="false"/> </target> Shortly after this there is a really long log of it running unjar against org.eclipse.jdt.core-3.8.3-SNAPSHOT-batch-compiler.jar putting the contents into a temporary directory. Copying over the classes from org.eclipse.jdt.compiler.tool into the temporary directory and then rezipping them back into org.eclipse.jdt.core-3.8.3-SNAPSHOT-batch-compiler.jar. Finally cleaning up the temporary directory. This same pattern is then found again for org.eclipse.jdt.compiler.apt. One thing I noticed that could perhaps be an optimization. I see numerous of the following logs when it tries to zip everything back into the batch-compiler.jar [zip] META-INF/ECLIPSE_.RSA omitted as /opt/public/eclipse/builds/422M/R4_2_maintenance/gitCache/eclipse.platform.releng.aggregator/eclipse.jdt.core/org.eclipse.jdt.core/target/org.eclipse.jdt.core-3.8.3-SNAPSHOT-batch-compiler.jar:META-INF/ECLIPSE_.RSA is up to date. This leads me to believe unjar-ing of the batch-compiler.jar is likely unnecessary since when we try to zip it back into itself all the files are unchanged so the existing files won't be pushed back in. Perhaps we could simply just add the org.eclipse.jdt.compiler.tools and org.eclipse.jdt.compiler.apt classes directly into the jar? Unless it's modifying a file from the batch-compiler.jar and I'm missing that part. (In reply to comment #26) > Unless it's modifying a file from the batch-compiler.jar and I'm missing > that part. The only file that will get modifies is the messages.properties. But apart from that we also exclude quite a few files from both org.eclipse.jdt.compiler.tools and org.eclipse.jdt.compiler.apt. Please refer to the export-ecj.xml for the inclusion and exclusion patterns. > Perhaps we could simply just add the org.eclipse.jdt.compiler.tools and > org.eclipse.jdt.compiler.apt classes directly into the jar? We could start with that and improve upon, I guess. (In reply to comment #27) > (In reply to comment #26) > > Unless it's modifying a file from the batch-compiler.jar and I'm missing > > that part. > > The only file that will get modifies is the messages.properties. But apart > from that we also exclude quite a few files from both > org.eclipse.jdt.compiler.tools and org.eclipse.jdt.compiler.apt. This is what I see in the log for messages.properties: [zip] org/eclipse/jdt/internal/compiler/messages.properties omitted as /opt/public/eclipse/builds/422M/R4_2_maintenance/gitCache/eclipse.platform.releng.aggregator/eclipse.jdt.core/org.eclipse.jdt.core/target/org.eclipse.jdt.core-3.8.3-SNAPSHOT-batch-compiler.jar:org/eclipse/jdt/internal/compiler/messages.properties is up to date. So looks like the CBI jdt.compiler.tools and jdt.compiler.apt modules don't modify this file. (In reply to comment #28) > This is what I see in the log for messages.properties: > > [zip] org/eclipse/jdt/internal/compiler/messages.properties omitted as > /opt/public/eclipse/builds/422M/R4_2_maintenance/gitCache/eclipse.platform. > releng.aggregator/eclipse.jdt.core/org.eclipse.jdt.core/target/org.eclipse. > jdt.core-3.8.3-SNAPSHOT-batch-compiler.jar:org/eclipse/jdt/internal/compiler/ > messages.properties is up to date. > > > So looks like the CBI jdt.compiler.tools and jdt.compiler.apt modules don't > modify this file. Sorry I didn't mention this before, but there are multiple messages.properties in the JDT/Core projects. The one I was talking about was: /org.eclipse.jdt.core/batch/org/eclipse/jdt/internal/compiler/batch/messages.properties So, where do we stand here? As I commented in bug 402693, I've noticed there is another "release train" product that depends on our batch compiler being in our repo. I see that this jar is being produced by the build: /org.eclipse.jdt.core/target/org.eclipse.jdt.core-3.9.0-SNAPSHOT-batch-compiler.jar And is 1.8 M in size ... so, I assume its something. Peeking inside it, I see it has a bundle version of Bundle-Version: 3.9.0.v20130227-1643. and correct ID (for repo): org.eclipse.jdt.core.compiler.batch Just wondering "what's next" and what I can do to help? (In reply to comment #30) > Just wondering "what's next" and what I can do to help? The link from the downloads page produces a 404. Once we fix that and I can download the JAR, I can test it tell everyone where we stand. (In reply to comment #31) > (In reply to comment #30) > > Just wondering "what's next" and what I can do to help? > > The link from the downloads page produces a 404. Once we fix that and I can > download the JAR, I can test it tell everyone where we stand. Well ... I'm not sure what I could link to from DL page, yet, ... that's part of what we need to iterate on, here, to figure out. I didn't see anything with "ecj*.jar" in its name. If that ecj.jar is just a renamed version of *-batch-compiler.jar, then I'd know what to do. And, in addition to that link on DL page, we need to figure out how what and how to "get into our repo" ... previously that was always a "custom step" in our PDE build, since this is a case where we did not want this signed (reading some old bugs, I guess we could, but I think there's some consumers who'd prefer it not, since in use on servers, they perceived some performance slow down using a s signed version, but not sure that's true, or if there is hard numbers for that) But ... to get to the point :) .. you don't need to wait for me to link on DL page ... you can always "see" our build machine directory at http://build.eclipse.org/eclipse/ our Kepler builds take place in http://build.eclipse.org/eclipse/builds/4I/master/gitCache/eclipse.platform.releng.aggregator/ or more specifically for your case, the URL would be http://build.eclipse.org/eclipse/builds/4I/master/gitCache/eclipse.platform.releng.aggregator/eclipse.jdt.core/org.eclipse.jdt.core/target/ Naturally, the content there will be removed and recreated during a build ... but once a build is done, you'd have that period of time until the next build starts to "see" what the results were, or fetch one of the jars there. In one of our final "publish to downloads server" steps, is where we rename jars, etc. and I'll take care of that, as soon as I know what to copy/collect and what to rename. Such as is the final "ecj.jar" just a renamed copy of *-batch-compiler.jar? If so, I can get started on that, while we finish up getting it right, and getting into repo. But in the mean time, please look directly on build server to confirm your output. (In reply to comment #32) > (In reply to comment #31) > Well ... I'm not sure what I could link to from DL page, yet, ... that's > part of what we need to iterate on, here, to figure out. I didn't see > anything with "ecj*.jar" in its name. If that ecj.jar is just a renamed > version of *-batch-compiler.jar, then I'd know what to do. Looks like what you are talking about is the right one. Initially I wasn't sure, but looking at the pom files, we are using org.eclipse.jdt.core-*-SNAPSHOT-batch-compiler.jar as the name. I just downloaded the archived and tested and it mostly went well. Just two things I would like to be fixed: 1. Obviously the name of the jar file - something like what we have on the download page. 2. The version inside the messages properties need to updated with the correct bundleVersionQualifer. I've added a "gather" step to our production builds, which copies and renames the jars ....
cp org.eclipse.jdt.core-3.9.0-SNAPSHOT-batch-compiler.jar "$BUILD_DIR"/ecj-${BUILD_ID}.jar
cp org.eclipse.jdt.core-3.9.0-SNAPSHOT-sources.jar "$BUILD_DIR"/ecjsrc-${BUILD_ID}.jar
The source one seems larger than was in PDE builds ... assuming you'll check that too and advise if I've picked the wrong one.
http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/commit/?id=7c47c7d90bd94ecf698305bc4fbced81cd968011
I'll let you worry about the version number in the message file. I've not looked at it, but assume that was part of your "customcallback" in the PDE based build?
I am more concerned about how to get the IU/bundle o.e.jdt.core.compiler.batch into our repository, as appears that was never planned for or worked on. Not sure what that will take. Or if we will have to "break" someone for M6. (bug 402693)
BTW, now that org.eclipse.jdt.core.compiler.batch compiler is getting put in our repo (bug 402693) you (any committer) should be able to download it directly from disk file location (and not have to use p2 director, or anything), with a command similar to scp build:/home/data/httpd/download.eclipse.org/eclipse/updates/4.3-I-builds/I20130313-0800/plugins/*batch* . [The 'build' part might have to be committerId@build.eclipse.org depending on how you've set up your SSH config.] I downloaded it and peeked, it is the same size as ecj-${buildId}.jar, and it even runs! :) but, not sure if its "internal versions" all match up with what you expect. Just thought I'd point out this download trick, in case you wanted to check anything. (In reply to comment #35) > Just thought I'd point out this download trick, in case you > wanted to check anything. Yes, David, I was able to download the JAR file. I did a comparison of the same with an older version and here is what I found missing from the CBI version: /org/eclipse/jdt/cofe > JDTCompilerAdapter$1.class > JDTCompilerAdapter.class /org/eclipse/jdt/internal/antadapter: > AntAdapterMessages.class > messages.properties There are other things missing or extra as well, but those are not as important. Looks like the pom.xml has to be tweaked. I will take a look at it. Just cleaning up some targets. I suggest Kepler M7, instead of 3.8.2, then if anything must be back ported to 3.8.2+ that a separate bug (or clone) be opened on that target. Hope you don't mind ... just trying to help. (In reply to comment #37) > Just cleaning up some targets. I suggest Kepler M7, instead of 3.8.2, then > if anything must be back ported to 3.8.2+ that a separate bug (or clone) be > opened on that target. Hope you don't mind ... just trying to help. Thanks, David! I have been meaning to do that as well. Updated the whiteboard content appropriately. Currently it does not work for 3.8.2 and 4.2.2, hence a backport is required and in that case we can just use this bug. At least for JDT, 3.8.2+ means it is in 3.8.2+ and all newer branches. There is no need to create clones here. Created attachment 229199 [details]
Proposed fix
Attaching a possible fix for the inclusion of missing files mentioned earlier. The fix is to include missing files from the directory:
${project.build.directory}/antbin
The antbin is the source specific output folder defined in the org.eclipse.jdt.core/.classpath and I am not very sure if the folder above is the right one, esp. since I am not able to run it locally.
Thanh/David, would it be possible for you to have a look at this test? TIA!
Appears there are more problems with the ECJ jar being produced by the CBI build: some of the classes in the JAR are not signed. A couple of examples being: org/eclipse/jdt/internal/compiler/tool/EclipseCompiler.class org/eclipse/jdt/internal/compiler/lookup/AptSourceLocalVariableBinding.class I am afraid I don't know what might be going on here and I have to request Thanh or David to help me with this. *** Bug 404729 has been marked as a duplicate of this bug. *** (In reply to comment #40) > Created attachment 229199 [details] > Proposed fix > > Attaching a possible fix for the inclusion of missing files mentioned > earlier. The fix is to include missing files from the directory: > > ${project.build.directory}/antbin > > The antbin is the source specific output folder defined in the > org.eclipse.jdt.core/.classpath and I am not very sure if the folder above > is the right one, esp. since I am not able to run it locally. > > Thanh/David, would it be possible for you to have a look at this test? TIA! I'm not sure what you are trying to accomplish, here. Are you saying "antbin" something produced during the build? and you want to include that in your custom jar, except for those included? If so, I do not see "antbin" on the build machine (under org.eclipse.jdt.core). If you can't build locally, I can maybe help track it down, with a few more hints about what its source is ... or what other "built" classes it contains? (And, then I can do some "finds" on the build machine. (In reply to comment #41) > Appears there are more problems with the ECJ jar being produced by the CBI > build: some of the classes in the JAR are not signed. A couple of examples > being: > > org/eclipse/jdt/internal/compiler/tool/EclipseCompiler.class > org/eclipse/jdt/internal/compiler/lookup/AptSourceLocalVariableBinding.class > > I am afraid I don't know what might be going on here and I have to request > Thanh or David to help me with this. I know in the past, we we didn't sign this "inner jar", and that's probably one of the reasons why ... because it was modified after it was created. The eclipse.inf for org.eclipse.jdt.core already contains jarprocessor.exclude.children=true which should prevent "normal" signing, but have a feeling that in creating your custom bundle, you may have to also add eclipse.inf (in META-INF) that has jarprocessor.exclude=true (which should prevent both being signed and being packed). I see this in antadapter/META-INF/eclipse.inf ... is this what's used in your "custom" script? Is it just getting "lost" and not copied? $ cat ./antadapter/META-INF/eclipse.inf jarprocessor.exclude.sign=true jarprocessor.exclude.children=true (In reply to comment #43) > I'm not sure what you are trying to accomplish, here. Are you saying > "antbin" something produced during the build? and you want to include that > in your custom jar, except for those included? If so, I do not see "antbin" > on the build machine (under org.eclipse.jdt.core). If you can't build > locally, I can maybe help track it down, with a few more hints about what > its source is ... or what other "built" classes it contains? (And, then I > can do some "finds" on the build machine. Certain ant adapter files get compiled into the antbin folder - at least that's what used to happen. These must be included in the ECJ jar too. Here is the list of files that need to be included: antbin\org\eclipse\jdt\core\BuildJarIndex.class antbin\org\eclipse\jdt\core\CheckDebugAttributes.class antbin\org\eclipse\jdt\core\JDTCompilerAdapter$1.class antbin\org\eclipse\jdt\core\JDTCompilerAdapter.class antbin\org\eclipse\jdt\internal\antadapter\AntAdapterMessages antbin\org\eclipse\jdt\internal\antadapter\messages.properties (In reply to comment #44) > I see this in antadapter/META-INF/eclipse.inf ... is this what's used in > your "custom" script? Is it just getting "lost" and not copied? > $ cat ./antadapter/META-INF/eclipse.inf > jarprocessor.exclude.sign=true > jarprocessor.exclude.children=true Yes, and I see other files from the same outer folder getting copied into the JAR. This should fix that problem: diff --git a/org.eclipse.jdt.compiler.tool/pom.xml b/org.eclipse.jdt.compiler.tool/pom.xml index ce330f5..b4a814f 100644 --- a/org.eclipse.jdt.compiler.tool/pom.xml +++ b/org.eclipse.jdt.compiler.tool/pom.xml @@ -57,6 +57,7 @@ <fileset dir="${basedir}"> <include name="META-INF/services/**"/> + <include name="META-INF/eclipse.inf"/> </fileset> </zip> If you can confirm that nothing is wrong with the above patch, I will go ahead and release this into master. Unfortunately currently I have no way of verifying my fix. (In reply to comment #45) > Yes, and I see other files from the same outer folder getting copied into > the JAR. This should fix that problem: Release this code change in master. The good news is, you didn't break the build :) But, the jar is still signed. Did you include that eclipse.inf file in your "fileset copy"? If so, it must be being created/signed before that point. I'll try and poke around the logs some and see if I can understand better what's happening. The "test build" if you want to look, is here, on build machine only: http://build.eclipse.org/eclipse/builds/4I/siteDir/eclipse/downloads/drops4/I20130403-1335/ (In reply to comment #45) > (In reply to comment #43) > > I'm not sure what you are trying to accomplish, here. Are you saying > > "antbin" something produced during the build? and you want to include that > > in your custom jar, except for those included? If so, I do not see "antbin" > > on the build machine (under org.eclipse.jdt.core). If you can't build > > locally, I can maybe help track it down, with a few more hints about what > > its source is ... or what other "built" classes it contains? (And, then I > > can do some "finds" on the build machine. > > Certain ant adapter files get compiled into the antbin folder - at least > that's what used to happen. These must be included in the ECJ jar too. Here > is the list of files that need to be included: > > antbin\org\eclipse\jdt\core\BuildJarIndex.class > antbin\org\eclipse\jdt\core\CheckDebugAttributes.class > antbin\org\eclipse\jdt\core\JDTCompilerAdapter$1.class > antbin\org\eclipse\jdt\core\JDTCompilerAdapter.class > > antbin\org\eclipse\jdt\internal\antadapter\AntAdapterMessages > antbin\org\eclipse\jdt\internal\antadapter\messages.properties > So these are ones you want to include in ecj.jar, and you need to know where Tycho/Maven put them? Looking on build machine, I see them in following locations: $ ls -1 ./org.eclipse.jdt.core/target/jdtCompilerAdapter.jar-classes/org/eclipse/jdt/core/ BuildJarIndex.class CheckDebugAttributes.class JDTCompilerAdapter$1.class JDTCompilerAdapter.class $ ls -1 ./org.eclipse.jdt.core/target/jdtCompilerAdapter.jar-classes/org/eclipse/jdt/internal/antadapter/ AntAdapterMessages.class messages.properties Its a little hard for me to "reconcile" this info with your current "custom script" ... there's one point you exclude them? Or, are you making several custom jars? Made further changes to org.eclipse.jdt.core/pom.xml via commit: http://git.eclipse.org/c/jdt/eclipse.jdt.core.git/commit/?id=2364afdbec314daa8d0b5edfe834ccbfea646954 This should take care of the missing files problem. This also reverts the commit I made for eclipse.inf, which didn't add any value anway. Will wait for the next build results. (In reply to comment #49) > Made further changes to org.eclipse.jdt.core/pom.xml via commit: > > http://git.eclipse.org/c/jdt/eclipse.jdt.core.git/commit/ > ?id=2364afdbec314daa8d0b5edfe834ccbfea646954 > > This should take care of the missing files problem. This also reverts the > commit I made for eclipse.inf, which didn't add any value anway. Will wait > for the next build results. In a local test build, I'm seeing this error (which prevents the build from continuing). [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (update-ecj) on project org.eclipse.jdt.compiler.tool: An Ant BuildException has occured: /data/shared/eclipse/builds/4I/master/gitCache/eclipse.platform.releng.aggregator/eclipse.jdt.core/org.eclipse.jdt.compiler.tool/target/jdtCompilerAdapter.jar-classes does not exist. [ERROR] around Ant part ...<zip update="true" destfile="${ecj.file}">... @ 14:45 in /data/shared/eclipse/builds/4I/master/gitCache/eclipse.platform.releng.aggregator/eclipse.jdt.core/org.eclipse.jdt.compiler.tool/target/antrun/build-main.xml (In reply to comment #50) > (In reply to comment #49) > > Made further changes to org.eclipse.jdt.core/pom.xml via commit: > > > > http://git.eclipse.org/c/jdt/eclipse.jdt.core.git/commit/ > > ?id=2364afdbec314daa8d0b5edfe834ccbfea646954 > > > > This should take care of the missing files problem. This also reverts the > > commit I made for eclipse.inf, which didn't add any value anway. Will wait > > for the next build results. > > In a local test build, I'm seeing this error (which prevents the build from > continuing). > > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-antrun-plugin:1.7:run (update-ecj) on project > org.eclipse.jdt.compiler.tool: An Ant BuildException has occured: > /data/shared/eclipse/builds/4I/master/gitCache/eclipse.platform.releng. > aggregator/eclipse.jdt.core/org.eclipse.jdt.compiler.tool/target/ > jdtCompilerAdapter.jar-classes does not exist. > [ERROR] around Ant part ...<zip update="true" destfile="${ecj.file}">... @ > 14:45 in > /data/shared/eclipse/builds/4I/master/gitCache/eclipse.platform.releng. > aggregator/eclipse.jdt.core/org.eclipse.jdt.compiler.tool/target/antrun/ > build-main.xml It seems, from error message, that ${basedir} you are using is set to .../eclipse.platform.releng.aggregator/eclipse.jdt.core/org.eclipse.jdt.compiler.tool but the jdtCompilerAdapter is "one up" from that ... in .../eclipse.platform.releng.aggregator/eclipse.jdt.core/org.eclipse.jdt.core/target/jdtCompilerAdapter.jar-classes Perhaps ${basedir}/../target/jdtCompilerAdapter.jar-classes would work? If there's not a better solution? I took a chance changing
${basedir}/target/jdtCompilerAdapter.jar-classes
into
${basedir}/../org.eclipse.jdt.core/target/jdtCompilerAdapter.jar-classes
which looked plausible in analogy to what we have 21 lines up.
(In reply to comment #52) > I took a chance changing > ${basedir}/target/jdtCompilerAdapter.jar-classes > into > ${basedir}/../org.eclipse.jdt.core/target/jdtCompilerAdapter.jar-classes > which looked plausible in analogy to what we have 21 lines up. Thanks, Stephan! I an see the files in the ECJ jar now. (In reply to comment #53) > Thanks, Stephan! I an see the files in the ECJ jar now. Lucky strike :) Additionally, it seems the jar is no longer signed, which indicates success re comment 41 / comment 44, right? (In reply to comment #54) > Additionally, it seems the jar is no longer signed, which indicates > success re comment 41 / comment 44, right? Yes, and I can confirm after the tests run fine. (In reply to comment #54) > (In reply to comment #53) > > Thanks, Stephan! I an see the files in the ECJ jar now. > > Lucky strike :) > > Additionally, it seems the jar is no longer signed, which indicates > success re comment 41 / comment 44, right? Nothing is signed for N builds ... you'll have to check the I build today. (In reply to comment #56) > (In reply to comment #54) > > (In reply to comment #53) > > > Thanks, Stephan! I an see the files in the ECJ jar now. > > > > Lucky strike :) > > > > Additionally, it seems the jar is no longer signed, which indicates > > success re comment 41 / comment 44, right? > > Nothing is signed for N builds ... you'll have to check the I build today. Too bad, in I20130409-0900 ecj.jar is still signed while the two classes mentioned in comment 41 are still not covered ... Created attachment 229613 [details] buildOneModule.sh (In reply to comment #49) > > This should take care of the missing files problem. This also reverts the > commit I made for eclipse.inf, which didn't add any value anway. Will wait > for the next build results. If you want to test your changes locally you can build the eclipse.jdt.core repo only. I've attached a script that I use (which uses -Pbree-libs), but basically the steps are: bash$ git clone ssh://git.eclipse.org/gitroot/jdt/eclipse.jdt.core.git bash$ LREPO=$(pwd)/localRepo bash$ cd eclipse.jdt.core bash$ mvn -Pbuild-individual-bundles clean install -Dmaven.test.skip=true -Dmaven.repo.local=$LREPO It took about 13 minutes to build all of eclipse.jdt.core on my machine. PW (In reply to comment #58) > bash$ mvn -Pbuild-individual-bundles clean install -Dmaven.test.skip=true > -Dmaven.repo.local=$LREPO > > It took about 13 minutes to build all of eclipse.jdt.core on my machine. This does it, thanks Paul! I had a trouble with BREE, though, which failed with the the error being discussed here: http://dev.eclipse.org/mhonarc/lists/cross-project-issues-dev/msg08242.html But looks like I need to do a bit more to make the build produce signed jars. Coming back to the original problem, all the files from the following locations are not being signed: org.eclipse.jdt.core/antadapter/ org.eclipse.jdt.compiler.tool/src And we don't have anything like org.eclipse.jdt.compiler.tool/META-INF/eclipse.inf David, this could be a problem, right?
>
> But looks like I need to do a bit more to make the build produce signed jars.
>
> Coming back to the original problem, all the files from the following
> locations are not being signed:
>
> org.eclipse.jdt.core/antadapter/
> org.eclipse.jdt.compiler.tool/src
>
> And we don't have anything like
> org.eclipse.jdt.compiler.tool/META-INF/eclipse.inf
>
> David, this could be a problem, right?
Not necessarily. But, I've lost track exactly what we are talking about.
If its something produced by production build, that goes into delivered IDE, then, yes, must be signed.
If this is something just for ecj.jar or org.eclipse.jdt.core.compiler.batch then does not have to be signed. We did not in the past, and since these are used "server side", there appears to be less demand for it from clients (though, if they were signed, I don't think that'd be bad).
Here is the URL of the test I-build (build machine only) that Jay wanted me to do this afternoon ... http://build.eclipse.org/eclipse/builds/4I/siteDir/eclipse/downloads/drops4/I20130415-1341/ And indeed the ecj jar is not signed! Congratulations! The ecj source jar is signed (and, I'm assuming only partially?) so it might need some similar treatment, where ever its source META-INF is. Similarly, the batch compiler jar in the repository http://build.eclipse.org/eclipse/builds/4I/siteDir/updates/4.3-I-builds/I20130415-1341/plugins/org.eclipse.jdt.core.compiler.batch_3.9.0.v20130414-2202.jar is also not signed (just like it used to not be signed, by PDE build). Assuming that's all accurate and has correct content. Please double check, I did a quick "diff -r" on the unzipped contents and it showed no difference and they were same size, 1830341, so I think equal. Looking good! (In reply to comment #61) > The ecj source jar is signed (and, I'm assuming only partially?) so it might > need some similar treatment, where ever its source META-INF is. Yep, and all of the content of the JAR except the ones inside the META-INF are signed too. Interestingly, the sources of the ant adapter along with the eclipse.inf are inside the source JAR, but inside a sub folder 'jdtCompilerAdaptersrc'. Have no idea how/why this happens. I will investigate. If it's going to take more time, I plan to close this bug after some testing and raise a new one for the ecjsrc*.jar. Looks like the ecj's source jar is produced as a regular source bundle instead of a custom bundle as was the case in PDE builds. Since it's only the source, we can take it up as a lower priority bug and close this one. I will close this bug after applying the fixes on R3_8_maintenance. Released the fix in R3_8_maintenance as well. Raised bug 405870 to handle the missing version property, which has not been updated in CBI build automatically. Downloaded ecj.jar from http://download.eclipse.org/eclipse/downloads/drops4/I20130501-2000/winPlatform.php#EclipseSDK and verified java -jar ecj-I20130501-2000.jar A.java Works. Here A.java is a helloworld example. The ecj.jar (ecj-I20130501-2000.jar) is unsigned. The ecj-src.jar (ecjsrc-I20130501-2000.jar) is signed. Verified for 3.7 M7 using build I20130428-2000. Incorrect build was mentioned. Verified for 3.7 M7 using build I20130501-2000. *** Bug 403350 has been marked as a duplicate of this bug. *** |