Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.
Bug 462783 - New certificate conflict for org.eclipse.jdt.internal.compiler.lookup.AptSourceLocalVariableBinding
Summary: New certificate conflict for org.eclipse.jdt.internal.compiler.lookup.AptSour...
Status: VERIFIED FIXED
Alias: None
Product: JDT
Classification: Eclipse Project
Component: Core (show other bugs)
Version: 4.5   Edit
Hardware: All All
: P3 critical (vote)
Target Milestone: 4.5 M7   Edit
Assignee: David Williams CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-03-22 10:00 EDT by Andrey Loskutov CLA
Modified: 2015-04-28 08:51 EDT (History)
7 users (show)

See Also:


Attachments
detailed signing data from 2 bundles (222.43 KB, text/plain)
2015-03-22 13:20 EDT, David Williams CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Andrey Loskutov CLA 2015-03-22 10:00:43 EDT
Similar to bug 462431, bug 461634, caused most likely by bug 450783 (updated certificates).

The compile of org.eclipse.core.databinding.observable fails because of:

[ERROR] Failed to execute goal org.eclipse.tycho:tycho-compiler-plugin:0.23.0-SNAPSHOT:compile (default-compile) on project org.eclipse.core.databinding.observable: Fatal error compiling: class "org.eclipse.jdt.internal.compiler.lookup.AptSourceLocalVariableBinding"'s signer information does not match signer information of other classes in the same package

See https://hudson.eclipse.org/platform/job/eclipse.platform.ui-Gerrit/5095/consoleFull
Comment 1 Stephan Herrmann CLA 2015-03-22 10:47:05 EDT
Hi Andrey,

Looks like mismatching versions of bundle org.eclipse.jdt.core and its fragment org.eclipse.jdt.compiler.apt are in use somehow.

In the linked console log I can see references to several versions of o.e.jdt.core:
- 3.10.0.v20140604-1726 
- 3.11.0.v20150317-0048
- 3.11.0.N20150321-1500
(need to figure out, which ones are used as your compiler, and which ones are regular dependencies).

o.e.jdt.compiler.apt is mentioned in only this versions:
- 1.1.0.v20140509-1235

Do you have an idea what should be changed in JDT/Core? o.e.j.compiler.apt has been touched twice (2015-03-12 and 2015-03-16) after the cert update (2015-03-04). I've no idea, why we don't see a more recent version of that fragment in the build logs.

Maybe we need to ask the tycho guys, which exact bits they're pulling in from their 0.23.0-SNAPSHOT.
Comment 2 David Williams CLA 2015-03-22 11:18:12 EDT
I'm guessing this is related to bug 462774. 

I'll revert that change, so we can just see if it works around the immediate ... but, then we still need to solve the problem.
Comment 3 Stephan Herrmann CLA 2015-03-22 11:24:15 EDT
(In reply to David Williams from comment #2)
> I'm guessing this is related to bug 462774. 
> 
> I'll revert that change, so we can just see if it works around the immediate
> ... but, then we still need to solve the problem.

BTW: We will ask for a compiler update after fixing bug 371614 (for reducing bogus compile time warnings), which should happen within a week or so.
Comment 4 David Williams CLA 2015-03-22 11:49:06 EDT
(In reply to David Williams from comment #2)
> I'm guessing this is related to bug 462774. 
> 
> I'll revert that change, so we can just see if it works around the immediate
> ... but, then we still need to solve the problem.

This seems to have allowed my local compiles to proceed. 

But, I am not sure how to solve the problem and will need a better understanding of how "apt" is "mixed in" with the compiler. (or, is the compiler mixed in with apt?) 

As you think about it, remember that the certificate was changed on March 4, 2015. According to that, of our "current code" (in M6) only jdt annotation 1.1.100 was signed by "the old" certificate: 

jdt.core: 3.11.0.v20150317-0048
jdt.annotation: 2.0.100.v20150311-1658
jdt.annotation: 1.1.100.v20140704-0625

I would be interesting the problem was solved merely by touching jdt.annotation 1.1.100 (to increase it's qualifier, thus making sure what was delivered was signed with new certificate).
Comment 5 Stephan Herrmann CLA 2015-03-22 11:54:09 EDT
(In reply to David Williams from comment #4)
> (In reply to David Williams from comment #2)
> > I'm guessing this is related to bug 462774. 
> > 
> > I'll revert that change, so we can just see if it works around the immediate
> > ... but, then we still need to solve the problem.
> 
> This seems to have allowed my local compiles to proceed. 
> 
> But, I am not sure how to solve the problem and will need a better
> understanding of how "apt" is "mixed in" with the compiler. (or, is the
> compiler mixed in with apt?)

o.e.j.compiler.apt is a fragment of o.e.j.core.
How these are pulled in during the build has to be answered by a tycho expert, I guess.
 
> As you think about it, remember that the certificate was changed on March 4,
> 2015. According to that, of our "current code" (in M6) only jdt annotation
> 1.1.100 was signed by "the old" certificate: 
> 
> jdt.core: 3.11.0.v20150317-0048
> jdt.annotation: 2.0.100.v20150311-1658
> jdt.annotation: 1.1.100.v20140704-0625
> 
> I would be interesting the problem was solved merely by touching
> jdt.annotation 1.1.100 (to increase it's qualifier, thus making sure what
> was delivered was signed with new certificate).

I don't see the annotation bundles anywhere in the picture. They don't participate in any package split to begin with.
Comment 6 David Williams CLA 2015-03-22 12:02:09 EDT
(In reply to Stephan Herrmann from comment #5)
> (In reply to David Williams from comment #4)
> > (In reply to David Williams from comment #2)
> > > I'm guessing this is related to bug 462774. 
> > > 
> > > I'll revert that change, so we can just see if it works around the immediate
> > > ... but, then we still need to solve the problem.
> > 
> > This seems to have allowed my local compiles to proceed. 
> > 
> > But, I am not sure how to solve the problem and will need a better
> > understanding of how "apt" is "mixed in" with the compiler. (or, is the
> > compiler mixed in with apt?)
> 
> o.e.j.compiler.apt is a fragment of o.e.j.core.
> How these are pulled in during the build has to be answered by a tycho
> expert, I guess.
>  
> > As you think about it, remember that the certificate was changed on March 4,
> > 2015. According to that, of our "current code" (in M6) only jdt annotation
> > 1.1.100 was signed by "the old" certificate: 
> > 
> > jdt.core: 3.11.0.v20150317-0048
> > jdt.annotation: 2.0.100.v20150311-1658
> > jdt.annotation: 1.1.100.v20140704-0625
> > 
> > I would be interesting the problem was solved merely by touching
> > jdt.annotation 1.1.100 (to increase it's qualifier, thus making sure what
> > was delivered was signed with new certificate).
> 
> I don't see the annotation bundles anywhere in the picture. They don't
> participate in any package split to begin with.

Oh, sorry, I was mentally confusing "apt" mentioned in comment 0, I guess. 

= = = =
Related topic:

See also Bug 461101 - Update JDT to Mars M5 or later 

That is, Tycho was thinking of moving up to M6 jdt.core ... but, not sure I'd recommend that, until this problem was understood?
Comment 7 David Williams CLA 2015-03-22 13:20:00 EDT
Created attachment 251810 [details]
detailed signing data from 2 bundles

I'm not such _which_ bundles matter, when it comes to "apt", but 

I looked at two of them, in detail, (using javasigner -verify -verbose  -certs)

org.eclipse.jdt.compiler.apt_1.1.100.v20150316-1618.jar
That's the fragment in question, I believe? 
and it was signed with latest cert, 

      [entry was signed on 3/16/15 8:50 PM]
      X.509, CN="Eclipse Foundation, Inc.", OU=IT, O="Eclipse Foundation, Inc.", L=Ottawa, ST=Ontario, C=CA 
      [certificate is valid from 3/3/15 7:00 PM to 3/8/18 7:00 AM]


But, also looked at 
org.eclipse.jdt.apt.core_3.3.700.v20150226-0921.jar

And it was signed with "old" certificate: 

      [entry was signed on 3/3/15 8:54 AM]
      X.509, CN="Eclipse.org Foundation, Inc.", OU=IT, O="Eclipse.org Foundation, Inc.", L=Ottawa, ST=Ontario, C=CA
      [certificate is valid from 2/28/12 7:00 PM to 3/5/15 7:00 AM]


So, if it is literally only the fragment that matters ... which is what I suspect ... then maybe Tycho "picking up a new version" is the solution? 

But, if any of these other "apt" bundles get "pulled" in by the fragment, may more needs to be "touched"? But, As far as I can see ... with my short-sighted understanding of jdt ... only org.eclipse.jdt.internal.compiler.tool get's "pulled in" ... but that fragment, and 
org.eclipse.jdt.compiler.tool is also signed by latest cert.
Comment 8 David Williams CLA 2015-03-22 13:32:51 EDT
Adding members of Tycho team to CC, which, as stated already, may know more about "apt" fragment? 

Also, Tycho team, does bug 371614 effect when you'd be able/willing to "pick up" a new version? If it is not a show stopper (for initial "pick up", at least) I'd say new (M6) version in 0.23.0-SNAPSHOT as soon as possible would be a good thing? To prevent you from having to look it up, the URL for the p2 repo would be 
http://download.eclipse.org/eclipse/updates/4.5milestones/S-4.5M6-201503200800/
Or, if you use EJC.jar directly, that'd be under 
http://download.eclipse.org/eclipse/downloads/drops4/S-4.5M6-201503200800/


(As an aside, if Tycho "ran under OSGi/Equinox" this wouldn't be a problem ... does that deserve a "feature request"? :)
Comment 9 David Williams CLA 2015-03-22 23:49:11 EDT
I am also intrigued why "things work" for the first two bundles of this build, but then fails on third: 

eclipse.platform.ui ................................ SUCCESS [ 49.853 s]
org.eclipse.core.commands .......................... SUCCESS [ 42.956 s]
org.eclipse.core.databinding.observable ............ FAILURE [  1.179 s]

I would think if simply a "mismatch" of compiler and apt fragment, that it'd always occur? 

Anyone know?
Comment 10 Ed Willink CLA 2015-03-23 02:16:00 EDT
(In reply to David Williams from comment #9)
> I would think if simply a "mismatch" of compiler and apt fragment, that it'd
> always occur? 

Surely the first use of any bundle succeeds and defines the 'correct' certificate?

It is the second use with the 'wrong' certificate fails.

This problem now seems so prevalent and so insidious, since it does not occur until a commit touches a bundle, that perhaps the platform should take the easy solution.

Touch ALL bundles. and issue an M6a.
Comment 11 Andrey Loskutov CLA 2015-03-23 03:25:48 EDT
Thanks to the quick help from David on bug 462774 comment 3 the build works now. 

Should we close this bug now, or do we want first: 
1) understand why it failed (comment 8)?
2) understand why not by the very first project in the build sequence, but by the third one (comment 9)?
3) should we respin M6 (comment 10)?
Comment 12 Jan Sievers CLA 2015-03-23 04:24:19 EDT
(In reply to Stephan Herrmann from comment #5)
> o.e.j.compiler.apt is a fragment of o.e.j.core.
> How these are pulled in during the build has to be answered by a tycho
> expert, I guess.

we use both o.e.j.compiler.apt and o.e.j.core as plain jars in a maven classloader environment.
A Tycho version "ships with" a certain version of both of these jars, so if you override one of these using pom.xml configuration, make sure to override the other one as well to match the first one.
Comment 13 David Williams CLA 2015-03-23 09:12:49 EDT
(In reply to Jan Sievers from comment #12)
> (In reply to Stephan Herrmann from comment #5)
> > o.e.j.compiler.apt is a fragment of o.e.j.core.
> > How these are pulled in during the build has to be answered by a tycho
> > expert, I guess.
> 
> we use both o.e.j.compiler.apt and o.e.j.core as plain jars in a maven
> classloader environment.
> A Tycho version "ships with" a certain version of both of these jars, so if
> you override one of these using pom.xml configuration, make sure to override
> the other one as well to match the first one.

Thanks Jan! 

I know how to change our parent pom to implement this, but we will also have to change our 'eclipse-staging' job, to deploy jdt.compiler.apt, along with jdt.core. Does any one know if that's possible? (to deploy both in one job, or do we need a another job). 
See 
https://hudson.eclipse.org/platform/job/deployJDTCompiler/configure
At the heart of it all, we use 
deploy:deploy-file
with this, current, input: 
version=${jdtversion}
url=${mavenurl}
groupId=org.eclipse.jdt
file=${jdtlocation}/org.eclipse.jdt.core_${jdtversion}.jar
packaging=jar
repositoryId=repo.eclipse.org
artifactId=org.eclipse.jdt.core

From looking at the job, and the doc for deploy:deploy-file I believe I should just add an additional "maven 3" build step, so it'd run "one job", but two steps. Sound right? 

Any way to make as "one transaction" ... in the rare event that one step fails, then neither artifact would be deployed?
Comment 14 Jan Sievers CLA 2015-03-23 09:25:10 EDT
(In reply to David Williams from comment #13)
> From looking at the job, and the doc for deploy:deploy-file I believe I
> should just add an additional "maven 3" build step, so it'd run "one job",
> but two steps. Sound right? 

yes, it should be as simpe as that

> Any way to make as "one transaction" ... in the rare event that one step
> fails, then neither artifact would be deployed?

not really. Nexus Pro has staging repos for deployment atomicity but I guess this is not an option at eclipse.org
Comment 15 David Williams CLA 2015-03-23 15:55:25 EDT
My local test build seems to be fine, after updating "both" (bug 462774). 

So, I'd say this could be marked "fixed", but the "verification" test can only be done after another nightly, and "deploy" of parent pom (at midnight), so, I'd suggest if that UI jobs runs ok on Tuesday morning, this could be marked "verified". 

Thanks, 

Isn't it fun to learn new things? :)
Comment 16 Jay Arthanareeswaran CLA 2015-04-28 04:09:01 EDT
David, I suppose the fix is working as expected and can be marked as verified? Thanks for confirming.
Comment 17 David Williams CLA 2015-04-28 08:51:09 EDT
Yes, it working well.