Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 532773

Summary: NPE during API analysis while using Java 10
Product: [Eclipse Project] PDE Reporter: Vikas Chandra <Vikas.Chandra>
Component: API ToolsAssignee: Vikas Chandra <Vikas.Chandra>
Status: RESOLVED FIXED QA Contact:
Severity: major    
Priority: P3 CC: akurtakov, daniel_megert, sarika.sinha, tjwatson
Version: 4.7.3   
Target Milestone: 4.7.3a   
Hardware: PC   
OS: Windows 7   
See Also: https://git.eclipse.org/r/120076
https://git.eclipse.org/c/pde/eclipse.pde.ui.git/commit/?id=26f6dfab4efeba9b7f188f97b44c1b868f949d51
https://git.eclipse.org/r/120167
https://git.eclipse.org/c/pde/eclipse.pde.ui.git/commit/?id=0e085d0187261f0f5d8358effcba91beb59075d5
https://git.eclipse.org/r/120179
https://git.eclipse.org/r/120180
https://git.eclipse.org/c/pde/eclipse.pde.ui.git/commit/?id=4103b1e8ca1cbfcefce21e50860ea1b8175a02a6
https://git.eclipse.org/r/120238
https://git.eclipse.org/c/pde/eclipse.pde.ui.git/commit/?id=59cf13561f08c25cd0cba6cf6319b29f9e1a3e17
https://git.eclipse.org/r/120253
https://git.eclipse.org/c/pde/eclipse.pde.ui.git/commit/?id=e58a1d274db4d2cf99801502a8ebf28a74088e14
https://git.eclipse.org/r/120280
https://git.eclipse.org/c/pde/eclipse.pde.ui.git/commit/?id=e789291de5ef274b764627babd4135f9b47fc7ab
https://git.eclipse.org/r/120323
https://git.eclipse.org/c/pde/eclipse.pde.ui.git/commit/?id=1df2296e60bcd7f149a58ffcf4e137151ac7a7af
https://git.eclipse.org/r/120413
https://git.eclipse.org/c/pde/eclipse.pde.ui.git/commit/?id=d44b26a150298a7e72c42cd6ea93d71a0ad8e2f4
Whiteboard: java10
Attachments:
Description Flags
Hardcoding the VM packages for 1.8->10 scenario none

Description Vikas Chandra CLA 2018-03-22 10:54:45 EDT
Build id 

Version: Oxygen.3a (4.7.3a)
Build id: M20180322-0400


java.lang.NullPointerException
	at org.eclipse.pde.api.tools.internal.model.SystemLibraryApiComponent.createApiDescription(SystemLibraryApiComponent.java:92)
	at org.eclipse.pde.api.tools.internal.model.Component.getApiDescription(Component.java:97)
	at org.eclipse.pde.api.tools.internal.builder.IllegalImplementsProblemDetector.findRestrictedSuperinterfaces(IllegalImplementsProblemDetector.java:215)
	at org.eclipse.pde.api.tools.internal.builder.IllegalImplementsProblemDetector.considerReference(IllegalImplementsProblemDetector.java:71)
Comment 1 Vikas Chandra CLA 2018-03-22 12:57:39 EDT
The reason for this is that VM packages is not coming for java 10 post the merge.
Pre-merge we used to get the following packages for Java 10.

value = getVMPackages();

returns null on 

Version: Oxygen.3 (4.7.3)
Build id: M20180221-1700

On
Eclipse SDK
Version: Oxygen.3 (4.7.3)
Build id: M20180221-1700
OS: Windows 7, v.6.1, x86_64 / win32

with java10 patch it used to return all the packages.

Both the setup had only Java10 as the installed JRE.

I am investigating this further.
Comment 2 Vikas Chandra CLA 2018-03-23 02:38:27 EDT
Even after hardcoding the values of "value" of comment 1, I am getting multiple java.lang.NoClassDefFoundError exceptions like below


!MESSAGE Problems occurred when invoking code from plug-in: "org.eclipse.core.resources".
!STACK 0
java.lang.NoClassDefFoundError: org.eclipse.jdt.internal.core.builder.AbstractImageBuilder$1
	at org.eclipse.jdt.internal.core.builder.AbstractImageBuilder.addAllSourceFiles(AbstractImageBuilder.java:222)
	at org.eclipse.jdt.internal.core.builder.BatchImageBuilder.build(BatchImageBuilder.java:52)
	at org.eclipse.jdt.internal.core.builder.JavaBuilder.buildAll(JavaBuilder.java:256)
	at org.eclipse.jdt.internal.core.builder.JavaBuilder.build(JavaBuilder.java:175)

MESSAGE Problems occurred when invoking code from plug-in: "org.eclipse.core.resources".
!STACK 0
java.lang.NoClassDefFoundError: org.eclipse.jdt.internal.core.util.HashSetOfArray
	at org.eclipse.jdt.internal.core.JavaProjectElementInfo.getProjectCache(JavaProjectElementInfo.java:220)
	at org.eclipse.jdt.internal.core.JavaProjectElementInfo.newNameLookup(JavaProjectElementInfo.java:287)
	at org.eclipse.jdt.internal.core.JavaProject.newNameLookup(JavaProject.java:2622)
	at org.eclipse.jdt.internal.core.JavaProject.newNameLookup(JavaProject.java:2632)
	at org.eclipse.jdt.internal.core.JavaProject.findType(JavaProject.java:1614)
	at org.eclipse.jdt.internal.core.JavaProject.findType(JavaProject.java:1521)
Comment 3 Vikas Chandra CLA 2018-03-23 04:13:40 EDT
I also see

Caused by: java.lang.ClassNotFoundException: org.eclipse.jdt.core.dom.Type cannot be found by org.eclipse.pde.api.tools_1.1.150.v20180322-0726
	at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:484)
	at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:395)
	at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:387)
	at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:150)


which is an indication of required version mismatch.
Comment 4 Vikas Chandra CLA 2018-03-23 06:02:48 EDT
When I launch directly in Java10, I don't get any of the problem.

The scenario that causes these problems are

1) Launch eclipse(M20180322-1835) with Java 1.8
2) Remove Java 1.8 from Installed JRE
3) Add Java 10 in Installed JRE
4) Take jdt.core plugin from jdt.core repo
5) In preference, change Missing API Baseline option from "Error" to "Warning" or "Ignore".

You will get comment#0.  

Even after restart, problem persists.
Comment 5 Vikas Chandra CLA 2018-03-23 06:05:12 EDT
Sometimes I also get
java.lang.BootstrapMethodError: java.lang.LinkageError: loading constraint violation when overriding method "jdk/internal/jimage/ImageReader$SharedImageReader$LocationVisitor.visit(Ljdk/internal/jimage/ImageLocation;)V" during creation of class "jdk/internal/jimage/ImageReader$SharedImageReader$$Lambda$650/000000000461FA00": loader "java/lang/InternalAnonymousClassLoader@e4e84284" of class "jdk/internal/jimage/ImageReader$SharedImageReader$$Lambda$650/000000000461FA00" and loader "java/net/URLClassLoader@eab61cb5" of class "jdk/internal/jimage/ImageReader$SharedImageReader$LocationVisitor" have different types for the method signature
	at jdk.internal.jimage.ImageReader$SharedImageReader.handlePackages(ImageReader.java:389)
	at jdk.internal.jimage.ImageReader$SharedImageReader.buildNode(ImageReader.java:304)
	at jdk.internal.jimage.ImageReader$SharedImageReader.findNode(ImageReader.java:489)
	at jdk.internal.jimage.ImageReader.findNode(ImageReader.java:104)
Comment 6 Vikas Chandra CLA 2018-03-23 06:27:49 EDT
See bug 532470
Comment 7 Vikas Chandra CLA 2018-03-23 06:31:03 EDT
If I launch eclipse with Java 9 and then add java 10, I don't see the problem.
If I launch eclipse with Java 10 , I dont see this problem.


If I launch eclipse with Java 1.8 and then add java 10, comment 0 happens and also comment 1 etc. Even restart doesn't fix it.
Comment 8 Vikas Chandra CLA 2018-03-23 09:05:37 EDT
Created attachment 273274 [details]
Hardcoding the VM packages for 1.8->10 scenario

On Build id: M20180322-1835, I hard-coded the vmpackages ( which was previously hard-coded in OSGI profile) and then launched eclipse in java 1.8. Then I added Java 10 and then built the workspace. The build went fine.

An identical set-up without the hard-coding resulted in comment 0. Also I am unable to see issues like comment 1 and comment 2.
Comment 9 Vikas Chandra CLA 2018-03-23 09:06:34 EDT
Another update: Running build after removing API nature works fine for 1.8->10 scenario ( see comment 7)
Comment 10 Vikas Chandra CLA 2018-03-23 09:08:03 EDT
>> issues like comment 1 and comment 2

I meant comment 2 and comment 3.
Comment 11 Eclipse Genie CLA 2018-03-23 09:34:16 EDT
New Gerrit change created: https://git.eclipse.org/r/120076
Comment 12 Sarika Sinha CLA 2018-03-23 09:44:07 EDT
Java Project works fine when moving from 1.8 to 10 at project level or launching level.
Comment 13 Eclipse Genie CLA 2018-03-23 09:51:07 EDT
Gerrit change https://git.eclipse.org/r/120076 was merged to [R4_7_maintenance].
Commit: http://git.eclipse.org/c/pde/eclipse.pde.ui.git/commit/?id=26f6dfab4efeba9b7f188f97b44c1b868f949d51
Comment 14 Thomas Watson CLA 2018-03-23 10:28:50 EDT
(In reply to Eclipse Genie from comment #13)
> Gerrit change https://git.eclipse.org/r/120076 was merged to
> [R4_7_maintenance].
> Commit:
> http://git.eclipse.org/c/pde/eclipse.pde.ui.git/commit/
> ?id=26f6dfab4efeba9b7f188f97b44c1b868f949d51

I don't understand this fix.  Why hard code the packages here in the java code instead of adding new profiles for the Java versions?

http://git.eclipse.org/c/pde/eclipse.pde.ui.git/tree/apitools/org.eclipse.pde.api.tools/src/org/eclipse/pde/api/tools/internal/util/profiles?h=R4_7_maintenance

The current strategy of using the ModuleLayer API will only work if you are running the code on a VM that has the ModuleLayer API.  That would be Java 9 or layer.  And even then, if you are targeting some VM greater than the VM running the PDE code the packages may not be correct.  The PDE code will discover packages available it the VM running the PDE code.  If PDE is launched with Java 9 it will find the packages for Java 9, but if you configured in Java 10, 11, etc as the installed JVM of your workspace and your project targets that version you will have used the wrong package list from Java 9.

I'm pretty sure JDT has some code somewhere that discovers the packages of the installed VMs.  I suggest you use that approach to fix this.
Comment 15 Vikas Chandra CLA 2018-03-23 10:58:53 EDT
>>Why hard code the packages here in the java code 

This is just a stop-gap arrangement for 473aRC1.

>>new profiles for the Java version

I am planning to do this on Monday.

>>I suggest you use that approach to fix this.

I will explore this methodology for long term solution of this issue.

Thanks for your suggestions. They were very helpful !!
Comment 16 Vikas Chandra CLA 2018-03-23 11:02:47 EDT
Making this defect from blocker to major for now.

On Monday, I plan to do the same via profile. If there are any concerns, please let me know.
Comment 17 Vikas Chandra CLA 2018-03-23 13:13:01 EDT
Verified that there is no NPE or java.lang.NoClassDefFoundError
on a large set of plugins ( all plugins of jdt.core + pde.ui + platform.releng) with Java 10 ( or while switching from Java 1.8-> Java 10 - see comment 7)

on 

Version: Oxygen.3a (4.7.3a)
Build id: M20180323-1030
Comment 18 Vikas Chandra CLA 2018-03-23 13:34:07 EDT
Pending action items on this bug ( for me)

1) Fix this issue ( comment 0) in master using profile

2) Revert the fix ( comment 13) ( When R4_7_maintenance stream is open for 473aRC2 contribution)

3) Cherry-pick 1) onto R4_7_maintenance  stream ( When R4_7_maintenance stream is open for 473aRC2 contribution)

4) Verify again in 4.7.3a as well as 4.8 builds
Comment 19 Dani Megert CLA 2018-03-24 06:17:23 EDT
(In reply to Vikas Chandra from comment #7)
> If I launch eclipse with Java 9 and then add java 10, I don't see the
> problem.
> If I launch eclipse with Java 10 , I dont see this problem.
> 
> 
> If I launch eclipse with Java 1.8 and then add java 10, comment 0 happens
> and also comment 1 etc. Even restart doesn't fix it.

Can you explain why it makes a difference whether Java 1.8 or 9 is used? Looks like some (wrong?) caching is done.
Comment 20 Thomas Watson CLA 2018-03-24 08:20:02 EDT
(In reply to Dani Megert from comment #19)
> (In reply to Vikas Chandra from comment #7)
> > If I launch eclipse with Java 9 and then add java 10, I don't see the
> > problem.
> > If I launch eclipse with Java 10 , I dont see this problem.
> > 
> > 
> > If I launch eclipse with Java 1.8 and then add java 10, comment 0 happens
> > and also comment 1 etc. Even restart doesn't fix it.
> 
> Can you explain why it makes a difference whether Java 1.8 or 9 is used?
> Looks like some (wrong?) caching is done.

See my comment 14.  The current strategy attempts to use reflection on Java 9 ModuleLayer types to figure out the packages available from the VM running PDE (Java 8).  That cannot work when running on Java 8 no matter what versions of Java is being targeted in the workspace.  Even if it did work, this strategy is incorrect.  PDE needs to figure out the packages that are available from the Java version being targeted in the workspace not the Java version running PDE/Eclipse.  That may be different than the Java version used to launch PDE/Eclipse.
Comment 21 Eclipse Genie CLA 2018-03-26 06:19:11 EDT
New Gerrit change created: https://git.eclipse.org/r/120167
Comment 23 Alexander Kurtakov CLA 2018-03-26 06:36:08 EDT
Why adding the profile in PDE instead of Equinox?
Comment 24 Vikas Chandra CLA 2018-03-26 07:25:28 EDT
>>Why adding the profile in PDE instead of Equinox?

The profile was added in OSGI earlier and it will be great if this is added there.

Thomas?
Comment 25 Eclipse Genie CLA 2018-03-26 08:47:36 EDT
New Gerrit change created: https://git.eclipse.org/r/120179
Comment 26 Eclipse Genie CLA 2018-03-26 09:01:50 EDT
New Gerrit change created: https://git.eclipse.org/r/120180
Comment 28 Vikas Chandra CLA 2018-03-26 09:08:48 EDT
>>https://git.eclipse.org/r/120180

In R4_7_maintenance branch, this is the gerrit patch that reverts the hard-coding and retrieves the VM packages via a profile in "org.eclipse.pde.api.tools" plugin.
Comment 29 Vikas Chandra CLA 2018-03-26 09:33:35 EDT
>>Why adding the profile in PDE instead of Equinox?


Ideally, I think that the replacement for OSGI profile ( which is jdt.launching's execution environment profile) should provide VM packages of the EE.

As of now, it provides all other attributes of the EE's profile except VM packages. Going forward ( 4.8 and later), I think it could be stored as just JavaSE-10-VM-packages and retrieved from jdt-launching's EE's profile as a property.

As Tom points out multiple times, the current strategy is incorrect and needs either dynamic evaluation of VM packages that is targeted in workspace and not the one launched with. 

>>I'm pretty sure JDT has some code somewhere that discovers the packages of the >>installed VMs

As of now, I am not even sure if this is possible unless JDT has these VM packages hard-coded somewhere ( because if eclipse is launched in 1.8 and then workspace JRE changes to java 10, it can't dynamically calculate the VM packages - similar to what happened in calculateVMpackages() code in PDE that was taken from OSGI). In any case, this needs investigation and I don't have any time estimate for this task at this present moment.

Given the present state of things, I had advocated comment 18. As a followup, I will recommend this patch ( comment 28) for 473aRC2.

Any objections?
Comment 30 Vikas Chandra CLA 2018-03-26 09:46:33 EDT
Note to self: Refactor code in master stream.
Comment 31 Thomas Watson CLA 2018-03-26 12:24:15 EDT
(In reply to Vikas Chandra from comment #24)
> >>Why adding the profile in PDE instead of Equinox?
> 
> The profile was added in OSGI earlier and it will be great if this is added
> there.
> 
> Thomas?

I will not be including new Java profiles in the Framework.  The framework no longer needs hard coded profiles and updating every 6 months with new profiles for usecases outside the framework is not the correct solution in my opinion.
Comment 32 Thomas Watson CLA 2018-03-26 12:31:13 EDT
Note that api.tools already has java profiles located here:

http://git.eclipse.org/c/pde/eclipse.pde.ui.git/tree/apitools/org.eclipse.pde.api.tools/src/org/eclipse/pde/api/tools/internal/util/profiles

It appears these profiles only get used when not running in an OSGi framework.  If running inside the Equinox framework then it looks for these profiles from org.eclipse.osgi directly instead.  I would suggest you always look for the profiles in one place which is controlled by api.tools instead of looking in multiple places depending on the environment that api.tools is running in.  I would suggest that location be 

http://git.eclipse.org/c/pde/eclipse.pde.ui.git/tree/apitools/org.eclipse.pde.api.tools/src/org/eclipse/pde/api/tools/internal/util/profiles
Comment 33 Alexander Kurtakov CLA 2018-03-27 04:29:31 EDT
With 4.8.0.I20180326-2000 I see a lot of :
java.lang.NullPointerException
	at org.eclipse.pde.api.tools.internal.model.ApiBaseline.initialize(ApiBaseline.java:365)
	at org.eclipse.pde.api.tools.internal.model.ApiBaseline.initialize(ApiBaseline.java:277)
	at org.eclipse.pde.api.tools.internal.model.ApiBaseline.resolveSystemLibrary(ApiBaseline.java:556)
	at org.eclipse.pde.api.tools.internal.model.WorkspaceBaseline.addApiComponents(WorkspaceBaseline.java:57)
	at org.eclipse.pde.api.tools.internal.ApiBaselineManager.createWorkspaceBaseline(ApiBaselineManager.java:678)
	at org.eclipse.pde.api.tools.internal.ApiBaselineManager.getWorkspaceBaseline(ApiBaselineManager.java:609)
	at org.eclipse.pde.api.tools.internal.builder.ApiAnalysisBuilder.build(ApiAnalysisBuilder.java:345)
	at org.eclipse.core.internal.events.BuildManager$2.run(BuildManager.java:795)
	at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
...

Preventing me from even launching child eclipse.
Comment 34 Eclipse Genie CLA 2018-03-27 04:41:48 EDT
New Gerrit change created: https://git.eclipse.org/r/120238
Comment 36 Eclipse Genie CLA 2018-03-27 07:44:31 EDT
New Gerrit change created: https://git.eclipse.org/r/120253
Comment 38 Vikas Chandra CLA 2018-03-27 08:29:55 EDT
*** Bug 532927 has been marked as a duplicate of this bug. ***
Comment 39 Dani Megert CLA 2018-03-27 08:51:22 EDT
(In reply to Thomas Watson from comment #32)
> I would suggest that location be 
> http://git.eclipse.org/c/pde/eclipse.pde.ui.git/tree/apitools/org.eclipse.pde.api.tools/src/org/eclipse/pde/api/tools/internal/util/profiles
> 

We might have to put them in a lower layer, see bug 525831.
Comment 40 Vikas Chandra CLA 2018-03-27 10:55:38 EDT
I downloaded Build id: I20180327-0805 and verified that comment 33 doesn't happen.

Can someone else check that too? Alex ?
Comment 41 Alexander Kurtakov CLA 2018-03-27 11:00:07 EDT
(In reply to Vikas Chandra from comment #40)
> I downloaded Build id: I20180327-0805 and verified that comment 33 doesn't
> happen.
> 
> Can someone else check that too? Alex ?

I don't see the issue on master anymore.
Comment 42 Eclipse Genie CLA 2018-03-27 11:30:00 EDT
New Gerrit change created: https://git.eclipse.org/r/120280
Comment 44 Eclipse Genie CLA 2018-03-28 02:25:42 EDT
New Gerrit change created: https://git.eclipse.org/r/120323
Comment 45 Vikas Chandra CLA 2018-03-28 02:32:11 EDT
For 473aRC2, I have created patch https://git.eclipse.org/r/120323

Although eclipse-SDK-M20180323-1030-win32-x86_64 onwards, things work in 473a builds but there were few things that needs to be corrected even with hard-coding

a) removes the dynamic calculation totally
b) have a javase-10 check
c) remove spaces

We have the exact same code in master stream ( in build Build id: I20180327-2000) and I tested it to work fine. I want to port this code into 473aRC2.
Comment 46 Vikas Chandra CLA 2018-03-28 03:52:25 EDT
Incorporated the suggestions of Sarika in 120323/4 which includes removal of redundant code.
Comment 47 Eclipse Genie CLA 2018-03-28 03:56:06 EDT
Gerrit change https://git.eclipse.org/r/120323 was merged to [R4_7_maintenance].
Commit: http://git.eclipse.org/c/pde/eclipse.pde.ui.git/commit/?id=1df2296e60bcd7f149a58ffcf4e137151ac7a7af
Comment 48 Vikas Chandra CLA 2018-03-28 03:58:19 EDT
Committing early to get it picked by 1:30pm IST M build.
Comment 49 Vikas Chandra CLA 2018-03-28 07:35:02 EDT
Verified that there is no NPE in

Version: Oxygen.3a (4.7.3a)
Build id: M20180328-0400

after the recent changes on maint stream.
Comment 50 Eclipse Genie CLA 2018-03-29 05:00:48 EDT
New Gerrit change created: https://git.eclipse.org/r/120413
Comment 52 Vikas Chandra CLA 2018-03-29 05:18:40 EDT
Opened bug 533052 for a discussion on long-term solution.