Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 326465

Summary: Deadlock on startup using DefaultClassLoader/StartupClassLoader locks
Product: [Eclipse Project] Equinox Reporter: Andrey Loskutov <loskutov>
Component: FrameworkAssignee: equinox.framework-inbox <equinox.framework-inbox>
Status: CLOSED NOT_ECLIPSE QA Contact:
Severity: major    
Priority: P3 CC: jamesblackburn+eclipse, Konstantin.Scheglov, tjwatson
Version: 3.6   
Target Milestone: ---   
Hardware: PC   
OS: Linux   
Whiteboard:
Attachments:
Description Flags
backtrace none

Description Andrey Loskutov CLA 2010-09-28 16:55:21 EDT
Per some really strange accident I've seen same deadlock today twice on different Linux based Eclipse installations. In both cases *it seems* that the root cause could be the slow network/slow version control system connections, but I'm not sure. I'm pretty sure I've never seen such deadlock on the Eclipse 3.5.x, so it seems to be 3.6 issue. 

Case 1: RHEL 5.3 64 bit Linux box with Eclipse 3.6.0 64 bit & Clearcase, 1.6.0_16 64 bit Sun VM, workstation with 8-core Intel CPU & 16 GB RAM.

Case 2: Ububtu 10.04 64 bit Linux box with Eclipse 3.6.1 64 bit & SVN, 1.6.0_20 64 bit Sun VM, workstation with 4-core Intel CPU & 8 GB RAM.

Both deadlocks occured immediately after the startup, on the first mouse click somewhere into the Eclipse window area. After this click, Eclipse UI was not responsible. In both cases I was unable to generate "usual" thread dump as the jstack was unable to connect to the JVM. In both cases same deadlock was found by jstack (see below).

I can't reproduce this deadlock, even that it happened twice today, it didn't occur before. I hope that the locks shown by the jstack could help to identify the root cause.

andrei@pinguin ~ $ java -version
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)

andrei@pinguin ~ $ jstack 3086
3086: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding

andrei@pinguin ~ $ jstack -F 3086
Attaching to process ID 3086, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 16.3-b01
Deadlock Detection:

Found one Java-level deadlock:
=============================

"Worker-4":
  waiting to lock Monitor@0x00007f2d604ae598 (Object@0x00007f2d79f71cd0, a org/eclipse/osgi/internal/baseadaptor/DefaultClassLoader),
  which is held by "Worker-1"
"Worker-1":
  waiting to lock Monitor@0x000000004163a2c8 (Object@0x00007f2d770f0e20, a org/eclipse/equinox/launcher/Main$StartupClassLoader),
  which is held by "Worker-4"

Found a total of 1 deadlock.

Thread 3195: (state = BLOCKED)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
	at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
	at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
	at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
	at sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
	at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
	at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
	at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
	at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
	at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
	at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
	at sun.tools.jstack.JStack.main(JStack.java:84)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
	at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
	at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
	at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
	at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)


Thread 3194: (state = BLOCKED)
Error occurred during stack walking:
... and so on, without any usable stack trace from any thread ...
Comment 1 Thomas Watson CLA 2010-09-29 08:37:59 EDT
It is difficult to tell what the deadlock is without the stacks from the two deadlocked threads.  It appears that this is another case where the VM locks the classloader and we are getting into a deadlock because of it.  The thing I don't understand is why we need to obtain a bundle's class loader lock (on DefaultClassLoader) while loading classes with the StartupClassLoader.  The StartupClassLoader is used to load the framework and the framework classes cannot have dependencies on classes loaded from a bundle class loader.  Something else must be going on but it is impossible to tell at this point.
Comment 2 James Blackburn CLA 2010-09-29 12:50:15 EDT
Created attachment 179866 [details]
backtrace

This just occurred to me.

Using Sun JDK 1.6.0_16 main deadlocked with Worker-2 between the Main$StartupClassLoader and DefaultClassLoader locks.

Backtrace attached.
Comment 3 Thomas Watson CLA 2010-09-29 14:48:11 EDT
(In reply to comment #2)
> Created an attachment (id=179866) [details]
> backtrace
> 
> This just occurred to me.
> 
> Using Sun JDK 1.6.0_16 main deadlocked with Worker-2 between the
> Main$StartupClassLoader and DefaultClassLoader locks.
> 
> Backtrace attached.

Thanks James, any idea what DesignerPlugin is?  or why it appears they have installed a security manager.  It looks like this is what is causing the deadlock in this case.  The security manager for DesignerPlugin looks to be getting used while trying to load classes for the framework which causes additional classes to need to be loaded from the DesignerPlugin.
Comment 4 Thomas Watson CLA 2010-09-29 15:52:21 EDT
I just noticed james is not cc'ed on the bug.  James please see comment 3.
Comment 5 Andrey Loskutov CLA 2010-09-29 15:53:39 EDT
(In reply to comment #3)
> any idea what DesignerPlugin is? 

I guess this is com.instantiations.designer.core.DesignerPlugin from
WindowBuilder from http://code.google.com/webtoolkit/tools/wbpro/index.html

I have it recently installed on both machines, so this could be the initial root cause for my issues.
Comment 6 James Blackburn CLA 2010-09-29 16:15:43 EDT
(In reply to comment #5)
> I have it recently installed on both machines, so this could be the initial
> root cause for my issues.

Same here!
Comment 7 Thomas Watson CLA 2010-09-29 16:45:00 EDT
Until Java 7 is available and we can take advantage of parallel class loaders I am not sure what we can do about this in eclipse.  Unfortunately the VM locks the startup class loader natively and we cannot really get around that.

I suggest an issue be opened against WindowBuilder.  I think the combination of installing a security manager and being an early startup plugin causes this deadlock to show up.  If they must do both in WindowBuilder then one possible work around is for them to eagerly load the complete implementation of their security manager before setting it as the security manager of the VM.  This way when it gets used to do security checks we don't need to load additional classes for the security manager while loading framework implementation classes.
Comment 8 Andrey Loskutov CLA 2010-09-30 16:52:34 EDT
(In reply to comment #7)
> I suggest an issue be opened against WindowBuilder. 

I did this, but have no idea if it was successful or not, as the bug submission can only be performed from their own plugin. There seems to be neither publicly accessible tracker nor email confirmation for the bug submission. So I can't put here any pointer to their bug tracker :-(
Comment 9 Konstantin Scheglov CLA 2010-10-01 05:20:16 EDT
I just want to confirm that as developer of WindowBuilder I've read this thread now and will try to fix problem.

We use SecurityManager to prevent user code from executing exit(). We saw couple examples of this in user code and during using JFrame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE) too. I will try to move installing SecurityManager to the time when we are going to parse and execute source.
Comment 10 James Blackburn CLA 2010-10-01 05:22:55 EDT
(In reply to comment #9)
> I just want to confirm that as developer of WindowBuilder I've read this thread
> now and will try to fix problem.

Thanks Konstantin.  It would be good if there were a public bug tracker for this tool as firing bug reports into the ether doesn't gives no positive feedback.
Comment 11 Konstantin Scheglov CLA 2010-10-01 05:36:21 EDT
Now we will install SecurityManager just before parsing, not during Bundle start. This change will be included into next public build.
Comment 12 Thomas Watson CLA 2010-10-01 08:26:59 EDT
I am going to close this as not eclipse.  Thanks all.