Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 329302

Summary:

Deadlock between UIThread and UICallback worker thread

Product:

[RT] RAP

Reporter:

Phil Xue <phil.xue>

Component:

RWT

Assignee:

Project Inbox <rap-inbox>

Status:

RESOLVED FIXED

QA Contact:

Severity:

critical

Priority:

CC:

chris, fwaibel, Istvan.Ballok, rsternberg

Version:

unspecified

Target Milestone:

1.4 RC1

Hardware:

OS:

Windows XP

Whiteboard:

sr132

Attachments:

Description	Flags
thread dump	none
Patch to fix the deadlock	none

Description Phil Xue

2010-11-02 17:16:48 EDT

Build Identifier: 

The deadlock occurred right after we migrated to RAP 1.3.1 (we've been using 1.3.0M6), as I know that there were quite a few bug fixes (e.g. 315418 and 301261) that didn't make it to 1.3.1 we then moved to rap-runtime-1.3.1-N-20101026-1455 which should include most of fixes on v13_Mainentance branch to date, but we hit the exact same deadlock again yesterday.

INFO   | jvm 2    | 2010/11/02 10:15:43 | "Worker-10":
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.rwt.internal.lifecycle.UICallBackManager.sendUICallBack(UICallBackManager.java:69)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	- waiting to lock <0x00002aaac027a490> (a java.lang.Object)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.swt.widgets.Display$3.run(Display.java:1165)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.rwt.internal.lifecycle.FakeContextUtil.runNonUIThreadWithFakeContext(FakeContextUtil.java:60)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.rwt.lifecycle.UICallBack.runNonUIThreadWithFakeContext(UICallBack.java:48)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.swt.widgets.Display.wakeThread(Display.java:1163)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.swt.widgets.Synchronizer.addLast(Synchronizer.java:73)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.swt.widgets.Synchronizer.asyncExec(Synchronizer.java:95)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.ui.internal.UISynchronizer.asyncExec(UISynchronizer.java:129)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.swt.widgets.Display.asyncExec(Display.java:984)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	- locked <0x00002aaac0282ee8> (a java.lang.Object)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.rap.ui.internal.progress.JobManagerAdapter.done(JobManagerAdapter.java:120)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	- locked <0x00002aaab8e33b38> (a java.lang.Object)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.core.internal.jobs.JobListeners$3.notify(JobListeners.java:39)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.core.internal.jobs.JobListeners.doNotify(JobListeners.java:96)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.core.internal.jobs.JobListeners.done(JobListeners.java:152)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.core.internal.jobs.JobManager.endJob(JobManager.java:647)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.core.internal.jobs.WorkerPool.endJob(WorkerPool.java:105)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:71)
INFO   | jvm 2    | 2010/11/02 10:15:43 | "UIThread [FEC7851B2DE927D3A59B83B4B02C3224]":
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.swt.graphics.Device.isDisposed(Device.java:300)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	- waiting to lock <0x00002aaac0282ee8> (a java.lang.Object)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.rwt.internal.lifecycle.UICallBackManager.hasRunnables(UICallBackManager.java:100)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.rwt.internal.lifecycle.UICallBackManager.notifyUIThreadEnd(UICallBackManager.java:91)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	- locked <0x00002aaac027a490> (a java.lang.Object)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.rwt.internal.lifecycle.RWTLifeCycle.sleep(RWTLifeCycle.java:297)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.swt.widgets.Display.sleep(Display.java:1128)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.ui.application.WorkbenchAdvisor.eventLoopIdle(WorkbenchAdvisor.java:361)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.java:2391)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:2351)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.ui.internal.Workbench.access$5(Workbench.java:2206)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.ui.internal.Workbench$4.run(Workbench.java:424)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.jface.internal.databinding.realmadapter.RealmAdapter.run(RealmAdapter.java:44)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.rap.ui.internal.RealmAdapterHook.runWithDefault(RealmAdapterHook.java:48)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:427)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:157)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at com.orchestral.core.conductor.impl.Conductor.start(Conductor.java:2)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.rap.ui.internal.application.EntrypointApplicationWrapper.createUI(EntrypointApplicationWrapper.java:27)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.rwt.internal.lifecycle.EntryPointManager.createUI(EntryPointManager.java:92)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.rwt.internal.lifecycle.RWTLifeCycle.createUI(RWTLifeCycle.java:245)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.rwt.internal.lifecycle.RWTLifeCycle$UIThreadController.run(RWTLifeCycle.java:114)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	- locked <0x00002aaac0279fe0> (a org.eclipse.rwt.internal.lifecycle.UIThread)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at java.lang.Thread.run(Thread.java:619)
INFO   | jvm 2    | 2010/11/02 10:15:43 | 	at org.eclipse.rwt.internal.lifecycle.UIThread.run(UIThread.java:102)

Reproducible: Sometimes

Steps to Reproduce:
It's not easy to come up with a sample code to reproduce, but we've hit the same deadlock at least twice a day.

Comment 1 Phil Xue

2010-11-02 17:19:42 EDT

Created attachment 182258 [details]
thread dump

thread dump file attached for ease of reading

Comment 2 Ralf Sternberg

2010-12-04 04:31:09 EST

Created attachment 184531 [details]
Patch to fix the deadlock

The problem is that the UICallBackManager calls Display methods (in UIThread) to get the runnable count while the Display calls UICallBackManager methods in a backgroundThread (asyncExec) to wake a blocked UICallBack request. Both Display and UICallBackManager acquire their own locks in the respective methods.

With this patch, the Synchronizer notifies the UICallBackManager when the runnables queue changes from empty to filled (or vice versa) so that the UICallBackManager does not need to access the Display anymore.

Comment 3 Ralf Sternberg

2010-12-05 12:15:04 EST

Applied patch to 1.3 branch after reviewing with Rüdiger.
For HEAD, I'll try to find a better solution, ideally one that doesn't involve two different locks for Display and UICallBackManager.

Comment 4 Chris Fairhall

2011-02-02 14:48:28 EST

(In reply to comment #3)
> Applied patch to 1.3 branch after reviewing with Rüdiger.
> For HEAD, I'll try to find a better solution, ideally one that doesn't involve
> two different locks for Display and UICallBackManager.

Is there any chance this will make it into 1.3.2 on feb 25th?

Comment 5 Ralf Sternberg

2011-02-02 15:13:00 EST

(In reply to comment #4)
> Is there any chance this will make it into 1.3.2 on feb 25th?

Yes, the fix is included in 1.3.2.

Comment 6 Chris Fairhall

2011-02-02 17:08:43 EST

(In reply to comment #5)
> (In reply to comment #4)
> > Is there any chance this will make it into 1.3.2 on feb 25th?
> 
> Yes, the fix is included in 1.3.2.

What about 1.4 M5?

Comment 7 Ralf Sternberg

2011-02-02 18:10:46 EST

(In reply to comment #6)
> What about 1.4 M5?

See comment 3. When this bug is fixed, the Target Milestone field will be set to the milestone that contains the fix.

Comment 8 Ralf Sternberg

2011-05-06 06:54:05 EDT

Applied patch to CVS HEAD. Will be available with 1.4 RC1.

Comment 9 Ralf Sternberg

2011-05-06 11:33:13 EDT

We decided to keep this fix as a basis for a rework of the UICallback mechanism in 1.5. Opened bug 344989 for this re-design.