Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.
Bug 62368 - JobManager deadlock
Summary: JobManager deadlock
Status: VERIFIED FIXED
Alias: None
Product: Platform
Classification: Eclipse Project
Component: Resources (show other bugs)
Version: 3.0   Edit
Hardware: PC Windows XP
: P1 major (vote)
Target Milestone: 3.0 M9   Edit
Assignee: Tod Creasey CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2004-05-14 16:48 EDT by Ed Burnette CLA
Modified: 2004-05-18 12:18 EDT (History)
3 users (show)

See Also:


Attachments
Deadlock view (38.98 KB, image/x-png)
2004-05-14 16:51 EDT, Ed Burnette CLA
no flags Details
Threads view (33.14 KB, image/x-png)
2004-05-14 16:51 EDT, Ed Burnette CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Ed Burnette CLA 2004-05-14 16:48:16 EDT
I20040514 (with org.eclipse.ui.workbench from HEAD):
A Self-hosted workspace is not responding because, according to the Threads 
and Monitors view, there is a deadlock condition.

Here are all the stack frames, copied from the Debug view:

org.eclipse.core.launcher.Main at localhost:3934 (Suspended)
	System Thread [Finalizer] (Suspended)
		Object.wait(long) line: not available [native method]
		ReferenceQueue.remove(long) line: not available
		ReferenceQueue.remove() line: not available
		Finalizer$FinalizerThread.run() line: not available
	System Thread [Reference Handler] (Suspended)
		Object.wait(long) line: not available [native method]
		Reference$Lock(Object).wait() line: not available
		Reference$ReferenceHandler.run() line: not available
	Thread [main] (Suspended)
		JobManager.endJob(InternalJob, IStatus, boolean) line: 392
		ProgressViewUpdater$1(InternalJob).done(IStatus) line: 134
		ProgressViewUpdater$1(Job).done(IStatus) line: 202
		UIJob$1.run() line: 101
		RunnableLock.run() line: 35
		UISynchronizer(Synchronizer).runAsyncMessages() line: 106
		Display.runAsyncMessages() line: 2702
		Display.readAndDispatch() line: 2394
		Workbench.runEventLoop(Window$IExceptionHandler, Display) 
line: 1363
		Workbench.runUI() line: 1334
		Workbench.createAndRunWorkbench(Display, WorkbenchAdvisor) 
line: 253
		PlatformUI.createAndRunWorkbench(Display, WorkbenchAdvisor) 
line: 141
		IDEApplication.run(Object) line: 90
		PlatformActivator$1.run(Object) line: 298
		EclipseStarter.run(Object) line: 249
		EclipseStarter.run(String[], Runnable) line: 126
		NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) 
line: not available [native method]
		NativeMethodAccessorImpl.invoke(Object, Object[]) line: not 
available
		DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 
not available
		Method.invoke(Object, Object[]) line: not available
		Main.basicRun(String[]) line: 269
		Main.run(String[]) line: 722
		Main.main(String[]) line: 706
	System Thread [Signal Dispatcher] (Suspended)
	Thread [Framework Event Dispatcher] (Suspended)
		Object.wait(long) line: not available [native method]
		EventThread(Object).wait() line: not available
		EventThread.getNextEvent() line: 149
		EventThread.run() line: 107
	Thread [Start Level Event Dispatcher] (Suspended)
		Object.wait(long) line: not available [native method]
		EventThread(Object).wait() line: not available
		EventThread.getNextEvent() line: 149
		EventThread.run() line: 107
	Thread [Thread-0] (Suspended)
		Object.wait(long) line: not available [native method]
		ReferenceQueue.remove(long) line: not available
		ReferenceQueue.remove() line: not available
		ActionContributionItem$ImageCache$ReferenceCleanerThread.run() 
line: 196
	Thread [Thread-1] (Suspended)
		Object.wait(long) line: not available [native method]
		ReferenceQueue.remove(long) line: not available
		ReferenceQueue.remove() line: not available
		ActionContributionItem$ImageCache$ReferenceCleanerThread.run() 
line: 196
	Thread [Worker-0] (Suspended)
		JobManager.findBlockingJob(InternalJob) line: 452
		ThreadJob.joinRun(IProgressMonitor) line: 152
		ImplicitJobs.begin(ISchedulingRule, IProgressMonitor, boolean) 
line: 87
		JobManager.beginRule(ISchedulingRule, IProgressMonitor) line: 
167
		WorkManager.checkIn(ISchedulingRule, IProgressMonitor) line: 95
		Workspace.prepareOperation(ISchedulingRule, IProgressMonitor) 
line: 1629
		Workspace.run(IWorkspaceRunnable, ISchedulingRule, int, 
IProgressMonitor) line: 1669
		Workspace.run(IWorkspaceRunnable, IProgressMonitor) line: 1694
		JavaCore$4.run(IProgressMonitor) line: 3644
		Worker.run() line: 66
	Thread [Java indexing] (Suspended)
		JobManager.setPriority(InternalJob, int) line: 839
		JobManager$1$ProgressJob(InternalJob).setPriority(int) line: 
346
		JobManager$1$ProgressJob(Job).setPriority(int) line: 440
		IndexManager(JobManager).run() line: 363
		Thread.run() line: not available
	Thread [Worker-1] (Suspended)
		JobManager.runNow(InternalJob) line: 747
		ThreadJob.joinRun(IProgressMonitor) line: 149
		ImplicitJobs.begin(ISchedulingRule, IProgressMonitor, boolean) 
line: 87
		JobManager.beginRule(ISchedulingRule, IProgressMonitor) line: 
167
		LastSaveReferenceProvider.readDocument(IProgressMonitor, 
boolean) line: 225
		LastSaveReferenceProvider.getReference(IProgressMonitor) line: 
111
		DocumentLineDiffer$1.run(IProgressMonitor) line: 370
		Worker.run() line: 66
	Thread [Worker-2] (Suspended)
		ProgressManager.refreshJobInfo(JobInfo) line: 526
		ProgressManager$JobMonitor.beginTask(String, int) line: 186
		DecorationScheduler$1.run(IProgressMonitor) line: 220
		Worker.run() line: 66
	Thread [Worker-3] (Suspended)
		ProgressManager.refreshJobInfo(JobInfo) line: 526
		ProgressManager$JobMonitor.subTask(String) line: 270
		JobManager$1$ProgressJob.run(IProgressMonitor) line: 319
		Worker.run() line: 66
	Thread [Worker-4] (Suspended)
		Object.wait(long) line: not available [native method]
		Semaphore.acquire(long) line: 38
		JobManager.join(InternalJob) line: 514
		TableContentProvider$1(InternalJob).join() line: 271
		TableContentProvider$1(Job).join() line: 353
		TableContentProvider.doUpdate(IProgressMonitor) line: 353
		TableContentProvider.access$4(TableContentProvider, 
IProgressMonitor) line: 346
		TableContentProvider$3.run(IProgressMonitor) line: 184
		RestartableJob$2.run(IProgressMonitor) line: 86
		Worker.run() line: 66
	Thread [Worker-5] (Suspended)
		ProgressManager.refreshJobInfo(JobInfo) line: 526
		JobInfo.cancel() line: 307
		ProgressManager$JobMonitor.setCanceled(boolean) line: 236
		JobManager.cancel(InternalJob) line: 183
		RestartableJob$2(InternalJob).cancel() line: 122
		RestartableJob$2(Job).cancel() line: 183
		RestartableJob.cancel() line: 152
		TableContentProvider.cancelPendingChanges() line: 336
		ProblemView(TableView).haltTableUpdates() line: 92
		ProblemView(MarkerView).internalRefresh(IProgressMonitor) 
line: 155
		MarkerView.access$2(MarkerView, IProgressMonitor) line: 151
		MarkerView$3.run(IProgressMonitor) line: 237
		RestartableJob$2.run(IProgressMonitor) line: 86
		Worker.run() line: 66
	Thread [Worker-6] (Suspended)
		JobManager.setPriority(InternalJob, int) line: 839
		ThreadJob(InternalJob).setPriority(int) line: 346
		ThreadJob(Job).setPriority(int) line: 440
		ThreadJob.<init>(JobManager, ISchedulingRule) line: 57
		ImplicitJobs.newThreadJob(ISchedulingRule) line: 158
		ImplicitJobs.begin(ISchedulingRule, IProgressMonitor, boolean) 
line: 67
		JobManager.beginRule(ISchedulingRule, IProgressMonitor) line: 
167
		WorkManager.checkIn(ISchedulingRule, IProgressMonitor) line: 95
		Workspace.prepareOperation(ISchedulingRule, IProgressMonitor) 
line: 1629
		AutoBuildJob.doBuild(IProgressMonitor) line: 146
		AutoBuildJob.run(IProgressMonitor) line: 196
		Worker.run() line: 66
	Thread [Worker-7] (Suspended)
		JobManager.nextJob() line: 647
		JobManager.startJob() line: 936
		WorkerPool.startJob(Worker) line: 237
		Worker.run() line: 59
	Thread [Worker-8] (Suspended)
		JobManager.schedule(InternalJob, long) line: 769
		ProgressViewUpdater$1(InternalJob).schedule(long) line: 313
		ProgressViewUpdater$1(Job).schedule(long) line: 417
		ProgressViewUpdater.scheduleUpdate() line: 197
		ProgressViewUpdater.refreshJobInfo(JobInfo) line: 302
		ProgressManager.refreshJobInfo(JobInfo) line: 531
		ProgressManager$1.aboutToRun(IJobChangeEvent) line: 344
		JobListeners$1.notify(IJobChangeListener, IJobChangeEvent) 
line: 34
		JobListeners.doNotify(JobListeners$IListenerDoit, 
IJobChangeEvent) line: 101
		JobListeners.aboutToRun(Job) line: 151
		JobManager.startJob() line: 942
		WorkerPool.startJob(Worker) line: 230
		Worker.run() line: 59
	Thread [Worker-9] (Suspended)
		JobManager.nextJob() line: 647
		JobManager.startJob() line: 936
		WorkerPool.startJob(Worker) line: 230
		Worker.run() line: 59
	Thread [All Types Caching] (Suspended)
		Thread.sleep(long) line: not available [native method]
		IndexManager(JobManager).performConcurrentJob(IJob, int, 
IProgressMonitor) line: 243
		SearchEngine.searchAllTypeNames(IWorkspace, char[], char[], 
int, boolean, int, IJavaSearchScope, ITypeNameRequestor, int, 
IProgressMonitor) line: 865
		AllTypesCache.search(ITypeNameRequestor, int, 
IProgressMonitor) line: 519
		AllTypesCache$TypeCacher.doSearchTypes() line: 188
		AllTypesCache$TypeCacher.run() line: 157

Unfortunately the Threads and Monitors view doesn't support copy (!). So I'll 
attach screenshots shortly.
Comment 1 Ed Burnette CLA 2004-05-14 16:51:07 EDT
Created attachment 10677 [details]
Deadlock view
Comment 2 Ed Burnette CLA 2004-05-14 16:51:21 EDT
Created attachment 10678 [details]
Threads view
Comment 3 John Arthorne CLA 2004-05-14 17:54:24 EDT
This deadlock was introduced by new locking code within the progress monitor
setCanceled() callback from core to UI (ProgressManager.refreshJobInfo).

I will refactor JobManager to avoid owning the job manager lock while calling
the progress monitor.

Tod, this new code pattern in ProgressManager is very deadlock prone. It holds a
lock while calling progress listeners, which may in turn acquire arbitrary locks
(or syncExec). A safer approach is to create a copy of the listener list within
the sync block, and then notify listeners outside the sync block. Alternatively,
use the org.eclipse.jface.util.ListenerList, which Nick is about to release
thread safety fixes for.
Comment 4 John Arthorne CLA 2004-05-14 18:30:09 EDT
I have fixed this deadlock by avoiding calling the progress monitor inside the
job manager sync block.

However, the ProgressManager code also needs to be fixed.  Users of jobs will
not be expecting thread safety issues while calling IProgressMonitor methods. If
they do call the progress monitor while holding a lock at the moment, deadlock
is likely.

Again, using ListenerList, or creating a copy of the listeners and then calling
them outside the sync block is recommended.
Comment 5 Tod Creasey CLA 2004-05-17 09:10:22 EDT
I have made the suggested changes and released them (thanks once again John). 
I have switched to doing a toArray() call and then iterating over the result 
as this will allocate a new array everytime.

John if you could check that I have made the changes you think are neccessary 
I would be most appreciative.
Comment 6 John Arthorne CLA 2004-05-17 11:14:16 EDT
Tod, your fix looks good.