Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 55635

Summary: Eclipse hangs on full rebuild
Product: [Eclipse Project] Platform Reporter: Andre Weinand <andre_weinand>
Component: ResourcesAssignee: John Arthorne <john.arthorne>
Status: RESOLVED FIXED QA Contact:
Severity: critical    
Priority: P3 CC: douglas.pollock, Michael.Valenta, Tod_Creasey
Version: 3.0   
Target Milestone: 3.0 M8   
Hardware: All   
OS: All   
Whiteboard:
Attachments:
Description Flags
full thread dump
none
Proposed fix to UILockListener none

Description Andre Weinand CLA 2004-03-23 04:38:59 EST
I20040322

I did a Project/Clear. Since I have autobuild on, eclipse immediately started to rebuild everything.
After some time it hanged (see attached full thread dump):


Full thread dump Java HotSpot(TM) Client VM (1.4.2-34 mixed mode):

"Worker-14" prio=5 tid=0x0c1f2c60 nid=0xf0ba400 in Object.wait() [f0a94000..f0a94b20]
	at java.lang.Object.wait(Native Method)
	- waiting on <0x609135a0> (a org.eclipse.core.internal.jobs.ImplicitJobs$ThreadJob)
	at org.eclipse.core.internal.jobs.ImplicitJobs$ThreadJob.joinRun(ImplicitJobs.java:118)
	- locked <0x609135a0> (a org.eclipse.core.internal.jobs.ImplicitJobs$ThreadJob)
	at org.eclipse.core.internal.jobs.ImplicitJobs.begin(ImplicitJobs.java:265)
	at org.eclipse.core.internal.jobs.JobManager.beginRule(JobManager.java:149)
	at org.eclipse.ui.views.markers.internal.MarkerView.internalRefresh(MarkerView.java:168)
	at org.eclipse.ui.views.markers.internal.MarkerView.access$2(MarkerView.java:157)
	at org.eclipse.ui.views.markers.internal.MarkerView$4.run(MarkerView.java:242)
	at org.eclipse.ui.views.markers.internal.RestartableJob$2.run(RestartableJob.java:86)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:62)

"Worker-12" prio=5 tid=0x0c1eced0 nid=0xdff4200 in Object.wait() [f080f000..f080fb20]
	at java.lang.Object.wait(Native Method)
	- waiting on <0x60d82b90> (a org.eclipse.core.internal.jobs.ImplicitJobs$ThreadJob)
	at org.eclipse.core.internal.jobs.ImplicitJobs$ThreadJob.joinRun(ImplicitJobs.java:118)
	- locked <0x60d82b90> (a org.eclipse.core.internal.jobs.ImplicitJobs$ThreadJob)
	at org.eclipse.core.internal.jobs.ImplicitJobs.begin(ImplicitJobs.java:265)
	at org.eclipse.core.internal.jobs.JobManager.beginRule(JobManager.java:149)
	at 
org.eclipse.team.internal.ccvs.core.syncinfo.ReentrantLock$ThreadInfo.pushRule(ReentrantLock.java:80)
	at org.eclipse.team.internal.ccvs.core.syncinfo.ReentrantLock.acquire(ReentrantLock.java:269)
	at 
org.eclipse.team.internal.ccvs.core.resources.EclipseSynchronizer.beginBatching(EclipseSynchronizer.jav
a:496)
	at 
org.eclipse.team.internal.ccvs.core.resources.EclipseSynchronizer.ignoreFilesChanged(EclipseSynchroni
zer.java:632)
	at 
org.eclipse.team.internal.ccvs.core.syncinfo.DeferredResourceChangeHandler.dispatchEvents(DeferredR
esourceChangeHandler.java:64)
	at 
org.eclipse.team.internal.core.BackgroundEventHandler.processEvents(BackgroundEventHandler.java:
278)
	at org.eclipse.team.internal.core.BackgroundEventHandler$1.run(BackgroundEventHandler.java:
131)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:62)

"Worker-11" prio=5 tid=0x0c1eca10 nid=0xf038e00 in Object.wait() [f070d000..f070db20]
	at java.lang.Object.wait(Native Method)
	at org.eclipse.core.internal.jobs.WorkerPool.sleep(WorkerPool.java:185)
	- locked <0x5d529748> (a org.eclipse.core.internal.jobs.WorkerPool)
	at org.eclipse.core.internal.jobs.WorkerPool.startJob(WorkerPool.java:211)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)

"Worker-10" prio=5 tid=0x0c1ec6d0 nid=0xf02ea00 in Object.wait() [f068c000..f068cb20]
	at java.lang.Object.wait(Native Method)
	- waiting on <0x5c59e350> (a org.eclipse.ui.internal.Semaphore)
	at org.eclipse.ui.internal.Semaphore.acquire(Semaphore.java:37)
	- locked <0x5c59e350> (a org.eclipse.ui.internal.Semaphore)
	at org.eclipse.ui.internal.UISynchronizer.syncExec(UISynchronizer.java:42)
	at org.eclipse.swt.widgets.Display.syncExec(Display.java:2647)
	at 
org.eclipse.ui.model.WorkbenchContentProvider.resourceChanged(WorkbenchContentProvider.java:107)
	at org.eclipse.core.internal.events.NotificationManager$2.run(NotificationManager.java:255)
	at org.eclipse.core.internal.runtime.InternalPlatform.run(InternalPlatform.java:610)
	at org.eclipse.core.runtime.Platform.run(Platform.java:521)
	at org.eclipse.core.internal.events.NotificationManager.notify(NotificationManager.java:248)
	at 
org.eclipse.core.internal.events.NotificationManager.broadcastChanges(NotificationManager.java:142)
	at org.eclipse.core.internal.resources.Workspace.broadcastChanges(Workspace.java:155)
	at org.eclipse.core.internal.resources.Workspace.endOperation(Workspace.java:854)
	at org.eclipse.core.internal.events.AutoBuildJob.doBuild(AutoBuildJob.java:144)
	at org.eclipse.core.internal.events.AutoBuildJob.run(AutoBuildJob.java:168)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:62)

"org.eclipse.jdt.internal.ui.text.JavaReconciler" daemon prio=1 tid=0x0c1d7c80 nid=0xc97bc00 in 
Object.wait() [f0a13000..f0a13b20]
	at java.lang.Object.wait(Native Method)
	- waiting on <0x606ceff8> (a org.eclipse.jface.text.reconciler.DirtyRegionQueue)
	at 
org.eclipse.jface.text.reconciler.AbstractReconciler$BackgroundThread.run(AbstractReconciler.java:161)
	- locked <0x606ceff8> (a org.eclipse.jface.text.reconciler.DirtyRegionQueue)

"org.eclipse.jdt.internal.ui.text.JavaReconciler" daemon prio=1 tid=0x0c13e1f0 nid=0xc8f2000 in 
Object.wait() [f058a000..f058ab20]
	at java.lang.Object.wait(Native Method)
	- waiting on <0x5e97d078> (a org.eclipse.jface.text.reconciler.DirtyRegionQueue)
	at 
org.eclipse.jface.text.reconciler.AbstractReconciler$BackgroundThread.run(AbstractReconciler.java:161)
	- locked <0x5e97d078> (a org.eclipse.jface.text.reconciler.DirtyRegionQueue)

"Java indexing" daemon prio=4 tid=0x00563aa0 nid=0x1d24000 in Object.wait() 
[f060b000..f060bb20]
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:429)
	at org.eclipse.jdt.internal.core.search.processing.JobManager.run(JobManager.java:327)
	- locked <0x5da55280> (a org.eclipse.jdt.internal.core.search.indexing.IndexManager)
	at java.lang.Thread.run(Thread.java:552)

"Start Level Event Dispatcher" daemon prio=5 tid=0x00513b00 nid=0x18a5200 in Object.wait() 
[f0509000..f0509b20]
	at java.lang.Object.wait(Native Method)
	- waiting on <0x5d1c0050> (a org.eclipse.osgi.framework.eventmgr.EventThread)
	at java.lang.Object.wait(Object.java:429)
	at org.eclipse.osgi.framework.eventmgr.EventThread.getNextEvent(EventThread.java:167)
	- locked <0x5d1c0050> (a org.eclipse.osgi.framework.eventmgr.EventThread)
	at org.eclipse.osgi.framework.eventmgr.EventThread.run(EventThread.java:118)

"Framework Event Dispatcher" daemon prio=5 tid=0x005116c0 nid=0x182ea00 in Object.wait() 
[f0488000..f0488b20]
	at java.lang.Object.wait(Native Method)
	- waiting on <0x5d1c02b0> (a org.eclipse.osgi.framework.eventmgr.EventThread)
	at java.lang.Object.wait(Object.java:429)
	at org.eclipse.osgi.framework.eventmgr.EventThread.getNextEvent(EventThread.java:167)
	- locked <0x5d1c02b0> (a org.eclipse.osgi.framework.eventmgr.EventThread)
	at org.eclipse.osgi.framework.eventmgr.EventThread.run(EventThread.java:118)

"Signal Dispatcher" daemon prio=10 tid=0x00507fc0 nid=0x1814c00 waiting on condition [0..0]

"Finalizer" daemon prio=8 tid=0x00506ad0 nid=0x180e000 in Object.wait() [f0203000..f0203b20]
	at java.lang.Object.wait(Native Method)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
	- locked <0x5d1c03d8> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x005067a0 nid=0x1802600 in Object.wait() 
[f0182000..f0182b20]
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:429)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:115)
	- locked <0x5d1c00a0> (a java.lang.ref.Reference$Lock)

"main" prio=6 tid=0x00500d90 nid=0xa000a1ec in Object.wait() [bfffb000..bffff548]
	at java.lang.Object.wait(Native Method)
	- waiting on <0x60b33698> (a org.eclipse.core.internal.jobs.Semaphore)
	at org.eclipse.core.internal.jobs.Semaphore.acquire(Semaphore.java:36)
	- locked <0x60b33698> (a org.eclipse.core.internal.jobs.Semaphore)
	at org.eclipse.core.internal.jobs.JobManager.join(JobManager.java:469)
	at org.eclipse.core.internal.jobs.InternalJob.join(InternalJob.java:200)
	at org.eclipse.core.runtime.jobs.Job.join(Job.java:298)
	at org.eclipse.core.internal.events.BuildManager.interrupt(BuildManager.java:335)
	at org.eclipse.core.internal.resources.Workspace.prepareOperation(Workspace.java:1520)
	at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:1554)
	at org.eclipse.ui.actions.WorkspaceModifyOperation.run(WorkspaceModifyOperation.java:105)
	- locked <0x60b336e8> (a org.eclipse.ui.actions.WorkspaceModifyDelegatingOperation)
	at org.eclipse.ui.editors.text.WorkspaceOperationRunner.run(WorkspaceOperationRunner.java:72)
	at org.eclipse.ui.editors.text.WorkspaceOperationRunner.run(WorkspaceOperationRunner.java:62)
	at 
org.eclipse.ui.editors.text.TextFileDocumentProvider.executeOperation(TextFileDocumentProvider.java:
391)
	at 
org.eclipse.ui.editors.text.TextFileDocumentProvider.saveDocument(TextFileDocumentProvider.java:
611)
	at org.eclipse.ui.texteditor.AbstractTextEditor.performSave(AbstractTextEditor.java:3136)
	at 
org.eclipse.jdt.internal.ui.javaeditor.CompilationUnitEditor.performSave(CompilationUnitEditor.java:
1181)
	at org.eclipse.jdt.internal.ui.javaeditor.CompilationUnitEditor.doSave(CompilationUnitEditor.java:
1231)
	- locked <0x606d9068> (a org.eclipse.jdt.internal.core.CompilationUnit)
	at org.eclipse.ui.internal.EditorManager$10.run(EditorManager.java:982)
	at org.eclipse.ui.internal.EditorManager$8.run(EditorManager.java:837)
	at org.eclipse.jface.operation.ModalContext.runInCurrentThread(ModalContext.java:302)
	at org.eclipse.jface.operation.ModalContext.run(ModalContext.java:252)
	at org.eclipse.jface.window.ApplicationWindow$1.run(ApplicationWindow.java:576)
	at org.eclipse.swt.custom.BusyIndicator.showWhile(BusyIndicator.java:84)
	at org.eclipse.jface.window.ApplicationWindow.run(ApplicationWindow.java:573)
	at org.eclipse.ui.internal.WorkbenchWindow.run(WorkbenchWindow.java:1511)
	at org.eclipse.ui.internal.EditorManager.runProgressMonitorOperation(EditorManager.java:843)
	at org.eclipse.ui.internal.EditorManager.savePart(EditorManager.java:987)
	at org.eclipse.ui.internal.WorkbenchPage.savePart(WorkbenchPage.java:2449)
	at org.eclipse.ui.internal.WorkbenchPage.saveEditor(WorkbenchPage.java:2461)
	at org.eclipse.ui.internal.SaveAction.run(SaveAction.java:72)
	at org.eclipse.jface.action.Action.runWithEvent(Action.java:881)
	at org.eclipse.ui.commands.ActionHandler.execute(ActionHandler.java:68)
	at org.eclipse.ui.internal.commands.Command.execute(Command.java:173)
	at org.eclipse.ui.internal.keys.WorkbenchKeyboard.executeCommand(WorkbenchKeyboard.java:
460)
	at org.eclipse.ui.internal.keys.WorkbenchKeyboard.press(WorkbenchKeyboard.java:834)
	at org.eclipse.ui.internal.keys.WorkbenchKeyboard.processKeyEvent(WorkbenchKeyboard.java:875)
	at 
org.eclipse.ui.internal.keys.WorkbenchKeyboard.filterKeySequenceBindings(WorkbenchKeyboard.java:
519)
	at org.eclipse.ui.internal.keys.WorkbenchKeyboard.access$2(WorkbenchKeyboard.java:485)
	at org.eclipse.ui.internal.keys.WorkbenchKeyboard$1.handleEvent(WorkbenchKeyboard.java:258)
	at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:82)
	at org.eclipse.swt.widgets.Display.filterEvent(Display.java:801)
	at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1194)
	at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1219)
	at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1204)
	at org.eclipse.swt.widgets.Control.sendKeyEvent(Control.java:1939)
	at org.eclipse.swt.widgets.Control.sendKeyEvent(Control.java:1934)
	at org.eclipse.swt.widgets.Control.kEventTextInputUnicodeForKeyEvent(Control.java:1472)
	at org.eclipse.swt.widgets.Composite.kEventTextInputUnicodeForKeyEvent(Composite.java:392)
	at org.eclipse.swt.widgets.Widget.textInputProc(Widget.java:1604)
	at org.eclipse.swt.widgets.Display.textInputProc(Display.java:2662)
	at org.eclipse.swt.internal.carbon.OS.SendEventToEventTarget(Native Method)
	at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:1917)
	at org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.java:1488)
	at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:1462)
	at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:259)
	at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:139)
	at org.eclipse.ui.internal.ide.IDEApplication.run(IDEApplication.java:90)
	at org.eclipse.core.internal.runtime.PlatformActivator$1.run(PlatformActivator.java:262)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:213)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:115)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:324)
	at org.eclipse.core.launcher.Main.basicRun(Main.java:315)
	at org.eclipse.core.launcher.Main.run(Main.java:740)
	at org.eclipse.core.launcher.Main.main(Main.java:724)

"VM Thread" prio=5 tid=0x00506020 nid=0x1802e00 runnable 

"VM Periodic Task Thread" prio=10 tid=0x00507f30 nid=0x1814800 waiting on condition 
"Exception Catcher Thread" prio=10 tid=0x00500ef0 nid=0x1802a00 runnable
Comment 1 Andre Weinand CLA 2004-03-23 04:39:37 EST
Created attachment 8775 [details]
full thread dump
Comment 2 Andre Weinand CLA 2004-03-23 04:43:00 EST
I forgt to mention:
After the full rebuild had started in background, I changed a Java file and saved it.
Comment 3 John Arthorne CLA 2004-03-23 10:56:27 EST
Anything in your log file around that time?
Comment 4 Andre Weinand CLA 2004-03-23 11:17:35 EST
No, nothing.
Comment 5 John Arthorne CLA 2004-03-23 15:30:57 EST
The UI thread is trying to modify the workspace, and thus must wait for the
autobuild to complete.  The autobuild has finished, and it is in the middle of
trying to do a syncExec.  The UI thread should be responding to the syncExecs
since we check for this while waiting on the join.  My only possible hunch is
that you are seeing something like bug 55637, where it is not hung, but there is
tons of work happening in syncExecs and the UI thread is processing them all
while waiting for the build to complete.
Comment 6 John Arthorne CLA 2004-03-24 10:57:25 EST
Michael Valenta and I have tracked down a deadlock with a very similar stack
trace during execution of the Team/CVS automated tests. In both cases it looks
on the surface like a classic deadlock

 - UI thread is waiting for a lock (in this case a join)
 - Thread owning the lock is trying to syncExec

The UILockListener/UISynchronizer hooks are designed to solve this deadlock, but
it didn't handle the case where there was a nested wait inside a syncExec:

 1) UI attempts to acquire a lock A
 2) In UILockListener.aboutToWait, the "ui" field is set to indicate that the UI
thread is waiting on a lock.
 3) Thread owning lock A does syncExec 1
 4) UI Thread grants syncExec because it notices that it is waiting for a lock
(ui field is non-null)
 5) syncExec 1 attempts to acquire a different lock B
 6) Lock B is granted in UI thread
 8) When lock B is released, UILockListener.aboutToRelease clears the field
UILockListener.ui, which records the fact that the UI thread is waiting for a lock.
 9) Thread owning lock does another syncExec
 10) UI thread DOES NOT service the syncExec because it believes it is not
currently waiting on a lock. (UILockListener.isUIWaiting()) returns false
because UI field is null).

-> The system deadlocks because the UI thread has "forgetten" that it was
waiting after releasing the nested lock, so it stops servicing syncExecs from
the lock owning thread.
Comment 7 John Arthorne CLA 2004-03-24 10:59:37 EST
Created attachment 8833 [details]
Proposed fix to UILockListener

Proposed fix.  Fix is to re-assign the UILockListener.ui field after servicing
pending work (sync execs), in case the nested execution acquired another lock
and cleared the field in the process. This guarantees that the UI thread knows
it is waiting before it falls asleep on the lock it is waiting for.
Comment 8 Michael Valenta CLA 2004-03-24 11:10:47 EST
I have tried the above fix with the CVS test cases that previously locked 
every time and they ran to completion with no failures.
Comment 9 Tod Creasey CLA 2004-03-24 11:51:36 EST
Do we think this may be the cause of Bug 55714?
Comment 10 Douglas Pollock CLA 2004-03-24 11:53:49 EST
John: Tod just noticed that we've been tracking a similar bug.  The bug number 
is Bug 55714.  We have a lot less specific information than is provided here.  
However, if it is the same bug, we've been seeing it a *lot*. 
Comment 11 John Arthorne CLA 2004-03-24 12:15:02 EST
I don't think so.  I think most of the swirl in that bug is due to bug 55605.
The only stack trace that was attached looked completely different (from Chris,
an SWT hang). 
Comment 12 John Arthorne CLA 2004-03-24 16:29:14 EST
A fix has been released for the next 3.0 M8 integration build (thanks Doug). I
will mark fixed, but please reopen if experienced again on I20040325 or greater