Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 375601

Summary: New deadlock in indexer job : PDOMManager.createIndexer() vs PDOMManager.savePreferences()
Product: [Tools] CDT Reporter: Andrey Loskutov <loskutov>
Component: cdt-indexerAssignee: Markus Schorn <mschorn.eclipse>
Status: RESOLVED FIXED QA Contact: Markus Schorn <mschorn.eclipse>
Severity: major    
Priority: P3 CC: cdtdoug, malaperle, mober.at+eclipse, pawel.1.piech, yevshif
Version: Next   
Target Milestone: 8.0.3   
Hardware: PC   
OS: All   
Whiteboard:
Bug Depends on: 359485    
Bug Blocks:    
Attachments:
Description Flags
Full thread dump none

Description Andrey Loskutov CLA 2012-03-29 03:24:05 EDT 
Build Identifier: 8.0.2

We've observed deadlock in indexer code: PDOMManager.createIndexer() vs PDOMManager.savePreferences().

Looking into the code the deadlock might be the side effect of the fix of the another deadlock in the bug 359485.

Found one Java-level deadlock:
=============================
"Worker-7":
  waiting to lock monitor 0x0000000044e06e20 (object 0x00002aaae369ecc0, a org.eclipse.core.internal.preferences.InstancePreferences),
  which is held by "Worker-4"
"Worker-4":
  waiting to lock monitor 0x00002aab2402e1b8 (object 0x00002aaae3694060, a org.eclipse.core.internal.resources.ProjectPreferences),
  which is held by "Worker-7"

Java stack information for the threads listed above:
===================================================
"Worker-7":
    at org.eclipse.core.internal.preferences.EclipsePreferences.getChild(EclipsePreferences.java:487)
    - waiting to lock <0x00002aaae369ecc0> (a org.eclipse.core.internal.preferences.InstancePreferences)
    at org.eclipse.core.internal.preferences.EclipsePreferences.internalNode(EclipsePreferences.java:629)
    at org.eclipse.core.internal.preferences.EclipsePreferences.node(EclipsePreferences.java:757)
    at org.eclipse.cdt.internal.core.pdom.indexer.IndexerPreferences.getLocalPreferences(IndexerPreferences.java:311)
    at org.eclipse.cdt.internal.core.pdom.indexer.IndexerPreferences.getScope(IndexerPreferences.java:93)
    at org.eclipse.cdt.internal.core.pdom.indexer.IndexerPreferences.getProperties(IndexerPreferences.java:167)
    at org.eclipse.cdt.internal.core.pdom.PDOMManager.createIndexer(PDOMManager.java:611)
    - locked <0x00002aaab7cba8e0> (a java.util.HashMap)
    - locked <0x00002aaae3694060> (a org.eclipse.core.internal.resources.ProjectPreferences)
    at org.eclipse.cdt.internal.core.pdom.PDOMSetupJob.run(PDOMSetupJob.java:58)
    at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)
"Worker-4":
    at org.eclipse.core.internal.preferences.EclipsePreferences.getChild(EclipsePreferences.java:488)
    - waiting to lock <0x00002aaae3694060> (a org.eclipse.core.internal.resources.ProjectPreferences)
    at org.eclipse.core.internal.preferences.EclipsePreferences.getChildren(EclipsePreferences.java:510)
    at org.eclipse.core.internal.preferences.EclipsePreferences.shareStrings(EclipsePreferences.java:1091)
    at org.eclipse.core.internal.preferences.EclipsePreferences.shareStrings(EclipsePreferences.java:1094)
    at org.eclipse.core.internal.preferences.EclipsePreferences.shareStrings(EclipsePreferences.java:1094)
    at org.eclipse.core.internal.preferences.EclipsePreferences.shareStrings(EclipsePreferences.java:1094)
    at org.eclipse.core.internal.preferences.PreferencesService.shareStrings(PreferencesService.java:949)
    at org.eclipse.core.internal.preferences.EclipsePreferences.convertToProperties(EclipsePreferences.java:349)
    at org.eclipse.core.internal.preferences.EclipsePreferences.convertToProperties(EclipsePreferences.java:347)
    at org.eclipse.core.internal.preferences.EclipsePreferences.save(EclipsePreferences.java:1068)
    at org.eclipse.core.internal.preferences.EclipsePreferences.save(EclipsePreferences.java:1057)
    at org.eclipse.core.internal.preferences.EclipsePreferences.internalFlush(EclipsePreferences.java:440)
    at org.eclipse.core.internal.preferences.EclipsePreferences.flush(EclipsePreferences.java:405)
    - locked <0x00002aaae369ecc0> (a org.eclipse.core.internal.preferences.InstancePreferences)
    at org.eclipse.cdt.internal.core.CCoreInternals$1.run(CCoreInternals.java:43)
    at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)

Found 1 deadlock.

Reproducible: Sometimes

Steps to Reproduce:
We've seen this twice in our automatic unit tests.
Comment 1 Andrey Loskutov CLA 2012-03-29 03:25:00 EDT 
Created attachment 213317 [details]
Full thread dump
Comment 2 Marc-André Laperle CLA 2012-03-29 12:20:53 EDT 
Are you running 3.7.1 or 3.7.2? The fix in bug 359485 was to work around Bug 359698 which got fixed in 3.7.2.
Comment 3 Andrey Loskutov CLA 2012-03-29 12:36:58 EDT 
CDT 8.0.2 on Eclipse 3.7.2
Comment 4 Andrey Loskutov CLA 2012-03-29 13:09:48 EDT 
I've just downloaded 3.7.2 classic package and still see "synchronized flush()" in the sources of org.eclipse.equinox.preferences/src/org/eclipse/core/internal/preferences/EclipsePreferences.java

	/*
	 * @see org.osgi.service.prefs.Preferences#flush()
	 */
	synchronized public void flush() throws BackingStoreException {
		internalFlush();
	}

Looks like the patch for bug 359698 was never properly integrated/propagated to 3.7.2 stream.
Comment 5 Andrey Loskutov CLA 2012-03-30 11:11:17 EDT 
Please check bug 333726 comment 27 for the discussion about possible solution.
Comment 6 Markus Schorn CLA 2012-04-04 11:28:07 EDT 
The problem should go away by removing the workaround that shall not be necessary any more.
Comment 7 Andrey Loskutov CLA 2012-04-04 12:58:25 EDT 
(In reply to comment #6)
> The problem should go away by removing the workaround that shall not be
> necessary any more.

*Should* ? 

Did you removed the code (please point me to the commit) OR do you suggest me to try with the removing the workaround by myself? What was exactly in the workaround: bug 359485 comment 8 contains links to git commits - is it all what we have to remove, or anything else?

Thanks!
Comment 8 Markus Schorn CLA 2012-04-05 01:51:56 EDT 
(In reply to comment #7)
I have removed the workaround.
Comment 9 Markus Schorn CLA 2012-04-05 03:05:57 EDT 
The workaround has been put back in, because the platform issue is not yet fixed in eclipse 3.8.
Comment 10 Martin Oberhuber CLA 2012-04-06 06:32:07 EDT 
So what particular Platform problem does this depend on now - bug 376206 or anything else ?
Comment 11 Markus Schorn CLA 2012-04-06 07:06:49 EDT 
(In reply to comment #10)
> So what particular Platform problem does this depend on now - bug 376206 or
> anything else ?

That's unclear (at least to me) at the moment, see 
http://dev.eclipse.org/mhonarc/lists/cdt-dev/msg24178.html
Comment 12 Markus Schorn CLA 2012-04-10 10:19:43 EDT 
Workaround for bug 359485 is removed again and tested with platform 3.8m6 and 3.7.2.

This bug occurs with CDT 8.0.2 (which contains the workaround for 359485) and platform 3.7.2 (which contains the fix for 359485).
Comment 13 CDT Genie CLA 2013-02-27 11:39:00 EST 
*** cdt git genie on behalf of Markus Schorn ***

    Bug 375601: Remove workaround for bug 359485.

[*] http://git.eclipse.org/c/cdt/org.eclipse.cdt.git/commit/?id=61f66feccebe3db1379669b8b53914690f4c9394
Comment 14 CDT Genie CLA 2013-02-27 11:39:11 EST 
*** cdt git genie on behalf of Markus Schorn ***

    Bug 375601: Remove workaround for bug 359485.

[*] http://git.eclipse.org/c/cdt/org.eclipse.cdt.git/commit/?id=9c2b66cf03f5423013365c0db8890a801b45a876
Comment 15 CDT Genie CLA 2013-02-27 11:39:16 EST 
*** cdt git genie on behalf of Markus Schorn ***

    Bug 375601: Remove workaround for bug 359485.

[*] http://git.eclipse.org/c/cdt/org.eclipse.cdt.git/commit/?id=b2a57f053d773794a78c38f91dff405e8c54401f