Bug 330505

Description John Liu 2010-11-17 13:58:48 EST

Build Identifier: 

I have ever experienced indexing deadlock in my testing, although it happens very rarely, but its consequence is very serious - the indexing freezes and RDp loses all of index related functions.

So I spent time to investigate it, I utilize break point to help me to produce a deadlock scenario, as the following steps:

1, Set break point at findCalledBy() function right after it acquires a read lock of the project index, run get call hierarchy, the process stops after it acquires a read lock of the project index.

2, Runs reindex, it will first do the index clearing up, which will need to acquire a write lock of the project index.

3, Reindex will not run until the first call hierarchy dstore command completes, this is good because it helps to keep a sequential command sequence, which prevent deadlock happens. 

4, But if the result of get call hierarchy needs to get its children's caller or callee, it will execute findCalledBy() again and this time, it will breaks the sequence of reindexing command and get call hierarchy command. As a result, we can produce a deadlock like this:

a    findCalledBy() acquires a read lock of the project index.
b    reindex clearing up step tries to acquire a write lock of the project index.The reindex process is waiting for write lock here....
c    findCalledBy() finishes the job and tries to release its read lock of the project index, but it won't be able to, so it is waiting here...    
          
See the deadlock happens here and no more index function is able to work properly anymore, the server side indexing process is hanging and even killing server won't help to kill them.

We can see the problem is in step c, in which findCalledBy() should be able to release the read lock. But it is strangely hanging there. 

After spending some time in debugging it, I find the problem is because CDT CIndex has a double synchronize control, Itself has synchronized lock acquiring and releasing function and its underneath fragment (PDOM) also synchronize an object to handle the sequence of acquiring and releasing locks. So this double synchronize control helps to create the above dealock, like this way:

   *  b step reindex calls acquiring write lock before c step findCalledBy() calls releasing read lock, so b step runs first before c step controlled by CIndex level.
   * but b step waits c step to release the lock to complete, which controlled by PDOM level.
   
I then notice that CDT always create a different CIndex to be used by PDOMManager for an index function, see IndexFactory class. Then it turns out that the solution for it is fairly simple, when RemoteIndexerManager gets an index for a scope, it should also create a new CIndex class wrapper for its PDOM fragment like CDT does, this will then breaks the synchronize control in CIndex level, since index in step b and step c will be two different objects(note only CIndex wrapper objects are different, their underneath PDOM fragments are still same). By this fix, the step c will be able to release the read lock and then step b will acquire the write lock after that.


Reproducible: Always