Bug 312462

Description Desmond Ng 2010-05-11 12:35:32 EDT

Build Identifier: Build id: 20090920-1017


A symptom of this issue is database locks (not deadlocks) on the database are not released.  Successive thread dump of shows three threads blocking many other threads, and the three blocking threads are at the same spot:
 
      java.lang.Thread.State: WAITING (on object monitor)
             at java.lang.Object.wait(Native Method)
             at java.lang.Object.wait(Object.java:485)
             at org.eclipse.persistence.internal.helper.ConcurrencyManager.acquire(ConcurrencyManager.java:89)
             - locked <0x738ef1f8> (a org.eclipse.persistence.internal.helper.ConcurrencyManager)
             at org.eclipse.persistence.internal.identitymaps.CacheKey.acquire(CacheKey.java:116)
         ...
 
There are about three other idle threads also stuck at ConcurrencyManager.java:89.
 
About two weeks ago, we also encountered another DB Lock, but that dump showed one thread stuck at ConcurrencyManager.releaseDeferredLock()
 
 
 "AJPRequestHandler-RMICallHandler-271" prio=10 tid=0x4e91a800 nid=0x5916 sleeping[0x4bbfe000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
         at java.lang.Thread.sleep(Native Method)
         at org.eclipse.persistence.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:429)
         at org.eclipse.persistence.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:413)
         at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:742)
         at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneNormally(ObjectBuilder.java:547)
 
 
Not sure if both are occurrence are related, but it results in the JVM being useless when it comes to using EclipseLink, as more threads are blocked at:
 
"AJPRequestHandler-RMICallHandler-303" prio=10 tid=0x08485c00 nid=0x1067 waiting for monitor entry [0x4c2e1000]
      java.lang.Thread.State: BLOCKED (on object monitor)
             at org.eclipse.persistence.internal.identitymaps.WeakIdentityMap.checkCleanup(WeakIdentityMap.java:94)
             - waiting to lock <0x7a357f40> (a org.eclipse.persistence.internal.identitymaps.SoftCacheWeakIdentityMap)
             at org.eclipse.persistence.internal.identitymaps.WeakIdentityMap.getCacheKeyIfAbsentPut(WeakIdentityMap.java:83)
         ...
 
 
Killing the database to release the locks from the database did not resolve the problem. 

Gordon Yorke analyzed the thread dump and said:

> ----- Original Message -----
> From: gordon.yorke@oracle.com
> ...
> Sent: Thursday, May 6, 2010 10:51:19 AM GMT -06:00 US/Canada Central
>
> It appears that EclipseLink may have a bug.  Although I am unable to 
> pinpoint the cause using the thread dumps from looking at the code I so 
> see a potential issue that would produce similar issue.  A bug should be 
> filed for this issue.  Unfortunately there does not appear to be a 
> workaround for the issue.
> --Gordon

He also mentioned that

> On May 6, 2010, at 1:25 PM, Gordon Yorke <gordon.yorke@oracle.com>  
> wrote:
>
> Yes, this issue is present in all  of the streams.  It is caused  
> when a thread has acquired a readlock on a cacheKey and another  
> thread attempts to update the cache.  Although the issue reported  
> [...] may be hard to pinpoint fixing this code will solve this  
> deadlock as well.
> --Gordon


Reproducible: Sometimes

Steps to Reproduce:
N/A