Description Thomas Watson 2011-10-20 08:26:55 EDT

Investigate for possible 3.7.2 backport.

+++ This bug was initially created as a clone of Bug #359535 +++

Build Identifier: 3.6.0.v20010613

I have come across a deadlock situation during the startup phase of our application that is caused by an interaction between Spring OSGi and Equinox. As far as I can tell from browsing through the Gemini Blueprint SVN repository the same situation also applies to Gemini Blueprint.

I will attach an annotated stacktrace of the two blocking threads. The numbers and letters in the following description refer to my annotations in the stacktrace.

During startup of the system the spring osgi extender tries to extend two bundles at the same time:

Thread [SpringOsgiExtenderThread-10]
1) starts refreshing Applicationcontext for business bundle A. 
2) This causes the loading of a bean which represents an OSGi service reference. Therefore an OsgiServiceProxyFactoryBean tries to create a proxy object for this service.
3) As a consequence of this a ServiceDynamicInterceptor is created which triggers loading of class org.springframework.osgi.service.importer.support.internal.aop.SwappingServiceReferenceProxy.
4) While loading this class a lock for the ClassLoader of spring-osgi-core is acquired.
5) During the class loading cascade another class from the same bundle is loaded (org.springframework.osgi.service.importer.support.internal.aop.BaseServiceReferenceProxy), which triggers
6) a reentry into the lock for the classloader, which does not do any harm.
7) When the class is found all ClassLoadingStatsHooks are called. One of them is the WeavingHookConfigurator. This causes
8) Access to the ServiceRegistry, which tries to
9) Lookup service registrations. The access to the service registry is synchronized. So SpringOsgiExtenderThread-10 blocks waiting for the release of the lock by the other thread.


SpringOsgiExtenderThread-20
a) tries to complete refreshing the application context for business bundle B. 
b) In the process of doing this a bean is found that represents a service registration <osgi:service .....> 
c) Since all preconditions are satisfied the service registration can take place. For this purpose a lock on the ServiceRegistry is acquired.
d) When initializing the ServiceProperties there is an iteraton over all keys in the Dictionary of properties used when registering the service. Spring OSGi and Gemini Blueprint does not use a JDK Dictionary class but a subclass org.springframework.osgi.util.internal.MapBasedDictionary. This class has been loaded before. 
6) But when calling the keys() method the inner class IteratorBasedEnumeration is used for the first time. This triggers class loading for this inner class. Classloading would require the lock on the class loader of spring-osgi-core, which is held by the other thread.


We did not see a really easy solution to this problem so far. One idea could be to reorder the sequence of steps in ServiceRegistrationImpl.register(...). Maybe it is possible to call createProperties() without holding the lock on the ServiceRegistry. But we are not aware of all side effects this might have.

Reproducible: Sometimes