Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 353054

Summary: Installation of plan 'org.eclipse.virgo.kernel.userregion.springdm' version '3.0.0' failed. java.util.ConcurrentModificationException: null
Product: [RT] Virgo Reporter: Eric Hough <k2eric>
Component: runtimeAssignee: Project Inbox <virgo-inbox>
Status: CLOSED FIXED QA Contact:
Severity: normal    
Priority: P3 CC: glyn.normington
Version: unspecified   
Target Milestone: 3.5.0.RELEASE   
Hardware: PC   
OS: Linux   
Whiteboard:
Bug Depends on: 353103, 358101    
Bug Blocks:    
Attachments:
Description Flags
stack trace observed at startup
none
Equinox fix suitable for Virgo 3.0.0.RC1 none

Description Eric Hough CLA 2011-07-25 20:11:11 EDT
Build Identifier: 3.0.0.RC1

Virgo Kernel throws this error on startup when loaded with a certain set of bundles in repository/usr. It's almost definitely related to com.springsource.org.hibernate.ejb-3.4.0.GA-A.jar, but the way Virgo handles the problem appears to be a bug.

It's easy to reproduce...

Reproducible: Always

Steps to Reproduce:
1. Unzip fresh copy of Virgo Kernel 3.0.0.RC1
2. Load up the following bundles into repository/usr. All can be pulled from Spring's EBR.

com.springsource.antlr-2.7.7.jar			   
com.springsource.javassist-3.10.0.GA.jar		   
com.springsource.javax.jms-1.1.0.jar			   
com.springsource.javax.persistence-2.0.0.jar		   
com.springsource.javax.servlet-2.5.0.jar		   
com.springsource.net.sf.cglib-2.2.0.jar			   
com.springsource.net.sf.ehcache-2.0.0.jar		   
com.springsource.org.apache.commons.codec-1.4.0.jar	   
com.springsource.org.apache.commons.collections-3.2.1.jar  
com.springsource.org.apache.commons.io-1.4.0.jar		   
com.springsource.org.apache.commons.lang-2.4.0.jar		   
com.springsource.org.codehaus.btm-1.3.3.jar			   
com.springsource.org.dom4j-1.6.1.jar				   
com.springsource.org.hibernate-3.3.2.GA.jar			   
com.springsource.org.hibernate.annotations-3.4.0.GA.jar	   
com.springsource.org.hibernate.annotations.common-3.3.0.ga.jar
com.springsource.org.hibernate.ejb-3.4.0.GA-A.jar		   
com.springsource.org.objectweb.asm-1.5.3.jar			   
com.springsource.org.xmlpull-1.1.4.c.jar
org.springframework.context.support-3.0.5.RELEASE.jar
org.springframework.data.spring-data-commons-core-1.1.0.M2.jar
org.springframework.data.spring-data-jpa-1.0.0.RC1.jar
org.springframework.jdbc-3.0.5.RELEASE.jar
org.springframework.orm-3.0.5.RELEASE.jar
org.springframework.transaction-3.0.5.RELEASE.jar
org.springframework.web-3.0.5.RELEASE.jar
org.springframework.web.servlet-3.0.5.RELEASE.jar

3. Start server. Observe stack trace and failure to start.

4. Remove repository/usr/com.springsource.org.hibernate.ejb-3.4.0.GA-A.jar then start the server again (after cleaning). Notice server starts successfully.
Comment 1 Eric Hough CLA 2011-07-25 20:12:02 EDT
Created attachment 200317 [details]
stack trace observed at startup
Comment 2 Glyn Normington CLA 2011-07-26 06:55:02 EDT
Thanks for raising. It looks like an Equinox resolver bug, but I'd like to recreate the behaviour to investigate further.

The following are not available from the SpringSource EBR. Please could you provide links to the versions you used:

org.springframework.data.spring-data-commons-core-1.1.0.M2.jar
org.springframework.data.spring-data-jpa-1.0.0.RC1.jar

Also, I noticed that you were not using the latest version (3.4.0.GA-A) of com.springsource.org.hibernate.annotations which matches version 3.4.0.GA-A of com.springsource.org.hibernate.ejb, so you should probably upgrade that dependency. See bug 335174 for background.
Comment 3 Glyn Normington CLA 2011-07-26 08:27:31 EDT
I found the missing JARs on Amazon S3 rather than the EBR web app.
Comment 4 Glyn Normington CLA 2011-07-26 08:54:36 EDT
(In reply to comment #0)
> Steps to Reproduce:
> 1. Unzip fresh copy of Virgo Kernel 3.0.0.RC1
> 2. Load up the following bundles into repository/usr...

For convenience, a zip of the usr directory is available at:

http://dl.dropbox.com/u/2487064/bug353054-repository-usr.zip

(I will delete this file once this bug has been closed)

> 3. Start server. Observe stack trace and failure to start. 
> 4. Remove repository/usr/com.springsource.org.hibernate.ejb-3.4.0.GA-A.jar then
> start the server again (after cleaning). Notice server starts successfully.
Comment 5 Glyn Normington CLA 2011-07-26 08:57:17 EDT
(In reply to comment #2)
> Also, I noticed that you were not using the latest version (3.4.0.GA-A) of
> com.springsource.org.hibernate.annotations which matches version 3.4.0.GA-A of
> com.springsource.org.hibernate.ejb, so you should probably upgrade that
> dependency. See bug 335174 for background.

Using the later version makes no difference. The resolver fails with the same symptoms.
Comment 6 Glyn Normington CLA 2011-07-26 10:48:20 EDT
Created attachment 200363 [details]
Equinox fix suitable for Virgo 3.0.0.RC1
Comment 7 Glyn Normington CLA 2011-07-26 10:52:14 EDT
The problem was a regression in Equinox and the official fix will appear in due course.

As a temporary workaround for Virgo RC1, please see the attachment https://bugs.eclipse.org/bugs/attachment.cgi?id=200363 and copy it over both (bug 353122 !) copies of Equinox inside Virgo:

find . -name "org.eclipse.osgi-*.jar" -exec cp org.eclipse.osgi_3.7.0.v20110224Bug353103Fix.jar {} \;

(or something like that, depending on your OS).

I did this and the kernel came up ok.
Comment 8 Glyn Normington CLA 2011-07-26 11:05:40 EDT
Targeting next release after 3.0 as we will need to pick up Equinox 3.7.1 which is due in September 2011.
Comment 9 Eric Hough CLA 2011-07-26 11:23:14 EDT
Some initial testing on my end shows that your patch works. I copied [1] to lib/org.eclipse.osgi-3.7.0.v20110224.jar and the kernel comes up properly. I'll continue to test and report back here if it comes up again, but I'm feeling optimistic!

Two lingering questions in my mind..

 1. How did you come up with this fixed copy of Equinox? I'm assuming you must have built it yourself, as it seems to me that ResolverBundle.java still uses an ArrayList<ResolverBundle> as the type of the fragments variable.

 2. Why is Virgo even looking in repository/usr? There's nothing in pickup, so as a user I would expect that Virgo should be able to load itself from lib and repository/ext.

Thank you very much for your prompt attention to this bug, and especially for providing me with a fix so I can keep moving forward in my work!

[1] https://bugs.eclipse.org/bugs/attachment.cgi?id=200363
[2] http://dev.eclipse.org/viewcvs/viewvc.cgi/org.eclipse.equinox/framework/bundles/org.eclipse.osgi/resolver/src/org/eclipse/osgi/internal/module/ResolverBundle.java?view=markup&root=RT_Project
Comment 10 Glyn Normington CLA 2011-07-26 11:46:11 EDT
(In reply to comment #9)
> Some initial testing on my end shows that your patch works. I copied [1] to
> lib/org.eclipse.osgi-3.7.0.v20110224.jar and the kernel comes up properly. I'll
> continue to test and report back here if it comes up again, but I'm feeling
> optimistic!

Good to hear.

> 
> Two lingering questions in my mind..
> 
>  1. How did you come up with this fixed copy of Equinox? I'm assuming you must
> have built it yourself, as it seems to me that ResolverBundle.java still uses
> an ArrayList<ResolverBundle> as the type of the fragments variable.

Tom Watson built that fixed version relative to the version of Equinox in Virgo RC1.

(He also noticed that we had failed to upgrade to the stable Indigo version, so I raised bug 35319 to cover this.)

> 
>  2. Why is Virgo even looking in repository/usr? There's nothing in pickup, so
> as a user I would expect that Virgo should be able to load itself from lib and
> repository/ext.

Great question! That's because Virgo has a kernel, which does *not* depend on the contents of repository/usr (or of repository/ext for that matter), and a user region where applications are deployed. The basic content of the user region provided with Virgo has its dependencies resolved from the Virgo repository chain which is typically repository/ext followed by repository/usr.

So, for example, a fresh install of the Virgo kernel deploys Spring DM together with the parts of the Spring framework that Spring DM needs into the user region. If the user wants to provide optional dependencies of the Spring framework, they add them to repository/usr. When the kernel is started, these optional dependencies need resolving as there can only be a single instance of Spring framework at a given version in the user region and that instance needs those optional dependencies satisfying so the user can exploit them via Spring.

Also, please bear in mind that the repository chain is totally configurable, so the user could do away with repository/ext and repository/usr and have a different collection of watched, external, and/or remote repositories, provided the basic set of bundles normally in repository/ext is available in the chain.

> 
> Thank you very much for your prompt attention to this bug, and especially for
> providing me with a fix so I can keep moving forward in my work!

You're welcome.

> 
> [1] https://bugs.eclipse.org/bugs/attachment.cgi?id=200363
> [2]
> http://dev.eclipse.org/viewcvs/viewvc.cgi/org.eclipse.equinox/framework/bundles/org.eclipse.osgi/resolver/src/org/eclipse/osgi/internal/module/ResolverBundle.java?view=markup&root=RT_Project
Comment 11 Glyn Normington CLA 2011-07-26 11:53:33 EDT
Another point: I am aware that we will not be able to fix this bug in 3.0.0.RELEASE. If this is an issue, users may like to use the available commercial support ([1]). Meanwhile, replacing Equinox with the attached version or a 3.7.1 build which contains the fix to bug 353102 (tomorrow's integration build, for example) should do the trick.

[1] http://www.eclipse.org/virgo/support/
Comment 12 Glyn Normington CLA 2011-09-19 08:29:36 EDT
Tested with Equinox maintenance build org.eclipse.osgi_3.7.1.R37x_v20110808-1106.jar and this resolves the problem.  Once we have upgraded Equinox to 3.7.1 or later (see bug 358101), we can close this bug.
Comment 13 Glyn Normington CLA 2011-12-06 04:56:18 EST
Fixed now that Equinox is upgraded to 3.7.1.R37x_v20110808-1106 by bug 358101.