Comment 1 Mike Melick 2011-02-04 17:48:40 EST

Correction: 

My customer is using a version based on Dali Java Persistence Tools
Version: 2.3.1.v201006300000-7N7UF71FD3wTgbUhpcawZT
Build id: 20100701091247 

I did additional testing on: 
(WTP 3.2.3 based) Dali Java Persistence Tools Version:
2.3.3.v201010220000-7N7UF76FD3wTgbVhnkf0cT Build id: 20100915173744

The issue is present in both versions.  The stack traces are similar in both cases.

Comment 2 David Williams 2011-02-07 02:46:26 EST

It might help if you attached complete stack trace ... to see what else was going on. 

Also, was this accessing a remote database? Seems I remember some other bug that showed exceptionally bad UI performance if the data access was slow ... but, of course, no idea if this is similar.

Comment 3 Neil Hauge 2011-02-07 09:50:14 EST

The dependent JAR files are definitely the culprit here.  Brian, can you take a quick look at this.

Comment 4 Neil Hauge 2011-02-08 10:27:13 EST

Just a pre-update on this.  There may not be a "safe" fix for maintenance for this issue, but it needs some additional investigation.  The likely fix is probably in our UI for the Project Explorer.  We need a way to indicate the loading of our model in the JPA Project Content node (without locking up the UI).  I think there are other WTP facets that do something like this.

I assume this JPA project has some referenced JAR's in the persistence.xml.  What type of size are we talking about here?

I'm thinking this is also a pretty rare project scenario wrt the size and scope of referenced JAR's.  We have to scan these referenced jars to look for entities, and depending on the size of the jar, this can take a while.

Comment 5 Mike Melick 2011-02-08 11:35:07 EST

Created attachment 188532 [details]
sample of typical customer persistence.xml

Class names were (obviously) modified, otherwise it's the typical persistence.xml my customer is using.  Depending on the project, they may have more or less elements.

Comment 6 Mike Melick 2011-02-08 12:01:13 EST

(In reply to comment #4)
> Just a pre-update on this.  There may not be a "safe" fix for maintenance for
> this issue, but it needs some additional investigation.  The likely fix is
> probably in our UI for the Project Explorer.  We need a way to indicate the
> loading of our model in the JPA Project Content node (without locking up the
> UI).  I think there are other WTP facets that do something like this.
> 
I understand the need for safety in the maintenance stream, but I do hope there is something that can be done for the customer.  As a general rule long running actions really shouldn't be on the UI thread, as the UI will freeze while waiting. In this case, it was probably anticipated that the operation would be a quick one, which turned out to be slow with this particular workspace. 

If it isn't possible to modify the UI, is there a way to limit the amount of work being done? Is there extra scanning being performed? can some of the work be deferred?  I know.... probably already questions that you've asked of yourself. :) 


> I assume this JPA project has some referenced JAR's in the persistence.xml. 
> What type of size are we talking about here?
> 
No. The customer has several (~7) projects with JPA artifacts in this workspace.  None of these projects have any jars referenced from the persistence.xml file.  I've attached a typical persistence.xml file for reference purposes. My working assumption would be that it is looking on the classpath or a portion of the classpath, specifically the shared libs used by these projects.  The projects have 10s or 100s of Megs of jar files they are referencing as shared libs.  I can get you some exact numbers if need be. 

> I'm thinking this is also a pretty rare project scenario wrt the size and scope
> of referenced JAR's.  We have to scan these referenced jars to look for
> entities, and depending on the size of the jar, this can take a while.
While this workspace seems to take things to an extreme with the amount of shared libs they have, this seems like it could be a more common use case if the classpath is being scanned rather than just jars specified in the persistence.xml file. 



While working with this workspace, I've observed the following (empirically), which raises some thoughts/questions: 

1. The customer uses shared libs with a large number of jars.  When I had the workspace incorrectly setup, and these were not present, the project opened quickly. My working assumption was that these were being scanned. This likely implies at least some of the classpath is being scanned.  Is this necessary? 

2. If two projects both have JPA artifacts, opening the first one after a restart will incur the delay, but opening the second will not.  It appears that JPA artifacts are scanned within the entire workspace.  Is this necessary, or could it be scoped to the individual project? Note: I'm only advocating scoping it down if that's more efficient. It might be quicker/better to do this in one pass. 

3. Visually, there doesn't seem to be much JPA related information shown in the navigator.  What is being looked for on project open? Is is just to populate the navigator, or is there other work that might be put into the background? Can this be read from the persistence.xml, or is scanning of files necessary? 

Would it help if we could reproduce this in a WTP only type environment with sample jars?  Not sure if this is possible, but with what we've seen thus far, we could certainly give it a try.

Comment 7 Troy Bishop 2011-02-08 12:58:34 EST

(In reply to comment #4)

> I assume this JPA project has some referenced JAR's in the persistence.xml. 
> What type of size are we talking about here?
> 
> I'm thinking this is also a pretty rare project scenario wrt the size and scope
> of referenced JAR's.  We have to scan these referenced jars to look for
> entities, and depending on the size of the jar, this can take a while.

I think you are scanning more than just the referenced JAR files from the persistence.xml file.  I think you are scanning all JAR files on the projects build path.

To prove this I created a JPA project using the 'Eclipse IDE for Java EE Developers' download package (from http://www.eclipse.org/downloads/).  On the build path are the JRE and EclipseLink (2.1.1) libraries.  I then created a few (6) "empty" entities, i.e.

<?xml version="1.0" encoding="UTF-8"?>
<persistence version="2.0" xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd">
	<persistence-unit name="Example">
		<class>some.pkg.SomeClass1</class>
		<class>some.pkg.SomeClass2</class>
		<class>some.pkg.SomeClass3</class>
		<class>some.pkg.SomeClass4</class>
		<class>some.pkg.SomeClass5</class>
		<class>some.pkg.SomeClass6</class>
	</persistence-unit>
</persistence>


@Entity
@Table(name = "SOMECLASS1")
public class SomeClass1 {

	@Id
	@GeneratedValue(strategy = GenerationType.AUTO)
	@Column(unique = true, nullable = false)
	private int id;

}

etc.

I then closed the product and re-opened the product.  Expanding this project took a little less than 2seconds (using a stopwatch) so the UI hang was not really noticable.

I then went and retrieved a large amount of JAR files (i.e. spring, hibernate, log4j, apache commons, etc), extracted them all to a temporary directory, and then made multiple copies of these JARs (about 10 copies).  I then added all of these JARs to the Java Build Path of my previously created JPA project, closed the product, restarted it, and expanded the JPA project.  This time it took about 15 seconds and the UI hang was definitely noticable.

In our adopter product we have Java Build Paths that may contain less JARs than this scenario but these JARs definitely contain more classes.  As well, our client had also added a lot of their own custom JAR files.  All of this combined is likely why we are seeing the UI hang for more than 30seconds.

Comment 8 Neil Hauge 2011-02-08 15:02:27 EST

Thanks for the detailed feedback.  This is not as rare of a condition as I originally assumed.  Local JAR's are being scanned when a JPA project is built in an effort to populate our resource model.  Perhaps we could defer and limit the resource model to JAR's that are explicitly referenced by the persistence.xml.  If not, we would need to implement an "Initializing JPT Tooling..." job that will build the projects.

Comment 9 Neil Hauge 2011-02-14 11:48:23 EST

I think we have a decent idea for a patch/maintenance fix.  We were thinking a safe way to resolve this for most users facing this issue would be to have a preference that could enable/disable the inclusion of binary (JAR) based mapping metadata into a project.  The great majority of users probably do not need this functionality, and as a result, it would make sense to have this be an opt-in preference.  For maintenance, in order to avoid changing the default behavior, this preference would be defaulted to enable binary content.  This would be a project level preference.

We would continue to investigate a long term fix for this issue, and upon solving could determine if it is feasible to backport to 2.3 maintenance.

Sound good?

Comment 10 Mike Melick 2011-02-16 11:30:53 EST

(In reply to comment #9)
> I think we have a decent idea for a patch/maintenance fix.  We were thinking a
> safe way to resolve this for most users facing this issue would be to have a
> preference that could enable/disable the inclusion of binary (JAR) based
> mapping metadata into a project.  The great majority of users probably do not
> need this functionality, and as a result, it would make sense to have this be
> an opt-in preference.  For maintenance, in order to avoid changing the default
> behavior, this preference would be defaulted to enable binary content.  This
> would be a project level preference.
> 
> We would continue to investigate a long term fix for this issue, and upon
> solving could determine if it is feasible to backport to 2.3 maintenance.
> 
> Sound good?

The trouble with using a preference is that it isn't intuitively obvious to a customer that they can leverage this preference.  First a customer would have to recognize that they have encountered this specific problem.  Then, find the preference, or be pointed to the preference through some documentation.  Understand if it is safe to disable the preference in their case, and finally actually change the preference for each of their projects. It would help the one specific customer I have now, as I'd tell him about this... but I don't believe it would offer too much help customers in general. 

I know our hands are often somewhat tied in terms of what can and can not be accomplished in a service stream, but are there any options for "fixing" the problem in service rather than "avoiding" the issue via a preference? If you thought about what you wanted to do longer term to address the issue, could you either stage or scale back something to deliver safely in the service stream that would improve the customer experience here?

Comment 11 Neil Hauge 2011-02-16 18:12:29 EST

(In reply to comment #10)

I agree with all of your comments regarding the preference.  It is far from a perfect solution, but offers an extremely low-risk change, compared to what is likely a non-isolated, moderate-risk alternative.  That said, we can certainly proceed with a true fix and determine its suitability for the maintenance stream.

Comment 12 David Williams 2011-02-23 09:30:38 EST

If the "true fix" is not feasible, and we need to go back to think about preferences ... instead of a per project preference (or, perhaps, in addition to) is a better alternative to have a "product preference" so if some adopter products do not need the "scan all jars" functionality, they can "ship" with the default set to "shallow scan". This might still need to be a visible user preference so they could change the default, if needed, for those few users that need it to behave that way.  Its probably over my head, but its unclear to me when the "deep scan" is needed. Is it purely a function of "type of project"? Options in persistence.xml?  or does it depend on having a certain type of runtime available? 

Just a suggestion ... didn't want to get stuck waiting for "best", if "better" is feasible. 

Thanks,

Comment 13 Neil Hauge 2011-02-23 10:40:23 EST

(In reply to comment #12)
> If the "true fix" is not feasible, and we need to go back to think about
> preferences ... instead of a per project preference (or, perhaps, in addition
> to) is a better alternative to have a "product preference" so if some adopter
> products do not need the "scan all jars" functionality, they can "ship" with
> the default set to "shallow scan". This might still need to be a visible user
> preference so they could change the default, if needed, for those few users
> that need it to behave that way.  Its probably over my head, but its unclear to
> me when the "deep scan" is needed. Is it purely a function of "type of
> project"? Options in persistence.xml?  or does it depend on having a certain
> type of runtime available? 
> 
> Just a suggestion ... didn't want to get stuck waiting for "best", if "better"
> is feasible. 
> 
> Thanks,

I think a true fix is feasible, just not easy, and will require a lot of testing.  It will likely be a risky change to make in a maintenance release, but given the severity is likely worth the risk.  Part of the problem right now is simply having the bandwidth to work on the true fix.  We should be getting to this soon, but probably not until next week.

As for a product preference, that would be fine with me if it was something that was desired by an adopter.  I think for future releases we would want to disable the deep scan by default and make it more visible during facet/project configuration.

The deep scan (binary scan) is needed if you want to include binary classes in your persistence.xml or orm.xml.  It is not a particular type of project nor does it require any particular runtime.  This is all just generic JPA functionality.  There is no JPA metadata indication that would tell us that a user wants to reference the available binary content, so a user preference would be the only way to know.  Otherwise, we need to have scanned to properly validate persistence.xml and orm.xml class references.

Comment 14 Neil Hauge 2011-04-28 14:11:07 EDT

We haven't been able to put a fix together for this in the 2.3.4 timeframe.  As a result I am re-targeting to 3.0 so we can work on a fix for Indigo.  It is possible that we could backport this future fix to the 2.3.x stream at a later date.

Comment 15 Neil Hauge 2011-07-01 16:24:48 EDT

Moving JPA specific bugs to new JPA component in bugzilla.

Comment 16 Jesper Skov 2012-01-30 05:59:31 EST

We observe a similar problem, I wonder if it is related.

Some of our projects experience a long (30-40 seconds) hang when refactoring some classes.

I'll try if I can reduce classpath size in these projects to see if it affects the performance.

We use Eclipse 3.7SR1. We had hoped for a solution in SR2, but finding this issue, I now doubt it :(

Comment 17 Neil Hauge 2012-02-01 13:41:25 EST

(In reply to comment #16)
> We observe a similar problem, I wonder if it is related.
> 
> Some of our projects experience a long (30-40 seconds) hang when refactoring
> some classes.
> 

This sounds like a different issue.  This issue is primarily related to an initial scan that is needed to build the JPA project model.  A refactoring wouldn't require this type of build.  I would like to know more about this issue though, which is likely unrelated.  It would be great if you could open a new bug against Dali and attach a thread dump during the time you experiencing the hang.

http://wiki.eclipse.org/How_to_report_a_deadlock

Comment 18 Jesper Skov 2012-02-02 02:33:40 EST

Reducing classpath size did not help, as you expected :)

I'll make a new issue with stack dump later today.

Comment 19 Karen Butzke 2012-04-25 17:24:28 EDT

Brian has done work in Dali 3.2 (WTP 3.4) to build JPA projects on a background thread instead of the UI thread. It still locks the workspace, but the UI will not hang and you will see the 'Building JPA projects' process running.

I have been doing some performance profiling for this particular use case and have uncovered a change that cuts the time building the JPA project down by at least 50% if there are large jars in the project. It is still expensive to scan all the classes in the jars for JPA annotations, but the time is now in JDT code if those classes are not annotated. This particular fix can probably be backported to the 3.0 and 2.3 streams. So, at least the building time would be less, but the locking up of the UI would still exist in those streams. That change is too large to be backported to maintenance.

I have checked the change in for 3.2M7 and will create patches for the 2.3 and 3.0 streams. I will enter another bug for the preference to have Dali not scan jars in the project.

Comment 20 Troy Bishop 2012-04-25 17:47:14 EDT

(In reply to comment #19)
> Brian has done work in Dali 3.2 (WTP 3.4) to build JPA projects on a background
> thread instead of the UI thread. It still locks the workspace, but the UI will
> not hang and you will see the 'Building JPA projects' process running.
> 

That's great progress, but locking the entire workspace is still not ideal so if at all possible, can the locking be finer grained (see bug 198591)?  The reason I say this is because there are a lot of other actions that lock on the workspace and can lead to a UI hang until your builder completes (i.e. if code executing in the UI thread attempts to lock on the workspace).  The change you mentioned to reduce the build time by approximately 50% will certainly help this situation... but I wanted you to be aware of the consequences of locking on the workspace and, if at all possible, change to a finer grained lock.

Comment 21 Brian Vosburgh 2012-05-10 13:14:21 EDT

(In reply to comment #20)
> That's great progress, but locking the entire workspace is still not ideal so
> if at all possible, can the locking be finer grained (see bug 198591)?

It is likely we can change the Dali startup code to lock on each project as Dali processes that particularly project, instead of locking the entire workspace while Dali processes every project. But that change is too risky for Juno. But it also looks reasonable for a maintenance release. We have to do a bit more research.

I am closing this bug as fixed; as Dali no longer locks the UI. Please open another bug if you would like us to look into finer-grained locking during start-up; and, if possible, provide a relevant use case that would demand the change.