Community
Participate
Working Groups
Build Identifier: 3.2.1 JSPIndexManager and JSIndexManager consume too much CPU during post-build. Inside my development environment, it takes 3 minutes that all both IndexManager finish their work. I did some profiling which proves - where the CPU time is consumed - what object allocations are made. - high gc activity Reproducible: Always
Created attachment 191698 [details] profiling results which proves the performance problem Please have a look at the attachment. It contains the result of my profiling session which has been started right after the build finished (according to the eclipse job monitor) and has been stopped as soon as both index manager finished their job. It contains: - cpu profiling of JSIndexManager and JSPIndexManager - object allocation profiling - CPU load and CPU load required by Garbage collector - thread graph - thread dump
Patric, The operation of going through every resource change event, translating the relivant documents to Java (for JSPs) or JS (for script regions in HTML/JSPs) is an expensive operation but it is needs to happen for each resource. I am curious you are saying this is happening 'post-build' do you mean after each build the indexer managers are taking 3 minutes to process changes? Or do you mean the first time you build a workspace with a bunch of imported files it takes that long? If this is happening on every build, does your build create a bunch of new resources that would then be indexed? If these are derived resources then that issue is being handled by bug 340647. If this is only happening after an import of a large amount of files then it is expected due to the amount of work that needs to be done to index a large batch of files.
I don't think this meets the conditions of Critical, which is "crashes, loss of data, severe memory leak." Downgrading to Major and hopefully we can get more information from Ian's questions.
Hi Ian, with 'post-build' I mean that the build seems to be finished (according to the job/progress manager in the bottom right of eclipse), but there are still jobs to finish. Wouldn't it be a good idea to display them in the progress manager, especially when these are an expensive operation? I am sure that many users aren't aware that why some jobs consume significant cpu time and slows down other things inside and outside eclipse. I did the profiling because I haven't any idea which benefit I get from them (All "build" validators and the "JSP java search" are disabled in the prefrence dialog). Because of that, I am wondering what makes them "expensive" and if they really can't be optimized or even disabled at all. (and to allow me a comparison: 2000 java classes are compiled faster on my machine then to index 200 jsps) They create an enormous amount of objects which causes high garbage collection load. I am especially talking of a "clean/full build" because this is what we often have to do. (BTW: How to configure WTP 3.2.x that all validators/builders will only be invoked on an incremental build? In erlier releases, I was sufficent so configure the validators to run on a "manual build", but this changed) The profiled build did not create any derived resources.
Patric, The JSPIndexManager and JSIndexManager are not directly related to a build operation which is why their status is not part of a build operation. The only way in which they are related is if the build created some new resources that need to be indexed. We do not display these jobs in the progress manager by default (you can see them if you turn on system jobs), because they are background running system jobs that can not be canceled so there is no reason to bother the user with them. While currently the implementation of these two managers is very different their two jobs are very similar. They are different because the JSPIndexManager recently went through a major overhaul and the JSIndexManager has yet to get the same treatment. The job of both of them is this: 1. listen for resource change events 2. if the resource change event relates to the type of files the manager cares about then put it in a queue to processes 3. process the queue of changes, which includes translating the file into the correct language, Java or JS as the case may be, then inject these translated files into the respective Java and JS indexes. It is that third step that is expensive. To open up HTML/JSP documents looking for Java and JS regions and then parsing those regions out, translating them into valid Java and JS documents that can then be given to the respective indexes. What this buys us now all of the embedded Java and JS in HTML/JSP documents is now indexed so any operation on the respective indexes will now handle the contents of these documents. This means embedded JS and Java gets included in search and refactoring operations. If this was not done and a user performed a search or refactoring operation then the embedded JS and Java could then possibly be broken by the changes. It is not out of the realm of possibilities to look into ways to make the managers back off when there is other stuff going on or possibly to have an option to completely turn them off at the loss of functionality. But in both of theses cases these would be enhancements and should not be categorized as critical or major. If you see these index managers kicking off (as apposed to already running) after every build and NO resources are changing because of the build then there is a bug here of some sort.
The fix for bug 340647 should help a little. While it won't reduce the cost of indexing a file, it will reduce the number of files considered for indexing, and possibly the number actually indexed.
Ian, thank you for your detailed answer. Please note that I cannot use the refactoring of JSPs by java refactoring actions anyway - because of several WTP bugs related with jsp fragments (e.g. bug 155664 ). How can I make those "system jobs" visible? I will do some more tests to identify what exactly those two jobs are indexing during build. BTW, you missed my (slightly off-topic) question in my last comment: How to configure WTP 3.2.x that all validators/builders will only be invoked during an incremental build? In erlier releases it was sufficent to configure the validators to run on a "manual build". But this doesn't work anymore.
Turning off automatic builds and setting all validators to manual only (double-check whether there are project-specific settings) should handle that. Indexing falls under neither of these categories. If you see something else happening, that could be a bug in whatever it is that you're seeing running.
Nitin, I verified the problem and filed bug 340743. Thanks.
I'm not sure it ever worked that way, or was intended to. I always thought Automatic was tied to builds, incremental or otherwise, while Manual meant the context menu item.
