Bugzilla – Full Text Bug Listing
| Summary: | [patch][Markers] Horrible performance removing markers from Problems View taking hours to do the job | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Eclipse Project] Platform | Reporter: | Andrew Gvozdev <angvoz.dev> | ||||||||||
| Component: | IDE | Assignee: | Platform-UI-Inbox <Platform-UI-Inbox> | ||||||||||
| Status: | CLOSED DUPLICATE | QA Contact: | |||||||||||
| Severity: | normal | ||||||||||||
| Priority: | P3 | CC: | aleherb+eclipse, bokowski, cdtdoug, daniel_megert, jamesblackburn+eclipse, Lars.Vogel, loskutov, malaperle, Michael_Rennie, ob1.eclipse, prakash, remy.suen, wywrzal, yevshif | ||||||||||
| Version: | 3.6 | Keywords: | helpwanted | ||||||||||
| Target Milestone: | --- | ||||||||||||
| Hardware: | All | ||||||||||||
| OS: | All | ||||||||||||
| See Also: |
https://bugs.eclipse.org/bugs/show_bug.cgi?id=431170 https://bugs.eclipse.org/bugs/show_bug.cgi?id=349869 |
||||||||||||
| Whiteboard: | candidate43 | ||||||||||||
| Attachments: |
|
||||||||||||
|
Description
Andrew Gvozdev
This is likely core.resources. Do you know (or can you tell programmatically) whether the markers are confined to a single resource? The markers implementation is array based, so removing a whole bunch from one resource is likely O(n^2) Well I found bug 305034 related to performance of removing markers programmatically but it is nowhere close to what I observe. I thought it might be related to rendering in the table. And no, the markers are for different resources spread out for some of ~20000 files (In reply to comment #2) > Well I found bug 305034 related to performance of removing markers > programmatically but it is nowhere close to what I observe. I thought it might > be related to rendering in the table. If your UI is locking up, then it could be GTK (if you're on linux). The tree view (which backs tables) is also O(n^2) :) I did some experiments on this, see: bug 269248 comment 4 Another fix might be to limit the number of errors logged for a given build in CDT ;) having 10-20k isn't all that useful to users... It would obviously be good if you could attach some backtraces to pinpoint what's taking all the time: while [ true ] ; do jstack JAVA_PID > `date "+%Y%m%d.%H%M.%S"` ; sleep 1; done Created attachment 195503 [details] jstack trace (In reply to comment #4) > If your UI is locking up, then it could be GTK (if you're on linux). The tree > view (which backs tables) is also O(n^2) :) I did some experiments on this, see: > bug 269248 comment 4 It's Windows. > Another fix might be to limit the number of errors logged for a given build in > CDT ;) having 10-20k isn't all that useful to users... That's not the point, just one example. Problems View should behave regardless. There is bug 341867 for Codan. (In reply to comment #5) > It would obviously be good if you could attach some backtraces to pinpoint > what's taking all the time: > while [ true ] ; do jstack JAVA_PID > `date "+%Y%m%d.%H%M.%S"` ; sleep 1; > done It appears that it is indeed trying to rearrange the tree (for each deleted marker?). I would not expect it to work that hard on that. Rather, I would expect it to show an empty table after all markers are deleted. "Reference Handler" daemon prio=10 tid=0x3ea02800 nid=0x1a0c in Object.wait() [0x3ec8f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source) - locked <0x10c40320> (a java.lang.ref.Reference$Lock) "main" prio=6 tid=0x008f6800 nid=0x1a04 runnable [0x0012e000] java.lang.Thread.State: RUNNABLE at org.eclipse.swt.internal.win32.OS.SendMessageW(Native Method) at org.eclipse.swt.internal.win32.OS.SendMessage(OS.java:3223) at org.eclipse.swt.widgets.Tree.getItems(Tree.java:3278) at org.eclipse.swt.widgets.TreeItem.getItems(TreeItem.java:790) at org.eclipse.jface.viewers.TreeViewer.getChildren(TreeViewer.java:168) at org.eclipse.jface.viewers.AbstractTreeViewer.createChildren(AbstractTreeViewer.java:774) at org.eclipse.jface.viewers.TreeViewer.createChildren(TreeViewer.java:644) at org.eclipse.jface.viewers.AbstractTreeViewer.createChildren(AbstractTreeViewer.java:753) at org.eclipse.jface.viewers.AbstractTreeViewer.internalExpand(AbstractTreeViewer.java:1631) at org.eclipse.jface.viewers.AbstractTreeViewer.setSelectionToWidget(AbstractTreeViewer.java:2507) at org.eclipse.jface.viewers.AbstractTreeViewer.setSelectionToWidget(AbstractTreeViewer.java:2944) at org.eclipse.jface.viewers.StructuredViewer.preservingSelection(StructuredViewer.java:1450) at org.eclipse.jface.viewers.TreeViewer.preservingSelection(TreeViewer.java:403) at org.eclipse.jface.viewers.StructuredViewer.preservingSelection(StructuredViewer.java:1404) at org.eclipse.jface.viewers.StructuredViewer.refresh(StructuredViewer.java:1533) at org.eclipse.jface.viewers.ColumnViewer.refresh(ColumnViewer.java:548) at org.eclipse.jface.viewers.StructuredViewer.refresh(StructuredViewer.java:1490) at org.eclipse.ui.internal.views.markers.UIUpdateJob.runInUIThread(UIUpdateJob.java:108) at org.eclipse.ui.progress.UIJob$1.run(UIJob.java:95) at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:35) at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:135) - locked <0x0dfe0108> (a org.eclipse.swt.widgets.RunnableLock) at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:4138) at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3755) at org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.java:2696) at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:2660) at org.eclipse.ui.internal.Workbench.access$4(Workbench.java:2494) at org.eclipse.ui.internal.Workbench$7.run(Workbench.java:674) at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332) at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:667) at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:149) at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:123) at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196) at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110) at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79) at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:344) at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:622) at org.eclipse.equinox.launcher.Main.basicRun(Main.java:577) at org.eclipse.equinox.launcher.Main.run(Main.java:1410) Investigating ... I created 25,000 markers on a project without any builders and used Delete from the context menu of the Problems View to delete all of them. It approximately happens in about a minutes time. Definitely not in the "an hour" range. Also the UIUpdateJob.runInUIThread() is called only once, after all the deletion happens. I guess there might be some listeners in the CDT that probably does some heavy weight work. Please reassign to Platform UI if you can reproduce this on a General Project. Maybe the ProblemsLabelDecorator is part of the game - bug 346011. (In reply to comment #9) > Maybe the ProblemsLabelDecorator is part of the game - bug 346011. I did some experiments and the ProblemsLabelDecorator is not the culprit. (In reply to comment #8) > I created 25,000 markers on a project without any builders and used Delete from > the context menu of the Problems View to delete all of them. I created 20000 markers and deleted 10000 -> The Problems view was stuck for a very long time in org.eclipse.jface.viewers.AbstractTreeViewer.setSelectionToWidget(List, boolean) trying to restore the selection. The complexity of restoring the selection seems to be number of removed items x number of remaining items That's why removing all markers is much faster. (In reply to comment #8) > I created 25,000 markers on a project without any builders and used Delete from > the context menu of the Problems View to delete all of them. It approximately > happens in about a minutes time. Definitely not in the "an hour" range. Also the > UIUpdateJob.runInUIThread() is called only once, after all the deletion happens. > > I guess there might be some listeners in the CDT that probably does some heavy > weight work. Please reassign to Platform UI if you can reproduce this on a > General Project. You are right. I just removed 6000 markers in Java project and it was not a problem, done in matter of seconds. (In reply to comment #10) > (In reply to comment #8) > > I created 25,000 markers on a project without any builders and used Delete > > from the context menu of the Problems View to delete all of them. > I created 20000 markers and deleted 10000 -> The Problems view was stuck for a > very long time in > org.eclipse.jface.viewers.AbstractTreeViewer.setSelectionToWidget(List, boolean) > trying to restore the selection. > The complexity of restoring the selection seems to be > number of removed items x number of remaining items > That's why removing all markers is much faster. Was it C++ project? From my multiple previous attempts it does not make a difference whether removing all markers or a chunk of them. On the other hand, there is no trace of CDT in 626 jstack files generated by James' suggestion in comment 5. (In reply to comment #11) > Was it C++ project? Yes, a C++ project with ~20000 codan problem markers. > From my multiple previous attempts it does not make a > difference whether removing all markers or a chunk of them. I can only talk about my own findings. The stack trace you attached seems to indicate that it is blocked in the same location. (In reply to comment #12) > On the other hand, there is no trace of CDT in 626 jstack files generated by > James' suggestion in comment 5. Are those stack traces similar to the attached one? (In reply to comment #13) >> (In reply to comment #12) > > On the other hand, there is no trace of CDT in 626 jstack files generated by > > James' suggestion in comment 5. > Are those stack traces similar to the attached one? Those which I bothered to look at - yes. Maybe a dozen or so. Obviously I haven't checked every one of them. I ran "grep cdt *" but that did not yield any match. (In reply to comment #11) > > I guess there might be some listeners in the CDT that probably does some heavy > > weight work. Please reassign to Platform UI if you can reproduce this on a > > General Project. > You are right. I just removed 6000 markers in Java project and it was not a > problem, done in matter of seconds. Just tried with a Java project and 20000 problem markers. Started deleting 10000 markers half an hour ago - still not responsive. Created attachment 196068 [details]
Code to create markers
1) Create a General Project
2) Select it
3) Use Menu->Bug Utils->Create Markers
4) Configure Problems view to show all the markers
5) Select all 25000 markers
6) Right click -> Delete
I see that the it takes a longer time in the preserving the selection, but even then the whole thing is done in a minute. Can someone try in their machines and let me know this works? If not it could be windows/linux specific problem (I'm on Mac)
Another interesting piece to optimize from the Platfrom UI side: Bug# 346372 (In reply to comment #16) > I see that the it takes a longer time in the preserving the selection, but even > then the whole thing is done in a minute. Can someone try in their machines and > let me know this works? If not it could be windows/linux specific problem (I'm > on Mac) I am on Windows 7 and this example completes in less than a minute, but when I show only 12500 markers in the view and delete all of them, I have to wait much longer. After about 10 minutes I set a breakpoint in the loop in org.eclipse.jface.viewers.AbstractTreeViewer.setSelectionToWidget(List, boolean). The value of loop variable "i" was 2780, ie. it would take another ~35 minutes to complete the loop. Created attachment 196533 [details]
patch
I don't have the time to test but would think that this change helps. Could somebody else please give it a try? Thanks!
Also, it would be good to know if this is a recent regression (i.e., was the same operation slow in 3.4 or 3.6?). (In reply to comment #19) > I don't have the time to test but would think that this change helps. Could > somebody else please give it a try? Thanks! I tried it and it does help. Note that deselecting all deselects also the group nodes which would otherwise stay selected, but that would hardly be noticed. (In reply to comment #20) > Also, it would be good to know if this is a recent regression (i.e., was the > same operation slow in 3.4 or 3.6?). I tried with 3.6.2 and it's just the same. (In reply to comment #21) > I tried it and it does help. > Note that deselecting all deselects also the group nodes which would otherwise > stay selected, but that would hardly be noticed. I forgot: The items are deselected before the confirmation dialog is opened. It should be moved after the dialog. Created attachment 196647 [details]
Performance tweak in AbstractTreeViewer
In an attempt towards a more general fix, I have made some experimental changes to AbstractTreeViewer.internalExpand() to make better use of the element hashmap (if it is enabled).
Moving to Platform UI. Note that we are awfully close to the 3.7 release. Would you want us to try and get this fixed for 3.7.0? See also http://www.eclipse.org/eclipse/development/plans/freeze_plan_3_7.php#FixPassAfterRC3 - we are past RC3 now. It is indeed very late and the problem is not a regression. I suggest to put this in early 3.8/4.2 and if the fix works well in the field, we can backport it to 3.7.1. (In reply to comment #24) > Moving to Platform UI. Note that we are awfully close to the 3.7 release. Would > you want us to try and get this fixed for 3.7.0? From my POV this is not high priority, but Andrew might have a different opinion as he opened the bug. (In reply to comment #26) > (In reply to comment #24) > > Moving to Platform UI. Note that we are awfully close to the 3.7 release. > > Would you want us to try and get this fixed for 3.7.0? > From my POV this is not high priority, but Andrew might have a different opinion > as he opened the bug. While this particular problem is very annoying to me personally, I do not believe that it is typical for a regular user. I would be just fine with early 3.8. (In reply to Anton Leherbauer from comment #23) > Created attachment 196647 [details] > Performance tweak in AbstractTreeViewer Anton, are you still around? Sorry for ignoring this patch until now. In case you are still around, could you convert this patch into a Gerrit review? See the following link for info how to do that: http://www.vogella.com/tutorials/EclipsePlatformDevelopment/article.html (In reply to Lars Vogel from comment #28) > (In reply to Anton Leherbauer from comment #23) > > Created attachment 196647 [details] > > Performance tweak in AbstractTreeViewer > > Anton, are you still around? Sorry for ignoring this patch until now. In > case you are still around, could you convert this patch into a Gerrit > review? See the following link for info how to do that: > http://www.vogella.com/tutorials/EclipsePlatformDevelopment/article.html Unfortunately the patch causes failures in the JFace unit tests. So, more work and investigation would be necessary, but currently I have no time for that. Simon, can you have a look at the patch and the failing unit tests? (In reply to Lars Vogel from comment #30) > Simon, can you have a look at the patch and the failing unit tests? Simon, do you plan to work on this? Otherwise I could try to look into. See also bug 431170 comment 7. The problem is most likely the view refresh/sort/group code, not the resources API. Andrey, please pick it up, I know Simon will not mind. (In reply to Lars Vogel from comment #32) > Andrey, please pick it up, I know Simon will not mind. This bug is just an extreme case of bug 349869. It is easy to let Problems view to cause *permanent* UI freezes, one do not need to delete all markers, at least on Linux => see bug 349869 comment 11. I would like to close this as a duplicate of bug 349869. (also close/duplicate for bug 383733, bug 392582, bug 431170) Any objections? (In reply to Andrey Loskutov from comment #33) > Any objections? No *** This bug has been marked as a duplicate of bug 349869 *** |