Bugzilla – Full Text Bug Listing
| Summary: | Uncancellable "Git repository changed jobs" dominate my CPU forever | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Technology] EGit | Reporter: | Markus Keller <markus.kell.r> | ||||||||||
| Component: | UI | Assignee: | Project Inbox <egit.ui-inbox> | ||||||||||
| Status: | CLOSED WORKSFORME | QA Contact: | |||||||||||
| Severity: | normal | ||||||||||||
| Priority: | P3 | CC: | aar, andreas.hoehmann, angvoz.dev, carolynmacleod4, cdtdoug, daniel_megert, dariusz.luksza, igor, jens.baumgart, kate.ebneter, matthias.sohn, mgobeil, remy.suen, robin.rosenberg, spisek, taylor.mitchell, thatnitind, twolf | ||||||||||
| Version: | 1.2 | Keywords: | performance | ||||||||||
| Target Milestone: | --- | ||||||||||||
| Hardware: | PC | ||||||||||||
| OS: | Windows 98 | ||||||||||||
| Whiteboard: | |||||||||||||
| Attachments: |
|
||||||||||||
This just resulted in blocking all 4 processors of my machine and I had to kill Eclipse.
Steps to reproduce:
1. start new workspace with EGit 1.2.0.201112061823
2. import team project from attachment 208048 [details]
3. search for "compilers.incompatible-environment=?"
==> should get +/- 50 matches
4. in the Search view: context menu > Replace All...
5. replace with: "compilers.incompatible-environment=2"
==> now the workspace is doomed!
The reason is that Git tries to re-index all repositories in parallel now, which brings the machine down to its knees.
Just ran into this again with two threads. I debugged the workbench and found that the problem is not the workload from the parallel indexing per se. Even after I suspended all threads and then resumed one of the indexing treads, it never left the HashMap.get() method! It just stayed on lines 319-322 (JDK build 1.7.0_02-b13). Unfortunately, the debugger doesn't let me inspect the current Entry, but it very much looks like the data structure is screwed and the Entry#next chain forms a loop. I don't know whether the parallel indexing is acceptable, but the unsynchronized access to the SessionResourceVariantByteStore#syncBytesCache definitely isn't. Maybe the multiple threads are even a red herring. At least my two threads were indexing different repositories. But a strange thing I saw was that IndexDiffCache.notifyListeners(Repository, IndexDiffData) line: 94 notified a GitSubscriberMergeContext$1 whose diffTree.pathTree contained resources from the other repository. I run into this bug every time I use the Synchronize view to show differences to another branch. I was in another branch, then selected the master branch in the Repositories view, and then chose "Synchronize with Workspace".
After working for a while (also making changes to the synchronized resources), Eclipse becomes unusable.
I confirmed that the problem is really a corrupt data structure in the HashMap (looping next pointers). Access to this map needs to be synchronized.
"Worker-39" prio=6 tid=0x5ded8400 nid=0x12dc runnable [0x66c0e000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.get(HashMap.java:320)
at org.eclipse.team.core.variants.SessionResourceVariantByteStore.internalGetSyncBytes(SessionResourceVariantByteStore.java:126)
at org.eclipse.team.core.variants.SessionResourceVariantByteStore.setBytes(SessionResourceVariantByteStore.java:104)
at org.eclipse.team.core.variants.ResourceVariantTree.setVariant(ResourceVariantTree.java:72)
at org.eclipse.team.core.variants.AbstractResourceVariantTree.collectChanges(AbstractResourceVariantTree.java:168)
at org.eclipse.team.core.variants.AbstractResourceVariantTree.collectChanges(AbstractResourceVariantTree.java:177)
at org.eclipse.team.core.variants.AbstractResourceVariantTree.collectChanges(AbstractResourceVariantTree.java:177)
at org.eclipse.team.core.variants.AbstractResourceVariantTree.collectChanges(AbstractResourceVariantTree.java:177)
at org.eclipse.team.core.variants.AbstractResourceVariantTree.collectChanges(AbstractResourceVariantTree.java:177)
at org.eclipse.team.core.variants.AbstractResourceVariantTree.collectChanges(AbstractResourceVariantTree.java:177)
at org.eclipse.team.core.variants.AbstractResourceVariantTree.collectChanges(AbstractResourceVariantTree.java:177)
at org.eclipse.team.core.variants.AbstractResourceVariantTree.collectChanges(AbstractResourceVariantTree.java:113)
at org.eclipse.team.core.variants.ResourceVariantTree.access$0(ResourceVariantTree.java:1)
at org.eclipse.team.core.variants.ResourceVariantTree$1.run(ResourceVariantTree.java:112)
at org.eclipse.team.core.variants.ResourceVariantByteStore.run(ResourceVariantByteStore.java:128)
at org.eclipse.team.core.variants.ResourceVariantTree.collectChanges(ResourceVariantTree.java:110)
at org.eclipse.team.core.variants.AbstractResourceVariantTree.refresh(AbstractResourceVariantTree.java:90)
at org.eclipse.team.core.variants.AbstractResourceVariantTree.refresh(AbstractResourceVariantTree.java:54)
at org.eclipse.team.core.variants.ResourceVariantTreeSubscriber.refresh(ResourceVariantTreeSubscriber.java:173)
at org.eclipse.team.core.variants.ResourceVariantTreeSubscriber.refresh(ResourceVariantTreeSubscriber.java:114)
at org.eclipse.egit.core.synchronize.GitResourceVariantTreeSubscriber.refresh(GitResourceVariantTreeSubscriber.java:187)
at org.eclipse.team.core.subscribers.Subscriber.refresh(Subscriber.java:466)
at org.eclipse.egit.core.synchronize.GitSubscriberMergeContext.handleRepositoryChange(GitSubscriberMergeContext.java:139)
at org.eclipse.egit.core.synchronize.GitSubscriberMergeContext.access$0(GitSubscriberMergeContext.java:129)
at org.eclipse.egit.core.synchronize.GitSubscriberMergeContext$1.indexDiffChanged(GitSubscriberMergeContext.java:65)
at org.eclipse.egit.core.internal.indexdiff.IndexDiffCache.notifyListeners(IndexDiffCache.java:94)
at org.eclipse.egit.core.internal.indexdiff.IndexDiffCache.access$0(IndexDiffCache.java:86)
at org.eclipse.egit.core.internal.indexdiff.IndexDiffCache$1.indexDiffChanged(IndexDiffCache.java:81)
at org.eclipse.egit.core.internal.indexdiff.IndexDiffCacheEntry.notifyListeners(IndexDiffCacheEntry.java:297)
at org.eclipse.egit.core.internal.indexdiff.IndexDiffCacheEntry.access$6(IndexDiffCacheEntry.java:290)
at org.eclipse.egit.core.internal.indexdiff.IndexDiffCacheEntry$3.run(IndexDiffCacheEntry.java:155)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)
(In reply to comment #1) > This just resulted in blocking all 4 processors of my machine and I had to kill > Eclipse. > > Steps to reproduce: > 1. start new workspace with EGit 1.2.0.201112061823 > 2. import team project from attachment 208048 [details] > 3. search for "compilers.incompatible-environment=?" > ==> should get +/- 50 matches > 4. in the Search view: context menu > Replace All... > 5. replace with: "compilers.incompatible-environment=2" > ==> now the workspace is doomed! I was unable to reproduce this but using above scenario on EGit 2.0.0.201202171844. Actually both operation search and replace all was pretty fast (completed in less then 60s) and workspace was still usable. *** Bug 372265 has been marked as a duplicate of this bug. *** Oh, hilarious. I removed the repo from view and I still get the jobs starting up, "Re-indexing repository org.eclipse.cdt". (In reply to comment #4) > I was unable to reproduce this [..] Since this is a concurrency bug, you will likely not be able to reproduce it (at least not without adding strategic delays). Please tackle the problem by checking the code. (In reply to comment #3) > I confirmed that the problem is really a corrupt data structure in the HashMap > (looping next pointers). Access to this map needs to be synchronized. Looks like you have to make sure that access to SessionResourceVariantByteStore.syncBytesCache is synchronized on a higher level (or if SessionResourceVariantByteStore should be thread-safe, fix the problem there). Created attachment 218374 [details]
Thread dump
A thread dump when becoming unresponsive during a Re-indexing git repository job. See also the corresponding Profiler snapshot (taken via jvisualvm)
Created attachment 218375 [details]
Profiler snapshot (via jvisualvm)
Profiler snapshot when eclipse becomes unresponsive during an EGit Re-indexing Job.
(In reply to comment #8) > Created attachment 218374 [details] > Thread dump > > A thread dump when becoming unresponsive during a Re-indexing git repository > job. See also the corresponding Profiler snapshot (taken via jvisualvm) I see the following issues here: 1. AbstractTextEditor.validateEditorInputState(AbstractTextEditor.java:4982) tries to lock the workspace in the UI thread and waits for OrderedLock. This is the reason for the UI freeze. I created bug 384473 for this issue. 2. Thread "Worker-103" owns the OrderedLock (is broadcasting changes). It is processing GitSubscriberMergeContext$2.resourceChanged 3. "Worker-61" (IndexDiffCacheEntry.waitForWorkspaceLock) has locked the workspace root and waits for OrderedLock (it performs an empty Workspace.run to synchronize index diff calculations with workspace modifications). This is not the reason for the freeze but it might be better just to acquire and free scheduling rule workspace root instead of executing an empty Workspace.run problem 3. was fixed with 0625b0e5f6f83f01c2b4ec29671c41b6751a6078 (https://git.eclipse.org/r/#/c/6655/) Created attachment 218629 [details]
Thread dump for a possible deadlock. Maybe related ?
I am constantly having similar freezes during git reindexing, just after changing a branch or rebasing.
Today, I had another freeze , and this time looks like there are a lot of worker threads blocked on :
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c4e0bf08> (a java.util.concurrent.locks.ReentrantLock$FairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
at java.util.concurrent.locks.ReentrantLock$FairSync.lock(ReentrantLock.java:201)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.eclipse.egit.core.internal.indexdiff.IndexDiffCacheEntry$5.run(IndexDiffCacheEntry.java:224)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)
The reindexing job is stucked at 7% and it cannot be cancelled.
I don't know if this is related or I should open a new ticket...
Could you retry with a recent nightly version, since https://git.eclipse.org/r/#/c/8558/ index diff updates are done incrementally and hence a lot faster (In reply to comment #13) > Could you retry with a recent nightly version, since > https://git.eclipse.org/r/#/c/8558/ index diff updates are done > incrementally and hence a lot faster I can't say whether they are faster but I sill often see many index jobs from EGit (even for the same repository) which seem to hang. I have tons of these: "Worker-10" prio=6 tid=0x0000000010e80800 nid=0x1e6c waiting on condition [0x0000000019aae000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007838e89a0> (a java.util.concurrent.locks.ReentrantLock$FairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197) at java.util.concurrent.locks.ReentrantLock$FairSync.lock(ReentrantLock.java:229) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290) at org.eclipse.egit.core.internal.indexdiff.IndexDiffCacheEntry$5.run(IndexDiffCacheEntry.java:338) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:53) (In reply to comment #13) > Could you retry with a recent nightly version, since > https://git.eclipse.org/r/#/c/8558/ index diff updates are done > incrementally and hence a lot faster This did definitely not fix the problem. Actually, it seems worse to me. My computer just almost blocked again with tons of index jobs running in parallel [1] and I had to quit Eclipse. This is really a blocker. [1] "Worker-53" prio=6 tid=0x0000000011635000 nid=0x2058 runnable [0x000000001af7e000] java.lang.Thread.State: RUNNABLE at java.util.HashMap.getEntry(HashMap.java:446) at java.util.HashMap.get(HashMap.java:405) at org.eclipse.team.core.variants.SessionResourceVariantByteStore.internalGetSyncBytes(SessionResourceVariantByteStore.java:126) at org.eclipse.team.core.variants.SessionResourceVariantByteStore.setBytes(SessionResourceVariantByteStore.java:104) at org.eclipse.team.core.variants.ResourceVariantTree.setVariant(ResourceVariantTree.java:72) at org.eclipse.team.core.variants.AbstractResourceVariantTree.collectChanges(AbstractResourceVariantTree.java:168) at org.eclipse.team.core.variants.AbstractResourceVariantTree.collectChanges(AbstractResourceVariantTree.java:177) at org.eclipse.team.core.variants.AbstractResourceVariantTree.collectChanges(AbstractResourceVariantTree.java:177) at org.eclipse.team.core.variants.AbstractResourceVariantTree.collectChanges(AbstractResourceVariantTree.java:177) at org.eclipse.team.core.variants.AbstractResourceVariantTree.collectChanges(AbstractResourceVariantTree.java:177) at org.eclipse.team.core.variants.AbstractResourceVariantTree.collectChanges(AbstractResourceVariantTree.java:113) at org.eclipse.team.core.variants.ResourceVariantTree.access$0(ResourceVariantTree.java:1) at org.eclipse.team.core.variants.ResourceVariantTree$1.run(ResourceVariantTree.java:112) at org.eclipse.team.core.variants.ResourceVariantByteStore.run(ResourceVariantByteStore.java:128) at org.eclipse.team.core.variants.ResourceVariantTree.collectChanges(ResourceVariantTree.java:110) at org.eclipse.team.core.variants.AbstractResourceVariantTree.refresh(AbstractResourceVariantTree.java:90) at org.eclipse.team.core.variants.AbstractResourceVariantTree.refresh(AbstractResourceVariantTree.java:54) at org.eclipse.team.core.variants.ResourceVariantTreeSubscriber.refresh(ResourceVariantTreeSubscriber.java:170) at org.eclipse.team.core.variants.ResourceVariantTreeSubscriber.refresh(ResourceVariantTreeSubscriber.java:114) at org.eclipse.egit.core.synchronize.GitResourceVariantTreeSubscriber.refresh(GitResourceVariantTreeSubscriber.java:185) at org.eclipse.team.core.subscribers.Subscriber.refresh(Subscriber.java:466) at org.eclipse.egit.core.synchronize.GitSubscriberMergeContext.handleRepositoryChange(GitSubscriberMergeContext.java:142) at org.eclipse.egit.core.synchronize.GitSubscriberMergeContext.access$0(GitSubscriberMergeContext.java:133) at org.eclipse.egit.core.synchronize.GitSubscriberMergeContext$1.indexDiffChanged(GitSubscriberMergeContext.java:68) at org.eclipse.egit.core.internal.indexdiff.IndexDiffCache.notifyListeners(IndexDiffCache.java:98) at org.eclipse.egit.core.internal.indexdiff.IndexDiffCache.access$0(IndexDiffCache.java:89) at org.eclipse.egit.core.internal.indexdiff.IndexDiffCache$1.indexDiffChanged(IndexDiffCache.java:84) at org.eclipse.egit.core.internal.indexdiff.IndexDiffCacheEntry.notifyListeners(IndexDiffCacheEntry.java:426) at org.eclipse.egit.core.internal.indexdiff.IndexDiffCacheEntry.access$8(IndexDiffCacheEntry.java:418) at org.eclipse.egit.core.internal.indexdiff.IndexDiffCacheEntry$5.run(IndexDiffCacheEntry.java:358) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:53) (In reply to comment #15) > (In reply to comment #13) > > Could you retry with a recent nightly version, since > > https://git.eclipse.org/r/#/c/8558/ index diff updates are done > > incrementally and hence a lot faster > > This did definitely not fix the problem. Actually, it seems worse to me. My > computer just almost blocked again with tons of index jobs running in > parallel [1] and I had to quit Eclipse. This is really a blocker. I'm using org.eclipse.egit.core_2.3.0.201301070013. Created attachment 225342 [details]
Picture showing jobs
(In reply to comment #15) > (In reply to comment #13) > > Could you retry with a recent nightly version, since > > https://git.eclipse.org/r/#/c/8558/ index diff updates are done > > incrementally and hence a lot faster > > This did definitely not fix the problem. Actually, it seems worse to me. My > computer just almost blocked again with tons of index jobs running in > parallel [1] and I had to quit Eclipse. This is really a blocker. > The thread dump indicates this job came from the synchronize view, do you also observe this problem when not using the synchronize view ? Do all these threads involve synchronize ? I don't use the Synchronize view any more, and I don't see the hung jobs any more. The last times I ran into it, I wrote comment 3 and comment 7. At least back then, it was definitely a concurrency issue. (In reply to comment #18) > The thread dump indicates this job came from the synchronize view, do you > also observe this problem when not using the synchronize view ? I need to check. I always us it. And right now, when using it, no extensive indexing takes place. > Do all these threads involve synchronize ? I will check next time when I see this again and attach the full stack trace. Lowering importance for lack of repeatability. There have been changes around this area so it may have been solved, or not, but the latest update is near in time enough to make me believe this could happen, but I want a fresher stack trace. No more reports since 2013. Seems to have been fixed some time during the past four years :-) |
org.eclipse.egit_1.2.0.201110190928 I can't tell exactly what I did to produce this, but I ended up in a state where I had 7 "Git repository changed jobs" in the Progress view that didn't react on Cancel. They use up all my CPUs. After about 30min, I had to shut down Eclipse, because I couldn't work on any more with the remaining 0.1% CPU. I was not running out of memory. All jobs had stacktraces like this (all in HashMap.get()): "Worker-55" prio=6 tid=0x4beeb400 nid=0x2088 runnable [0x569af000] java.lang.Thread.State: RUNNABLE at java.util.HashMap.get(HashMap.java:303) at org.eclipse.team.core.variants.SessionResourceVariantByteStore.internalGetSyncBytes(SessionResourceVariantByteStore.java:126) at org.eclipse.team.core.variants.SessionResourceVariantByteStore.setBytes(SessionResourceVariantByteStore.java:104) at org.eclipse.team.core.variants.ResourceVariantTree.setVariant(ResourceVariantTree.java:72) at org.eclipse.team.core.variants.AbstractResourceVariantTree.collectChanges(AbstractResourceVariantTree.java:168) at org.eclipse.team.core.variants.AbstractResourceVariantTree.collectChanges(AbstractResourceVariantTree.java:177) at org.eclipse.team.core.variants.AbstractResourceVariantTree.collectChanges(AbstractResourceVariantTree.java:177) at org.eclipse.team.core.variants.AbstractResourceVariantTree.collectChanges(AbstractResourceVariantTree.java:177) at org.eclipse.team.core.variants.AbstractResourceVariantTree.collectChanges(AbstractResourceVariantTree.java:177) at org.eclipse.team.core.variants.AbstractResourceVariantTree.collectChanges(AbstractResourceVariantTree.java:113) at org.eclipse.team.core.variants.ResourceVariantTree.access$0(ResourceVariantTree.java:1) at org.eclipse.team.core.variants.ResourceVariantTree$1.run(ResourceVariantTree.java:112) at org.eclipse.team.core.variants.ResourceVariantByteStore.run(ResourceVariantByteStore.java:128) at org.eclipse.team.core.variants.ResourceVariantTree.collectChanges(ResourceVariantTree.java:110) at org.eclipse.team.core.variants.AbstractResourceVariantTree.refresh(AbstractResourceVariantTree.java:90) at org.eclipse.team.core.variants.AbstractResourceVariantTree.refresh(AbstractResourceVariantTree.java:54) at org.eclipse.team.core.variants.ResourceVariantTreeSubscriber.refresh(ResourceVariantTreeSubscriber.java:162) at org.eclipse.team.core.variants.ResourceVariantTreeSubscriber.refresh(ResourceVariantTreeSubscriber.java:126) at org.eclipse.egit.core.synchronize.GitResourceVariantTreeSubscriber.refresh(GitResourceVariantTreeSubscriber.java:144) at org.eclipse.team.core.subscribers.Subscriber.refresh(Subscriber.java:466) at org.eclipse.egit.core.synchronize.GitSubscriberMergeContext.update(GitSubscriberMergeContext.java:135) at org.eclipse.egit.core.synchronize.GitSubscriberMergeContext.access$0(GitSubscriberMergeContext.java:116) at org.eclipse.egit.core.synchronize.GitSubscriberMergeContext$1.repositoryChanged(GitSubscriberMergeContext.java:58) at org.eclipse.egit.core.project.GitProjectData$1.run(GitProjectData.java:158) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)