Community
Participate
Working Groups
Version: Oxygen Release Milestone 4 (4.7.0 M4) Build id: 20161215-0740 There seems to be a severe memory leak in Oxygen. I noticed it a week ago but thought it was some kind of hickup. This Weekend I kept Oxygen open (just sitting there, doing nothing) and my machine imploded ;) Currently, Oxygen is set to use Xmx2G, and htop shows a memory usage of >60%, which is on my machine >30G. top even switched to using terabyte to show memory usage xD PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 30877 mduft 20 0 41.123g 0.030t 39320 S 14.2 65.3 625:36.76 /opt/jdk8/bin/java -Declipse.p2.max.threads=10 -Doomph.update.url=http://download.eclipse.org/oomph/updates/milestone/la+ full command line: mduft 30877 8752 16 Feb03 ? 10:25:29 /opt/jdk8/bin/java -Declipse.p2.max.threads=10 -Doomph.update.url=http://download.eclipse.org/oomph/updates/milestone/latest -Doomph.redirection.index.redirection=index:/->http://git.eclipse.org/c/oomph/org.eclipse.oomph.git/plain/setups/ -Dosgi.requiredJavaVersion=1.8 -XX:+UseG1GC -XX:+UseStringDeduplication -Dosgi.requiredJavaVersion=1.8 -Xms256m -Xmx2048m -jar /ssd/eclipse.org/eclipse/platform-master/eclipse//plugins/org.eclipse.equinox.launcher_1.4.0.v20161129-1342.jar -os linux -ws gtk -arch x86_64 -showsplash -launcher /ssd/eclipse.org/eclipse/platform-master/eclipse/eclipse -name Eclipse --launcher.library /home/mduft/.p2/pool/plugins/org.eclipse.equinox.launcher.gtk.linux.x86_64_1.1.500.v20161026-0946/eclipse_1619.so -startup /ssd/eclipse.org/eclipse/platform-master/eclipse//plugins/org.eclipse.equinox.launcher_1.4.0.v20161129-1342.jar --launcher.appendVmargs -exitdata 8290016 -product org.eclipse.epp.package.committers.product -data file:/ssd/eclipse.org/eclipse/platform-master/workspaces/platform.debug/ -vm /opt/jdk8/bin/java -vmargs -Declipse.p2.max.threads=10 -Doomph.update.url=http://download.eclipse.org/oomph/updates/milestone/latest -Doomph.redirection.index.redirection=index:/->http://git.eclipse.org/c/oomph/org.eclipse.oomph.git/plain/setups/ -Dosgi.requiredJavaVersion=1.8 -XX:+UseG1GC -XX:+UseStringDeduplication -Dosgi.requiredJavaVersion=1.8 -Xms256m -Xmx2048m -jar /ssd/eclipse.org/eclipse/platform-master/eclipse//plugins/org.eclipse.equinox.launcher_1.4.0.v20161129-1342.jar I have a memory dump of the JVM, but it does not look suspicious: it has only ~250MB, so we're nowhere near the 2G. The problem must be in a native library somewhere (no, I did not install anything extra. I used oomph to install eclipse for platform.debug development, no extras). I also have a core file of the process, if that helps - just no idea how to give it to you, it has 43GB The problem ONLY occurs with Oxygen. There is no specific action I did in the IDE. Some "normal development" (fetch from git, do some changes, commit, push to eclipse.org gerrit) and afterwards let the IDE rest for 1, 2 days. Also possibly of interest: $ /opt/jdk8/bin/java -version java version "1.8.0_101" Java(TM) SE Runtime Environment (build 1.8.0_101-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode) Whatever it is, IMHO this is a release blocker.
Aaaand sorry if this is the wrong component, I'm pretty sure it is, I'm just completely unable to locate the area where this issue occurs
I hope this is not Eclipse 4.7. We've also recently encountered huge native memory leaks with Eclipse 3.8 and 1.8.0_91 JVM, with JVM using up to 64 GB RAM (and max heap set to only 4GB!). What we found is that hotspot C2 compiler leaked native memory, and that restricting the number of HS compiler threads to 2 reduces the probability of the leak to 0.0001%. The JVM argument is -XX:CICompilerCount=2. Also we found that 1.8.0_111 JVM seem to have fixed this leak. I didn't found anything useful in the JDK bug database or release notes, the closest issue could be https://bugs.openjdk.java.net/browse/JDK-8159244.
Does this also happen with the plain SDK? http://download.eclipse.org/eclipse/downloads/
Mhm, it could be the VM too. The Neon IDEs I'm using use an older JDK version. I am currently running it with valgrind, checking whether I can see something
You can add the VM options below to the eclipse startup arguments and if you start eclipse from the shell, you will see at the VM shutdown native memory summary printed out: -XX:+UnlockDiagnosticVMOptions -XX:+PrintNMTStatistics -XX:NativeMemoryTracking=summary In our case the "Compiler" section showed 1-60 gigabytes of memory use instead of few kilobytes, below is the example with 10 GB : - Compiler (reserved=10000100KB, committed=10000100KB) (malloc=35KB #320) (arena=10000065KB #14) As said, update to 1.8.0_111 build helped in our case.
I was not able to reproduce it for a few tries, but it seems that now it happens again :) I have diagnostic options on, so I will be able to check it relatively soon. I just want to wait until it definitely outgrew Xmx by at least a factor of three to be sure.
Created attachment 266700 [details] Long run test long.txt contains the memory summary of an Xmx2G Oxygen run where I did a set of actions, and after that let Eclipse rest. It grew with a pace of ~400MB/hr from 2062MB to >4.5G before I quit.
Created attachment 266701 [details] Short run test short.txt contains the very same memory summary for the very same actions performed, except that I did not keep Eclipse running, but rather quit immediately (process real memory consumption <2G). My expectation would have been to immediately see the part the grows (because it's huge in long.txt and small in short.txt). But this isn't the case. Anything you can see from this?
(In reply to Markus Duft from comment #8) > short.txt contains the very same memory summary for the very same actions > performed, except that I did not keep Eclipse running, but rather quit > immediately (process real memory consumption <2G). My expectation would have > been to immediately see the part the grows (because it's huge in long.txt > and small in short.txt). But this isn't the case. Anything you can see from > this? Why do you think you have a memory leak at all in this particular "LARGE" case? The two summaries are almost identical.
(In reply to Andrey Loskutov from comment #9) > (In reply to Markus Duft from comment #8) > > short.txt contains the very same memory summary for the very same actions > > performed, except that I did not keep Eclipse running, but rather quit > > immediately (process real memory consumption <2G). My expectation would have > > been to immediately see the part the grows (because it's huge in long.txt > > and small in short.txt). But this isn't the case. Anything you can see from > > this? > > Why do you think you have a memory leak at all in this particular "LARGE" > case? The two summaries are almost identical. because RAM on my machine gets consumed by the process :D obviously it must be in a place where the JVM does NOT detect the leak. The "long.txt" process used nearly 4 times the amount of RAM as the "short.txt" one. Identical Eclipse, identical steps I performed in Eclipse. The only difference is that I gave the "long.txt" Eclipse some time to fill up RAM ;)
So it may be a native library leaking memory outside of Java? I would tip on the GTK. Interesting would be to see if you can reproduce the problem with gtk2. Also please provide GTK versions you have installed (rpm -q gtk2; rpm -q gtk3).
(In reply to Andrey Loskutov from comment #11) > So it may be a native library leaking memory outside of Java? I would tip on > the GTK. Interesting would be to see if you can reproduce the problem with > gtk2. Also please provide GTK versions you have installed (rpm -q gtk2; rpm > -q gtk3). Mhm, true. fril0041 opt # rpm -q gtk2 gtk2-2.24.31-1.fc23.x86_64 fril0041 opt # rpm -q gtk3 gtk3-3.18.9-1.fc23.x86_64 I downloaded JDK8u121 now, checking whether I can reproduce it with that. If it's a SWT/GTK issue it should not change. Problem is, it does not happen /always/... I restarted and retried a few times to reproduce the issue before. So earliest I have some results tomorrow I think... once I reproduced with GTK3, I'll try with 2 too.
(In reply to Markus Duft from comment #12) > I downloaded JDK8u121 now, checking whether I can reproduce it with that. If > it's a SWT/GTK issue it should not change. Problem is, it does not happen > /always/... I restarted and retried a few times to reproduce the issue > before. So earliest I have some results tomorrow I think... once I > reproduced with GTK3, I'll try with 2 too. OK, with JDK8u121 and GTK3 I can reproduce. Yesterday at 16:37 when I left the company, I left behind Eclipse at 1634MB RES, today in the morning at 7:26 it is already at 8468MB RES. Anything I should check on the running process before terminating it? Any way to determine who is the bad library?
(In reply to Markus Duft from comment #13) > OK, with JDK8u121 and GTK3 I can reproduce. Yesterday at 16:37 when I left > the company, I left behind Eclipse at 1634MB RES, today in the morning at > 7:26 it is already at 8468MB RES. Anything I should check on the running > process before terminating it? Any way to determine who is the bad library? Just to make sure, before killing Eclipse, please run jstack -l, jmap and jinfo on the JVM process, may be some of them would give some hints, run pstack -p to see all processes in the tree spawned by Eclipse, run top/meminfo to get some more data about system memory use, capture /proc/meminfo data too. I'm not a C/C++ expert, so I would also ask Google how to get some data about process memory details on Linux.
I have core dumps, heap dumps, basic htop output, command line, and such. everything looks completely unsuspicious :( (the core file is of course a few GB large ;)) I'm currently running the same thing with GTK2, and it seems that this is not growing. The GTK3 assumption/direction seems to be correct. I will restart with GTK2 one or two more times just to be sure, since it didn't grow ALWAYS with GTK3... Also I noticed that there is something special about this Eclipse. It has the "darkest dark theme" (https://marketplace.eclipse.org/content/darkest-dark-theme) installed - maybe it's related (altough until now I did not see any pointer in that direction).
OK, it's not the theme. Currently it looks like GTK3 is the issue (it's growing again). But I still have no definitive answer...
(In reply to Markus Duft from comment #16) > OK, it's not the theme. Currently it looks like GTK3 is the issue (it's > growing again). But I still have no definitive answer... It would be interesting to see whether the same happens on Fedora 25 . Have you considered updating? Or running F25 in a vm and coping your eclipse in it to see whether same happens?
And another thing - please move to M5 so it doesn't happen that smth that's already fixed is investigated.
(In reply to Alexander Kurtakov from comment #17) > (In reply to Markus Duft from comment #16) > > OK, it's not the theme. Currently it looks like GTK3 is the issue (it's > > growing again). But I still have no definitive answer... > > It would be interesting to see whether the same happens on Fedora 25 . Have > you considered updating? Or running F25 in a vm and coping your eclipse in > it to see whether same happens? I definitely cannot update, but i can setup a virtual machine and copy Eclipse in there. If it indeed is reproducible inside the VM, I can share it for you to reproduce.
From what I gather, you're using Fedora 23 and Gtk3.18. There's a known memory leak issue with this combination that has been fixed in newer versions: (Not exactly sure which new version..) https://github.com/Beep6581/RawTherapee/issues/2990 (In reply to Markus Duft from comment #19) > I definitely cannot update, but i can setup a virtual machine and copy > Eclipse in there. If it indeed is reproducible inside the VM, I can share it > for you to reproduce. Note, Fedora is generally intended for fast-moving development and frequent updates. As a note, fedora 23 has reached end of life: https://fedoraproject.org/wiki/End_of_life This issue doesn't seem to occur for me (on Fedora 25 / Gtk 3.22). But I remember in the past my Eclipse used to grow and I had to restart it from time to time. I recommend you try to setup a F25 VM with similar settings to what you have, and see if the issue still occurs. If it still occurs on F25, then we should hunt down further. If issue still occurs on F25, then we could inspect process memory map: 1) Produce a pmap and see sizes of libraries. - find pid of Eclipse: pgrep java -A (Pid is first number shown) (If you have multiple java applications running, look for the one listing eclipse in it's path) 2) Run pmap (Process map) and output to file. pmap -x 18141 > pmapout 3) Please attach this file to bug. 4) The second column shows memory usage of each library. (or Anon means anonymous memory allocation). We can sort by this column to find out which libs requested the most memory: sort -k2 -n pmapout Also, if memory leak occurs on f25, please let us know if it occurs with a recent integration buidld with no added plugins: (See "Integration builds": http://download.eclipse.org/eclipse/downloads/ )
(In reply to Leo Ufimtsev from comment #20) > From what I gather, you're using Fedora 23 and Gtk3.18. There's a known > memory leak issue with this combination that has been fixed in newer > versions: (Not exactly sure which new version..) > https://github.com/Beep6581/RawTherapee/issues/2990 OK, so I might hit that. It's strange though. I have exactly one Eclipse installation that keeps growing. It is identical to another Oxygen installation that I made with the same Oomph installer, same package selection, etc. It has a different workspace, but that should be it. This installation never grew. Only one of the two is affected. > > (In reply to Markus Duft from comment #19) > > I definitely cannot update, but i can setup a virtual machine and copy > > Eclipse in there. If it indeed is reproducible inside the VM, I can share it > > for you to reproduce. > > Note, Fedora is generally intended for fast-moving development and frequent > updates. As a note, fedora 23 has reached end of life: > https://fedoraproject.org/wiki/End_of_life I know that very well. Still I am not free to upgrade my company system whenever I want :( > > This issue doesn't seem to occur for me (on Fedora 25 / Gtk 3.22). But I > remember in the past my Eclipse used to grow and I had to restart it from > time to time. It also seems that uninstalling the darkest dark theme helped at least a little bit. It grows at half the pace now... xD > > I recommend you try to setup a F25 VM with similar settings to what you > have, and see if the issue still occurs. If it still occurs on F25, then we > should hunt down further. It seems that the issue is NOT reproducible in the F25 VM. On the other hand, just to be sure, I also set up a F23 VM, and it is not reproducible there either. I switched both VMs to XFCE now (since I'm using that on my laptop for better multi-screen support) and restarted the test. I will report once I'm sure it's growing/not growing. > > If issue still occurs on F25, then we could inspect process memory map: > 1) Produce a pmap and see sizes of libraries. > - find pid of Eclipse: > pgrep java -A > (Pid is first number shown) > (If you have multiple java applications running, look for the one listing > eclipse in it's path) > > 2) Run pmap (Process map) and output to file. > pmap -x 18141 > pmapout > > 3) Please attach this file to bug. > > 4) The second column shows memory usage of each library. (or Anon means > anonymous memory allocation). We can sort by this column to find out which > libs requested the most memory: > sort -k2 -n pmapout Thanks, this is some very useful information! > > Also, if memory leak occurs on f25, please let us know if it occurs with a > recent integration buidld with no added plugins: > (See "Integration builds": http://download.eclipse.org/eclipse/downloads/ ) Will try if I can reproduce it at all.
(In reply to Markus Duft from comment #21) > It seems that the issue is NOT reproducible in the F25 VM. On the other > hand, just to be sure, I also set up a F23 VM, and it is not reproducible > there either. I switched both VMs to XFCE now (since I'm using that on my > laptop for better multi-screen support) and restarted the test. I will > report once I'm sure it's growing/not growing. > Hmm, interesting. Maybe there is some plugin that uses some widget that causes a memory leak? Not in the sense that the plugin itself is to blame. More like maybe a plugin is using some widget in some way that should not have a memory leak, but in fact it does have a leak. (I recall fixing one of those memory leaks). If you can somehow reproduce on F25, then this would certainly be an interesting investigation. (In reply to Markus Duft from comment #21) > > Note, Fedora is generally intended for fast-moving development and frequent > > updates. As a note, fedora 23 has reached end of life: > > https://fedoraproject.org/wiki/End_of_life > > I know that very well. Still I am not free to upgrade my company system > whenever I want :( :'-(. I feel your pain.
It gets stranger and stranger :( ... all on XFCE, all with JDK8u121... * Native Fedora 23, Oxygen M4, dark theme: reproducible (~400MB/hr) * Native Fedora 23, Oxygen M4, no dark theme: reproducible (~300MB/hr) * Native Fedora 23, Oxygen M5, no dark theme: ...running... * VM Fedora 23, Oxygen M4, no dark theme: not sure. growing ~10MB/hr but with pauses. Still within valid range. * VM Fedora 25, Oxygen M4, no dark theme: not reproducible. grew a little but stopped * VM Fedora 25, Oxygen M4, dark theme: ...running... pmap on native fedora shows all the 7G it used today morning on [anon]. The next best block is only ~100M, which is completely unsuspicious IMHO.
I managed to reproduce the issue on the F25 VM (with darkest dark theme installed). It's still Oxygen M4 though unfortunately. I will take a snapshot and upgrade to M5
(In reply to Markus Duft from comment #24) > I managed to reproduce the issue on the F25 VM (with darkest dark theme > installed). It's still Oxygen M4 though unfortunately. I will take a > snapshot and upgrade to M5 Interesting..
After letting things run over the weekend, I think we can resolve this bug for now. I thought I have it reproducible, but the process stopped growing at a still reasonable size (~2500M for Xmx2G, which is OK including VM overhead). Currently, I cannot reproduce it at all.
(In reply to Markus Duft from comment #26) > After letting things run over the weekend, I think we can resolve this bug > for now. I thought I have it reproducible, but the process stopped growing > at a still reasonable size (~2500M for Xmx2G, which is OK including VM > overhead). Currently, I cannot reproduce it at all. Thank you for taking the time to investigate this issue and update us on the bug status.
You're welcome - wouldn't want to suffer from such a leak for months until the next release xD
Version: Oxygen Release (4.7.0) Build id: 20170620-1800 I'm commenting here because I also have a huge memory leak on the release version of oxygen, and I do have a way to reproduce it every time (well, on my system at least). To 'trigger' the memory leak, all I have to do is having my mouse hovering the eclipse window (I have a dual screen, just passing my mouse on the other screen stops the leak). If I hover then stops moving the cursor, the leak will stop, and resume as soon as I move again (still on top of the eclipse window). I tried with and without the Darkest Dark theme (in reference to comment #24). Same results. GTKs version: ~$ dpkg -s libgtk2.0-0|grep '^Version' Version: 2.24.30-1ubuntu1 ~$ dpkg -s libgtk-3-0|grep '^Version' Version: 3.18.9-1ubuntu3.3 If I can provide any other information let me know.
I just upgraded to Ubuntu 17.04, the problem is the same, but here is the updated GTKs versions: $dpkg -s libgtk2.0-0|grep '^Version' Version: 2.24.31-1ubuntu1 $dpkg -s libgtk-3-0|grep '^Version' Version: 3.22.11-0ubuntu3
Actually, even without the mouse hovering, just going up and down in a file using the arrow will result in a continuous increase in the heap size
(In reply to Etienne Borel from comment #29) > Version: Oxygen Release (4.7.0) > Build id: 20170620-1800 > > I'm commenting here because I also have a huge memory leak on the release > version of oxygen, and I do have a way to reproduce it every time (well, on > my system at least). > > To 'trigger' the memory leak, all I have to do is having my mouse hovering > the eclipse window (I have a dual screen, just passing my mouse on the other > screen stops the leak). If I hover then stops moving the cursor, the leak > will stop, and resume as soon as I move again (still on top of the eclipse > window). > > I tried with and without the Darkest Dark theme (in reference to comment > #24). Same results. > > GTKs version: > > ~$ dpkg -s libgtk2.0-0|grep '^Version' > Version: 2.24.30-1ubuntu1 > > ~$ dpkg -s libgtk-3-0|grep '^Version' > Version: 3.18.9-1ubuntu3.3 > > If I can provide any other information let me know. This bug is closed. In the interest of house-keeping, would you be able to submit a new bug (maybe link back to this bug as 'related'), with relevant info attached? Thank you
