This Bugzilla instance is deprecated, and most Eclipse projects now use GitHub or Eclipse GitLab. Please see the deprecation plan for details.
Bug 383890 - JVM (Eclipse) crash with SIGSEGV in ld-linux-x86-64.so.2
Summary: JVM (Eclipse) crash with SIGSEGV in ld-linux-x86-64.so.2
Status: VERIFIED FIXED
Alias: None
Product: Platform
Classification: Eclipse Project
Component: SWT (show other bugs)
Version: 3.8   Edit
Hardware: PC Linux-GTK
: P3 critical with 2 votes (vote)
Target Milestone: 4.2.1   Edit
Assignee: Silenio Quarti CLA
QA Contact:
URL:
Whiteboard:
Keywords:
: 387377 (view as bug list)
Depends on:
Blocks:
 
Reported: 2012-06-29 08:27 EDT by Andrey Loskutov CLA
Modified: 2014-06-25 12:15 EDT (History)
20 users (show)

See Also:
daniel_megert: pmc_approved+


Attachments
JVM crash logs (213.52 KB, application/zip)
2012-06-29 08:28 EDT, Andrey Loskutov CLA
no flags Details
.xsession-errors (23.19 KB, application/octet-stream)
2012-07-02 10:28 EDT, Andrey Loskutov CLA
no flags Details
VM dump with core file (97.75 KB, text/plain)
2012-07-03 09:18 EDT, Andrey Loskutov CLA
no flags Details
VM crash with core dump *on shutdown* (88.33 KB, text/plain)
2012-07-03 10:36 EDT, Andrey Loskutov CLA
no flags Details
VM dump with core file #3 (113.81 KB, text/plain)
2012-07-04 10:19 EDT, Andrey Loskutov CLA
no flags Details
IBM crash logs on shutdown (564.85 KB, application/zip)
2012-08-23 14:06 EDT, Andrey Loskutov CLA
no flags Details
IBM crash logs on expanding the package explorer (616.60 KB, application/zip)
2012-08-24 02:50 EDT, Andrey Loskutov CLA
no flags Details
IBM crash logs on editor switch (517.53 KB, application/zip)
2012-08-24 04:43 EDT, Andrey Loskutov CLA
no flags Details
log file from N20120827-200 build on eclipse.org (104.26 KB, application/octet-stream)
2012-08-28 13:20 EDT, Andrey Loskutov CLA
no flags Details
logs (22.23 KB, application/zip)
2012-08-31 12:57 EDT, Silenio Quarti CLA
no flags Details
One of many crash logs (121.41 KB, text/x-log)
2012-09-04 09:57 EDT, Radim Hopp CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Andrey Loskutov CLA 2012-06-29 08:27:21 EDT
Build Identifier: I20120531-0600 (3.8 RC3)

While evaluating Eclipse 3.8 I'm observe sporadic VM crashes (once a day).

Common for all crashes is that they happen only with Eclipse 3.8 (Eclipse 3.7.0 running fine in same environment) and that the root cause is always:

# Problematic frame:
# C  [ld-linux-x86-64.so.2+0x8c0b]

I've googled and found only one related report http://www.eclipse.org/forums/index.php/m/875330/

There are also two closed / worksforme bugs: bug 380710 and bug 363491. Bug 380710  is also reported for 3.8.

Our environment is RHEL 5.8 64 bit, JVM is 1.7.0_04 from Oracle.

$uname -a
Linux socbl593 2.6.18-308.el5 #1 SMP Fri Jan 27 17:17:51 EST 2012 x86_64 x86_64 x86_64 GNU/Linux

$ java -version
java version "1.7.0_04"
Java(TM) SE Runtime Environment (build 1.7.0_04-b20)
Java HotSpot(TM) 64-Bit Server VM (build 23.0-b21, mixed mode)

As we do not see similar crashes in our lab with ~50 people using Eclipse 3.7.0 every day (with the same environment!), I can only correlate the crashes to the newer Eclipse version.


Reproducible: Sometimes

Steps to Reproduce:
Please note, that half of the crash reports happens on VM shutdown. Typically I've seen it often on regular switch of the workspace or regular closing Eclipse.
I've also seen crashes where you simply move the mouse or switch to Eclipse and it disappears from the screen without any notice except the VM crash dump. 

Interestingly, many crashes point to the same address ld-linux-x86-64.so.2+0x8c0b. Unfortunately I have no idea what this address could be.

In our case /lib64/ld-linux-x86-64.so.2 is a symbolic link to /lib64/ld-2.5.so

$ file /lib64/ld-2.5.so
/lib64/ld-2.5.so: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), not stripped

Any idea what it could be? Which additional information you would like to see?
Comment 1 Andrey Loskutov CLA 2012-06-29 08:28:56 EDT
Created attachment 218075 [details]
JVM crash logs
Comment 2 Andrey Loskutov CLA 2012-06-29 10:01:03 EDT
One more hint / observation: all crashes on shutdown (5 out of 12) have an entry in "Register to memory mapping" pointing to "R8 =0x0000003069c0d4ff: <offset 0xd4ff> in /usr/lib64/libORBit-2.so.0 at 0x0000003069c00000".

Similar entry can be found in the report http://www.eclipse.org/forums/index.php/m/875330/:

----------- Stack Backtrace -----------
 (0x00000039C9208C0B [ld-linux-x86-64.so.2+0x8c0b])
 (0x00000039C9209029 [ld-linux-x86-64.so.2+0x9029])
 (0x00000039C92092B2 [ld-linux-x86-64.so.2+0x92b2])
 (0x00000039C920CF65 [ld-linux-x86-64.so.2+0xcf65])
 (0x00000039C92129E2 [ld-linux-x86-64.so.2+0x129e2])
 (0x00000039D682D932 [libORBit-2.so.0+0x2d932])

Any idea if this is related to the libORBit-2.so.0? Why only on shutdown?
Comment 3 Andrey Loskutov CLA 2012-07-02 10:20:32 EDT
Today's Eclipse crash (after rebooting the workstation) left no JVM dump but new error message I've never seen before:

***MEMORY-ERROR***: Eclipse[6111]: GSlice: assertion failed: sinfo->n_allocated > 0

I found only two bugs with similar errors: bug 325028 and bug 381407. Neither one seems to be related here, but might be someone can find something useful there. Not sure if this ***MEMORY-ERROR*** crash is a new type of crash or the same as before, but just with different symptom.
Comment 4 Andrey Loskutov CLA 2012-07-02 10:28:55 EDT
Created attachment 218164 [details]
.xsession-errors

Got the ***MEMORY-ERROR*** second time:

***MEMORY-ERROR***: Eclipse[27414]: GSlice: assertion failed: sinfo->n_allocated > 0

Looking at .xsession-errors I can see there two entries (for two crashes today):

kwin: X_SetInputFocus(0x110006f): BadMatch (invalid parameter attributes)
kwin: X_SetInputFocus(0x1146e99): BadMatch (invalid parameter attributes)

It happened both times during debugging an SWT based application, where I was staying in Eclipse on breakpoint (inside the paint() routine) ant after hitting "continue" in the debugger the focus was transferred to the application. Immediately after (or during) the transfer Eclipse crashed. I will try to reproduce it and provide a small SWT snippet, but it looks like it could be independent from the original crash.
Comment 5 Andrey Loskutov CLA 2012-07-03 09:18:47 EDT
Created attachment 218221 [details]
VM dump with core file

We've got our first core dump today. The attached crash dump corresponds to the core file I have locally (~2GB/200MB as tar.gz). I can provide the core dump too.

This is what my colleague got from the core:

/vobs/devtools/gdb-7.4.50/gdb /opt/java1.7_x86_64/jre/bin/java
 gdb> target core <insert-your-core-file-path-here>
 gdb> info thread
 gdb> tread ...
 gdb> bt
#0 0x000000369ac30265 in raise () from /lib64/libc.so.6
 #1 0x000000369ac31d10 in abort () from /lib64/libc.so.6
 #2 0x00002b32900ddd25 in os::abort(bool) () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
 #3 0x00002b329023d747 in VMError::report_and_die() () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
 #4 0x00002b329023dd1e in crash_handler(int, siginfo*, void*) () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
 #5 <signal handler called>
 #6 0x00002b32900d60e1 in os::is_first_C_frame(frame*) () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
 #7 0x00002b329023bf8d in VMError::report(outputStream*) () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
 #8 0x00002b329023d34a in VMError::report_and_die() () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
 #9 0x00002b32900e1ad0 in JVM_handle_linux_signal () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
 #10 <signal handler called>
 #11 0x000000369a408abb in check_match.8512 () from /lib64/ld-linux-x86-64.so.2
 #12 0x000000369a408ed9 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2
 #13 0x000000369a409162 in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
 #14 0x000000369a40ccc5 in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
 #15 0x000000369a412882 in _dl_runtime_resolve () from /lib64/ld-linux-x86-64.so.2
 #16 0x00002aab893439a6 in ?? () from /usr/lib64/gtk-2.0/2.10.0/engines/libclearlooks.so
 #17 0x000000341ce85797 in ?? () from /usr/lib64/libgtk-x11-2.0.so.0
 #18 0x000000341ce85a3e in ?? () from /usr/lib64/libgtk-x11-2.0.so.0
 #19 0x000000341cf2ffcd in ?? () from /usr/lib64/libgtk-x11-2.0.so.0
 #20 0x00000036a140b148 in g_closure_invoke () from /lib64/libgobject-2.0.so.0
 #21 0x00000036a141b8e6 in ?? () from /lib64/libgobject-2.0.so.0
 #22 0x00000036a141c516 in g_signal_emit_valist () from /lib64/libgobject-2.0.so.0
 #23 0x00000036a141c923 in g_signal_emit () from /lib64/libgobject-2.0.so.0
 #24 0x000000341d02d79e in ?? () from /usr/lib64/libgtk-x11-2.0.so.0
 #25 0x000000341cea6f81 in gtk_container_propagate_expose () from /usr/lib64/libgtk-x11-2.0.so.0
 #26 0x000000341ceeb177 in ?? () from /usr/lib64/libgtk-x11-2.0.so.0
 #27 0x000000341cea79de in ?? () from /usr/lib64/libgtk-x11-2.0.so.0
 #28 0x000000341cf2ffcd in ?? () from /usr/lib64/libgtk-x11-2.0.so.0
 #29 0x00000036a140b08a in g_closure_invoke () from /lib64/libgobject-2.0.so.0
 #30 0x00000036a141b8e6 in ?? () from /lib64/libgobject-2.0.so.0
 #31 0x00000036a141c516 in g_signal_emit_valist () from /lib64/libgobject-2.0.so.0
 #32 0x00000036a141c923 in g_signal_emit () from /lib64/libgobject-2.0.so.0
 #33 0x000000341d02d79e in ?? () from /usr/lib64/libgtk-x11-2.0.so.0
 #34 0x000000341cf2a880 in gtk_main_do_event () from /usr/lib64/libgtk-x11-2.0.so.0
 #35 0x00002aab89e97240 in Java_org_eclipse_swt_internal_gtk_OS__1gtk_1main_1do_1event ()
 from /home/jelmenth/Projects/ws-ide-dev/.metadata/.plugins/org.eclipse.pde.core/New_configuration (1)/org.eclipse.osgi/bundles/424/2/.cp/libswt-pi-gtk-3833.so
 #36 0x00002aaaab5c0fbf in ?? ()
 #37 0x000000001b951800 in ?? ()
 #38 0x000000004085a668 in ?? ()
 #39 0x000000004085a670 in ?? ()
 #40 0x0000000000000000 in ?? ()
Comment 6 Andrey Loskutov CLA 2012-07-03 10:36:52 EDT
Created attachment 218226 [details]
VM crash with core dump *on shutdown*

Ok, now I've got core on shutdown. Now we see that libORBit-2.so.0 calls _dl_runtime_resolve() and for some reason this crashes the VM on shutdown...

gdb backtrace:
[Switching to thread 1 (Thread 0x40766940 (LWP 31494))]
#0  0x000000305b230265 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x000000305b230265 in raise () from /lib64/libc.so.6
#1  0x000000305b231d10 in abort () from /lib64/libc.so.6
#2  0x00002aec0542dd25 in os::abort(bool) ()
   from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#3  0x00002aec0558d747 in VMError::report_and_die() ()
   from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#4  0x00002aec0558dd1e in crash_handler(int, siginfo*, void*) ()
   from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#5  <signal handler called>
#6  0x00002aec054260e1 in os::is_first_C_frame(frame*) ()
   from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#7  0x00002aec0558bf8d in VMError::report(outputStream*) ()
   from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#8  0x00002aec0558d34a in VMError::report_and_die() ()
   from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#9  0x00002aec05431ad0 in JVM_handle_linux_signal ()
   from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#10 <signal handler called>
#11 0x000000305ac08c0b in check_match.8514 () from /lib64/ld-linux-x86-64.so.2
#12 0x000000305ac09029 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2
#13 0x000000305ac092b2 in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
#14 0x000000305ac0cf65 in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
#15 0x000000305ac129e2 in _dl_runtime_resolve () from /lib64/ld-linux-x86-64.so.2
#16 0x0000003069c2d932 in ?? () from /usr/lib64/libORBit-2.so.0
#17 0x000000305b2334f5 in exit () from /lib64/libc.so.6
#18 0x00002aec05238817 in vm_direct_exit(int) ()
   from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#19 0x00002aec0559629c in VM_Operation::evaluate() ()
   from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#20 0x00002aec05594cb0 in VMThread::evaluate_operation(VM_Operation*) ()
   from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#21 0x00002aec05595236 in VMThread::loop() ()
   from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#22 0x00002aec055958d0 in VMThread::run() ()
   from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#23 0x00002aec0542eff0 in java_start(Thread*) ()
   from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#24 0x000000305c20677d in start_thread () from /lib64/libpthread.so.0
#25 0x000000305b2d49ad in clone () from /lib64/libc.so.6
#26 0x0000000000000000 in ?? ()
Comment 7 Andrey Loskutov CLA 2012-07-04 10:19:00 EDT
Created attachment 218276 [details]
VM dump with core file #3

New dump today, still ld-linux-x86-64.so.2 but again different backtrace. VM crash log attached. Looking at the dumps the only common thing is *where* it crashes (ld-linux-x86-64.so.2), but nothing in common *who* calls it... 

Can it be that our environment (RHEL 5.8) is somehow incompatible with the native SWT/platform binaries?

#0 0x000000369ac30265 in raise () from /lib64/libc.so.6
 #1 0x000000369ac31d10 in abort () from /lib64/libc.so.6
 #2 0x00002b2759078d25 in os::abort(bool) () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
 #3 0x00002b27591d8747 in VMError::report_and_die() () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
 #4 0x00002b27591d8d1e in crash_handler(int, siginfo*, void*) () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
 #5 <signal handler called>
 #6 0x00002b27590710e1 in os::is_first_C_frame(frame*) () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
 #7 0x00002b27591d6f8d in VMError::report(outputStream*) () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
 #8 0x00002b27591d834a in VMError::report_and_die() () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
 #9 0x00002b275907cad0 in JVM_handle_linux_signal () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
 #10 <signal handler called>
 #11 0x000000369a408abb in check_match.8512 () from /lib64/ld-linux-x86-64.so.2
 #12 0x000000369a408ed9 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2
 #13 0x000000369a409162 in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
 #14 0x000000369a40ccc5 in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
 #15 0x000000369a412882 in _dl_runtime_resolve () from /lib64/ld-linux-x86-64.so.2
 #16 0x00002aab8d111adf in Java_java_net_PlainSocketImpl_socketBind () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/libnet.so
 #17 0x00002aaaab175d8e in ?? ()
 #18 0x000000000f4d4848 in ?? ()
 #19 0x0000000000000000 in ?? ()
Comment 8 Andrey Loskutov CLA 2012-07-11 17:22:24 EDT
With Eclipse 3.8 recompiled completely from 3.8 sources on our RHEL 5.8 (see http://dev.eclipse.org/mhonarc/lists/cbi-dev/msg00493.html) I've observed new kind of crash (no VM dump, Eclipse froze and disappeared with leaving this output on the text console):

*** glibc detected *** /usr/java/latest_x86_64/bin/java: munmap_chunk(): invalid pointer: 0x00002aaac435d4e0 ***
======= Backtrace: =========
/lib64/libc.so.6(cfree+0x166)[0x305b2729f6]
/usr/lib64/libpango-1.0.so.0(pango_glyph_string_free+0x16)[0x3063c10f76]
/usr/lib64/libpango-1.0.so.0[0x3063c1963a]
/lib64/libglib-2.0.so.0(g_slist_foreach+0x1d)[0x3060e4350d]
/usr/lib64/libpango-1.0.so.0(pango_layout_line_unref+0x35)[0x3063c18ee5]
/usr/lib64/libpango-1.0.so.0[0x3063c19675]
/usr/lib64/libpango-1.0.so.0(pango_layout_set_attributes+0x33)[0x3063c1af73]
/data/.eclipse_ide/com.verigy.ide_3.8_x86_64-custom/configuration/org.eclipse.osgi/bundles/433/2/.cp/libswt-pi-gtk-3833.so(Java_org_eclipse_swt_internal_gtk_OS__1pango_1layout_1set_1attributes+0xf)[0x2aaabdb3f86c]
[0x2aaaab7f41ed]
======= Memory map: ========
00400000-00401000 r-xp 00000000 08:02 461339                             /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/bin/java
00600000-00601000 rw-p 00000000 08:02 461339                             /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/bin/java
02d7b000-070cb000 rw-p 02d7b000 00:00 0                                  [heap]

I will continue to test tomorrow, but it looks like it's SWT problem with our RHEL 5.8 libraries.

If this an SWT problem, can someone explain me how I can *replace* patched and built SWT jar? I've tried to build SWT binaries alone as described in http://www.eclipse.org/swt/faq.php#howbuilddll and then I've tried to crate a feature patch for SWT, but I was not successful to install this feature patch into my Eclipse. Any hint is welcome.
Comment 9 Andrey Loskutov CLA 2012-07-16 11:18:05 EDT
It crashed again few times. I have core dumps and VM logs. Recent crashes have all libswt-pi-gtk-3833.so as the root cause, but this could be just occasion.

rpm -q cairo
cairo-1.2.4-5.el5

rpm -q gtk2
gtk2-2.10.4-21.el5_7.7

2 example gdb stacks from core files:

[Switching to thread 1 (Thread 0x402db940 (LWP 20899))]
#0  0x000000305b230265 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x000000305b230265 in raise () from /lib64/libc.so.6
#1  0x000000305b231d10 in abort () from /lib64/libc.so.6
#2  0x00002ab4542a0d25 in os::abort(bool) () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#3  0x00002ab454400747 in VMError::report_and_die() () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#4  0x00002ab454400d1e in crash_handler(int, siginfo*, void*) () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#5  <signal handler called>
#6  0x00002ab4542990e1 in os::is_first_C_frame(frame*) () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#7  0x00002ab4543fef8d in VMError::report(outputStream*) () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#8  0x00002ab45440034a in VMError::report_and_die() () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#9  0x00002ab4542a4ad0 in JVM_handle_linux_signal () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#10 <signal handler called>
#11 0x000000305ac08c0b in check_match.8514 () from /lib64/ld-linux-x86-64.so.2
#12 0x000000305ac09029 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2
#13 0x000000305ac092b2 in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
#14 0x000000305ac0cf65 in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
#15 0x000000305ac129e2 in _dl_runtime_resolve () from /lib64/ld-linux-x86-64.so.2
#16 0x00002aaab9fe354b in Java_org_eclipse_swt_internal_gtk_OS__1g_1utf16_1offset_1to_1pointer



[Switching to thread 1 (Thread 0x4133a940 (LWP 14196))]
#0  0x000000305b230265 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x000000305b230265 in raise () from /lib64/libc.so.6
#1  0x000000305b231d10 in abort () from /lib64/libc.so.6
#2  0x00002af96172ed25 in os::abort(bool) () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#3  0x00002af96188e747 in VMError::report_and_die() () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#4  0x00002af96188ed1e in crash_handler(int, siginfo*, void*) () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#5  <signal handler called>
#6  0x00002af9617270e1 in os::is_first_C_frame(frame*) () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#7  0x00002af96188cf8d in VMError::report(outputStream*) () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#8  0x00002af96188e34a in VMError::report_and_die() () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#9  0x00002af961732ad0 in JVM_handle_linux_signal () from /opt/hp93000rt/el5/x86_64/zenith_jdk1.7.0_04/jre/lib/amd64/server/libjvm.so
#10 <signal handler called>
#11 0x000000305ac08c0b in check_match.8514 () from /lib64/ld-linux-x86-64.so.2
#12 0x000000305ac09029 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2
#13 0x000000305ac092b2 in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
#14 0x000000305ac0cf65 in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
#15 0x000000305ac129e2 in _dl_runtime_resolve () from /lib64/ld-linux-x86-64.so.2
#16 0x00002aaab97de0a6 in Java_org_eclipse_swt_internal_gtk_OS_memmove__Lorg_eclipse_swt_internal_gtk_GtkSelectionData_2JJ ()
Comment 10 Arun Thondapu CLA 2012-07-17 08:09:11 EDT
Since you mentioned you were using the 3.8 RC3 build, can you please try to use the released version now? It might not solve the problem but it would make it easier for us to try to reproduce and debug the issue.

I'm assuming you're running the 64-bit version of Eclipse, are you running a vanilla version or have you installed any other plugins on top of Eclipse?

The interesting thing here is that the sequence of calls from the stack trace seems different each time but the crash ultimately happens always in the same function viz. do_lookup_x (). Since this is a libc function, I'm just curious whether there were any changes or updates recently in this area? Which version of libc is currently installed? Are there any upgrades available for this library?
Can you also please check the versions of pango and glib libraries on your machine? They may need to be upgraded to support Eclipse Juno.
Comment 11 Andrey Loskutov CLA 2012-07-17 08:47:12 EDT
(In reply to comment #10)
> Since you mentioned you were using the 3.8 RC3 build, can you please try to use
> the released version now? It might not solve the problem but it would make it
> easier for us to try to reproduce and debug the issue.

I was using M7, RC3 and final 3.8.0 with no differences at all. I've also recently built the 3.8.0 locally (including the native bits) by following instructions at CBI wiki http://wiki.eclipse.org/CBI/Eclipse_Platform_Build with exact same results. They all crash in ld-linux-x86-64.so.2.

> I'm assuming you're running the 64-bit version of Eclipse, are you running a
> vanilla version or have you installed any other plugins on top of Eclipse?

Yes, we use 64 bit Eclipse on 64 bit JVM on 64 bit RHEL 5.8. We use not the "vanilla" platform: we have Clearcase + CDT + GEF + few custom plugins. The crashes were observed first after we started trying to switch the *platform* from 3.7.0 to 3.8.0.

> The interesting thing here is that the sequence of calls from the stack trace
> seems different each time but the crash ultimately happens always in the same
> function viz. do_lookup_x (). 

Yes, this is what is really common for all of this crashes. I was unable to find any other common patterns. Also the crashes seem to happen randomly (no specific UI action triggering them). One common thing for the 50% of the crashes: they happen while regular Eclipse shutdown. But again, there is no guarantee that we will see crash on shutdown, just IF Eclipse crashes on shutdown, those crashes always point to libORBit.

> Since this is a libc function, I'm just curious
> whether there were any changes or updates recently in this area? 

No, our Linux admins told me we are using RHEL 5.8 "standard" libc. Even if we had such change, we would also see crashes on 3.7.0, which is not the case. Right now our lab (>50 people) is using 3.7.0 with no crashes, but few guys like me who are trying to switch the platform experience constant 3.8.0 crashes with the rate 1-2 times at day.

> Which version of libc is currently installed? 

glibc-2.5-81.

> Are there any upgrades available for this
> library?

In general we are pretty conservative with the platform (RHEL 5.8 is the *latest* we have). So even if there were RHEL update for glibc, we were unlikely switch to it "for Eclipse only", as glibc is one of the important platform parts and Eclipse is "just another tool".

> Can you also please check the versions of pango and glib libraries on your
> machine? They may need to be upgraded to support Eclipse Juno.

rpm -q glibc
glibc-2.5-81
rpm -q glib
glib-1.2.10-20.el5
rpm -q pango
pango-1.14.9-8.el5_7.3 
rpm -q cairo
cairo-1.2.4-5.el5
rpm -q gtk2
gtk2-2.10.4-21.el5_7.7
rpm -q ORBit2
ORBit2-2.14.3-5.el5
Comment 12 Andrey Loskutov CLA 2012-07-17 17:35:54 EDT
Google found some probably related crashes on Eclipse.org:

http://www.google.com/search?&q=%22ld-linux-x86-64.so.2%22+sigsegv+java+site:download.eclipse.org&hl=en

Below few quotes. It would be interesting to look into the hs_err_pid*.log files, but I have no access to the build machines. Anyone with access to the build machines can confirm if this are related crashes, or just some other problems?


http://download.eclipse.org/eclipse/downloads/drops4/N20120702-2000/testresults/linux.gtk.x86_6.0/org.eclipse.jdt.ui.tests.AutomatedSuite.txt

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f23d81af02b, pid=10114, tid=139791903262464
#
# JRE version: 6.0_27-b07
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.2-b06 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [ld-linux-x86-64.so.2+0x902b]  char+0x1b
#
# An error report file with more information is saved as:
# /opt/users/hudsonbuild/workspace/eclipse-JUnit-Linux2/workarea/N20120702-2000/eclipse-testing/test-eclipse/eclipse/hs_err_pid10114.log

http://download.eclipse.org/eclipse/downloads/drops4/N20120702-2000/testresults/linux.gtk.x86_6.0/org.eclipse.jdt.ui.tests.refactoring.all.AllAllRefactoringTests.txt

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f707d16e02b, pid=26010, tid=140121088751360
#
# JRE version: 6.0_27-b07
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.2-b06 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [ld-linux-x86-64.so.2+0x902b]  char+0x1b
#
# An error report file with more information is saved as:
# /opt/users/hudsonbuild/workspace/eclipse-JUnit-Linux2/workarea/N20120702-2000/eclipse-testing/test-eclipse/eclipse/hs_err_pid26010.log

http://download.eclipse.org/eclipse/downloads/drops4/R-4.2-201206081400/testresults/linux.gtk.x86_6.0/org.eclipse.jdt.debug.tests.AutomatedSuite.txt

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fa44c82802b, pid=23419, tid=140343221110528
#
# JRE version: 6.0_27-b07
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.2-b06 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [ld-linux-x86-64.so.2+0x902b]  char+0x1b
#
# An error report file with more information is saved as:
# /opt/users/hudsonbuild/workspace/eclipse-JUnit-Linux2/workarea/I20120608-1400/eclipse-testing/test-eclipse/eclipse/hs_err_pid23419.log
#

http://download.eclipse.org/eclipse/downloads/drops4/R-4.2-201206081400/testresults/linux.gtk.x86_6.0/org.eclipse.jdt.ui.tests.AutomatedSuite.txt

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fa39750902b, pid=785, tid=140340572071680
#
# JRE version: 6.0_27-b07
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.2-b06 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [ld-linux-x86-64.so.2+0x902b]  char+0x1b
#
# An error report file with more information is saved as:
# /opt/users/hudsonbuild/workspace/eclipse-JUnit-Linux2/workarea/I20120608-1400/eclipse-testing/test-eclipse/eclipse/hs_err_pid785.log
#

http://download.eclipse.org/eclipse/downloads/drops/M20120705-1600/testresults/linux.gtk.x86_6.0/org.eclipse.ui.tests.session.SessionTests.txt

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007ffab58b502b, pid=19198, tid=140714741393152
#
# JRE version: 6.0_27-b07
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.2-b06 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [ld-linux-x86-64.so.2+0x902b]  char+0x1b
#
# An error report file with more information is saved as:
# /opt/users/hudsonbuild/workspace/eclipse-JUnit-Linux/workarea/M20120705-1600/eclipse-testing/test-eclipse/eclipse/hs_err_pid19198.log
#

http://download.eclipse.org/eclipse/downloads/drops4/M20120712-1200/testresults/linux.gtk.x86_6.0/org.eclipse.pde.ui.tests.AllPDETests.txt

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f8fa08d402b, pid=27471, tid=140254827697920
#
# JRE version: 6.0_27-b07
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.2-b06 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [ld-linux-x86-64.so.2+0x902b]  char+0x1b
#
# An error report file with more information is saved as:
# /opt/users/hudsonbuild/workspace/eclipse-JUnit-Linux2/workarea/M20120712-1200/eclipse-testing/test-eclipse/eclipse/hs_err_pid27471.log
#
Comment 13 Andrey Loskutov CLA 2012-08-02 16:48:33 EDT
Just a small status update:

1) We still crash on RHEL 5.8, one to two times per week per developer. Meanwhile we tried each and every known setting related to the native SWT:

export GDK_NATIVE_WINDOWS=1
export LD_BIND_NOW=1
-Dorg.eclipse.swt.internal.gtk.cairoGraphics=false
-Dorg.eclipse.swt.internal.gtk.useCairo=false

with no success so far. If you have any other flags to play with, please do not hesitate to share it - we are so far that we will try *everything* to get rid of crashes.

2) Google still finds new crashes on Eclipse.org build server, most recent one from 26 Jul 2012:

http://www.google.com/search?&q=%22ld-linux-x86-64.so.2%22+sigsegv+java+site:download.eclipse.org&hl=en

Can anyone from commiters please try to analyze the eclipse.org crashes - how often, in which builds, in which tests, on which build machines, since when? May be this could help to identify the root cause. I can't do this as I have no access to the eclipse.org build infrastructure.

3) My colleague tried to analyze crash dumps/loader sources and according to him the loader tries to lookup something in a table but the table doesn't contain expected data, as if someone corrupted/deleted the entries - so it crashes.

4) May be this is random, but I observed some strange behavior. E.g. I usually have 2-3 Eclipse workspaces open, and so if one crashes, few minutes after that the second one will crash and later also the third one. As if all of them would use the same shared memory region and this one get corrupted by unknown reason. Any idea?
Comment 14 John Arthorne CLA 2012-08-03 11:41:13 EDT
(In reply to comment #13)
> Can anyone from commiters please try to analyze the eclipse.org crashes -
> how often, in which builds, in which tests, on which build machines, since
> when? May be this could help to identify the root cause. I can't do this as
> I have no access to the eclipse.org build infrastructure.

David I looked around for these crash logs but I don't know where they would be. They should be in the current working directory of the Eclipse launched to run the tests.
Comment 15 John Arthorne CLA 2012-08-03 11:42:06 EDT
CCing Dani as FYI. Most of these crashes are happening in the JDT tests (not implying it's a JDT problem, but in case you were wondering what was happening to your tests).
Comment 16 Andrey Loskutov CLA 2012-08-03 16:25:55 EDT
(In reply to comment #14)
> David I looked around for these crash logs but I don't know where they would
> be. They should be in the current working directory of the Eclipse launched
> to run the tests.

Here is the list from Google, sorted by path:

/opt/users/hudsonbuild/workspace/eclipse-JUnit-Linux/workarea/I20120608-1200/eclipse-testing/test-eclipse/eclipse/hs_err_pid16044.log
/opt/users/hudsonbuild/workspace/eclipse-JUnit-Linux/workarea/M20120705-1600/eclipse-testing/test-eclipse/eclipse/hs_err_pid19198.log
/opt/users/hudsonbuild/workspace/eclipse-JUnit-Linux/workarea/M20120712-1000/eclipse-testing/test-eclipse/eclipse/hs_err_pid28703.log
/opt/users/hudsonbuild/workspace/eclipse-JUnit-Linux2/workarea/I20120608-1400/eclipse-testing/test-eclipse/eclipse/hs_err_pid32288.log
/opt/users/hudsonbuild/workspace/eclipse-JUnit-Linux2/workarea/M20120712-1200/eclipse-testing/test-eclipse/eclipse/hs_err_pid11490.log
/opt/users/hudsonbuild/workspace/eclipse-JUnit-Linux2/workarea/M20120712-1200/eclipse-testing/test-eclipse/eclipse/hs_err_pid22730.log
/opt/users/hudsonbuild/workspace/eclipse-JUnit-Linux2/workarea/M20120712-1200/eclipse-testing/test-eclipse/eclipse/hs_err_pid27471.log
/opt/users/hudsonbuild/workspace/eclipse-JUnit-Linux2/workarea/M20120712-1200/eclipse-testing/test-eclipse/eclipse/hs_err_pid31050.log
/opt/users/hudsonbuild/workspace/eclipse-JUnit-Linux2/workarea/M20120720-1300/eclipse-testing/test-eclipse/eclipse/hs_err_pid9476.log
/opt/users/hudsonbuild/workspace/eclipse-JUnit-Linux2/workarea/M20120726-1200/eclipse-testing/test-eclipse/eclipse/hs_err_pid5788.log
/opt/users/hudsonbuild/workspace/eclipse-JUnit-Linux2/workarea/N20120723-2000/eclipse-testing/test-eclipse/eclipse/hs_err_pid3974.log
Comment 17 David Williams CLA 2012-08-06 23:13:07 EDT
(In reply to comment #14)
> (In reply to comment #13)

> David I looked around for these crash logs but I don't know where they would
> be. They should be in the current working directory of the Eclipse launched
> to run the tests.

If someone looked real quick, and drilled down in our Hudson "workarea" they might be able to look/save one of these logs before the next test was ran (at which point the workarea is cleaned and the logs would be lost). 

For example, one approximate URL would end up being

https://hudson.eclipse.org/hudson/view/Eclipse%20and%20Equinox/job/eclipse-JUnit-Linux2/ws/workarea/I20120806-2000/eclipse-testing/test-eclipse/eclipse/

(But, note it changes for each build id, such as "I20120806-2000" in this case), and of course platform and stream. 

I'll investigate if there's some way to save crash logs and make them part of what we save for our "test results".
Comment 18 Markus Keller CLA 2012-08-07 03:39:21 EDT
> For example, one approximate URL would end up being
> 
> https://hudson.eclipse.org/hudson/view/Eclipse%20and%20Equinox/job/eclipse-
> JUnit-Linux2/ws/workarea/I20120806-2000/eclipse-testing/test-eclipse/eclipse/

The workarea is not accessible for mere mortals, see bug 386722.
Comment 19 David Williams CLA 2012-08-17 11:40:17 EDT
I'm a bit stumped. I added some code to copy the logs and was surprised none had been showing up and just now, looked more closely. From last nightly, this test result, 

http://download.eclipse.org/eclipse/downloads/drops4/N20120816-2100/testresults/linux.gtk.x86_6.0/org.eclipse.jdt.ui.tests.refactoring.all.AllAllRefactoringTests.txt

says a log was written to 

# An error report file with more information is saved as:
# /opt/users/hudsonbuild/workspace/eclipse-JUnit-Linux2/workarea/N20120816-2100/eclipse-testing/test-eclipse/eclipse/hs_err_pid12066.log


I don't have direct shell access to the slave running the test, but according to the web interface, that directory would be 

https://hudson.eclipse.org/hudson/view/Eclipse%20and%20Equinox/job/eclipse-JUnit-Linux2/ws/workarea/N20120816-2100/eclipse-testing/test-eclipse/eclipse/

But I see no hs_err_pid12066.log there. 

Only thing I can think of is that the file has some sort of restricted access, for security reasons? 

Hmmm ... I could think of another thing ... I wonder if that "test eclipse" instance is recreated each suite, thereby deleting any logs that were there. If that's the case, the code to "save" the logs will need to go "deeper" into the test framework code. (I was trying/hoping to do it via what we capture in archived results via Hudson: 
We had been capturing
**/eclipse-testing/results/**
and to that I added
,**/eclipse-testing/**/*log
which did pick up 2 or 3 extra "log" files so I thought it was working ... but that'd only run once, after all tests are finished. 

So, that's status. Thought I'd document what I know about capturing the logs.
Comment 20 Markus Keller CLA 2012-08-17 13:16:06 EDT
(In reply to comment #19)
> Hmmm ... I could think of another thing ... I wonder if that "test eclipse"
> instance is recreated each suite, thereby deleting any logs that were there.

I was investigating this in parallel, and came to the same conclusion. I added bug 387496 to track the JDT UI AllAllRefactoringTests (they crash in a different library than what was reported here before).

I changed our particular test script to save the hs_err_pid*.log . If this works out, then we should move that up into library.xml.
Comment 21 David Williams CLA 2012-08-18 21:55:39 EDT
To cross reference, I've opened bug 387544 to discuss or document the general solution to capturing crash logs in library.xml.
Comment 22 Silenio Quarti CLA 2012-08-22 12:08:26 EDT
*** Bug 387377 has been marked as a duplicate of this bug. ***
Comment 23 Silenio Quarti CLA 2012-08-22 15:58:12 EDT
I have been running Eclipse on RHEL 5.8 64bit for several days and I have not crashed once.  I am using an IBM JRE. It seems that every crash log attached here is running Oracle HotSpot.  I am going to install an Oracle VM and see if I can reproduce the problem.  Please could you try an IBM VM and indicate whether you still are able to crash?


My VM:
java version "1.7.0"
Java(TM) SE Runtime Environment (build pxa6470sr1-20120330_01(SR1))
IBM J9 VM (build 2.6, JRE 1.7.0 Linux amd64-64 20120322_106209 (JIT enabled, AOT enabled)
J9VM - R26_Java726_SR1_20120322_1720_B106209
JIT  - r11_20120322_22976
GC   - R26_Java726_SR1_20120322_1720_B106209
J9CL - 20120322_106209)
JCL - 20120322_01 based on Oracle 7u3-b05
Comment 24 Andrey Loskutov CLA 2012-08-22 17:07:53 EDT
(In reply to comment #23)
> I have been running Eclipse on RHEL 5.8 64bit for several days and I have
> not crashed once.  I am using an IBM JRE. It seems that every crash log
> attached here is running Oracle HotSpot.  I am going to install an Oracle VM
> and see if I can reproduce the problem.  Please could you try an IBM VM and
> indicate whether you still are able to crash?

I will do tomorrow, and report in a week, as we still cannot *enforce* crashes (e.g it crashed 3 times in a row this Monday but no crashes so far for two days with unchanged environment).

We have tried 1.6.0_16, 1.7.0_04 from Sun/Oracle with identical results (crashes), but only Eclipse 3.8 is crashing with SIGSEGV in ld-linux,  never 3.7.0 or 3.7.2. So *if* this is JDK related, why it crashes with 3.8/4.2 only? Any idea?
Comment 25 Dennis Hendriks CLA 2012-08-23 04:05:04 EDT
I have similar issues to the issues described in this bug. I created Bug 387377, but it was closed as a duplicate. I though I'd share my system details, to see what the commonalities/differences are:

Eclipse Platform Version: 4.2.0 Build id: I20120608-1400

Eclipse Modeling Tools (Linux x86), with:
  ATL SDK - ATLAS Transformation Language SDK	3.3.0.v201205241358	org.eclipse.m2m.atl.sdk.feature.group	Eclipse Modeling Project
  Buckminster - Core	1.5.0.v20120628-1017	org.eclipse.buckminster.core.feature.feature.group	Eclipse Buckminster Project
  Buckminster - PDE support	1.5.0.v20120628-1241	org.eclipse.buckminster.pde.feature.feature.group	Eclipse Buckminster Project
  Buckminster - Subversive support	1.5.0.v20120603-0635	org.eclipse.buckminster.subversive.feature.feature.group	Eclipse Buckminster Project
  Eclipse Modeling Tools	1.5.0.20120620-0855	epp.package.modeling	null
  EMFText Access	1.4.0.v201107221427	org.emftext.access.feature.group	Software Technology Group - TU Dresden
  EMFText SDK	1.4.0.v201107221427	org.emftext.sdk.feature.group	Software Technology Group - TU Dresden
  EMFText Shared ANTLR 3.3.0 Runtime	3.3.0.v201107221427	org.emftext.commons.antlr3_3_0.feature.group	Software Technology Group - TU Dresden
  Graphical Modeling Framework (GMF) Tooling SDK	3.0.0.v201206221900	org.eclipse.gmf.sdk.feature.group	Eclipse Modeling Project
  OCL Examples and Editors	3.2.0.v20120613-0850	org.eclipse.ocl.examples.feature.group	Eclipse Modeling Project
  Operational QVT SDK	3.1.0.v201206041614-7U7A-CL5V0ETk4K7wL253-yIi90n	org.eclipse.m2m.qvt.oml.sdk.feature.group	Eclipse Modeling Project
  Xpand SDK	1.2.1.v201206110941	org.eclipse.xpand.sdk.feature.group	Eclipse Modeling Project
  Xtext SDK	2.3.0.v201206120633	org.eclipse.xtext.sdk.feature.group	Eclipse Modeling Project

$ uname -a
Linux se-168.se.wtb.tue.nl 2.6.18-308.13.1.el5PAE #1 SMP Tue Aug 21 17:50:26 EDT 2012 i686 i686 i386 GNU/Linux

$ cat /etc/redhat-release
CentOS release 5.8 (Final)

$ java -version
java version "1.6.0_25"
Java(TM) SE Runtime Environment (build 1.6.0_25-b06)
Java HotSpot(TM) Server VM (build 20.0-b11, mixed mode)

$ file /usr/java/jdk1.6.0_25/jre/bin/java
/usr/java/jdk1.6.0_25/jre/bin/java: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), not stripped

$ rpm -q cairo
cairo-1.2.4-5.el5

$ rpm -q gtk2
gtk2-2.10.4-21.el5_7.7

$ rpm -q glibc
glibc-2.5-81.el5_8.4

$ rpm -q glib
glib-1.2.10-20.el5

$ rpm -q pango
pango-1.14.9-8.el5.centos.3

$ rpm -q ORBit2
ORBit2-2.14.3-5.el5

$ grep   "\#\ \C\ \ \[" hs_err_pid*.log
hs_err_pid10251.log:# C  [ld-linux.so.2+0x94e1]
hs_err_pid10286.log:# C  [ld-linux.so.2+0x94e1]
hs_err_pid10775.log:# C  [ld-linux.so.2+0x94e1]
hs_err_pid11283.log:# C  [ld-linux.so.2+0x94e1]
(... 30 logs with .so same file and address omitted ...)
hs_err_pid8199.log:# C  [ld-linux.so.2+0x94e1]
hs_err_pid8252.log:# C  [ld-linux.so.2+0x94e1]
hs_err_pid8567.log:# C  [ld-linux.so.2+0x94e1]
hs_err_pid8787.log:# C  [ld-linux.so.2+0x94e1]
hs_err_pid9255.log:# C  [ld-linux.so.2+0x94e1]

To me, from the JVM crash logs, it seems to have nothing to do with ORBit...
Comment 26 Silenio Quarti CLA 2012-08-23 11:06:31 EDT
(In reply to comment #24)
> We have tried 1.6.0_16, 1.7.0_04 from Sun/Oracle with identical results
> (crashes), but only Eclipse 3.8 is crashing with SIGSEGV in ld-linux,  never
> 3.7.0 or 3.7.2. So *if* this is JDK related, why it crashes with 3.8/4.2
> only? Any idea?

No ideas. I am only trying to reproduce the problem.  The idea was to reproduce it in a consistent way and then revert back the SWT code from 3.8 to 3.7.2 to try to find the change that caused it.

The stacks are funny, the native OS._gtk_im_multicontext_append_menuitems() for example only calls the GTK API gtk_im_multicontext_append_menuitems(). It seems that we crash before calling the GTK API while the VM is looking up the SWT native implementation or after calling the GTK API.   Either way, the crash seems to be inside the VM.  So it could be a bug in the VM.  But it could also be some memory trashing that happened earlier. Or even a JIT problem. 


C  [ld-linux.so.2+0x94e1]
C  [ld-linux.so.2+0x98d7]
C  [ld-linux.so.2+0x9a77]
C  [ld-linux.so.2+0xde48]
C  [ld-linux.so.2+0x13610]
j  org.eclipse.swt.internal.gtk.OS._gtk_im_multicontext_append_menuitems(II)V+0
Comment 27 Andrey Loskutov CLA 2012-08-23 14:06:55 EDT
Created attachment 220217 [details]
IBM crash logs on shutdown

I've got my first two crashes with IBM JRE, both on a regular Eclipse shutdown, both with core dumps (can provide if needed). I had two Eclipse instances whole day running fine, but both crashed one after each other on closing workbench. Funny enough, one of them has libORBit-2.so in the immediate crash stack.

I will continue to run with IBM JRE in a hope to get crash during "usual" work (not on shutdown).
Comment 28 Andrey Loskutov CLA 2012-08-24 02:50:57 EDT
Created attachment 220238 [details]
IBM crash logs on expanding the package explorer

Ok,
few minutes after starting my 3 Eclipse workspaces I was switching between them and just after switch, while expanding the node in Package Explorer Eclipse crashed with IBM JRE. So it is definitely not a VM bug, as we observe it now on 3 different VM's (Sun 1.6, Oracle and IBM 1.7).

I have to correct myself: last 2 crash files are from the single crash, as it looks like IBM VM creates two dumps on a crash.

So now I've attached two crash dumps from the *same* crash. The relevant part of the stack trace is below, but it's still crashing in native ld-linux code:

1XMCURTHDINFO  Current thread
NULL           ----------------------
3XMTHREADINFO      "main" J9VMThread:0x0000000019469D00, j9thread_t:0x0000000019408B10, java/lang/Thread:0x00002AAAACBA5C90, state:R, prio=6
3XMTHREADINFO1            (native thread ID:0x7271, native priority:0x6, native policy:UNKNOWN)
3XMTHREADINFO2            (native stack address range from:0x00000000419C6000, to:0x00000000423C7000, size:0xA01000)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/lang/ClassLoader.loadLibraryWithPath(Native Method)
4XESTACKTRACE                at java/lang/ClassLoader.loadLibraryWithPath(ClassLoader.java:1157)
4XESTACKTRACE                at java/lang/ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1125)
5XESTACKTRACE                   (entered lock: java/lang/ClassLoader@0x00002AAAACB71160, entry count: 1)
4XESTACKTRACE                at java/lang/System.loadLibrary(System.java:491)
4XESTACKTRACE                at org/eclipse/swt/internal/Library.load(Library.java:216)
4XESTACKTRACE                at org/eclipse/swt/internal/Library.loadLibrary(Library.java:302)
4XESTACKTRACE                at org/eclipse/swt/internal/Library.loadLibrary(Library.java:240)
4XESTACKTRACE                at org/eclipse/swt/internal/gnome/GNOME.<clinit>(GNOME.java:21)
4XESTACKTRACE                at java/lang/J9VMInternals.initializeImpl(Native Method)
4XESTACKTRACE                at java/lang/J9VMInternals.initialize(J9VMInternals.java:228(Compiled Code))
4XESTACKTRACE                at org/eclipse/swt/program/Program.gnome_init(Program.java:566)
4XESTACKTRACE                at org/eclipse/swt/program/Program.getDesktop(Program.java:125)
4XESTACKTRACE                at org/eclipse/swt/program/Program.findProgram(Program.java:616)
4XESTACKTRACE                at org/eclipse/swt/program/Program.findProgram(Program.java:605)
4XESTACKTRACE                at org/eclipse/ui/internal/registry/EditorRegistry.getSystemExternalEditorImageDescriptor(EditorRegistry.java:1272)
4XESTACKTRACE                at org/eclipse/ui/internal/registry/EditorRegistry.getImageDescriptor(EditorRegistry.java:1480)
4XESTACKTRACE                at org/eclipse/ui/internal/ide/model/WorkbenchFile.getBaseImage(WorkbenchFile.java:63)
4XESTACKTRACE                at org/eclipse/ui/internal/ide/model/WorkbenchResource.getImageDescriptor(WorkbenchResource.java:42(Compiled Code))
4XESTACKTRACE                at org/eclipse/jdt/internal/ui/viewsupport/JavaElementImageProvider.getWorkbenchImageDescriptor(JavaElementImageProvider.java:182(Compiled Code))
4XESTACKTRACE                at org/eclipse/jdt/internal/ui/viewsupport/JavaElementImageProvider.computeDescriptor(JavaElementImageProvider.java:122(Compiled Code))
4XESTACKTRACE                at org/eclipse/jdt/internal/ui/viewsupport/JavaElementImageProvider.getImageLabel(JavaElementImageProvider.java:97(Compiled Code))
4XESTACKTRACE                at org/eclipse/jdt/internal/ui/viewsupport/JavaUILabelProvider.getImage(JavaUILabelProvider.java:144(Compiled Code))
4XESTACKTRACE                at org/eclipse/jdt/internal/ui/packageview/PackageExplorerLabelProvider.getImage(PackageExplorerLabelProvider.java:140(Compiled Code))
4XESTACKTRACE                at org/eclipse/jface/viewers/DelegatingStyledCellLabelProvider.getImage(DelegatingStyledCellLabelProvider.java:184(Compiled Code))
4XESTACKTRACE                at org/eclipse/jface/viewers/DecoratingStyledCellLabelProvider.getImage(DecoratingStyledCellLabelProvider.java:167)
4XESTACKTRACE                at org/eclipse/jface/viewers/DelegatingStyledCellLabelProvider.update(DelegatingStyledCellLabelProvider.java:118)
4XESTACKTRACE                at org/eclipse/jface/viewers/DecoratingStyledCellLabelProvider.update(DecoratingStyledCellLabelProvider.java:134(Compiled Code))
4XESTACKTRACE                at org/eclipse/jface/viewers/ViewerColumn.refresh(ViewerColumn.java:152(Compiled Code))
4XESTACKTRACE                at org/eclipse/jface/viewers/AbstractTreeViewer.doUpdateItem(AbstractTreeViewer.java:953(Compiled Code))
4XESTACKTRACE                at org/eclipse/jface/viewers/AbstractTreeViewer$UpdateItemSafeRunnable.run(AbstractTreeViewer.java:113(Compiled Code))
4XESTACKTRACE                at org/eclipse/core/runtime/SafeRunner.run(SafeRunner.java:42(Compiled Code))
4XESTACKTRACE                at org/eclipse/ui/internal/JFaceUtil$1.run(JFaceUtil.java:49(Compiled Code))
4XESTACKTRACE                at org/eclipse/jface/util/SafeRunnable.run(SafeRunnable.java:175(Compiled Code))
4XESTACKTRACE                at org/eclipse/jface/viewers/AbstractTreeViewer.doUpdateItem(AbstractTreeViewer.java:1033(Compiled Code))
4XESTACKTRACE                at org/eclipse/jface/viewers/StructuredViewer$UpdateItemSafeRunnable.run(StructuredViewer.java:485)
4XESTACKTRACE                at org/eclipse/core/runtime/SafeRunner.run(SafeRunner.java:42(Compiled Code))
4XESTACKTRACE                at org/eclipse/ui/internal/JFaceUtil$1.run(JFaceUtil.java:49(Compiled Code))
4XESTACKTRACE                at org/eclipse/jface/util/SafeRunnable.run(SafeRunnable.java:175(Compiled Code))
4XESTACKTRACE                at org/eclipse/jface/viewers/StructuredViewer.updateItem(StructuredViewer.java:2167)
4XESTACKTRACE                at org/eclipse/jface/viewers/AbstractTreeViewer.createTreeItem(AbstractTreeViewer.java:848)
4XESTACKTRACE                at org/eclipse/jface/viewers/AbstractTreeViewer$1.run(AbstractTreeViewer.java:823)
4XESTACKTRACE                at org/eclipse/swt/custom/BusyIndicator.showWhile(BusyIndicator.java:70)
4XESTACKTRACE                at org/eclipse/jface/viewers/AbstractTreeViewer.createChildren(AbstractTreeViewer.java:797)
4XESTACKTRACE                at org/eclipse/jface/viewers/TreeViewer.createChildren(TreeViewer.java:644)
4XESTACKTRACE                at org/eclipse/jface/viewers/AbstractTreeViewer.createChildren(AbstractTreeViewer.java:768)
4XESTACKTRACE                at org/eclipse/jface/viewers/AbstractTreeViewer.handleTreeExpand(AbstractTreeViewer.java:1500)
4XESTACKTRACE                at org/eclipse/jface/viewers/TreeViewer.handleTreeExpand(TreeViewer.java:952)
4XESTACKTRACE                at org/eclipse/jface/viewers/AbstractTreeViewer$4.treeExpanded(AbstractTreeViewer.java:1511)
4XESTACKTRACE                at org/eclipse/swt/widgets/TypedListener.handleEvent(TypedListener.java:132(Compiled Code))
4XESTACKTRACE                at org/eclipse/swt/widgets/EventTable.sendEvent(EventTable.java:84(Compiled Code))
4XESTACKTRACE                at org/eclipse/swt/widgets/Widget.sendEvent(Widget.java:1276(Compiled Code))
4XESTACKTRACE                at org/eclipse/swt/widgets/Widget.sendEvent(Widget.java:1300(Compiled Code))
4XESTACKTRACE                at org/eclipse/swt/widgets/Widget.sendEvent(Widget.java:1285(Compiled Code))
4XESTACKTRACE                at org/eclipse/swt/widgets/Tree.gtk_test_expand_row(Tree.java:2051)
4XESTACKTRACE                at org/eclipse/swt/widgets/Widget.windowProc(Widget.java:1790)
4XESTACKTRACE                at org/eclipse/swt/widgets/Display.windowProc(Display.java:4375)
4XESTACKTRACE                at org/eclipse/swt/internal/gtk/OS._gtk_main_do_event(Native Method)
4XESTACKTRACE                at org/eclipse/swt/internal/gtk/OS.gtk_main_do_event(OS.java:8295(Compiled Code))
4XESTACKTRACE                at org/eclipse/swt/widgets/Display.eventProc(Display.java:1192(Compiled Code))
4XESTACKTRACE                at org/eclipse/swt/internal/gtk/OS._g_main_context_iteration(Native Method)
4XESTACKTRACE                at org/eclipse/swt/internal/gtk/OS.g_main_context_iteration(OS.java:2332(Compiled Code))
4XESTACKTRACE                at org/eclipse/swt/widgets/Display.readAndDispatch(Display.java:3177(Compiled Code))
4XESTACKTRACE                at org/eclipse/ui/internal/Workbench.runEventLoop(Workbench.java:2701)
4XESTACKTRACE                at org/eclipse/ui/internal/Workbench.runUI(Workbench.java:2665)
4XESTACKTRACE                at org/eclipse/ui/internal/Workbench.access$4(Workbench.java:2499)
4XESTACKTRACE                at org/eclipse/ui/internal/Workbench$7.run(Workbench.java:679)
4XESTACKTRACE                at org/eclipse/core/databinding/observable/Realm.runWithDefault(Realm.java:332)
4XESTACKTRACE                at org/eclipse/ui/internal/Workbench.createAndRunWorkbench(Workbench.java:668)
4XESTACKTRACE                at org/eclipse/ui/PlatformUI.createAndRunWorkbench(PlatformUI.java:149)
4XESTACKTRACE                at org/eclipse/ui/internal/ide/application/IDEApplication.start(IDEApplication.java:124)
4XESTACKTRACE                at org/eclipse/equinox/internal/app/EclipseAppHandle.run(EclipseAppHandle.java:196)
4XESTACKTRACE                at org/eclipse/core/runtime/internal/adaptor/EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
4XESTACKTRACE                at org/eclipse/core/runtime/internal/adaptor/EclipseAppLauncher.start(EclipseAppLauncher.java:79)
4XESTACKTRACE                at org/eclipse/core/runtime/adaptor/EclipseStarter.run(EclipseStarter.java:353)
4XESTACKTRACE                at org/eclipse/core/runtime/adaptor/EclipseStarter.run(EclipseStarter.java:180)
4XESTACKTRACE                at sun/reflect/NativeMethodAccessorImpl.invoke0(Native Method)
4XESTACKTRACE                at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
4XESTACKTRACE                at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
4XESTACKTRACE                at java/lang/reflect/Method.invoke(Method.java:613)
4XESTACKTRACE                at org/eclipse/equinox/launcher/Main.invokeFramework(Main.java:629)
4XESTACKTRACE                at org/eclipse/equinox/launcher/Main.basicRun(Main.java:584)
4XESTACKTRACE                at org/eclipse/equinox/launcher/Main.run(Main.java:1438)
4XESTACKTRACE                at org/eclipse/equinox/launcher/Main.main(Main.java:1414)
3XMTHREADINFO3           Native callstack:
4XENATIVESTACK               (0x00002AAAAB098762 [libj9prt26.so+0x12762])
4XENATIVESTACK               (0x00002AAAAB0A5CBF [libj9prt26.so+0x1fcbf])
4XENATIVESTACK               (0x00002AAAAB0984AB [libj9prt26.so+0x124ab])
4XENATIVESTACK               (0x00002AAAAB0985A7 [libj9prt26.so+0x125a7])
4XENATIVESTACK               (0x00002AAAAB0A5CBF [libj9prt26.so+0x1fcbf])
4XENATIVESTACK               (0x00002AAAAB0980CB [libj9prt26.so+0x120cb])
4XENATIVESTACK               (0x00002AAAAB091639 [libj9prt26.so+0xb639])
4XENATIVESTACK               (0x00002AAAAB092EF4 [libj9prt26.so+0xcef4])
4XENATIVESTACK               (0x00002AAAAB0A5CBF [libj9prt26.so+0x1fcbf])
4XENATIVESTACK               (0x00002AAAAB57C329 [libj9dmp26.so+0x15329])
4XENATIVESTACK               (0x00002AAAAB5769CA [libj9dmp26.so+0xf9ca])
4XENATIVESTACK               (0x00002AAAAB0A5CBF [libj9prt26.so+0x1fcbf])
4XENATIVESTACK               (0x00002AAAAB57DBBE [libj9dmp26.so+0x16bbe])
4XENATIVESTACK               (0x00002AAAAB57DED9 [libj9dmp26.so+0x16ed9])
4XENATIVESTACK               (0x00002AAAAB56C8D8 [libj9dmp26.so+0x58d8])
4XENATIVESTACK               (0x00002AAAAB0A5CBF [libj9prt26.so+0x1fcbf])
4XENATIVESTACK               (0x00002AAAAB56B088 [libj9dmp26.so+0x4088])
4XENATIVESTACK               (0x00002AAAAB56F818 [libj9dmp26.so+0x8818])
4XENATIVESTACK               (0x00002AAAAB580032 [libj9dmp26.so+0x19032])
4XENATIVESTACK               (0x00002AAAAB570BCC [libj9dmp26.so+0x9bcc])
4XENATIVESTACK               (0x000000305C20EBE0 [libpthread.so.0+0xebe0])
4XENATIVESTACK               gsignal+0x35 (0x000000305B230265 [libc.so.6+0x30265])
4XENATIVESTACK               abort+0x110 (0x000000305B231D10 [libc.so.6+0x31d10])
4XENATIVESTACK               (0x00002AAAAB0A5161 [libj9prt26.so+0x1f161])
4XENATIVESTACK               (0x000000305C20EBE0 [libpthread.so.0+0xebe0])
4XENATIVESTACK               (0x000000305AC08C0B [ld-linux-x86-64.so.2+0x8c0b])
4XENATIVESTACK               (0x000000305AC09029 [ld-linux-x86-64.so.2+0x9029])
4XENATIVESTACK               (0x000000305AC092B2 [ld-linux-x86-64.so.2+0x92b2])
4XENATIVESTACK               (0x000000305AC0CF65 [ld-linux-x86-64.so.2+0xcf65])
4XENATIVESTACK               (0x000000305AC129E2 [ld-linux-x86-64.so.2+0x129e2])
4XENATIVESTACK               (0x0000003069C2D932 [libORBit-2.so.0+0x2d932])
4XENATIVESTACK               exit+0xe5 (0x000000305B2334F5 [libc.so.6+0x334f5])
4XENATIVESTACK               (0x00002AAAAB08DBDF [libj9prt26.so+0x7bdf])
4XENATIVESTACK               (0x00002AAAAACF7070 [libj9vm26.so+0x1a070])
4XENATIVESTACK               (0x00002AAAAACF7851 [libj9vm26.so+0x1a851])
4XENATIVESTACK               (0x00002AAAAB0A50DF [libj9prt26.so+0x1f0df])
4XENATIVESTACK               (0x000000305C20EBE0 [libpthread.so.0+0xebe0])
4XENATIVESTACK               (0x000000305AC08C0B [ld-linux-x86-64.so.2+0x8c0b])
4XENATIVESTACK               0x000000305AC09029
Comment 29 Andrey Loskutov CLA 2012-08-24 04:43:54 EDT
Created attachment 220244 [details]
IBM crash logs on editor switch

2 hours later next Eclipse workbench crashed, with ld-linux but on switching editors (Java stack is different as usually, native stack same).

I'm giving up with IBM JRE, my home is full with core dumps and I'm expecting  admins to call me soon because of disk quota (IBM JRE seems to inore "no core dumps" setting from Linux).

Relevant stack part:

1XMCURTHDINFO  Current thread
NULL           ----------------------
3XMTHREADINFO      "main" J9VMThread:0x000000000DD7BD00, j9thread_t:0x000000000DD1AB10, java/lang/Thread:0x00002AAAACB99FD8, state:R, prio=6
3XMTHREADINFO1            (native thread ID:0x1215, native priority:0x6, native policy:UNKNOWN)
3XMTHREADINFO2            (native stack address range from:0x0000000041AAF000, to:0x00000000424B0000, size:0xA01000)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at org/eclipse/swt/internal/gtk/OS._gdk_gc_get_values(Native Method)
4XESTACKTRACE                at org/eclipse/swt/internal/gtk/OS.gdk_gc_get_values(OS.java:4275)
4XESTACKTRACE                at org/eclipse/swt/widgets/Composite.drawBackground(Composite.java:408)
4XESTACKTRACE                at org/eclipse/swt/widgets/Canvas.drawBackground(Canvas.java:99)
4XESTACKTRACE                at org/eclipse/swt/custom/StyledTextRenderer.drawLine(StyledTextRenderer.java:384)
4XESTACKTRACE                at org/eclipse/swt/custom/StyledText.handlePaint(StyledText.java:6084)
4XESTACKTRACE                at org/eclipse/swt/custom/StyledText$7.handleEvent(StyledText.java:5635)
4XESTACKTRACE                at org/eclipse/swt/widgets/EventTable.sendEvent(EventTable.java:84(Compiled Code))
4XESTACKTRACE                at org/eclipse/swt/widgets/Widget.sendEvent(Widget.java:1276(Compiled Code))
4XESTACKTRACE                at org/eclipse/swt/widgets/Widget.sendEvent(Widget.java:1300(Compiled Code))
4XESTACKTRACE                at org/eclipse/swt/widgets/Widget.sendEvent(Widget.java:1285(Compiled Code))
4XESTACKTRACE                at org/eclipse/swt/widgets/Control.gtk_expose_event(Control.java:2992)
4XESTACKTRACE                at org/eclipse/swt/widgets/Composite.gtk_expose_event(Composite.java:706)
4XESTACKTRACE                at org/eclipse/swt/widgets/Canvas.gtk_expose_event(Canvas.java:167)
4XESTACKTRACE                at org/eclipse/swt/widgets/Widget.windowProc(Widget.java:1754(Compiled Code))
4XESTACKTRACE                at org/eclipse/swt/widgets/Control.windowProc(Control.java:5116(Compiled Code))
4XESTACKTRACE                at org/eclipse/swt/widgets/Display.windowProc(Display.java:4369(Compiled Code))
4XESTACKTRACE                at org/eclipse/swt/internal/gtk/OS._gtk_main_do_event(Native Method)
4XESTACKTRACE                at org/eclipse/swt/internal/gtk/OS.gtk_main_do_event(OS.java:8295(Compiled Code))
4XESTACKTRACE                at org/eclipse/swt/widgets/Display.eventProc(Display.java:1192(Compiled Code))
4XESTACKTRACE                at org/eclipse/swt/internal/gtk/OS._g_main_context_iteration(Native Method)
4XESTACKTRACE                at org/eclipse/swt/internal/gtk/OS.g_main_context_iteration(OS.java:2332(Compiled Code))
4XESTACKTRACE                at org/eclipse/swt/widgets/Shell.setVisible(Shell.java:2055)
4XESTACKTRACE                at org/eclipse/jface/text/AbstractInformationControl.setVisible(AbstractInformationControl.java:505)
4XESTACKTRACE                at org/eclipse/jface/text/DefaultInformationControl.setVisible(DefaultInformationControl.java:363)
4XESTACKTRACE                at org/eclipse/jface/text/AbstractInformationControlManager.showInformationControl(AbstractInformationControlManager.java:1270)
4XESTACKTRACE                at org/eclipse/jface/text/TextViewerHoverManager.showInformationControl(TextViewerHoverManager.java:283)
4XESTACKTRACE                at org/eclipse/jface/text/AbstractInformationControlManager.internalShowInformationControl(AbstractInformationControlManager.java:1221)
4XESTACKTRACE                at org/eclipse/jface/text/AbstractInformationControlManager.presentInformation(AbstractInformationControlManager.java:1150)
4XESTACKTRACE                at org/eclipse/jface/text/AbstractHoverInformationControlManager.presentInformation(AbstractHoverInformationControlManager.java:902)
4XESTACKTRACE                at org/eclipse/jface/text/TextViewerHoverManager.doPresentInformation(TextViewerHoverManager.java:243)
4XESTACKTRACE                at org/eclipse/jface/text/TextViewerHoverManager$5.run(TextViewerHoverManager.java:233)
4XESTACKTRACE                at org/eclipse/swt/widgets/RunnableLock.run(RunnableLock.java:35(Compiled Code))
4XESTACKTRACE                at org/eclipse/swt/widgets/Synchronizer.runAsyncMessages(Synchronizer.java:135(Compiled Code))
5XESTACKTRACE                   (entered lock: org/eclipse/swt/widgets/RunnableLock@0x00002AAB673589C0, entry count: 1)
4XESTACKTRACE                at org/eclipse/swt/widgets/Display.runAsyncMessages(Display.java:3529(Compiled Code))
4XESTACKTRACE                at org/eclipse/swt/widgets/Display.readAndDispatch(Display.java:3182(Compiled Code))
4XESTACKTRACE                at org/eclipse/ui/internal/Workbench.runEventLoop(Workbench.java:2701)
4XESTACKTRACE                at org/eclipse/ui/internal/Workbench.runUI(Workbench.java:2665)
4XESTACKTRACE                at org/eclipse/ui/internal/Workbench.access$4(Workbench.java:2499)
4XESTACKTRACE                at org/eclipse/ui/internal/Workbench$7.run(Workbench.java:679)
4XESTACKTRACE                at org/eclipse/core/databinding/observable/Realm.runWithDefault(Realm.java:332)
4XESTACKTRACE                at org/eclipse/ui/internal/Workbench.createAndRunWorkbench(Workbench.java:668)
4XESTACKTRACE                at org/eclipse/ui/PlatformUI.createAndRunWorkbench(PlatformUI.java:149)
4XESTACKTRACE                at org/eclipse/ui/internal/ide/application/IDEApplication.start(IDEApplication.java:124)
4XESTACKTRACE                at org/eclipse/equinox/internal/app/EclipseAppHandle.run(EclipseAppHandle.java:196)
4XESTACKTRACE                at org/eclipse/core/runtime/internal/adaptor/EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
4XESTACKTRACE                at org/eclipse/core/runtime/internal/adaptor/EclipseAppLauncher.start(EclipseAppLauncher.java:79)
4XESTACKTRACE                at org/eclipse/core/runtime/adaptor/EclipseStarter.run(EclipseStarter.java:353)
4XESTACKTRACE                at org/eclipse/core/runtime/adaptor/EclipseStarter.run(EclipseStarter.java:180)
4XESTACKTRACE                at sun/reflect/NativeMethodAccessorImpl.invoke0(Native Method)
4XESTACKTRACE                at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
4XESTACKTRACE                at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
4XESTACKTRACE                at java/lang/reflect/Method.invoke(Method.java:613)
4XESTACKTRACE                at org/eclipse/equinox/launcher/Main.invokeFramework(Main.java:629)
4XESTACKTRACE                at org/eclipse/equinox/launcher/Main.basicRun(Main.java:584)
4XESTACKTRACE                at org/eclipse/equinox/launcher/Main.run(Main.java:1438)
4XESTACKTRACE                at org/eclipse/equinox/launcher/Main.main(Main.java:1414)
3XMTHREADINFO3           Native callstack:
4XENATIVESTACK               (0x00002AAAAB098762 [libj9prt26.so+0x12762])
4XENATIVESTACK               (0x00002AAAAB0A5CBF [libj9prt26.so+0x1fcbf])
4XENATIVESTACK               (0x00002AAAAB0984AB [libj9prt26.so+0x124ab])
4XENATIVESTACK               (0x00002AAAAB0985A7 [libj9prt26.so+0x125a7])
4XENATIVESTACK               (0x00002AAAAB0A5CBF [libj9prt26.so+0x1fcbf])
4XENATIVESTACK               (0x00002AAAAB0980CB [libj9prt26.so+0x120cb])
4XENATIVESTACK               (0x00002AAAAB091639 [libj9prt26.so+0xb639])
4XENATIVESTACK               (0x00002AAAAB092EF4 [libj9prt26.so+0xcef4])
4XENATIVESTACK               (0x00002AAAAB0A5CBF [libj9prt26.so+0x1fcbf])
4XENATIVESTACK               (0x00002AAAAB57C329 [libj9dmp26.so+0x15329])
4XENATIVESTACK               (0x00002AAAAB5769CA [libj9dmp26.so+0xf9ca])
4XENATIVESTACK               (0x00002AAAAB0A5CBF [libj9prt26.so+0x1fcbf])
4XENATIVESTACK               (0x00002AAAAB57DBBE [libj9dmp26.so+0x16bbe])
4XENATIVESTACK               (0x00002AAAAB57DED9 [libj9dmp26.so+0x16ed9])
4XENATIVESTACK               (0x00002AAAAB56C8D8 [libj9dmp26.so+0x58d8])
4XENATIVESTACK               (0x00002AAAAB0A5CBF [libj9prt26.so+0x1fcbf])
4XENATIVESTACK               (0x00002AAAAB56B088 [libj9dmp26.so+0x4088])
4XENATIVESTACK               (0x00002AAAAB56F818 [libj9dmp26.so+0x8818])
4XENATIVESTACK               (0x00002AAAAB580032 [libj9dmp26.so+0x19032])
4XENATIVESTACK               (0x00002AAAAACF6DD9 [libj9vm26.so+0x19dd9])
4XENATIVESTACK               (0x00002AAAAB0A5CBF [libj9prt26.so+0x1fcbf])
4XENATIVESTACK               (0x00002AAAAACF6FF6 [libj9vm26.so+0x19ff6])
4XENATIVESTACK               (0x00002AAAAACF7851 [libj9vm26.so+0x1a851])
4XENATIVESTACK               (0x00002AAAAB0A50DF [libj9prt26.so+0x1f0df])
4XENATIVESTACK               (0x000000305C20EBE0 [libpthread.so.0+0xebe0])
4XENATIVESTACK               (0x000000305AC08C0B [ld-linux-x86-64.so.2+0x8c0b])
4XENATIVESTACK               (0x000000305AC09029 [ld-linux-x86-64.so.2+0x9029])
4XENATIVESTACK               (0x000000305AC092B2 [ld-linux-x86-64.so.2+0x92b2])
4XENATIVESTACK               (0x000000305AC0CF65 [ld-linux-x86-64.so.2+0xcf65])
4XENATIVESTACK               (0x000000305AC129E2 [ld-linux-x86-64.so.2+0x129e2])
4XENATIVESTACK               setGdkGCValuesFields+0x24 (0x00002AAB6E31052E [libswt-pi-gtk-3833.so+0x5c52e])
4XENATIVESTACK               Java_org_eclipse_swt_internal_gtk_OS__1gdk_1gc_1get_1values+0x57 (0x00002AAB6E2F836D [libswt-pi-gtk-3833.so+0x4436d])
4XENATIVESTACK               (0x00002AAAAAD0FBF2 [libj9vm26.so+0x32bf2])
NULL
Comment 30 thomas lafarge CLA 2012-08-24 09:54:12 EDT
Hi everyone,
I have the same problem here.

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000003f11608c0b, pid=6165, tid=1093544256
#
# JRE version: 6.0_22-b22
# Java VM: OpenJDK 64-Bit Server VM (20.0-b11 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea6 1.10.8
# Distribution: CentOS release 5.8 (Final), package rhel-1.27.1.10.8.el5_8-x86_64
# Problematic frame:
# C  [ld-linux-x86-64.so.2+0x8c0b]
#

I use eclipse Juno (4.2)
I can still use eclipse indigo without crashing.

My system is :

$ uname -a
Linux h048068.nist.gov 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
 java -version
java version "1.6.0_22"
OpenJDK Runtime Environment (IcedTea6 1.10.8) (rhel-1.27.1.10.8.el5_8-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode)

$ rpm -q glibc
glibc-2.5-81.el5_8.4.x86_64
glibc-2.5-81.el5_8.4.i686

$ rpm -q glib
glib-1.2.10-20.el5.x86_64

$ rpm -q pango
pango-1.14.9-8.el5.centos.3.x86_64
pango-1.14.9-8.el5.centos.3.i386

$ rpm -q cairo
cairo-1.2.4-5.el5.x86_64
cairo-1.2.4-5.el5.i386

$ rpm -q gtk2
gtk2-2.10.4-21.el5_7.7.x86_64
gtk2-2.10.4-21.el5_7.7.i386

Feel free to ask me if you want more information. It usually crashs between 5 and 30 minutes the same way it is described in this thread.

Any idea what it could be? Which additional information you would like to see?
Comment 31 Andrey Loskutov CLA 2012-08-24 10:12:44 EDT
3 different reporters observed this crash were using same kernel version (2.6.18-308 both 32/64 bits). This is RHEL 5.8 / CentOS 5.8 kernel.

Question is: are there *new or different* native libraries / compile flags / header files used to compile native SWT/equinox bits for 3.8 / 4.2 SWT stream,   which might be not supported by ancient libc/kernel we have?

I also know that there were a lot of SWT changes due the shift to the new Cairo/GTK3 stack (see work in the bug 340067). Is it something in this area? Some new Cairo/GTK dependency we can't properly satisfy or which is not properly working with our ancient Cairo/GTK?
Comment 32 Andrey Loskutov CLA 2012-08-24 10:19:52 EDT
Just another idea - we are running Eclipse on KDE, which is of course not "native" desktop for Gnome applications like Eclipse. 

As Silenio can't reproduce the crash, I'm wondering if this is desktop related?

@Thomas, Dennis, Silenio: which desktops are you using?
Comment 33 Silenio Quarti CLA 2012-08-24 11:37:50 EDT
(In reply to comment #31)
> 3 different reporters observed this crash were using same kernel version
> (2.6.18-308 both 32/64 bits). This is RHEL 5.8 / CentOS 5.8 kernel.
> 
> Question is: are there *new or different* native libraries / compile flags /
> header files used to compile native SWT/equinox bits for 3.8 / 4.2 SWT
> stream,   which might be not supported by ancient libc/kernel we have?
> 

We compile the SWT libraries on RedHat 4.

> I also know that there were a lot of SWT changes due the shift to the new
> Cairo/GTK3 stack (see work in the bug 340067). Is it something in this area?
> Some new Cairo/GTK dependency we can't properly satisfy or which is not
> properly working with our ancient Cairo/GTK?

We should not be running the new cairo code since it only runs for gtk 2.24 (you have 2.10).  The other changes for GTK 3 support could be the reason.  We also fixed many places we were supposedly leaking in Tree/Table.  Maybe we were wrong and we are double freeing now.  Google indicates that a crash in _dl_fixup() usually means we are trashing memory.  Comment#8  indicates some problem with attribute lists, we have related changes in TextLayout.  I have reviewed the changes between 3.7 and 3.8 twice already, but I could not find anything obviously wrong.
Comment 34 Silenio Quarti CLA 2012-08-24 12:04:55 EDT
(In reply to comment #32)
> Just another idea - we are running Eclipse on KDE, which is of course not
> "native" desktop for Gnome applications like Eclipse. 
> 
> As Silenio can't reproduce the crash, I'm wondering if this is desktop
> related?
> 
> @Thomas, Dennis, Silenio: which desktops are you using?

Gnome.

David, do you know if the build machine uses KDE?
Comment 35 Markus Keller CLA 2012-08-24 12:28:31 EDT
Until recently, the test machine didn't even run a window manager. Now, the test script starts metacity. Bug 379026 has old and new screenshots.
Comment 36 David Williams CLA 2012-08-24 19:05:47 EDT
taking a cue from comment 20, I changed my "query script" for machine/software info with the results below. 

Anything look unusual? Out of date? Is the missing glib odd? 


uname -a information
Linux hudson-slave6 2.6.32.54-0.3-default #1 SMP 2012-01-27 17:38:56 +0100 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/lsb-release
cat /etc/SuSE-release
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 1
rpm -q cairo
cairo-1.8.8-2.1.48
rpm -q gtk2
gtk2-2.18.9-0.16.1
rpm -q glibc
glibc-2.11.1-0.34.1
rpm -q glib
package glib is not installed
rpm -q pango
pango-1.26.2-1.3.1
rpm -q ORBit2
package ORBit2 is not installed
Comment 37 David Williams CLA 2012-08-24 21:35:16 EDT
FYI, I've opened bug 387747 about how each slave gives different results for "lsb_release" (which by itself, should not matter to Eclipse ... just makes me wonder how they were set up/installed?) 

But also found none of them have glib installed. If any of you Display or SWT experts know if that's important, please comment in bug 387747.
Comment 38 Silenio Quarti CLA 2012-08-25 00:37:39 EDT
(In reply to comment #37)
> But also found none of them have glib installed. If any of you Display or
> SWT experts know if that's important, please comment in bug 387747.

Query for glib2 instead.
Comment 39 David Williams CLA 2012-08-25 01:55:35 EDT
(In reply to comment #38)
> (In reply to comment #37)
> > But also found none of them have glib installed. If any of you Display or
> > SWT experts know if that's important, please comment in bug 387747.
> 
> Query for glib2 instead.

Thanks. There's one mystery solved. 

rpm -q glib2
glib2-2.22.5-0.2.23


You asked about "Gnome or KDE", and I do not know which desktop environment is installed/running. I see no sign of either. Though KDE is traditional default with OpenSUSe (but believe its a choice during the install). And Paul already mentioned metacity is the window manager, just because that's what we start. (As far as I know, it could be window manager with either Gnome or KDE, but not sure). 

Even though I tell Hudson to start/use Xvnc during the test, a grep shows no processes with gnome or kde in the processes ... almost like we don't have a desktop? Guess we'll have to ask webmasters? 
ps -ef | egrep -i "gnome|kde" | grep -v egrep
Comment 40 Dennis Hendriks CLA 2012-08-25 04:16:25 EDT
(In reply to comment #32)
> @Thomas, Dennis, Silenio: which desktops are you using?

I use KDE.
Comment 41 David Williams CLA 2012-08-25 12:06:27 EDT
Not sure what it means (since, I see no processes running that mention gnome or kde), but I do see some environment variables defined on hudson slaves that are interesting. Such as 

hudson-slave1
WINDOWMANAGER=/usr/bin/gnome
XDG_CONFIG_DIRS=/etc/xdg
XDG_DATA_DIRS=/usr/share:/etc/opt/kde3/share:/opt/kde3/share


hudson-slave6
XDG_CONFIG_DIRS=/etc/xdg
XDG_DATA_DIRS=/usr/share:/etc/opt/kde3/share:/opt/kde3/share
[note, no "WINDOWMANAGER" defined]

Note, '6' is what we primarily use for our unit tests. Maybe I'll switch to '1' for a while :/
Comment 42 Dennis Hendriks CLA 2012-08-27 04:03:16 EDT
> > Query for glib2 instead.

$ rpm -q glib2
glib2-2.12.3-4.el5_3.1
Comment 43 Dennis Hendriks CLA 2012-08-28 07:46:27 EDT
Just updated glibc, when CentOS informed me of a new version. However, this does not resolve the issue:

$ rpm -q glibc
glibc-2.5-81.el5_8.4

$ rpm -q glibc
glibc-2.5-81.el5_8.7
Comment 45 Andrey Loskutov CLA 2012-08-28 16:19:57 EDT
Just for your information: there are three new different bug reports showing now in oracle bug database, probably pointing to the same issue as this bug. Two of the reports were closed/not reproducible, third one will probably be closed too (too less information). All of them are showing similar stack trace caused by some Eclipse code.

closed dup: 
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7189671
closed not reproducible:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7163321
open:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7189603
Comment 46 Silenio Quarti CLA 2012-08-28 17:35:30 EDT
Found bug#363491 and bug#368705 which have similar crash stacks. What is interesting about those is that it is on eclipse 3.7.1 (v3738a and v3738). Andrey, please could confirm whether Eclipse 3.7.1 crashes for you?
Comment 47 Eugene Ostroukhov CLA 2012-08-28 18:29:26 EDT
I see these crashes in 3.8-based CDT DSF debug UI. Do not see it in 3.8 JDT debug UI that I use a whole lot more.

Ubuntu 11.10 x86/64
Comment 48 Alexander Kurtakov CLA 2012-08-29 02:57:54 EDT
Some info based on Fedora problems that might help:
* seeing liborbit(aka libswt-gnome-gtk*.so) is really suspicious. We don't ship gnome integration in Fedora for few releases as we were having many crash reports and in fact it was supposed to use glib/gio and not gnome on recent versions but it was falling back in some cases. The easiest way on recent systems was to not ship libswt-gnome-gtk at all and fix the 1-2 cases where it was erroneously falling back to gnome instead of using gio. But I think one needs at least glib2 2.16 which is not available in RHEL 5. So most recent RPM based Linux distros(we share the build system) do not ship this library at all.
* oracle jvm/openjdk jit sometimes miscompiles certain methods so in Fedora we add the following to eclipse.ini. It certainly reduced the crashes for us when we added it but not fixed all crashes.
-XX:CompileCommand=exclude,org/eclipse/core/internal/dtree/DataTreeNode,forwardDeltaWith
-XX:CompileCommand=exclude,org/eclipse/jdt/internal/compiler/lookup/ParameterizedMethodBinding,<init>
-XX:CompileCommand=exclude,org/eclipse/cdt/internal/core/dom/parser/cpp/semantics/CPPTemplates,instantiateTemplate
-XX:CompileCommand=exclude,org/eclipse/cdt/internal/core/pdom/dom/cpp/PDOMCPPLinkage,addBinding
-XX:CompileCommand=exclude,org/python/pydev/editor/codecompletion/revisited/PythonPathHelper,isValidSourceFile
-XX:CompileCommand=exclude,org/python/pydev/ui/filetypes/FileTypesPreferencesPage,getDottedValidSourceFiles
Comment 49 Andrey Loskutov CLA 2012-08-29 04:16:56 EDT
(In reply to comment #46)
> Found bug#363491 and bug#368705 which have similar crash stacks. What is
> interesting about those is that it is on eclipse 3.7.1 (v3738a and v3738).
> Andrey, please could confirm whether Eclipse 3.7.1 crashes for you?

We using only 3.7.0 and 3.7.2, but none of them is crashing with ld-linux-x86-64.so.2, it's only 3.8 which causes trouble. The earliest 3.8 version I've tested was M7 and the crash was already there. So I think the range to check is (3.7.2, 3.8 M7].

(In reply to comment #47)
> I see these crashes in 3.8-based CDT DSF debug UI. Do not see it in 3.8 JDT
> debug UI that I use a whole lot more.

Although we have both CDT and JDT in the Eclipse installed, I use 99.9% of time JDT and it crashes there. Not sure if this is CDT related.

(In reply to comment #48)
> jit sometimes miscompiles certain methods

We had this problem with some older Eclipse/JVM configurations, but the root cause was always easy to spot as it had a clear and unique stack trace for each "bad" method.

> seeing liborbit(aka libswt-gnome-gtk*.so) is really suspicious.

So is it a new / wrong dependency in 3.8, or do you mean we can try to remove libswt-gnome-gtk* from the Eclipse installation and check if it will still run on RHEL 5.8?
Comment 50 Rastislav Wagner CLA 2012-08-31 04:48:07 EDT
Im also getting JVM crash with SIGSEGV in ld-linux-x86-64.so.2

My environment:

Ubuntu 12.04 x64

java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.3) (6b24-1.11.3-1ubuntu0.12.04.1)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)


Reproducible: Always

Steps to reproduce:
Im running these SWTBot tests http://anonsvn.jboss.org/repos/jbosstools/trunk/maven/tests/org.jboss.tools.maven.ui.bot.test/ with maven. It always crashes in the same place. 

to run swtbot tests:
1.checkout maven and build component
2.mvn clean install -P default,jbosstools-staging-aggregate -Dswtbot.test.skip=false on maven/tests/org.jboss.tools.maven.ui.bot.tests/


Everything was ok with eclipse 3.7
Comment 51 Silenio Quarti CLA 2012-08-31 12:57:12 EDT
Created attachment 220614 [details]
logs

I have run the tests a couple a times on same setup (Ubuntu 12.04) and it did not crash.  From the logs, do you see anything that could explain why it does not crash for me? Thanks!
Comment 52 Rastislav Wagner CLA 2012-09-04 09:13:06 EDT
Unfortunately I cant see anything different in your out.log except in my case it always crashes on line
12:32:16 - (SWTBotExt.java:70) - Menu "Window" selected
Comment 53 Radim Hopp CLA 2012-09-04 09:57:32 EDT
Created attachment 220682 [details]
One of many crash logs

Happens to me also when I'm having two instances of Eclipse Juno opened in the same time. It happens few clicks (anywhere) after starting the second instance. Always crashes the older instance and the new one stays.

It also happens, when I'm running SWTBot tests right from eclipse (Run As -> SWTBot test) (different case, than case Rastislav gave, he was executing them via maven)

My environment: 
Fedora 17 64b
java version "1.7.0_02"
Java(TM) SE Runtime Environment (build 1.7.0_02-b13)
Java HotSpot(TM) 64-Bit Server VM (build 22.0-b10, mixed mode) 
(happens also with openjdk)
Comment 54 Silenio Quarti CLA 2012-09-10 21:02:42 EDT
I was finally able to reproduce a crash in ld-linux by running the UI SessionTests junits on a RedHat 5.8 with a HotSpot VM.  This is the change that is causing the problem:

http://git.eclipse.org/c/platform/eclipse.platform.swt.git/commit/?id=ff8ec8fc1484c7eedcc70981458ecc23d5a64f37

It is a workaround for XULRunner 10 support which should not be running. We are working on a fix.

If you want to confirm this is your problem. Please remove the libswt-xulrunner-fix.so from your swt plugin JAR and from your <user.home>/.mozilla/eclipse dir.  Make sure that library is not loaded anymore (unless you have XULRunner 10 installed).
Comment 55 Andrej Podhradsky CLA 2012-09-11 08:09:15 EDT
I can confirm that after removing the file 'libswt-xulrunner-fix.so' eclipse is not crashing. Thank you!
Comment 56 Silenio Quarti CLA 2012-09-11 11:04:18 EDT
We have released a fix for this to master (4.3).

http://git.eclipse.org/c/platform/eclipse.platform.swt.git/commit/?id=2c1e3b3d7e8ee238ed9a1b0193fa5c0af85695fa

The problem happens because the workaround library is overwritten without checking whether it already exists. Any running instance of eclipse that already loaded the old library will eventually crash.

Leaving bug open to include in 4.2.2.
Comment 57 thomas lafarge CLA 2012-09-11 12:03:11 EDT
Just to confirm that deleting:
libswt-xulrunner-fix.so 
from both the swt plugin JAR 
(Eclipse_Juno/plugins/org.eclipse.swt.gtk.linux.x86_64_3.100.0.v4233d.jar)
and from the <user.home>/.mozilla/eclipse dir.  

Appear to works for me as eclipse hasn't crash the whole morning.
Thank you
PS: eclipse 4.2 juno
Comment 58 Eugene Ostroukhov CLA 2012-09-11 13:28:14 EDT
I am becoming increasingly worried that I am seeing another issue with the same symthoms. I removed libswt-xulrunner-fix.so and triple checked it is not loaded in the process but I still see the crash.

I do not see a crash report though. All I see is a message "***MEMORY-ERROR***: Nsight[16878]: GSlice: assertion failed: sinfo->n_allocated > 0" sent to stderr and quiet crash of the Eclipse. There's no JDK crash report dialog, nothing in the ~/.xsession-log.

I tried installing debug versions of the Glib (and GTK) and I tried debugging with gdb (for whatever reason, Eclipse crashes with SEGABORT when I run it in GDB).

My system is Ubuntu 11.10/64. Is there some way to improve error reporting?
Comment 59 Andrey Loskutov CLA 2012-09-11 13:42:57 EDT
(In reply to comment #58)
> I am becoming increasingly worried that I am seeing another issue with the
> same symthoms. I removed libswt-xulrunner-fix.so and triple checked it is
> not loaded in the process but I still see the crash.
> 
> I do not see a crash report though. All I see is a message
> "***MEMORY-ERROR***: Nsight[16878]: GSlice: assertion failed:
> sinfo->n_allocated > 0" sent to stderr and quiet crash of the Eclipse.

Can this be the bug 381407?
Comment 60 Silenio Quarti CLA 2012-09-11 14:02:32 EDT
(In reply to comment #59)
> > I do not see a crash report though. All I see is a message
> > "***MEMORY-ERROR***: Nsight[16878]: GSlice: assertion failed:
> > sinfo->n_allocated > 0" sent to stderr and quiet crash of the Eclipse.
> 
> Can this be the bug 381407?

Yes, it is. Same warning logs.
Comment 61 Eugene Ostroukhov CLA 2012-09-11 14:22:14 EDT
(In reply to comment #60)
> (In reply to comment #59)
> > > I do not see a crash report though. All I see is a message
> > > "***MEMORY-ERROR***: Nsight[16878]: GSlice: assertion failed:
> > > sinfo->n_allocated > 0" sent to stderr and quiet crash of the Eclipse.
> > 
> > Can this be the bug 381407?
> 
> Yes, it is. Same warning logs.

I cherry-picked the three changes to the 3.8.x branch and rebuilt the SWT, then I renamed the JNI sos to make sure the updated version is picked - I still see the crash... Either I messed up somehow or I'm seeing a different problem. Is there a way to obtain a better stack trace for that failure? I wonder is Oracle JDK may provide more diagnostics (or, at least, not crash in GDB). I'm currently using OpenJDK.
Comment 62 Silenio Quarti CLA 2012-09-11 14:27:33 EDT
(In reply to comment #61)
> I cherry-picked the three changes to the 3.8.x branch and rebuilt the SWT,
> then I renamed the JNI sos to make sure the updated version is picked - I
> still see the crash... Either I messed up somehow or I'm seeing a different
> problem. Is there a way to obtain a better stack trace for that failure? I
> wonder is Oracle JDK may provide more diagnostics (or, at least, not crash
> in GDB). I'm currently using OpenJDK.

Do you get a stack trace if you turn on debugging?

http://www.eclipse.org/swt/faq.php#debugmode
Comment 63 Andrey Loskutov CLA 2012-09-11 14:29:19 EDT
(In reply to comment #61)
> (In reply to comment #60)
> > (In reply to comment #59)
> > > > I do not see a crash report though. All I see is a message
> > > > "***MEMORY-ERROR***: Nsight[16878]: GSlice: assertion failed:
> > > > sinfo->n_allocated > 0" sent to stderr and quiet crash of the Eclipse.
> > > 
> > > Can this be the bug 381407?
> > 
> > Yes, it is. Same warning logs.
> 
> I cherry-picked the three changes to the 3.8.x branch and rebuilt the SWT,
> then I renamed the JNI sos to make sure the updated version is picked - I
> still see the crash... Either I messed up somehow 

This is my main pain point. I've managed to build SWT plugin and fragment using SWT-FAQ. But I never managed to properly integrate the new SWT plugin + fragment into the SDK (so that can be re-distributed). I tried feature patches of all possible flavours, including / excluding plugin etc, lot of crazy stuff. At the end I found the big hammer method - I simply do CBI build every time I patch SWT. It's crazy, I know, so I would be happy to know how one can properly integrate patched SWT bits into the deployable eclipse SDK package. 

> or I'm seeing a different
> problem. 

Forget to delete config area with the cashed jars in ~/.eclipse?

> Is there a way to obtain a better stack trace for that failure? I
> wonder is Oracle JDK may provide more diagnostics (or, at least, not crash
> in GDB). I'm currently using OpenJDK.
Comment 64 Eugene Ostroukhov CLA 2012-09-11 14:31:21 EDT
(In reply to comment #62)
> (In reply to comment #61)
> > I cherry-picked the three changes to the 3.8.x branch and rebuilt the SWT,
> > then I renamed the JNI sos to make sure the updated version is picked - I
> > still see the crash... Either I messed up somehow or I'm seeing a different
> > problem. Is there a way to obtain a better stack trace for that failure? I
> > wonder is Oracle JDK may provide more diagnostics (or, at least, not crash
> > in GDB). I'm currently using OpenJDK.
> 
> Do you get a stack trace if you turn on debugging?
> 
> http://www.eclipse.org/swt/faq.php#debugmode

I trimmed everything prior to debug session start:
-->Deregister=ToolBar {} 139649708703744
-->Deregister=CLabel {} 139649679876640
-->Deregister=CLabel {} 139649679876896
-->Deregister=CLabel {} 139649679877152
-->Deregister=CLabel {} 139649679876512
-->Deregister=CLabel {} 139649679877152
-->Deregister=CLabel {} 139649679877024
-->Deregister=CLabel {} 139649679877024
-->Deregister=CLabel {} 139649679876640
-->Deregister=CLabel {} 139649712377920
-->Deregister=CLabel {} 139649666037392
-->Deregister=CLabel {} 139649676975936
-->Deregister=CLabel {} 139649712378432

***MEMORY-ERROR***: Nsight[17871]: GSlice: assertion failed: sinfo->n_allocated > 0
Comment 65 Eugene Ostroukhov CLA 2012-09-11 14:32:04 EDT
> Forget to delete config area with the cashed jars in ~/.eclipse?

GDB only shows "patched" so loaded in the process. That is, before JVM crashes :)
Comment 66 Silenio Quarti CLA 2012-09-11 14:45:44 EDT
Are you using these instructions to build a patched version?

http://www.eclipse.org/swt/faq.php#howbuildplugin
Comment 67 Eugene Ostroukhov CLA 2012-09-11 14:48:33 EDT
(In reply to comment #66)
> Are you using these instructions to build a patched version?
> 
> http://www.eclipse.org/swt/faq.php#howbuildplugin

They do not work for me entirely. I was able to reach a point where I build GTK 64-bit version and then I simply imported gtk-x86-64 fragment in my workspace, changed its version (so I can make sure its the one loaded) and manually replaced the shared lib.
Comment 68 Andrey Loskutov CLA 2012-09-11 14:49:31 EDT
(In reply to comment #66)
> Are you using these instructions to build a patched version?
> http://www.eclipse.org/swt/faq.php#howbuildplugin

Yes. Building is not a problem, integrating the build result is.
Comment 69 Eugene Ostroukhov CLA 2012-09-11 14:59:33 EDT
Good grief, looks like those changes are in Java files - and I only updated the native part. I'll try to properly rebuild the whole SWT (native and Java) and will report if I still see the issue.
Comment 70 Andrey Loskutov CLA 2012-09-11 15:15:48 EDT
(In reply to comment #56)
> The problem happens because the workaround library is overwritten without
> checking whether it already exists. Any running instance of eclipse that
> already loaded the old library will eventually crash.

BTW, if I properly understood the problem root cause, the workaround for the bug would be easy: 
(-: never work in two Eclipse instances in parallel :-)

And the way to force the crash would be:
 * start first Eclipse, open Javadoc hover for some java method (uses xulrunner)
 * start second Eclipse
 * switch to previous one and try to see the hover again. In theory (I'm not in the office) it should crash soon. This explains why it often crashes while switching from window to window.

> Leaving bug open to include in 4.2.2 
and please to 3.8.2 too.
Comment 71 Silenio Quarti CLA 2012-09-11 15:18:43 EDT
(In reply to comment #68)
> (In reply to comment #66)
> > Are you using these instructions to build a patched version?
> > http://www.eclipse.org/swt/faq.php#howbuildplugin
> 
> Yes. Building is not a problem, integrating the build result is.

I just followed those instructions and it worked for me. Added a println() to Display.init() to make sure the new code is running. Note that you have to build with the same qualifier (vXXXX) of what is installed and just overwrite the jar file in the eclipse/plugins directory.
Comment 72 Andrey Loskutov CLA 2012-09-11 15:57:44 EDT
(In reply to comment #71)
> > Yes. Building is not a problem, integrating the build result is.
> 
> I just followed those instructions and it worked for me. Added a println()
> to Display.init() to make sure the new code is running. Note that you have
> to build with the same qualifier (vXXXX) of what is installed and just
> overwrite the jar file in the eclipse/plugins directory.

That's my problem. I wanted to have a patch number/version to differentiate "our" build from "origin" so one can easily see on the eclipse configuration data if the build is customized or not. So there is no way to use changed build number for SWT?
Comment 73 Eugene Ostroukhov CLA 2012-09-11 16:49:16 EDT
(In reply to comment #66)
> Are you using these instructions to build a patched version?
> 
> http://www.eclipse.org/swt/faq.php#howbuildplugin

Yep, my problem was a bug 381407. I can no longer crash our product after I rebuilt SWT 3.8.x with those 3 commits cherry-picked. Thank you and Grant for fixes.
Comment 74 Silenio Quarti CLA 2012-09-11 16:54:19 EDT
(In reply to comment #72)
> That's my problem. I wanted to have a patch number/version to differentiate
> "our" build from "origin" so one can easily see on the eclipse configuration
> data if the build is customized or not. So there is no way to use changed
> build number for SWT?

My guess is that you need to install the new plugin with a proper p2 update site, but I do not know enough about p2 to help you with this.
Comment 75 Dani Megert CLA 2012-09-12 04:58:51 EDT
(In reply to comment #72)
> (In reply to comment #71)
> > > Yes. Building is not a problem, integrating the build result is.
> > 
> > I just followed those instructions and it worked for me. Added a println()
> > to Display.init() to make sure the new code is running. Note that you have
> > to build with the same qualifier (vXXXX) of what is installed and just
> > overwrite the jar file in the eclipse/plugins directory.
> 
> That's my problem. I wanted to have a patch number/version to differentiate
> "our" build from "origin" so one can easily see on the eclipse configuration
> data if the build is customized or not. So there is no way to use changed
> build number for SWT?

WARNING: This is not an official way to to things ;-)

You can use a different qualifier but you also have to update the 
<intall>\configuration\org.eclipse.equinox.simpleconfigurator\bundles.info
Comment 76 Martin Oberhuber CLA 2012-09-14 13:03:20 EDT
CQ:WIND00361119

Could this fix be deployed as an official patch feature from the Eclipse Platform update site ? See also bug 381407 comment 19 . This is a must-fix for our product which shall be released on Eclipse Juno BEFORE SR2.
Comment 77 Silenio Quarti CLA 2012-09-14 14:27:00 EDT
Requesting PMC approval for SR1.
Comment 78 Silenio Quarti CLA 2012-09-14 14:51:32 EDT
Backported to R3_8_maintenance and R4_2_maintenance.
Comment 79 Silenio Quarti CLA 2012-09-17 10:58:05 EDT
Verified in 4.2.1 build M20120914.
Comment 80 David Wootton CLA 2014-06-25 11:39:46 EDT
I think I'm seeing this bug at Eclipse startup on a Linux x86_64 SUSE 10 system, kernel 2.6.16.60-0-0.58.1.3835.0.PTF.638363-smp. I start Eclipse and get the spash screen, but before any workspace prompt or main window, Eclipse crashes. The error message and the stack traceback show do_lookup_x as the top of stack.

I've tried this with a Kepler SR2 install as well as with what I think is the official Luna release I downloaded this morning (eclipse-cpp-luna-R-linux-gtk-x86_64.tar.gz

Trying to debug this a little bit, it looks like do_lookup_x is performing some dynamic symbol lookup and failing at 0xcf offset on a 'div %rbx' instruction where %rbx is zero, resulting in a SIGFPE for zerodivide.

Any suggestions how to get around this or if there is a fix?
Comment 81 Markus Keller CLA 2014-06-25 12:15:46 EDT
(In reply to David Wootton from comment #80)
This bug has been fixed in a past release. Please open a new bug if you see a new or related problem in Luna.