Comment 1 Grant Gayed 2012-06-01 13:06:57 EDT

Created attachment 216687 [details]
full jre dump

Comment 2 Eugene Ostroukhov 2012-08-28 18:10:15 EDT

I've been using 3.8 for months to develop CDT-based product that was based on 3.7. I have not seen the crash once.

Now I see it all the time in our product that we migrated on top of 3.8. I don't see a pattern in this failures other then we see them more often when debugger UI is on. The UI is slightly extended CDT DSF debugger.

Comment 3 Eugene Ostroukhov 2012-08-28 18:25:50 EDT

I've seen this issue on Ubuntu 11.10 x86/64

Comment 4 Grant Gayed 2012-08-29 16:30:14 EDT

Fixed in the 4.3 stream > 20120829, patch: http://git.eclipse.org/c/platform/eclipse.platform.swt.git/commit/?id=0ffd76f04d04c924c21c761e53a4e547cd5ce501 .  Verified the fix using the comment 0 test case on PPC64.

Keeping report open and setting Target Milestone to 4.2.2 so that this fix will be included there once its tree becomes open post-4.2.1.

re: comment 3
I can't say for sure if this fix will make your x86_64 crashes go away, but if the error output you see is similar to what's here then I'd say there's a good chance that it has the same underlying cause.

Comment 5 Andrey Loskutov 2012-08-29 16:34:34 EDT

Looks like I was also beaten by this bug, see bug 383890 comment 3 and 4. However we are not using ppc, we are using x86_64 GTK2 build on RHEL 5.8. 

So we can add x86_64 as another platform to this bug.

Interestingly, I saw this crash only two times, while the crashes with SIGSEGV in ld-linux-x86-64.so.2 from bug 383890 are happening twice a week.

BTW, in bug 383890 comment 33 Silenio mentioned that there is a possibility that:

"We also fixed many places we were supposedly leaking in Tree/Table.  Maybe we were wrong and we are double freeing now.  Google indicates that a crash in _dl_fixup() usually means we are trashing memory." Good candidate for code reviews :-) ?

Comment 6 Andrey Loskutov 2012-08-29 16:36:46 EDT

(In reply to comment #4)
> Fixed in the 4.3 stream > 20120829

Can we get the patch into 3.8 maintenance stream please?

Comment 7 Andrey Loskutov 2012-08-30 02:57:22 EDT

Created attachment 220498 [details]
Crash in breadcrumb

I've played a bit trying to reproduce the crashes with the variables view layout switching, and it looks like they really can be enforced on Eclipse 3.8 RHEL 5.8 x86_64.

Interestingly, it is not enough to switch layout once. I had to switch multiple times, expand the table items, collapse items and then select a line in the Java editor. But in almost all cases I've managed to crash Eclipse in less then one minute.

Eclipse either immediately crashes without a VM dump at all or with a dump but a different one that was shown here:

#  SIGSEGV (0xb) at pc=0x0000003060e41d4c, pid=24872, tid=1083169088
#
# JRE version: 7.0_04-b20
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.0-b21 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libglib-2.0.so.0+0x41d4c]  chmod+0x41d4c

...

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libglib-2.0.so.0+0x41d4c]  chmod+0x41d4c

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  org.eclipse.swt.internal.gtk.OS._gtk_widget_destroy(J)V+0
j  org.eclipse.swt.internal.gtk.OS.gtk_widget_destroy(J)V+8
j  org.eclipse.swt.widgets.Widget.destroyWidget()V+26
j  org.eclipse.swt.widgets.Widget.release(Z)V+75
j  org.eclipse.swt.widgets.Control.release(Z)V+94
j  org.eclipse.swt.widgets.Widget.dispose()V+23
j  org.eclipse.jdt.internal.ui.javaeditor.breadcrumb.BreadcrumbItem.dispose()V+4

I've attached the dump for reference.

Comment 8 Andrey Loskutov 2012-09-04 11:20:21 EDT

Created attachment 220685 [details]
Crash dump in libglib-2.0

(In reply to comment #6)
> (In reply to comment #4)
> > Fixed in the 4.3 stream > 20120829
> 
> Can we get the patch into 3.8 maintenance stream please?

I've tried to apply proposed patch to the 3.8 SWT code used by the CBI build (see [1]) but the patched Eclipse still crashes (RHEL 5.8 x86_64). Sometimes it reports VM dumps (see attachment), sometimes just the eclipse crash dialog is shown, sometimes stderr is filled with some stack traces like below.

My steps to reproduce:

- put a breakpoint at Snippet1's "shell.open();" line
- debug Snippet1, hit the breakpoint
- in the Variables view deselect menu item <drop down arrow> > Layout > Show Columns
-> removes the view's columns
- expand nodes, scroll in the tree
- now re-select the same menu item
- expand nodes, scroll in the table
- select package explorer/outline view or move the mouse to the different place in the Java editor => crash
- in case no crash happens, redo layout change/expand/scroll items again. It will crash sooner or later.


*** glibc detected *** /opt/java1.7_x86_64/bin/java: double free or corruption (out): 0x00002aaab8b674e0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x305b27247f]
/lib64/libc.so.6(cfree+0x4b)[0x305b2728db]
/lib64/libglib-2.0.so.0(g_datalist_id_set_data_full+0x3c4)[0x3060e1bac4]
/lib64/libgobject-2.0.so.0(g_object_unref+0xa8)[0x306220cee8]
/usr/lib64/libgdk-x11-2.0.so.0[0x3064c44c14]
/usr/lib64/libgdk-x11-2.0.so.0[0x3064c46452]
/usr/lib64/libgdk-x11-2.0.so.0[0x3064c4686e]
/lib64/libglib-2.0.so.0(g_main_context_dispatch+0x1b4)[0x3060e2cdb4]
/lib64/libglib-2.0.so.0[0x3060e2fc0d]
/lib64/libglib-2.0.so.0(g_main_context_iteration+0x6e)[0x3060e3011e]
/data/.eclipse_ide/com.verigy.ide_3.8_x86_64-beta/configuration/org.eclipse.osgi/bundles/166/2/.cp/libswt-pi-gtk-3834.so(Java_org_eclipse_swt_internal_gtk_OS__1g_1main_1context_1iteration+0xf)[0x2aaabd417b5e]
[0x2aaaab8a7fad]
======= Memory map: ========


Should I open a new bug for 3.8 on x86_64 or what would you propose?

[1] http://wiki.eclipse.org/CBI/Eclipse_Platform_Build

Comment 9 Andrey Loskutov 2012-09-04 11:25:54 EDT

Created attachment 220686 [details]
Another crash in breadcrumb

Another VM crash dump for the comment above

Comment 10 Grant Gayed 2012-09-04 12:09:13 EDT

(In reply to comment #6)
> Can we get the patch into 3.8 maintenance stream please?

Sorry for the late reply, we were waiting to see if the change would affect the unit tests on 4.3, and to discuss whether to request PMC approval for this fix (required at this stage in the 4.2.1 release cycle).  This change will not be added to the 4.2.1/3.8.1 stream, it will wait for 4.2.2/3.8.2.


(In reply to comment #8)
> Should I open a new bug for 3.8 on x86_64 or what would you propose?

I'll investigate further first, on both PPC64 and x86_64, to try to determine if there's a connection, and will follow up here.

Comment 11 Grant Gayed 2012-09-05 17:00:30 EDT

I'm able to make PPC64 crash with the comment 8 steps, so the comment 4 fix is not complete.  I've investigated further and found some additional 64-bit fixes that are needed in this area, but they do not fix the crash.  Regardless, they're committed to the 4.3 stream: http://git.eclipse.org/c/platform/eclipse.platform.swt.git/commit/?id=001a26d56e2c5f6288db0d3f1cb8c5f85fea05dc .

Don't open a separate report for now, my hope is to fix for 4.2.2.

Comment 12 Silenio Quarti 2012-09-05 23:12:14 EDT

I believe I found the remaining problem. We were freeing the wrong object in Tree.copyModel(). This bug was caused by bug#364593.

This could be the cause of bug#383890 as well.

http://git.eclipse.org/c/platform/eclipse.platform.swt.git/commit/?id=cce2597307b6e46e6f5e666c978d0b2de6c62776

Comment 13 Grant Gayed 2012-09-06 11:17:24 EDT

The comment 12 fix appears to fix the comment 8 case, so I think the three patches linked in this report are what's needed here.

They are all in the 4.3 stream, and will be added to the 4.2.2/3.8.2 stream when it becomes available.  It's too late to consider this for 4.2.1/3.8.1, especially given the size and nature of the changes.

Comment 14 Andrey Loskutov 2012-09-10 09:47:39 EDT

(In reply to comment #13)
> The comment 12 fix appears to fix the comment 8 case, so I think the three
> patches linked in this report are what's needed here.

After short testing on manually backported code (into 3.8 SWT) it also seems to fix my issues reported here on RHEL 5.8 x86_64.

> They are all in the 4.3 stream, and will be added to the 4.2.2/3.8.2 stream
> when it becomes available.  It's too late to consider this for 4.2.1/3.8.1,
> especially given the size and nature of the changes.

Thanks, and please do not forget the 3.8.x stream :-)

(In reply to comment #12)
> This could be the cause of bug#383890 as well.

I'm monitoring now if the crash described in bug 383890 will be affected by the patch.

Comment 15 Silenio Quarti 2012-09-10 10:33:39 EDT

(In reply to comment #14)
> I'm monitoring now if the crash described in bug 383890 will be affected by
> the patch.

It has not fixed the crashes while running the junit tests as the nightly builds still show a crash log when running SessionTests.

http://download.eclipse.org/eclipse/downloads/drops4/N20120906-2200/logs.php#console

crashlogs_org.eclipse.ui.tests.session.SessionTests.hs_err_pid17199.log

Comment 16 Andrey Loskutov 2012-09-10 10:47:09 EDT

(In reply to comment #15)
> (In reply to comment #14)
> > I'm monitoring now if the crash described in bug 383890 will be affected by
> > the patch.
> 
> It has not fixed the crashes while running the junit tests as the nightly
> builds still show a crash log when running SessionTests.

The patched Eclipse just crashed on my workstation, as usually in ld-linux. So while the current reproducible crash seems to be fixed, the other one in bug 383890 is still there...

Comment 17 Eugene Ostroukhov 2012-09-11 16:50:49 EDT

I've cherry-picked these 3 commits into 3.8.x branch and rebuilt SWT - my problem from comment#2 is now gone. Thank you a lot!

Is there any dates when the fixes will be in 3.8,2 nightlies? I do not trust SWT package I produce locally...

Comment 18 Silenio Quarti 2012-09-11 17:04:43 EDT

(In reply to comment #17)
> I've cherry-picked these 3 commits into 3.8.x branch and rebuilt SWT - my
> problem from comment#2 is now gone. Thank you a lot!
> 
> Is there any dates when the fixes will be in 3.8,2 nightlies? I do not trust
> SWT package I produce locally...

They will be released to the R4_2_maintanance and R3_8_maintenance after Juno SR1 (4.2.1) ships (end of Sept) and those branches are open for June SR2 (4.2.2) development. 

http://www.eclipse.org/eclipse/development/plans/freeze_plan_4_2_1.php

Comment 19 Martin Oberhuber 2012-09-14 12:35:23 EDT

Could an official "patch feature" for this be considered and put on the Eclipse Platform Update Site soon after Juno SR1 ?

We are also going to release a product on Juno BEFORE SR2 and this fix is a must-have. Rather than deploying an inofficial patch of ours I'd prefer to see an official patch from Eclipse.org.

Comment 20 Martin Oberhuber 2012-09-14 12:49:08 EDT

CQ:WIND00361119

Comment 21 Andrey Loskutov 2012-09-14 12:56:53 EDT

(In reply to comment #19)
> Could an official "patch feature" for this be considered and put on the
> Eclipse Platform Update Site soon after Juno SR1 ?
> 
> We are also going to release a product on Juno BEFORE SR2 and this fix is a
> must-have. Rather than deploying an inofficial patch of ours I'd prefer to
> see an official patch from Eclipse.org.

Martin, not sure if you also aware of bug 383890 - this two bugs together served the most crashes in our lab. I would say in case your product allows multiple Eclipse instances running, then bug 383890 is a "must have" for you too.

I would also like to see an official patch feature for those two bugs for Juno SR1 release (both 3.x and 4.x streams).

Comment 22 Silenio Quarti 2012-09-14 14:30:02 EDT

Requesting PMC approval for SR1.

Comment 23 Eugene Ostroukhov 2012-09-14 14:38:40 EDT

Is this for both 4.2.1 and 3.8.1 or 4.2.1 and 3.8.2?

Comment 24 Silenio Quarti 2012-09-14 14:50:11 EDT

Backported to R3_8_maintenance and R4_2_maintenance.  

This is for 3.8.1 and 4.2.1.

Comment 25 Silenio Quarti 2012-09-17 11:04:56 EDT

Verified in 4.2.1 build M20120914.

Comment 26 Grant Gayed 2012-09-19 11:09:41 EDT

*** Bug 389861 has been marked as a duplicate of this bug. ***