This Bugzilla instance is deprecated, and most Eclipse projects now use GitHub or Eclipse GitLab. Please see the deprecation plan for details.
Bug 517013 - Regression: Equinox Launcher fails on RHEL6 due to GLIBC_2.14 dependency
Summary: Regression: Equinox Launcher fails on RHEL6 due to GLIBC_2.14 dependency
Status: VERIFIED FIXED
Alias: None
Product: Equinox
Classification: Eclipse Project
Component: Launcher (show other bugs)
Version: 4.7.0 Oxygen   Edit
Hardware: PC Linux
: P3 critical (vote)
Target Milestone: Oxygen RC3   Edit
Assignee: Arun Thondapu CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on: 515155
Blocks: 517284
  Show dependency tree
 
Reported: 2017-05-19 17:46 EDT by Martin Oberhuber CLA
Modified: 2017-06-02 13:16 EDT (History)
6 users (show)

See Also:
daniel_megert: pmc_approved+
akurtakov: pmc_approved+
arunkumar.thondapu: review+


Attachments
Patch with recompiled launcher binaries for eclipse-4.7m7 (105.76 KB, application/zip)
2017-05-25 16:38 EDT, Martin Oberhuber CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Martin Oberhuber CLA 2017-05-19 17:46:46 EDT
Build ID: eclipse-SDK-4.7RC1-linux-gtk-x86_64

The Equinox Launcher fails on RHEL6 since recently:

./eclipse 
./eclipse: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by ./eclipse)

This used to work with eclipse-SDK-I20170422 on the same host! It looks like a recent regression due to the way the Launcher binaries are built, since I cannot find any commit that actually changes the Launcher code. I do notice that the size of the eclipse executable changed:

- I20170422 : 79841 bytes
- 4.7RC1    : 71431 bytes

Please get this resolved, since we just invested some effort getting Eclipse to work on RHEL6, through https://bugs.eclipse.org/bugs/show_bug.cgi?id=515155 . We had succeeded with I20170422 . Maybe it is sufficient replacing any occurence of memcpy() by memmove() , or adding a -fno-builtin-memmove to the build command (as per bug 515155).
Comment 1 Martin Oberhuber CLA 2017-05-20 23:00:10 EDT
Workaround:
jre/bin/java -jar plugins/org.eclipse.equinox.launcher_1.4.0.v20161219-1356.jar

unblocks me so it's only CRITICAL but should still be fixed!
Comment 2 Martin Oberhuber CLA 2017-05-23 11:23:59 EDT
The "java -jar" workaround has 2 sigificant limitations,
 - no splashscreen
 - File > Restart doesn't work (also affects installing software)

To fix the problem, I thought about adding an #include “eclipse-memmove.h” to the 3 affected .c files (those which have memcpy() usage. The contents of "eclipse-memmove.h" would look like this (pseudocode, sorry I don’t have more time now):

#if defined(LINUX) && defined (X86_64)
#  if defined(memcpy)
#    undef memcpy
#  endif
#  define memcpy memmove 
#endif

That way, the impact remains limited to Linux x64 only.
Comment 3 Arun Thondapu CLA 2017-05-25 11:00:07 EDT
I couldn't find an RHEL 6 machine that can be used to rebuild the launcher binaries. I'll try to rebuild on RHEL 7.2 itself with -fno-builtin-memmove but I cannot verify that it actually fixes the issue.

Martin, can you help me in verifying the binaries before actually making the changes in an official build?
Comment 4 Martin Oberhuber CLA 2017-05-25 11:18:16 EDT
(In reply to Arun Thondapu from comment #3)
Hi Arun, I'll be happy to verify any binaries. BTW on your end you can also
verify easily without running Eclipse:

  % nm eclipse | grep GLIBC_2.14
  U memcpy@@GLIBC_2.14

Same test for the Launcher's companion sharedlib. The output above is from M7. With I20170422, the nm command doesn't print anything. That's what I'd like to get back.

And I'm afraid that just adding -fno-builtin-memmove will not be sufficient, you'll also need to replace the usage of memcpy() by memmove() in the code.

Thanks!
Martin
Comment 5 Eclipse Genie CLA 2017-05-25 16:29:25 EDT
New Gerrit change created: https://git.eclipse.org/r/97996
Comment 6 Martin Oberhuber CLA 2017-05-25 16:38:02 EDT
Created attachment 268574 [details]
Patch with recompiled launcher binaries for eclipse-4.7m7

Above Gerrit fixes the problem. Attached is the new launcher compiled on REL-7.3 (Maipo) and tested to work fine on both REL-7.3 and REL-6.3.

The fix only changes the compile for Linux x86_64 -- I've validated that 32-bit Linux GTK is unaffected by the problem, so the fix is only needed for x86_64. Could this contribution be considered for RC3 ?
Comment 7 Dani Megert CLA 2017-05-26 03:42:31 EDT
I can't test the fix but if Arun and a second reviewer (Alex?) is happy with the fix, I am +1 for RC3.
Comment 8 Daniel Wille CLA 2017-05-26 13:55:16 EDT
Myself and a work colleague also observed this critical issue on CentOS 6. I cloned the rt.equinox.framework, applied Martin's changes, and compiled by hand. I manually swapped in the resulting executable and library and found that solved the issue.
Comment 9 Dani Megert CLA 2017-05-27 10:12:04 EDT
(In reply to Daniel Wille from comment #8)
> Myself and a work colleague also observed this critical issue on CentOS 6. I
> cloned the rt.equinox.framework, applied Martin's changes, and compiled by
> hand. I manually swapped in the resulting executable and library and found
> that solved the issue.

David, you are a committer. Please set the review+ flag.

Alex, you reviewed the Gerrit change, so please add your review+ here.

Once we have that, we can merge the change.
Comment 10 Dani Megert CLA 2017-05-27 10:13:45 EDT
(In reply to Dani Megert from comment #9)
> (In reply to Daniel Wille from comment #8)
> > Myself and a work colleague also observed this critical issue on CentOS 6. I
> > cloned the rt.equinox.framework, applied Martin's changes, and compiled by
> > hand. I manually swapped in the resulting executable and library and found
> > that solved the issue.
> 
> David, you are a committer. Please set the review+ flag.
> 
> Alex, you reviewed the Gerrit change, so please add your review+ here.
> 
> Once we have that, we can merge the change.

Sorry, wrong bug report.
Comment 11 Dani Megert CLA 2017-05-27 10:14:31 EDT
> Alex, you reviewed the Gerrit change, so please add your review+ here.

This still applies.
Comment 12 Arun Thondapu CLA 2017-05-29 06:54:05 EDT
Martin, can you please submit an updated patchset according to the comments in gerrit? Lets try to have this in the next scheduled build if possible (which starts in an hour approximately).
Comment 13 Martin Oberhuber CLA 2017-05-29 11:03:55 EDT
(In reply to Arun Thondapu from comment #12)
> Martin, can you please submit an updated patchset according to the comments
> in gerrit? Lets try to have this in the next scheduled build if possible
> (which starts in an hour approximately).

Hi Arun, as I've also told Dani on the PMC meeting, I'm currently maxed out with other (non-Eclipse) work culminating in a release, so I'm afraid I have little or no time to spare.

I could probably add the CFLAGS to the build.sh as you wanted tonight, but I'll not be able to do as much testing as with the original submission. I'm not sure if taking this risk is the right path at the moment. 

I can commit to reworking the contribution after Thursday (my Release is on Wednesday) but I'm afraid that won't help you much for RC3...
Comment 14 Arun Thondapu CLA 2017-05-29 11:45:22 EDT
(In reply to Martin Oberhuber from comment #13)
> Hi Arun, as I've also told Dani on the PMC meeting, I'm currently maxed out
> with other (non-Eclipse) work culminating in a release, so I'm afraid I have
> little or no time to spare.

No problem, I will update the patch myself and build the new binaries.

> I could probably add the CFLAGS to the build.sh as you wanted tonight, but
> I'll not be able to do as much testing as with the original submission. I'm
> not sure if taking this risk is the right path at the moment. 

I will do some testing before pushing the changes but I can't test on RHEL 6.x. Will you be able to quickly verify that the problem is fixed in the next build on RHEL 6.x?

IMHO the risk associated with the suggested change is minimal as it is only cosmetic and does not have any real effect on the built binaries as such...
Comment 15 Martin Oberhuber CLA 2017-05-29 11:47:29 EDT
(In reply to Arun Thondapu from comment #14)
> No problem, I will update the patch myself and build the new binaries.
Thanks!

> 6.x. Will you be able to quickly verify that the problem is fixed in the
> next build on RHEL 6.x?
Yes, I can easily download and run on my host.
Comment 17 Arun Thondapu CLA 2017-05-29 13:59:40 EDT
Linux x86_64 launcher binaries have been rebuilt and committed via https://git.eclipse.org/c/equinox/rt.equinox.binaries.git/commit/?id=83d6c4d71b970fe2ed8a6a687cae384c3cec0b68
Comment 18 Arun Thondapu CLA 2017-05-29 23:41:25 EDT
Martin, can you please download and verify the fix with this build - http://download.eclipse.org/eclipse/downloads/drops4/I20170529-2000/
Comment 19 Martin Oberhuber CLA 2017-05-30 10:50:04 EDT
(In reply to Arun Thondapu from comment #18)
> Martin, can you please download and verify the fix with this build -
> http://download.eclipse.org/eclipse/downloads/drops4/I20170529-2000/

Confirmed, eclipse-SDK-I20170529-2000-linux-gtk-x86_64.tar.gz launches just fine on my REL6.3 box -- marking VERIFIED, thanks for all the help here!!
Comment 20 Dani Megert CLA 2017-06-01 05:05:08 EDT
(In reply to Martin Oberhuber from comment #19)
> (In reply to Arun Thondapu from comment #18)
> > Martin, can you please download and verify the fix with this build -
> > http://download.eclipse.org/eclipse/downloads/drops4/I20170529-2000/
> 
> Confirmed, eclipse-SDK-I20170529-2000-linux-gtk-x86_64.tar.gz launches just
> fine on my REL6.3 box -- marking VERIFIED, thanks for all the help here!!

Martin, as the launcher changed again, please verify with I20170531-2000 or newer.
Comment 21 Arun Thondapu CLA 2017-06-01 06:06:37 EDT
(In reply to Dani Megert from comment #20)
> Martin, as the launcher changed again, please verify with I20170531-2000 or
> newer.

I've verified using the nm command (see comment 4) that the dependency on GLIBC_2.14 does not exist in the newly built binaries, so we should be good here. Of course, we can be doubly sure if Martin is able to verify by running the latest build on RHEL 6.3...
Comment 22 Martin Oberhuber CLA 2017-06-01 17:09:37 EDT
Hi Dani,

I'm afraid I cannot do this verification easily, as I just changed employers and I don't have access to a REL6 or CentOS6 machine any more.

I had a quick look on an old Ubuntu 12.04, but that also has memcpy@GLIBC_2.14 in its libc.so.6 so it doesn't tell us much:
objdump -T /lib/x86_64-linux-gnu/libc.so.6 | grep memcpy
just FYI, on Ubuntu 12 there's also no icons in the welcome (see my other bug).

I could probably try download a copy of CentOS6 and install into a Virtualbox, but I'm not sure if I'll find the time for that in the next couple days...

Anyways, since the plan was reverting the checked-in launcher binaries, and I know from bug 515155 that I20170425 worked fine on REL6, what about doing a binary diff to that spin?
Comment 23 Martin Oberhuber CLA 2017-06-01 17:11:50 EDT
PS like Arun, I've also downloaded I20170531-2000 and verified using "objdump -T eclipse | grep memcpy" as well as "objdump -T eclipse_1624.so | grep memcpy" that the offending symbol isn't in there.
Comment 24 Dani Megert CLA 2017-06-02 04:20:42 EDT
(In reply to Martin Oberhuber from comment #22)
> Anyways, since the plan was reverting the checked-in launcher binaries

That did not work. We would have lost important fixes like scrolling performance fixes for Mac. We ended up removing the Java 9 specific code and rebuilt the launcher.
Comment 25 Alexander Kurtakov CLA 2017-06-02 04:25:08 EDT
(In reply to Dani Megert from comment #24)
> (In reply to Martin Oberhuber from comment #22)
> > Anyways, since the plan was reverting the checked-in launcher binaries
> 
> That did not work. We would have lost important fixes like scrolling
> performance fixes for Mac. We ended up removing the Java 9 specific code and
> rebuilt the launcher.

I'm really surprised that the plan was to revert to checked in older binaries. We should clearly never do such a thing. I would even say we should rebuild the binaries as part of the regular build instead.
Comment 26 Arun Thondapu CLA 2017-06-02 09:09:21 EDT
(In reply to Alexander Kurtakov from comment #25)
> (In reply to Dani Megert from comment #24)
> > (In reply to Martin Oberhuber from comment #22)
> > > Anyways, since the plan was reverting the checked-in launcher binaries
> > 
> > That did not work. We would have lost important fixes like scrolling
> > performance fixes for Mac. We ended up removing the Java 9 specific code and
> > rebuilt the launcher.
> 
> I'm really surprised that the plan was to revert to checked in older
> binaries. We should clearly never do such a thing. I would even say we
> should rebuild the binaries as part of the regular build instead.

Rebuilding the binaries for every build may be an overkill IMHO, but I completely agree that reverting to older launcher binaries instead of doing a rebuild, is definitely *not* the right thing to do...
Comment 27 Alexander Kurtakov CLA 2017-06-02 09:12:22 EDT
(In reply to Arun Thondapu from comment #26)
> (In reply to Alexander Kurtakov from comment #25)
> > (In reply to Dani Megert from comment #24)
> > > (In reply to Martin Oberhuber from comment #22)
> > > > Anyways, since the plan was reverting the checked-in launcher binaries
> > > 
> > > That did not work. We would have lost important fixes like scrolling
> > > performance fixes for Mac. We ended up removing the Java 9 specific code and
> > > rebuilt the launcher.
> > 
> > I'm really surprised that the plan was to revert to checked in older
> > binaries. We should clearly never do such a thing. I would even say we
> > should rebuild the binaries as part of the regular build instead.
> 
> Rebuilding the binaries for every build may be an overkill IMHO, but I
> completely agree that reverting to older launcher binaries instead of doing
> a rebuild, is definitely *not* the right thing to do...

Note, that when I say rebuild the binaries for every build I mean it happening internal inside the maven call and not the current process of keeping binaries in git.
Comment 28 Martin Oberhuber CLA 2017-06-02 11:15:33 EDT
@Alex and Arun, I generally agree that rebuilding from source is better than reverting binaries, for community adoption.

In this case, though, Dani had argued on the PMC call that the change of build environment had caused late risk with M7 (see bug 515155), and he would therefore prefer going back to the pre-M7 binaries which had been working fine for 7 milestones.

Personally, I'm also fine with the rebuild from source in the new build environment, and it clearly puts us in a better position reacting to issues moving forward if we are able to build incrmental changes.
Comment 29 Daniel Wille CLA 2017-06-02 13:16:10 EDT
I copied the following from rt.equinox.binaries, commit 6f5a92c:

  org.eclipse.equinox.executable/bin/gtk/linux/x86_64/eclipse
  org.eclipse.equinox.launcher.gtk.linux.x86_64/eclipse_1624.so

into an existing Eclipse product, deleted the old shared lib (eclipse_1623.so), and launched it on a CentOS 6, x86_64 machine. The product started up fine. 

Dropping the files into the product by hand was the easiest route to double-check, so I hope that's alright. My initial attempt to incorporate the p2 repo into an existing build I have didn't pull in the updated launcher.