Bugzilla – Full Text Bug Listing
| Summary: | Contribute symlink-enabled liblocalfile for Linux x86_64 | ||
|---|---|---|---|
| Product: | [Eclipse Project] Platform | Reporter: | Martin Oberhuber <mober.at+eclipse> |
| Component: | Resources | Assignee: | John Arthorne <john.arthorne> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | blocker | ||
| Priority: | P1 | CC: | gheorghe, john.arthorne |
| Version: | 3.3 | Flags: | dj.houghton:
review+
|
| Target Milestone: | 3.3 RC1 | ||
| Hardware: | PC | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Bug Depends on: | |||
| Bug Blocks: | 105554 | ||
| Attachments: | |||
|
Description
Martin Oberhuber
Created attachment 65211 [details]
liblocalfile_1_0_0.so built on RHEL5 tested on openSUSE 10.2
Attached archive contains liblocalfile_1_0_0.so built for Linux x86_64 on Redhat Enterprise Linux 5 Client, tested on openSUSE 10.2
Created attachment 65212 [details]
Patch making EFS aware of the new lib for Linux x86_64
Attached patch makes the EFS Local provider's Java code, as well as the test cases, aware of the new library. It also updates the MANIFEST of the Linux x86_64 fragment and adds a BUILD_INFO.txt with exact build information for the library.
Legal Message: I, Martin Oberhuber, declare that I developed attached code from scratch, without referencing any 3rd party materials except material licensed under the EPL. I am authorized by my employer, Wind River Systems Inc., to make this contribution under the EPL. You're keeping me busy! ;) Assigning to myself for review/release. Well an x86_64 machine happens to be the fastest in our lab, and I wanted to use this for my work on bug 105554, so this is more or less a by-product :-) Reviewed and released. I simplified the condition on LocalFileSystem.attributes() to: os.equals(Constants.OS_LINUX) || (os.equals(Constants.OS_SOLARIS) && arch.equals(Constants.ARCH_SPARC)) I.e., all known Linux architectures support the symlink attribute. (In reply to comment #6) > I simplified the condition on LocalFileSystem.attributes() to: > I.e., all known Linux architectures support the symlink attribute. While this seems logical at first sight, I'm not so sure if this is a good idea... with 3.2, a Linux-IA64 build was released, see http://download.eclipse.org/eclipse/downloads/drops/R-3.2-200606291905/ I'm not sure who contributed that build, or what other Linux arches might be built. I'm also not sure if that build would contain any native liblocalfile at all, or live with the plain Java implementation. Basically your change means that whoever assembles a new Linux build of Eclipse, has to either build the latest liblocalfile, or live with the Java-only implementation. For other arches (AIX, HP-UX) it's even worse... they cannot build their own liblocalfile fragment and get to the symlink benefits without touching the Java code :-( I think what we really need, is having the liblocalfile push its version number into the Java fragment host when it's loaded. That way, the check is more robust: * no liblocalfile -> Java-only version * liblocalfile but no version pushed -> no symlinks * liblocalfile, and version (or attributes) pushed -> handle as needed I don't think this is extremely critical, but at least it's worth a bug report and we should consider such a solution. Even if it means re-building the fragments we currently have (I'd volunteer for Linux x86, x86_64 and solaris since I can do these pretty fast; Corey would need to do Linux ppc). Right now I have the environment handy for building these, I'm not sure what it'll be like a few months from now... For other potential Linux architectures, we essentially have to guess whether they support symlinks. If we guess that they do support symlinks (my version), then they will be forced to recompile their native library for 3.3. If we guess that they do not support symlinks (your version), then it will be impossible for them to support symlinks because they can't modify the Java code. I still prefer the first option... I'm not sure what you mean by "having the liblocalfile push its version number into the Java fragment host when it's loaded". Are you suggesting that we increment the liblocalfile version number to match the fragment version number? That would certainly break HPUX, for example, which hasn't been recompiled in 3.3. Hm.. ok, wrt Linux your version is the better one.
By "pushing", I meant that the JNI code, when the native lib is first loaded, either does the equivalent JNI for
LocalFileSystem.getInstance().attributes = (what native thinks);
or we add another instance variable to the LocalFileNatives class, that holds
the attributes a Native thinks they should be, and use these in LocalFileSystem.
I think that the latter is the cleaner solution, since native code remains with its "own" class. Pushing the attributes can be done in the JNI_OnLoad() method which is invoked by the VM when loading the DLL, see
http://java.sun.com/j2se/1.5.0/docs/guide/jni/spec/invocation.html
(This callback only exists since j2se 1.4)
Hm, looks like JNI_OnLoad wont work... what about this:
public abstract class LocalFileNatives {
private static final native int nativeAttributes();
public static final int attributes() {
try {
return nativeAttributes();
} catch(NoSuchMethodError) {
return -1;
}
}
}
This would allow querying the native lib for what it thinks its attributes are, and in case the lib is an old one that does not have this method yet, return -1 for unknown --> allowing fallback to the current way of computing attributes.
Created attachment 65231 [details]
Additional patch allowing to get supported attributes from native lib
Attached additional patch contains the source code that allows getting supported attributes right from the native library implementation. This is much more robust than just guessing in Java. Includes a fallback to use the old guessing method when the native library does not support the new call for getting nativeAttributes. Because of this fallback, this should be a safe fix.
Tested on x86_64, both with an old native lib that did not have the new nativeAttributes() method defined, and with a new lib that attributes are returned correctly.
Legal Message: I, Martin Oberhuber, declare that I developed attached code from
scratch, without referencing any 3rd party materials except material licensed
under the EPL. I am authorized by my employer, Wind River Systems Inc., to make
this contribution under the EPL.
Created attachment 65234 [details]
Updated liblocalfile_1_0_0.so from RHEL5, from the patched sources
Attached archive contains an updated build of liblocalfile_1_0_0.so, built on RHEL5, tested on openSUSE 10.2 (automated SymlinkTest).
The version number in MANIFEST.MF should probably be reved up to 1.0.101.qualifier
Can you enter a new bug report for this? No need to copy over the patches, I just want to track it separately from this bug about x86_64. This isn't a new problem, and I want to focus right now on getting the recursive symlink problem solved. *** Bug 180806 has been marked as a duplicate of this bug. *** (In reply to comment #13) > Can you enter a new bug report for this? Done, it's bug 184534. Follow-up discussions will be there. I should have done that right away, sorry. Comment on attachment 65231 [details] Additional patch allowing to get supported attributes from native lib Obsoleting the patch for reading liblocalfile capabilites, which is really offtopic here. This patch is now provided via the correct bug, on bug 184534. (In reply to comment #6) > Reviewed and released. I simplified the condition on Verification FAILED with I20070502-0010 While you did apply the patch and BUILD_INFO.txt was added, the native lib at org.eclipse.core.filesystem.linux.x86_64/os/linux/x86_64/liblocalfile_1_0_0.so is still at version CVS 1.1 so it is still the old one. Please commit the shared lib from attachment 1 [details] or attachment 4 [details] of this bug -- this was not part of the patch because it is binary and binary files do not play well in patches... (In reply to comment #17) > Please commit the shared lib from attachment... Correct attachment ids in this bug are attachment 65211 [details] or attachment 65234 [details]. Darn, my fault. I contributed the source patch but not the updated library. If there is a rebuild of M7 I will ask for this to be included. Otherwise I will wait until RC1. I don't want to request a rebuild for this bug only since it is low impact. I have committed the new library to HEAD now. Map updated. This has been contributed to the upcoming M7 build. I have verified that the new library appears in I20070503-1400. Martin, if you could verify this works as expected it would be appreciated. Verified with I20070503-1400 on openSUSE 10.2 x86_64 3.3M7 crashes for me on RH4_x64 when I try to create a new project or class. I've tried recompiling the native on this platform and it seems to fix it for me. Here is the stack trace from the crash: Stack: [0x000000004113b000,0x000000004123c000), sp=0x0000000041238a30, free space=1014k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) C [ld-linux-x86-64.so.2+0x74d7] C [ld-linux-x86-64.so.2+0x8c70] C [libc.so.6+0xf7ad8] C [ld-linux-x86-64.so.2+0xaab0] C [libdl.so.2+0x1054] Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) j java.lang.ClassLoader$NativeLibrary.load(Ljava/lang/String;)V+0 j java.lang.ClassLoader.loadLibrary0(Ljava/lang/Class;Ljava/io/File;)Z+300 j java.lang.ClassLoader.loadLibrary(Ljava/lang/Class;Ljava/lang/String;Z)V+148 j java.lang.Runtime.loadLibrary0(Ljava/lang/Class;Ljava/lang/String;)V+54 j java.lang.System.loadLibrary(Ljava/lang/String;)V+7 j org.eclipse.core.internal.filesystem.local.LocalFileNatives.<clinit>()V+10 v ~StubRoutines::call_stub j org.eclipse.core.internal.filesystem.local.LocalFile.fetchInfo(ILorg/eclipse/core/runtime/IProgressMonitor;)Lorg/eclipse/core/filesystem/IFileInfo;+0 j org.eclipse.core.filesystem.provider.FileStore.fetchInfo()Lorg/eclipse/core/filesystem/IFileInfo;+3 j org.eclipse.jdt.internal.ui.wizards.JavaProjectWizardSecondPage.rememberExistingFiles(Ljava/net/URI;)V+16 j org.eclipse.jdt.internal.ui.wizards.JavaProjectWizardSecondPage.updateProject(Lorg/eclipse/core/runtime/IProgressMonitor;)V+138 j org.eclipse.jdt.internal.ui.wizards.JavaProjectWizardSecondPage.performFinish(Lorg/eclipse/core/runtime/IProgressMonitor;)V+27 j org.eclipse.jdt.internal.ui.wizards.JavaProjectWizard.finishPage(Lorg/eclipse/core/runtime/IProgressMonitor;)V+5 j org.eclipse.jdt.internal.ui.wizards.NewElementWizard$2.run(Lorg/eclipse/core/runtime/IProgressMonitor;)V+5 j org.eclipse.jdt.internal.core.BatchOperation.executeOperation()V+8 j org.eclipse.jdt.internal.core.JavaModelOperation.run(Lorg/eclipse/core/runtime/IProgressMonitor;)V+45 j org.eclipse.core.internal.resources.Workspace.run(Lorg/eclipse/core/resources/IWorkspaceRunnable;Lorg/eclipse/core/runtime/jobs/ISchedulingRule;ILorg/eclipse/core/runtime/IProgressMonitor;)V+80 j org.eclipse.jdt.core.JavaCore.run(Lorg/eclipse/core/resources/IWorkspaceRunnable;Lorg/eclipse/core/runtime/jobs/ISchedulingRule;Lorg/eclipse/core/runtime/IProgressMonitor;)V+40 j org.eclipse.jdt.internal.ui.actions.WorkbenchRunnableAdapter.run(Lorg/eclipse/core/runtime/IProgressMonitor;)V+9 j org.eclipse.jface.operation.ModalContext$ModalContextThread.run()V+15 v ~StubRoutines::call_stub Created attachment 66509 [details]
Recompiled native library on RH4_x64
Martin, are you sure you compiled/verified on a 64-bit machine? I looked at this with Bogdan and we couldn't see any reason why the library recompiled on RHEL4 worked fine, while your library compiled on RHEL5 did not. We ran LDD on the libraries and it showed the same link dependencies in both cases. Setting as P1/blocker since Eclipse is currently unusable on certain x86_64 machines. Yes, I've definitely been compiling on RHEL5 and testing on openSUSE 10.2 x86_64 -- it's actually been my main work machine over the past two weeks.
I'm sorry that it broke Bogdan, but I didn't have an older x86_64 machine available. It would be interesting to know Bogdan's glibc version: Try
rpm -q glibc
My RHEL5 box was glibc-2.5-12 (as documented in BUILD_INFO.txt), my openSUSE box was slightly newer.
On my openSUSE box, ldd liblocalfile_1.0.0.so gives:
libc.so.6 => /lib64/libc.so.6 (0x00002b7d9a155000)
/lib64/ld-linux-x86-64.so.2 (0x000055555554000)
strings liblocalfile_1.0.0.so includes
GLIBC_2.2.5
so I'd assume this is the minimum glibc version this can run on.
Bogdan: Can you let us know your glibc version (see above comment)? Martin: Can you try running with the library Bogdan attached on your machine. If that version works for you, I'd prefer to release that copy to ensure we work on the most possible OS versions. Bogdan's lib is running fine for me on openSUSE 10.2 -- the Unittests related to symlinks all pass, and the manual tests also pass. Bogdan would you mind also providing the compiler version you're using so when your lib is released we can fill in the information into BUILD_INFO.txt? See http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.core.filesystem.linux.x86_64/BUILD_INFO.txt?view=markup Here's the requested info: glibc-2.3.4-2.25 gcc-3.4.6-3 RHEL 4 Linux 2.6.9-42.EL x86_64 I have released the updated library and BUID_INFO.txt. |