Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 338010

Summary: Resource.createLink() does not preserve symbolic links
Product: [Eclipse Project] Platform Reporter: Terry Parker <tparker>
Component: ResourcesAssignee: Sergey Prigogin <eclipse.sprigogin>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: P3 CC: daniel_megert, dash.alpha, eclipse.sprigogin, jamesblackburn+eclipse, john.arthorne, linux.news, markus.kell.r, miles.daffin, mober.at+eclipse, serge, sptaszkiewicz, Szymon.Brandys
Version: 3.6.1Flags: john.arthorne: review+
Target Milestone: 4.5 M5   
Hardware: PC   
OS: Linux   
Whiteboard:
Bug Depends on:    
Bug Blocks: 233019, 322821    
Attachments:
Description Flags
Proposed fix
none
inefficient fix for failing Mac tests (case-insensitive file system) markus.kell.r: review-

Description Terry Parker CLA 2011-02-23 13:58:28 EST
The behavior of Resource.createLink() differs depending on whether the link is being replaced or newly created.  
- If the link is being replaced, setLinkLocation() is called directly with the "localLocation" URI parameter
- If the link is being created, the setLinkLocation() is called with the results of FileUtil.canonicalURI(localLocation)

On Linux, canonicalizing the URI changes the path in an unexpected way and makes for an inconsistent user experience.
Comment 1 John Arthorne CLA 2011-02-24 14:10:32 EST
The inconsistency is bad - it should always be canonicalized. The reasons for this are discussed for example in bug 202095 and bug 119705.
Comment 2 Sergey Prigogin CLA 2011-02-24 14:58:41 EST
Created attachment 189739 [details]
Proposed fix

Unfortunately, canonicalization of file path has very different semantics on Windows and on Unix. On Windows it's an innocent case normalization, but on Unix it resolves symbolic links, which in many cases is highly undesirable.

Consider a scenario when there is a symbolically linked directory on Unix that is managed outside of Eclipse and may get remapped to a new location from time to time. If the user's goal is to create a linked Eclipse folder pointing to the symbolically linked directory, resolving the link in Eclipse would do a great disservice to the user. The resolved location will become invalid when the Unix symbolic link is remapped outside of Eclipse.

The proposed patch applies URI canonicalization on Windows only and resolves inconsistency between the initial creation and a later update of a linked folder.
Comment 3 John Arthorne CLA 2011-02-24 16:24:00 EST
This change worries me because there are some heavily used API methods that map from a file system location to an IResource (IWorkspaceRoot#find*). Without converting to canonical paths these methods will not always be able to map a file system location to a resource.  One common case is that the workspace location itself is a symbolic link, meaning any manipulation of non-canonical paths is sensitive to whether you are viewing the file from the perspective of that particular symlink or not. We've had endless bugs in this area before, and canonicalizing aggressively in all cases seems to be the only answer that has worked (bug 100807, bug 128460, bug 233939, likely many others).
Comment 4 Sergey Prigogin CLA 2011-02-24 17:02:47 EST
(In reply to comment #3)

There is a drastic difference between using canonicalization when comparing different paths and storing paths persistently in canonicalized form. Mandatory canonicalization of Unix paths before storing them is unacceptable since it ignores a possibility of future symbolic link changes outside of Eclipse.

It would be impossible to setup a stable working Eclipse environment in my organization if the possibility to create a linked folder pointing to a symbolically linked directory was taken away (currently we are using a workaround by creating symbolic links twice). I would like to make it clear that we are not talking about some exotic scenarios or corner cases, but about ability of hundreds of developers to continue to use Eclipse.
Comment 5 Sergey Prigogin CLA 2011-02-24 17:14:34 EST
John, would you be open to an idea to control the canonicalization policy by a new update flag, lets say RESOLVE_SYMBOLIC_LINKS or CANONICALIZE_DESTINATION_PATH?
Comment 6 Sergey Prigogin CLA 2011-03-03 12:48:42 EST
Ping.
Comment 7 Sergey Prigogin CLA 2011-03-22 20:57:34 EDT
John, following our offline conversation, could you please give a green light to the attached patch. Thanks.
Comment 8 John Arthorne CLA 2011-03-22 21:17:00 EDT
I don't like that particular fix, but I agree on the problem. I think we need to keep a canonical path in memory for use in path comparisons and look-ups. Perhaps this can be held separately in each FileStoreRoot object. However I understand that it is desirable to retain the original link location, and persist only this link location across sessions, to handle the case where the link target changes over time.

Problems with the current patch:
 - I think it will cause regression for some of the reverse-lookup scenarios mentioned in comment #3 (finding resource(s) corresponding to a given file system location)
 - Sym-links are also supported on Windows 7 so creating canonical locations on windows might still not be desirable.
Comment 9 Sergey Prigogin CLA 2011-03-22 21:41:38 EDT
(In reply to comment #8)
> I don't like that particular fix, but I agree on the problem. I think we need
> to keep a canonical path in memory for use in path comparisons and look-ups.
> Perhaps this can be held separately in each FileStoreRoot object. However I
> understand that it is desirable to retain the original link location, and
> persist only this link location across sessions, to handle the case where the
> link target changes over time.

Keeping the canonical path in memory looks like a premature optimization to me.

> Problems with the current patch:
>  - I think it will cause regression for some of the reverse-lookup scenarios
> mentioned in comment #3 (finding resource(s) corresponding to a given file
> system location)

The patch passes all tests. If it does cause issues in some exotic cases, our chances to find and fix these issues increase if the patch is applied sooner.

>  - Sym-links are also supported on Windows 7 so creating canonical locations on
> windows might still not be desirable.

What do you propose? Explicit conversion to upper case and removal of repeated slashes on Windows?
Comment 10 Szymon Brandys CLA 2011-03-23 04:57:50 EDT
Sergey, please add a test for it.
Comment 11 Serge Beauchamp CLA 2011-04-01 09:56:36 EDT
I think that one way to reconcile those 2 requirements, is to keep 2 sets of values at runtime for the linked resource location.

1 that is canonicalized, and used for alias resolution, etc...

Another that isn't, and is stored in the .project file, and used for displaying the location to the user, and editing a new location.

This would allow customers to setup linked resources that point to sym links in the file system without having those links automatically resolved and stored in the .project file, while at the same time avoiding the issues that John is referring to.

What do you guys think?
Comment 12 James Blackburn CLA 2011-04-01 10:02:45 EDT
As per the core.resources call yesterday I'm marking this bug as blocking 322821 as that bug also deals with canonicalizing links.
Comment 13 John Arthorne CLA 2011-04-01 10:57:29 EDT
(In reply to comment #11)
> I think that one way to reconcile those 2 requirements, is to keep 2 sets of
> values at runtime for the linked resource location.
> 
> 1 that is canonicalized, and used for alias resolution, etc...
> 
> Another that isn't, and is stored in the .project file, and used for displaying
> the location to the user, and editing a new location.
> 
> What do you guys think?

Yes, that makes sense. This is what I was implying in comment #8.
Comment 14 Sergey Prigogin CLA 2011-04-01 11:09:38 EDT
Agree. Recently I ran across a performance issue that persuaded me that results of canonicalization should be cached.
Comment 15 Szymon Brandys CLA 2011-04-04 07:38:54 EDT
(In reply to comment #12)
> As per the core.resources call yesterday I'm marking this bug as blocking
> 322821 as that bug also deals with canonicalizing links.

We also agreed to move it to 3.8.
Comment 16 Martin Oberhuber CLA 2011-07-12 08:35:22 EDT
So it looks like most internal in-memory code will have to continue operating on canonicalized paths. 

I'm wondering whether it would be sufficient to keep the second, non-canonicalized form in the external ProjectDescription only. 

Some clients will want to display / retrieve / operate on the non-canonical form. Looks like new API will be needed to retrieve the external (non-canonicalized) form.
Comment 17 Sergey Prigogin CLA 2011-07-12 10:01:02 EDT
(In reply to comment #16)
> So it looks like most internal in-memory code will have to continue operating
> on canonicalized paths. 
> 
> I'm wondering whether it would be sufficient to keep the second,
> non-canonicalized form in the external ProjectDescription only. 
> 
> Some clients will want to display / retrieve / operate on the non-canonical
> form. Looks like new API will be needed to retrieve the external
> (non-canonicalized) form.

I'd say it should be the other way around. Old APIs should be reserved for the original non-canonicalized form and the new ones introduced for the canonicalized form. Any client code would then be able to use just the old API without sacrificing correctness. The new APIs would be used for performance optimization to avoid excessive File.getCanonicalPath calls.
Comment 18 Martin Oberhuber CLA 2011-07-15 15:13:36 EDT
(In reply to comment #17)

I tend to agree with John that eager normalization simplifies the model, thus avoiding problems (defects) down the road.

On the other hand, I believe that original end user input needs to be preserved at any rate (in the persisted form). So when "Resource.createLink()" leads to persisting something, the original input needs to be persisted and not a canonicalized variant.

> canonicalized form. Any client code would then be able to use just the old API
> without sacrificing correctness. 

Well, what is "correctness" in this respect? Maintaining original user input? But if you have a linked resource with link target under the workspace root, it would IMO be equally valid to expect that link target be expressed as a child under the workspace root?

Whenever there's aliasing, there is IMO no clear answer of what is "correct", and using Java getCanonicalPath() is IMO not the worst idea of normalizing the model. I'd be really afreaid that although it's not documented there is client code which does expect canonical paths to be returned from Eclipse, such that paths can be compared or path prefixes can be computed ("isPathBelow").

Thus I'd rather keep existing API behavior unchanged as much as possible and add new methods that allow clients access the original user input / persisted form where it can differ from the internal (normalized) model.
Comment 19 Sergey Prigogin CLA 2011-07-15 15:36:53 EDT
(In reply to comment #18)
> Thus I'd rather keep existing API behavior unchanged as much as possible and
> add new methods that allow clients access the original user input / persisted
> form where it can differ from the internal (normalized) model.

I would agree with that if there was a consistent existing API behavior. The current behavior is to canonicalize the target location when a linked resource is created from scratch, but not when the location is modified. Any client code that  relies on canonicalization of linked resource locations is already broken since these locations are not guaranteed to be canonical.

When choosing between two possible behaviors of the existing APIs we should prefer the one that preserves more information. Canonicalization increases entropy and should be avoided.
Comment 20 Miles Daffin CLA 2011-12-21 09:51:43 EST
I see no problem in using canonicalization under the hood. However, if the user specifies a path for a particular purpose (e.g. osgi.instance.area.default in config.ini, or the base directory for a perforce workspace) this should be respected because it can create problems when it is not. For more on the problems please see the following post from 2008:

http://www.eclipse.org/forums/index.php/t/78994/
Comment 21 Szymon Brandys CLA 2012-04-25 11:25:31 EDT
We most likely will not address these bugs during this dev cycle.
Comment 22 Ben Bucksch CLA 2012-12-18 16:45:34 EST
+1 on comment 2 and 4
Sergey describes my situation.

Please do not resolve symlinks at all on Unix. You may do so when comparing 2 paths for whether they are identical, but generally do not resolve them, not even internally. I don't want to see the physical path anywhere, not when it's stored, not when other paths are derived from it, not when it's being accessed on disk using fopen() et al.

The whole point of a symlink is to hide the physical location.

This bug has created hassle to no end for me, because workspaces and projects broke again and again, and the software kept offering me physical paths, I kept changing them back to the symbolic path, and the software again stored the physical one against my will.
Comment 23 Sergey Prigogin CLA 2014-05-24 00:42:49 EDT
A proposed fix is in https://git.eclipse.org/r/#/c/27227/. It is intended for post-Luna, but it would be nice to get an early feedback. This change should also fix bug 233019 and bug 322821.
Comment 24 David Henderson CLA 2014-06-10 00:22:39 EDT
+1.  Our IT team manages installs of 3rd party software libraries, JDK install, etc as well as the mapping of physical filesystem paths for resources such as our home directories, source code repositories, etc.  Any time the physical paths change it causes all the developers a lot of pain changing user library paths, installed JRE path, source paths, workspace paths, etc.  Sometimes the old paths for the fully resolved symlinks no longer physically exist and I have no idea how to tell eclipse where the new locations for resources can be found.

(In reply to Ben Bucksch from comment #22)
> +1 on comment 2 and 4
> Sergey describes my situation.
> 
> Please do not resolve symlinks at all on Unix. You may do so when comparing
> 2 paths for whether they are identical, but generally do not resolve them,
> not even internally. I don't want to see the physical path anywhere, not
> when it's stored, not when other paths are derived from it, not when it's
> being accessed on disk using fopen() et al.
> 
> The whole point of a symlink is to hide the physical location.
> 
> This bug has created hassle to no end for me, because workspaces and
> projects broke again and again, and the software kept offering me physical
> paths, I kept changing them back to the symbolic path, and the software
> again stored the physical one against my will.
Comment 25 Sergey Prigogin CLA 2014-08-15 20:02:57 EDT
A proposed fix for this bug is sitting in Gerrit without any feedback since May. Does it mean that Platform Resources project is not interested in contributions?
Comment 26 Dani Megert CLA 2014-08-17 10:02:26 EDT
(In reply to Sergey Prigogin from comment #25)
> A proposed fix for this bug is sitting in Gerrit without any feedback since
> May. Does it mean that Platform Resources project is not interested in
> contributions?

We are just very limited on resources in this component. Szymon, please try to review during M2.
Comment 27 Szymon Ptaszkiewicz CLA 2014-09-16 09:18:16 EDT
Let's try in M3.
Comment 28 Sergey Prigogin CLA 2014-09-23 13:18:51 EDT
The proposed fix for this bug has been sitting in Gerrit for 4 months without any feedback. Please prioritize its review.
Comment 29 Sergey Prigogin CLA 2014-10-16 13:19:01 EDT
(In reply to Szymon Ptaszkiewicz from comment #27)
> Let's try in M3.

Hi Szymon. It's getting late for M3. If the patch is not reviewed now, it is going to hit the code freeze week before M3.
Comment 30 Szymon Ptaszkiewicz CLA 2014-10-17 12:24:49 EDT
(In reply to Sergey Prigogin from comment #29)
> (In reply to Szymon Ptaszkiewicz from comment #27)
> > Let's try in M3.
> 
> Hi Szymon. It's getting late for M3. If the patch is not reviewed now, it is
> going to hit the code freeze week before M3.

Hi Sergey! I'm busy with other tasks so I can't review it right now, but I may find some time after EclipseCon Europe. After a quick look, one thing that can be already improved is to use the new canCreateSymLinks method instead of the legacy conditions.
Comment 31 Sergey Prigogin CLA 2014-10-17 16:51:15 EDT
(In reply to Szymon Ptaszkiewicz from comment #30)
LinkedResourceTest updated to use canCreateSymLink.
Comment 32 Sergey Prigogin CLA 2014-11-03 13:23:18 EST
(In reply to Szymon Ptaszkiewicz from comment #30)
Hi Szymon,
I hope you enjoyed EclipseCon Europe and can now review https://git.eclipse.org/r/#/c/27227/
Comment 33 Sergey Prigogin CLA 2014-11-23 21:47:22 EST
The proposed fix has been sitting for 6 months in Gerrit waiting to be reviewed. Szymon, please indicate when you intend to review the patch.
Comment 34 Dani Megert CLA 2014-12-01 08:25:09 EST
(In reply to Sergey Prigogin from comment #33)
> The proposed fix has been sitting for 6 months in Gerrit waiting to be
> reviewed. Szymon, please indicate when you intend to review the patch.

John will review the patch before Xmas (can't promise for M4, which is already next week).
Comment 35 John Arthorne CLA 2014-12-19 16:21:56 EST
I have reviewed and released. There is just one small issue that the new test case fails if you happen to have some existing files in your temp dir with the same name used by tests. I will apply a patch on top. Thanks Sergey and sorry for the long delay on this.
Comment 37 Dani Megert CLA 2014-12-22 04:53:09 EST
This breaks Mac completely. One can't create a new workspace. That also explains the 20162 tests failures.

java.lang.NullPointerException
at org.eclipse.core.internal.utils.FileUtil.realPath(FileUtil.java:97)
at org.eclipse.core.internal.utils.FileUtil.realURI(FileUtil.java:154)
at org.eclipse.core.internal.localstore.FileSystemResourceManager.setLocation(FileSystemResourceManager.java:1035)
at org.eclipse.core.internal.localstore.FileSystemResourceManager.getStoreRoot(FileSystemResourceManager.java:565)
at org.eclipse.core.internal.localstore.FileSystemResourceManager.getStoreRoot(FileSystemResourceManager.java:568)
at org.eclipse.core.internal.localstore.FileSystemResourceManager.getStore(FileSystemResourceManager.java:522)
at org.eclipse.core.internal.resources.Resource.getStore(Resource.java:1289)
at org.eclipse.core.internal.resources.Project.assertCreateRequirements(Project.java:59)
at org.eclipse.core.internal.resources.Project.create(Project.java:290)


The problem is this line in FileUtil.realPath(FileUtil.java:97):
	realPath = realPath.setDevice(path.getDevice().toUpperCase());
the device can be 'null'. Interesting enough, this works on Linux, where I would also expect the device to be 'null'. This should be investigated.

I've released a local fix with http://git.eclipse.org/c/platform/eclipse.platform.resources.git/commit/?id=e5ab4acc688cd7ddc98435ac3b8422dcb369615d
but you need to review that code again related to this.

Also, one of the new tests constantly fails on Linux:
LinkedResourceTest.testBug338010_linkedFolderWithSymlink
Comment 38 John Arthorne CLA 2014-12-22 08:03:22 EST
The null check on the device is the right fix, thanks Dani. The device is only relevant on Windows. I don't have a Linux machine available right now, but I released a bit of cleanup in the test and more information when that test fails.
Comment 39 Sergey Prigogin CLA 2014-12-22 12:53:30 EST
(In reply to Dani Megert from comment #37)
> The problem is this line in FileUtil.realPath(FileUtil.java:97):
> 	realPath = realPath.setDevice(path.getDevice().toUpperCase());
> the device can be 'null'. Interesting enough, this works on Linux, where I
> would also expect the device to be 'null'. This should be investigated.

The NPE didn't happen on Linux because the method returns early since the file system is case sensitive.
Comment 40 Sergey Prigogin CLA 2014-12-22 13:02:25 EST
Thank you, Dani, for fixing the NPE. https://git.eclipse.org/r/#/c/38689/ contains a slight improvement for the fix.
Comment 41 Dani Megert CLA 2014-12-22 15:37:03 EST
(In reply to Sergey Prigogin from comment #40)
> Thank you, Dani, for fixing the NPE. https://git.eclipse.org/r/#/c/38689/
> contains a slight improvement for the fix.

Thanks Sergey. Submitted with http://git.eclipse.org/c/platform/eclipse.platform.resources.git/commit/?id=04fbd06ad46b7b661ac82d86224d4adbb4939582
Comment 42 Sergey Prigogin CLA 2014-12-23 17:03:18 EST
The original and the follow-up fixes are submitted.
Comment 43 Szymon Ptaszkiewicz CLA 2014-12-24 02:52:34 EST
The new tests still fail consistently on Linux and Mac. This needs to be fixed before we declare success.
Comment 44 Szymon Ptaszkiewicz CLA 2014-12-24 05:20:59 EST
(In reply to Szymon Ptaszkiewicz from comment #43)
> The new tests still fail consistently on Linux and Mac. This needs to be
> fixed before we declare success.

LinkedResourceTest.testBug338010_linkedFolderWithSymlink fails also on Windows (you need to run tests with administrator privileges there to be able to test symlink support).
Comment 45 Sergey Prigogin CLA 2014-12-24 23:57:01 EST
(In reply to Szymon Ptaszkiewicz from comment #43)
> The new tests still fail consistently on Linux and Mac. This needs to be
> fixed before we declare success.

I wasn't able to reproduce the failure on Linux when running as a plug-in test from Eclipse. Judging by the error message at http://download.eclipse.org/eclipse/downloads/drops4/I20141223-0800/testresults/html/org.eclipse.core.tests.resources_linux.gtk.x86_64_8.0.html 

"Could not create link at location: ROOT/1741477380961/symlink/dir2 

junit.framework.AssertionFailedError: Could not create link at location: ROOT/1741477380961/symlink/dir2
at org.eclipse.core.tests.resources.LinkedResourceTest.testBug338010_linkedFolderWithSymlink(LinkedResourceTest.java:1963)",

it looks like getRandomLocation() returned ROOT/1741477380961, which is obviously wrong. The problem was most likely in getTempDir(), which apparently returned ROOT.

Will take a look at Mac and Windows next.
Comment 46 Sergey Prigogin CLA 2014-12-29 00:31:24 EST
(In reply to Szymon Ptaszkiewicz from comment #44)
> LinkedResourceTest.testBug338010_linkedFolderWithSymlink fails also on
> Windows (you need to run tests with administrator privileges there to be
> able to test symlink support).

All LinkedResourceTest test pass for me on Windows when run as Administrator.
Comment 47 Sergey Prigogin CLA 2014-12-29 00:34:09 EST
(In reply to Szymon Ptaszkiewicz from comment #43)

Unfortunately, I wasn't able to run any tests on Mac since all plugin tests fail there with:

java.lang.IllegalStateException: Unable to acquire application service. Ensure that the org.eclipse.core.runtime bundle is resolved and started (see config.ini).
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:78)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:380)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:235)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:648)
	at org.eclipse.equinox.launcher.Main.basicRun(Main.java:603)
	at org.eclipse.equinox.launcher.Main.run(Main.java:1465)
	at org.eclipse.equinox.launcher.Main.main(Main.java:1438)
Comment 48 Sergey Prigogin CLA 2015-01-04 20:53:52 EST
> (In reply to Szymon Ptaszkiewicz from comment #43)
To investigate what is wrong with getRandomLocation() method in the Linux nightly build I'd suggest temporarily replacing the last line of org.eclipse.core.tests.harness.FileSystemHelper.getTempDir() with:

    Path path = new Path(tempPath);
    org.junit.Assert.assertTrue(path.toOSString() + " is not absolute", path.isAbsolute());
    return path;

and adding

    org.junit.Assert.assertTrue(path.toOSString() + " is not absolute", path.isAbsolute());

before the last line of FileSystemHelper.getRandomLocation(IPath)
Comment 49 Markus Keller CLA 2015-01-08 06:37:06 EST
(In reply to Sergey Prigogin from comment #45)
The failure on Linux is not where it appears to be. LinkedResourceWithPathVariableTest extends LinkedResourceTest, and the failure only happens in the former. This is hard to see because the Ant JUnit runner drops the test suite hierarchy and only shows a flat list. You can see the actual test class in the tooltip on the failing test's name in http://download.eclipse.org/eclipse/downloads/drops4/I20150106-0800/testresults/html/org.eclipse.core.tests.resources_linux.gtk.x86_64_8.0.html

Please check if the released changes cause trouble with path variables. If they don't, then you can adjust the test.
Comment 50 Markus Keller CLA 2015-01-08 09:54:27 EST
Created attachment 249788 [details]
inefficient fix for failing Mac tests (case-insensitive file system)

(In reply to Sergey Prigogin from comment #47)
> Unfortunately, I wasn't able to run any tests on Mac since all plugin tests
> fail there with:

They run fine for me out of the box. Please delete your launch configuration and create a new one. Make sure you run with Java 7 or 8 (I used 1.8.0_20).

The bad file name looks like a bug in
LocalFile#fetchInfo(int, IProgressMonitor). At line 156, the FileInfo wrongly gets the file name of the LocalFile, instead of the actual native file name. Only the latter is correct on case-insensitive file systems.

IFileInfo#getName() doesn't specify that the name can be empty, so all implementations of FileStore#fetchInfo(int, IProgressMonitor) that return an incomplete IFileInfo are actually wrong.
Comment 51 Sergey Prigogin CLA 2015-01-08 21:17:08 EST
(In reply to Markus Keller from comment #49)
Thanks for pointing me to LinkedResourceWithPathVariableTest.

A proposed fix is in https://git.eclipse.org/r/#/c/39246/
Please review and commit.
Comment 52 Sergey Prigogin CLA 2015-01-08 23:41:35 EST
(In reply to Markus Keller from comment #50)
The workaround proposed in attachment 249788 [details] for the bug in LocalFile.fetchInfo on Mac is quite expensive since File.getCanonicalPath method is pretty slow. The fix should be moved to the Mac-specific code to avoid unnecessary overhead on other platforms. On Mac we can choose among the following possible solutions:

1. new Path(file.getCanonicalPath()).lastSegment()
2. file.toPath().toRealPath(LinkOption.NOFOLLOW_LINKS).toFile().getName()  (Java 1.7+)
3. String[] names = file.getParentFile().list(new FilenameFilter() {
       public boolean accept(File  dir, String name) {
           return name.equalsIgnoreCase(file.getName());
       }
   });
   if (names.length != 1)
       // Handle error
   return names[0];

We can determine experimentally which of the methods is faster.

Unfortunately, I can't do much on Mac since all plug-in tests fail with the same error on my Mac (with Java 1.8.0_25). I tried recreating launch configurations, resetting target platform, and reinstalling Eclipse from scratch. Nothing helps.
Comment 53 Dani Megert CLA 2015-01-09 03:46:48 EST
(In reply to Sergey Prigogin from comment #51)
> (In reply to Markus Keller from comment #49)
> Thanks for pointing me to LinkedResourceWithPathVariableTest.
> 
> A proposed fix is in https://git.eclipse.org/r/#/c/39246/
> Please review and commit.

Submitted with http://git.eclipse.org/c/platform/eclipse.platform.resources.git/commit/?id=a228eabaa2a0d2e06e8537893826bd357cf9207c
Comment 54 Markus Keller CLA 2015-01-09 14:25:43 EST
(In reply to Sergey Prigogin from comment #52)
I agree that File#getCanonicalPath() could be a performance problem, since it resolves the whole path instead of only the file name, and since it's not necessary on Windows. I'm not sure about Linux. AFAIK, you can also have case-insensitive file systems there. Unfortunately, I don't see a more direct way than your (3.) via file.getParentFile().list(..).

> Unfortunately, I can't do much on Mac since all plug-in tests fail with the
> same error on my Mac (with Java 1.8.0_25). I tried recreating launch
> configurations, resetting target platform, and reinstalling Eclipse from
> scratch. Nothing helps.

E.g. FileUtilTest still launches without problems for me with I20150106-0800 and 1.8.0_25 on OS X 10.10. It's hard to tell from here what's wrong with your setup. Make sure you use a recent I-build. If you e.g. use 4.4.1 and then check out bundles like org.eclipse.core.runtime and org.eclipse.core.tests.harness from eclipse.platform.runtime from master, then you're running with an unsupported mix.

Try launching your main Eclipse with the "-clean" command line argument. In the launch configuration, verify these settings:
- Main tab: Clear workspace, Run a product: "org.eclipse.sdk.ide"
- Plug-ins tab: Launch with: "all workspace and enabled target plug-ins"
- Configuration tab: Clear config area before launching, Generate config.ini with default content
Comment 55 Sergey Prigogin CLA 2015-01-09 21:07:43 EST
(In reply to Markus Keller from comment #54)
Figured out why I was not able to run plugin tests on Mac. It was an unfortunate combination of bugs 457175 and 457176.
Comment 56 Sergey Prigogin CLA 2015-01-11 20:09:52 EST
It turned out that among the three solutions proposed in comment #52 only the third one is viable.

The solution #1 would return a wrong name for a symbolic link.
The solution #2 doesn't work due to a bug in java.nio.file.Path.toRealPath(LinkOption.NOFOLLOW_LINKS). This method produces a wrong result on Mac, identical to the one returned by FileUtil.realPath! I've submitted a bug report against JDK 1.8.0_25 and will include a reference to the bug once the report goes through review.

The solution #3 is implemented in https://git.eclipse.org/r/#/c/39359/
Please review and submit.
Comment 57 Markus Keller CLA 2015-01-12 15:23:03 EST
(In reply to Sergey Prigogin from comment #56)
> The solution #3 is implemented in https://git.eclipse.org/r/#/c/39359/
> Please review and submit.

Looks good, except for the NPE if someone ever tries to access the root of the file system like this:

    new LocalFile(new File("/")).fetchInfo();

Fixed in patch set 3.

org.eclipse.core.tests.resources.AutomatedTests are mostly green now on Mac, but I see 5 failures in org.eclipse.core.tests.resources.content.IContentTypeManagerTest now. Not sure if they have anything to do with this bug. First failure:

junit.framework.AssertionFailedError: 2.setup.4
	at junit.framework.Assert.fail(Assert.java:57)
	at junit.framework.Assert.assertTrue(Assert.java:22)
	at org.eclipse.core.tests.harness.BundleTestingHelper.runWithBundles(BundleTestingHelper.java:132)
	at org.eclipse.core.tests.resources.content.IContentTypeManagerTest.testAlias(IContentTypeManagerTest.java:174)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
Comment 58 Sergey Prigogin CLA 2015-01-12 17:30:09 EST
(In reply to Markus Keller from comment #57)

All org.eclipse.core.tests.resources.AutomatedTests are green for me on Mac with the patch. On which platform do you see failures in IContentTypeManagerTest?
Comment 59 Sergey Prigogin CLA 2015-01-12 17:39:27 EST
(In reply to Sergey Prigogin from comment #58)

Five IContentTypeManagerTest tests fail when run as org.eclipse.ui.ide.workbench but pass in headless mode. This strange behavior is independent from the patch.
Comment 60 Szymon Ptaszkiewicz CLA 2015-01-13 05:01:02 EST
(In reply to Markus Keller from comment #57)
> org.eclipse.core.tests.resources.AutomatedTests are mostly green now on Mac,
> but I see 5 failures in
> org.eclipse.core.tests.resources.content.IContentTypeManagerTest now. Not
> sure if they have anything to do with this bug.

These are known failures tracked in bug 296811.
Comment 61 Markus Keller CLA 2015-01-13 06:19:26 EST
Patch set 4 looks good, but the

    lastName.toLowerCase().equals(lastName.toUpperCase())

needs an explanation in the code, e.g.:

/*
 * matches special cases:
 * - root with name "", where file.getParentFile() would return null
 * - names whose upper and lower case are the same, i.e. there can't
 *     be different casing variants on the file system
 */
Comment 62 Dani Megert CLA 2015-01-16 03:38:31 EST
The failures are gone in N20150115-2000.