Comment 1 Denis Roy 2011-11-08 16:57:18 EST

The .nfs0000000 etc.  file represents a file that was deleted, but was still in use.  The OS is just waiting for the file to no longer be used to completely remove it.

Comment 2 Greg Watson 2011-11-08 18:12:17 EST

Unfortunately the exception causes the delete to abort, so a bunch of the old files are left there.

Comment 3 Greg Watson 2011-11-08 18:13:30 EST

Also, it's been three hours and the delete is still failing. What could still be using the file?

Comment 4 Denis Roy 2011-11-08 18:58:39 EST

The hudson master process is still listing the file as being in use...  Odd.

Comment 5 Denis Roy 2011-11-09 09:28:58 EST

It appears Hudson has released its hold on your file.  Have you been able to delete your workspace?

Comment 6 Greg Watson 2011-11-09 11:22:30 EST

Yes, in that case. However I'm having the same problem again now.

Comment 7 Greg Watson 2011-11-12 17:19:23 EST

Closing this as it seems to be ok now. I'll open a bug against hudson to handle this situation more gracefully.

Comment 8 Greg Watson 2011-12-05 14:38:44 EST

I'm getting this problem again for the ptp-photran-release job. I'm also not able to build because the workspace is in an inconsistent state, and I'm not able to delete the workspace because of this problem.

Comment 9 Eclipse Webmaster 2011-12-05 15:11:11 EST

Looks like the master process was holding the file.  I"ve restarted the master, and cleared the workspace.

Did you actually file a bug with the Hudson folks?

-M.

Comment 10 Eclipse Webmaster 2011-12-05 16:39:42 EST

*** Bug 365668 has been marked as a duplicate of this bug. ***

Comment 11 Greg Watson 2011-12-05 18:51:11 EST

(In reply to comment #9)
> Looks like the master process was holding the file.  I"ve restarted the master,
> and cleared the workspace.
> 
> Did you actually file a bug with the Hudson folks?

Yes, 362652. 

I'm now getting this on ptp-master-4.1. I'm trying to debug the build, but I'm getting this all the time now. I basically can't do anything until whatever is holding the file goes away, which sometimes takes a day.

Comment 12 Eclipse Webmaster 2011-12-13 11:10:59 EST

Greg are you still seeing this issue since the changes last week?

-M.

Comment 13 Greg Watson 2011-12-13 11:27:11 EST

No, not since the change to a non-NFS filesystem.

Comment 14 Denis Roy 2011-12-14 10:17:28 EST

Adding a dep. to bug 366696, since that could impact functionality over here.

Comment 15 Dennis Huebner 2011-12-15 05:26:51 EST

(In reply to comment #14)
> Adding a dep. to bug 366696, since that could impact functionality over here.

Hi guys,
now I'm also getting NFS IOException:
java.io.IOException: Unable to delete /opt/users/hudsonbuild/.hudson/jobs/Xtext-test/workspace/org.eclipse.xtext.git/.git/objects/pack/.nfs00000000062b20030000d8c2
	at hudson.Util.deleteFile(Util.java:263)
	at hudson.Util.deleteRecursive(Util.java:305)
	at hudson.Util.deleteContentsRecursive(Util.java:224)

Is this somehow related to the git plugin we are using on hudson?

Comment 16 Eclipse Webmaster 2011-12-15 10:02:34 EST

(In reply to comment #15)
> 
> Is this somehow related to the git plugin we are using on hudson?

It seems like Hudson(Java) is making some assumptions about the 'speed' of file releases/closes/unlink operations that don't work well with NFS files.

The thing is this should only really happen on the master.  The slaves all have local workspaces, only the master uses NFS.

Here's a thought: I see that jobs can be configured to use a 'custom' workspace, are you willing to trying setting that to: /opt/users/hudson/workspace/mybuild ?  That path is local to all Hudson instances, so perhaps it will remove the 'indirect' NFS dependency that seems to be created by default(job workspaces seem to default to /hudson/job/path/jobs/myjob/workspace which in our case on NFS).

-M.

Comment 17 Dennis Huebner 2011-12-15 10:10:06 EST

(In reply to comment #16)
> (In reply to comment #15)
> > 
> > Is this somehow related to the git plugin we are using on hudson?
> 
> It seems like Hudson(Java) is making some assumptions about the 'speed' of file
> releases/closes/unlink operations that don't work well with NFS files.
> 
> The thing is this should only really happen on the master.  The slaves all have
> local workspaces, only the master uses NFS.
> 
> Here's a thought: I see that jobs can be configured to use a 'custom'
> workspace, are you willing to trying setting that to:
> /opt/users/hudson/workspace/mybuild ?  That path is local to all Hudson
> instances, so perhaps it will remove the 'indirect' NFS dependency that seems
> to be created by default(job workspaces seem to default to
> /hudson/job/path/jobs/myjob/workspace which in our case on NFS).
> 
> -M.

I will try this solution... 

Notice: Using custom workspace location prevents jobs from using "parallel launching (beta) feature".

Comment 18 Dennis Huebner 2011-12-15 10:52:57 EST

(In reply to comment #17)
> (In reply to comment #16)
> > (In reply to comment #15)
> > > 
> > > Is this somehow related to the git plugin we are using on hudson?
> > 
> > It seems like Hudson(Java) is making some assumptions about the 'speed' of file
> > releases/closes/unlink operations that don't work well with NFS files.
> > 
> > The thing is this should only really happen on the master.  The slaves all have
> > local workspaces, only the master uses NFS.
> > 
> > Here's a thought: I see that jobs can be configured to use a 'custom'
> > workspace, are you willing to trying setting that to:
> > /opt/users/hudson/workspace/mybuild ?  That path is local to all Hudson
> > instances, so perhaps it will remove the 'indirect' NFS dependency that seems
> > to be created by default(job workspaces seem to default to
> > /hudson/job/path/jobs/myjob/workspace which in our case on NFS).
> > 
> > -M.
> 
> I will try this solution... 
> 
> Notice: Using custom workspace location prevents jobs from using "parallel
> launching (beta) feature".

Wont work:

Started by user dhubner
Building remotely on Fastlane
java.io.IOException: Failed to mkdirs: /opt/users/hudson/workspace/Xtext-tests
	at hudson.FilePath.mkdirs(FilePath.java:817)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1227)
	at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:507)
	at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:424)
	at hudson.model.Run.run(Run.java:1367)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:145)
[FINDBUGS] Skipping publisher since build result is FAILURE

Comment 19 Eclipse Webmaster 2011-12-15 11:00:14 EST

Sorry my bad.  That path should be /opt/users/hudsonbuild/workspace .

It also just occurred to me that this may remove the ability to access the build artifacts directly(since this location isn't shared with build.eclipse.org)


-M.

Comment 20 Dennis Huebner 2012-01-16 04:55:03 EST

*** Bug 363683 has been marked as a duplicate of this bug. ***

Comment 21 Dennis Huebner 2012-01-16 05:00:53 EST

Due to less commit activity findbugs can be triggered locally before check in and/or release.

Comment 22 Dennis Huebner 2012-01-16 05:02:10 EST

(In reply to comment #21)
> Due to less commit activity findbugs can be triggered locally before check in
> and/or release.

Sorry wrong bug. Please ignore.

Comment 23 Denis Roy 2012-01-20 14:16:15 EST

*** Bug 369199 has been marked as a duplicate of this bug. ***

Comment 24 Alexander Nyßen 2012-01-28 05:53:34 EST

gef4-nightly-tycho seems to have run into the problem again:

Exception: 
Stacktrace:
java.io.IOException: Unable to delete /opt/users/hudsonbuild/.hudson/jobs/gef4-nightly-tycho/workspace/.git/objects/pack/.nfs00000000027da26e0000ff95
	at hudson.Util.deleteFile(Util.java:263)
	at hudson.Util.deleteRecursive(Util.java:305)
	at hudson.Util.deleteContentsRecursive(Util.java:224)
	at hudson.Util.deleteRecursive(Util.java:304)
	at hudson.Util.deleteContentsRecursive(Util.java:224)
	at hudson.Util.deleteRecursive(Util.java:304)
	at hudson.Util.deleteContentsRecursive(Util.java:224)
	at hudson.Util.deleteRecursive(Util.java:304)
	at hudson.Util.deleteContentsRecursive(Util.java:224)

Comment 25 Eclipse Webmaster 2012-01-31 14:24:13 EST

(In reply to comment #24)
> gef4-nightly-tycho seems to have run into the problem again:

The .nfs files have 'expired'.

-M.

Comment 26 Alexander Nyßen 2012-02-01 08:51:35 EST

(In reply to comment #25)
> (In reply to comment #24)
> > gef4-nightly-tycho seems to have run into the problem again:
> 
> The .nfs files have 'expired'.
> 
> -M.

While it has worked for one build in between, now gef4-nightly-tycho is broken again (same issue). Seems to be a never ending story... What can we do to get rid of this?

Comment 27 Dennis Huebner 2012-02-01 09:18:45 EST

(In reply to comment #26)
> (In reply to comment #25)
> > (In reply to comment #24)
> > > gef4-nightly-tycho seems to have run into the problem again:
> > 
> > The .nfs files have 'expired'.
> > 
> > -M.
> 
> While it has worked for one build in between, now gef4-nightly-tycho is broken
> again (same issue). Seems to be a never ending story... What can we do to get
> rid of this?

It happens mostly in the job's workspace folder, because there is a lot of data traffic during the build run. The only reason why rsync to build.eclipse.org it necessary, at least for my projects, is that I need file access to latest builds or build folder in the whole. However nobody wants access to WORKSPACE which is normally cleaned after every build. So if we can somehow exclude workspace/workspaces from rsync it wouldn't happen anymore/not so often. I'm ready to play a guinea pig in this experiment. :)

Comment 28 Alexander Nyßen 2012-03-08 01:52:37 EST

gef4-nightly-tycho had it again on 12-03-04, build #278: 

See <https://hudson.eclipse.org/hudson/job/gef4-nightly-tycho/278/>

------------------------------------------
Started by timer
Building on master
Checkout:workspace / <https://hudson.eclipse.org/hudson/job/gef4-nightly-tycho/ws/> - hudson.remoting.LocalChannel@77422494
Using strategy: Default
Last Built Revision: Revision 35cf5126045bba246c1b55d9a210ef30c90ecaa2 (origin/master)
Checkout:workspace / <https://hudson.eclipse.org/hudson/job/gef4-nightly-tycho/ws/> - hudson.remoting.LocalChannel@77422494
Wiping out workspace first.
java.io.IOException: Unable to delete <https://hudson.eclipse.org/hudson/job/gef4-nightly-tycho/ws/.git/objects/pack> - files in dir: [<https://hudson.eclipse.org/hudson/job/gef4-nightly-tycho/ws/.git/objects/pack/.nfs0000000000db620300010961]>
	at hudson.Util.deleteFile(Util.java:262)
	at hudson.Util.deleteRecursive(Util.java:305)
	at hudson.Util.deleteContentsRecursive(Util.java:224)
	at hudson.Util.deleteRecursive(Util.java:304)
	at hudson.Util.deleteContentsRecursive(Util.java:224)
	at hudson.Util.deleteRecursive(Util.java:304)
	at hudson.Util.deleteContentsRecursive(Util.java:224)
	at hudson.FilePath$10.invoke(FilePath.java:838)
	at hudson.FilePath$10.invoke(FilePath.java:836)
	at hudson.FilePath.act(FilePath.java:758)
	at hudson.FilePath.act(FilePath.java:740)
	at hudson.FilePath.deleteContents(FilePath.java:836)
	at hudson.plugins.git.GitSCM$3.invoke(GitSCM.java:858)
	at hudson.plugins.git.GitSCM$3.invoke(GitSCM.java:845)
	at hudson.FilePath.act(FilePath.java:758)
	at hudson.FilePath.act(FilePath.java:740)
	at hudson.plugins.git.GitSCM.gerRevisionToBuild(GitSCM.java:845)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:622)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1229)
	at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:507)
	at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:424)
	at hudson.model.Run.run(Run.java:1367)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:145)
Recording test results
Archiving artifacts

Comment 29 Ed Willink 2012-03-31 06:33:23 EDT

After failing to wiope out the workspace becuiasde of 

    Unable to delete /opt/users/hudsonbuild/.hudson/jobs/buckminster-mdt-ocl-tools-3.2-master/workspace/org.eclipse.ocl.git/.git/objects/pack/.nfs0000000000f7000e00000001

thwe build now fails due to GIT startup problems.

How do we delete these .nfst files which are far from current?

Comment 30 Eclipse Webmaster 2012-04-02 11:36:47 EDT

The easiest solution is probably to add some cleanup commands to the shell section of your build job (something like 'find /opt/users/hudsonbuild/.hudson/jobs/buckminster-mdt-ocl-tools-3.2-master -name ".nfs*" -exec rm {} \;' )

I've cleared the current lock files.

-M.

Comment 31 Alexander Nyßen 2012-04-06 10:08:29 EDT

Well, I actually do not regard this as being resolved (today the gef4 build failed again because of the same exception). 

However, I was wondering why that always happened to be the case in case of my gef4-nightly-tycho build and not in case of the gef-nighly-tycho build (which are pretty much the same). It seems to be that the cause was that gef4 was restricted to run on master only (I had set this because of problems with some slaves that occurred when setting up the build). Having changed that, gef4 does seem to build as well. Therefore it seems to be kind of "working for me" now. 

However, any ideas why this happens when builds are restricted to the master node?

Comment 32 Markus Knauer 2013-04-25 01:09:42 EDT

(In reply to comment #30)
> The easiest solution is probably to add some cleanup commands to the shell
> section of your build job (something like 'find
> /opt/users/hudsonbuild/.hudson/jobs/buckminster-mdt-ocl-tools-3.2-master
> -name ".nfs*" -exec rm {} \;' )

This workaround didn't work for us (RAP Runtime). If the files are locked for the Hudson build job process, it is locked for every other process and executing a native "rm" doesn't help either.