Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 390955

Summary: Unable to wipe out workspace
Product: Community Reporter: Eike Stepper <stepper>
Component: CI-JenkinsAssignee: CI Admin Inbox <ci.admin-inbox>
Status: RESOLVED FIXED QA Contact:
Severity: blocker    
Priority: P3 CC: david_williams, Ed.Merks, henshin.ck, mknauer, webmaster
Version: unspecified   
Target Milestone: ---   
Hardware: PC   
OS: Windows 7   
Whiteboard:

Description Eike Stepper CLA 2012-10-02 14:04:05 EDT
(Similar to bug 363222)

I can't wipe my Job workspace completely and now it's in a state where I can not build anymore because of a partial Git clone in it.

https://hudson.eclipse.org/hudson/job/emf-cdo-integration/doWipeOutWorkspace

The output is:

Unable to wipe out workspace.

java.io.IOException: Unable to delete /opt/users/hudsonbuild/.hudson/jobs/emf-cdo-integration/workspace/git/.git/objects/pack/.nfs0000000000e0845300000050
    at hudson.Util.deleteFile(Util.java:263)
    at hudson.Util.deleteRecursive(Util.java:305)
    at hudson.Util.deleteContentsRecursive(Util.java:224)
    at hudson.Util.deleteRecursive(Util.java:304)
    at hudson.Util.deleteContentsRecursive(Util.java:224)
    at hudson.Util.deleteRecursive(Util.java:304)
    at hudson.Util.deleteContentsRecursive(Util.java:224)
    at hudson.Util.deleteRecursive(Util.java:304)
    at hudson.Util.deleteContentsRecursive(Util.java:224)
    at hudson.Util.deleteRecursive(Util.java:304)
    at hudson.Util.deleteContentsRecursive(Util.java:224)
    at hudson.Util.deleteRecursive(Util.java:304)
    at hudson.FilePath$9.invoke(FilePath.java:826)
    at hudson.FilePath$9.invoke(FilePath.java:824)
    at hudson.FilePath.act(FilePath.java:758)
    at hudson.FilePath.act(FilePath.java:740)
    at hudson.FilePath.deleteRecursive(FilePath.java:824)
    at hudson.model.AbstractProject.cleanWorkspace(AbstractProject.java:2065)
    at hudson.model.AbstractProject.doDoWipeOutWorkspace(AbstractProject.java:2046)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:282)
    at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:149)
    at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:88)
    at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:103)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:573)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658)
    at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:233)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:573)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:489)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:160)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1360)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:94)
    at org.hudsonci.servlets.internal.ServletRegistrationFilterAdapter.doFilter(ServletRegistrationFilterAdapter.java:180)
    at org.hudsonci.servlets.internal.ServletRegistrationFilterAdapter.doFilter(ServletRegistrationFilterAdapter.java:148)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:97)
    at org.hudsonci.servlets.internal.ServletRegistrationFilterAdapter.doFilter(ServletRegistrationFilterAdapter.java:180)
    at org.hudsonci.servlets.internal.ServletRegistrationFilterAdapter.doFilter(ServletRegistrationFilterAdapter.java:148)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:97)
    at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:64)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:97)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:86)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:477)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:499)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
    at org.eclipse.jetty.server.Server.handle(Server.java:349)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:910)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:634)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
    at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:191)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
    at java.lang.Thread.run(Thread.java:619)
Comment 1 David Williams CLA 2012-10-02 14:11:21 EDT
Sometimes, when I've gotten this, I was (still) able to clear the workspace using Hudson's web app directly. (and then next runs would be fine). I've only seen it a couple of times.
Comment 2 David Williams CLA 2012-10-02 14:12:51 EDT
at first I thought you were talking about the "delete git repo from windows" problem, which is much worse :) [see bug 377670, if you ever need to know]
Comment 3 Eike Stepper CLA 2012-10-02 14:13:33 EDT
(In reply to comment #1)
> to clear the workspace using Hudson's web app directly.

Hmm, not sure what you mean by this. I tried it with the Hudson web UI.
Comment 4 Eike Stepper CLA 2012-10-02 14:14:20 EDT
(In reply to comment #2)

Ah ;-)

That's not related to my problem, which is probably a pure Hudson issue.
Comment 5 Eclipse Webmaster CLA 2012-10-02 14:15:50 EDT
I've removed the .nfs lockfile.  If you wait (long enough) the file should expire by itself.

-M.
Comment 6 Eike Stepper CLA 2012-10-02 14:18:21 EDT
Thanks!
Comment 7 Eike Stepper CLA 2012-10-15 02:00:11 EDT
This is annoying. It happened again. And it always seems to happen when there's no webmasters around. How long do I have to wait until I can build again?
Comment 8 Markus Knauer CLA 2012-10-15 02:25:41 EDT
(In reply to comment #7)
> How long do I have to wait until I can build again?

Yepp, that would be interesting to know... right now it seems to be caused by this NFS lock file which is alreayd over 22 hours old:

> Unable to delete 
> /opt/users/hudsonbuild/.hudson/jobs/emf-cdo-integration/workspace/git/.git/objects/pack/.nfs0000000000dca07e000000ca

> -rw-r--r-- 1 hudsonBuild callisto-dev 55547 2012-10-14 04:01 
> /shared/jobs/emf-cdo-integration/workspace/git/.git/objects/pack/.nfs0000000000dca07e000000ca

If the timeout is 24 hours you may have luck in less than 2 hours... ;-)
Comment 9 Eike Stepper CLA 2012-10-15 09:12:45 EDT
Still not possible...
Comment 10 Denis Roy CLA 2012-10-15 10:01:17 EDT
> This is annoying. 

Don't build on the master.
Comment 11 Eclipse Webmaster CLA 2012-10-15 10:16:55 EDT
I've cleared the .nfs file.

-M.
Comment 12 Eike Stepper CLA 2012-10-15 10:29:02 EDT
Thank you!
Comment 13 Eike Stepper CLA 2012-10-15 10:29:38 EDT
(In reply to comment #10)
> Don't build on the master.

How's that related? What is the master for?
Comment 14 Denis Roy CLA 2012-10-15 10:37:52 EDT
(In reply to comment #13)
> How's that related? What is the master for?

The master is there to delegate jobs to the slaves -- that's why it only has one executor thread.  It is also the only node one to have its job storage on NFS so that we can back it up.
Comment 15 Eike Stepper CLA 2012-10-17 01:05:32 EDT
Okay, thanks for your hint, too. I remember problems with our promotion service when promoting builds that don't come from the master node but I tried again now and it works perfectly. I've permanently switched to the slaves 1 and 2 now ;-)
Comment 16 Christian Krause CLA 2014-07-10 02:46:55 EDT
Hi,
I've got the same problem:

https://hudson.eclipse.org/hudson/job/cbi_henshin_nightly

I waited about 12 hours but I still cannot wipe the WS. Could you wipe it for me?

Cheers,
Christian
Comment 17 Eclipse Webmaster CLA 2014-07-10 10:07:25 EDT
Hudson has been restarted and I've wiped out the workspace,

-M.