Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 328334

Summary:	Hudson slave 1 SSH failures
Product:	Community	Reporter:	Glyn Normington <glyn.normington>
Component:	CI-Jenkins	Assignee:	Eclipse Webmaster <webmaster>
Status:	CLOSED FIXED	QA Contact:
Severity:	normal
Priority:	P3	CC:	zteve.powell
Version:	unspecified
Target Milestone:	---
Hardware:	PC
OS:	Linux
Whiteboard:

Description Glyn Normington

2010-10-21 06:27:24 EDT

Various Virgo jobs are failing very early on slave 1.

For example, see https://hudson.eclipse.org/hudson/view/Virgo/job/virgo.util.snapshot/. Job 132 failed on slave 1, whereas job 133 passed on slave 2.

The failure shows the following console log:

Started by an SCM change
Building remotely on hudson-slave1
hudson.util.IOException2: remote file operation failed: /opt/users/hudsonbuild/workspace/virgo.util.snapshot at hudson.remoting.Channel@782931c7:hudson-slave1
	at hudson.FilePath.act(FilePath.java:749)
	at hudson.FilePath.act(FilePath.java:735)
	at hudson.FilePath.mkdirs(FilePath.java:801)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1059)
	at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
	at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
	at hudson.model.Run.run(Run.java:1273)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:129)
Caused by: java.io.IOException: SSH channel is closed. (Close requested by remote)
	at com.trilead.ssh2.channel.ChannelManager.sendData(ChannelManager.java:383)
	at com.trilead.ssh2.channel.ChannelOutputStream.write(ChannelOutputStream.java:63)
	at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1838)
	at java.io.ObjectOutputStream$BlockDataOutputStream.writeByte(ObjectOutputStream.java:1876)
	at java.io.ObjectOutputStream.writeFatalException(ObjectOutputStream.java:1537)
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329)
	at hudson.remoting.Channel.send(Channel.java:419)
	at hudson.remoting.Request.call(Request.java:105)
	at hudson.remoting.Channel.call(Channel.java:557)
	at hudson.FilePath.act(FilePath.java:742)
	... 9 more
Archiving artifacts
ERROR: Failed to archive artifacts: build-util/target/
java.io.IOException: SSH channel is closed. (Close requested by remote)
	at com.trilead.ssh2.channel.ChannelManager.sendData(ChannelManager.java:383)
	at com.trilead.ssh2.channel.ChannelOutputStream.write(ChannelOutputStream.java:63)
	at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1838)
	at java.io.ObjectOutputStream$BlockDataOutputStream.writeByte(ObjectOutputStream.java:1876)
	at java.io.ObjectOutputStream.writeFatalException(ObjectOutputStream.java:1537)
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329)
	at hudson.remoting.Channel.send(Channel.java:419)
	at hudson.remoting.Request.call(Request.java:105)
	at hudson.remoting.Channel.call(Channel.java:557)
	at hudson.EnvVars.getRemote(EnvVars.java:202)
	at hudson.model.Computer.getEnvironment(Computer.java:745)
	at hudson.model.Run.getEnvironment(Run.java:1673)
	at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:682)
	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:116)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
	at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:601)
	at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:580)
	at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:558)
	at hudson.model.Build$RunnerImpl.post2(Build.java:158)
	at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:528)
	at hudson.model.Run.run(Run.java:1296)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:129)
Recording test results
ERROR: Publisher hudson.tasks.junit.JUnitResultArchiver aborted due to exception
java.io.IOException: SSH channel is closed. (Close requested by remote)
	at com.trilead.ssh2.channel.ChannelManager.sendData(ChannelManager.java:383)
	at com.trilead.ssh2.channel.ChannelOutputStream.write(ChannelOutputStream.java:63)
	at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1838)
	at java.io.ObjectOutputStream$BlockDataOutputStream.writeByte(ObjectOutputStream.java:1876)
	at java.io.ObjectOutputStream.writeFatalException(ObjectOutputStream.java:1537)
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329)
	at hudson.remoting.Channel.send(Channel.java:419)
	at hudson.remoting.Request.call(Request.java:105)
	at hudson.remoting.Channel.call(Channel.java:557)
	at hudson.EnvVars.getRemote(EnvVars.java:202)
	at hudson.model.Computer.getEnvironment(Computer.java:745)
	at hudson.model.Run.getEnvironment(Run.java:1673)
	at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:682)
	at hudson.tasks.junit.JUnitResultArchiver.perform(JUnitResultArchiver.java:130)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
	at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:601)
	at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:580)
	at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:558)
	at hudson.model.Build$RunnerImpl.post2(Build.java:158)
	at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:528)
	at hudson.model.Run.run(Run.java:1296)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:129)
ERROR: Publisher hudson.tasks.Mailer aborted due to exception
java.io.IOException: SSH channel is closed. (Close requested by remote)
	at com.trilead.ssh2.channel.ChannelManager.sendData(ChannelManager.java:383)
	at com.trilead.ssh2.channel.ChannelOutputStream.write(ChannelOutputStream.java:63)
	at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1838)
	at java.io.ObjectOutputStream$BlockDataOutputStream.writeByte(ObjectOutputStream.java:1876)
	at java.io.ObjectOutputStream.writeFatalException(ObjectOutputStream.java:1537)
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329)
	at hudson.remoting.Channel.send(Channel.java:419)
	at hudson.remoting.Request.call(Request.java:105)
	at hudson.remoting.Channel.call(Channel.java:557)
	at hudson.EnvVars.getRemote(EnvVars.java:202)
	at hudson.model.Computer.getEnvironment(Computer.java:745)
	at hudson.model.Run.getEnvironment(Run.java:1673)
	at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:682)
	at hudson.tasks.Mailer.perform(Mailer.java:108)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
	at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:601)
	at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:580)
	at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:558)
	at hudson.model.Build$RunnerImpl.post2(Build.java:158)
	at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:528)
	at hudson.model.Run.run(Run.java:1296)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:129)
Finished: FAILURE

Comment 1 Eclipse Webmaster

2010-10-21 09:44:45 EDT

I've restarted the slave.

-M.

Comment 2 Glyn Normington

2010-10-21 10:01:57 EDT

Thanks Matt.

For future reference, what led you to believe that a restart would help? And can I restart a slave if this recurs or will I have to bother you guys?

Comment 3 Eclipse Webmaster

2010-10-21 10:47:18 EDT

A similar error was reported in another bug(regarding the ssh channel) and our digging indicated the the solution was to disconnect and reconnect the slave.

It's something you'll either need to file a bug or send an email for.

-M.

Comment 4 Glyn Normington

2010-10-21 10:53:02 EDT

Ok, thanks.

Comment 5 Steve Powell

2010-10-21 12:25:19 EDT

This time...