How to overcome node’s networking problem

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

How to overcome node’s networking problem

Gal Netanel

Hi,

I am using Jenkins to run jobs on nodes that last for long period of time (even weeks).

Sometimes, the node’ network interface is going down and immediately coming up again, as a result, the Jenkins master console show the below exception.

Does anyone have an idea how can I overcome such a problem ? can Jenkins be configured to handle short period of time that node does not have a link ?

Thanks, Gal.

P.S:
the execption being thrown that appear on the master:

FATAL: Unable to delete script file C:\WINDOWS\TEMP\hudson17637.bat
hudson.util.IOException2: remote file operation failed: C:\WINDOWS\TEMP\hudson17637.bat at hudson.remoting.Channel@39b30dfc:UTE_Agent_01_149.49.78.11
         at hudson.FilePath.act(FilePath.java:838)
         at hudson.FilePath.act(FilePath.java:824)
         at hudson.FilePath.delete(FilePath.java:1129)
         at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92)
         at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
         at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
         at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:717)
         at hudson.model.Build$BuildExecution.build(Build.java:199)
         at hudson.model.Build$BuildExecution.doRun(Build.java:160)
         at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
         at hudson.model.Run.execute(Run.java:1488)
         at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
         at hudson.model.ResourceController.execute(ResourceController.java:88)
         at hudson.model.Executor.run(Executor.java:236)
Caused by: hudson.remoting.ChannelClosedException: channel is already closed
         at hudson.remoting.Channel.send(Channel.java:492)
         at hudson.remoting.Request.call(Request.java:129)
         at hudson.remoting.Channel.call(Channel.java:663)
         at hudson.FilePath.act(FilePath.java:831)
         ... 13 more
Caused by: java.net.SocketException: Connection reset
         at java.net.SocketInputStream.read(SocketInputStream.java:185)
         at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
         at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
         at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2265)
         at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2558)
         at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2568)
         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314)
         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)
         at hudson.remoting.Command.readFrom(Command.java:90)
         at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)
         at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
FATAL: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
         at hudson.remoting.Request.call(Request.java:174)
         at hudson.remoting.Channel.call(Channel.java:663)
         at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:158)
         at $Proxy42.join(Unknown Source)
         at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:861)
         at hudson.Launcher$ProcStarter.join(Launcher.java:345)
         at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:82)
         at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
         at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
         at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:717)
         at hudson.model.Build$BuildExecution.build(Build.java:199)
         at hudson.model.Build$BuildExecution.doRun(Build.java:160)
         at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
         at hudson.model.Run.execute(Run.java:1488)
         at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
         at hudson.model.ResourceController.execute(ResourceController.java:88)
         at hudson.model.Executor.run(Executor.java:236)
Caused by: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
         at hudson.remoting.Request.abort(Request.java:299)
         at hudson.remoting.Channel.terminate(Channel.java:719)
         at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
Caused by: java.net.SocketException: Connection reset
         at java.net.SocketInputStream.read(SocketInputStream.java:185)
         at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
         at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
         at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2265)
         at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2558)
         at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2568)
         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314)
         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)
         at hudson.remoting.Command.readFrom(Command.java:90)
         at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)
         at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

 

FATAL: Unable to delete script file C:\WINDOWS\TEMP\hudson17637.bat

hudson.util.IOException2: remote file operation failed: C:\WINDOWS\TEMP\hudson17637.bat at hudson.remoting.Channel@39b30dfc:UTE_Agent_01_149.49.78.11

         at hudson.FilePath.act(FilePath.java:838)

         at hudson.FilePath.act(FilePath.java:824)

         at hudson.FilePath.delete(FilePath.java:1129)

         at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92)

         at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)

         at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)

         at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:717)

         at hudson.model.Build$BuildExecution.build(Build.java:199)

         at hudson.model.Build$BuildExecution.doRun(Build.java:160)

         at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)

         at hudson.model.Run.execute(Run.java:1488)

         at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)

         at hudson.model.ResourceController.execute(ResourceController.java:88)

         at hudson.model.Executor.run(Executor.java:236)

Caused by: hudson.remoting.ChannelClosedException: channel is already closed

         at hudson.remoting.Channel.send(Channel.java:492)

         at hudson.remoting.Request.call(Request.java:129)

         at hudson.remoting.Channel.call(Channel.java:663)

         at hudson.FilePath.act(FilePath.java:831)

         ... 13 more

Caused by: java.net.SocketException: Connection reset

         at java.net.SocketInputStream.read(SocketInputStream.java:185)

         at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)

         at java.io.BufferedInputStream.read(BufferedInputStream.java:254)

         at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2265)

         at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2558)

         at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2568)

         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314)

         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)

         at hudson.remoting.Command.readFrom(Command.java:90)

         at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)

         at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

FATAL: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset

hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset

         at hudson.remoting.Request.call(Request.java:174)

         at hudson.remoting.Channel.call(Channel.java:663)

         at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:158)

         at $Proxy42.join(Unknown Source)

         at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:861)

         at hudson.Launcher$ProcStarter.join(Launcher.java:345)

         at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:82)

         at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)

         at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)

         at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:717)

         at hudson.model.Build$BuildExecution.build(Build.java:199)

         at hudson.model.Build$BuildExecution.doRun(Build.java:160)

         at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)

         at hudson.model.Run.execute(Run.java:1488)

         at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)

         at hudson.model.ResourceController.execute(ResourceController.java:88)

         at hudson.model.Executor.run(Executor.java:236)

Caused by: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset

         at hudson.remoting.Request.abort(Request.java:299)

         at hudson.remoting.Channel.terminate(Channel.java:719)

         at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)

Caused by: java.net.SocketException: Connection reset

         at java.net.SocketInputStream.read(SocketInputStream.java:185)

         at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)

         at java.io.BufferedInputStream.read(BufferedInputStream.java:254)

         at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2265)

         at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2558)

         at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2568)

         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314)

         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)

         at hudson.remoting.Command.readFrom(Command.java:90)

         at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)

         at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)