Windows slaves continually losing connection

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Windows slaves continually losing connection

Daniel Becroft
Hi,

Jenkins 2.179
Windows Server 2008 R2 controller,  Windows 2016 agent

I think I've raised this issue before, but I don't believe I got it resolved (and I can't locate the original post).

We've got a long running job (approx 18hours), and every second day, almost without fail, the jeniins agent process is getting disconnected. The messages from the jenkins-slave.err.log are below - the time between the INFO: Connected and INFO: Terminated lines seem to be consistently about 48 hours aparet (give/take a few hours).

\jenkins-slave.err.log

Jul 31, 2019 2:33:37 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Aug 01, 2019 6:53:39 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
Aug 01, 2019 6:53:49 AM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: jenkins/slaves/restarter/JnlpSlaveRestarterInstaller
java.lang.NoClassDefFoundError: jenkins/slaves/restarter/JnlpSlaveRestarterInstaller
at jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$FindEffectiveRestarters$1.onReconnect(JnlpSlaveRestarterInstaller.java:93)
at hudson.remoting.EngineListenerSplitter.onReconnect(EngineListenerSplitter.java:49)
at hudson.remoting.Engine.innerRun(Engine.java:647)
at hudson.remoting.Engine.run(Engine.java:455)
Caused by: java.lang.ClassNotFoundException: jenkins.slaves.restarter.JnlpSlaveRestarterInstaller
at java.net.URLClassLoader.findClass(Unknown Source)
at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:157)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 4 more
[...]
Aug 01, 2019 8:39:26 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Aug 02, 2019 6:34:30 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
Aug 02, 2019 6:34:41 AM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: jenkins/slaves/restarter/JnlpSlaveRestarterInstaller
java.lang.NoClassDefFoundError: jenkins/slaves/restarter/JnlpSlaveRestarterInstaller
at jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$FindEffectiveRestarters$1.onReconnect(JnlpSlaveRestarterInstaller.java:93)
at hudson.remoting.EngineListenerSplitter.onReconnect(EngineListenerSplitter.java:49)
at hudson.remoting.Engine.innerRun(Engine.java:647)
at hudson.remoting.Engine.run(Engine.java:455)
Caused by: java.lang.ClassNotFoundException: jenkins.slaves.restarter.JnlpSlaveRestarterInstaller
at java.net.URLClassLoader.findClass(Unknown Source)
at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:157)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 4 more
[...]
Aug 02, 2019 4:38:01 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Aug 04, 2019 6:48:05 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
Aug 04, 2019 6:48:15 AM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: jenkins/slaves/restarter/JnlpSlaveRestarterInstaller
java.lang.NoClassDefFoundError: jenkins/slaves/restarter/JnlpSlaveRestarterInstaller
at jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$FindEffectiveRestarters$1.onReconnect(JnlpSlaveRestarterInstaller.java:93)
at hudson.remoting.EngineListenerSplitter.onReconnect(EngineListenerSplitter.java:49)
at hudson.remoting.Engine.innerRun(Engine.java:647)
at hudson.remoting.Engine.run(Engine.java:455)
Caused by: java.lang.ClassNotFoundException: jenkins.slaves.restarter.JnlpSlaveRestarterInstaller
at java.net.URLClassLoader.findClass(Unknown Source)
at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:157)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 4 more

Is there anything I should investigate as to why these agents are continually failing? Can I increase the verbosity of the logging to get more information, or tweak some timeout settings?

What I can't figure out is why the error on startup again is "NoClassDefFoundError", as that indicates a classpath problem, right?

I've got a Windows 2008 server agent running a similar job, and it has no similar issues.

Cheers,
Daniel .

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/5617c5aa-0346-4ec6-b717-137e16083cef%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Windows slaves continually losing connection

Ivan Fernandez Calvo
Hi,

You updated to the latest weekly Jenkins version, Did you updated the jar agent too? if you installed the agent service on your agents a long time ago probably it does not update the jar file on the agent so you are running an old version of the remoting jar file, this file changed with every version of Jenkins, it uses to be compatible but some times is not, it is recommended to update this jar file with each Jenkins update. I'd recommend to uninstall the service, remove all files in the agent folder,  and install it again, in that way you will update the service to the latest version and the remoting jar file too, see https://github.com/kohsuke/winsw for more details

El lunes, 5 de agosto de 2019, 13:46:22 (UTC+2), Daniel Becroft escribió:
Hi,

Jenkins 2.179
Windows Server 2008 R2 controller,  Windows 2016 agent

I think I've raised this issue before, but I don't believe I got it resolved (and I can't locate the original post).

We've got a long running job (approx 18hours), and every second day, almost without fail, the jeniins agent process is getting disconnected. The messages from the jenkins-slave.err.log are below - the time between the INFO: Connected and INFO: Terminated lines seem to be consistently about 48 hours aparet (give/take a few hours).

\jenkins-slave.err.log

Jul 31, 2019 2:33:37 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Aug 01, 2019 6:53:39 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
Aug 01, 2019 6:53:49 AM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: jenkins/slaves/restarter/JnlpSlaveRestarterInstaller
java.lang.NoClassDefFoundError: jenkins/slaves/restarter/JnlpSlaveRestarterInstaller
at jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$FindEffectiveRestarters$1.onReconnect(JnlpSlaveRestarterInstaller.java:93)
at hudson.remoting.EngineListenerSplitter.onReconnect(EngineListenerSplitter.java:49)
at hudson.remoting.Engine.innerRun(Engine.java:647)
at hudson.remoting.Engine.run(Engine.java:455)
Caused by: java.lang.ClassNotFoundException: jenkins.slaves.restarter.JnlpSlaveRestarterInstaller
at java.net.URLClassLoader.findClass(Unknown Source)
at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:157)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 4 more
[...]
Aug 01, 2019 8:39:26 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Aug 02, 2019 6:34:30 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
Aug 02, 2019 6:34:41 AM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: jenkins/slaves/restarter/JnlpSlaveRestarterInstaller
java.lang.NoClassDefFoundError: jenkins/slaves/restarter/JnlpSlaveRestarterInstaller
at jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$FindEffectiveRestarters$1.onReconnect(JnlpSlaveRestarterInstaller.java:93)
at hudson.remoting.EngineListenerSplitter.onReconnect(EngineListenerSplitter.java:49)
at hudson.remoting.Engine.innerRun(Engine.java:647)
at hudson.remoting.Engine.run(Engine.java:455)
Caused by: java.lang.ClassNotFoundException: jenkins.slaves.restarter.JnlpSlaveRestarterInstaller
at java.net.URLClassLoader.findClass(Unknown Source)
at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:157)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 4 more
[...]
Aug 02, 2019 4:38:01 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Aug 04, 2019 6:48:05 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
Aug 04, 2019 6:48:15 AM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: jenkins/slaves/restarter/JnlpSlaveRestarterInstaller
java.lang.NoClassDefFoundError: jenkins/slaves/restarter/JnlpSlaveRestarterInstaller
at jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$FindEffectiveRestarters$1.onReconnect(JnlpSlaveRestarterInstaller.java:93)
at hudson.remoting.EngineListenerSplitter.onReconnect(EngineListenerSplitter.java:49)
at hudson.remoting.Engine.innerRun(Engine.java:647)
at hudson.remoting.Engine.run(Engine.java:455)
Caused by: java.lang.ClassNotFoundException: jenkins.slaves.restarter.JnlpSlaveRestarterInstaller
at java.net.URLClassLoader.findClass(Unknown Source)
at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:157)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 4 more

Is there anything I should investigate as to why these agents are continually failing? Can I increase the verbosity of the logging to get more information, or tweak some timeout settings?

What I can't figure out is why the error on startup again is "NoClassDefFoundError", as that indicates a classpath problem, right?

I've got a Windows 2008 server agent running a similar job, and it has no similar issues.

Cheers,
Daniel .

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/915db2e8-7ed3-4beb-b978-f06fada7753e%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Windows slaves continually losing connection

Daniel Becroft
Thanks Ivan. I checked the .exe version, and it was already the latest. The slave.jar wasn't being updated though, so I've enabled automatic updates on that, and upgraded everything to the latest weekly (2.187).

I'll keep an eye on it, and hopefully that's solved the problem.
---
Daniel Becroft


On Tue, Aug 6, 2019 at 3:32 AM Ivan Fernandez Calvo <[hidden email]> wrote:
Hi,

You updated to the latest weekly Jenkins version, Did you updated the jar agent too? if you installed the agent service on your agents a long time ago probably it does not update the jar file on the agent so you are running an old version of the remoting jar file, this file changed with every version of Jenkins, it uses to be compatible but some times is not, it is recommended to update this jar file with each Jenkins update. I'd recommend to uninstall the service, remove all files in the agent folder,  and install it again, in that way you will update the service to the latest version and the remoting jar file too, see https://github.com/kohsuke/winsw for more details

El lunes, 5 de agosto de 2019, 13:46:22 (UTC+2), Daniel Becroft escribió:
Hi,

Jenkins 2.179
Windows Server 2008 R2 controller,  Windows 2016 agent

I think I've raised this issue before, but I don't believe I got it resolved (and I can't locate the original post).

We've got a long running job (approx 18hours), and every second day, almost without fail, the jeniins agent process is getting disconnected. The messages from the jenkins-slave.err.log are below - the time between the INFO: Connected and INFO: Terminated lines seem to be consistently about 48 hours aparet (give/take a few hours).

\jenkins-slave.err.log

Jul 31, 2019 2:33:37 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Aug 01, 2019 6:53:39 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
Aug 01, 2019 6:53:49 AM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: jenkins/slaves/restarter/JnlpSlaveRestarterInstaller
java.lang.NoClassDefFoundError: jenkins/slaves/restarter/JnlpSlaveRestarterInstaller
at jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$FindEffectiveRestarters$1.onReconnect(JnlpSlaveRestarterInstaller.java:93)
at hudson.remoting.EngineListenerSplitter.onReconnect(EngineListenerSplitter.java:49)
at hudson.remoting.Engine.innerRun(Engine.java:647)
at hudson.remoting.Engine.run(Engine.java:455)
Caused by: java.lang.ClassNotFoundException: jenkins.slaves.restarter.JnlpSlaveRestarterInstaller
at java.net.URLClassLoader.findClass(Unknown Source)
at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:157)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 4 more
[...]
Aug 01, 2019 8:39:26 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Aug 02, 2019 6:34:30 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
Aug 02, 2019 6:34:41 AM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: jenkins/slaves/restarter/JnlpSlaveRestarterInstaller
java.lang.NoClassDefFoundError: jenkins/slaves/restarter/JnlpSlaveRestarterInstaller
at jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$FindEffectiveRestarters$1.onReconnect(JnlpSlaveRestarterInstaller.java:93)
at hudson.remoting.EngineListenerSplitter.onReconnect(EngineListenerSplitter.java:49)
at hudson.remoting.Engine.innerRun(Engine.java:647)
at hudson.remoting.Engine.run(Engine.java:455)
Caused by: java.lang.ClassNotFoundException: jenkins.slaves.restarter.JnlpSlaveRestarterInstaller
at java.net.URLClassLoader.findClass(Unknown Source)
at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:157)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 4 more
[...]
Aug 02, 2019 4:38:01 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Aug 04, 2019 6:48:05 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
Aug 04, 2019 6:48:15 AM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: jenkins/slaves/restarter/JnlpSlaveRestarterInstaller
java.lang.NoClassDefFoundError: jenkins/slaves/restarter/JnlpSlaveRestarterInstaller
at jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$FindEffectiveRestarters$1.onReconnect(JnlpSlaveRestarterInstaller.java:93)
at hudson.remoting.EngineListenerSplitter.onReconnect(EngineListenerSplitter.java:49)
at hudson.remoting.Engine.innerRun(Engine.java:647)
at hudson.remoting.Engine.run(Engine.java:455)
Caused by: java.lang.ClassNotFoundException: jenkins.slaves.restarter.JnlpSlaveRestarterInstaller
at java.net.URLClassLoader.findClass(Unknown Source)
at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:157)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 4 more

Is there anything I should investigate as to why these agents are continually failing? Can I increase the verbosity of the logging to get more information, or tweak some timeout settings?

What I can't figure out is why the error on startup again is "NoClassDefFoundError", as that indicates a classpath problem, right?

I've got a Windows 2008 server agent running a similar job, and it has no similar issues.

Cheers,
Daniel .

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/915db2e8-7ed3-4beb-b978-f06fada7753e%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAJM0iSQ4M6s3XqPtqBB2LrERVeec472bBkJwaowyFg7TvtXSMg%40mail.gmail.com.