Issue with master-agent communication over ssh

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Issue with master-agent communication over ssh

JR ibkr
Hello Jenkins community, 

I have setup Jenkins agents over SSH and this agent frequently goes offline with Resource temporarily unavailable error. I had to configure my Jenkins agents to communicate over JNLP. Any advise on how to fix this issue?

Jenkins v2.249.1

[12/04/20 15:23:00] [SSH] Checking java version of java
[12/04/20 15:23:01] [SSH] java -version returned 1.8.0_202.
[12/04/20 15:23:01] [SSH] Starting sftp client.
[12/04/20 15:23:03] [SSH] Remote file system root $JENKINS_SSH_DATA does not exist. Will try to create it...
[12/04/20 15:23:03] [SSH] Copying latest remoting.jar...
[12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes.
Expanded the channel window size to 4MB
[12/04/20 15:23:03] [SSH] Starting agent process: cd "$JENKINS_SSH_DATA" && java  -jar remoting.jar -workDir $JENKINS_SSH_DATA -jar-cache $JENKINS_SSH_DATA/remoting/jarCache
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using $JENKINS_SSH_DATA/remoting as a remoting work directory
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to $JENKINS_SSH_DATA/remoting
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 4.5
This is a Unix agent
Evacuated stdout
Agent successfully connected and online
The Agent is connected, disconnect it before to try to connect it again.
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS discoverGitSystemConfig
WARNING: Exception caught during execution of command '[git, config, --system, --edit]' in '$GIT_PATH/bin', return code '128', error message 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory
'
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS$FileStoreAttributes saveToConfig
WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config] failed after 5 retries
Dec 04, 2020 3:33:59 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@493c5a4e
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:313)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89)
    at hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@2511e2d4
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:326)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90)
    at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85)
    at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54)
    at java.io.OutputStream.write(OutputStream.java:75)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
ERROR: Connection terminated
java.io.StreamCorruptedException: invalid stream header: 00025B42
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
    at hudson.remoting.Command.readFrom(Command.java:142)
    at hudson.remoting.Command.readFrom(Command.java:128)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel channel.
java.util.concurrent.TimeoutException: Ping started at 1607120910589 hasn't completed by 1607121150590
    at hudson.remoting.PingThread.ping(PingThread.java:134)
    at hudson.remoting.PingThread.run(PingThread.java:90)

Thanks,
Jigar R

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d11ea78f-af76-44d9-895d-8270ee98872cn%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Issue with master-agent communication over ssh

jeremy mordkoff
This has been discussed before. Did you search this group's archives? You might be better off re-starting a conversation in one of those threads. 

My own experience is that I too saw this issue for many months. It cleared up on its own. In that interim I did upgrade jenkins more than once and I did switch from dualstack ipv4/ipv6 to ipv4 only for all jenkins work as I was getting errors deploying containers when ipv6 was enabled. I have no idea if either of these actually mattered, and I'm sure other things changed that I am assuming are completely unrelated but may not be. 


On Wednesday, December 9, 2020 at 11:53:19 AM UTC-5 [hidden email] wrote:
Hello Jenkins community, 

I have setup Jenkins agents over SSH and this agent frequently goes offline with Resource temporarily unavailable error. I had to configure my Jenkins agents to communicate over JNLP. Any advise on how to fix this issue?

Jenkins v2.249.1

[12/04/20 15:23:00] [SSH] Checking java version of java
[12/04/20 15:23:01] [SSH] java -version returned 1.8.0_202.
[12/04/20 15:23:01] [SSH] Starting sftp client.
[12/04/20 15:23:03] [SSH] Remote file system root $JENKINS_SSH_DATA does not exist. Will try to create it...
[12/04/20 15:23:03] [SSH] Copying latest remoting.jar...
[12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes.
Expanded the channel window size to 4MB
[12/04/20 15:23:03] [SSH] Starting agent process: cd "$JENKINS_SSH_DATA" && java  -jar remoting.jar -workDir $JENKINS_SSH_DATA -jar-cache $JENKINS_SSH_DATA/remoting/jarCache
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using $JENKINS_SSH_DATA/remoting as a remoting work directory
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to $JENKINS_SSH_DATA/remoting
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 4.5
This is a Unix agent
Evacuated stdout
Agent successfully connected and online
The Agent is connected, disconnect it before to try to connect it again.
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS discoverGitSystemConfig
WARNING: Exception caught during execution of command '[git, config, --system, --edit]' in '$GIT_PATH/bin', return code '128', error message 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory
'
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS$FileStoreAttributes saveToConfig
WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config] failed after 5 retries
Dec 04, 2020 3:33:59 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@493c5a4e
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:313)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89)
    at hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@2511e2d4
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:326)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90)
    at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85)
    at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54)
    at java.io.OutputStream.write(OutputStream.java:75)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
ERROR: Connection terminated
java.io.StreamCorruptedException: invalid stream header: 00025B42
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
    at hudson.remoting.Command.readFrom(Command.java:142)
    at hudson.remoting.Command.readFrom(Command.java:128)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel channel.
java.util.concurrent.TimeoutException: Ping started at 1607120910589 hasn't completed by 1607121150590
    at hudson.remoting.PingThread.ping(PingThread.java:134)
    at hudson.remoting.PingThread.run(PingThread.java:90)

Thanks,
Jigar R

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/b1f9aa56-7092-4bdd-95c8-d1e9f0c96ac8n%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Issue with master-agent communication over ssh

Ivan Fernandez Calvo
In reply to this post by JR ibkr
The most common issue is related to disconnections because there is no traffic between the Jenkins instance and the agent, for that, you have to tune the TCP stack of your OS (see https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected), or enable the keepalive option in the SSH protocol this can be configured by setting ClientAliveInterval or TCPKeepAlive on the SSH server (/etc/ssh/sshd_config), also by setting ServerAliveInterval or TCPKeepAlive options for the user connection (/etc/ssh/ssh_config or ~/.ssh/ssh_config)

https://www.freebsd.org/cgi/man.cgi?sshd_config(5)
https://www.freebsd.org/cgi/man.cgi?ssh_config(5)

Also, check you follow the best practices to configure your SSH agents and enable SSH verbose log output in your service (see https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md)


El miércoles, 9 de diciembre de 2020 a las 17:53:19 UTC+1, [hidden email] escribió:
Hello Jenkins community, 

I have setup Jenkins agents over SSH and this agent frequently goes offline with Resource temporarily unavailable error. I had to configure my Jenkins agents to communicate over JNLP. Any advise on how to fix this issue?

Jenkins v2.249.1

[12/04/20 15:23:00] [SSH] Checking java version of java
[12/04/20 15:23:01] [SSH] java -version returned 1.8.0_202.
[12/04/20 15:23:01] [SSH] Starting sftp client.
[12/04/20 15:23:03] [SSH] Remote file system root $JENKINS_SSH_DATA does not exist. Will try to create it...
[12/04/20 15:23:03] [SSH] Copying latest remoting.jar...
[12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes.
Expanded the channel window size to 4MB
[12/04/20 15:23:03] [SSH] Starting agent process: cd "$JENKINS_SSH_DATA" && java  -jar remoting.jar -workDir $JENKINS_SSH_DATA -jar-cache $JENKINS_SSH_DATA/remoting/jarCache
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using $JENKINS_SSH_DATA/remoting as a remoting work directory
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to $JENKINS_SSH_DATA/remoting
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 4.5
This is a Unix agent
Evacuated stdout
Agent successfully connected and online
The Agent is connected, disconnect it before to try to connect it again.
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS discoverGitSystemConfig
WARNING: Exception caught during execution of command '[git, config, --system, --edit]' in '$GIT_PATH/bin', return code '128', error message 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory
'
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS$FileStoreAttributes saveToConfig
WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config] failed after 5 retries
Dec 04, 2020 3:33:59 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@493c5a4e
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:313)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89)
    at hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@2511e2d4
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:326)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90)
    at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85)
    at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54)
    at java.io.OutputStream.write(OutputStream.java:75)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
ERROR: Connection terminated
java.io.StreamCorruptedException: invalid stream header: 00025B42
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
    at hudson.remoting.Command.readFrom(Command.java:142)
    at hudson.remoting.Command.readFrom(Command.java:128)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel channel.
java.util.concurrent.TimeoutException: Ping started at 1607120910589 hasn't completed by 1607121150590
    at hudson.remoting.PingThread.ping(PingThread.java:134)
    at hudson.remoting.PingThread.run(PingThread.java:90)

Thanks,
Jigar R

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/0cca6599-cdb2-4dc2-a7df-d756e4e793b3n%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Issue with master-agent communication over ssh

JR ibkr
Interestingly, whenever Jenkins agent is executing PMD/Findbugs task, it runs into " Resource temporarily unavailable". 
I have
  • verified TCP parameters i.e. tcp_keepalive_time, tcp_keepalive_intvl, tcp_keepalive_probes & tcp_fin_timeout ; All of them have higher values then what cloudbees recommended at https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected
  • set heap space to 2-4Gb
  • disabled ping thread from Jenkins agent by setting "-Dhudson.remoting.Launcher.pingIntervalSec=-1"
Please let me know if there is anything else I should consider.
On Friday, December 11, 2020 at 12:11:39 PM UTC-5 [hidden email] wrote:
The most common issue is related to disconnections because there is no traffic between the Jenkins instance and the agent, for that, you have to tune the TCP stack of your OS (see https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected), or enable the keepalive option in the SSH protocol this can be configured by setting ClientAliveInterval or TCPKeepAlive on the SSH server (/etc/ssh/sshd_config), also by setting ServerAliveInterval or TCPKeepAlive options for the user connection (/etc/ssh/ssh_config or ~/.ssh/ssh_config)

https://www.freebsd.org/cgi/man.cgi?sshd_config(5)
https://www.freebsd.org/cgi/man.cgi?ssh_config(5)

Also, check you follow the best practices to configure your SSH agents and enable SSH verbose log output in your service (see https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md)


El miércoles, 9 de diciembre de 2020 a las 17:53:19 UTC+1, [hidden email] escribió:
Hello Jenkins community, 

I have setup Jenkins agents over SSH and this agent frequently goes offline with Resource temporarily unavailable error. I had to configure my Jenkins agents to communicate over JNLP. Any advise on how to fix this issue?

Jenkins v2.249.1

[12/04/20 15:23:00] [SSH] Checking java version of java
[12/04/20 15:23:01] [SSH] java -version returned 1.8.0_202.
[12/04/20 15:23:01] [SSH] Starting sftp client.
[12/04/20 15:23:03] [SSH] Remote file system root $JENKINS_SSH_DATA does not exist. Will try to create it...
[12/04/20 15:23:03] [SSH] Copying latest remoting.jar...
[12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes.
Expanded the channel window size to 4MB
[12/04/20 15:23:03] [SSH] Starting agent process: cd "$JENKINS_SSH_DATA" && java  -jar remoting.jar -workDir $JENKINS_SSH_DATA -jar-cache $JENKINS_SSH_DATA/remoting/jarCache
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using $JENKINS_SSH_DATA/remoting as a remoting work directory
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to $JENKINS_SSH_DATA/remoting
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 4.5
This is a Unix agent
Evacuated stdout
Agent successfully connected and online
The Agent is connected, disconnect it before to try to connect it again.
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS discoverGitSystemConfig
WARNING: Exception caught during execution of command '[git, config, --system, --edit]' in '$GIT_PATH/bin', return code '128', error message 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory
'
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS$FileStoreAttributes saveToConfig
WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config] failed after 5 retries
Dec 04, 2020 3:33:59 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@493c5a4e
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:313)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89)
    at hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@2511e2d4
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:326)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90)
    at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85)
    at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54)
    at java.io.OutputStream.write(OutputStream.java:75)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
ERROR: Connection terminated
java.io.StreamCorruptedException: invalid stream header: 00025B42
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
    at hudson.remoting.Command.readFrom(Command.java:142)
    at hudson.remoting.Command.readFrom(Command.java:128)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel channel.
java.util.concurrent.TimeoutException: Ping started at 1607120910589 hasn't completed by 1607121150590
    at hudson.remoting.PingThread.ping(PingThread.java:134)
    at hudson.remoting.PingThread.run(PingThread.java:90)

Thanks,
Jigar R

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Issue with master-agent communication over ssh

Ivan Fernandez Calvo
If the agent is running something there is traffic, so it si not related to keep alive settings, I wonder if after the issue happens there is a remoting process running on the agent.
Did you check for hs_err_pid error files in the root fs of the agent? http://www.oracle.com/technetwork/java/javase/felog-138657.html#gbwcy
How much memory you set for your remoting process?
How much memory your maven build needs?

El miércoles, 30 de diciembre de 2020 a las 23:40:38 UTC+1, [hidden email] escribió:
Interestingly, whenever Jenkins agent is executing PMD/Findbugs task, it runs into " Resource temporarily unavailable". 
I have
Please let me know if there is anything else I should consider.
On Friday, December 11, 2020 at 12:11:39 PM UTC-5 [hidden email] wrote:
The most common issue is related to disconnections because there is no traffic between the Jenkins instance and the agent, for that, you have to tune the TCP stack of your OS (see https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected), or enable the keepalive option in the SSH protocol this can be configured by setting ClientAliveInterval or TCPKeepAlive on the SSH server (/etc/ssh/sshd_config), also by setting ServerAliveInterval or TCPKeepAlive options for the user connection (/etc/ssh/ssh_config or ~/.ssh/ssh_config)

https://www.freebsd.org/cgi/man.cgi?sshd_config(5)
https://www.freebsd.org/cgi/man.cgi?ssh_config(5)

Also, check you follow the best practices to configure your SSH agents and enable SSH verbose log output in your service (see https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md)


El miércoles, 9 de diciembre de 2020 a las 17:53:19 UTC+1, [hidden email] escribió:
Hello Jenkins community, 

I have setup Jenkins agents over SSH and this agent frequently goes offline with Resource temporarily unavailable error. I had to configure my Jenkins agents to communicate over JNLP. Any advise on how to fix this issue?

Jenkins v2.249.1

[12/04/20 15:23:00] [SSH] Checking java version of java
[12/04/20 15:23:01] [SSH] java -version returned 1.8.0_202.
[12/04/20 15:23:01] [SSH] Starting sftp client.
[12/04/20 15:23:03] [SSH] Remote file system root $JENKINS_SSH_DATA does not exist. Will try to create it...
[12/04/20 15:23:03] [SSH] Copying latest remoting.jar...
[12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes.
Expanded the channel window size to 4MB
[12/04/20 15:23:03] [SSH] Starting agent process: cd "$JENKINS_SSH_DATA" && java  -jar remoting.jar -workDir $JENKINS_SSH_DATA -jar-cache $JENKINS_SSH_DATA/remoting/jarCache
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using $JENKINS_SSH_DATA/remoting as a remoting work directory
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to $JENKINS_SSH_DATA/remoting
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 4.5
This is a Unix agent
Evacuated stdout
Agent successfully connected and online
The Agent is connected, disconnect it before to try to connect it again.
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS discoverGitSystemConfig
WARNING: Exception caught during execution of command '[git, config, --system, --edit]' in '$GIT_PATH/bin', return code '128', error message 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory
'
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS$FileStoreAttributes saveToConfig
WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config] failed after 5 retries
Dec 04, 2020 3:33:59 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@493c5a4e
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:313)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89)
    at hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@2511e2d4
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:326)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90)
    at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85)
    at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54)
    at java.io.OutputStream.write(OutputStream.java:75)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
ERROR: Connection terminated
java.io.StreamCorruptedException: invalid stream header: 00025B42
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
    at hudson.remoting.Command.readFrom(Command.java:142)
    at hudson.remoting.Command.readFrom(Command.java:128)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel channel.
java.util.concurrent.TimeoutException: Ping started at 1607120910589 hasn't completed by 1607121150590
    at hudson.remoting.PingThread.ping(PingThread.java:134)
    at hudson.remoting.PingThread.run(PingThread.java:90)

Thanks,
Jigar R

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/2b23ef01-3890-4fbd-962b-970a5efec3d8n%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Issue with master-agent communication over ssh

Jeff Thompson
In reply to this post by JR ibkr

When I saw something like this in the past, it was because the process was running out of resources. Specifically when running SpotBugs, I got out-of-memory errors. I had to modify the pom to allocate more memory.

I recommend ensuring your build runs normally on the agent without the additional complexities of the Jenkins environment. This might show where additional resources are needed. If that all passes, then continue on to the additional troubleshooting steps involving the Jenkins controller and agent.

Jeff Thompson

On 12/30/20 3:40 PM, [hidden email] wrote:
Interestingly, whenever Jenkins agent is executing PMD/Findbugs task, it runs into " Resource temporarily unavailable". 
I have
Please let me know if there is anything else I should consider.
On Friday, December 11, 2020 at 12:11:39 PM UTC-5 [hidden email] wrote:
The most common issue is related to disconnections because there is no traffic between the Jenkins instance and the agent, for that, you have to tune the TCP stack of your OS (see https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected), or enable the keepalive option in the SSH protocol this can be configured by setting ClientAliveInterval or TCPKeepAlive on the SSH server (/etc/ssh/sshd_config), also by setting ServerAliveInterval or TCPKeepAlive options for the user connection (/etc/ssh/ssh_config or ~/.ssh/ssh_config)

https://www.freebsd.org/cgi/man.cgi?sshd_config(5)
https://www.freebsd.org/cgi/man.cgi?ssh_config(5)

Also, check you follow the best practices to configure your SSH agents and enable SSH verbose log output in your service (see https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md)


El miércoles, 9 de diciembre de 2020 a las 17:53:19 UTC+1, jiga...@... escribió:
Hello Jenkins community, 

I have setup Jenkins agents over SSH and this agent frequently goes offline with Resource temporarily unavailable error. I had to configure my Jenkins agents to communicate over JNLP. Any advise on how to fix this issue?

Jenkins v2.249.1

[12/04/20 15:23:00] [SSH] Checking java version of java
[12/04/20 15:23:01] [SSH] java -version returned 1.8.0_202.
[12/04/20 15:23:01] [SSH] Starting sftp client.
[12/04/20 15:23:03] [SSH] Remote file system root $JENKINS_SSH_DATA does not exist. Will try to create it...
[12/04/20 15:23:03] [SSH] Copying latest remoting.jar...
[12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes.
Expanded the channel window size to 4MB
[12/04/20 15:23:03] [SSH] Starting agent process: cd "$JENKINS_SSH_DATA" && java  -jar remoting.jar -workDir $JENKINS_SSH_DATA -jar-cache $JENKINS_SSH_DATA/remoting/jarCache
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using $JENKINS_SSH_DATA/remoting as a remoting work directory
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to $JENKINS_SSH_DATA/remoting
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 4.5
This is a Unix agent
Evacuated stdout
Agent successfully connected and online
The Agent is connected, disconnect it before to try to connect it again.
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS discoverGitSystemConfig
WARNING: Exception caught during execution of command '[git, config, --system, --edit]' in '$GIT_PATH/bin', return code '128', error message 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory
'
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS$FileStoreAttributes saveToConfig
WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config] failed after 5 retries
Dec 04, 2020 3:33:59 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@493c5a4e
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:313)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89)
    at hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@2511e2d4
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:326)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90)
    at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85)
    at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54)
    at java.io.OutputStream.write(OutputStream.java:75)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
ERROR: Connection terminated
java.io.StreamCorruptedException: invalid stream header: 00025B42
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
    at hudson.remoting.Command.readFrom(Command.java:142)
    at hudson.remoting.Command.readFrom(Command.java:128)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel channel.
java.util.concurrent.TimeoutException: Ping started at 1607120910589 hasn't completed by 1607121150590
    at hudson.remoting.PingThread.ping(PingThread.java:134)
    at hudson.remoting.PingThread.run(PingThread.java:90)

Thanks,
Jigar R

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/88bf8ba5-d077-cc33-964b-1667138c1aa3%40cloudbees.com.
Reply | Threaded
Open this post in threaded view
|

Re: Issue with master-agent communication over ssh

Jigar R


On Monday, January 4, 2021 at 12:37:52 PM UTC-5 [hidden email] wrote:

When I saw something like this in the past, it was because the process was running out of resources. Specifically when running SpotBugs, I got out-of-memory errors. I had to modify the pom to allocate more memory.

I have 2 different kind of jenkins agents.
1. java web start
2. ssh
If the memory was issue then wouldn't it fail on both cases?

I do see "Agent went offline during build Connection was broker: java.io.StreamCorruptedException: invalid stream header:".

I recommend ensuring your build runs normally on the agent without the additional complexities of the Jenkins environment. This might show where additional resources are needed. If that all passes, then continue on to the additional troubleshooting steps involving the Jenkins controller and agent.

Jeff Thompson

On 12/30/20 3:40 PM, jiga...@... wrote:
Interestingly, whenever Jenkins agent is executing PMD/Findbugs task, it runs into " Resource temporarily unavailable". 
I have
Please let me know if there is anything else I should consider.
On Friday, December 11, 2020 at 12:11:39 PM UTC-5 kuisat...@... wrote:
The most common issue is related to disconnections because there is no traffic between the Jenkins instance and the agent, for that, you have to tune the TCP stack of your OS (see https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected), or enable the keepalive option in the SSH protocol this can be configured by setting ClientAliveInterval or TCPKeepAlive on the SSH server (/etc/ssh/sshd_config), also by setting ServerAliveInterval or TCPKeepAlive options for the user connection (/etc/ssh/ssh_config or ~/.ssh/ssh_config)

https://www.freebsd.org/cgi/man.cgi?sshd_config(5)
https://www.freebsd.org/cgi/man.cgi?ssh_config(5)

Also, check you follow the best practices to configure your SSH agents and enable SSH verbose log output in your service (see https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md)


El miércoles, 9 de diciembre de 2020 a las 17:53:19 UTC+1, [hidden email] escribió:
Hello Jenkins community, 

I have setup Jenkins agents over SSH and this agent frequently goes offline with Resource temporarily unavailable error. I had to configure my Jenkins agents to communicate over JNLP. Any advise on how to fix this issue?

Jenkins v2.249.1

[12/04/20 15:23:00] [SSH] Checking java version of java
[12/04/20 15:23:01] [SSH] java -version returned 1.8.0_202.
[12/04/20 15:23:01] [SSH] Starting sftp client.
[12/04/20 15:23:03] [SSH] Remote file system root $JENKINS_SSH_DATA does not exist. Will try to create it...
[12/04/20 15:23:03] [SSH] Copying latest remoting.jar...
[12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes.
Expanded the channel window size to 4MB
[12/04/20 15:23:03] [SSH] Starting agent process: cd "$JENKINS_SSH_DATA" && java  -jar remoting.jar -workDir $JENKINS_SSH_DATA -jar-cache $JENKINS_SSH_DATA/remoting/jarCache
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using $JENKINS_SSH_DATA/remoting as a remoting work directory
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to $JENKINS_SSH_DATA/remoting
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 4.5
This is a Unix agent
Evacuated stdout
Agent successfully connected and online
The Agent is connected, disconnect it before to try to connect it again.
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS discoverGitSystemConfig
WARNING: Exception caught during execution of command '[git, config, --system, --edit]' in '$GIT_PATH/bin', return code '128', error message 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory
'
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS$FileStoreAttributes saveToConfig
WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config] failed after 5 retries
Dec 04, 2020 3:33:59 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@493c5a4e
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:313)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89)
    at hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@2511e2d4
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:326)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90)
    at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85)
    at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54)
    at java.io.OutputStream.write(OutputStream.java:75)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
ERROR: Connection terminated
java.io.StreamCorruptedException: invalid stream header: 00025B42
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
    at hudson.remoting.Command.readFrom(Command.java:142)
    at hudson.remoting.Command.readFrom(Command.java:128)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel channel.
java.util.concurrent.TimeoutException: Ping started at 1607120910589 hasn't completed by 1607121150590
    at hudson.remoting.PingThread.ping(PingThread.java:134)
    at hudson.remoting.PingThread.run(PingThread.java:90)

Thanks,
Jigar R

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/6169de50-f83a-4b4c-b199-59b2e2b65cdbn%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Issue with master-agent communication over ssh

Ivan Fernandez Calvo
>I have 2 different kind of jenkins agents.
>1. java web start
>2. ssh
>If the memory was issue then wouldn't it fail on both cases?

Not necessarily, starting by the point that are different ways to establish the connection, the JNLP agents could not update the remoting jar file (depends on your configurations), so you can be running different versions of remoting. I agree with Jeff looks like an OOM issue, review my comments at https://groups.google.com/g/jenkinsci-users/c/nD3s06hSUXE/m/BQKk5GSYBwAJ my recommendation is to fix the mem for the remoting process to 1024M (-Xmx1024m -Xms1024m) see if the issue disappear or change, if disappear, you would have to adjust the remoting process memory to the right one between 256M-1024M, to use 512M usually is safe a not too much (but depends on your agents' memory we do not know how much they have)

El lunes, 1 de febrero de 2021 a las 23:52:19 UTC+1, [hidden email] escribió:
On Monday, January 4, 2021 at 12:37:52 PM UTC-5 [hidden email] wrote:

When I saw something like this in the past, it was because the process was running out of resources. Specifically when running SpotBugs, I got out-of-memory errors. I had to modify the pom to allocate more memory.

I have 2 different kind of jenkins agents.
1. java web start
2. ssh
If the memory was issue then wouldn't it fail on both cases?

I do see "Agent went offline during build Connection was broker: java.io.StreamCorruptedException: invalid stream header:".

I recommend ensuring your build runs normally on the agent without the additional complexities of the Jenkins environment. This might show where additional resources are needed. If that all passes, then continue on to the additional troubleshooting steps involving the Jenkins controller and agent.

Jeff Thompson

On 12/30/20 3:40 PM, [hidden email] wrote:
Interestingly, whenever Jenkins agent is executing PMD/Findbugs task, it runs into " Resource temporarily unavailable". 
I have
Please let me know if there is anything else I should consider.
On Friday, December 11, 2020 at 12:11:39 PM UTC-5 [hidden email] wrote:
The most common issue is related to disconnections because there is no traffic between the Jenkins instance and the agent, for that, you have to tune the TCP stack of your OS (see https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected), or enable the keepalive option in the SSH protocol this can be configured by setting ClientAliveInterval or TCPKeepAlive on the SSH server (/etc/ssh/sshd_config), also by setting ServerAliveInterval or TCPKeepAlive options for the user connection (/etc/ssh/ssh_config or ~/.ssh/ssh_config)

https://www.freebsd.org/cgi/man.cgi?sshd_config(5)
https://www.freebsd.org/cgi/man.cgi?ssh_config(5)

Also, check you follow the best practices to configure your SSH agents and enable SSH verbose log output in your service (see https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md)


El miércoles, 9 de diciembre de 2020 a las 17:53:19 UTC+1, [hidden email] escribió:
Hello Jenkins community, 

I have setup Jenkins agents over SSH and this agent frequently goes offline with Resource temporarily unavailable error. I had to configure my Jenkins agents to communicate over JNLP. Any advise on how to fix this issue?

Jenkins v2.249.1

[12/04/20 15:23:00] [SSH] Checking java version of java
[12/04/20 15:23:01] [SSH] java -version returned 1.8.0_202.
[12/04/20 15:23:01] [SSH] Starting sftp client.
[12/04/20 15:23:03] [SSH] Remote file system root $JENKINS_SSH_DATA does not exist. Will try to create it...
[12/04/20 15:23:03] [SSH] Copying latest remoting.jar...
[12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes.
Expanded the channel window size to 4MB
[12/04/20 15:23:03] [SSH] Starting agent process: cd "$JENKINS_SSH_DATA" && java  -jar remoting.jar -workDir $JENKINS_SSH_DATA -jar-cache $JENKINS_SSH_DATA/remoting/jarCache
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using $JENKINS_SSH_DATA/remoting as a remoting work directory
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to $JENKINS_SSH_DATA/remoting
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 4.5
This is a Unix agent
Evacuated stdout
Agent successfully connected and online
The Agent is connected, disconnect it before to try to connect it again.
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS discoverGitSystemConfig
WARNING: Exception caught during execution of command '[git, config, --system, --edit]' in '$GIT_PATH/bin', return code '128', error message 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory
'
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS$FileStoreAttributes saveToConfig
WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config] failed after 5 retries
Dec 04, 2020 3:33:59 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@493c5a4e
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:313)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89)
    at hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@2511e2d4
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:326)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90)
    at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85)
    at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54)
    at java.io.OutputStream.write(OutputStream.java:75)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
ERROR: Connection terminated
java.io.StreamCorruptedException: invalid stream header: 00025B42
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
    at hudson.remoting.Command.readFrom(Command.java:142)
    at hudson.remoting.Command.readFrom(Command.java:128)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel channel.
java.util.concurrent.TimeoutException: Ping started at 1607120910589 hasn't completed by 1607121150590
    at hudson.remoting.PingThread.ping(PingThread.java:134)
    at hudson.remoting.PingThread.run(PingThread.java:90)

Thanks,
Jigar R

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/099267e4-36f2-4b1e-adf1-4ed6f6d5bf90n%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Issue with master-agent communication over ssh

Jigar R


On Tuesday, February 2, 2021 at 12:20:19 PM UTC-5 [hidden email] wrote:
>I have 2 different kind of jenkins agents.
>1. java web start
>2. ssh
>If the memory was issue then wouldn't it fail on both cases?

Not necessarily, starting by the point that are different ways to establish the connection, the JNLP agents could not update the remoting jar file (depends on your configurations), so you can be running different versions of remoting. I agree with Jeff looks like an OOM issue, review my comments at https://groups.google.com/g/jenkinsci-users/c/nD3s06hSUXE/m/BQKk5GSYBwAJ my recommendation is to fix the mem for the remoting process to 1024M (-Xmx1024m -Xms1024m) see if the issue disappear or change, if disappear, you would have to adjust the remoting process memory to the right one between 256M-1024M, to use 512M usually is safe a not too much (but depends on your agents' memory we do not know how much they have)

I created SSH agent with -Xmx1024m -Xms1024m.
I do see bunch of Warnings about "Ping failed. Terminating the channel channel".
I got following error:
"
  WARNING: Failed to send back a reply to te rquest hudson.remoting.Request$....
  java.io.IOException: Resource temporarily unavailable
          at java.io.FileOutputStream.writeBytes(Native method)
          .....
"
 

El lunes, 1 de febrero de 2021 a las 23:52:19 UTC+1, jigarra...@... escribió:
On Monday, January 4, 2021 at 12:37:52 PM UTC-5 [hidden email] wrote:

When I saw something like this in the past, it was because the process was running out of resources. Specifically when running SpotBugs, I got out-of-memory errors. I had to modify the pom to allocate more memory.

I have 2 different kind of jenkins agents.
1. java web start
2. ssh
If the memory was issue then wouldn't it fail on both cases?

I do see "Agent went offline during build Connection was broker: java.io.StreamCorruptedException: invalid stream header:".

I recommend ensuring your build runs normally on the agent without the additional complexities of the Jenkins environment. This might show where additional resources are needed. If that all passes, then continue on to the additional troubleshooting steps involving the Jenkins controller and agent.

Jeff Thompson

On 12/30/20 3:40 PM, [hidden email] wrote:
Interestingly, whenever Jenkins agent is executing PMD/Findbugs task, it runs into " Resource temporarily unavailable". 
I have
Please let me know if there is anything else I should consider.
On Friday, December 11, 2020 at 12:11:39 PM UTC-5 [hidden email] wrote:
The most common issue is related to disconnections because there is no traffic between the Jenkins instance and the agent, for that, you have to tune the TCP stack of your OS (see https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected), or enable the keepalive option in the SSH protocol this can be configured by setting ClientAliveInterval or TCPKeepAlive on the SSH server (/etc/ssh/sshd_config), also by setting ServerAliveInterval or TCPKeepAlive options for the user connection (/etc/ssh/ssh_config or ~/.ssh/ssh_config)

https://www.freebsd.org/cgi/man.cgi?sshd_config(5)
https://www.freebsd.org/cgi/man.cgi?ssh_config(5)

Also, check you follow the best practices to configure your SSH agents and enable SSH verbose log output in your service (see https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md)


El miércoles, 9 de diciembre de 2020 a las 17:53:19 UTC+1, [hidden email] escribió:
Hello Jenkins community, 

I have setup Jenkins agents over SSH and this agent frequently goes offline with Resource temporarily unavailable error. I had to configure my Jenkins agents to communicate over JNLP. Any advise on how to fix this issue?

Jenkins v2.249.1

[12/04/20 15:23:00] [SSH] Checking java version of java
[12/04/20 15:23:01] [SSH] java -version returned 1.8.0_202.
[12/04/20 15:23:01] [SSH] Starting sftp client.
[12/04/20 15:23:03] [SSH] Remote file system root $JENKINS_SSH_DATA does not exist. Will try to create it...
[12/04/20 15:23:03] [SSH] Copying latest remoting.jar...
[12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes.
Expanded the channel window size to 4MB
[12/04/20 15:23:03] [SSH] Starting agent process: cd "$JENKINS_SSH_DATA" && java  -jar remoting.jar -workDir $JENKINS_SSH_DATA -jar-cache $JENKINS_SSH_DATA/remoting/jarCache
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using $JENKINS_SSH_DATA/remoting as a remoting work directory
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to $JENKINS_SSH_DATA/remoting
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 4.5
This is a Unix agent
Evacuated stdout
Agent successfully connected and online
The Agent is connected, disconnect it before to try to connect it again.
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS discoverGitSystemConfig
WARNING: Exception caught during execution of command '[git, config, --system, --edit]' in '$GIT_PATH/bin', return code '128', error message 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory
'
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS$FileStoreAttributes saveToConfig
WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config] failed after 5 retries
Dec 04, 2020 3:33:59 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@493c5a4e
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:313)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89)
    at hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@2511e2d4
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:326)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90)
    at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85)
    at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54)
    at java.io.OutputStream.write(OutputStream.java:75)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
ERROR: Connection terminated
java.io.StreamCorruptedException: invalid stream header: 00025B42
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
    at hudson.remoting.Command.readFrom(Command.java:142)
    at hudson.remoting.Command.readFrom(Command.java:128)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel channel.
java.util.concurrent.TimeoutException: Ping started at 1607120910589 hasn't completed by 1607121150590
    at hudson.remoting.PingThread.ping(PingThread.java:134)
    at hudson.remoting.PingThread.run(PingThread.java:90)

Thanks,
Jigar R

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/615a4333-587c-4ea2-a11f-fea45fe38a56n%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Issue with master-agent communication over ssh

Ivan Fernandez Calvo
Weird, Could you share a screencapture of what you configurate? Also the whole exception those lines alone mean nothing. Know the version of Jenkins and the version of the SSH build agents plugins you use can help, the memory you have in your agents, if they are bare metal or cloud. In overall if you want help please provide more context. 

El mar, 2 feb 2021 a las 22:52, Jigar R (<[hidden email]>) escribió:


On Tuesday, February 2, 2021 at 12:20:19 PM UTC-5 [hidden email] wrote:
>I have 2 different kind of jenkins agents.
>1. java web start
>2. ssh
>If the memory was issue then wouldn't it fail on both cases?

Not necessarily, starting by the point that are different ways to establish the connection, the JNLP agents could not update the remoting jar file (depends on your configurations), so you can be running different versions of remoting. I agree with Jeff looks like an OOM issue, review my comments at https://groups.google.com/g/jenkinsci-users/c/nD3s06hSUXE/m/BQKk5GSYBwAJ my recommendation is to fix the mem for the remoting process to 1024M (-Xmx1024m -Xms1024m) see if the issue disappear or change, if disappear, you would have to adjust the remoting process memory to the right one between 256M-1024M, to use 512M usually is safe a not too much (but depends on your agents' memory we do not know how much they have)

I created SSH agent with -Xmx1024m -Xms1024m.
I do see bunch of Warnings about "Ping failed. Terminating the channel channel".
I got following error:
"
  WARNING: Failed to send back a reply to te rquest hudson.remoting.Request$....
  java.io.IOException: Resource temporarily unavailable
          at java.io.FileOutputStream.writeBytes(Native method)
          .....
"
 

El lunes, 1 de febrero de 2021 a las 23:52:19 UTC+1, [hidden email] escribió:
On Monday, January 4, 2021 at 12:37:52 PM UTC-5 [hidden email] wrote:

When I saw something like this in the past, it was because the process was running out of resources. Specifically when running SpotBugs, I got out-of-memory errors. I had to modify the pom to allocate more memory.

I have 2 different kind of jenkins agents.
1. java web start
2. ssh
If the memory was issue then wouldn't it fail on both cases?

I do see "Agent went offline during build Connection was broker: java.io.StreamCorruptedException: invalid stream header:".

I recommend ensuring your build runs normally on the agent without the additional complexities of the Jenkins environment. This might show where additional resources are needed. If that all passes, then continue on to the additional troubleshooting steps involving the Jenkins controller and agent.

Jeff Thompson

On 12/30/20 3:40 PM, [hidden email] wrote:
Interestingly, whenever Jenkins agent is executing PMD/Findbugs task, it runs into " Resource temporarily unavailable". 
I have
Please let me know if there is anything else I should consider.
On Friday, December 11, 2020 at 12:11:39 PM UTC-5 [hidden email] wrote:
The most common issue is related to disconnections because there is no traffic between the Jenkins instance and the agent, for that, you have to tune the TCP stack of your OS (see https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected), or enable the keepalive option in the SSH protocol this can be configured by setting ClientAliveInterval or TCPKeepAlive on the SSH server (/etc/ssh/sshd_config), also by setting ServerAliveInterval or TCPKeepAlive options for the user connection (/etc/ssh/ssh_config or ~/.ssh/ssh_config)

https://www.freebsd.org/cgi/man.cgi?sshd_config(5)
https://www.freebsd.org/cgi/man.cgi?ssh_config(5)

Also, check you follow the best practices to configure your SSH agents and enable SSH verbose log output in your service (see https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md)


El miércoles, 9 de diciembre de 2020 a las 17:53:19 UTC+1, [hidden email] escribió:
Hello Jenkins community, 

I have setup Jenkins agents over SSH and this agent frequently goes offline with Resource temporarily unavailable error. I had to configure my Jenkins agents to communicate over JNLP. Any advise on how to fix this issue?

Jenkins v2.249.1

[12/04/20 15:23:00] [SSH] Checking java version of java
[12/04/20 15:23:01] [SSH] java -version returned 1.8.0_202.
[12/04/20 15:23:01] [SSH] Starting sftp client.
[12/04/20 15:23:03] [SSH] Remote file system root $JENKINS_SSH_DATA does not exist. Will try to create it...
[12/04/20 15:23:03] [SSH] Copying latest remoting.jar...
[12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes.
Expanded the channel window size to 4MB
[12/04/20 15:23:03] [SSH] Starting agent process: cd "$JENKINS_SSH_DATA" && java  -jar remoting.jar -workDir $JENKINS_SSH_DATA -jar-cache $JENKINS_SSH_DATA/remoting/jarCache
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using $JENKINS_SSH_DATA/remoting as a remoting work directory
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to $JENKINS_SSH_DATA/remoting
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 4.5
This is a Unix agent
Evacuated stdout
Agent successfully connected and online
The Agent is connected, disconnect it before to try to connect it again.
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS discoverGitSystemConfig
WARNING: Exception caught during execution of command '[git, config, --system, --edit]' in '$GIT_PATH/bin', return code '128', error message 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory
'
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS$FileStoreAttributes saveToConfig
WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config] failed after 5 retries
Dec 04, 2020 3:33:59 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@493c5a4e
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:313)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89)
    at hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@2511e2d4
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:326)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90)
    at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85)
    at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54)
    at java.io.OutputStream.write(OutputStream.java:75)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
ERROR: Connection terminated
java.io.StreamCorruptedException: invalid stream header: 00025B42
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
    at hudson.remoting.Command.readFrom(Command.java:142)
    at hudson.remoting.Command.readFrom(Command.java:128)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel channel.
java.util.concurrent.TimeoutException: Ping started at 1607120910589 hasn't completed by 1607121150590
    at hudson.remoting.PingThread.ping(PingThread.java:134)
    at hudson.remoting.PingThread.run(PingThread.java:90)

Thanks,
Jigar R

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/615a4333-587c-4ea2-a11f-fea45fe38a56n%40googlegroups.com.


--

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAKo5QrpE%2Bxys4CwNGe6Mx7NRozmeFxaJO0uZ77Rz3NUK057WEA%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Issue with master-agent communication over ssh

Jigar R
Hello Ivan,

Attached the logs in the email 
  • jenkins.log - jenkins build output
  • Jenkins-agent.log - output from jenkins ssh agent 
Jenkins SSH agent was created with following:
  • launch method : launch agents via ssh
  • JavaPath: $JAVA_HOME
  • JVM options: -Xmx2048m -Xms2048m
  • Use TCP_NODELAY flag on the ssh connection - enabled
Environment information

  • Jenkins v2.249.1
  • RH6
  • SSH agent plugin 1.20
  • SSH slaves plugin 1.30.4

On Tuesday, February 2, 2021 at 5:27:07 PM UTC-5 [hidden email] wrote:
Weird, Could you share a screencapture of what you configurate? Also the whole exception those lines alone mean nothing. Know the version of Jenkins and the version of the SSH build agents plugins you use can help, the memory you have in your agents, if they are bare metal or cloud. In overall if you want help please provide more context. 

El mar, 2 feb 2021 a las 22:52, Jigar R (<[hidden email]>) escribió:


On Tuesday, February 2, 2021 at 12:20:19 PM UTC-5 [hidden email] wrote:
>I have 2 different kind of jenkins agents.
>1. java web start
>2. ssh
>If the memory was issue then wouldn't it fail on both cases?

Not necessarily, starting by the point that are different ways to establish the connection, the JNLP agents could not update the remoting jar file (depends on your configurations), so you can be running different versions of remoting. I agree with Jeff looks like an OOM issue, review my comments at https://groups.google.com/g/jenkinsci-users/c/nD3s06hSUXE/m/BQKk5GSYBwAJ my recommendation is to fix the mem for the remoting process to 1024M (-Xmx1024m -Xms1024m) see if the issue disappear or change, if disappear, you would have to adjust the remoting process memory to the right one between 256M-1024M, to use 512M usually is safe a not too much (but depends on your agents' memory we do not know how much they have)

I created SSH agent with -Xmx1024m -Xms1024m.
I do see bunch of Warnings about "Ping failed. Terminating the channel channel".
I got following error:
"
  WARNING: Failed to send back a reply to te rquest hudson.remoting.Request$....
  java.io.IOException: Resource temporarily unavailable
          at java.io.FileOutputStream.writeBytes(Native method)
          .....
"
 

El lunes, 1 de febrero de 2021 a las 23:52:19 UTC+1, [hidden email] escribió:
On Monday, January 4, 2021 at 12:37:52 PM UTC-5 [hidden email] wrote:

When I saw something like this in the past, it was because the process was running out of resources. Specifically when running SpotBugs, I got out-of-memory errors. I had to modify the pom to allocate more memory.

I have 2 different kind of jenkins agents.
1. java web start
2. ssh
If the memory was issue then wouldn't it fail on both cases?

I do see "Agent went offline during build Connection was broker: java.io.StreamCorruptedException: invalid stream header:".

I recommend ensuring your build runs normally on the agent without the additional complexities of the Jenkins environment. This might show where additional resources are needed. If that all passes, then continue on to the additional troubleshooting steps involving the Jenkins controller and agent.

Jeff Thompson

On 12/30/20 3:40 PM, [hidden email] wrote:
Interestingly, whenever Jenkins agent is executing PMD/Findbugs task, it runs into " Resource temporarily unavailable". 
I have
Please let me know if there is anything else I should consider.
On Friday, December 11, 2020 at 12:11:39 PM UTC-5 [hidden email] wrote:
The most common issue is related to disconnections because there is no traffic between the Jenkins instance and the agent, for that, you have to tune the TCP stack of your OS (see https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected), or enable the keepalive option in the SSH protocol this can be configured by setting ClientAliveInterval or TCPKeepAlive on the SSH server (/etc/ssh/sshd_config), also by setting ServerAliveInterval or TCPKeepAlive options for the user connection (/etc/ssh/ssh_config or ~/.ssh/ssh_config)

https://www.freebsd.org/cgi/man.cgi?sshd_config(5)
https://www.freebsd.org/cgi/man.cgi?ssh_config(5)

Also, check you follow the best practices to configure your SSH agents and enable SSH verbose log output in your service (see https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md)


El miércoles, 9 de diciembre de 2020 a las 17:53:19 UTC+1, [hidden email] escribió:
Hello Jenkins community, 

I have setup Jenkins agents over SSH and this agent frequently goes offline with Resource temporarily unavailable error. I had to configure my Jenkins agents to communicate over JNLP. Any advise on how to fix this issue?

Jenkins v2.249.1

[12/04/20 15:23:00] [SSH] Checking java version of java
[12/04/20 15:23:01] [SSH] java -version returned 1.8.0_202.
[12/04/20 15:23:01] [SSH] Starting sftp client.
[12/04/20 15:23:03] [SSH] Remote file system root $JENKINS_SSH_DATA does not exist. Will try to create it...
[12/04/20 15:23:03] [SSH] Copying latest remoting.jar...
[12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes.
Expanded the channel window size to 4MB
[12/04/20 15:23:03] [SSH] Starting agent process: cd "$JENKINS_SSH_DATA" && java  -jar remoting.jar -workDir $JENKINS_SSH_DATA -jar-cache $JENKINS_SSH_DATA/remoting/jarCache
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using $JENKINS_SSH_DATA/remoting as a remoting work directory
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to $JENKINS_SSH_DATA/remoting
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 4.5
This is a Unix agent
Evacuated stdout
Agent successfully connected and online
The Agent is connected, disconnect it before to try to connect it again.
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS discoverGitSystemConfig
WARNING: Exception caught during execution of command '[git, config, --system, --edit]' in '$GIT_PATH/bin', return code '128', error message 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory
'
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS$FileStoreAttributes saveToConfig
WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config] failed after 5 retries
Dec 04, 2020 3:33:59 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@493c5a4e
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:313)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89)
    at hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@2511e2d4
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:326)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90)
    at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85)
    at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54)
    at java.io.OutputStream.write(OutputStream.java:75)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
ERROR: Connection terminated
java.io.StreamCorruptedException: invalid stream header: 00025B42
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
    at hudson.remoting.Command.readFrom(Command.java:142)
    at hudson.remoting.Command.readFrom(Command.java:128)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel channel.
java.util.concurrent.TimeoutException: Ping started at 1607120910589 hasn't completed by 1607121150590
    at hudson.remoting.PingThread.ping(PingThread.java:134)
    at hudson.remoting.PingThread.run(PingThread.java:90)

Thanks,
Jigar R

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/615a4333-587c-4ea2-a11f-fea45fe38a56n%40googlegroups.com.


--

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/32d22541-1143-4c96-be3b-23c10bb5619cn%40googlegroups.com.

jenkins-agent.log (6K) Download Attachment
jenkins.log (11K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Issue with master-agent communication over ssh

Ivan Fernandez Calvo
I see some serialization fails and this breaks the channel, the plugin that causes the exception seems https://github.com/jenkinsci/tasks-plugin, and the `[Deprecated] Scan workspace for open tasks` I think matters, this plugins has been integrated into https://github.com/jenkinsci/warnings-ng-plugin and https://github.com/jenkinsci/analysis-model

ERROR: Step ‘[Deprecated] Scan workspace for open tasks’ aborted due to exception:
java.io.StreamCorruptedException: invalid type code: 6D
...
    at hudson.plugins.tasks.TasksPublisher.perform(TasksPublisher.java:182)
    at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:69)
    at hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:298)
    at jenkins.tasks.SimpleBuildStep.perform(SimpleBuildStep.java:112)
    at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
    at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
    at hudson.model.Build$BuildExecution.post2(Build.java:186)
    at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
    at hudson.model.Run.execute(Run.java:1919)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:428)

El jue, 4 feb 2021 a las 14:58, Jigar R (<[hidden email]>) escribió:
Hello Ivan,

Attached the logs in the email 
  • jenkins.log - jenkins build output
  • Jenkins-agent.log - output from jenkins ssh agent 
Jenkins SSH agent was created with following:
  • launch method : launch agents via ssh
  • JavaPath: $JAVA_HOME
  • JVM options: -Xmx2048m -Xms2048m
  • Use TCP_NODELAY flag on the ssh connection - enabled
Environment information

  • Jenkins v2.249.1
  • RH6
  • SSH agent plugin 1.20
  • SSH slaves plugin 1.30.4

On Tuesday, February 2, 2021 at 5:27:07 PM UTC-5 [hidden email] wrote:
Weird, Could you share a screencapture of what you configurate? Also the whole exception those lines alone mean nothing. Know the version of Jenkins and the version of the SSH build agents plugins you use can help, the memory you have in your agents, if they are bare metal or cloud. In overall if you want help please provide more context. 

El mar, 2 feb 2021 a las 22:52, Jigar R (<[hidden email]>) escribió:


On Tuesday, February 2, 2021 at 12:20:19 PM UTC-5 [hidden email] wrote:
>I have 2 different kind of jenkins agents.
>1. java web start
>2. ssh
>If the memory was issue then wouldn't it fail on both cases?

Not necessarily, starting by the point that are different ways to establish the connection, the JNLP agents could not update the remoting jar file (depends on your configurations), so you can be running different versions of remoting. I agree with Jeff looks like an OOM issue, review my comments at https://groups.google.com/g/jenkinsci-users/c/nD3s06hSUXE/m/BQKk5GSYBwAJ my recommendation is to fix the mem for the remoting process to 1024M (-Xmx1024m -Xms1024m) see if the issue disappear or change, if disappear, you would have to adjust the remoting process memory to the right one between 256M-1024M, to use 512M usually is safe a not too much (but depends on your agents' memory we do not know how much they have)

I created SSH agent with -Xmx1024m -Xms1024m.
I do see bunch of Warnings about "Ping failed. Terminating the channel channel".
I got following error:
"
  WARNING: Failed to send back a reply to te rquest hudson.remoting.Request$....
  java.io.IOException: Resource temporarily unavailable
          at java.io.FileOutputStream.writeBytes(Native method)
          .....
"
 

El lunes, 1 de febrero de 2021 a las 23:52:19 UTC+1, [hidden email] escribió:
On Monday, January 4, 2021 at 12:37:52 PM UTC-5 [hidden email] wrote:

When I saw something like this in the past, it was because the process was running out of resources. Specifically when running SpotBugs, I got out-of-memory errors. I had to modify the pom to allocate more memory.

I have 2 different kind of jenkins agents.
1. java web start
2. ssh
If the memory was issue then wouldn't it fail on both cases?

I do see "Agent went offline during build Connection was broker: java.io.StreamCorruptedException: invalid stream header:".

I recommend ensuring your build runs normally on the agent without the additional complexities of the Jenkins environment. This might show where additional resources are needed. If that all passes, then continue on to the additional troubleshooting steps involving the Jenkins controller and agent.

Jeff Thompson

On 12/30/20 3:40 PM, [hidden email] wrote:
Interestingly, whenever Jenkins agent is executing PMD/Findbugs task, it runs into " Resource temporarily unavailable". 
I have
Please let me know if there is anything else I should consider.
On Friday, December 11, 2020 at 12:11:39 PM UTC-5 [hidden email] wrote:
The most common issue is related to disconnections because there is no traffic between the Jenkins instance and the agent, for that, you have to tune the TCP stack of your OS (see https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected), or enable the keepalive option in the SSH protocol this can be configured by setting ClientAliveInterval or TCPKeepAlive on the SSH server (/etc/ssh/sshd_config), also by setting ServerAliveInterval or TCPKeepAlive options for the user connection (/etc/ssh/ssh_config or ~/.ssh/ssh_config)

https://www.freebsd.org/cgi/man.cgi?sshd_config(5)
https://www.freebsd.org/cgi/man.cgi?ssh_config(5)

Also, check you follow the best practices to configure your SSH agents and enable SSH verbose log output in your service (see https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md)


El miércoles, 9 de diciembre de 2020 a las 17:53:19 UTC+1, [hidden email] escribió:
Hello Jenkins community, 

I have setup Jenkins agents over SSH and this agent frequently goes offline with Resource temporarily unavailable error. I had to configure my Jenkins agents to communicate over JNLP. Any advise on how to fix this issue?

Jenkins v2.249.1

[12/04/20 15:23:00] [SSH] Checking java version of java
[12/04/20 15:23:01] [SSH] java -version returned 1.8.0_202.
[12/04/20 15:23:01] [SSH] Starting sftp client.
[12/04/20 15:23:03] [SSH] Remote file system root $JENKINS_SSH_DATA does not exist. Will try to create it...
[12/04/20 15:23:03] [SSH] Copying latest remoting.jar...
[12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes.
Expanded the channel window size to 4MB
[12/04/20 15:23:03] [SSH] Starting agent process: cd "$JENKINS_SSH_DATA" && java  -jar remoting.jar -workDir $JENKINS_SSH_DATA -jar-cache $JENKINS_SSH_DATA/remoting/jarCache
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using $JENKINS_SSH_DATA/remoting as a remoting work directory
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to $JENKINS_SSH_DATA/remoting
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 4.5
This is a Unix agent
Evacuated stdout
Agent successfully connected and online
The Agent is connected, disconnect it before to try to connect it again.
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS discoverGitSystemConfig
WARNING: Exception caught during execution of command '[git, config, --system, --edit]' in '$GIT_PATH/bin', return code '128', error message 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory
'
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS$FileStoreAttributes saveToConfig
WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config] failed after 5 retries
Dec 04, 2020 3:33:59 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@493c5a4e
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:313)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89)
    at hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@2511e2d4
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:326)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90)
    at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85)
    at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54)
    at java.io.OutputStream.write(OutputStream.java:75)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
ERROR: Connection terminated
java.io.StreamCorruptedException: invalid stream header: 00025B42
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
    at hudson.remoting.Command.readFrom(Command.java:142)
    at hudson.remoting.Command.readFrom(Command.java:128)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel channel.
java.util.concurrent.TimeoutException: Ping started at 1607120910589 hasn't completed by 1607121150590
    at hudson.remoting.PingThread.ping(PingThread.java:134)
    at hudson.remoting.PingThread.run(PingThread.java:90)

Thanks,
Jigar R

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/615a4333-587c-4ea2-a11f-fea45fe38a56n%40googlegroups.com.


--

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/32d22541-1143-4c96-be3b-23c10bb5619cn%40googlegroups.com.


--

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAKo5Qrr7zpoT__4EeG-oXzPeW-518g56oPphTV0989yLgyr9wQ%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Issue with master-agent communication over ssh

Jeff Thompson
In reply to this post by Ivan Fernandez Calvo

I agree with Ivan. There are differences, sometimes subtle, between how different agents behave for various reasons. The basic operations of the plugins and the protocol should be the same. There are can be differences in resource usage, platforms, etc. Sometimes plugins will behave differently on different one.

Try out some of Ivan's suggests or other troubleshooting like that and figure out how to isolate the problem.

Jeff Thompson

On 2/2/21 10:20 AM, Ivan Fernandez Calvo wrote:
>I have 2 different kind of jenkins agents.
>1. java web start
>2. ssh
>If the memory was issue then wouldn't it fail on both cases?

Not necessarily, starting by the point that are different ways to establish the connection, the JNLP agents could not update the remoting jar file (depends on your configurations), so you can be running different versions of remoting. I agree with Jeff looks like an OOM issue, review my comments at https://groups.google.com/g/jenkinsci-users/c/nD3s06hSUXE/m/BQKk5GSYBwAJ my recommendation is to fix the mem for the remoting process to 1024M (-Xmx1024m -Xms1024m) see if the issue disappear or change, if disappear, you would have to adjust the remoting process memory to the right one between 256M-1024M, to use 512M usually is safe a not too much (but depends on your agents' memory we do not know how much they have)

El lunes, 1 de febrero de 2021 a las 23:52:19 UTC+1, [hidden email] escribió:
On Monday, January 4, 2021 at 12:37:52 PM UTC-5 jtho...@... wrote:

When I saw something like this in the past, it was because the process was running out of resources. Specifically when running SpotBugs, I got out-of-memory errors. I had to modify the pom to allocate more memory.

I have 2 different kind of jenkins agents.
1. java web start
2. ssh
If the memory was issue then wouldn't it fail on both cases?

I do see "Agent went offline during build Connection was broker: java.io.StreamCorruptedException: invalid stream header:".

I recommend ensuring your build runs normally on the agent without the additional complexities of the Jenkins environment. This might show where additional resources are needed. If that all passes, then continue on to the additional troubleshooting steps involving the Jenkins controller and agent.

Jeff Thompson

On 12/30/20 3:40 PM, [hidden email] wrote:
Interestingly, whenever Jenkins agent is executing PMD/Findbugs task, it runs into " Resource temporarily unavailable". 
I have
Please let me know if there is anything else I should consider.
On Friday, December 11, 2020 at 12:11:39 PM UTC-5 [hidden email] wrote:
The most common issue is related to disconnections because there is no traffic between the Jenkins instance and the agent, for that, you have to tune the TCP stack of your OS (see https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected), or enable the keepalive option in the SSH protocol this can be configured by setting ClientAliveInterval or TCPKeepAlive on the SSH server (/etc/ssh/sshd_config), also by setting ServerAliveInterval or TCPKeepAlive options for the user connection (/etc/ssh/ssh_config or ~/.ssh/ssh_config)

https://www.freebsd.org/cgi/man.cgi?sshd_config(5)
https://www.freebsd.org/cgi/man.cgi?ssh_config(5)

Also, check you follow the best practices to configure your SSH agents and enable SSH verbose log output in your service (see https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md)


El miércoles, 9 de diciembre de 2020 a las 17:53:19 UTC+1, [hidden email] escribió:
Hello Jenkins community, 

I have setup Jenkins agents over SSH and this agent frequently goes offline with Resource temporarily unavailable error. I had to configure my Jenkins agents to communicate over JNLP. Any advise on how to fix this issue?

Jenkins v2.249.1

[12/04/20 15:23:00] [SSH] Checking java version of java
[12/04/20 15:23:01] [SSH] java -version returned 1.8.0_202.
[12/04/20 15:23:01] [SSH] Starting sftp client.
[12/04/20 15:23:03] [SSH] Remote file system root $JENKINS_SSH_DATA does not exist. Will try to create it...
[12/04/20 15:23:03] [SSH] Copying latest remoting.jar...
[12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes.
Expanded the channel window size to 4MB
[12/04/20 15:23:03] [SSH] Starting agent process: cd "$JENKINS_SSH_DATA" && java  -jar remoting.jar -workDir $JENKINS_SSH_DATA -jar-cache $JENKINS_SSH_DATA/remoting/jarCache
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using $JENKINS_SSH_DATA/remoting as a remoting work directory
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to $JENKINS_SSH_DATA/remoting
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 4.5
This is a Unix agent
Evacuated stdout
Agent successfully connected and online
The Agent is connected, disconnect it before to try to connect it again.
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS discoverGitSystemConfig
WARNING: Exception caught during execution of command '[git, config, --system, --edit]' in '$GIT_PATH/bin', return code '128', error message 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory
'
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS$FileStoreAttributes saveToConfig
WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config] failed after 5 retries
Dec 04, 2020 3:33:59 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@493c5a4e
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:313)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89)
    at hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@2511e2d4
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:326)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90)
    at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85)
    at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54)
    at java.io.OutputStream.write(OutputStream.java:75)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
ERROR: Connection terminated
java.io.StreamCorruptedException: invalid stream header: 00025B42
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
    at hudson.remoting.Command.readFrom(Command.java:142)
    at hudson.remoting.Command.readFrom(Command.java:128)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel channel.
java.util.concurrent.TimeoutException: Ping started at 1607120910589 hasn't completed by 1607121150590
    at hudson.remoting.PingThread.ping(PingThread.java:134)
    at hudson.remoting.PingThread.run(PingThread.java:90)

Thanks,
Jigar R

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/099267e4-36f2-4b1e-adf1-4ed6f6d5bf90n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/2999d743-ff8c-aa70-fcd5-e2ed64bcb854%40cloudbees.com.
Reply | Threaded
Open this post in threaded view
|

Re: Issue with master-agent communication over ssh

Jigar R
In reply to this post by Ivan Fernandez Calvo


On Thu, Feb 4, 2021 at 9:29 AM kuisathaverat <[hidden email]> wrote:
I see some serialization fails and this breaks the channel, the plugin that causes the exception seems https://github.com/jenkinsci/tasks-plugin, and the `[Deprecated] Scan workspace for open tasks` I think matters, this plugins has been integrated into https://github.com/jenkinsci/warnings-ng-plugin and https://github.com/jenkinsci/analysis-model

ERROR: Step ‘[Deprecated] Scan workspace for open tasks’ aborted due to exception:
java.io.StreamCorruptedException: invalid type code: 6D
...
    at hudson.plugins.tasks.TasksPublisher.perform(TasksPublisher.java:182)
    at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:69)
    at hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:298)
    at jenkins.tasks.SimpleBuildStep.perform(SimpleBuildStep.java:112)
    at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
    at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
    at hudson.model.Build$BuildExecution.post2(Build.java:186)
    at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
    at hudson.model.Run.execute(Run.java:1919)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:428)

Thanks for this information. I will move to use warning-ng plugin & see if it breaks this or not.

El jue, 4 feb 2021 a las 14:58, Jigar R (<[hidden email]>) escribió:
Hello Ivan,

Attached the logs in the email 
  • jenkins.log - jenkins build output
  • Jenkins-agent.log - output from jenkins ssh agent 
Jenkins SSH agent was created with following:
  • launch method : launch agents via ssh
  • JavaPath: $JAVA_HOME
  • JVM options: -Xmx2048m -Xms2048m
  • Use TCP_NODELAY flag on the ssh connection - enabled
Environment information

  • Jenkins v2.249.1
  • RH6
  • SSH agent plugin 1.20
  • SSH slaves plugin 1.30.4

On Tuesday, February 2, 2021 at 5:27:07 PM UTC-5 [hidden email] wrote:
Weird, Could you share a screencapture of what you configurate? Also the whole exception those lines alone mean nothing. Know the version of Jenkins and the version of the SSH build agents plugins you use can help, the memory you have in your agents, if they are bare metal or cloud. In overall if you want help please provide more context. 

El mar, 2 feb 2021 a las 22:52, Jigar R (<[hidden email]>) escribió:


On Tuesday, February 2, 2021 at 12:20:19 PM UTC-5 [hidden email] wrote:
>I have 2 different kind of jenkins agents.
>1. java web start
>2. ssh
>If the memory was issue then wouldn't it fail on both cases?

Not necessarily, starting by the point that are different ways to establish the connection, the JNLP agents could not update the remoting jar file (depends on your configurations), so you can be running different versions of remoting. I agree with Jeff looks like an OOM issue, review my comments at https://groups.google.com/g/jenkinsci-users/c/nD3s06hSUXE/m/BQKk5GSYBwAJ my recommendation is to fix the mem for the remoting process to 1024M (-Xmx1024m -Xms1024m) see if the issue disappear or change, if disappear, you would have to adjust the remoting process memory to the right one between 256M-1024M, to use 512M usually is safe a not too much (but depends on your agents' memory we do not know how much they have)

I created SSH agent with -Xmx1024m -Xms1024m.
I do see bunch of Warnings about "Ping failed. Terminating the channel channel".
I got following error:
"
  WARNING: Failed to send back a reply to te rquest hudson.remoting.Request$....
  java.io.IOException: Resource temporarily unavailable
          at java.io.FileOutputStream.writeBytes(Native method)
          .....
"
 

El lunes, 1 de febrero de 2021 a las 23:52:19 UTC+1, [hidden email] escribió:
On Monday, January 4, 2021 at 12:37:52 PM UTC-5 [hidden email] wrote:

When I saw something like this in the past, it was because the process was running out of resources. Specifically when running SpotBugs, I got out-of-memory errors. I had to modify the pom to allocate more memory.

I have 2 different kind of jenkins agents.
1. java web start
2. ssh
If the memory was issue then wouldn't it fail on both cases?

I do see "Agent went offline during build Connection was broker: java.io.StreamCorruptedException: invalid stream header:".

I recommend ensuring your build runs normally on the agent without the additional complexities of the Jenkins environment. This might show where additional resources are needed. If that all passes, then continue on to the additional troubleshooting steps involving the Jenkins controller and agent.

Jeff Thompson

On 12/30/20 3:40 PM, [hidden email] wrote:
Interestingly, whenever Jenkins agent is executing PMD/Findbugs task, it runs into " Resource temporarily unavailable". 
I have
Please let me know if there is anything else I should consider.
On Friday, December 11, 2020 at 12:11:39 PM UTC-5 [hidden email] wrote:
The most common issue is related to disconnections because there is no traffic between the Jenkins instance and the agent, for that, you have to tune the TCP stack of your OS (see https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected), or enable the keepalive option in the SSH protocol this can be configured by setting ClientAliveInterval or TCPKeepAlive on the SSH server (/etc/ssh/sshd_config), also by setting ServerAliveInterval or TCPKeepAlive options for the user connection (/etc/ssh/ssh_config or ~/.ssh/ssh_config)

https://www.freebsd.org/cgi/man.cgi?sshd_config(5)
https://www.freebsd.org/cgi/man.cgi?ssh_config(5)

Also, check you follow the best practices to configure your SSH agents and enable SSH verbose log output in your service (see https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md)


El miércoles, 9 de diciembre de 2020 a las 17:53:19 UTC+1, [hidden email] escribió:
Hello Jenkins community, 

I have setup Jenkins agents over SSH and this agent frequently goes offline with Resource temporarily unavailable error. I had to configure my Jenkins agents to communicate over JNLP. Any advise on how to fix this issue?

Jenkins v2.249.1

[12/04/20 15:23:00] [SSH] Checking java version of java
[12/04/20 15:23:01] [SSH] java -version returned 1.8.0_202.
[12/04/20 15:23:01] [SSH] Starting sftp client.
[12/04/20 15:23:03] [SSH] Remote file system root $JENKINS_SSH_DATA does not exist. Will try to create it...
[12/04/20 15:23:03] [SSH] Copying latest remoting.jar...
[12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes.
Expanded the channel window size to 4MB
[12/04/20 15:23:03] [SSH] Starting agent process: cd "$JENKINS_SSH_DATA" && java  -jar remoting.jar -workDir $JENKINS_SSH_DATA -jar-cache $JENKINS_SSH_DATA/remoting/jarCache
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using $JENKINS_SSH_DATA/remoting as a remoting work directory
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to $JENKINS_SSH_DATA/remoting
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 4.5
This is a Unix agent
Evacuated stdout
Agent successfully connected and online
The Agent is connected, disconnect it before to try to connect it again.
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS discoverGitSystemConfig
WARNING: Exception caught during execution of command '[git, config, --system, --edit]' in '$GIT_PATH/bin', return code '128', error message 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory
'
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS$FileStoreAttributes saveToConfig
WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config] failed after 5 retries
Dec 04, 2020 3:33:59 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@493c5a4e
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:313)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89)
    at hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@2511e2d4
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:326)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90)
    at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85)
    at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54)
    at java.io.OutputStream.write(OutputStream.java:75)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
ERROR: Connection terminated
java.io.StreamCorruptedException: invalid stream header: 00025B42
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
    at hudson.remoting.Command.readFrom(Command.java:142)
    at hudson.remoting.Command.readFrom(Command.java:128)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel channel.
java.util.concurrent.TimeoutException: Ping started at 1607120910589 hasn't completed by 1607121150590
    at hudson.remoting.PingThread.ping(PingThread.java:134)
    at hudson.remoting.PingThread.run(PingThread.java:90)

Thanks,
Jigar R

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/615a4333-587c-4ea2-a11f-fea45fe38a56n%40googlegroups.com.


--

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/32d22541-1143-4c96-be3b-23c10bb5619cn%40googlegroups.com.


--

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAKo5Qrr7zpoT__4EeG-oXzPeW-518g56oPphTV0989yLgyr9wQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAEOZ%3Dj64hNXvQ2%3DrmU%3D1x1A5ypopjx2QfzSODg6URtb944Ugww%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Issue with master-agent communication over ssh

Jigar R


On Thursday, February 4, 2021 at 12:20:21 PM UTC-5 Jigar R wrote:
On Thu, Feb 4, 2021 at 9:29 AM kuisathaverat <kuisat...@...> wrote:
I see some serialization fails and this breaks the channel, the plugin that causes the exception seems https://github.com/jenkinsci/tasks-plugin, and the `[Deprecated] Scan workspace for open tasks` I think matters, this plugins has been integrated into https://github.com/jenkinsci/warnings-ng-plugin and https://github.com/jenkinsci/analysis-model

ERROR: Step ‘[Deprecated] Scan workspace for open tasks’ aborted due to exception:
java.io.StreamCorruptedException: invalid type code: 6D
...
    at hudson.plugins.tasks.TasksPublisher.perform(TasksPublisher.java:182)
    at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:69)
    at hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:298)
    at jenkins.tasks.SimpleBuildStep.perform(SimpleBuildStep.java:112)
    at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
    at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
    at hudson.model.Build$BuildExecution.post2(Build.java:186)
    at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
    at hudson.model.Run.execute(Run.java:1919)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:428)

Thanks for this information. I will move to use warning-ng plugin & see if it breaks this or not.

I updated jenkins job to use warnings NG instead of deprecated plugins. After lots of trial and error, I found that jenkins SSH agent would throw EOFException while running JaCoCo plugin v3.0.7 (https://plugins.jenkins.io/jacoco/). Attached logs.


 

El jue, 4 feb 2021 a las 14:58, Jigar R (<jigarra...@...>) escribió:
Hello Ivan,

Attached the logs in the email 
  • jenkins.log - jenkins build output
  • Jenkins-agent.log - output from jenkins ssh agent 
Jenkins SSH agent was created with following:
  • launch method : launch agents via ssh
  • JavaPath: $JAVA_HOME
  • JVM options: -Xmx2048m -Xms2048m
  • Use TCP_NODELAY flag on the ssh connection - enabled
Environment information

  • Jenkins v2.249.1
  • RH6
  • SSH agent plugin 1.20
  • SSH slaves plugin 1.30.4

On Tuesday, February 2, 2021 at 5:27:07 PM UTC-5 kuisat...@... wrote:
Weird, Could you share a screencapture of what you configurate? Also the whole exception those lines alone mean nothing. Know the version of Jenkins and the version of the SSH build agents plugins you use can help, the memory you have in your agents, if they are bare metal or cloud. In overall if you want help please provide more context. 

El mar, 2 feb 2021 a las 22:52, Jigar R (<[hidden email]>) escribió:


On Tuesday, February 2, 2021 at 12:20:19 PM UTC-5 [hidden email] wrote:
>I have 2 different kind of jenkins agents.
>1. java web start
>2. ssh
>If the memory was issue then wouldn't it fail on both cases?

Not necessarily, starting by the point that are different ways to establish the connection, the JNLP agents could not update the remoting jar file (depends on your configurations), so you can be running different versions of remoting. I agree with Jeff looks like an OOM issue, review my comments at https://groups.google.com/g/jenkinsci-users/c/nD3s06hSUXE/m/BQKk5GSYBwAJ my recommendation is to fix the mem for the remoting process to 1024M (-Xmx1024m -Xms1024m) see if the issue disappear or change, if disappear, you would have to adjust the remoting process memory to the right one between 256M-1024M, to use 512M usually is safe a not too much (but depends on your agents' memory we do not know how much they have)

I created SSH agent with -Xmx1024m -Xms1024m.
I do see bunch of Warnings about "Ping failed. Terminating the channel channel".
I got following error:
"
  WARNING: Failed to send back a reply to te rquest hudson.remoting.Request$....
  java.io.IOException: Resource temporarily unavailable
          at java.io.FileOutputStream.writeBytes(Native method)
          .....
"
 

El lunes, 1 de febrero de 2021 a las 23:52:19 UTC+1, [hidden email] escribió:
On Monday, January 4, 2021 at 12:37:52 PM UTC-5 [hidden email] wrote:

When I saw something like this in the past, it was because the process was running out of resources. Specifically when running SpotBugs, I got out-of-memory errors. I had to modify the pom to allocate more memory.

I have 2 different kind of jenkins agents.
1. java web start
2. ssh
If the memory was issue then wouldn't it fail on both cases?

I do see "Agent went offline during build Connection was broker: java.io.StreamCorruptedException: invalid stream header:".

I recommend ensuring your build runs normally on the agent without the additional complexities of the Jenkins environment. This might show where additional resources are needed. If that all passes, then continue on to the additional troubleshooting steps involving the Jenkins controller and agent.

Jeff Thompson

On 12/30/20 3:40 PM, [hidden email] wrote:
Interestingly, whenever Jenkins agent is executing PMD/Findbugs task, it runs into " Resource temporarily unavailable". 
I have
Please let me know if there is anything else I should consider.
On Friday, December 11, 2020 at 12:11:39 PM UTC-5 [hidden email] wrote:
The most common issue is related to disconnections because there is no traffic between the Jenkins instance and the agent, for that, you have to tune the TCP stack of your OS (see https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected), or enable the keepalive option in the SSH protocol this can be configured by setting ClientAliveInterval or TCPKeepAlive on the SSH server (/etc/ssh/sshd_config), also by setting ServerAliveInterval or TCPKeepAlive options for the user connection (/etc/ssh/ssh_config or ~/.ssh/ssh_config)

https://www.freebsd.org/cgi/man.cgi?sshd_config(5)
https://www.freebsd.org/cgi/man.cgi?ssh_config(5)

Also, check you follow the best practices to configure your SSH agents and enable SSH verbose log output in your service (see https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md)


El miércoles, 9 de diciembre de 2020 a las 17:53:19 UTC+1, [hidden email] escribió:
Hello Jenkins community, 

I have setup Jenkins agents over SSH and this agent frequently goes offline with Resource temporarily unavailable error. I had to configure my Jenkins agents to communicate over JNLP. Any advise on how to fix this issue?

Jenkins v2.249.1

[12/04/20 15:23:00] [SSH] Checking java version of java
[12/04/20 15:23:01] [SSH] java -version returned 1.8.0_202.
[12/04/20 15:23:01] [SSH] Starting sftp client.
[12/04/20 15:23:03] [SSH] Remote file system root $JENKINS_SSH_DATA does not exist. Will try to create it...
[12/04/20 15:23:03] [SSH] Copying latest remoting.jar...
[12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes.
Expanded the channel window size to 4MB
[12/04/20 15:23:03] [SSH] Starting agent process: cd "$JENKINS_SSH_DATA" && java  -jar remoting.jar -workDir $JENKINS_SSH_DATA -jar-cache $JENKINS_SSH_DATA/remoting/jarCache
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using $JENKINS_SSH_DATA/remoting as a remoting work directory
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to $JENKINS_SSH_DATA/remoting
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 4.5
This is a Unix agent
Evacuated stdout
Agent successfully connected and online
The Agent is connected, disconnect it before to try to connect it again.
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS discoverGitSystemConfig
WARNING: Exception caught during execution of command '[git, config, --system, --edit]' in '$GIT_PATH/bin', return code '128', error message 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory
'
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS$FileStoreAttributes saveToConfig
WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config] failed after 5 retries
Dec 04, 2020 3:33:59 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@493c5a4e
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:313)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89)
    at hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@2511e2d4
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:326)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90)
    at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85)
    at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54)
    at java.io.OutputStream.write(OutputStream.java:75)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
ERROR: Connection terminated
java.io.StreamCorruptedException: invalid stream header: 00025B42
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
    at hudson.remoting.Command.readFrom(Command.java:142)
    at hudson.remoting.Command.readFrom(Command.java:128)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel channel.
java.util.concurrent.TimeoutException: Ping started at 1607120910589 hasn't completed by 1607121150590
    at hudson.remoting.PingThread.ping(PingThread.java:134)
    at hudson.remoting.PingThread.run(PingThread.java:90)

Thanks,
Jigar R

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/615a4333-587c-4ea2-a11f-fea45fe38a56n%40googlegroups.com.


--

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/32d22541-1143-4c96-be3b-23c10bb5619cn%40googlegroups.com.


--

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@....

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/78598c38-5627-42f1-84cd-e18408c04ec4n%40googlegroups.com.

jenkins2.log (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Issue with master-agent communication over ssh

Jigar R
Any recommendations on how should I go about this new error?

On Thursday, February 4, 2021 at 5:26:13 PM UTC-5 Jigar R wrote:
On Thursday, February 4, 2021 at 12:20:21 PM UTC-5 Jigar R wrote:
On Thu, Feb 4, 2021 at 9:29 AM kuisathaverat <[hidden email]> wrote:
I see some serialization fails and this breaks the channel, the plugin that causes the exception seems https://github.com/jenkinsci/tasks-plugin, and the `[Deprecated] Scan workspace for open tasks` I think matters, this plugins has been integrated into https://github.com/jenkinsci/warnings-ng-plugin and https://github.com/jenkinsci/analysis-model

ERROR: Step ‘[Deprecated] Scan workspace for open tasks’ aborted due to exception:
java.io.StreamCorruptedException: invalid type code: 6D
...
    at hudson.plugins.tasks.TasksPublisher.perform(TasksPublisher.java:182)
    at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:69)
    at hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:298)
    at jenkins.tasks.SimpleBuildStep.perform(SimpleBuildStep.java:112)
    at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
    at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
    at hudson.model.Build$BuildExecution.post2(Build.java:186)
    at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
    at hudson.model.Run.execute(Run.java:1919)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:428)

Thanks for this information. I will move to use warning-ng plugin & see if it breaks this or not.

I updated jenkins job to use warnings NG instead of deprecated plugins. After lots of trial and error, I found that jenkins SSH agent would throw EOFException while running JaCoCo plugin v3.0.7 (https://plugins.jenkins.io/jacoco/). Attached logs.


 

El jue, 4 feb 2021 a las 14:58, Jigar R (<[hidden email]>) escribió:
Hello Ivan,

Attached the logs in the email 
  • jenkins.log - jenkins build output
  • Jenkins-agent.log - output from jenkins ssh agent 
Jenkins SSH agent was created with following:
  • launch method : launch agents via ssh
  • JavaPath: $JAVA_HOME
  • JVM options: -Xmx2048m -Xms2048m
  • Use TCP_NODELAY flag on the ssh connection - enabled
Environment information

  • Jenkins v2.249.1
  • RH6
  • SSH agent plugin 1.20
  • SSH slaves plugin 1.30.4

On Tuesday, February 2, 2021 at 5:27:07 PM UTC-5 [hidden email] wrote:
Weird, Could you share a screencapture of what you configurate? Also the whole exception those lines alone mean nothing. Know the version of Jenkins and the version of the SSH build agents plugins you use can help, the memory you have in your agents, if they are bare metal or cloud. In overall if you want help please provide more context. 

El mar, 2 feb 2021 a las 22:52, Jigar R (<[hidden email]>) escribió:


On Tuesday, February 2, 2021 at 12:20:19 PM UTC-5 [hidden email] wrote:
>I have 2 different kind of jenkins agents.
>1. java web start
>2. ssh
>If the memory was issue then wouldn't it fail on both cases?

Not necessarily, starting by the point that are different ways to establish the connection, the JNLP agents could not update the remoting jar file (depends on your configurations), so you can be running different versions of remoting. I agree with Jeff looks like an OOM issue, review my comments at https://groups.google.com/g/jenkinsci-users/c/nD3s06hSUXE/m/BQKk5GSYBwAJ my recommendation is to fix the mem for the remoting process to 1024M (-Xmx1024m -Xms1024m) see if the issue disappear or change, if disappear, you would have to adjust the remoting process memory to the right one between 256M-1024M, to use 512M usually is safe a not too much (but depends on your agents' memory we do not know how much they have)

I created SSH agent with -Xmx1024m -Xms1024m.
I do see bunch of Warnings about "Ping failed. Terminating the channel channel".
I got following error:
"
  WARNING: Failed to send back a reply to te rquest hudson.remoting.Request$....
  java.io.IOException: Resource temporarily unavailable
          at java.io.FileOutputStream.writeBytes(Native method)
          .....
"
 

El lunes, 1 de febrero de 2021 a las 23:52:19 UTC+1, [hidden email] escribió:
On Monday, January 4, 2021 at 12:37:52 PM UTC-5 [hidden email] wrote:

When I saw something like this in the past, it was because the process was running out of resources. Specifically when running SpotBugs, I got out-of-memory errors. I had to modify the pom to allocate more memory.

I have 2 different kind of jenkins agents.
1. java web start
2. ssh
If the memory was issue then wouldn't it fail on both cases?

I do see "Agent went offline during build Connection was broker: java.io.StreamCorruptedException: invalid stream header:".

I recommend ensuring your build runs normally on the agent without the additional complexities of the Jenkins environment. This might show where additional resources are needed. If that all passes, then continue on to the additional troubleshooting steps involving the Jenkins controller and agent.

Jeff Thompson

On 12/30/20 3:40 PM, [hidden email] wrote:
Interestingly, whenever Jenkins agent is executing PMD/Findbugs task, it runs into " Resource temporarily unavailable". 
I have
Please let me know if there is anything else I should consider.
On Friday, December 11, 2020 at 12:11:39 PM UTC-5 [hidden email] wrote:
The most common issue is related to disconnections because there is no traffic between the Jenkins instance and the agent, for that, you have to tune the TCP stack of your OS (see https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected), or enable the keepalive option in the SSH protocol this can be configured by setting ClientAliveInterval or TCPKeepAlive on the SSH server (/etc/ssh/sshd_config), also by setting ServerAliveInterval or TCPKeepAlive options for the user connection (/etc/ssh/ssh_config or ~/.ssh/ssh_config)

https://www.freebsd.org/cgi/man.cgi?sshd_config(5)
https://www.freebsd.org/cgi/man.cgi?ssh_config(5)

Also, check you follow the best practices to configure your SSH agents and enable SSH verbose log output in your service (see https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md)


El miércoles, 9 de diciembre de 2020 a las 17:53:19 UTC+1, [hidden email] escribió:
Hello Jenkins community, 

I have setup Jenkins agents over SSH and this agent frequently goes offline with Resource temporarily unavailable error. I had to configure my Jenkins agents to communicate over JNLP. Any advise on how to fix this issue?

Jenkins v2.249.1

[12/04/20 15:23:00] [SSH] Checking java version of java
[12/04/20 15:23:01] [SSH] java -version returned 1.8.0_202.
[12/04/20 15:23:01] [SSH] Starting sftp client.
[12/04/20 15:23:03] [SSH] Remote file system root $JENKINS_SSH_DATA does not exist. Will try to create it...
[12/04/20 15:23:03] [SSH] Copying latest remoting.jar...
[12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes.
Expanded the channel window size to 4MB
[12/04/20 15:23:03] [SSH] Starting agent process: cd "$JENKINS_SSH_DATA" && java  -jar remoting.jar -workDir $JENKINS_SSH_DATA -jar-cache $JENKINS_SSH_DATA/remoting/jarCache
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using $JENKINS_SSH_DATA/remoting as a remoting work directory
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to $JENKINS_SSH_DATA/remoting
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 4.5
This is a Unix agent
Evacuated stdout
Agent successfully connected and online
The Agent is connected, disconnect it before to try to connect it again.
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS discoverGitSystemConfig
WARNING: Exception caught during execution of command '[git, config, --system, --edit]' in '$GIT_PATH/bin', return code '128', error message 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory
'
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS$FileStoreAttributes saveToConfig
WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config] failed after 5 retries
Dec 04, 2020 3:33:59 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@493c5a4e
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:313)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89)
    at hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@2511e2d4
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:326)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90)
    at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85)
    at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54)
    at java.io.OutputStream.write(OutputStream.java:75)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
ERROR: Connection terminated
java.io.StreamCorruptedException: invalid stream header: 00025B42
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
    at hudson.remoting.Command.readFrom(Command.java:142)
    at hudson.remoting.Command.readFrom(Command.java:128)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel channel.
java.util.concurrent.TimeoutException: Ping started at 1607120910589 hasn't completed by 1607121150590
    at hudson.remoting.PingThread.ping(PingThread.java:134)
    at hudson.remoting.PingThread.run(PingThread.java:90)

Thanks,
Jigar R

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/615a4333-587c-4ea2-a11f-fea45fe38a56n%40googlegroups.com.


--

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/32d22541-1143-4c96-be3b-23c10bb5619cn%40googlegroups.com.


--

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/c23aa67c-49e3-4a81-ad6e-5e3b4c81b762n%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Issue with master-agent communication over ssh

Ivan Fernandez Calvo
you attached the Jenkins build log and the Agent log, should be also an exception in the Jenkins log, Is the same you posted before `invalid type code: 6D`?

El mar, 9 feb 2021 a las 21:23, Jigar R (<[hidden email]>) escribió:
Any recommendations on how should I go about this new error?

On Thursday, February 4, 2021 at 5:26:13 PM UTC-5 Jigar R wrote:
On Thursday, February 4, 2021 at 12:20:21 PM UTC-5 Jigar R wrote:
On Thu, Feb 4, 2021 at 9:29 AM kuisathaverat <[hidden email]> wrote:
I see some serialization fails and this breaks the channel, the plugin that causes the exception seems https://github.com/jenkinsci/tasks-plugin, and the `[Deprecated] Scan workspace for open tasks` I think matters, this plugins has been integrated into https://github.com/jenkinsci/warnings-ng-plugin and https://github.com/jenkinsci/analysis-model

ERROR: Step ‘[Deprecated] Scan workspace for open tasks’ aborted due to exception:
java.io.StreamCorruptedException: invalid type code: 6D
...
    at hudson.plugins.tasks.TasksPublisher.perform(TasksPublisher.java:182)
    at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:69)
    at hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:298)
    at jenkins.tasks.SimpleBuildStep.perform(SimpleBuildStep.java:112)
    at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
    at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
    at hudson.model.Build$BuildExecution.post2(Build.java:186)
    at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
    at hudson.model.Run.execute(Run.java:1919)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:428)

Thanks for this information. I will move to use warning-ng plugin & see if it breaks this or not.

I updated jenkins job to use warnings NG instead of deprecated plugins. After lots of trial and error, I found that jenkins SSH agent would throw EOFException while running JaCoCo plugin v3.0.7 (https://plugins.jenkins.io/jacoco/). Attached logs.


 

El jue, 4 feb 2021 a las 14:58, Jigar R (<[hidden email]>) escribió:
Hello Ivan,

Attached the logs in the email 
  • jenkins.log - jenkins build output
  • Jenkins-agent.log - output from jenkins ssh agent 
Jenkins SSH agent was created with following:
  • launch method : launch agents via ssh
  • JavaPath: $JAVA_HOME
  • JVM options: -Xmx2048m -Xms2048m
  • Use TCP_NODELAY flag on the ssh connection - enabled
Environment information

  • Jenkins v2.249.1
  • RH6
  • SSH agent plugin 1.20
  • SSH slaves plugin 1.30.4

On Tuesday, February 2, 2021 at 5:27:07 PM UTC-5 [hidden email] wrote:
Weird, Could you share a screencapture of what you configurate? Also the whole exception those lines alone mean nothing. Know the version of Jenkins and the version of the SSH build agents plugins you use can help, the memory you have in your agents, if they are bare metal or cloud. In overall if you want help please provide more context. 

El mar, 2 feb 2021 a las 22:52, Jigar R (<[hidden email]>) escribió:


On Tuesday, February 2, 2021 at 12:20:19 PM UTC-5 [hidden email] wrote:
>I have 2 different kind of jenkins agents.
>1. java web start
>2. ssh
>If the memory was issue then wouldn't it fail on both cases?

Not necessarily, starting by the point that are different ways to establish the connection, the JNLP agents could not update the remoting jar file (depends on your configurations), so you can be running different versions of remoting. I agree with Jeff looks like an OOM issue, review my comments at https://groups.google.com/g/jenkinsci-users/c/nD3s06hSUXE/m/BQKk5GSYBwAJ my recommendation is to fix the mem for the remoting process to 1024M (-Xmx1024m -Xms1024m) see if the issue disappear or change, if disappear, you would have to adjust the remoting process memory to the right one between 256M-1024M, to use 512M usually is safe a not too much (but depends on your agents' memory we do not know how much they have)

I created SSH agent with -Xmx1024m -Xms1024m.
I do see bunch of Warnings about "Ping failed. Terminating the channel channel".
I got following error:
"
  WARNING: Failed to send back a reply to te rquest hudson.remoting.Request$....
  java.io.IOException: Resource temporarily unavailable
          at java.io.FileOutputStream.writeBytes(Native method)
          .....
"
 

El lunes, 1 de febrero de 2021 a las 23:52:19 UTC+1, [hidden email] escribió:
On Monday, January 4, 2021 at 12:37:52 PM UTC-5 [hidden email] wrote:

When I saw something like this in the past, it was because the process was running out of resources. Specifically when running SpotBugs, I got out-of-memory errors. I had to modify the pom to allocate more memory.

I have 2 different kind of jenkins agents.
1. java web start
2. ssh
If the memory was issue then wouldn't it fail on both cases?

I do see "Agent went offline during build Connection was broker: java.io.StreamCorruptedException: invalid stream header:".

I recommend ensuring your build runs normally on the agent without the additional complexities of the Jenkins environment. This might show where additional resources are needed. If that all passes, then continue on to the additional troubleshooting steps involving the Jenkins controller and agent.

Jeff Thompson

On 12/30/20 3:40 PM, [hidden email] wrote:
Interestingly, whenever Jenkins agent is executing PMD/Findbugs task, it runs into " Resource temporarily unavailable". 
I have
Please let me know if there is anything else I should consider.
On Friday, December 11, 2020 at 12:11:39 PM UTC-5 [hidden email] wrote:
The most common issue is related to disconnections because there is no traffic between the Jenkins instance and the agent, for that, you have to tune the TCP stack of your OS (see https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected), or enable the keepalive option in the SSH protocol this can be configured by setting ClientAliveInterval or TCPKeepAlive on the SSH server (/etc/ssh/sshd_config), also by setting ServerAliveInterval or TCPKeepAlive options for the user connection (/etc/ssh/ssh_config or ~/.ssh/ssh_config)

https://www.freebsd.org/cgi/man.cgi?sshd_config(5)
https://www.freebsd.org/cgi/man.cgi?ssh_config(5)

Also, check you follow the best practices to configure your SSH agents and enable SSH verbose log output in your service (see https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md)


El miércoles, 9 de diciembre de 2020 a las 17:53:19 UTC+1, [hidden email] escribió:
Hello Jenkins community, 

I have setup Jenkins agents over SSH and this agent frequently goes offline with Resource temporarily unavailable error. I had to configure my Jenkins agents to communicate over JNLP. Any advise on how to fix this issue?

Jenkins v2.249.1

[12/04/20 15:23:00] [SSH] Checking java version of java
[12/04/20 15:23:01] [SSH] java -version returned 1.8.0_202.
[12/04/20 15:23:01] [SSH] Starting sftp client.
[12/04/20 15:23:03] [SSH] Remote file system root $JENKINS_SSH_DATA does not exist. Will try to create it...
[12/04/20 15:23:03] [SSH] Copying latest remoting.jar...
[12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes.
Expanded the channel window size to 4MB
[12/04/20 15:23:03] [SSH] Starting agent process: cd "$JENKINS_SSH_DATA" && java  -jar remoting.jar -workDir $JENKINS_SSH_DATA -jar-cache $JENKINS_SSH_DATA/remoting/jarCache
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using $JENKINS_SSH_DATA/remoting as a remoting work directory
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to $JENKINS_SSH_DATA/remoting
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 4.5
This is a Unix agent
Evacuated stdout
Agent successfully connected and online
The Agent is connected, disconnect it before to try to connect it again.
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS discoverGitSystemConfig
WARNING: Exception caught during execution of command '[git, config, --system, --edit]' in '$GIT_PATH/bin', return code '128', error message 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory
'
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS$FileStoreAttributes saveToConfig
WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config] failed after 5 retries
Dec 04, 2020 3:33:59 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@493c5a4e
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:313)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89)
    at hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@2511e2d4
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:326)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90)
    at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85)
    at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54)
    at java.io.OutputStream.write(OutputStream.java:75)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
ERROR: Connection terminated
java.io.StreamCorruptedException: invalid stream header: 00025B42
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
    at hudson.remoting.Command.readFrom(Command.java:142)
    at hudson.remoting.Command.readFrom(Command.java:128)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel channel.
java.util.concurrent.TimeoutException: Ping started at 1607120910589 hasn't completed by 1607121150590
    at hudson.remoting.PingThread.ping(PingThread.java:134)
    at hudson.remoting.PingThread.run(PingThread.java:90)

Thanks,
Jigar R

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/615a4333-587c-4ea2-a11f-fea45fe38a56n%40googlegroups.com.


--

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/32d22541-1143-4c96-be3b-23c10bb5619cn%40googlegroups.com.


--

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/c23aa67c-49e3-4a81-ad6e-5e3b4c81b762n%40googlegroups.com.


--

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAKo5Qrr-GttQ6Yj3G5%2B6h8n_qPrbqcgFTjAg2G-EQY6L1wH_yA%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Issue with master-agent communication over ssh

Jigar R


On Wednesday, February 10, 2021 at 9:30:28 AM UTC-5 [hidden email] wrote:
you attached the Jenkins build log and the Agent log, should be also an exception in the Jenkins log, Is the same you posted before `invalid type code: 6D`?

This is all I see in jenkins.err log

2021-02-11 21:44:00.121+0000 [id=67650] INFO    h.r.SynchronousCommandTransport$ReaderThread#run: I/O error in channel ssh_agent
java.io.EOFException
    at java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:3106)
    at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1956)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
    at hudson.remoting.Command.readFromObjectStream(Command.java:155)
    at hudson.remoting.Command.readFrom(Command.java:142)
    at hudson.remoting.Command.readFrom(Command.java:128)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: java.io.IOException: Unexpected termination of the channel
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
2021-02-11 21:44:00.122+0000 [id=67569] WARNING h.m.AbstractBuild$AbstractBuildExecution#reportError: Step ‘Record compiler warnings and static analysis results’ aborted due to exception:
java.io.IOException: No workspace found for JOB_NAME #116
    at io.jenkins.plugins.analysis.core.steps.IssuesRecorder.perform(IssuesRecorder.java:577)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
    at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
    at hudson.model.Build$BuildExecution.post2(Build.java:186)
    at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
    at hudson.model.Run.execute(Run.java:1919)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:428)
2021-02-11 21:44:00.283+0000 [id=67569] WARNING hudson.ivy.IvyBuildTrigger#recomputeModuleDescriptor: Cannot read ivy file backup...removing ModuleDescriptor
2021-02-11 21:44:00.287+0000 [id=67569] WARNING hudson.ivy.IvyBuildTrigger#getModuleDescriptor: Node is offline for JOB_NAME, using project to get Module Descriptor

 
El mar, 9 feb 2021 a las 21:23, Jigar R (<jigarra...@...>) escribió:
Any recommendations on how should I go about this new error?

On Thursday, February 4, 2021 at 5:26:13 PM UTC-5 Jigar R wrote:
On Thursday, February 4, 2021 at 12:20:21 PM UTC-5 Jigar R wrote:
On Thu, Feb 4, 2021 at 9:29 AM kuisathaverat <[hidden email]> wrote:
I see some serialization fails and this breaks the channel, the plugin that causes the exception seems https://github.com/jenkinsci/tasks-plugin, and the `[Deprecated] Scan workspace for open tasks` I think matters, this plugins has been integrated into https://github.com/jenkinsci/warnings-ng-plugin and https://github.com/jenkinsci/analysis-model

ERROR: Step ‘[Deprecated] Scan workspace for open tasks’ aborted due to exception:
java.io.StreamCorruptedException: invalid type code: 6D
...
    at hudson.plugins.tasks.TasksPublisher.perform(TasksPublisher.java:182)
    at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:69)
    at hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:298)
    at jenkins.tasks.SimpleBuildStep.perform(SimpleBuildStep.java:112)
    at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
    at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
    at hudson.model.Build$BuildExecution.post2(Build.java:186)
    at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
    at hudson.model.Run.execute(Run.java:1919)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:428)

Thanks for this information. I will move to use warning-ng plugin & see if it breaks this or not.

I updated jenkins job to use warnings NG instead of deprecated plugins. After lots of trial and error, I found that jenkins SSH agent would throw EOFException while running JaCoCo plugin v3.0.7 (https://plugins.jenkins.io/jacoco/). Attached logs.


 

El jue, 4 feb 2021 a las 14:58, Jigar R (<[hidden email]>) escribió:
Hello Ivan,

Attached the logs in the email 
  • jenkins.log - jenkins build output
  • Jenkins-agent.log - output from jenkins ssh agent 
Jenkins SSH agent was created with following:
  • launch method : launch agents via ssh
  • JavaPath: $JAVA_HOME
  • JVM options: -Xmx2048m -Xms2048m
  • Use TCP_NODELAY flag on the ssh connection - enabled
Environment information

  • Jenkins v2.249.1
  • RH6
  • SSH agent plugin 1.20
  • SSH slaves plugin 1.30.4

On Tuesday, February 2, 2021 at 5:27:07 PM UTC-5 [hidden email] wrote:
Weird, Could you share a screencapture of what you configurate? Also the whole exception those lines alone mean nothing. Know the version of Jenkins and the version of the SSH build agents plugins you use can help, the memory you have in your agents, if they are bare metal or cloud. In overall if you want help please provide more context. 

El mar, 2 feb 2021 a las 22:52, Jigar R (<[hidden email]>) escribió:


On Tuesday, February 2, 2021 at 12:20:19 PM UTC-5 [hidden email] wrote:
>I have 2 different kind of jenkins agents.
>1. java web start
>2. ssh
>If the memory was issue then wouldn't it fail on both cases?

Not necessarily, starting by the point that are different ways to establish the connection, the JNLP agents could not update the remoting jar file (depends on your configurations), so you can be running different versions of remoting. I agree with Jeff looks like an OOM issue, review my comments at https://groups.google.com/g/jenkinsci-users/c/nD3s06hSUXE/m/BQKk5GSYBwAJ my recommendation is to fix the mem for the remoting process to 1024M (-Xmx1024m -Xms1024m) see if the issue disappear or change, if disappear, you would have to adjust the remoting process memory to the right one between 256M-1024M, to use 512M usually is safe a not too much (but depends on your agents' memory we do not know how much they have)

I created SSH agent with -Xmx1024m -Xms1024m.
I do see bunch of Warnings about "Ping failed. Terminating the channel channel".
I got following error:
"
  WARNING: Failed to send back a reply to te rquest hudson.remoting.Request$....
  java.io.IOException: Resource temporarily unavailable
          at java.io.FileOutputStream.writeBytes(Native method)
          .....
"
 

El lunes, 1 de febrero de 2021 a las 23:52:19 UTC+1, [hidden email] escribió:
On Monday, January 4, 2021 at 12:37:52 PM UTC-5 [hidden email] wrote:

When I saw something like this in the past, it was because the process was running out of resources. Specifically when running SpotBugs, I got out-of-memory errors. I had to modify the pom to allocate more memory.

I have 2 different kind of jenkins agents.
1. java web start
2. ssh
If the memory was issue then wouldn't it fail on both cases?

I do see "Agent went offline during build Connection was broker: java.io.StreamCorruptedException: invalid stream header:".

I recommend ensuring your build runs normally on the agent without the additional complexities of the Jenkins environment. This might show where additional resources are needed. If that all passes, then continue on to the additional troubleshooting steps involving the Jenkins controller and agent.

Jeff Thompson

On 12/30/20 3:40 PM, [hidden email] wrote:
Interestingly, whenever Jenkins agent is executing PMD/Findbugs task, it runs into " Resource temporarily unavailable". 
I have
Please let me know if there is anything else I should consider.
On Friday, December 11, 2020 at 12:11:39 PM UTC-5 [hidden email] wrote:
The most common issue is related to disconnections because there is no traffic between the Jenkins instance and the agent, for that, you have to tune the TCP stack of your OS (see https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected), or enable the keepalive option in the SSH protocol this can be configured by setting ClientAliveInterval or TCPKeepAlive on the SSH server (/etc/ssh/sshd_config), also by setting ServerAliveInterval or TCPKeepAlive options for the user connection (/etc/ssh/ssh_config or ~/.ssh/ssh_config)

https://www.freebsd.org/cgi/man.cgi?sshd_config(5)
https://www.freebsd.org/cgi/man.cgi?ssh_config(5)

Also, check you follow the best practices to configure your SSH agents and enable SSH verbose log output in your service (see https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md)


El miércoles, 9 de diciembre de 2020 a las 17:53:19 UTC+1, [hidden email] escribió:
Hello Jenkins community, 

I have setup Jenkins agents over SSH and this agent frequently goes offline with Resource temporarily unavailable error. I had to configure my Jenkins agents to communicate over JNLP. Any advise on how to fix this issue?

Jenkins v2.249.1

[12/04/20 15:23:00] [SSH] Checking java version of java
[12/04/20 15:23:01] [SSH] java -version returned 1.8.0_202.
[12/04/20 15:23:01] [SSH] Starting sftp client.
[12/04/20 15:23:03] [SSH] Remote file system root $JENKINS_SSH_DATA does not exist. Will try to create it...
[12/04/20 15:23:03] [SSH] Copying latest remoting.jar...
[12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes.
Expanded the channel window size to 4MB
[12/04/20 15:23:03] [SSH] Starting agent process: cd "$JENKINS_SSH_DATA" && java  -jar remoting.jar -workDir $JENKINS_SSH_DATA -jar-cache $JENKINS_SSH_DATA/remoting/jarCache
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using $JENKINS_SSH_DATA/remoting as a remoting work directory
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to $JENKINS_SSH_DATA/remoting
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 4.5
This is a Unix agent
Evacuated stdout
Agent successfully connected and online
The Agent is connected, disconnect it before to try to connect it again.
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS discoverGitSystemConfig
WARNING: Exception caught during execution of command '[git, config, --system, --edit]' in '$GIT_PATH/bin', return code '128', error message 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory
'
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS$FileStoreAttributes saveToConfig
WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config] failed after 5 retries
Dec 04, 2020 3:33:59 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@493c5a4e
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:313)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89)
    at hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@2511e2d4
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:326)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90)
    at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85)
    at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54)
    at java.io.OutputStream.write(OutputStream.java:75)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
ERROR: Connection terminated
java.io.StreamCorruptedException: invalid stream header: 00025B42
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
    at hudson.remoting.Command.readFrom(Command.java:142)
    at hudson.remoting.Command.readFrom(Command.java:128)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel channel.
java.util.concurrent.TimeoutException: Ping started at 1607120910589 hasn't completed by 1607121150590
    at hudson.remoting.PingThread.ping(PingThread.java:134)
    at hudson.remoting.PingThread.run(PingThread.java:90)

Thanks,
Jigar R

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/615a4333-587c-4ea2-a11f-fea45fe38a56n%40googlegroups.com.


--

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/32d22541-1143-4c96-be3b-23c10bb5619cn%40googlegroups.com.


--

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@....


--

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1ed964b4-e619-4680-a1b0-fc93765bd01cn%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Issue with master-agent communication over ssh

Ivan Fernandez Calvo
Looks like an abrupt disconnect, this point to the resources you give to the remote process and the resources you have in the agent. The resource management on a JVM is tricky, the JDK gives a 75% of the memory resources if you do not pass Xmx and Xms settings. So I wonder how much memory you have on those agents and if you set those limits. Based on my experience less than 4GB  for sunfire tasks use to be tight.

Un Saludo 
Ivan Fernandez Calvo

El 12 feb 2021, a las 17:32, Jigar R <[hidden email]> escribió:



On Wednesday, February 10, 2021 at 9:30:28 AM UTC-5 [hidden email] wrote:
you attached the Jenkins build log and the Agent log, should be also an exception in the Jenkins log, Is the same you posted before `invalid type code: 6D`?

This is all I see in jenkins.err log

2021-02-11 21:44:00.121+0000 [id=67650] INFO    h.r.SynchronousCommandTransport$ReaderThread#run: I/O error in channel ssh_agent
java.io.EOFException
    at java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:3106)
    at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1956)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
    at hudson.remoting.Command.readFromObjectStream(Command.java:155)
    at hudson.remoting.Command.readFrom(Command.java:142)
    at hudson.remoting.Command.readFrom(Command.java:128)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: java.io.IOException: Unexpected termination of the channel
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
2021-02-11 21:44:00.122+0000 [id=67569] WARNING h.m.AbstractBuild$AbstractBuildExecution#reportError: Step ‘Record compiler warnings and static analysis results’ aborted due to exception:
java.io.IOException: No workspace found for JOB_NAME #116
    at io.jenkins.plugins.analysis.core.steps.IssuesRecorder.perform(IssuesRecorder.java:577)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
    at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
    at hudson.model.Build$BuildExecution.post2(Build.java:186)
    at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
    at hudson.model.Run.execute(Run.java:1919)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:428)
2021-02-11 21:44:00.283+0000 [id=67569] WARNING hudson.ivy.IvyBuildTrigger#recomputeModuleDescriptor: Cannot read ivy file backup...removing ModuleDescriptor
2021-02-11 21:44:00.287+0000 [id=67569] WARNING hudson.ivy.IvyBuildTrigger#getModuleDescriptor: Node is offline for JOB_NAME, using project to get Module Descriptor

 
El mar, 9 feb 2021 a las 21:23, Jigar R (<jigarra...@...>) escribió:
Any recommendations on how should I go about this new error?

On Thursday, February 4, 2021 at 5:26:13 PM UTC-5 Jigar R wrote:
On Thursday, February 4, 2021 at 12:20:21 PM UTC-5 Jigar R wrote:
On Thu, Feb 4, 2021 at 9:29 AM kuisathaverat <[hidden email]> wrote:
I see some serialization fails and this breaks the channel, the plugin that causes the exception seems https://github.com/jenkinsci/tasks-plugin, and the `[Deprecated] Scan workspace for open tasks` I think matters, this plugins has been integrated into https://github.com/jenkinsci/warnings-ng-plugin and https://github.com/jenkinsci/analysis-model

ERROR: Step ‘[Deprecated] Scan workspace for open tasks’ aborted due to exception:
java.io.StreamCorruptedException: invalid type code: 6D
...
    at hudson.plugins.tasks.TasksPublisher.perform(TasksPublisher.java:182)
    at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:69)
    at hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:298)
    at jenkins.tasks.SimpleBuildStep.perform(SimpleBuildStep.java:112)
    at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
    at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
    at hudson.model.Build$BuildExecution.post2(Build.java:186)
    at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
    at hudson.model.Run.execute(Run.java:1919)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:428)

Thanks for this information. I will move to use warning-ng plugin & see if it breaks this or not.

I updated jenkins job to use warnings NG instead of deprecated plugins. After lots of trial and error, I found that jenkins SSH agent would throw EOFException while running JaCoCo plugin v3.0.7 (https://plugins.jenkins.io/jacoco/). Attached logs.


 

El jue, 4 feb 2021 a las 14:58, Jigar R (<[hidden email]>) escribió:
Hello Ivan,

Attached the logs in the email 
  • jenkins.log - jenkins build output
  • Jenkins-agent.log - output from jenkins ssh agent 
Jenkins SSH agent was created with following:
  • launch method : launch agents via ssh
  • JavaPath: $JAVA_HOME
  • JVM options: -Xmx2048m -Xms2048m
  • Use TCP_NODELAY flag on the ssh connection - enabled
Environment information

  • Jenkins v2.249.1
  • RH6
  • SSH agent plugin 1.20
  • SSH slaves plugin 1.30.4

On Tuesday, February 2, 2021 at 5:27:07 PM UTC-5 [hidden email] wrote:
Weird, Could you share a screencapture of what you configurate? Also the whole exception those lines alone mean nothing. Know the version of Jenkins and the version of the SSH build agents plugins you use can help, the memory you have in your agents, if they are bare metal or cloud. In overall if you want help please provide more context. 

El mar, 2 feb 2021 a las 22:52, Jigar R (<[hidden email]>) escribió:


On Tuesday, February 2, 2021 at 12:20:19 PM UTC-5 [hidden email] wrote:
>I have 2 different kind of jenkins agents.
>1. java web start
>2. ssh
>If the memory was issue then wouldn't it fail on both cases?

Not necessarily, starting by the point that are different ways to establish the connection, the JNLP agents could not update the remoting jar file (depends on your configurations), so you can be running different versions of remoting. I agree with Jeff looks like an OOM issue, review my comments at https://groups.google.com/g/jenkinsci-users/c/nD3s06hSUXE/m/BQKk5GSYBwAJ my recommendation is to fix the mem for the remoting process to 1024M (-Xmx1024m -Xms1024m) see if the issue disappear or change, if disappear, you would have to adjust the remoting process memory to the right one between 256M-1024M, to use 512M usually is safe a not too much (but depends on your agents' memory we do not know how much they have)

I created SSH agent with -Xmx1024m -Xms1024m.
I do see bunch of Warnings about "Ping failed. Terminating the channel channel".
I got following error:
"
  WARNING: Failed to send back a reply to te rquest hudson.remoting.Request$....
  java.io.IOException: Resource temporarily unavailable
          at java.io.FileOutputStream.writeBytes(Native method)
          .....
"
 

El lunes, 1 de febrero de 2021 a las 23:52:19 UTC+1, [hidden email] escribió:
On Monday, January 4, 2021 at 12:37:52 PM UTC-5 [hidden email] wrote:

When I saw something like this in the past, it was because the process was running out of resources. Specifically when running SpotBugs, I got out-of-memory errors. I had to modify the pom to allocate more memory.

I have 2 different kind of jenkins agents.
1. java web start
2. ssh
If the memory was issue then wouldn't it fail on both cases?

I do see "Agent went offline during build Connection was broker: java.io.StreamCorruptedException: invalid stream header:".

I recommend ensuring your build runs normally on the agent without the additional complexities of the Jenkins environment. This might show where additional resources are needed. If that all passes, then continue on to the additional troubleshooting steps involving the Jenkins controller and agent.

Jeff Thompson

On 12/30/20 3:40 PM, [hidden email] wrote:
Interestingly, whenever Jenkins agent is executing PMD/Findbugs task, it runs into " Resource temporarily unavailable". 
I have
Please let me know if there is anything else I should consider.
On Friday, December 11, 2020 at 12:11:39 PM UTC-5 [hidden email] wrote:
The most common issue is related to disconnections because there is no traffic between the Jenkins instance and the agent, for that, you have to tune the TCP stack of your OS (see https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected), or enable the keepalive option in the SSH protocol this can be configured by setting ClientAliveInterval or TCPKeepAlive on the SSH server (/etc/ssh/sshd_config), also by setting ServerAliveInterval or TCPKeepAlive options for the user connection (/etc/ssh/ssh_config or ~/.ssh/ssh_config)

https://www.freebsd.org/cgi/man.cgi?sshd_config(5)
https://www.freebsd.org/cgi/man.cgi?ssh_config(5)

Also, check you follow the best practices to configure your SSH agents and enable SSH verbose log output in your service (see https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md)


El miércoles, 9 de diciembre de 2020 a las 17:53:19 UTC+1, [hidden email] escribió:
Hello Jenkins community, 

I have setup Jenkins agents over SSH and this agent frequently goes offline with Resource temporarily unavailable error. I had to configure my Jenkins agents to communicate over JNLP. Any advise on how to fix this issue?

Jenkins v2.249.1

[12/04/20 15:23:00] [SSH] Checking java version of java
[12/04/20 15:23:01] [SSH] java -version returned 1.8.0_202.
[12/04/20 15:23:01] [SSH] Starting sftp client.
[12/04/20 15:23:03] [SSH] Remote file system root $JENKINS_SSH_DATA does not exist. Will try to create it...
[12/04/20 15:23:03] [SSH] Copying latest remoting.jar...
[12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes.
Expanded the channel window size to 4MB
[12/04/20 15:23:03] [SSH] Starting agent process: cd "$JENKINS_SSH_DATA" && java  -jar remoting.jar -workDir $JENKINS_SSH_DATA -jar-cache $JENKINS_SSH_DATA/remoting/jarCache
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using $JENKINS_SSH_DATA/remoting as a remoting work directory
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to $JENKINS_SSH_DATA/remoting
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 4.5
This is a Unix agent
Evacuated stdout
Agent successfully connected and online
The Agent is connected, disconnect it before to try to connect it again.
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS discoverGitSystemConfig
WARNING: Exception caught during execution of command '[git, config, --system, --edit]' in '$GIT_PATH/bin', return code '128', error message 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory
'
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS$FileStoreAttributes saveToConfig
WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config] failed after 5 retries
Dec 04, 2020 3:33:59 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@493c5a4e
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:313)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89)
    at hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@2511e2d4
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:326)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90)
    at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85)
    at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54)
    at java.io.OutputStream.write(OutputStream.java:75)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
ERROR: Connection terminated
java.io.StreamCorruptedException: invalid stream header: 00025B42
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
    at hudson.remoting.Command.readFrom(Command.java:142)
    at hudson.remoting.Command.readFrom(Command.java:128)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel channel.
java.util.concurrent.TimeoutException: Ping started at 1607120910589 hasn't completed by 1607121150590
    at hudson.remoting.PingThread.ping(PingThread.java:134)
    at hudson.remoting.PingThread.run(PingThread.java:90)

Thanks,
Jigar R

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/615a4333-587c-4ea2-a11f-fea45fe38a56n%40googlegroups.com.


--

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/32d22541-1143-4c96-be3b-23c10bb5619cn%40googlegroups.com.


--

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@....


--

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1ed964b4-e619-4680-a1b0-fc93765bd01cn%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/3E29DA25-7732-4BA6-9C29-47544088D8D7%40gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Issue with master-agent communication over ssh

Jigar R


Sent from my iPhone

On Feb 12, 2021, at 1:09 PM, Iván Fernández Calvo <[hidden email]> wrote:

Looks like an abrupt disconnect, this point to the resources you give to the remote process and the resources you have in the agent. The resource management on a JVM is tricky, the JDK gives a 75% of the memory resources if you do not pass Xmx and Xms settings. So I wonder how much memory you have on those agents and if you set those limits. Based on my experience less than 4GB  for sunfire tasks use to be tight.

Un Saludo 
Ivan Fernandez Calvo
I am using -Xmx2048m -Xms2048m. I will increase it to 4096 and see what happens. 

Thanks, 
Jigar R


El 12 feb 2021, a las 17:32, Jigar R <[hidden email]> escribió:



On Wednesday, February 10, 2021 at 9:30:28 AM UTC-5 [hidden email] wrote:
you attached the Jenkins build log and the Agent log, should be also an exception in the Jenkins log, Is the same you posted before `invalid type code: 6D`?

This is all I see in jenkins.err log

2021-02-11 21:44:00.121+0000 [id=67650] INFO    h.r.SynchronousCommandTransport$ReaderThread#run: I/O error in channel ssh_agent
java.io.EOFException
    at java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:3106)
    at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1956)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
    at hudson.remoting.Command.readFromObjectStream(Command.java:155)
    at hudson.remoting.Command.readFrom(Command.java:142)
    at hudson.remoting.Command.readFrom(Command.java:128)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: java.io.IOException: Unexpected termination of the channel
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
2021-02-11 21:44:00.122+0000 [id=67569] WARNING h.m.AbstractBuild$AbstractBuildExecution#reportError: Step ‘Record compiler warnings and static analysis results’ aborted due to exception:
java.io.IOException: No workspace found for JOB_NAME #116
    at io.jenkins.plugins.analysis.core.steps.IssuesRecorder.perform(IssuesRecorder.java:577)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
    at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
    at hudson.model.Build$BuildExecution.post2(Build.java:186)
    at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
    at hudson.model.Run.execute(Run.java:1919)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:428)
2021-02-11 21:44:00.283+0000 [id=67569] WARNING hudson.ivy.IvyBuildTrigger#recomputeModuleDescriptor: Cannot read ivy file backup...removing ModuleDescriptor
2021-02-11 21:44:00.287+0000 [id=67569] WARNING hudson.ivy.IvyBuildTrigger#getModuleDescriptor: Node is offline for JOB_NAME, using project to get Module Descriptor

 
El mar, 9 feb 2021 a las 21:23, Jigar R (<jigarra...@...>) escribió:
Any recommendations on how should I go about this new error?

On Thursday, February 4, 2021 at 5:26:13 PM UTC-5 Jigar R wrote:
On Thursday, February 4, 2021 at 12:20:21 PM UTC-5 Jigar R wrote:
On Thu, Feb 4, 2021 at 9:29 AM kuisathaverat <[hidden email]> wrote:
I see some serialization fails and this breaks the channel, the plugin that causes the exception seems https://github.com/jenkinsci/tasks-plugin, and the `[Deprecated] Scan workspace for open tasks` I think matters, this plugins has been integrated into https://github.com/jenkinsci/warnings-ng-plugin and https://github.com/jenkinsci/analysis-model

ERROR: Step ‘[Deprecated] Scan workspace for open tasks’ aborted due to exception:
java.io.StreamCorruptedException: invalid type code: 6D
...
    at hudson.plugins.tasks.TasksPublisher.perform(TasksPublisher.java:182)
    at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:69)
    at hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:298)
    at jenkins.tasks.SimpleBuildStep.perform(SimpleBuildStep.java:112)
    at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
    at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
    at hudson.model.Build$BuildExecution.post2(Build.java:186)
    at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
    at hudson.model.Run.execute(Run.java:1919)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:428)

Thanks for this information. I will move to use warning-ng plugin & see if it breaks this or not.

I updated jenkins job to use warnings NG instead of deprecated plugins. After lots of trial and error, I found that jenkins SSH agent would throw EOFException while running JaCoCo plugin v3.0.7 (https://plugins.jenkins.io/jacoco/). Attached logs.


 

El jue, 4 feb 2021 a las 14:58, Jigar R (<[hidden email]>) escribió:
Hello Ivan,

Attached the logs in the email 
  • jenkins.log - jenkins build output
  • Jenkins-agent.log - output from jenkins ssh agent 
Jenkins SSH agent was created with following:
  • launch method : launch agents via ssh
  • JavaPath: $JAVA_HOME
  • JVM options: -Xmx2048m -Xms2048m
  • Use TCP_NODELAY flag on the ssh connection - enabled
Environment information

  • Jenkins v2.249.1
  • RH6
  • SSH agent plugin 1.20
  • SSH slaves plugin 1.30.4

On Tuesday, February 2, 2021 at 5:27:07 PM UTC-5 [hidden email] wrote:
Weird, Could you share a screencapture of what you configurate? Also the whole exception those lines alone mean nothing. Know the version of Jenkins and the version of the SSH build agents plugins you use can help, the memory you have in your agents, if they are bare metal or cloud. In overall if you want help please provide more context. 

El mar, 2 feb 2021 a las 22:52, Jigar R (<[hidden email]>) escribió:


On Tuesday, February 2, 2021 at 12:20:19 PM UTC-5 [hidden email] wrote:
>I have 2 different kind of jenkins agents.
>1. java web start
>2. ssh
>If the memory was issue then wouldn't it fail on both cases?

Not necessarily, starting by the point that are different ways to establish the connection, the JNLP agents could not update the remoting jar file (depends on your configurations), so you can be running different versions of remoting. I agree with Jeff looks like an OOM issue, review my comments at https://groups.google.com/g/jenkinsci-users/c/nD3s06hSUXE/m/BQKk5GSYBwAJ my recommendation is to fix the mem for the remoting process to 1024M (-Xmx1024m -Xms1024m) see if the issue disappear or change, if disappear, you would have to adjust the remoting process memory to the right one between 256M-1024M, to use 512M usually is safe a not too much (but depends on your agents' memory we do not know how much they have)

I created SSH agent with -Xmx1024m -Xms1024m.
I do see bunch of Warnings about "Ping failed. Terminating the channel channel".
I got following error:
"
  WARNING: Failed to send back a reply to te rquest hudson.remoting.Request$....
  java.io.IOException: Resource temporarily unavailable
          at java.io.FileOutputStream.writeBytes(Native method)
          .....
"
 

El lunes, 1 de febrero de 2021 a las 23:52:19 UTC+1, [hidden email] escribió:
On Monday, January 4, 2021 at 12:37:52 PM UTC-5 [hidden email] wrote:

When I saw something like this in the past, it was because the process was running out of resources. Specifically when running SpotBugs, I got out-of-memory errors. I had to modify the pom to allocate more memory.

I have 2 different kind of jenkins agents.
1. java web start
2. ssh
If the memory was issue then wouldn't it fail on both cases?

I do see "Agent went offline during build Connection was broker: java.io.StreamCorruptedException: invalid stream header:".

I recommend ensuring your build runs normally on the agent without the additional complexities of the Jenkins environment. This might show where additional resources are needed. If that all passes, then continue on to the additional troubleshooting steps involving the Jenkins controller and agent.

Jeff Thompson

On 12/30/20 3:40 PM, [hidden email] wrote:
Interestingly, whenever Jenkins agent is executing PMD/Findbugs task, it runs into " Resource temporarily unavailable". 
I have
Please let me know if there is anything else I should consider.
On Friday, December 11, 2020 at 12:11:39 PM UTC-5 [hidden email] wrote:
The most common issue is related to disconnections because there is no traffic between the Jenkins instance and the agent, for that, you have to tune the TCP stack of your OS (see https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected), or enable the keepalive option in the SSH protocol this can be configured by setting ClientAliveInterval or TCPKeepAlive on the SSH server (/etc/ssh/sshd_config), also by setting ServerAliveInterval or TCPKeepAlive options for the user connection (/etc/ssh/ssh_config or ~/.ssh/ssh_config)

https://www.freebsd.org/cgi/man.cgi?sshd_config(5)
https://www.freebsd.org/cgi/man.cgi?ssh_config(5)

Also, check you follow the best practices to configure your SSH agents and enable SSH verbose log output in your service (see https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md)


El miércoles, 9 de diciembre de 2020 a las 17:53:19 UTC+1, [hidden email] escribió:
Hello Jenkins community, 

I have setup Jenkins agents over SSH and this agent frequently goes offline with Resource temporarily unavailable error. I had to configure my Jenkins agents to communicate over JNLP. Any advise on how to fix this issue?

Jenkins v2.249.1

[12/04/20 15:23:00] [SSH] Checking java version of java
[12/04/20 15:23:01] [SSH] java -version returned 1.8.0_202.
[12/04/20 15:23:01] [SSH] Starting sftp client.
[12/04/20 15:23:03] [SSH] Remote file system root $JENKINS_SSH_DATA does not exist. Will try to create it...
[12/04/20 15:23:03] [SSH] Copying latest remoting.jar...
[12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes.
Expanded the channel window size to 4MB
[12/04/20 15:23:03] [SSH] Starting agent process: cd "$JENKINS_SSH_DATA" && java  -jar remoting.jar -workDir $JENKINS_SSH_DATA -jar-cache $JENKINS_SSH_DATA/remoting/jarCache
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using $JENKINS_SSH_DATA/remoting as a remoting work directory
Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to $JENKINS_SSH_DATA/remoting
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 4.5
This is a Unix agent
Evacuated stdout
Agent successfully connected and online
The Agent is connected, disconnect it before to try to connect it again.
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS discoverGitSystemConfig
WARNING: Exception caught during execution of command '[git, config, --system, --edit]' in '$GIT_PATH/bin', return code '128', error message 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory
'
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS$FileStoreAttributes saveToConfig
WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config] failed after 5 retries
Dec 04, 2020 3:33:59 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@493c5a4e
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:313)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89)
    at hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@2511e2d4
java.io.IOException: Resource temporarily unavailable
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:326)
    at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88)
    at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90)
    at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85)
    at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54)
    at java.io.OutputStream.write(OutputStream.java:75)
    at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
    at hudson.remoting.Channel.send(Channel.java:766)
    at hudson.remoting.Request$2.run(Request.java:388)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
 
ERROR: Connection terminated
java.io.StreamCorruptedException: invalid stream header: 00025B42
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
    at hudson.remoting.Command.readFrom(Command.java:142)
    at hudson.remoting.Command.readFrom(Command.java:128)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel channel.
java.util.concurrent.TimeoutException: Ping started at 1607120910589 hasn't completed by 1607121150590
    at hudson.remoting.PingThread.ping(PingThread.java:134)
    at hudson.remoting.PingThread.run(PingThread.java:90)

Thanks,
Jigar R

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/615a4333-587c-4ea2-a11f-fea45fe38a56n%40googlegroups.com.


--

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/32d22541-1143-4c96-be3b-23c10bb5619cn%40googlegroups.com.


--

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@....


--

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1ed964b4-e619-4680-a1b0-fc93765bd01cn%40googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/3E29DA25-7732-4BA6-9C29-47544088D8D7%40gmail.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/22ACBD5A-0D12-4BCD-BCFE-AFC490606099%40gmail.com.
12