Issue with master-agent communication over ssh
Locked
 
|
Hello Jenkins community,
I have setup Jenkins agents over SSH and this agent frequently goes offline with Resource temporarily unavailable error. I had to configure my Jenkins agents to communicate over JNLP. Any advise on how to fix this issue? Jenkins v2.249.1 [12/04/20 15:23:00] [SSH] Checking java version of java [12/04/20 15:23:01] [SSH] java -version returned 1.8.0_202. [12/04/20 15:23:01] [SSH] Starting sftp client. [12/04/20 15:23:03] [SSH] Remote file system root $JENKINS_SSH_DATA does not exist. Will try to create it... [12/04/20 15:23:03] [SSH] Copying latest remoting.jar... [12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes. Expanded the channel window size to 4MB [12/04/20 15:23:03] [SSH] Starting agent process: cd "$JENKINS_SSH_DATA" && java -jar remoting.jar -workDir $JENKINS_SSH_DATA -jar-cache $JENKINS_SSH_DATA/remoting/jarCache Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using $JENKINS_SSH_DATA/remoting as a remoting work directory Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging INFO: Both error and output logs will be printed to $JENKINS_SSH_DATA/remoting <===[JENKINS REMOTING CAPACITY]===>channel started Remoting version: 4.5 This is a Unix agent Evacuated stdout Agent successfully connected and online The Agent is connected, disconnect it before to try to connect it again. Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS discoverGitSystemConfig WARNING: Exception caught during execution of command '[git, config, --system, --edit]' in '$GIT_PATH/bin', return code '128', error message 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory ' Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS$FileStoreAttributes saveToConfig WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config] failed after 5 retries Dec 04, 2020 3:33:59 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warn WARNING: Attempt to (de-)serialize anonymous class com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/ Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@493c5a4e java.io.IOException: Resource temporarily unavailable at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:313) at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83) at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89) at hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62) at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46) at hudson.remoting.Channel.send(Channel.java:766) at hudson.remoting.Request$2.run(Request.java:388) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@2511e2d4 java.io.IOException: Resource temporarily unavailable at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:326) at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88) at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90) at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85) at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54) at java.io.OutputStream.write(OutputStream.java:75) at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46) at hudson.remoting.Channel.send(Channel.java:766) at hudson.remoting.Request$2.run(Request.java:388) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ERROR: Connection terminated java.io.StreamCorruptedException: invalid stream header: 00025B42 at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63) Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead INFO: Ping failed. Terminating the channel channel. java.util.concurrent.TimeoutException: Ping started at 1607120910589 hasn't completed by 1607121150590 at hudson.remoting.PingThread.ping(PingThread.java:134) at hudson.remoting.PingThread.run(PingThread.java:90) Thanks, Jigar R You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d11ea78f-af76-44d9-895d-8270ee98872cn%40googlegroups.com. |
Re: Issue with master-agent communication over ssh
|
|
This has been discussed before. Did you search this group's archives? You might be better off re-starting a conversation in one of those threads.
My own experience is that I too saw this issue for many months. It cleared up on its own. In that interim I did upgrade jenkins more than once and I did switch from dualstack ipv4/ipv6 to ipv4 only for all jenkins work as I was getting errors deploying containers when ipv6 was enabled. I have no idea if either of these actually mattered, and I'm sure other things changed that I am assuming are completely unrelated but may not be. On Wednesday, December 9, 2020 at 11:53:19 AM UTC-5 [hidden email] wrote: Hello Jenkins community, You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/b1f9aa56-7092-4bdd-95c8-d1e9f0c96ac8n%40googlegroups.com. |
Re: Issue with master-agent communication over ssh
|
|
In reply to this post by JR ibkr
The most common issue is related to disconnections because there is no traffic between the Jenkins instance and the agent, for that, you have to tune the TCP stack of your OS (see https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected), or enable the keepalive option in the SSH protocol this can be configured by setting ClientAliveInterval or TCPKeepAlive on the SSH server (/etc/ssh/sshd_config), also by setting ServerAliveInterval or TCPKeepAlive options for the user connection (/etc/ssh/ssh_config or ~/.ssh/ssh_config) https://www.freebsd.org/cgi/man.cgi?sshd_config(5) Also, check you follow the best practices to configure your SSH agents and enable SSH verbose log output in your service (see https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md) El miércoles, 9 de diciembre de 2020 a las 17:53:19 UTC+1, [hidden email] escribió: Hello Jenkins community, You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/0cca6599-cdb2-4dc2-a7df-d756e4e793b3n%40googlegroups.com. |
|
|
Interestingly, whenever Jenkins agent is executing PMD/Findbugs task, it runs into " Resource temporarily unavailable".
I have
On Friday, December 11, 2020 at 12:11:39 PM UTC-5 [hidden email] wrote:
You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com. |
Re: Issue with master-agent communication over ssh
|
|
If the agent is running something there is traffic, so it si not related to keep alive settings, I wonder if after the issue happens there is a remoting process running on the agent. Did you check the SSHD service logs on your agent? https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md#enable-verbose-ssh-server-log-output Did you check the remoting logs? https://github.com/jenkinsci/remoting/blob/master/docs/logging.md Did you check for hs_err_pid error files in the root fs of the agent? http://www.oracle.com/technetwork/java/javase/felog-138657.html#gbwcy How much memory you set for your remoting process? How much memory your maven build needs? El miércoles, 30 de diciembre de 2020 a las 23:40:38 UTC+1, [hidden email] escribió: Interestingly, whenever Jenkins agent is executing PMD/Findbugs task, it runs into " Resource temporarily unavailable". You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/2b23ef01-3890-4fbd-962b-970a5efec3d8n%40googlegroups.com. |
Re: Issue with master-agent communication over ssh
|
|
In reply to this post by JR ibkr
When I saw something like this in the past, it was because the process was running out of resources. Specifically when running SpotBugs, I got out-of-memory errors. I had to modify the pom to allocate more memory. I recommend ensuring your build runs normally on the agent without the additional complexities of the Jenkins environment. This might show where additional resources are needed. If that all passes, then continue on to the additional troubleshooting steps involving the Jenkins controller and agent. Jeff Thompson On 12/30/20 3:40 PM, [hidden email]
wrote:
Interestingly, whenever Jenkins agent is executing PMD/Findbugs task, it runs into " Resource temporarily unavailable".-- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/88bf8ba5-d077-cc33-964b-1667138c1aa3%40cloudbees.com. |
«
Return to Jenkins users
|
1 view|%1 views
| Free forum by Nabble | Edit this page |