Memory issues with Jenkins

classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Memory issues with Jenkins

Sverre Moe-2
We have assigned 8GB of memory to our Jenkins instance.
JAVA_OPTIONS=-Xmx8g

Still we experience memory issues after a while running.
java.lang.OutOfMemoryError: unable to create new native thread

We have:
aprox 40 connected build agents
aprox 400 pipeline jobs

We have a test Jenkins instance running with the same jobs, this one connects to the same build agents (though on a different home directory).

Lately we have been getting disconnected build agents, that we cannot get up again without restarting Jenkins.

Can we assign more memory to a build agent? Would it have any affect on this issue?

This we got from one of our latest Pipeline builds that failed on a sh("find **** -exec ***") step. It failed on that build agent that is now disconnected.


java.lang.OutOfMemoryError: unable to create new native thread
		at java.lang.Thread.start0(Native Method)
		at java.lang.Thread.start(Thread.java:714)
		at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
		at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
		at java.lang.UNIXProcess.initStreams(UNIXProcess.java:288)
		at java.lang.UNIXProcess.lambda$new$2(UNIXProcess.java:258)
		at java.security.AccessController.doPrivileged(Native Method)
		at java.lang.UNIXProcess.<init>(UNIXProcess.java:257)
		at java.lang.ProcessImpl.start(ProcessImpl.java:134)
		at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
		at hudson.Proc$LocalProc.<init>(Proc.java:249)
Also:   java.io.IOException: error=11, Resource temporarily unavailable



SEVERE: Unexpected error when retrieving changeset
hudson.plugins.git.GitException: Error: git whatchanged --no-abbrev -M "--format=commit %H%ntree %T%nparent %P%nauthor %aN <%aE> %
ai%ncommitter %cN <%cE> %ci%n%n%w(76,4,4)%s%n%n%b" -n 1 b2c871830a03ea5f2fd2b21245afb09d51d69686 in /home/build/jenkins/workspace/
project_user_work
       at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$6.execute(CliGitAPIImpl.java:1012)
       at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
       at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
       at hudson.remoting.UserRequest.perform(UserRequest.java:212)
       at hudson.remoting.UserRequest.perform(UserRequest.java:54)
       at hudson.remoting.Request$2.run(Request.java:369)
       at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
       Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to master-sles12.3-x86_64_3
               at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
               at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
               at hudson.remoting.Channel.call(Channel.java:955)
               at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
               at sun.reflect.GeneratedMethodAccessor678.invoke(Unknown Source)
               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
               at java.lang.reflect.Method.invoke(Method.java:498)
               at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
               at com.sun.proxy.$Proxy104.execute(Unknown Source)
               at io.jenkins.blueocean.autofavorite.FavoritingScmListener.getChangeSet(FavoritingScmListener.java:159)
               at io.jenkins.blueocean.autofavorite.FavoritingScmListener.onCheckout(FavoritingScmListener.java:84)
               at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:140)
               at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
               at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
               at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingS
tepExecution.java:47)

Jul 01, 2019 11:51:12 AM hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler uncaughtException
SEVERE: A thread (Timer-9692/111139) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and
is usually indicative of a bug in the code.
java.lang.OutOfMemoryError: unable to create new native thread
       at java.lang.Thread.start0(Native Method)
       at java.lang.Thread.start(Thread.java:714)
       at java.util.Timer.<init>(Timer.java:160)
       at java.util.Timer.<init>(Timer.java:132)
       at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.scheduleRetryQueueProcessing(EventDispatcher.java:296)
       at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.processRetries(EventDispatcher.java:437)
       at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher$1.run(EventDispatcher.java:299)
       at java.util.TimerThread.mainLoop(Timer.java:555)
       at java.util.TimerThread.run(Timer.java:505)

INFO: Ping failed. Terminating the channel master-sles12.3-x86_64_3.
java.util.concurrent.TimeoutException: Ping started at 1561982408948 hasn't completed by 1561982648948
       at hudson.remoting.PingThread.ping(PingThread.java:134)
       at hudson.remoting.PingThread.run(PingThread.java:90)

Jul 01, 2019 2:04:11 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
INFO: I/O error in channel master-sles12.3-x86_64_3
java.io.IOException: Unexpected termination of the channel
WARNING: Failed to monitor master-sles12.3-x86_64_3 for Free Temp Space

Jul 01, 2019 2:04:11 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitorDetailed
WARNING: Failed to monitor master-sles12.3-x86_64_3 for Free Swap Space



The latest problem we got. It did not take down the build node. On all occasions of this problem it happens when the Pipeline is doing some IO on the Jenkins master. Here we manually restart the build again, and it builds fine.

Running on Jenkins in /var/lib/jenkins/workspace/project_master
[Pipeline] {
[Pipeline] parallel
[Pipeline] { (Branch: Setup)
[Pipeline] End of Pipeline
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:714)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1366)
	at com.google.common.eventbus.AsyncEventBus.dispatch(AsyncEventBus.java:90)
	at com.google.common.eventbus.AsyncEventBus.dispatchQueuedEvents(AsyncEventBus.java:81)
	at com.google.common.eventbus.EventBus.post(EventBus.java:264)
	at org.jenkinsci.plugins.pubsub.GuavaPubsubBus$1.publish(GuavaPubsubBus.java:70)
	at org.jenkinsci.plugins.pubsub.PubsubBus.publish(PubsubBus.java:141)
	at io.jenkins.blueocean.events.PipelineEventListener.publishEvent(PipelineEventListener.java:196)
	at io.jenkins.blueocean.events.PipelineEventListener.onNewHead(PipelineEventListener.java:85)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1463)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:458)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Finished: FAILURE

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/b60fa6e1-a8b3-4b2a-95a9-d8880d7dc7a6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Memory issues with Jenkins

Sverre Moe-2
Today it has been chaotic.
Several build agents disconnected

Unexpected termination of the channel

Many builds failed because of Memory error.

I have tried restarting Jenkins several times today.

Anyone have any suggestions?

tirsdag 2. juli 2019 14.34.25 UTC+2 skrev Sverre Moe følgende:
We have assigned 8GB of memory to our Jenkins instance.
JAVA_OPTIONS=-Xmx8g

Still we experience memory issues after a while running.
java.lang.OutOfMemoryError: unable to create new native thread

We have:
aprox 40 connected build agents
aprox 400 pipeline jobs

We have a test Jenkins instance running with the same jobs, this one connects to the same build agents (though on a different home directory).

Lately we have been getting disconnected build agents, that we cannot get up again without restarting Jenkins.

Can we assign more memory to a build agent? Would it have any affect on this issue?

This we got from one of our latest Pipeline builds that failed on a sh("find **** -exec ***") step. It failed on that build agent that is now disconnected.


java.lang.OutOfMemoryError: unable to create new native thread
		at java.lang.Thread.start0(Native Method)
		at java.lang.Thread.start(Thread.java:714)
		at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
		at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
		at java.lang.UNIXProcess.initStreams(UNIXProcess.java:288)
		at java.lang.UNIXProcess.lambda$new$2(UNIXProcess.java:258)
		at java.security.AccessController.doPrivileged(Native Method)
		at java.lang.UNIXProcess.<init>(UNIXProcess.java:257)
		at java.lang.ProcessImpl.start(ProcessImpl.java:134)
		at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
		at hudson.Proc$LocalProc.<init>(Proc.java:249)
Also:   java.io.IOException: error=11, Resource temporarily unavailable



SEVERE: Unexpected error when retrieving changeset
hudson.plugins.git.GitException: Error: git whatchanged --no-abbrev -M "--format=commit %H%ntree %T%nparent %P%nauthor %aN <%aE> %
ai%ncommitter %cN <%cE> %ci%n%n%w(76,4,4)%s%n%n%b" -n 1 b2c871830a03ea5f2fd2b21245afb09d51d69686 in /home/build/jenkins/workspace/
project_user_work
       at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$6.execute(CliGitAPIImpl.java:1012)
       at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
       at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
       at hudson.remoting.UserRequest.perform(UserRequest.java:212)
       at hudson.remoting.UserRequest.perform(UserRequest.java:54)
       at hudson.remoting.Request$2.run(Request.java:369)
       at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
       Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to master-sles12.3-x86_64_3
               at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
               at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
               at hudson.remoting.Channel.call(Channel.java:955)
               at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
               at sun.reflect.GeneratedMethodAccessor678.invoke(Unknown Source)
               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
               at java.lang.reflect.Method.invoke(Method.java:498)
               at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
               at com.sun.proxy.$Proxy104.execute(Unknown Source)
               at io.jenkins.blueocean.autofavorite.FavoritingScmListener.getChangeSet(FavoritingScmListener.java:159)
               at io.jenkins.blueocean.autofavorite.FavoritingScmListener.onCheckout(FavoritingScmListener.java:84)
               at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:140)
               at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
               at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
               at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingS
tepExecution.java:47)

Jul 01, 2019 11:51:12 AM hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler uncaughtException
SEVERE: A thread (Timer-9692/111139) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and
is usually indicative of a bug in the code.
java.lang.OutOfMemoryError: unable to create new native thread
       at java.lang.Thread.start0(Native Method)
       at java.lang.Thread.start(Thread.java:714)
       at java.util.Timer.<init>(Timer.java:160)
       at java.util.Timer.<init>(Timer.java:132)
       at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.scheduleRetryQueueProcessing(EventDispatcher.java:296)
       at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.processRetries(EventDispatcher.java:437)
       at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher$1.run(EventDispatcher.java:299)
       at java.util.TimerThread.mainLoop(Timer.java:555)
       at java.util.TimerThread.run(Timer.java:505)

INFO: Ping failed. Terminating the channel master-sles12.3-x86_64_3.
java.util.concurrent.TimeoutException: Ping started at 1561982408948 hasn't completed by 1561982648948
       at hudson.remoting.PingThread.ping(PingThread.java:134)
       at hudson.remoting.PingThread.run(PingThread.java:90)

Jul 01, 2019 2:04:11 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
INFO: I/O error in channel master-sles12.3-x86_64_3
java.io.IOException: Unexpected termination of the channel
WARNING: Failed to monitor master-sles12.3-x86_64_3 for Free Temp Space

Jul 01, 2019 2:04:11 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitorDetailed
WARNING: Failed to monitor master-sles12.3-x86_64_3 for Free Swap Space



The latest problem we got. It did not take down the build node. On all occasions of this problem it happens when the Pipeline is doing some IO on the Jenkins master. Here we manually restart the build again, and it builds fine.

Running on <a href="https://build-ci.spacetec.no:8443/computer/(master)/" style="font-size: 15px; color: rgb(149, 115, 115); --darkreader-inline-color:#b7b2a7;" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fcomputer%2F(master)%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNENLMD_obJmFOJ3xZVO3f0gqKIMZQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fcomputer%2F(master)%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNENLMD_obJmFOJ3xZVO3f0gqKIMZQ&#39;;return true;" data-darkreader-inline-color="">Jenkins in /var/lib/jenkins/workspace/project_master
[Pipeline] {<a href="https://build-ci.spacetec.no:8443/job/mcap_hrdfep/job/JPSS/75/console#" style="font-size: 15px; color: rgb(149, 115, 115); --darkreader-inline-color:#b7b2a7;" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fjob%2Fmcap_hrdfep%2Fjob%2FJPSS%2F75%2Fconsole%23\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFR-BnC66406Li_mm4E4t2uztg3Dg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fjob%2Fmcap_hrdfep%2Fjob%2FJPSS%2F75%2Fconsole%23\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFR-BnC66406Li_mm4E4t2uztg3Dg&#39;;return true;" data-darkreader-inline-color="">
[Pipeline] parallel<a href="https://build-ci.spacetec.no:8443/job/mcap_hrdfep/job/JPSS/75/console#" style="font-size: 15px; color: rgb(149, 115, 115); --darkreader-inline-color:#b7b2a7;" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fjob%2Fmcap_hrdfep%2Fjob%2FJPSS%2F75%2Fconsole%23\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFR-BnC66406Li_mm4E4t2uztg3Dg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fjob%2Fmcap_hrdfep%2Fjob%2FJPSS%2F75%2Fconsole%23\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFR-BnC66406Li_mm4E4t2uztg3Dg&#39;;return true;" data-darkreader-inline-color="">
[Pipeline] { (Branch: Setup)<a href="https://build-ci.spacetec.no:8443/job/mcap_hrdfep/job/JPSS/75/console#" style="font-size: 15px; color: rgb(149, 115, 115); --darkreader-inline-color:#b7b2a7;" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fjob%2Fmcap_hrdfep%2Fjob%2FJPSS%2F75%2Fconsole%23\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFR-BnC66406Li_mm4E4t2uztg3Dg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fjob%2Fmcap_hrdfep%2Fjob%2FJPSS%2F75%2Fconsole%23\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFR-BnC66406Li_mm4E4t2uztg3Dg&#39;;return true;" data-darkreader-inline-color="">
[Pipeline] End of Pipeline
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:714)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1366)
	at com.google.common.eventbus.AsyncEventBus.dispatch(AsyncEventBus.java:90)
	at com.google.common.eventbus.AsyncEventBus.dispatchQueuedEvents(AsyncEventBus.java:81)
	at com.google.common.eventbus.EventBus.post(EventBus.java:264)
	at org.jenkinsci.plugins.pubsub.GuavaPubsubBus$1.publish(GuavaPubsubBus.java:70)
	at org.jenkinsci.plugins.pubsub.PubsubBus.publish(PubsubBus.java:141)
	at io.jenkins.blueocean.events.PipelineEventListener.publishEvent(PipelineEventListener.java:196)
	at io.jenkins.blueocean.events.PipelineEventListener.onNewHead(PipelineEventListener.java:85)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1463)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:458)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Finished: FAILURE

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/6b1f3729-e456-41a9-a464-c63d061e2912%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Memory issues with Jenkins

Baptiste MATHUS
Did you enable GC logging to have a better understanding of the profile of your memory consumption? If not, I would recommend you do it first and analyze them.
https://jenkins.io/blog/2016/11/21/gc-tuning/ explained this part (and much more) quite well.

Then, once you understand better when it crashes, possibly you'll want to analyze a heap dump to see what is causing the problem.

Cheers


Le mar. 2 juil. 2019 à 15:30, Sverre Moe <[hidden email]> a écrit :
Today it has been chaotic.
Several build agents disconnected

Unexpected termination of the channel

Many builds failed because of Memory error.

I have tried restarting Jenkins several times today.

Anyone have any suggestions?

tirsdag 2. juli 2019 14.34.25 UTC+2 skrev Sverre Moe følgende:
We have assigned 8GB of memory to our Jenkins instance.
JAVA_OPTIONS=-Xmx8g

Still we experience memory issues after a while running.
java.lang.OutOfMemoryError: unable to create new native thread

We have:
aprox 40 connected build agents
aprox 400 pipeline jobs

We have a test Jenkins instance running with the same jobs, this one connects to the same build agents (though on a different home directory).

Lately we have been getting disconnected build agents, that we cannot get up again without restarting Jenkins.

Can we assign more memory to a build agent? Would it have any affect on this issue?

This we got from one of our latest Pipeline builds that failed on a sh("find **** -exec ***") step. It failed on that build agent that is now disconnected.


java.lang.OutOfMemoryError: unable to create new native thread
		at java.lang.Thread.start0(Native Method)
		at java.lang.Thread.start(Thread.java:714)
		at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
		at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
		at java.lang.UNIXProcess.initStreams(UNIXProcess.java:288)
		at java.lang.UNIXProcess.lambda$new$2(UNIXProcess.java:258)
		at java.security.AccessController.doPrivileged(Native Method)
		at java.lang.UNIXProcess.<init>(UNIXProcess.java:257)
		at java.lang.ProcessImpl.start(ProcessImpl.java:134)
		at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
		at hudson.Proc$LocalProc.<init>(Proc.java:249)
Also:   java.io.IOException: error=11, Resource temporarily unavailable



SEVERE: Unexpected error when retrieving changeset
hudson.plugins.git.GitException: Error: git whatchanged --no-abbrev -M "--format=commit %H%ntree %T%nparent %P%nauthor %aN <%aE> %
ai%ncommitter %cN <%cE> %ci%n%n%w(76,4,4)%s%n%n%b" -n 1 b2c871830a03ea5f2fd2b21245afb09d51d69686 in /home/build/jenkins/workspace/
project_user_work
       at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$6.execute(CliGitAPIImpl.java:1012)
       at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
       at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
       at hudson.remoting.UserRequest.perform(UserRequest.java:212)
       at hudson.remoting.UserRequest.perform(UserRequest.java:54)
       at hudson.remoting.Request$2.run(Request.java:369)
       at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
       Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to master-sles12.3-x86_64_3
               at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
               at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
               at hudson.remoting.Channel.call(Channel.java:955)
               at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
               at sun.reflect.GeneratedMethodAccessor678.invoke(Unknown Source)
               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
               at java.lang.reflect.Method.invoke(Method.java:498)
               at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
               at com.sun.proxy.$Proxy104.execute(Unknown Source)
               at io.jenkins.blueocean.autofavorite.FavoritingScmListener.getChangeSet(FavoritingScmListener.java:159)
               at io.jenkins.blueocean.autofavorite.FavoritingScmListener.onCheckout(FavoritingScmListener.java:84)
               at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:140)
               at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
               at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
               at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingS
tepExecution.java:47)

Jul 01, 2019 11:51:12 AM hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler uncaughtException
SEVERE: A thread (Timer-9692/111139) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and
is usually indicative of a bug in the code.
java.lang.OutOfMemoryError: unable to create new native thread
       at java.lang.Thread.start0(Native Method)
       at java.lang.Thread.start(Thread.java:714)
       at java.util.Timer.<init>(Timer.java:160)
       at java.util.Timer.<init>(Timer.java:132)
       at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.scheduleRetryQueueProcessing(EventDispatcher.java:296)
       at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.processRetries(EventDispatcher.java:437)
       at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher$1.run(EventDispatcher.java:299)
       at java.util.TimerThread.mainLoop(Timer.java:555)
       at java.util.TimerThread.run(Timer.java:505)

INFO: Ping failed. Terminating the channel master-sles12.3-x86_64_3.
java.util.concurrent.TimeoutException: Ping started at 1561982408948 hasn't completed by 1561982648948
       at hudson.remoting.PingThread.ping(PingThread.java:134)
       at hudson.remoting.PingThread.run(PingThread.java:90)

Jul 01, 2019 2:04:11 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
INFO: I/O error in channel master-sles12.3-x86_64_3
java.io.IOException: Unexpected termination of the channel
WARNING: Failed to monitor master-sles12.3-x86_64_3 for Free Temp Space

Jul 01, 2019 2:04:11 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitorDetailed
WARNING: Failed to monitor master-sles12.3-x86_64_3 for Free Swap Space



The latest problem we got. It did not take down the build node. On all occasions of this problem it happens when the Pipeline is doing some IO on the Jenkins master. Here we manually restart the build again, and it builds fine.

Running on Jenkins in /var/lib/jenkins/workspace/project_master
[Pipeline] {
[Pipeline] parallel
[Pipeline] { (Branch: Setup)
[Pipeline] End of Pipeline
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:714)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1366)
	at com.google.common.eventbus.AsyncEventBus.dispatch(AsyncEventBus.java:90)
	at com.google.common.eventbus.AsyncEventBus.dispatchQueuedEvents(AsyncEventBus.java:81)
	at com.google.common.eventbus.EventBus.post(EventBus.java:264)
	at org.jenkinsci.plugins.pubsub.GuavaPubsubBus$1.publish(GuavaPubsubBus.java:70)
	at org.jenkinsci.plugins.pubsub.PubsubBus.publish(PubsubBus.java:141)
	at io.jenkins.blueocean.events.PipelineEventListener.publishEvent(PipelineEventListener.java:196)
	at io.jenkins.blueocean.events.PipelineEventListener.onNewHead(PipelineEventListener.java:85)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1463)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:458)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Finished: FAILURE

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/6b1f3729-e456-41a9-a464-c63d061e2912%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CANWgJS5qokC%3D6vVcku1A6OXm%2Boi%3D03YzuSNQrCarDtCyCPy4pQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Memory issues with Jenkins

Jan Monterrubio
Correct me if I’m wrong but I don’t think increasing heap size will actually affect your ability to create more native threads. 


On Thu, Jul 4, 2019 at 16:03 Baptiste Mathus <[hidden email]> wrote:
Did you enable GC logging to have a better understanding of the profile of your memory consumption? If not, I would recommend you do it first and analyze them.
https://jenkins.io/blog/2016/11/21/gc-tuning/ explained this part (and much more) quite well.

Then, once you understand better when it crashes, possibly you'll want to analyze a heap dump to see what is causing the problem.

Cheers


Le mar. 2 juil. 2019 à 15:30, Sverre Moe <[hidden email]> a écrit :
Today it has been chaotic.
Several build agents disconnected

Unexpected termination of the channel

Many builds failed because of Memory error.

I have tried restarting Jenkins several times today.

Anyone have any suggestions?

tirsdag 2. juli 2019 14.34.25 UTC+2 skrev Sverre Moe følgende:
We have assigned 8GB of memory to our Jenkins instance.
JAVA_OPTIONS=-Xmx8g

Still we experience memory issues after a while running.
java.lang.OutOfMemoryError: unable to create new native thread

We have:
aprox 40 connected build agents
aprox 400 pipeline jobs

We have a test Jenkins instance running with the same jobs, this one connects to the same build agents (though on a different home directory).

Lately we have been getting disconnected build agents, that we cannot get up again without restarting Jenkins.

Can we assign more memory to a build agent? Would it have any affect on this issue?

This we got from one of our latest Pipeline builds that failed on a sh("find **** -exec ***") step. It failed on that build agent that is now disconnected.


java.lang.OutOfMemoryError: unable to create new native thread
		at java.lang.Thread.start0(Native Method)
		at java.lang.Thread.start(Thread.java:714)
		at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
		at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
		at java.lang.UNIXProcess.initStreams(UNIXProcess.java:288)
		at java.lang.UNIXProcess.lambda$new$2(UNIXProcess.java:258)
		at java.security.AccessController.doPrivileged(Native Method)
		at java.lang.UNIXProcess.<init>(UNIXProcess.java:257)
		at java.lang.ProcessImpl.start(ProcessImpl.java:134)
		at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
		at hudson.Proc$LocalProc.<init>(Proc.java:249)
Also:   java.io.IOException: error=11, Resource temporarily unavailable



SEVERE: Unexpected error when retrieving changeset
hudson.plugins.git.GitException: Error: git whatchanged --no-abbrev -M "--format=commit %H%ntree %T%nparent %P%nauthor %aN <%aE> %
ai%ncommitter %cN <%cE> %ci%n%n%w(76,4,4)%s%n%n%b" -n 1 b2c871830a03ea5f2fd2b21245afb09d51d69686 in /home/build/jenkins/workspace/
project_user_work
       at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$6.execute(CliGitAPIImpl.java:1012)
       at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
       at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
       at hudson.remoting.UserRequest.perform(UserRequest.java:212)
       at hudson.remoting.UserRequest.perform(UserRequest.java:54)
       at hudson.remoting.Request$2.run(Request.java:369)
       at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
       Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to master-sles12.3-x86_64_3
               at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
               at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
               at hudson.remoting.Channel.call(Channel.java:955)
               at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
               at sun.reflect.GeneratedMethodAccessor678.invoke(Unknown Source)
               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
               at java.lang.reflect.Method.invoke(Method.java:498)
               at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
               at com.sun.proxy.$Proxy104.execute(Unknown Source)
               at io.jenkins.blueocean.autofavorite.FavoritingScmListener.getChangeSet(FavoritingScmListener.java:159)
               at io.jenkins.blueocean.autofavorite.FavoritingScmListener.onCheckout(FavoritingScmListener.java:84)
               at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:140)
               at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
               at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
               at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingS
tepExecution.java:47)

Jul 01, 2019 11:51:12 AM hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler uncaughtException
SEVERE: A thread (Timer-9692/111139) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and
is usually indicative of a bug in the code.
java.lang.OutOfMemoryError: unable to create new native thread
       at java.lang.Thread.start0(Native Method)
       at java.lang.Thread.start(Thread.java:714)
       at java.util.Timer.<init>(Timer.java:160)
       at java.util.Timer.<init>(Timer.java:132)
       at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.scheduleRetryQueueProcessing(EventDispatcher.java:296)
       at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.processRetries(EventDispatcher.java:437)
       at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher$1.run(EventDispatcher.java:299)
       at java.util.TimerThread.mainLoop(Timer.java:555)
       at java.util.TimerThread.run(Timer.java:505)

INFO: Ping failed. Terminating the channel master-sles12.3-x86_64_3.
java.util.concurrent.TimeoutException: Ping started at 1561982408948 hasn't completed by 1561982648948
       at hudson.remoting.PingThread.ping(PingThread.java:134)
       at hudson.remoting.PingThread.run(PingThread.java:90)

Jul 01, 2019 2:04:11 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
INFO: I/O error in channel master-sles12.3-x86_64_3
java.io.IOException: Unexpected termination of the channel
WARNING: Failed to monitor master-sles12.3-x86_64_3 for Free Temp Space

Jul 01, 2019 2:04:11 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitorDetailed
WARNING: Failed to monitor master-sles12.3-x86_64_3 for Free Swap Space



The latest problem we got. It did not take down the build node. On all occasions of this problem it happens when the Pipeline is doing some IO on the Jenkins master. Here we manually restart the build again, and it builds fine.

Running on Jenkins in /var/lib/jenkins/workspace/project_master
[Pipeline] {
[Pipeline] parallel
[Pipeline] { (Branch: Setup)
[Pipeline] End of Pipeline
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:714)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1366)
	at com.google.common.eventbus.AsyncEventBus.dispatch(AsyncEventBus.java:90)
	at com.google.common.eventbus.AsyncEventBus.dispatchQueuedEvents(AsyncEventBus.java:81)
	at com.google.common.eventbus.EventBus.post(EventBus.java:264)
	at org.jenkinsci.plugins.pubsub.GuavaPubsubBus$1.publish(GuavaPubsubBus.java:70)
	at org.jenkinsci.plugins.pubsub.PubsubBus.publish(PubsubBus.java:141)
	at io.jenkins.blueocean.events.PipelineEventListener.publishEvent(PipelineEventListener.java:196)
	at io.jenkins.blueocean.events.PipelineEventListener.onNewHead(PipelineEventListener.java:85)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1463)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:458)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Finished: FAILURE

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/6b1f3729-e456-41a9-a464-c63d061e2912%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CANWgJS5qokC%3D6vVcku1A6OXm%2Boi%3D03YzuSNQrCarDtCyCPy4pQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CADgiF9L_1PDwbfY4nrBVrJrajOZ5dMjBJtzTybm_95qNNad9Fw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Memory issues with Jenkins

Sverre Moe-2
In reply to this post by Baptiste MATHUS
I will try turning on GC logging.

torsdag 4. juli 2019 23.04.55 UTC+2 skrev Baptiste Mathus følgende:
Did you enable GC logging to have a better understanding of the profile of your memory consumption? If not, I would recommend you do it first and analyze them.
<a href="https://jenkins.io/blog/2016/11/21/gc-tuning/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fjenkins.io%2Fblog%2F2016%2F11%2F21%2Fgc-tuning%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHxAz8FwdtwJFu_D6n6pvfXqnsNfQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fjenkins.io%2Fblog%2F2016%2F11%2F21%2Fgc-tuning%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHxAz8FwdtwJFu_D6n6pvfXqnsNfQ&#39;;return true;">https://jenkins.io/blog/2016/11/21/gc-tuning/ explained this part (and much more) quite well.

Then, once you understand better when it crashes, possibly you'll want to analyze a heap dump to see what is causing the problem.

Cheers


Le mar. 2 juil. 2019 à 15:30, Sverre Moe <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="ZWrphmPtCQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">sver...@...> a écrit :
Today it has been chaotic.
Several build agents disconnected

Unexpected termination of the channel

Many builds failed because of Memory error.

I have tried restarting Jenkins several times today.

Anyone have any suggestions?

tirsdag 2. juli 2019 14.34.25 UTC+2 skrev Sverre Moe følgende:
We have assigned 8GB of memory to our Jenkins instance.
JAVA_OPTIONS=-Xmx8g

Still we experience memory issues after a while running.
java.lang.OutOfMemoryError: unable to create new native thread

We have:
aprox 40 connected build agents
aprox 400 pipeline jobs

We have a test Jenkins instance running with the same jobs, this one connects to the same build agents (though on a different home directory).

Lately we have been getting disconnected build agents, that we cannot get up again without restarting Jenkins.

Can we assign more memory to a build agent? Would it have any affect on this issue?

This we got from one of our latest Pipeline builds that failed on a sh("find **** -exec ***") step. It failed on that build agent that is now disconnected.


java.lang.OutOfMemoryError: unable to create new native thread
		at java.lang.Thread.start0(Native Method)
		at java.lang.Thread.start(Thread.java:714)
		at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
		at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
		at java.lang.UNIXProcess.initStreams(UNIXProcess.java:288)
		at java.lang.UNIXProcess.lambda$new$2(UNIXProcess.java:258)
		at java.security.AccessController.doPrivileged(Native Method)
		at java.lang.UNIXProcess.<init>(UNIXProcess.java:257)
		at java.lang.ProcessImpl.start(ProcessImpl.java:134)
		at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
		at hudson.Proc$LocalProc.<init>(Proc.java:249)
Also:   java.io.IOException: error=11, Resource temporarily unavailable



SEVERE: Unexpected error when retrieving changeset
hudson.plugins.git.GitException: Error: git whatchanged --no-abbrev -M "--format=commit %H%ntree %T%nparent %P%nauthor %aN <%aE> %
ai%ncommitter %cN <%cE> %ci%n%n%w(76,4,4)%s%n%n%b" -n 1 b2c871830a03ea5f2fd2b21245afb09d51d69686 in /home/build/jenkins/workspace/
project_user_work
       at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$6.execute(CliGitAPIImpl.java:1012)
       at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
       at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
       at hudson.remoting.UserRequest.perform(UserRequest.java:212)
       at hudson.remoting.UserRequest.perform(UserRequest.java:54)
       at hudson.remoting.Request$2.run(Request.java:369)
       at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
       Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to master-sles12.3-x86_64_3
               at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
               at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
               at hudson.remoting.Channel.call(Channel.java:955)
               at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
               at sun.reflect.GeneratedMethodAccessor678.invoke(Unknown Source)
               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
               at java.lang.reflect.Method.invoke(Method.java:498)
               at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
               at com.sun.proxy.$Proxy104.execute(Unknown Source)
               at io.jenkins.blueocean.autofavorite.FavoritingScmListener.getChangeSet(FavoritingScmListener.java:159)
               at io.jenkins.blueocean.autofavorite.FavoritingScmListener.onCheckout(FavoritingScmListener.java:84)
               at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:140)
               at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
               at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
               at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingS
tepExecution.java:47)

Jul 01, 2019 11:51:12 AM hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler uncaughtException
SEVERE: A thread (Timer-9692/111139) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and
is usually indicative of a bug in the code.
java.lang.OutOfMemoryError: unable to create new native thread
       at java.lang.Thread.start0(Native Method)
       at java.lang.Thread.start(Thread.java:714)
       at java.util.Timer.<init>(Timer.java:160)
       at java.util.Timer.<init>(Timer.java:132)
       at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.scheduleRetryQueueProcessing(EventDispatcher.java:296)
       at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.processRetries(EventDispatcher.java:437)
       at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher$1.run(EventDispatcher.java:299)
       at java.util.TimerThread.mainLoop(Timer.java:555)
       at java.util.TimerThread.run(Timer.java:505)

INFO: Ping failed. Terminating the channel master-sles12.3-x86_64_3.
java.util.concurrent.TimeoutException: Ping started at 1561982408948 hasn't completed by 1561982648948
       at hudson.remoting.PingThread.ping(PingThread.java:134)
       at hudson.remoting.PingThread.run(PingThread.java:90)

Jul 01, 2019 2:04:11 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
INFO: I/O error in channel master-sles12.3-x86_64_3
java.io.IOException: Unexpected termination of the channel
WARNING: Failed to monitor master-sles12.3-x86_64_3 for Free Temp Space

Jul 01, 2019 2:04:11 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitorDetailed
WARNING: Failed to monitor master-sles12.3-x86_64_3 for Free Swap Space



The latest problem we got. It did not take down the build node. On all occasions of this problem it happens when the Pipeline is doing some IO on the Jenkins master. Here we manually restart the build again, and it builds fine.

Running on <a href="https://build-ci.spacetec.no:8443/computer/(master)/" style="font-size: 15px; color: rgb(149, 115, 115); --darkreader-inline-color:#b7b2a7;" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fcomputer%2F(master)%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNENLMD_obJmFOJ3xZVO3f0gqKIMZQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fcomputer%2F(master)%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNENLMD_obJmFOJ3xZVO3f0gqKIMZQ&#39;;return true;" data-darkreader-inline-color="">Jenkins in /var/lib/jenkins/workspace/project_master
[Pipeline] {<a href="https://build-ci.spacetec.no:8443/job/mcap_hrdfep/job/JPSS/75/console#" style="font-size: 15px; color: rgb(149, 115, 115); --darkreader-inline-color:#b7b2a7;" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fjob%2Fmcap_hrdfep%2Fjob%2FJPSS%2F75%2Fconsole%23\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFR-BnC66406Li_mm4E4t2uztg3Dg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fjob%2Fmcap_hrdfep%2Fjob%2FJPSS%2F75%2Fconsole%23\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFR-BnC66406Li_mm4E4t2uztg3Dg&#39;;return true;" data-darkreader-inline-color="">
[Pipeline] parallel<a href="https://build-ci.spacetec.no:8443/job/mcap_hrdfep/job/JPSS/75/console#" style="font-size: 15px; color: rgb(149, 115, 115); --darkreader-inline-color:#b7b2a7;" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fjob%2Fmcap_hrdfep%2Fjob%2FJPSS%2F75%2Fconsole%23\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFR-BnC66406Li_mm4E4t2uztg3Dg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fjob%2Fmcap_hrdfep%2Fjob%2FJPSS%2F75%2Fconsole%23\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFR-BnC66406Li_mm4E4t2uztg3Dg&#39;;return true;" data-darkreader-inline-color="">
[Pipeline] { (Branch: Setup)<a href="https://build-ci.spacetec.no:8443/job/mcap_hrdfep/job/JPSS/75/console#" style="font-size: 15px; color: rgb(149, 115, 115); --darkreader-inline-color:#b7b2a7;" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fjob%2Fmcap_hrdfep%2Fjob%2FJPSS%2F75%2Fconsole%23\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFR-BnC66406Li_mm4E4t2uztg3Dg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fjob%2Fmcap_hrdfep%2Fjob%2FJPSS%2F75%2Fconsole%23\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFR-BnC66406Li_mm4E4t2uztg3Dg&#39;;return true;" data-darkreader-inline-color="">
[Pipeline] End of Pipeline
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:714)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1366)
	at com.google.common.eventbus.AsyncEventBus.dispatch(AsyncEventBus.java:90)
	at com.google.common.eventbus.AsyncEventBus.dispatchQueuedEvents(AsyncEventBus.java:81)
	at com.google.common.eventbus.EventBus.post(EventBus.java:264)
	at org.jenkinsci.plugins.pubsub.GuavaPubsubBus$1.publish(GuavaPubsubBus.java:70)
	at org.jenkinsci.plugins.pubsub.PubsubBus.publish(PubsubBus.java:141)
	at io.jenkins.blueocean.events.PipelineEventListener.publishEvent(PipelineEventListener.java:196)
	at io.jenkins.blueocean.events.PipelineEventListener.onNewHead(PipelineEventListener.java:85)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1463)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:458)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Finished: FAILURE

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="ZWrphmPtCQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jenkins...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/6b1f3729-e456-41a9-a464-c63d061e2912%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/6b1f3729-e456-41a9-a464-c63d061e2912%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/6b1f3729-e456-41a9-a464-c63d061e2912%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/6b1f3729-e456-41a9-a464-c63d061e2912%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/f9cdac34-e674-4749-a425-495ec20932a1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Memory issues with Jenkins

Sverre Moe-2
In reply to this post by Jan Monterrubio

Since we don't use 32bit, the reason must be
- the virtual memory of the OS has been fully depleted

How can I check for this, and remedy it?

fredag 5. juli 2019 03.17.38 UTC+2 skrev Jan Monterrubio følgende:
Correct me if I’m wrong but I don’t think increasing heap size will actually affect your ability to create more native threads. 

See this for a possible explanation: 
<a href="https://plumbr.io/outofmemoryerror/unable-to-create-new-native-thread" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fplumbr.io%2Foutofmemoryerror%2Funable-to-create-new-native-thread\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG1MvleHJDFhJVoZklfX73e5IiZ-g&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fplumbr.io%2Foutofmemoryerror%2Funable-to-create-new-native-thread\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG1MvleHJDFhJVoZklfX73e5IiZ-g&#39;;return true;">https://plumbr.io/outofmemoryerror/unable-to-create-new-native-thread

On Thu, Jul 4, 2019 at 16:03 Baptiste Mathus <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="n4TX9i37CQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">m...@...> wrote:
Did you enable GC logging to have a better understanding of the profile of your memory consumption? If not, I would recommend you do it first and analyze them.
<a href="https://jenkins.io/blog/2016/11/21/gc-tuning/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fjenkins.io%2Fblog%2F2016%2F11%2F21%2Fgc-tuning%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHxAz8FwdtwJFu_D6n6pvfXqnsNfQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fjenkins.io%2Fblog%2F2016%2F11%2F21%2Fgc-tuning%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHxAz8FwdtwJFu_D6n6pvfXqnsNfQ&#39;;return true;">https://jenkins.io/blog/2016/11/21/gc-tuning/ explained this part (and much more) quite well.

Then, once you understand better when it crashes, possibly you'll want to analyze a heap dump to see what is causing the problem.

Cheers


Le mar. 2 juil. 2019 à 15:30, Sverre Moe <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="n4TX9i37CQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">sver...@...> a écrit :
Today it has been chaotic.
Several build agents disconnected

Unexpected termination of the channel

Many builds failed because of Memory error.

I have tried restarting Jenkins several times today.

Anyone have any suggestions?

tirsdag 2. juli 2019 14.34.25 UTC+2 skrev Sverre Moe følgende:
We have assigned 8GB of memory to our Jenkins instance.
JAVA_OPTIONS=-Xmx8g

Still we experience memory issues after a while running.
java.lang.OutOfMemoryError: unable to create new native thread

We have:
aprox 40 connected build agents
aprox 400 pipeline jobs

We have a test Jenkins instance running with the same jobs, this one connects to the same build agents (though on a different home directory).

Lately we have been getting disconnected build agents, that we cannot get up again without restarting Jenkins.

Can we assign more memory to a build agent? Would it have any affect on this issue?

This we got from one of our latest Pipeline builds that failed on a sh("find **** -exec ***") step. It failed on that build agent that is now disconnected.


java.lang.OutOfMemoryError: unable to create new native thread
		at java.lang.Thread.start0(Native Method)
		at java.lang.Thread.start(Thread.java:714)
		at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
		at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
		at java.lang.UNIXProcess.initStreams(UNIXProcess.java:288)
		at java.lang.UNIXProcess.lambda$new$2(UNIXProcess.java:258)
		at java.security.AccessController.doPrivileged(Native Method)
		at java.lang.UNIXProcess.<init>(UNIXProcess.java:257)
		at java.lang.ProcessImpl.start(ProcessImpl.java:134)
		at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
		at hudson.Proc$LocalProc.<init>(Proc.java:249)
Also:   java.io.IOException: error=11, Resource temporarily unavailable



SEVERE: Unexpected error when retrieving changeset
hudson.plugins.git.GitException: Error: git whatchanged --no-abbrev -M "--format=commit %H%ntree %T%nparent %P%nauthor %aN <%aE> %
ai%ncommitter %cN <%cE> %ci%n%n%w(76,4,4)%s%n%n%b" -n 1 b2c871830a03ea5f2fd2b21245afb09d51d69686 in /home/build/jenkins/workspace/
project_user_work
       at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$6.execute(CliGitAPIImpl.java:1012)
       at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
       at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
       at hudson.remoting.UserRequest.perform(UserRequest.java:212)
       at hudson.remoting.UserRequest.perform(UserRequest.java:54)
       at hudson.remoting.Request$2.run(Request.java:369)
       at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
       Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to master-sles12.3-x86_64_3
               at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
               at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
               at hudson.remoting.Channel.call(Channel.java:955)
               at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
               at sun.reflect.GeneratedMethodAccessor678.invoke(Unknown Source)
               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
               at java.lang.reflect.Method.invoke(Method.java:498)
               at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
               at com.sun.proxy.$Proxy104.execute(Unknown Source)
               at io.jenkins.blueocean.autofavorite.FavoritingScmListener.getChangeSet(FavoritingScmListener.java:159)
               at io.jenkins.blueocean.autofavorite.FavoritingScmListener.onCheckout(FavoritingScmListener.java:84)
               at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:140)
               at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
               at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
               at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingS
tepExecution.java:47)

Jul 01, 2019 11:51:12 AM hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler uncaughtException
SEVERE: A thread (Timer-9692/111139) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and
is usually indicative of a bug in the code.
java.lang.OutOfMemoryError: unable to create new native thread
       at java.lang.Thread.start0(Native Method)
       at java.lang.Thread.start(Thread.java:714)
       at java.util.Timer.<init>(Timer.java:160)
       at java.util.Timer.<init>(Timer.java:132)
       at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.scheduleRetryQueueProcessing(EventDispatcher.java:296)
       at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.processRetries(EventDispatcher.java:437)
       at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher$1.run(EventDispatcher.java:299)
       at java.util.TimerThread.mainLoop(Timer.java:555)
       at java.util.TimerThread.run(Timer.java:505)

INFO: Ping failed. Terminating the channel master-sles12.3-x86_64_3.
java.util.concurrent.TimeoutException: Ping started at 1561982408948 hasn't completed by 1561982648948
       at hudson.remoting.PingThread.ping(PingThread.java:134)
       at hudson.remoting.PingThread.run(PingThread.java:90)

Jul 01, 2019 2:04:11 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
INFO: I/O error in channel master-sles12.3-x86_64_3
java.io.IOException: Unexpected termination of the channel
WARNING: Failed to monitor master-sles12.3-x86_64_3 for Free Temp Space

Jul 01, 2019 2:04:11 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitorDetailed
WARNING: Failed to monitor master-sles12.3-x86_64_3 for Free Swap Space



The latest problem we got. It did not take down the build node. On all occasions of this problem it happens when the Pipeline is doing some IO on the Jenkins master. Here we manually restart the build again, and it builds fine.

Running on <a href="https://build-ci.spacetec.no:8443/computer/(master)/" style="font-size: 15px; color: rgb(149, 115, 115); --darkreader-inline-color:#b7b2a7;" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fcomputer%2F(master)%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNENLMD_obJmFOJ3xZVO3f0gqKIMZQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fcomputer%2F(master)%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNENLMD_obJmFOJ3xZVO3f0gqKIMZQ&#39;;return true;" data-darkreader-inline-color="">Jenkins in /var/lib/jenkins/workspace/project_master
[Pipeline] {<a href="https://build-ci.spacetec.no:8443/job/mcap_hrdfep/job/JPSS/75/console#" style="font-size: 15px; color: rgb(149, 115, 115); --darkreader-inline-color:#b7b2a7;" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fjob%2Fmcap_hrdfep%2Fjob%2FJPSS%2F75%2Fconsole%23\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFR-BnC66406Li_mm4E4t2uztg3Dg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fjob%2Fmcap_hrdfep%2Fjob%2FJPSS%2F75%2Fconsole%23\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFR-BnC66406Li_mm4E4t2uztg3Dg&#39;;return true;" data-darkreader-inline-color="">
[Pipeline] parallel<a href="https://build-ci.spacetec.no:8443/job/mcap_hrdfep/job/JPSS/75/console#" style="font-size: 15px; color: rgb(149, 115, 115); --darkreader-inline-color:#b7b2a7;" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fjob%2Fmcap_hrdfep%2Fjob%2FJPSS%2F75%2Fconsole%23\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFR-BnC66406Li_mm4E4t2uztg3Dg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fjob%2Fmcap_hrdfep%2Fjob%2FJPSS%2F75%2Fconsole%23\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFR-BnC66406Li_mm4E4t2uztg3Dg&#39;;return true;" data-darkreader-inline-color="">
[Pipeline] { (Branch: Setup)<a href="https://build-ci.spacetec.no:8443/job/mcap_hrdfep/job/JPSS/75/console#" style="font-size: 15px; color: rgb(149, 115, 115); --darkreader-inline-color:#b7b2a7;" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fjob%2Fmcap_hrdfep%2Fjob%2FJPSS%2F75%2Fconsole%23\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFR-BnC66406Li_mm4E4t2uztg3Dg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fjob%2Fmcap_hrdfep%2Fjob%2FJPSS%2F75%2Fconsole%23\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFR-BnC66406Li_mm4E4t2uztg3Dg&#39;;return true;" data-darkreader-inline-color="">
[Pipeline] End of Pipeline
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:714)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1366)
	at com.google.common.eventbus.AsyncEventBus.dispatch(AsyncEventBus.java:90)
	at com.google.common.eventbus.AsyncEventBus.dispatchQueuedEvents(AsyncEventBus.java:81)
	at com.google.common.eventbus.EventBus.post(EventBus.java:264)
	at org.jenkinsci.plugins.pubsub.GuavaPubsubBus$1.publish(GuavaPubsubBus.java:70)
	at org.jenkinsci.plugins.pubsub.PubsubBus.publish(PubsubBus.java:141)
	at io.jenkins.blueocean.events.PipelineEventListener.publishEvent(PipelineEventListener.java:196)
	at io.jenkins.blueocean.events.PipelineEventListener.onNewHead(PipelineEventListener.java:85)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1463)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:458)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Finished: FAILURE

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="n4TX9i37CQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jenkins...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/6b1f3729-e456-41a9-a464-c63d061e2912%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/6b1f3729-e456-41a9-a464-c63d061e2912%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/6b1f3729-e456-41a9-a464-c63d061e2912%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/6b1f3729-e456-41a9-a464-c63d061e2912%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="n4TX9i37CQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jenkins...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/CANWgJS5qokC%3D6vVcku1A6OXm%2Boi%3D03YzuSNQrCarDtCyCPy4pQ%40mail.gmail.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/CANWgJS5qokC%3D6vVcku1A6OXm%2Boi%3D03YzuSNQrCarDtCyCPy4pQ%40mail.gmail.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/CANWgJS5qokC%3D6vVcku1A6OXm%2Boi%3D03YzuSNQrCarDtCyCPy4pQ%40mail.gmail.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/CANWgJS5qokC%3D6vVcku1A6OXm%2Boi%3D03YzuSNQrCarDtCyCPy4pQ%40mail.gmail.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/09896d2e-2d44-44ba-86e6-8e10c8f4617d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Memory issues with Jenkins

Sverre Moe-2
Could it be issue with the virtual memory in the jenkins server? Because Jenkins does consume a lot of virtual memory.

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                        
13565 jenkins   20   0 12.641g 0.011t  13552 S 0.000 56.62 625:25.18 /usr/bin/java -XX:+UseG1GC -Xmx10g

It has amassed 12G of virtual memory. It is a problem with java and glibc
that can be remedied with
MALLOC_ARENA_MAX=1

I have tried adding ENVIRONMENT to /etc/systemd/system/jenkins.service, but it is not set before running jenkins.

https://www.ibm.com/developerworks/community/blogs/kevgrig/entry/linux_glibc_2_10_rhel_6_malloc_may_show_excessive_virtual_memory_usage?lang=en

https://stackoverflow.com/questions/561245/virtual-memory-usage-from-java-under-linux-too-much-memory-used/28935176#28935176


tirsdag 9. juli 2019 13.24.18 UTC+2 skrev Sverre Moe følgende:

Since we don't use 32bit, the reason must be
- the virtual memory of the OS has been fully depleted

How can I check for this, and remedy it?

fredag 5. juli 2019 03.17.38 UTC+2 skrev Jan Monterrubio følgende:
Correct me if I’m wrong but I don’t think increasing heap size will actually affect your ability to create more native threads. 

See this for a possible explanation: 
<a href="https://plumbr.io/outofmemoryerror/unable-to-create-new-native-thread" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fplumbr.io%2Foutofmemoryerror%2Funable-to-create-new-native-thread\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG1MvleHJDFhJVoZklfX73e5IiZ-g&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fplumbr.io%2Foutofmemoryerror%2Funable-to-create-new-native-thread\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG1MvleHJDFhJVoZklfX73e5IiZ-g&#39;;return true;">https://plumbr.io/outofmemoryerror/unable-to-create-new-native-thread

On Thu, Jul 4, 2019 at 16:03 Baptiste Mathus <[hidden email]> wrote:
Did you enable GC logging to have a better understanding of the profile of your memory consumption? If not, I would recommend you do it first and analyze them.
<a href="https://jenkins.io/blog/2016/11/21/gc-tuning/" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fjenkins.io%2Fblog%2F2016%2F11%2F21%2Fgc-tuning%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHxAz8FwdtwJFu_D6n6pvfXqnsNfQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fjenkins.io%2Fblog%2F2016%2F11%2F21%2Fgc-tuning%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHxAz8FwdtwJFu_D6n6pvfXqnsNfQ&#39;;return true;">https://jenkins.io/blog/2016/11/21/gc-tuning/ explained this part (and much more) quite well.

Then, once you understand better when it crashes, possibly you'll want to analyze a heap dump to see what is causing the problem.

Cheers


Le mar. 2 juil. 2019 à 15:30, Sverre Moe <[hidden email]> a écrit :
Today it has been chaotic.
Several build agents disconnected

Unexpected termination of the channel

Many builds failed because of Memory error.

I have tried restarting Jenkins several times today.

Anyone have any suggestions?

tirsdag 2. juli 2019 14.34.25 UTC+2 skrev Sverre Moe følgende:
We have assigned 8GB of memory to our Jenkins instance.
JAVA_OPTIONS=-Xmx8g

Still we experience memory issues after a while running.
java.lang.OutOfMemoryError: unable to create new native thread

We have:
aprox 40 connected build agents
aprox 400 pipeline jobs

We have a test Jenkins instance running with the same jobs, this one connects to the same build agents (though on a different home directory).

Lately we have been getting disconnected build agents, that we cannot get up again without restarting Jenkins.

Can we assign more memory to a build agent? Would it have any affect on this issue?

This we got from one of our latest Pipeline builds that failed on a sh("find **** -exec ***") step. It failed on that build agent that is now disconnected.


java.lang.OutOfMemoryError: unable to create new native thread
		at java.lang.Thread.start0(Native Method)
		at java.lang.Thread.start(Thread.java:714)
		at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
		at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
		at java.lang.UNIXProcess.initStreams(UNIXProcess.java:288)
		at java.lang.UNIXProcess.lambda$new$2(UNIXProcess.java:258)
		at java.security.AccessController.doPrivileged(Native Method)
		at java.lang.UNIXProcess.<init>(UNIXProcess.java:257)
		at java.lang.ProcessImpl.start(ProcessImpl.java:134)
		at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
		at hudson.Proc$LocalProc.<init>(Proc.java:249)
Also:   java.io.IOException: error=11, Resource temporarily unavailable



SEVERE: Unexpected error when retrieving changeset
hudson.plugins.git.GitException: Error: git whatchanged --no-abbrev -M "--format=commit %H%ntree %T%nparent %P%nauthor %aN <%aE> %
ai%ncommitter %cN <%cE> %ci%n%n%w(76,4,4)%s%n%n%b" -n 1 b2c871830a03ea5f2fd2b21245afb09d51d69686 in /home/build/jenkins/workspace/
project_user_work
       at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$6.execute(CliGitAPIImpl.java:1012)
       at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
       at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
       at hudson.remoting.UserRequest.perform(UserRequest.java:212)
       at hudson.remoting.UserRequest.perform(UserRequest.java:54)
       at hudson.remoting.Request$2.run(Request.java:369)
       at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
       Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to master-sles12.3-x86_64_3
               at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
               at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
               at hudson.remoting.Channel.call(Channel.java:955)
               at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
               at sun.reflect.GeneratedMethodAccessor678.invoke(Unknown Source)
               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
               at java.lang.reflect.Method.invoke(Method.java:498)
               at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
               at com.sun.proxy.$Proxy104.execute(Unknown Source)
               at io.jenkins.blueocean.autofavorite.FavoritingScmListener.getChangeSet(FavoritingScmListener.java:159)
               at io.jenkins.blueocean.autofavorite.FavoritingScmListener.onCheckout(FavoritingScmListener.java:84)
               at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:140)
               at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
               at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
               at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingS
tepExecution.java:47)

Jul 01, 2019 11:51:12 AM hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler uncaughtException
SEVERE: A thread (Timer-9692/111139) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and
is usually indicative of a bug in the code.
java.lang.OutOfMemoryError: unable to create new native thread
       at java.lang.Thread.start0(Native Method)
       at java.lang.Thread.start(Thread.java:714)
       at java.util.Timer.<init>(Timer.java:160)
       at java.util.Timer.<init>(Timer.java:132)
       at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.scheduleRetryQueueProcessing(EventDispatcher.java:296)
       at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.processRetries(EventDispatcher.java:437)
       at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher$1.run(EventDispatcher.java:299)
       at java.util.TimerThread.mainLoop(Timer.java:555)
       at java.util.TimerThread.run(Timer.java:505)

INFO: Ping failed. Terminating the channel master-sles12.3-x86_64_3.
java.util.concurrent.TimeoutException: Ping started at 1561982408948 hasn't completed by 1561982648948
       at hudson.remoting.PingThread.ping(PingThread.java:134)
       at hudson.remoting.PingThread.run(PingThread.java:90)

Jul 01, 2019 2:04:11 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
INFO: I/O error in channel master-sles12.3-x86_64_3
java.io.IOException: Unexpected termination of the channel
WARNING: Failed to monitor master-sles12.3-x86_64_3 for Free Temp Space

Jul 01, 2019 2:04:11 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitorDetailed
WARNING: Failed to monitor master-sles12.3-x86_64_3 for Free Swap Space



The latest problem we got. It did not take down the build node. On all occasions of this problem it happens when the Pipeline is doing some IO on the Jenkins master. Here we manually restart the build again, and it builds fine.

Running on <a href="https://build-ci.spacetec.no:8443/computer/(master)/" style="font-size: 15px; color: rgb(149, 115, 115); --darkreader-inline-color:#b7b2a7;" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fcomputer%2F(master)%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNENLMD_obJmFOJ3xZVO3f0gqKIMZQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fcomputer%2F(master)%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNENLMD_obJmFOJ3xZVO3f0gqKIMZQ&#39;;return true;" data-darkreader-inline-color="">Jenkins in /var/lib/jenkins/workspace/project_master
[Pipeline] {<a href="https://build-ci.spacetec.no:8443/job/mcap_hrdfep/job/JPSS/75/console#" style="font-size: 15px; color: rgb(149, 115, 115); --darkreader-inline-color:#b7b2a7;" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fjob%2Fmcap_hrdfep%2Fjob%2FJPSS%2F75%2Fconsole%23\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFR-BnC66406Li_mm4E4t2uztg3Dg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fjob%2Fmcap_hrdfep%2Fjob%2FJPSS%2F75%2Fconsole%23\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFR-BnC66406Li_mm4E4t2uztg3Dg&#39;;return true;" data-darkreader-inline-color="">
[Pipeline] parallel<a href="https://build-ci.spacetec.no:8443/job/mcap_hrdfep/job/JPSS/75/console#" style="font-size: 15px; color: rgb(149, 115, 115); --darkreader-inline-color:#b7b2a7;" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fjob%2Fmcap_hrdfep%2Fjob%2FJPSS%2F75%2Fconsole%23\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFR-BnC66406Li_mm4E4t2uztg3Dg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fjob%2Fmcap_hrdfep%2Fjob%2FJPSS%2F75%2Fconsole%23\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFR-BnC66406Li_mm4E4t2uztg3Dg&#39;;return true;" data-darkreader-inline-color="">
[Pipeline] { (Branch: Setup)<a href="https://build-ci.spacetec.no:8443/job/mcap_hrdfep/job/JPSS/75/console#" style="font-size: 15px; color: rgb(149, 115, 115); --darkreader-inline-color:#b7b2a7;" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fjob%2Fmcap_hrdfep%2Fjob%2FJPSS%2F75%2Fconsole%23\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFR-BnC66406Li_mm4E4t2uztg3Dg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbuild-ci.spacetec.no%3A8443%2Fjob%2Fmcap_hrdfep%2Fjob%2FJPSS%2F75%2Fconsole%23\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFR-BnC66406Li_mm4E4t2uztg3Dg&#39;;return true;" data-darkreader-inline-color="">
[Pipeline] End of Pipeline
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:714)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1366)
	at com.google.common.eventbus.AsyncEventBus.dispatch(AsyncEventBus.java:90)
	at com.google.common.eventbus.AsyncEventBus.dispatchQueuedEvents(AsyncEventBus.java:81)
	at com.google.common.eventbus.EventBus.post(EventBus.java:264)
	at org.jenkinsci.plugins.pubsub.GuavaPubsubBus$1.publish(GuavaPubsubBus.java:70)
	at org.jenkinsci.plugins.pubsub.PubsubBus.publish(PubsubBus.java:141)
	at io.jenkins.blueocean.events.PipelineEventListener.publishEvent(PipelineEventListener.java:196)
	at io.jenkins.blueocean.events.PipelineEventListener.onNewHead(PipelineEventListener.java:85)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1463)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:458)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Finished: FAILURE

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/6b1f3729-e456-41a9-a464-c63d061e2912%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/6b1f3729-e456-41a9-a464-c63d061e2912%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/6b1f3729-e456-41a9-a464-c63d061e2912%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/6b1f3729-e456-41a9-a464-c63d061e2912%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/CANWgJS5qokC%3D6vVcku1A6OXm%2Boi%3D03YzuSNQrCarDtCyCPy4pQ%40mail.gmail.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/CANWgJS5qokC%3D6vVcku1A6OXm%2Boi%3D03YzuSNQrCarDtCyCPy4pQ%40mail.gmail.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/CANWgJS5qokC%3D6vVcku1A6OXm%2Boi%3D03YzuSNQrCarDtCyCPy4pQ%40mail.gmail.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/CANWgJS5qokC%3D6vVcku1A6OXm%2Boi%3D03YzuSNQrCarDtCyCPy4pQ%40mail.gmail.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/e49d203f-db67-41ae-b223-313c5535e537%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Memory issues with Jenkins

Ivan Fernandez Calvo
Hi,

When the issue happens, Did you check the number of threads that Jenkins has open? How many file descriptors can your process open (run ulimit -a with the user jenkins)? here you have a good KB about memory and user limit on Jenkins Prepare Jenkins for Support

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/7ef87f30-febf-410d-8996-566d21eac359%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Memory issues with Jenkins

Sverre Moe-2
jenkins@meoscorebs12:~> ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 80229
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 80229
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited


I have check the number of processes, with "ps aux", but not the number of threads.
Right now, (Jenkins was restarted 2 day ago and few builds have run), there are 264 threads for Jenkins currently.
Is there any way I can find out what each thread is for?

lørdag 13. juli 2019 13.02.15 UTC+2 skrev Ivan Fernandez Calvo følgende:
Hi,

When the issue happens, Did you check the number of threads that Jenkins has open? How many file descriptors can your process open (run ulimit -a with the user jenkins)? here you have a good KB about memory and user limit on Jenkins <a href="https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEfQICfugtdoTEmraZRb7IFzUWMtA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEfQICfugtdoTEmraZRb7IFzUWMtA&#39;;return true;">Prepare Jenkins for Support

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/363d9039-e391-4682-97bc-d332039bc83f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Memory issues with Jenkins

Sverre Moe-2
I checked out that link

Tried the ulimit settings
https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support#bulimitsettingsjustforlinuxos

Will try the additional JVM flags suggested in
https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support#ajavaparameters

søndag 14. juli 2019 13.29.22 UTC+2 skrev Sverre Moe følgende:
jenkins@meoscorebs12:~> ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 80229
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 80229
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited


I have check the number of processes, with "ps aux", but not the number of threads.
Right now, (Jenkins was restarted 2 day ago and few builds have run), there are 264 threads for Jenkins currently.
Is there any way I can find out what each thread is for?

lørdag 13. juli 2019 13.02.15 UTC+2 skrev Ivan Fernandez Calvo følgende:
Hi,

When the issue happens, Did you check the number of threads that Jenkins has open? How many file descriptors can your process open (run ulimit -a with the user jenkins)? here you have a good KB about memory and user limit on Jenkins <a href="https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEfQICfugtdoTEmraZRb7IFzUWMtA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEfQICfugtdoTEmraZRb7IFzUWMtA&#39;;return true;">Prepare Jenkins for Support

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/ef859447-1b6c-4fe0-b386-c804692c8258%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Memory issues with Jenkins

Sverre Moe-2
I have analyzed the GC log

https://www.gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTkvMDcvMjgvLS1nYy0yMDE5LTA3LTI2XzE3LTMxLTIyLmxvZy4wLmN1cnJlbnQtLTE1LTI0LTQy&channel=WEB

This may be related to another issue I am having when build agents are getting disconnected.
https://groups.google.com/forum/#!topic/jenkinsci-users/UDyaH-kfqqA

I also have an jstack output analyzed:
https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTkvMDcvMjkvLS1qc3RhY2sudHh0LS05LTE2LTI3

søndag 14. juli 2019 13.48.57 UTC+2 skrev Sverre Moe følgende:
I checked out that link

Tried the ulimit settings
<a href="https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support#bulimitsettingsjustforlinuxos" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support%23bulimitsettingsjustforlinuxos\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGHZMO36L8VcyFlmIvtPNJtR4KcFw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support%23bulimitsettingsjustforlinuxos\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGHZMO36L8VcyFlmIvtPNJtR4KcFw&#39;;return true;">https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support#bulimitsettingsjustforlinuxos

Will try the additional JVM flags suggested in
<a href="https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support#ajavaparameters" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support%23ajavaparameters\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFMbXrQKA5Oh6NC-05Zt_Q-JZy4tQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support%23ajavaparameters\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFMbXrQKA5Oh6NC-05Zt_Q-JZy4tQ&#39;;return true;">https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support#ajavaparameters

søndag 14. juli 2019 13.29.22 UTC+2 skrev Sverre Moe følgende:
jenkins@meoscorebs12:~> ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 80229
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 80229
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited


I have check the number of processes, with "ps aux", but not the number of threads.
Right now, (Jenkins was restarted 2 day ago and few builds have run), there are 264 threads for Jenkins currently.
Is there any way I can find out what each thread is for?

lørdag 13. juli 2019 13.02.15 UTC+2 skrev Ivan Fernandez Calvo følgende:
Hi,

When the issue happens, Did you check the number of threads that Jenkins has open? How many file descriptors can your process open (run ulimit -a with the user jenkins)? here you have a good KB about memory and user limit on Jenkins <a href="https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEfQICfugtdoTEmraZRb7IFzUWMtA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEfQICfugtdoTEmraZRb7IFzUWMtA&#39;;return true;">Prepare Jenkins for Support

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1465add4-191b-46c3-bd08-31816f0cc82b%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Memory issues with Jenkins

guerkan demirci
Hi Sverre Moe,

could you resolve the issue?
I have the same problem with Jenkins.

Best


Am Montag, 29. Juli 2019 11:30:58 UTC+2 schrieb Sverre Moe:
I have analyzed the GC log

<a href="https://www.gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTkvMDcvMjgvLS1nYy0yMDE5LTA3LTI2XzE3LTMxLTIyLmxvZy4wLmN1cnJlbnQtLTE1LTI0LTQy&amp;channel=WEB" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.gceasy.io%2Fmy-gc-report.jsp%3Fp%3Dc2hhcmVkLzIwMTkvMDcvMjgvLS1nYy0yMDE5LTA3LTI2XzE3LTMxLTIyLmxvZy4wLmN1cnJlbnQtLTE1LTI0LTQy%26channel%3DWEB\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGywHPBBoEdaq2eB6Sw7iiQiPbWGA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.gceasy.io%2Fmy-gc-report.jsp%3Fp%3Dc2hhcmVkLzIwMTkvMDcvMjgvLS1nYy0yMDE5LTA3LTI2XzE3LTMxLTIyLmxvZy4wLmN1cnJlbnQtLTE1LTI0LTQy%26channel%3DWEB\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGywHPBBoEdaq2eB6Sw7iiQiPbWGA&#39;;return true;">https://www.gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTkvMDcvMjgvLS1nYy0yMDE5LTA3LTI2XzE3LTMxLTIyLmxvZy4wLmN1cnJlbnQtLTE1LTI0LTQy&channel=WEB

This may be related to another issue I am having when build agents are getting disconnected.
<a href="https://groups.google.com/forum/#!topic/jenkinsci-users/UDyaH-kfqqA" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/forum/#!topic/jenkinsci-users/UDyaH-kfqqA&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/forum/#!topic/jenkinsci-users/UDyaH-kfqqA&#39;;return true;">https://groups.google.com/forum/#!topic/jenkinsci-users/UDyaH-kfqqA

I also have an jstack output analyzed:
<a href="https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTkvMDcvMjkvLS1qc3RhY2sudHh0LS05LTE2LTI3" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Ffastthread.io%2Fmy-thread-report.jsp%3Fp%3Dc2hhcmVkLzIwMTkvMDcvMjkvLS1qc3RhY2sudHh0LS05LTE2LTI3\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHlQoB9XJrQLNkQ_33CAaNs2mbR8g&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Ffastthread.io%2Fmy-thread-report.jsp%3Fp%3Dc2hhcmVkLzIwMTkvMDcvMjkvLS1qc3RhY2sudHh0LS05LTE2LTI3\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHlQoB9XJrQLNkQ_33CAaNs2mbR8g&#39;;return true;">https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTkvMDcvMjkvLS1qc3RhY2sudHh0LS05LTE2LTI3

søndag 14. juli 2019 13.48.57 UTC+2 skrev Sverre Moe følgende:
I checked out that link

Tried the ulimit settings
<a href="https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support#bulimitsettingsjustforlinuxos" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support%23bulimitsettingsjustforlinuxos\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGHZMO36L8VcyFlmIvtPNJtR4KcFw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support%23bulimitsettingsjustforlinuxos\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGHZMO36L8VcyFlmIvtPNJtR4KcFw&#39;;return true;">https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support#bulimitsettingsjustforlinuxos

Will try the additional JVM flags suggested in
<a href="https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support#ajavaparameters" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support%23ajavaparameters\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFMbXrQKA5Oh6NC-05Zt_Q-JZy4tQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support%23ajavaparameters\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFMbXrQKA5Oh6NC-05Zt_Q-JZy4tQ&#39;;return true;">https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support#ajavaparameters

søndag 14. juli 2019 13.29.22 UTC+2 skrev Sverre Moe følgende:
jenkins@meoscorebs12:~> ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 80229
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 80229
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited


I have check the number of processes, with "ps aux", but not the number of threads.
Right now, (Jenkins was restarted 2 day ago and few builds have run), there are 264 threads for Jenkins currently.
Is there any way I can find out what each thread is for?

lørdag 13. juli 2019 13.02.15 UTC+2 skrev Ivan Fernandez Calvo følgende:
Hi,

When the issue happens, Did you check the number of threads that Jenkins has open? How many file descriptors can your process open (run ulimit -a with the user jenkins)? here you have a good KB about memory and user limit on Jenkins <a href="https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEfQICfugtdoTEmraZRb7IFzUWMtA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEfQICfugtdoTEmraZRb7IFzUWMtA&#39;;return true;">Prepare Jenkins for Support

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/4625a630-c1e1-4433-b751-f835755fe15c%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Memory issues with Jenkins

Sverre Moe-2
Have not yet resolved the issue. Not found a solution, nor what the cause actually is.

lørdag 3. august 2019 13.04.10 UTC+2 skrev guerkan demirci følgende:
Hi Sverre Moe,

could you resolve the issue?
I have the same problem with Jenkins.

Best


Am Montag, 29. Juli 2019 11:30:58 UTC+2 schrieb Sverre Moe:
I have analyzed the GC log

<a href="https://www.gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTkvMDcvMjgvLS1nYy0yMDE5LTA3LTI2XzE3LTMxLTIyLmxvZy4wLmN1cnJlbnQtLTE1LTI0LTQy&amp;channel=WEB" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.gceasy.io%2Fmy-gc-report.jsp%3Fp%3Dc2hhcmVkLzIwMTkvMDcvMjgvLS1nYy0yMDE5LTA3LTI2XzE3LTMxLTIyLmxvZy4wLmN1cnJlbnQtLTE1LTI0LTQy%26channel%3DWEB\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGywHPBBoEdaq2eB6Sw7iiQiPbWGA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.gceasy.io%2Fmy-gc-report.jsp%3Fp%3Dc2hhcmVkLzIwMTkvMDcvMjgvLS1nYy0yMDE5LTA3LTI2XzE3LTMxLTIyLmxvZy4wLmN1cnJlbnQtLTE1LTI0LTQy%26channel%3DWEB\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGywHPBBoEdaq2eB6Sw7iiQiPbWGA&#39;;return true;">https://www.gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTkvMDcvMjgvLS1nYy0yMDE5LTA3LTI2XzE3LTMxLTIyLmxvZy4wLmN1cnJlbnQtLTE1LTI0LTQy&channel=WEB

This may be related to another issue I am having when build agents are getting disconnected.
<a href="https://groups.google.com/forum/#!topic/jenkinsci-users/UDyaH-kfqqA" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/forum/#!topic/jenkinsci-users/UDyaH-kfqqA&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/forum/#!topic/jenkinsci-users/UDyaH-kfqqA&#39;;return true;">https://groups.google.com/forum/#!topic/jenkinsci-users/UDyaH-kfqqA

I also have an jstack output analyzed:
<a href="https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTkvMDcvMjkvLS1qc3RhY2sudHh0LS05LTE2LTI3" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Ffastthread.io%2Fmy-thread-report.jsp%3Fp%3Dc2hhcmVkLzIwMTkvMDcvMjkvLS1qc3RhY2sudHh0LS05LTE2LTI3\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHlQoB9XJrQLNkQ_33CAaNs2mbR8g&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Ffastthread.io%2Fmy-thread-report.jsp%3Fp%3Dc2hhcmVkLzIwMTkvMDcvMjkvLS1qc3RhY2sudHh0LS05LTE2LTI3\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHlQoB9XJrQLNkQ_33CAaNs2mbR8g&#39;;return true;">https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTkvMDcvMjkvLS1qc3RhY2sudHh0LS05LTE2LTI3

søndag 14. juli 2019 13.48.57 UTC+2 skrev Sverre Moe følgende:
I checked out that link

Tried the ulimit settings
<a href="https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support#bulimitsettingsjustforlinuxos" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support%23bulimitsettingsjustforlinuxos\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGHZMO36L8VcyFlmIvtPNJtR4KcFw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support%23bulimitsettingsjustforlinuxos\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGHZMO36L8VcyFlmIvtPNJtR4KcFw&#39;;return true;">https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support#bulimitsettingsjustforlinuxos

Will try the additional JVM flags suggested in
<a href="https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support#ajavaparameters" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support%23ajavaparameters\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFMbXrQKA5Oh6NC-05Zt_Q-JZy4tQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support%23ajavaparameters\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFMbXrQKA5Oh6NC-05Zt_Q-JZy4tQ&#39;;return true;">https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support#ajavaparameters

søndag 14. juli 2019 13.29.22 UTC+2 skrev Sverre Moe følgende:
jenkins@meoscorebs12:~> ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 80229
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 80229
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited


I have check the number of processes, with "ps aux", but not the number of threads.
Right now, (Jenkins was restarted 2 day ago and few builds have run), there are 264 threads for Jenkins currently.
Is there any way I can find out what each thread is for?

lørdag 13. juli 2019 13.02.15 UTC+2 skrev Ivan Fernandez Calvo følgende:
Hi,

When the issue happens, Did you check the number of threads that Jenkins has open? How many file descriptors can your process open (run ulimit -a with the user jenkins)? here you have a good KB about memory and user limit on Jenkins <a href="https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEfQICfugtdoTEmraZRb7IFzUWMtA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEfQICfugtdoTEmraZRb7IFzUWMtA&#39;;return true;">Prepare Jenkins for Support

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/6b4b772c-3f8c-4d1f-aafd-94e4eea7298d%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Memory issues with Jenkins

Ivan Fernandez Calvo
if it is the same instance with the agent disconnection errors, your problem is the NFS performance, it blocks tons of threads in IO operations that make agents cannot disconnect or connects blocking thousands of threads, the Jenkins instance works until is collapsed by the number of threads blocked or the amount of memory used by them. As I said in the other email thread rid of the NFS and your problems go away, if it is not possible, make that your NFS performs like a hard drive (+150MB/s) with more than 100 concurrent IO operations on small files for a period of 5-10 min, if you cannot make it that NFS will not work properly with Jenkins.


El lunes, 5 de agosto de 2019, 11:08:27 (UTC+2), Sverre Moe escribió:
Have not yet resolved the issue. Not found a solution, nor what the cause actually is.

lørdag 3. august 2019 13.04.10 UTC+2 skrev guerkan demirci følgende:
Hi Sverre Moe,

could you resolve the issue?
I have the same problem with Jenkins.

Best


Am Montag, 29. Juli 2019 11:30:58 UTC+2 schrieb Sverre Moe:
I have analyzed the GC log

<a href="https://www.gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTkvMDcvMjgvLS1nYy0yMDE5LTA3LTI2XzE3LTMxLTIyLmxvZy4wLmN1cnJlbnQtLTE1LTI0LTQy&amp;channel=WEB" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.gceasy.io%2Fmy-gc-report.jsp%3Fp%3Dc2hhcmVkLzIwMTkvMDcvMjgvLS1nYy0yMDE5LTA3LTI2XzE3LTMxLTIyLmxvZy4wLmN1cnJlbnQtLTE1LTI0LTQy%26channel%3DWEB\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGywHPBBoEdaq2eB6Sw7iiQiPbWGA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.gceasy.io%2Fmy-gc-report.jsp%3Fp%3Dc2hhcmVkLzIwMTkvMDcvMjgvLS1nYy0yMDE5LTA3LTI2XzE3LTMxLTIyLmxvZy4wLmN1cnJlbnQtLTE1LTI0LTQy%26channel%3DWEB\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGywHPBBoEdaq2eB6Sw7iiQiPbWGA&#39;;return true;">https://www.gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTkvMDcvMjgvLS1nYy0yMDE5LTA3LTI2XzE3LTMxLTIyLmxvZy4wLmN1cnJlbnQtLTE1LTI0LTQy&channel=WEB

This may be related to another issue I am having when build agents are getting disconnected.
<a href="https://groups.google.com/forum/#!topic/jenkinsci-users/UDyaH-kfqqA" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/forum/#!topic/jenkinsci-users/UDyaH-kfqqA&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/forum/#!topic/jenkinsci-users/UDyaH-kfqqA&#39;;return true;">https://groups.google.com/forum/#!topic/jenkinsci-users/UDyaH-kfqqA

I also have an jstack output analyzed:
<a href="https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTkvMDcvMjkvLS1qc3RhY2sudHh0LS05LTE2LTI3" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Ffastthread.io%2Fmy-thread-report.jsp%3Fp%3Dc2hhcmVkLzIwMTkvMDcvMjkvLS1qc3RhY2sudHh0LS05LTE2LTI3\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHlQoB9XJrQLNkQ_33CAaNs2mbR8g&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Ffastthread.io%2Fmy-thread-report.jsp%3Fp%3Dc2hhcmVkLzIwMTkvMDcvMjkvLS1qc3RhY2sudHh0LS05LTE2LTI3\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHlQoB9XJrQLNkQ_33CAaNs2mbR8g&#39;;return true;">https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTkvMDcvMjkvLS1qc3RhY2sudHh0LS05LTE2LTI3

søndag 14. juli 2019 13.48.57 UTC+2 skrev Sverre Moe følgende:
I checked out that link

Tried the ulimit settings
<a href="https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support#bulimitsettingsjustforlinuxos" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support%23bulimitsettingsjustforlinuxos\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGHZMO36L8VcyFlmIvtPNJtR4KcFw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support%23bulimitsettingsjustforlinuxos\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGHZMO36L8VcyFlmIvtPNJtR4KcFw&#39;;return true;">https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support#bulimitsettingsjustforlinuxos

Will try the additional JVM flags suggested in
<a href="https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support#ajavaparameters" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support%23ajavaparameters\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFMbXrQKA5Oh6NC-05Zt_Q-JZy4tQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support%23ajavaparameters\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFMbXrQKA5Oh6NC-05Zt_Q-JZy4tQ&#39;;return true;">https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support#ajavaparameters

søndag 14. juli 2019 13.29.22 UTC+2 skrev Sverre Moe følgende:
jenkins@meoscorebs12:~> ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 80229
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 80229
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited


I have check the number of processes, with "ps aux", but not the number of threads.
Right now, (Jenkins was restarted 2 day ago and few builds have run), there are 264 threads for Jenkins currently.
Is there any way I can find out what each thread is for?

lørdag 13. juli 2019 13.02.15 UTC+2 skrev Ivan Fernandez Calvo følgende:
Hi,

When the issue happens, Did you check the number of threads that Jenkins has open? How many file descriptors can your process open (run ulimit -a with the user jenkins)? here you have a good KB about memory and user limit on Jenkins <a href="https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEfQICfugtdoTEmraZRb7IFzUWMtA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEfQICfugtdoTEmraZRb7IFzUWMtA&#39;;return true;">Prepare Jenkins for Support

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/8659bf13-1111-4076-a2c3-32e07d564aca%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Memory issues with Jenkins

guerkan demirci
We don't have NFS for JENKINS_HOME. Master and all clients are using local file system.

Anyway, I can see the stack count going extremely high before going into OutOfMemory and then disconnecting all agents.

But we do have a network cifs folder where all agents are reading and writing to lot's of data.

I don't understand exactly what kind of disc access operation on Jenkins would spawn lot's of threads without waiting for the end of the IO operation.
Might this be caused by a Script in one of our projects Jenkinsfile?

Please, could you share some additional information or any idea for metering additional numbers othen then the number of threads?

This started to happen three weeks ago after an update. Meanwhile we added some more agents into the pool and started using docker for some jobs.

How can I know what's the reason of all the threads other then NFS?

Thank you



Am Montag, 5. August 2019 19:39:47 UTC+2 schrieb Ivan Fernandez Calvo:
if it is the same instance with the agent disconnection errors, your problem is the NFS performance, it blocks tons of threads in IO operations that make agents cannot disconnect or connects blocking thousands of threads, the Jenkins instance works until is collapsed by the number of threads blocked or the amount of memory used by them. As I said in the other email thread rid of the NFS and your problems go away, if it is not possible, make that your NFS performs like a hard drive (+150MB/s) with more than 100 concurrent IO operations on small files for a period of 5-10 min, if you cannot make it that NFS will not work properly with Jenkins.


El lunes, 5 de agosto de 2019, 11:08:27 (UTC+2), Sverre Moe escribió:
Have not yet resolved the issue. Not found a solution, nor what the cause actually is.

lørdag 3. august 2019 13.04.10 UTC+2 skrev guerkan demirci følgende:
Hi Sverre Moe,

could you resolve the issue?
I have the same problem with Jenkins.

Best


Am Montag, 29. Juli 2019 11:30:58 UTC+2 schrieb Sverre Moe:
I have analyzed the GC log

<a href="https://www.gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTkvMDcvMjgvLS1nYy0yMDE5LTA3LTI2XzE3LTMxLTIyLmxvZy4wLmN1cnJlbnQtLTE1LTI0LTQy&amp;channel=WEB" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.gceasy.io%2Fmy-gc-report.jsp%3Fp%3Dc2hhcmVkLzIwMTkvMDcvMjgvLS1nYy0yMDE5LTA3LTI2XzE3LTMxLTIyLmxvZy4wLmN1cnJlbnQtLTE1LTI0LTQy%26channel%3DWEB\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGywHPBBoEdaq2eB6Sw7iiQiPbWGA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.gceasy.io%2Fmy-gc-report.jsp%3Fp%3Dc2hhcmVkLzIwMTkvMDcvMjgvLS1nYy0yMDE5LTA3LTI2XzE3LTMxLTIyLmxvZy4wLmN1cnJlbnQtLTE1LTI0LTQy%26channel%3DWEB\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGywHPBBoEdaq2eB6Sw7iiQiPbWGA&#39;;return true;">https://www.gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTkvMDcvMjgvLS1nYy0yMDE5LTA3LTI2XzE3LTMxLTIyLmxvZy4wLmN1cnJlbnQtLTE1LTI0LTQy&channel=WEB

This may be related to another issue I am having when build agents are getting disconnected.
<a href="https://groups.google.com/forum/#!topic/jenkinsci-users/UDyaH-kfqqA" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/forum/#!topic/jenkinsci-users/UDyaH-kfqqA&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/forum/#!topic/jenkinsci-users/UDyaH-kfqqA&#39;;return true;">https://groups.google.com/forum/#!topic/jenkinsci-users/UDyaH-kfqqA

I also have an jstack output analyzed:
<a href="https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTkvMDcvMjkvLS1qc3RhY2sudHh0LS05LTE2LTI3" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Ffastthread.io%2Fmy-thread-report.jsp%3Fp%3Dc2hhcmVkLzIwMTkvMDcvMjkvLS1qc3RhY2sudHh0LS05LTE2LTI3\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHlQoB9XJrQLNkQ_33CAaNs2mbR8g&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Ffastthread.io%2Fmy-thread-report.jsp%3Fp%3Dc2hhcmVkLzIwMTkvMDcvMjkvLS1qc3RhY2sudHh0LS05LTE2LTI3\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHlQoB9XJrQLNkQ_33CAaNs2mbR8g&#39;;return true;">https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTkvMDcvMjkvLS1qc3RhY2sudHh0LS05LTE2LTI3

søndag 14. juli 2019 13.48.57 UTC+2 skrev Sverre Moe følgende:
I checked out that link

Tried the ulimit settings
<a href="https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support#bulimitsettingsjustforlinuxos" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support%23bulimitsettingsjustforlinuxos\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGHZMO36L8VcyFlmIvtPNJtR4KcFw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support%23bulimitsettingsjustforlinuxos\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGHZMO36L8VcyFlmIvtPNJtR4KcFw&#39;;return true;">https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support#bulimitsettingsjustforlinuxos

Will try the additional JVM flags suggested in
<a href="https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support#ajavaparameters" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support%23ajavaparameters\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFMbXrQKA5Oh6NC-05Zt_Q-JZy4tQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support%23ajavaparameters\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFMbXrQKA5Oh6NC-05Zt_Q-JZy4tQ&#39;;return true;">https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support#ajavaparameters

søndag 14. juli 2019 13.29.22 UTC+2 skrev Sverre Moe følgende:
jenkins@meoscorebs12:~> ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 80229
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 80229
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited


I have check the number of processes, with "ps aux", but not the number of threads.
Right now, (Jenkins was restarted 2 day ago and few builds have run), there are 264 threads for Jenkins currently.
Is there any way I can find out what each thread is for?

lørdag 13. juli 2019 13.02.15 UTC+2 skrev Ivan Fernandez Calvo følgende:
Hi,

When the issue happens, Did you check the number of threads that Jenkins has open? How many file descriptors can your process open (run ulimit -a with the user jenkins)? here you have a good KB about memory and user limit on Jenkins <a href="https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEfQICfugtdoTEmraZRb7IFzUWMtA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F222446987-Prepare-Jenkins-for-Support\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEfQICfugtdoTEmraZRb7IFzUWMtA&#39;;return true;">Prepare Jenkins for Support

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/966e8bae-faf6-45d1-a65e-1aa6130f37ab%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Memory issues with Jenkins

Ivan Fernandez Calvo
Hi,

Severe has another email thread open, I think it is the same Jenkins instance https://groups.google.com/d/msgid/jenkinsci-users/cc2d0bdb-b15f-4bec-a0a3-0562ea8c7df7%40googlegroups.com?utm_medium=email&utm_source=footer. I dunno what happens on your instance but probably it isn’t better that you open another email thread with the description of your issue

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/25a199ff-64cc-46a4-9811-bf0a71bdf0f7%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Memory issues with Jenkins

Sverre Moe-2
Sadly I was mistaken. We do not use NFS for JENKINS_HOME.

We do however use NFS for the location where builds copy the RPM build artifacts.

mandag 5. august 2019 22.17.46 UTC+2 skrev Ivan Fernandez Calvo følgende:
Hi,

Severe has another email thread open, I think it is the same Jenkins instance <a href="https://groups.google.com/d/msgid/jenkinsci-users/cc2d0bdb-b15f-4bec-a0a3-0562ea8c7df7%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/cc2d0bdb-b15f-4bec-a0a3-0562ea8c7df7%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/cc2d0bdb-b15f-4bec-a0a3-0562ea8c7df7%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/cc2d0bdb-b15f-4bec-a0a3-0562ea8c7df7%40googlegroups.com?utm_medium=email&utm_source=footer. I dunno what happens on your instance but probably it isn’t better that you open another email thread with the description of your issue

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/81039869-a11d-47e0-9c8b-7c2b4774da36%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Memory issues with Jenkins

Sverre Moe-2
We got an 30 minute free CloudBees support. It was too short to dig deeper to find the problem, but the person I was talking to (after examining our logs) mentioned what he thought was the problem and gave a suggestion.

We should not use Jenkins master at all for builds (allocated with the node("master") step). We had 15 Executors for Jenkins master.

We could also try to Increase limits of hard nofile and nproc for jenkins user, but the main recomondation was to remove all Executors for Jenkins master.
> /etc/security/limits.conf
jenkins          soft    core            unlimited
jenkins          hard    core            unlimited
jenkins          soft    fsize           unlimited
jenkins          hard    fsize           unlimited
jenkins          soft    nofile          4096
jenkins          hard    nofile          10240 #Was 8192
jenkins          soft    nproc           30654
jenkins          hard    nproc           60654 #Was 30654


To remove Jenkins master Executors will take some time. We use Jenkins master when we publish our build artifacts RPMs to our NFS file storage.
Since our RPM NFS is only attached to the Jenkins master it is not possible at the moment. Unless we can use any other agent, then do a SCP onto our Jenkins master with the RPM artifacts.


We had a few other circumstances where we used Jenkins master. Like checking out a file to determine which build agent to actually use. These I have already changed to use any available build agent instead.

tirsdag 6. august 2019 09.48.50 UTC+2 skrev Sverre Moe følgende:
Sadly I was mistaken. We do not use NFS for JENKINS_HOME.

We do however use NFS for the location where builds copy the RPM build artifacts.

mandag 5. august 2019 22.17.46 UTC+2 skrev Ivan Fernandez Calvo følgende:
Hi,

Severe has another email thread open, I think it is the same Jenkins instance <a href="https://groups.google.com/d/msgid/jenkinsci-users/cc2d0bdb-b15f-4bec-a0a3-0562ea8c7df7%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/cc2d0bdb-b15f-4bec-a0a3-0562ea8c7df7%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/cc2d0bdb-b15f-4bec-a0a3-0562ea8c7df7%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/cc2d0bdb-b15f-4bec-a0a3-0562ea8c7df7%40googlegroups.com?utm_medium=email&utm_source=footer. I dunno what happens on your instance but probably it isn’t better that you open another email thread with the description of your issue

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/3e728790-b2f5-4ae1-a9fe-512a5c912d61%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Memory issues with Jenkins

Devin Nusbaum
I have not read the whole thread in detail, but the “Unable to create new native thread” OutOfMemoryErrors from your original thread where one of the stack traces involves org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.scheduleRetryQueueProcessing looks like it could be related to https://issues.jenkins-ci.org/browse/JENKINS-58684, which is a thread leak caused by the SSE Gateway Plugin. You could try reverting the SSE Gateway Plugin to version 1.17 to see if that helps, although that might reintroduce a different, somewhat rarer memory leak (https://issues.jenkins-ci.org/browse/JENKINS-51057). To test my hypothesis, if you are running SSE Gateway Plugin version 1.19, you can collect thread dumps over time and see if you seem to have a large number of threads named “EventDispatcher.retryProcessor” (unfortunately in version 1.18 and below the threads are automatically named “Timer #n”, which is less useful), which would confirm that you are hitting JENKINS-58684.

The advice to stop building on master is definitely a good idea as well.

On Aug 14, 2019, at 07:11, Sverre Moe <[hidden email]> wrote:

We got an 30 minute free CloudBees support. It was too short to dig deeper to find the problem, but the person I was talking to (after examining our logs) mentioned what he thought was the problem and gave a suggestion.

We should not use Jenkins master at all for builds (allocated with the node("master") step). We had 15 Executors for Jenkins master.

We could also try to Increase limits of hard nofile and nproc for jenkins user, but the main recomondation was to remove all Executors for Jenkins master.
> /etc/security/limits.conf
jenkins          soft    core            unlimited
jenkins          hard    core            unlimited
jenkins          soft    fsize           unlimited
jenkins          hard    fsize           unlimited
jenkins          soft    nofile          4096
jenkins          hard    nofile          10240 #Was 8192
jenkins          soft    nproc           30654
jenkins          hard    nproc           60654 #Was 30654


To remove Jenkins master Executors will take some time. We use Jenkins master when we publish our build artifacts RPMs to our NFS file storage.
Since our RPM NFS is only attached to the Jenkins master it is not possible at the moment. Unless we can use any other agent, then do a SCP onto our Jenkins master with the RPM artifacts.


We had a few other circumstances where we used Jenkins master. Like checking out a file to determine which build agent to actually use. These I have already changed to use any available build agent instead.

tirsdag 6. august 2019 09.48.50 UTC+2 skrev Sverre Moe følgende:
Sadly I was mistaken. We do not use NFS for JENKINS_HOME.

We do however use NFS for the location where builds copy the RPM build artifacts.

mandag 5. august 2019 22.17.46 UTC+2 skrev Ivan Fernandez Calvo følgende:
Hi,

Severe has another email thread open, I think it is the same Jenkins instance https://groups.google.com/d/msgid/jenkinsci-users/cc2d0bdb-b15f-4bec-a0a3-0562ea8c7df7%40googlegroups.com?utm_medium=email&utm_source=footer. I dunno what happens on your instance but probably it isn’t better that you open another email thread with the description of your issue

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/3e728790-b2f5-4ae1-a9fe-512a5c912d61%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/9FACF73D-081E-4DA8-92F0-86027EAC0868%40cloudbees.com.
Reply | Threaded
Open this post in threaded view
|

Re: Memory issues with Jenkins

Félix Belzunce Arcos
Hi Sverre Moe,

I am the person who talked to you this morning :-)

Long term solution is to avoid building on the master to avoid performance issue and the need to increase the number of processes and open files in the machine where the jenkins master is located. Building on the master is also not recommended from a security point of view.

Short term solution would be to increase the number of new processes on this machine + take thread dumps from the master each 10 minutes. For this, you can create a cron freestyle job executed every 10 minutes executing jstack <JENKINS_PID>. When the issue happens, you could take a look at the latest 10 builds with their thread dumps and try to figure out what is actually consuming so many threads on the master.

I hope this helps,


El miércoles, 14 de agosto de 2019, 15:38:17 (UTC+2), Devin Nusbaum escribió:
I have not read the whole thread in detail, but the “Unable to create new native thread” OutOfMemoryErrors from your original thread where one of the stack traces involves org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.scheduleRetryQueueProcessing looks like it could be related to <a href="https://issues.jenkins-ci.org/browse/JENKINS-58684" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fissues.jenkins-ci.org%2Fbrowse%2FJENKINS-58684\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG4SDin_35S6cDlAg40GlM0ueqi0A&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fissues.jenkins-ci.org%2Fbrowse%2FJENKINS-58684\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG4SDin_35S6cDlAg40GlM0ueqi0A&#39;;return true;">https://issues.jenkins-ci.org/browse/JENKINS-58684, which is a thread leak caused by the SSE Gateway Plugin. You could try reverting the SSE Gateway Plugin to version 1.17 to see if that helps, although that might reintroduce a different, somewhat rarer memory leak (<a href="https://issues.jenkins-ci.org/browse/JENKINS-51057" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fissues.jenkins-ci.org%2Fbrowse%2FJENKINS-51057\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGVe4k46XUg0Sy5Hnngnr3WlJ0j8g&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fissues.jenkins-ci.org%2Fbrowse%2FJENKINS-51057\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGVe4k46XUg0Sy5Hnngnr3WlJ0j8g&#39;;return true;">https://issues.jenkins-ci.org/browse/JENKINS-51057). To test my hypothesis, if you are running SSE Gateway Plugin version 1.19, you can collect thread dumps over time and see if you seem to have a large number of threads named “EventDispatcher.retryProcessor” (unfortunately in version 1.18 and below the threads are automatically named “Timer #n”, which is less useful), which would confirm that you are hitting <a href="https://issues.jenkins-ci.org/browse/JENKINS-58684" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fissues.jenkins-ci.org%2Fbrowse%2FJENKINS-58684\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG4SDin_35S6cDlAg40GlM0ueqi0A&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fissues.jenkins-ci.org%2Fbrowse%2FJENKINS-58684\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG4SDin_35S6cDlAg40GlM0ueqi0A&#39;;return true;">JENKINS-58684.

The advice to stop building on master is definitely a good idea as well.

On Aug 14, 2019, at 07:11, Sverre Moe <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="VUlMetXeDAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">sver...@...> wrote:

We got an 30 minute free CloudBees support. It was too short to dig deeper to find the problem, but the person I was talking to (after examining our logs) mentioned what he thought was the problem and gave a suggestion.

We should not use Jenkins master at all for builds (allocated with the node("master") step). We had 15 Executors for Jenkins master.

We could also try to Increase limits of hard nofile and nproc for jenkins user, but the main recomondation was to remove all Executors for Jenkins master.
> /etc/security/limits.conf
jenkins          soft    core            unlimited
jenkins          hard    core            unlimited
jenkins          soft    fsize           unlimited
jenkins          hard    fsize           unlimited
jenkins          soft    nofile          4096
jenkins          hard    nofile          10240 #Was 8192
jenkins          soft    nproc           30654
jenkins          hard    nproc           60654 #Was 30654


To remove Jenkins master Executors will take some time. We use Jenkins master when we publish our build artifacts RPMs to our NFS file storage.
Since our RPM NFS is only attached to the Jenkins master it is not possible at the moment. Unless we can use any other agent, then do a SCP onto our Jenkins master with the RPM artifacts.


We had a few other circumstances where we used Jenkins master. Like checking out a file to determine which build agent to actually use. These I have already changed to use any available build agent instead.

tirsdag 6. august 2019 09.48.50 UTC+2 skrev Sverre Moe følgende:
Sadly I was mistaken. We do not use NFS for JENKINS_HOME.

We do however use NFS for the location where builds copy the RPM build artifacts.

mandag 5. august 2019 22.17.46 UTC+2 skrev Ivan Fernandez Calvo følgende:
Hi,

Severe has another email thread open, I think it is the same Jenkins instance <a href="https://groups.google.com/d/msgid/jenkinsci-users/cc2d0bdb-b15f-4bec-a0a3-0562ea8c7df7%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/cc2d0bdb-b15f-4bec-a0a3-0562ea8c7df7%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/cc2d0bdb-b15f-4bec-a0a3-0562ea8c7df7%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/cc2d0bdb-b15f-4bec-a0a3-0562ea8c7df7%40googlegroups.com?utm_medium=email&utm_source=footer. I dunno what happens on your instance but probably it isn’t better that you open another email thread with the description of your issue

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="VUlMetXeDAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jenkins...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/3e728790-b2f5-4ae1-a9fe-512a5c912d61%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/3e728790-b2f5-4ae1-a9fe-512a5c912d61%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/3e728790-b2f5-4ae1-a9fe-512a5c912d61%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/3e728790-b2f5-4ae1-a9fe-512a5c912d61%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/21a93499-136d-4f69-958e-5160e38d71e1%40googlegroups.com.
12