Problems with memory usage on 1.560

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Problems with memory usage on 1.560

Tobias Hieta
Hello,

I installed 1.560 the other day and I have seen a few of the following exceptions when building my (rather large) matrix builds:

00:12:32.091 FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination
00:12:32.130 hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination
00:12:32.810 at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
00:12:32.810 at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
00:12:32.810 at hudson.remoting.Request.call(Request.java:174)
00:12:32.810 at hudson.remoting.Channel.call(Channel.java:738)
00:12:32.810 at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:168)
00:12:32.810 at com.sun.proxy.$Proxy73.join(Unknown Source)
00:12:32.810 at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:951)
00:12:32.810 at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:137)
00:12:32.810 at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97)
00:12:32.810 at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
00:12:32.810 at org.jenkins_ci.plugins.run_condition.BuildStepRunner$2.run(BuildStepRunner.java:110)
00:12:32.810 at org.jenkins_ci.plugins.run_condition.BuildStepRunner$Fail.conditionalRun(BuildStepRunner.java:154)
00:12:32.810 at org.jenkins_ci.plugins.run_condition.BuildStepRunner.perform(BuildStepRunner.java:105)
00:12:32.810 at org.jenkinsci.plugins.conditionalbuildstep.singlestep.SingleConditionalBuilder.perform(SingleConditionalBuilder.java:108)
00:12:32.810 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
00:12:32.810 at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:740)
00:12:32.810 at hudson.model.Build$BuildExecution.build(Build.java:198)
00:12:32.810 at hudson.model.Build$BuildExecution.doRun(Build.java:159)
00:12:32.810 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:519)
00:12:32.810 at hudson.model.Run.execute(Run.java:1703)
00:12:32.810 at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
00:12:32.810 at hudson.model.ResourceController.execute(ResourceController.java:88)
00:12:32.810 at hudson.model.Executor.run(Executor.java:231)
00:12:32.810 Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination
00:12:32.810 at hudson.remoting.Request.abort(Request.java:299)
00:12:32.810 at hudson.remoting.Channel.terminate(Channel.java:801)
00:12:32.810 at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:76)
00:12:32.810 Caused by: java.io.IOException: Unexpected reader termination
00:12:32.810 ... 1 more
00:12:32.810 Caused by: java.lang.OutOfMemoryError: Java heap space
00:12:32.810 at java.util.Arrays.copyOf(Arrays.java:2271)
00:12:32.810 at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
00:12:32.810 at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
00:12:32.810 at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
00:12:32.810 at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87)
00:12:32.810 at hudson.remoting.ChunkedInputStream.read(ChunkedInputStream.java:46)
00:12:32.810 at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:88)
00:12:32.810 at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33)
00:12:32.810 at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
00:12:32.810 at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

I have tried to up the heap space both on the master node and on the slaves. I have currently given the master over 8GB of Heap Space and the Slaves around the same, but this still happens from time to time. First question, is this running out of memory on the master node or on the slaves? Secondly, should a matrix build with ~15 builds really use this much memory? Should I just up the heap space on the master node until it goes away or is there some code that have recently changed the memory usage?

Thanks,
Tobias

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Problems with memory usage on 1.560

Larry Shatzer, Jr.
I've run into issues with the Clone Workspace plugin. Once I rolled back to 1.599, it went away. Have you tried to downgrade back to 1.559?

I've noticed that 1.559 uses Remoting 2.37, and 1.560 uses 2.39, and master is currently on 2.40. I've not tried running this with what is on master yet. I've been meaning to ping the IRC channel about this.

Here's my stacktrace:

Archiving workspace
ERROR: Publisher hudson.plugins.cloneworkspace.CloneWorkspacePublisher aborted due to exception
hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination
	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
	at hudson.remoting.Request.call(Request.java:174)
	at hudson.remoting.Channel.call(Channel.java:738)
	at hudson.FilePath.act(FilePath.java:909)
	at hudson.FilePath.act(FilePath.java:893)
	at hudson.FilePath.archive(FilePath.java:423)
	at hudson.FilePath.tar(FilePath.java:2047)
	at hudson.plugins.cloneworkspace.CloneWorkspacePublisher.snapshot(CloneWorkspacePublisher.java:231)
	at hudson.plugins.cloneworkspace.CloneWorkspacePublisher.perform(CloneWorkspacePublisher.java:178)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:740)
	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:710)
	at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.post2(MavenModuleSetBuild.java:1036)
	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:659)
	at hudson.model.Run.execute(Run.java:1728)
	at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:525)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:231)
Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination
	at hudson.remoting.Request.abort(Request.java:299)
	at hudson.remoting.Channel.terminate(Channel.java:801)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:76)
Caused by: java.io.IOException: Unexpected reader termination
	... 1 more
Caused by: java.lang.OutOfMemoryError: Java heap space


On Wed, Apr 23, 2014 at 1:00 AM, Tobias Hieta <[hidden email]> wrote:
Hello,

I installed 1.560 the other day and I have seen a few of the following exceptions when building my (rather large) matrix builds:

00:12:32.091 FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination
00:12:32.130 hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination
00:12:32.810 at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
00:12:32.810 at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
00:12:32.810 at hudson.remoting.Request.call(Request.java:174)
00:12:32.810 at hudson.remoting.Channel.call(Channel.java:738)
00:12:32.810 at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:168)
00:12:32.810 at com.sun.proxy.$Proxy73.join(Unknown Source)
00:12:32.810 at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:951)
00:12:32.810 at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:137)
00:12:32.810 at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97)
00:12:32.810 at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
00:12:32.810 at org.jenkins_ci.plugins.run_condition.BuildStepRunner$2.run(BuildStepRunner.java:110)
00:12:32.810 at org.jenkins_ci.plugins.run_condition.BuildStepRunner$Fail.conditionalRun(BuildStepRunner.java:154)
00:12:32.810 at org.jenkins_ci.plugins.run_condition.BuildStepRunner.perform(BuildStepRunner.java:105)
00:12:32.810 at org.jenkinsci.plugins.conditionalbuildstep.singlestep.SingleConditionalBuilder.perform(SingleConditionalBuilder.java:108)
00:12:32.810 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
00:12:32.810 at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:740)
00:12:32.810 at hudson.model.Build$BuildExecution.build(Build.java:198)
00:12:32.810 at hudson.model.Build$BuildExecution.doRun(Build.java:159)
00:12:32.810 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:519)
00:12:32.810 at hudson.model.Run.execute(Run.java:1703)
00:12:32.810 at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
00:12:32.810 at hudson.model.ResourceController.execute(ResourceController.java:88)
00:12:32.810 at hudson.model.Executor.run(Executor.java:231)
00:12:32.810 Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected reader termination
00:12:32.810 at hudson.remoting.Request.abort(Request.java:299)
00:12:32.810 at hudson.remoting.Channel.terminate(Channel.java:801)
00:12:32.810 at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:76)
00:12:32.810 Caused by: java.io.IOException: Unexpected reader termination
00:12:32.810 ... 1 more
00:12:32.810 Caused by: java.lang.OutOfMemoryError: Java heap space
00:12:32.810 at java.util.Arrays.copyOf(Arrays.java:2271)
00:12:32.810 at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
00:12:32.810 at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
00:12:32.810 at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
00:12:32.810 at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:87)
00:12:32.810 at hudson.remoting.ChunkedInputStream.read(ChunkedInputStream.java:46)
00:12:32.810 at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:88)
00:12:32.810 at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33)
00:12:32.810 at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
00:12:32.810 at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

I have tried to up the heap space both on the master node and on the slaves. I have currently given the master over 8GB of Heap Space and the Slaves around the same, but this still happens from time to time. First question, is this running out of memory on the master node or on the slaves? Secondly, should a matrix build with ~15 builds really use this much memory? Should I just up the heap space on the master node until it goes away or is there some code that have recently changed the memory usage?

Thanks,
Tobias

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Problems with memory usage on 1.560

Morten Andersen
In reply to this post by Tobias Hieta
Hi Tobias,

Could it be this issue that you are seing: [JENKINS-22734] OutOfMemory on file transfer

Best Regards
Morten

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Problems with memory usage on 1.560

Tobias Hieta
Yes this seems to be exactly my issue. I reverted back to a earlier version of Jenkins and it seems to build fine now.


On Fri, Apr 25, 2014 at 9:50 AM, Morten Andersen <[hidden email]> wrote:
Hi Tobias,

Could it be this issue that you are seing: [JENKINS-22734] OutOfMemory on file transfer

Best Regards
Morten

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/QSZu57CmmfA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.



--
Tobias Hieta
http://about.me/thieta

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.