Severe memory leak in hudson

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Severe memory leak in hudson

Swindells, Thomas
We are suffering from a very severe memory leak in hudson.

I believe it relates to issue http://issues.hudson-ci.org/browse/HUDSON-8061

This issue is affecting us very badly as it appears it is causing a
major memory leak - within about a day the hudson machine becomes
practically unusable due to the memory use.

From what I can tell the master has an export table entry for each
attempted poll. Each entry looks like:
#17604 (ref.2) : hudson.scm.PerJobCredentialStore@672b5b46
java.lang.Exception
at hudson.remoting.ExportTable$Entry.<init>(ExportTable.java:66)
at hudson.remoting.ExportTable.export(ExportTable.java:157)
at hudson.remoting.ExportTable.export(ExportTable.java:144)
at hudson.remoting.Channel.export(Channel.java:522)
at hudson.remoting.Channel.export(Channel.java:517)
at hudson.remoting.Channel.export(Channel.java:488)
at
hudson.scm.PerJobCredentialStore.writeReplace(PerJobCredentialStore.java:
82)
at sun.reflect.GeneratedMethodAccessor121.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:
1032)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
1115)
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:
1518)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:
1483)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:
1400)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
1158)
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:
1518)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:
1483)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:
1400)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
1158)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
at hudson.remoting.UserRequest._serialize(UserRequest.java:151)
at hudson.remoting.UserRequest.serialize(UserRequest.java:160)
at hudson.remoting.UserRequest.<init>(UserRequest.java:62)
at hudson.remoting.Channel.call(Channel.java:628)
at
hudson.scm.SubversionSCM.compareRemoteRevisionWith(SubversionSCM.java:
1078)
at hudson.scm.SCM._compareRemoteRevisionWith(SCM.java:354)
at hudson.scm.SCM.poll(SCM.java:371)
at hudson.model.AbstractProject.poll(AbstractProject.java:1249)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:425)
at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:454)
at hudson.util.SequentialExecutionQueue
$QueueEntry.run(SequentialExecutionQueue.java:118)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:
441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

The dump of the export table is roughly 24mb in size and the heap dump
shows that there 917500 StackTraceElement's referenced via the
exception from approximately 25000 ExportTable entries.

The log shows the same stack trace as raised in the Jira issue.

We are currently on Subversion plugin version 1.20 but I've also tried
the latest (1.22) and that hasn't helped either.

Thanks,

Thomas
Reply | Threaded
Open this post in threaded view
|

RE: Severe memory leak in hudson

Matthew.Webber
There have been a number of reported problems with the Subversion plugin
after version 1.17 (basically, it's broken for quite a few people, in a
variety of ways). I would suggest going back to 1.17 until this is all
sorted out, which since it's bundled with Hudson, means going back to
Hudson 1.384.

You may find that after rolling back, you need to delete ~/.subversion
on your master and slaves, and then re-enter your SVN credentials.

Sorry I don't have any better suggestions.

Matthew
 

> -----Original Message-----
> From: [hidden email] [mailto:hudson-
> [hidden email]] On Behalf Of Thomas Swindells
> Sent: 23 December 2010 16:16
> To: Hudson Users
> Subject: Severe memory leak in hudson
>
> We are suffering from a very severe memory leak in hudson.
>
> I believe it relates to issue http://issues.hudson-
> ci.org/browse/HUDSON-8061
>
> This issue is affecting us very badly as it appears it is causing a
> major memory leak - within about a day the hudson machine becomes
> practically unusable due to the memory use.
>
> From what I can tell the master has an export table entry for each
> attempted poll. Each entry looks like:
> #17604 (ref.2) : hudson.scm.PerJobCredentialStore@672b5b46
> java.lang.Exception
> at hudson.remoting.ExportTable$Entry.<init>(ExportTable.java:66)
> at hudson.remoting.ExportTable.export(ExportTable.java:157)
> at hudson.remoting.ExportTable.export(ExportTable.java:144)
> at hudson.remoting.Channel.export(Channel.java:522)
> at hudson.remoting.Channel.export(Channel.java:517)
> at hudson.remoting.Channel.export(Channel.java:488)
> at
>
hudson.scm.PerJobCredentialStore.writeReplace(PerJobCredentialStore.jav
> a:
> 82)
> at sun.reflect.GeneratedMethodAccessor121.invoke(Unknown Source)
> at
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
> rImpl.java:
> 25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:

> 1032)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
> 1115)
> at
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:
> 1518)
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:
> 1483)
> at
>
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:

> 1400)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
> 1158)
> at
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:
> 1518)
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:
> 1483)
> at
>
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:

> 1400)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
> 1158)
> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
> at hudson.remoting.UserRequest._serialize(UserRequest.java:151)
> at hudson.remoting.UserRequest.serialize(UserRequest.java:160)
> at hudson.remoting.UserRequest.<init>(UserRequest.java:62)
> at hudson.remoting.Channel.call(Channel.java:628)
> at
> hudson.scm.SubversionSCM.compareRemoteRevisionWith(SubversionSCM.java:
> 1078)
> at hudson.scm.SCM._compareRemoteRevisionWith(SCM.java:354)
> at hudson.scm.SCM.poll(SCM.java:371)
> at hudson.model.AbstractProject.poll(AbstractProject.java:1249)
> at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:425)
> at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:454)
> at hudson.util.SequentialExecutionQueue
> $QueueEntry.run(SequentialExecutionQueue.java:118)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:
> 441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor
> $Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor
> $Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> The dump of the export table is roughly 24mb in size and the heap dump
> shows that there 917500 StackTraceElement's referenced via the
> exception from approximately 25000 ExportTable entries.
>
> The log shows the same stack trace as raised in the Jira issue.
>
> We are currently on Subversion plugin version 1.20 but I've also tried
> the latest (1.22) and that hasn't helped either.
>
> Thanks,
>
> Thomas

--
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom




Reply | Threaded
Open this post in threaded view
|

RE: Severe memory leak in hudson

Swindells, Thomas
What's the easiest way to rollback to a specific version without breaking everything in the process?

> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On
> Behalf Of [hidden email]
> Sent: 23 December 2010 16:23
> To: [hidden email]
> Subject: RE: Severe memory leak in hudson
>
> There have been a number of reported problems with the Subversion plugin
> after version 1.17 (basically, it's broken for quite a few people, in a
> variety of ways). I would suggest going back to 1.17 until this is all
> sorted out, which since it's bundled with Hudson, means going back to
> Hudson 1.384.
>
> You may find that after rolling back, you need to delete ~/.subversion
> on your master and slaves, and then re-enter your SVN credentials.
>
> Sorry I don't have any better suggestions.
>
> Matthew
>
> > -----Original Message-----
> > From: [hidden email] [mailto:hudson-
> > [hidden email]] On Behalf Of Thomas Swindells
> > Sent: 23 December 2010 16:16
> > To: Hudson Users
> > Subject: Severe memory leak in hudson
> >
> > We are suffering from a very severe memory leak in hudson.
> >
> > I believe it relates to issue http://issues.hudson-
> > ci.org/browse/HUDSON-8061
> >
> > This issue is affecting us very badly as it appears it is causing a
> > major memory leak - within about a day the hudson machine becomes
> > practically unusable due to the memory use.
> >
> > From what I can tell the master has an export table entry for each
> > attempted poll. Each entry looks like:
> > #17604 (ref.2) : hudson.scm.PerJobCredentialStore@672b5b46
> > java.lang.Exception
> > at hudson.remoting.ExportTable$Entry.<init>(ExportTable.java:66)
> > at hudson.remoting.ExportTable.export(ExportTable.java:157)
> > at hudson.remoting.ExportTable.export(ExportTable.java:144)
> > at hudson.remoting.Channel.export(Channel.java:522)
> > at hudson.remoting.Channel.export(Channel.java:517)
> > at hudson.remoting.Channel.export(Channel.java:488)
> > at
> >
> hudson.scm.PerJobCredentialStore.writeReplace(PerJobCredentialStore.jav
> > a:
> > 82)
> > at sun.reflect.GeneratedMethodAccessor121.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
> > rImpl.java:
> > 25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:
> > 1032)
> > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
> > 1115)
> > at
> > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:
> > 1518)
> > at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:
> > 1483)
> > at
> >
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:
> > 1400)
> > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
> > 1158)
> > at
> > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:
> > 1518)
> > at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:
> > 1483)
> > at
> >
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:
> > 1400)
> > at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
> > 1158)
> > at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
> > at hudson.remoting.UserRequest._serialize(UserRequest.java:151)
> > at hudson.remoting.UserRequest.serialize(UserRequest.java:160)
> > at hudson.remoting.UserRequest.<init>(UserRequest.java:62)
> > at hudson.remoting.Channel.call(Channel.java:628)
> > at
> > hudson.scm.SubversionSCM.compareRemoteRevisionWith(SubversionSCM.java:
> > 1078)
> > at hudson.scm.SCM._compareRemoteRevisionWith(SCM.java:354)
> > at hudson.scm.SCM.poll(SCM.java:371)
> > at hudson.model.AbstractProject.poll(AbstractProject.java:1249)
> > at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:425)
> > at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:454)
> > at hudson.util.SequentialExecutionQueue
> > $QueueEntry.run(SequentialExecutionQueue.java:118)
> > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:
> > 441)
> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > at java.util.concurrent.ThreadPoolExecutor
> > $Worker.runTask(ThreadPoolExecutor.java:886)
> > at java.util.concurrent.ThreadPoolExecutor
> > $Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:662)
> >
> > The dump of the export table is roughly 24mb in size and the heap dump
> > shows that there 917500 StackTraceElement's referenced via the
> > exception from approximately 25000 ExportTable entries.
> >
> > The log shows the same stack trace as raised in the Jira issue.
> >
> > We are currently on Subversion plugin version 1.20 but I've also tried
> > the latest (1.22) and that hasn't helped either.
> >
> > Thanks,
> >
> > Thomas
>
> --
> This e-mail and any attachments may contain confidential, copyright and or
> privileged material, and are for the use of the intended addressee only. If
> you are not the intended addressee or an authorised recipient of the addressee
> please notify us of receipt by returning the e-mail and do not use, copy,
> retain, distribute or disclose the information in or attached to the e-mail.
> Any opinions expressed within this e-mail are those of the individual and not
> necessarily of Diamond Light Source Ltd.
> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments
> are free from viruses and we cannot accept liability for any damage which you
> may sustain as a result of software viruses which may be transmitted in or
> with the message.
> Diamond Light Source Limited (company no. 4375679). Registered in England and
> Wales with its registered office at Diamond House, Harwell Science and
> Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>
>
>



**************************************************************************************
This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the [hidden email] and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by NDS for employment and security purposes. To protect the environment please do not print this e-mail unless necessary.

NDS Limited. Registered Office: One London Road, Staines, Middlesex, TW18 4EX, United Kingdom. A company registered in England and Wales. Registered no. 3080780. VAT no. GB 603 8808 40-00
**************************************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: RE: Severe memory leak in hudson

Douglas Borg
The easiest way I have found to revert plugins is to go grab the old version of the plugin here (I grabbed 1.17):


Then rename the subversion.hpi to subversion.bak and copy it into your ${HUDSON_HOME}/plugins directory overwriting the old subversion.bak.

Then go to  your Hudson Instance and Login. Go to: Manage Hudson >> Manage Plugins >> Installed Plugins. Click the "Unpin" next to the Subversion plugin and then click the "Downgrade to 1.17" button and let Hudson restart itself. It may make sense to reboot the entire server as well - we noticed our memory usage was really high.

I just tried the above steps on a 1.391 system and it seems to be working; I have not had any SVN related issues so far. I will post again if I run into any further issues. FYI I got the 1.391 war file by extracting it out of the redhat rpm:

-Doug Borg
Reply | Threaded
Open this post in threaded view
|

Re: RE: Severe memory leak in hudson

Douglas Borg
Looks like I misunderstood the whole "Pinned Plugins" thing: http://wiki.hudson-ci.org/display/HUDSON/Pinned+Plugins

After you revert, you should "pin" the 1.17 plugin by creating an empty file called: subversion.hpi.pinned so that Hudson doesn't over-write your downgraded plugin the next time the appserver is restarted.