How to resolve hung subversion polling threads?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

How to resolve hung subversion polling threads?

Rob Oxspring
Hi all,

What am I supposed to do when presented with the following warning?
"There are more SCM polling activities scheduled than handled, so the
threads are not keeping up with the demands. Check if your polling is
hanging, and/or increase the number of threads if necessary."

Clearly in my case there are two threads that have hung (23hrs and 1day
12hrs respectively) but what is it that I'm expected to do? there is no
option to cancel the presented threads that I can find and I don't see
any place to implement sensible timeouts to avoid the problem in the
future.  For the moment I'm left waiting for a chance to restart Hudson
in the hope that that'll resolve problems for now.

Am I missing something?

Thanks,

Rob

P.S. What would cause it to hang anyway?
  Rob Oxspring
  [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: How to resolve hung subversion polling threads?

carlq
I have the exact same problem. Often it seems when slaves get wonky, threads get left hanging around. The only way I've found to clean them up is to just restart Hudson every couple of nights.

I wonder if something could be done with a little groovy script in the scripting console?


On Thu, Feb 25, 2010 at 7:11 AM, Rob Oxspring <[hidden email]> wrote:
Hi all,

What am I supposed to do when presented with the following warning?
"There are more SCM polling activities scheduled than handled, so the
threads are not keeping up with the demands. Check if your polling is
hanging, and/or increase the number of threads if necessary."

Clearly in my case there are two threads that have hung (23hrs and 1day
12hrs respectively) but what is it that I'm expected to do? there is no
option to cancel the presented threads that I can find and I don't see
any place to implement sensible timeouts to avoid the problem in the
future.  For the moment I'm left waiting for a chance to restart Hudson
in the hope that that'll resolve problems for now.

Am I missing something?

Thanks,

Rob

P.S. What would cause it to hang anyway?
 Rob Oxspring
 [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: How to resolve hung subversion polling threads?

Mike Dillon-5
We've seen this problem on a few servers here at Yahoo! and ended up
patching the Subversion plugin internally to revert all polling to the
master as a stop-gap. A number of us spent a couple days trying to
diagnose the root cause and were not able to do so.

-md

begin Carl Quinn quotation:

> I have the exact same problem. Often it seems when slaves get wonky, threads
> get left hanging around. The only way I've found to clean them up is to just
> restart Hudson every couple of nights.
>
> I wonder if something could be done with a little groovy script in the
> scripting console?
>
>
> On Thu, Feb 25, 2010 at 7:11 AM, Rob Oxspring <[hidden email]>wrote:
>
> > Hi all,
> >
> > What am I supposed to do when presented with the following warning?
> > "There are more SCM polling activities scheduled than handled, so the
> > threads are not keeping up with the demands. Check if your polling is
> > hanging, and/or increase the number of threads if necessary."
> >
> > Clearly in my case there are two threads that have hung (23hrs and 1day
> > 12hrs respectively) but what is it that I'm expected to do? there is no
> > option to cancel the presented threads that I can find and I don't see
> > any place to implement sensible timeouts to avoid the problem in the
> > future.  For the moment I'm left waiting for a chance to restart Hudson
> > in the hope that that'll resolve problems for now.
> >
> > Am I missing something?
> >
> > Thanks,
> >
> > Rob
> >
> > P.S. What would cause it to hang anyway?
> >  Rob Oxspring
> >  [hidden email]
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: How to resolve hung subversion polling threads?

Kohsuke Kawaguchi-2

If you see the problem, can you get the thread dump so that we know
where it's hanging?

SVNKit is supposed to set socket time out, but in the past we found a
few bugs where it didn't.


Mike Dillon wrote:

> We've seen this problem on a few servers here at Yahoo! and ended up
> patching the Subversion plugin internally to revert all polling to the
> master as a stop-gap. A number of us spent a couple days trying to
> diagnose the root cause and were not able to do so.
>
> -md
>
> begin Carl Quinn quotation:
>> I have the exact same problem. Often it seems when slaves get wonky, threads
>> get left hanging around. The only way I've found to clean them up is to just
>> restart Hudson every couple of nights.
>>
>> I wonder if something could be done with a little groovy script in the
>> scripting console?
>>
>>
>> On Thu, Feb 25, 2010 at 7:11 AM, Rob Oxspring <[hidden email]>wrote:
>>
>> > Hi all,
>> >
>> > What am I supposed to do when presented with the following warning?
>> > "There are more SCM polling activities scheduled than handled, so the
>> > threads are not keeping up with the demands. Check if your polling is
>> > hanging, and/or increase the number of threads if necessary."
>> >
>> > Clearly in my case there are two threads that have hung (23hrs and 1day
>> > 12hrs respectively) but what is it that I'm expected to do? there is no
>> > option to cancel the presented threads that I can find and I don't see
>> > any place to implement sensible timeouts to avoid the problem in the
>> > future.  For the moment I'm left waiting for a chance to restart Hudson
>> > in the hope that that'll resolve problems for now.
>> >
>> > Am I missing something?
>> >
>> > Thanks,
>> >
>> > Rob
>> >
>> > P.S. What would cause it to hang anyway?
>> >  Rob Oxspring
>> >  [hidden email]
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: [hidden email]
>> > For additional commands, e-mail: [hidden email]
>> >
>> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
Kohsuke Kawaguchi
Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: How to resolve hung subversion polling threads?

Mike Dillon-5
I'm heading out for the day soon, but I think we've got some thread
dumps around for that one somewhere. I'll take a look tomorrow.

-md

begin Kohsuke Kawaguchi quotation:

>
> If you see the problem, can you get the thread dump so that we know
> where it's hanging?
>
> SVNKit is supposed to set socket time out, but in the past we found a
> few bugs where it didn't.
>
>
> Mike Dillon wrote:
> > We've seen this problem on a few servers here at Yahoo! and ended up
> > patching the Subversion plugin internally to revert all polling to the
> > master as a stop-gap. A number of us spent a couple days trying to
> > diagnose the root cause and were not able to do so.
> >
> > -md
> >
> > begin Carl Quinn quotation:
> >> I have the exact same problem. Often it seems when slaves get wonky, threads
> >> get left hanging around. The only way I've found to clean them up is to just
> >> restart Hudson every couple of nights.
> >>
> >> I wonder if something could be done with a little groovy script in the
> >> scripting console?
> >>
> >>
> >> On Thu, Feb 25, 2010 at 7:11 AM, Rob Oxspring <[hidden email]>wrote:
> >>
> >> > Hi all,
> >> >
> >> > What am I supposed to do when presented with the following warning?
> >> > "There are more SCM polling activities scheduled than handled, so the
> >> > threads are not keeping up with the demands. Check if your polling is
> >> > hanging, and/or increase the number of threads if necessary."
> >> >
> >> > Clearly in my case there are two threads that have hung (23hrs and 1day
> >> > 12hrs respectively) but what is it that I'm expected to do? there is no
> >> > option to cancel the presented threads that I can find and I don't see
> >> > any place to implement sensible timeouts to avoid the problem in the
> >> > future.  For the moment I'm left waiting for a chance to restart Hudson
> >> > in the hope that that'll resolve problems for now.
> >> >
> >> > Am I missing something?
> >> >
> >> > Thanks,
> >> >
> >> > Rob
> >> >
> >> > P.S. What would cause it to hang anyway?
> >> >  Rob Oxspring
> >> >  [hidden email]
> >> >
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: [hidden email]
> >> > For additional commands, e-mail: [hidden email]
> >> >
> >> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
>
> --
> Kohsuke Kawaguchi
> Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: How to resolve hung subversion polling threads?

carlq
In reply to this post by Kohsuke Kawaguchi-2
My polling is with Perforce, and all the stuck threads look like this: (thanks to the JavaMelody monitoring plugin!)

SCM polling for hudson.model.FreeStyleProject@12129d7[WE-APP-emailListProcessor] / waiting for hudson.remoting.Channel@15eb5a:hudson02
java.lang.Object.wait(Native Method) 
hudson.remoting.Request.call(Request.java:122) 
hudson.remoting.Channel.call(Channel.java:551) 
hudson.FilePath.act(FilePath.java:683) 
hudson.FilePath.act(FilePath.java:676) 
hudson.FilePath.exists(FilePath.java:944) 
hudson.model.AbstractProject.pollSCMChanges(AbstractProject.java:1042) 
hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:319) 
hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:346) 
hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) 
java.util.concurrent.FutureTask.run(FutureTask.java:138) 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) 
java.lang.Thread.run(Thread.java:619) 


On Thu, Feb 25, 2010 at 4:46 PM, Kohsuke Kawaguchi <[hidden email]> wrote:

If you see the problem, can you get the thread dump so that we know
where it's hanging?

SVNKit is supposed to set socket time out, but in the past we found a
few bugs where it didn't.


Mike Dillon wrote:
> We've seen this problem on a few servers here at Yahoo! and ended up
> patching the Subversion plugin internally to revert all polling to the
> master as a stop-gap. A number of us spent a couple days trying to
> diagnose the root cause and were not able to do so.
>
> -md
>
> begin Carl Quinn quotation:
>> I have the exact same problem. Often it seems when slaves get wonky, threads
>> get left hanging around. The only way I've found to clean them up is to just
>> restart Hudson every couple of nights.
>>
>> I wonder if something could be done with a little groovy script in the
>> scripting console?
>>
>>
>> On Thu, Feb 25, 2010 at 7:11 AM, Rob Oxspring <[hidden email]>wrote:
>>
>> > Hi all,
>> >
>> > What am I supposed to do when presented with the following warning?
>> > "There are more SCM polling activities scheduled than handled, so the
>> > threads are not keeping up with the demands. Check if your polling is
>> > hanging, and/or increase the number of threads if necessary."
>> >
>> > Clearly in my case there are two threads that have hung (23hrs and 1day
>> > 12hrs respectively) but what is it that I'm expected to do? there is no
>> > option to cancel the presented threads that I can find and I don't see
>> > any place to implement sensible timeouts to avoid the problem in the
>> > future.  For the moment I'm left waiting for a chance to restart Hudson
>> > in the hope that that'll resolve problems for now.
>> >
>> > Am I missing something?
>> >
>> > Thanks,
>> >
>> > Rob
>> >
>> > P.S. What would cause it to hang anyway?
>> >  Rob Oxspring
>> >  [hidden email]
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: [hidden email]
>> > For additional commands, e-mail: [hidden email]
>> >
>> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
Kohsuke Kawaguchi
Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: How to resolve hung subversion polling threads?

Dean Yu
Re: How to resolve hung subversion polling threads? I think the issue is not with any particular SCM implementation, but with problems in the underlying remoting code. I filed a bug for it a couple of weeks ago:

http://issues.hudson-ci.org/browse/HUDSON-5413

  -- Dean


On 2/25/10 5:10 PM, "Carl Quinn" <carl.quinn@...> wrote:

My polling is with Perforce, and all the stuck threads look like this: (thanks to the JavaMelody monitoring plugin!)

SCM polling for hudson.model.FreeStyleProject@12129d7[WE-APP-emailListProcessor] / waiting for hudson.remoting.Channel@15eb5a:hudson02
java.lang.Object.wait(Native Method) 
hudson.remoting.Request.call(Request.java:122) 
hudson.remoting.Channel.call(Channel.java:551) 
hudson.FilePath.act(FilePath.java:683) 
hudson.FilePath.act(FilePath.java:676) 
hudson.FilePath.exists(FilePath.java:944) 
hudson.model.AbstractProject.pollSCMChanges(AbstractProject.java:1042) 
hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:319) 
hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:346) 
hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) 
java.util.concurrent.FutureTask.run(FutureTask.java:138) 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) 
java.lang.Thread.run(Thread.java:619) 


On Thu, Feb 25, 2010 at 4:46 PM, Kohsuke Kawaguchi <Kohsuke.Kawaguchi@...> wrote:

If you see the problem, can you get the thread dump so that we know
where it's hanging?

SVNKit is supposed to set socket time out, but in the past we found a
few bugs where it didn't.


Mike Dillon wrote:
> We've seen this problem on a few servers here at Yahoo! and ended up
> patching the Subversion plugin internally to revert all polling to the
> master as a stop-gap. A number of us spent a couple days trying to
> diagnose the root cause and were not able to do so.
>
> -md
>
> begin Carl Quinn quotation:
>> I have the exact same problem. Often it seems when slaves get wonky, threads
>> get left hanging around. The only way I've found to clean them up is to just
>> restart Hudson every couple of nights.
>>
>> I wonder if something could be done with a little groovy script in the
>> scripting console?
>>
>>
>> On Thu, Feb 25, 2010 at 7:11 AM, Rob Oxspring <roxspring@...>wrote:
>>
>> > Hi all,
>> >
>> > What am I supposed to do when presented with the following warning?
>> > "There are more SCM polling activities scheduled than handled, so the
>> > threads are not keeping up with the demands. Check if your polling is
>> > hanging, and/or increase the number of threads if necessary."
>> >
>> > Clearly in my case there are two threads that have hung (23hrs and 1day
>> > 12hrs respectively) but what is it that I'm expected to do? there is no
>> > option to cancel the presented threads that I can find and I don't see
>> > any place to implement sensible timeouts to avoid the problem in the
>> > future.  For the moment I'm left waiting for a chance to restart Hudson
>> > in the hope that that'll resolve problems for now.
>> >
>> > Am I missing something?
>> >
>> > Thanks,
>> >
>> > Rob
>> >
>> > P.S. What would cause it to hang anyway?
>> >  Rob Oxspring
>> >  roxspring@...
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@...
>> > For additional commands, e-mail: dev-help@...
>> >
>> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@...
> For additional commands, e-mail: dev-help@...
>
>


--
Kohsuke Kawaguchi
Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@...
For additional commands, e-mail: dev-help@...



Reply | Threaded
Open this post in threaded view
|

Re: How to resolve hung subversion polling threads?

Mike Dillon-5
In reply to this post by Kohsuke Kawaguchi-2
I've added a slightly vetted thread dump to the JIRA issue Dean
mentioned in his message (HUDSON-5413). I got it from an internal ticket
from a few weeks back before we worked around this.

-md

begin Kohsuke Kawaguchi quotation:

>
> If you see the problem, can you get the thread dump so that we know
> where it's hanging?
>
> SVNKit is supposed to set socket time out, but in the past we found a
> few bugs where it didn't.
>
>
> Mike Dillon wrote:
> > We've seen this problem on a few servers here at Yahoo! and ended up
> > patching the Subversion plugin internally to revert all polling to the
> > master as a stop-gap. A number of us spent a couple days trying to
> > diagnose the root cause and were not able to do so.
> >
> > -md
> >
> > begin Carl Quinn quotation:
> >> I have the exact same problem. Often it seems when slaves get wonky, threads
> >> get left hanging around. The only way I've found to clean them up is to just
> >> restart Hudson every couple of nights.
> >>
> >> I wonder if something could be done with a little groovy script in the
> >> scripting console?
> >>
> >>
> >> On Thu, Feb 25, 2010 at 7:11 AM, Rob Oxspring <[hidden email]>wrote:
> >>
> >> > Hi all,
> >> >
> >> > What am I supposed to do when presented with the following warning?
> >> > "There are more SCM polling activities scheduled than handled, so the
> >> > threads are not keeping up with the demands. Check if your polling is
> >> > hanging, and/or increase the number of threads if necessary."
> >> >
> >> > Clearly in my case there are two threads that have hung (23hrs and 1day
> >> > 12hrs respectively) but what is it that I'm expected to do? there is no
> >> > option to cancel the presented threads that I can find and I don't see
> >> > any place to implement sensible timeouts to avoid the problem in the
> >> > future.  For the moment I'm left waiting for a chance to restart Hudson
> >> > in the hope that that'll resolve problems for now.
> >> >
> >> > Am I missing something?
> >> >
> >> > Thanks,
> >> >
> >> > Rob
> >> >
> >> > P.S. What would cause it to hang anyway?
> >> >  Rob Oxspring
> >> >  [hidden email]
> >> >
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: [hidden email]
> >> > For additional commands, e-mail: [hidden email]
> >> >
> >> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
>
> --
> Kohsuke Kawaguchi
> Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]