Subversion polling stops/stalls eventually

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Subversion polling stops/stalls eventually

Rhett Sutphin
Hi,

I've got another intermittent issue that I haven't been able to  
correlate with anything in particular.  All of my jobs are using  
subversion polling to kick off builds.  One job will occasionally  
stop polling.  Once this happens, the only way I can get it to start  
polling again is to restart tomcat.  After the polling stalls, manual  
builds still work fine (including svn updates).  A haven't kept  
detailed records, but this seems to happen every 10-14 days.

This job is a native m2 build (the most complex & active one I have,  
FWIW).  There are two freestyle builds pointing at the same repo  
which have never similarly stalled.  There are a couple of other  
native m2 builds (against other repos) whose polling has never stalled.

There's nothing special in the logs (/hudson/log or catalina.out) at  
the time the polling stalls.  The subversion polling log looks normal  
-- it just shows the last polling result (i.e., at the time it  
stalled).  Is there some extra debug information I can enable to  
catch the reason for this?  I found a previous thread that suggested /
hudson/threadDump, but that doesn't work for me, which I gather is  
because I'm running on Java 5.  (There's no production-quality Java 6  
for OS X yet, so I can't get around that.)

Thanks,
Rhett

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Subversion polling stops/stalls eventually

Kohsuke Kawaguchi
Administrator
Rhett Sutphin wrote:
> Hi,
>
> I've got another intermittent issue that I haven't been able to  
> correlate with anything in particular.  All of my jobs are using  
> subversion polling to kick off builds.  One job will occasionally  
> stop polling.  Once this happens, the only way I can get it to start  
> polling again is to restart tomcat.  After the polling stalls, manual  
> builds still work fine (including svn updates).  A haven't kept  
> detailed records, but this seems to happen every 10-14 days.

OK. When that happens what does the log show? Is it in the middle of SVN
operation? Or is it like the previous one ended normally but the next
polling just doesn't happen?

> This job is a native m2 build (the most complex & active one I have,  
> FWIW).  There are two freestyle builds pointing at the same repo  
> which have never similarly stalled.  There are a couple of other  
> native m2 builds (against other repos) whose polling has never stalled.

Thanks for the info. The polling stuff is somewhat detached from the
rest of the build, so I don't think it's likely that the complexity of
the build is affecting this (although it's too early to say anything.)

> There's nothing special in the logs (/hudson/log or catalina.out) at  
> the time the polling stalls.  The subversion polling log looks normal  
> -- it just shows the last polling result (i.e., at the time it  
> stalled).  Is there some extra debug information I can enable to  
> catch the reason for this?

Ah, OK, thanks, you answered my first question.



 > I found a previous thread that suggested /
> hudson/threadDump, but that doesn't work for me, which I gather is  
> because I'm running on Java 5.  (There's no production-quality Java 6  
> for OS X yet, so I can't get around that.)

You can try hudson/classicThreadDump. Please send me the output, I think
this is the first step to understand what's going on.

I filed https://hudson.dev.java.net/issues/show_bug.cgi?id=644 in
relation to this.


--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Subversion polling stops/stalls eventually

Rhett Sutphin
Hi,

On Jul 17, 2007, at 9:57 AM, Kohsuke Kawaguchi wrote:

>> This job is a native m2 build (the most complex & active one I  
>> have,  FWIW).  There are two freestyle builds pointing at the same  
>> repo  which have never similarly stalled.  There are a couple of  
>> other  native m2 builds (against other repos) whose polling has  
>> never stalled.
>
> Thanks for the info. The polling stuff is somewhat detached from  
> the rest of the build, so I don't think it's likely that the  
> complexity of the build is affecting this (although it's too early  
> to say anything.)

Yeah, I figured probably not.  Since it was the only way I could see  
that that job was different, I mentioned it.

> > I found a previous thread that suggested /
>> hudson/threadDump, but that doesn't work for me, which I gather  
>> is  because I'm running on Java 5.  (There's no production-quality  
>> Java 6  for OS X yet, so I can't get around that.)
>
> You can try hudson/classicThreadDump. Please send me the output, I  
> think this is the first step to understand what's going on.

In an odd coincidence, it stalled again early this morning (much  
quicker than usual).  I'm sending the thread dump off-list.

Rhett

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Re: Subversion polling stops/stalls eventually

Nord, James-2
Hi,

I'm also having issues with the polling stopping (but not on all
projects)

3 top level maven projects, one is still polling other 2 are not.  
The 2 that are not polliing actually do involve projects that requires
some building, the one that is still polling is a pom project that
contains a single empty maven project.

Under the tomcat process there are 2 Java processes (not sure what they
are both doing but here is the stack output of both.)

Tomcat 5.5.23
Hudson 1.115
Java 1.6_u1
WinXP SP2

Other Odd things:
        The "Subversion Polling Log" link shows up on 2 of the 3
projects. (it is not the same 2 as the ones that don't build...)

Regards,

        /James


> -----Original Message-----
> From: Rhett Sutphin [mailto:[hidden email]]
> Sent: 17 July 2007 18:57
> To: [hidden email]
> Subject: Re: Subversion polling stops/stalls eventually
>
> Hi,
>
> On Jul 17, 2007, at 9:57 AM, Kohsuke Kawaguchi wrote:
> >> This job is a native m2 build (the most complex & active
> one I have,  
> >> FWIW).  There are two freestyle builds pointing at the same repo  
> >> which have never similarly stalled.  There are a couple of other  
> >> native m2 builds (against other repos) whose polling has never
> >> stalled.
> >
> > Thanks for the info. The polling stuff is somewhat detached
> from the
> > rest of the build, so I don't think it's likely that the
> complexity of
> > the build is affecting this (although it's too early to say
> anything.)
>
> Yeah, I figured probably not.  Since it was the only way I
> could see that that job was different, I mentioned it.
>
> > > I found a previous thread that suggested /
> >> hudson/threadDump, but that doesn't work for me, which I
> gather is  
> >> because I'm running on Java 5.  (There's no
> production-quality Java 6  
> >> for OS X yet, so I can't get around that.)
> >
> > You can try hudson/classicThreadDump. Please send me the output, I
> > think this is the first step to understand what's going on.
>
> In an odd coincidence, it stalled again early this morning
> (much quicker than usual).  I'm sending the thread dump off-list.
>
> Rhett
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
*********************************************************************************************************
This e-mail is confidential, the property of NDS Ltd and intended for the addressee only.  Any dissemination, copying or distribution of this message or any attachments by anyone other than the intended recipient is strictly prohibited.  If you have received this message in error, please immediately notify the [hidden email] and destroy the original message.  Messages sent to and from NDS may be monitored.  NDS cannot guarantee any message delivery method is secure or error-free.  Information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses.  We do not accept responsibility for any errors or omissions in this message and/or attachment that arise as a result of transmission.  You should carry out your own virus checks before opening any attachment.  Any views or opinions presented are solely those of the author and do not necessarily represent those of NDS.

To protect the environment please do not print this e-mail unless necessary.

NDS Limited Registered office: One Heathrow Boulevard, 286 Bath Road, West Drayton, Middlesex, UB7 0DQ, United Kingdom. A company registered in England and Wales  Registered no. 3080780   VAT no. GB 603 8808 40-00
**********************************************************************************************************


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

stack_pid840.txt (4K) Download Attachment
stack_pid1952.txt (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: RE: Re: Subversion polling stops/stalls eventually

Nord, James-2
Just seen 1.119...  Upgrade in progress. D'OH!

> -----Original Message-----
> From: Nord, James [mailto:[hidden email]]
> Sent: 18 July 2007 13:52
> To: [hidden email]
> Subject: RE: Re: Subversion polling stops/stalls eventually
>
> Hi,
>
> I'm also having issues with the polling stopping (but not on all
> projects)
>
> 3 top level maven projects, one is still polling other 2 are not.  
> The 2 that are not polliing actually do involve projects that
> requires some building, the one that is still polling is a
> pom project that contains a single empty maven project.
>
> Under the tomcat process there are 2 Java processes (not sure
> what they are both doing but here is the stack output of both.)
>
> Tomcat 5.5.23
> Hudson 1.115
> Java 1.6_u1
> WinXP SP2
>
> Other Odd things:
> The "Subversion Polling Log" link shows up on 2 of the
> 3 projects. (it is not the same 2 as the ones that don't build...)
>
> Regards,
>
> /James
>
>
> > -----Original Message-----
> > From: Rhett Sutphin [mailto:[hidden email]]
> > Sent: 17 July 2007 18:57
> > To: [hidden email]
> > Subject: Re: Subversion polling stops/stalls eventually
> >
> > Hi,
> >
> > On Jul 17, 2007, at 9:57 AM, Kohsuke Kawaguchi wrote:
> > >> This job is a native m2 build (the most complex & active
> > one I have,
> > >> FWIW).  There are two freestyle builds pointing at the same repo
> > >> which have never similarly stalled.  There are a couple of other
> > >> native m2 builds (against other repos) whose polling has never
> > >> stalled.
> > >
> > > Thanks for the info. The polling stuff is somewhat detached
> > from the
> > > rest of the build, so I don't think it's likely that the
> > complexity of
> > > the build is affecting this (although it's too early to say
> > anything.)
> >
> > Yeah, I figured probably not.  Since it was the only way I
> could see
> > that that job was different, I mentioned it.
> >
> > > > I found a previous thread that suggested /
> > >> hudson/threadDump, but that doesn't work for me, which I
> > gather is
> > >> because I'm running on Java 5.  (There's no
> > production-quality Java 6
> > >> for OS X yet, so I can't get around that.)
> > >
> > > You can try hudson/classicThreadDump. Please send me the
> output, I
> > > think this is the first step to understand what's going on.
> >
> > In an odd coincidence, it stalled again early this morning (much
> > quicker than usual).  I'm sending the thread dump off-list.
> >
> > Rhett
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
> **************************************************************
> *******************************************
> This e-mail is confidential, the property of NDS Ltd and
> intended for the addressee only.  Any dissemination, copying
> or distribution of this message or any attachments by anyone
> other than the intended recipient is strictly prohibited.  If
> you have received this message in error, please immediately
> notify the [hidden email] and destroy the original
> message.  Messages sent to and from NDS may be monitored.  
> NDS cannot guarantee any message delivery method is secure or
> error-free.  Information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or contain
> viruses.  We do not accept responsibility for any errors or
> omissions in this message and/or attachment that arise as a
> result of transmission.  You should carry out your own virus
> checks before opening any attachment.  Any views or opinions
> presented are solely those of the author and do not
> necessarily represent those of NDS.
>
> To protect the environment please do not print this e-mail
> unless necessary.
>
> NDS Limited Registered office: One Heathrow Boulevard, 286
> Bath Road, West Drayton, Middlesex, UB7 0DQ, United Kingdom.
> A company registered in England and Wales  Registered no.
> 3080780   VAT no. GB 603 8808 40-00
> **************************************************************
> ********************************************
>
> *** NDS IN IT scanned this email for malicious content ***
> *** IMPORTANT: Do not open attachments from unrecognized
> senders  ***Disclaimer:
> **************************************************************
> *******************************************
> This e-mail is confidential, the property of NDS Ltd and
> intended for the addressee only.  Any dissemination, copying
> or distribution of this message or any attachments by anyone
> other than the intended recipient is strictly prohibited.  If
> you have received this message in error, please immediately
> notify the [hidden email] and destroy the original
> message.  Messages sent to and from NDS may be monitored.  
> NDS cannot guarantee any message delivery method is secure or
> error-free.  Information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or contain
> viruses.  We do not accept responsibility for any errors or
> omissions in this message and/or attachment that arise as a
> result of transmission.  You should carry out your own virus
> checks before opening any attachment.  Any views or opinions
> presented are solely those of the author and do not
> necessarily represent those of NDS.
>
> To protect the environment please do not print this e-mail
> unless necessary.
>
> NDS Limited Registered office: One Heathrow Boulevard, 286
> Bath Road, West Drayton, Middlesex, UB7 0DQ, United Kingdom.
> A company registered in England and Wales  Registered no.
> 3080780   VAT no. GB 603 8808 40-00
> **************************************************************
> ********************************************
>
>
*********************************************************************************************************
This e-mail is confidential, the property of NDS Ltd and intended for the addressee only.  Any dissemination, copying or distribution of this message or any attachments by anyone other than the intended recipient is strictly prohibited.  If you have received this message in error, please immediately notify the [hidden email] and destroy the original message.  Messages sent to and from NDS may be monitored.  NDS cannot guarantee any message delivery method is secure or error-free.  Information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses.  We do not accept responsibility for any errors or omissions in this message and/or attachment that arise as a result of transmission.  You should carry out your own virus checks before opening any attachment.  Any views or opinions presented are solely those of the author and do not necessarily represent those of NDS.

To protect the environment please do not print this e-mail unless necessary.

NDS Limited Registered office: One Heathrow Boulevard, 286 Bath Road, West Drayton, Middlesex, UB7 0DQ, United Kingdom. A company registered in England and Wales  Registered no. 3080780   VAT no. GB 603 8808 40-00
**********************************************************************************************************

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Subversion polling stops/stalls eventually

Kohsuke Kawaguchi
Administrator
In reply to this post by Nord, James-2
Nord, James wrote:
> Under the tomcat process there are 2 Java processes (not sure what they
> are both doing but here is the stack output of both.)
>
> Tomcat 5.5.23
> Hudson 1.115
> Java 1.6_u1
> WinXP SP2

Thanks. Those two processes are the Maven process parked so that future
m2 builds can happen quickly.

But the thread dump that I wanted to see is that of Tomcat, so you
missed the crucial one. If the problem happens again, please get me the
thread dump of your Tomcat.

> Other Odd things:
> The "Subversion Polling Log" link shows up on 2 of the 3
> projects. (it is not the same 2 as the ones that don't build...)

The link is supposed to show up whenever the polling is configured. So I
don't know what's going on there. Are you sure polling is configured on
all 3 projects?

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: Re: Subversion polling stops/stalls eventually

Nord, James-2
Hi Kohsuke ,

> > Under the tomcat process there are 2 Java processes (not sure what
> > they are both doing but here is the stack output of both.)
> >
> > Tomcat 5.5.23
> > Hudson 1.115
> > Java 1.6_u1
> > WinXP SP2
>
> Thanks. Those two processes are the Maven process parked so
> that future
> m2 builds can happen quickly.
>
> But the thread dump that I wanted to see is that of Tomcat,
> so you missed the crucial one. If the problem happens again,
> please get me the thread dump of your Tomcat.

I tried that unfortunately I couldn't get a stack from it with jstack as
it internally loads the jvm.dll and not the java.exe process.

Anyone know how to get a tomcat thread dump on windows when run as a
service?

 
> > Other Odd things:
> > The "Subversion Polling Log" link shows up on 2 of the
> 3 projects.
> > (it is not the same 2 as the ones that don't build...)
>
> The link is supposed to show up whenever the polling is
> configured. So I don't know what's going on there. Are you
> sure polling is configured on all 3 projects?


All 3 where configured, and you could see from the main log that at one
point they where all running.

I've updated to 1.119 and the link is shown for all 3 projects - I'm
sure the tomcat was restarted several times when polling was failing so
I think whatever it was must have been solved by 1.119.
*********************************************************************************************************
This e-mail is confidential, the property of NDS Ltd and intended for the addressee only.  Any dissemination, copying or distribution of this message or any attachments by anyone other than the intended recipient is strictly prohibited.  If you have received this message in error, please immediately notify the [hidden email] and destroy the original message.  Messages sent to and from NDS may be monitored.  NDS cannot guarantee any message delivery method is secure or error-free.  Information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses.  We do not accept responsibility for any errors or omissions in this message and/or attachment that arise as a result of transmission.  You should carry out your own virus checks before opening any attachment.  Any views or opinions presented are solely those of the author and do not necessarily represent those of NDS.

To protect the environment please do not print this e-mail unless necessary.

NDS Limited Registered office: One Heathrow Boulevard, 286 Bath Road, West Drayton, Middlesex, UB7 0DQ, United Kingdom. A company registered in England and Wales  Registered no. 3080780   VAT no. GB 603 8808 40-00
**********************************************************************************************************

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Subversion polling stops/stalls eventually

Kohsuke Kawaguchi
Administrator
Nord, James wrote:

> Hi Kohsuke ,
>
>> > Under the tomcat process there are 2 Java processes (not sure what
>> > they are both doing but here is the stack output of both.)
>> >
>> > Tomcat 5.5.23
>> > Hudson 1.115
>> > Java 1.6_u1
>> > WinXP SP2
>>
>> Thanks. Those two processes are the Maven process parked so
>> that future
>> m2 builds can happen quickly.
>>
>> But the thread dump that I wanted to see is that of Tomcat,
>> so you missed the crucial one. If the problem happens again,
>> please get me the thread dump of your Tomcat.
>
> I tried that unfortunately I couldn't get a stack from it with jstack as
> it internally loads the jvm.dll and not the java.exe process.
>
> Anyone know how to get a tomcat thread dump on windows when run as a
> service?
Does jconsole work? Or maybe they are both using the same underlying
transport.

Did I already suggested http://yourhost/hudson/threadDump ?

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment