build hang after temporary network failure

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

build hang after temporary network failure

tec-3

Hi Folks,

I'm seeing a problem where hudson (v1.237) jobs hang until killed if there
is a network problem during svn polling.

The network I'm on is currently congested with the sporadic problem that
arp packets sometimes collide causing ssh connections to occasionally get
"no route to host" messages instead of connections.  A retry a few seconds
later will be fine, but the momentary glitch does block the first attempt.

My problem with Hudson is that I have a small job polling my svn repository
every 5 minutes.  Most of the time, things go as I expect, but when a
network glitch coincides with a hudson svn poll or a hudson svn update, I
get the following error message and the build hangs.  The last time this
happened, the normal 30 second job was at over 4 hours and counting when I
killed the hudson master and restarted.

What I'd like to see is a simple failure or abort, followed by a retry at
a later time.

Is this a known bug?

Best Wishes,

     --Tom

Buildbot Hudson writes:
 > See http://HUDSON_MASTER:8080/job/testBlame/393/changes
 >
 > ------------------------------------------
 > started
 > Building remotely on HUDSON_SLAVE_HOST
 > Updating svn+ssh://REPOSITORY_HOST/REPOSITORY_PATH
 > ERROR: Failed to update svn+ssh://REPOSITORY_HOST/REPOSITORY_PATH
 > org.tmatesoft.svn.core.SVNException: svn: Cannot connect to 'svn+ssh://REPOSITORY_HOST': There was a problem while connecting to REPOSITORY_HOST:22
 > at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:69)
 > at org.tmatesoft.svn.core.internal.io.svn.SVNSSHSession.openConnection(SVNSSHSession.java:281)
 > at org.tmatesoft.svn.core.internal.io.svn.SVNSSHSession.getConnection(SVNSSHSession.java:85)
 > at org.tmatesoft.svn.core.internal.io.svn.SVNSSHConnector.open(SVNSSHConnector.java:74)
 > at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.open(SVNConnection.java:60)
 > at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.openConnection(SVNRepositoryImpl.java:974)
 > at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.getDatedRevision(SVNRepositoryImpl.java:159)
 > at org.tmatesoft.svn.core.wc.SVNBasicClient.getRevisionNumber(SVNBasicClient.java:346)
 > at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:159)
 > at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:387)
 > at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:359)
 > at hudson.FilePath$FileCallableWrapper.call(FilePath.java:1122)
 > at hudson.remoting.UserRequest.perform(UserRequest.java:69)
 > at hudson.remoting.UserRequest.perform(UserRequest.java:23)
 > at hudson.remoting.Request$2.run(Request.java:200)
 > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
 > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
 > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
 > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
 > at java.lang.Thread.run(Thread.java:619)
 > Caused by: java.io.IOException: There was a problem while connecting to REPOSITORY_HOST:22
 > at com.trilead.ssh2.Connection.connect(Connection.java:753)
 > at com.trilead.ssh2.Connection.connect(Connection.java:544)
 > at org.tmatesoft.svn.core.internal.io.svn.SVNSSHSession.openConnection(SVNSSHSession.java:241)
 > ... 19 more
 > Caused by: java.net.NoRouteToHostException: No route to host
 > at java.net.PlainSocketImpl.socketConnect(Native Method)
 > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
 > at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
 > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
 > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
 > at java.net.Socket.connect(Socket.java:519)
 > at com.trilead.ssh2.transport.TransportManager.establishConnection(TransportManager.java:341)
 > at com.trilead.ssh2.transport.TransportManager.initialize(TransportManager.java:449)
 > at com.trilead.ssh2.Connection.connect(Connection.java:697)
 > ... 21 more
 > Recording fingerprints
 >

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: build hang after temporary network failure

Kohsuke Kawaguchi
Administrator

I suppose "java.net.NoRouteToHostException: No route to host" exception
is reported but the build hangs after that?

When it hangs again, can you get us the thread dump on the server and
the slave? You can go to http://server/hudson/threadDump and
http://server/computer/YOURSLAVENAME/systemInfo to get those.

Tom Cavin wrote:

> Hi Folks,
>
> I'm seeing a problem where hudson (v1.237) jobs hang until killed if there
> is a network problem during svn polling.
>
> The network I'm on is currently congested with the sporadic problem that
> arp packets sometimes collide causing ssh connections to occasionally get
> "no route to host" messages instead of connections.  A retry a few seconds
> later will be fine, but the momentary glitch does block the first attempt.
>
> My problem with Hudson is that I have a small job polling my svn repository
> every 5 minutes.  Most of the time, things go as I expect, but when a
> network glitch coincides with a hudson svn poll or a hudson svn update, I
> get the following error message and the build hangs.  The last time this
> happened, the normal 30 second job was at over 4 hours and counting when I
> killed the hudson master and restarted.
>
> What I'd like to see is a simple failure or abort, followed by a retry at
> a later time.
>
> Is this a known bug?
>
> Best Wishes,
>
>      --Tom
>
> Buildbot Hudson writes:
>  > See http://HUDSON_MASTER:8080/job/testBlame/393/changes
>  >
>  > ------------------------------------------
>  > started
>  > Building remotely on HUDSON_SLAVE_HOST
>  > Updating svn+ssh://REPOSITORY_HOST/REPOSITORY_PATH
>  > ERROR: Failed to update svn+ssh://REPOSITORY_HOST/REPOSITORY_PATH
>  > org.tmatesoft.svn.core.SVNException: svn: Cannot connect to 'svn+ssh://REPOSITORY_HOST': There was a problem while connecting to REPOSITORY_HOST:22
>  > at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:69)
>  > at org.tmatesoft.svn.core.internal.io.svn.SVNSSHSession.openConnection(SVNSSHSession.java:281)
>  > at org.tmatesoft.svn.core.internal.io.svn.SVNSSHSession.getConnection(SVNSSHSession.java:85)
>  > at org.tmatesoft.svn.core.internal.io.svn.SVNSSHConnector.open(SVNSSHConnector.java:74)
>  > at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.open(SVNConnection.java:60)
>  > at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.openConnection(SVNRepositoryImpl.java:974)
>  > at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.getDatedRevision(SVNRepositoryImpl.java:159)
>  > at org.tmatesoft.svn.core.wc.SVNBasicClient.getRevisionNumber(SVNBasicClient.java:346)
>  > at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:159)
>  > at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:387)
>  > at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:359)
>  > at hudson.FilePath$FileCallableWrapper.call(FilePath.java:1122)
>  > at hudson.remoting.UserRequest.perform(UserRequest.java:69)
>  > at hudson.remoting.UserRequest.perform(UserRequest.java:23)
>  > at hudson.remoting.Request$2.run(Request.java:200)
>  > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>  > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>  > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>  > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>  > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  > at java.lang.Thread.run(Thread.java:619)
>  > Caused by: java.io.IOException: There was a problem while connecting to REPOSITORY_HOST:22
>  > at com.trilead.ssh2.Connection.connect(Connection.java:753)
>  > at com.trilead.ssh2.Connection.connect(Connection.java:544)
>  > at org.tmatesoft.svn.core.internal.io.svn.SVNSSHSession.openConnection(SVNSSHSession.java:241)
>  > ... 19 more
>  > Caused by: java.net.NoRouteToHostException: No route to host
>  > at java.net.PlainSocketImpl.socketConnect(Native Method)
>  > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
>  > at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
>  > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
>  > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
>  > at java.net.Socket.connect(Socket.java:519)
>  > at com.trilead.ssh2.transport.TransportManager.establishConnection(TransportManager.java:341)
>  > at com.trilead.ssh2.transport.TransportManager.initialize(TransportManager.java:449)
>  > at com.trilead.ssh2.Connection.connect(Connection.java:697)
>  > ... 21 more
>  > Recording fingerprints
>  >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

--
Kohsuke Kawaguchi
Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: build hang after temporary network failure

tec-3

Hi Kohsuke,

It happened again, so here is the information requested.

The server is actually running two jobs and using two executors.  The main
production job is still running, but the minor test job (poll repository,
update checkout, record file checksums and save in reference file,
fingerprint and archive reference file, repeat ...) has failed and has tied
up one of my executors.

Information follows as:

                   failed Job console output (sanitized)
     ****************************************************************
                     thread dump of server (sanitized)
     ****************************************************************
                     system info of client (sanitized)
     ****************************************************************

Please let me know if you need more information.  Since the production job
seems to be running and polling properly, I can probably leave the hung
executor in its current state for a few hours.

Best Wishes,

     --Tom

     ****************************************************************
                   failed Job console output (sanitized)
started
Building remotely on PROBLEM-HOST
Updating svn+ssh://SVN_REPOSITORY_HOST/Path/to/Code
ERROR: Failed to update svn+ssh://SVN_REPOSITORY_HOST/Path/to/Code
org.tmatesoft.svn.core.SVNException: svn: Cannot connect to 'svn+ssh://SVN_REPOSITORY_HOST': There was a problem while connecting to SVN_REPOSITORY_HOST:22
        at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:69)
        at org.tmatesoft.svn.core.internal.io.svn.SVNSSHSession.openConnection(SVNSSHSession.java:281)
        at org.tmatesoft.svn.core.internal.io.svn.SVNSSHSession.getConnection(SVNSSHSession.java:85)
        at org.tmatesoft.svn.core.internal.io.svn.SVNSSHConnector.open(SVNSSHConnector.java:74)
        at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.open(SVNConnection.java:60)
        at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.openConnection(SVNRepositoryImpl.java:974)
        at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.getDatedRevision(SVNRepositoryImpl.java:159)
        at org.tmatesoft.svn.core.wc.SVNBasicClient.getRevisionNumber(SVNBasicClient.java:346)
        at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:159)
        at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:387)
        at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:359)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:1122)
        at hudson.remoting.UserRequest.perform(UserRequest.java:69)
        at hudson.remoting.UserRequest.perform(UserRequest.java:23)
        at hudson.remoting.Request$2.run(Request.java:200)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: There was a problem while connecting to SVN_REPOSITORY_HOST:22
        at com.trilead.ssh2.Connection.connect(Connection.java:753)
        at com.trilead.ssh2.Connection.connect(Connection.java:544)
        at org.tmatesoft.svn.core.internal.io.svn.SVNSSHSession.openConnection(SVNSSHSession.java:241)
        ... 19 more
Caused by: java.net.NoRouteToHostException: No route to host
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:519)
        at com.trilead.ssh2.transport.TransportManager.establishConnection(TransportManager.java:341)
        at com.trilead.ssh2.transport.TransportManager.initialize(TransportManager.java:449)
        at com.trilead.ssh2.Connection.connect(Connection.java:697)
        ... 21 more
Recording fingerprints
Sending e-mails to: Me@my-internal-email
finished: FAILURE

     ****************************************************************
                     thread dump of server (sanitized)
  Thread Dump
    Thread[RequestHandlerThread[#3],5,main] (WAITING)
      java.lang.Object.wait(Native Method)
      java.lang.Object.wait(Object.java:474)
      winstone.RequestHandlerThread.run(RequestHandlerThread.java:216)
      java.lang.Thread.run(Thread.java:595)
    Thread[Executor #0 for UNRELATED-DEAD-HOST,5,main] (TIMED_WAITING)
      java.lang.Object.wait(Native Method)
      hudson.util.OneShotEvent.block(OneShotEvent.java:40)
      hudson.model.Queue.pop(Queue.java:478)
      hudson.model.Executor.run(Executor.java:72)
    Thread[Hudson cron thread,5,main] (TIMED_WAITING)
      java.lang.Object.wait(Native Method)
      java.util.TimerThread.mainLoop(Timer.java:509)
      java.util.TimerThread.run(Timer.java:462)
    Thread[DestroyJavaVM,5,main] (RUNNABLE)
    Thread[Executor #0 for PROBLEM-HOST,5,main] (WAITING)
      java.lang.Object.wait(Native Method)
      java.lang.Object.wait(Object.java:474)
      hudson.util.OneShotEvent.block(OneShotEvent.java:28)
      hudson.model.Queue.pop(Queue.java:476)
      hudson.model.Executor.run(Executor.java:72)
    Thread[RequestHandlerThread[#0],5,main] (WAITING)
      java.lang.Object.wait(Native Method)
      java.lang.Object.wait(Object.java:474)
      winstone.RequestHandlerThread.run(RequestHandlerThread.java:216)
      java.lang.Thread.run(Thread.java:595)
    Thread[RequestHandlerThread[#1],5,main] (WAITING)
      java.lang.Object.wait(Native Method)
      java.lang.Object.wait(Object.java:474)
      winstone.RequestHandlerThread.run(RequestHandlerThread.java:216)
      java.lang.Thread.run(Thread.java:595)
    Thread[stderr copier for remote agent on PROBLEM-HOST,5,main] (RUNNABLE)
      java.io.FileInputStream.readBytes(Native Method)
      java.io.FileInputStream.read(FileInputStream.java:177)
      hudson.util.StreamCopyThread.run(StreamCopyThread.java:27)
    Thread[RequestHandlerThread[#6],5,main] (RUNNABLE)
      java.lang.Thread.dumpThreads(Native Method)
      java.lang.Thread.getAllStackTraces(Thread.java:1434)
      hudson.Functions.dumpAllThreads(Functions.java:630)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      java.lang.reflect.Method.invoke(Method.java:585)
      org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:259)
      org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104)
      org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
      org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
      org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
      org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
      hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:71)
      org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61)
      org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsIterator(ExpressionSupport.java:94)
      org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:89)
      org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
      org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41)
      org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
      org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)
      org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
      org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
      org.apache.commons.jelly.tags.core.IfTag.doTag(IfTag.java:42)
      org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
      org.apache.commons.jelly.tags.core.JellyTag.doTag(JellyTag.java:45)
      org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
      org.apache.commons.jelly.impl.DynamicTag.doTag(DynamicTag.java:80)
      org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
      org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
      org.apache.commons.jelly.impl.StaticTag.doTag(StaticTag.java:65)
      org.apache.commons.jelly.impl.StaticTagScript.run(StaticTagScript.java:112)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
      org.apache.commons.jelly.impl.StaticTag.doTag(StaticTag.java:65)
      org.apache.commons.jelly.impl.StaticTagScript.run(StaticTagScript.java:112)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
      org.apache.commons.jelly.impl.StaticTag.doTag(StaticTag.java:65)
      org.apache.commons.jelly.impl.StaticTagScript.run(StaticTagScript.java:112)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
      org.apache.commons.jelly.impl.StaticTag.doTag(StaticTag.java:65)
      org.apache.commons.jelly.impl.StaticTagScript.run(StaticTagScript.java:112)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
      org.apache.commons.jelly.impl.StaticTag.doTag(StaticTag.java:65)
      org.apache.commons.jelly.impl.StaticTagScript.run(StaticTagScript.java:112)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
      org.apache.commons.jelly.tags.core.JellyTag.doTag(JellyTag.java:45)
      org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
      org.apache.commons.jelly.impl.DynamicTag.doTag(DynamicTag.java:80)
      org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
      org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
      org.apache.commons.jelly.tags.core.JellyTag.doTag(JellyTag.java:45)
      org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
      org.kohsuke.stapler.jelly.JellyClassTearOff.invokeScript(JellyClassTearOff.java:78)
      org.kohsuke.stapler.MetaClass$3.dispatch(MetaClass.java:125)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:440)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:361)
      org.kohsuke.stapler.Stapler.service(Stapler.java:121)
      javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
      winstone.ServletConfiguration.execute(ServletConfiguration.java:249)
      winstone.RequestDispatcher.forward(RequestDispatcher.java:335)
      winstone.RequestDispatcher.doFilter(RequestDispatcher.java:378)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:53)
      hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:45)
      winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
      winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:52)
      hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:28)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:55)
      org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:55)
      org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:55)
      org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:55)
      org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:55)
      org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:55)
      org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:42)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:55)
      hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:44)
      hudson.security.HudsonFilter.doFilter(HudsonFilter.java:85)
      winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
      winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
      winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
      winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:244)
      winstone.RequestHandlerThread.run(RequestHandlerThread.java:150)
      java.lang.Thread.run(Thread.java:595)
    Thread[WinstoneHostConfigurationMgmt:default,5,main] (TIMED_WAITING)
      java.lang.Thread.sleep(Native Method)
      winstone.HostConfiguration.run(HostConfiguration.java:176)
      java.lang.Thread.run(Thread.java:595)
    Thread[process reaper,5,main] (RUNNABLE)
      java.lang.UNIXProcess.waitForProcessExit(Native Method)
      java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
      java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
    Thread[Reference Handler,10,system] (WAITING)
      java.lang.Object.wait(Native Method)
      java.lang.Object.wait(Object.java:474)
      java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
    Thread[Finalizer,8,system] (WAITING)
      java.lang.Object.wait(Native Method)
      java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
      java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
      java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
    Thread[Thread-2,5,main] (TIMED_WAITING)
      java.lang.Object.wait(Native Method)
      hudson.model.ViewJob$ReloadThread.getNext(ViewJob.java:136)
      hudson.model.ViewJob$ReloadThread.run(ViewJob.java:152)
    Thread[Channel reader thread: PROBLEM-HOST,5,main] (RUNNABLE)
      java.io.FileInputStream.readBytes(Native Method)
      java.io.FileInputStream.read(FileInputStream.java:194)
      java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
      java.io.BufferedInputStream.read(BufferedInputStream.java:235)
      java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2200)
      java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2490)
      java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2500)
      java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1267)
      java.io.ObjectInputStream.readObject(ObjectInputStream.java:339)
      hudson.remoting.Channel$ReaderThread.run(Channel.java:576)
    Thread[RequestHandlerThread[#5],5,main] (WAITING)
      java.lang.Object.wait(Native Method)
      java.lang.Object.wait(Object.java:474)
      winstone.RequestHandlerThread.run(RequestHandlerThread.java:216)
      java.lang.Thread.run(Thread.java:595)
    Thread[process reaper,5,main] (RUNNABLE)
      java.lang.UNIXProcess.waitForProcessExit(Native Method)
      java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
      java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
    Thread[ConnectorThread:[ajp13-8009],5,main] (RUNNABLE)
      java.net.PlainSocketImpl.socketAccept(Native Method)
      java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
      java.net.ServerSocket.implAccept(ServerSocket.java:450)
      java.net.ServerSocket.accept(ServerSocket.java:421)
      winstone.ajp13.Ajp13Listener.run(Ajp13Listener.java:111)
      java.lang.Thread.run(Thread.java:595)
    Thread[RequestHandlerThread[#4],5,main] (WAITING)
      java.lang.Object.wait(Native Method)
      java.lang.Object.wait(Object.java:474)
      winstone.RequestHandlerThread.run(RequestHandlerThread.java:216)
      java.lang.Thread.run(Thread.java:595)
    Thread[pool-2-thread-301,5,main] (TIMED_WAITING)
      sun.misc.Unsafe.park(Native Method)
      java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:772)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1087)
      java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
      java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
      java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
      java.lang.Thread.run(Thread.java:595)
    Thread[TCP slave agent listener port=0,5,main] (RUNNABLE)
      java.net.PlainSocketImpl.socketAccept(Native Method)
      java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
      java.net.ServerSocket.implAccept(ServerSocket.java:450)
      java.net.ServerSocket.accept(ServerSocket.java:421)
      hudson.TcpSlaveAgentListener.run(TcpSlaveAgentListener.java:82)
    Thread[ConnectorThread:[http-8080],5,main] (RUNNABLE)
      java.net.PlainSocketImpl.socketAccept(Native Method)
      java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
      java.net.ServerSocket.implAccept(ServerSocket.java:450)
      java.net.ServerSocket.accept(ServerSocket.java:421)
      winstone.HttpListener.run(HttpListener.java:127)
      java.lang.Thread.run(Thread.java:595)
    Thread[RequestHandlerThread[#7],5,main] (WAITING)
      java.lang.Object.wait(Native Method)
      java.lang.Object.wait(Object.java:474)
      winstone.RequestHandlerThread.run(RequestHandlerThread.java:216)
      java.lang.Thread.run(Thread.java:595)
    Thread[LauncherControlThread[ControlPort=-1],5,main] (TIMED_WAITING)
      java.lang.Thread.sleep(Native Method)
      winstone.Launcher.run(Launcher.java:279)
      java.lang.Thread.run(Thread.java:595)
    Thread[Signal Dispatcher,10,system] (RUNNABLE)
    Thread[RequestHandlerThread[#2],5,main] (WAITING)
      java.lang.Object.wait(Native Method)
      java.lang.Object.wait(Object.java:474)
      winstone.RequestHandlerThread.run(RequestHandlerThread.java:216)
      java.lang.Thread.run(Thread.java:595)
    Thread[Executor #0 for WORKING-HOST,5,main] (WAITING)
      java.lang.Object.wait(Native Method)
      java.lang.Object.wait(Object.java:474)
      hudson.util.OneShotEvent.block(OneShotEvent.java:28)
      hudson.model.Queue.pop(Queue.java:476)
      hudson.model.Executor.run(Executor.java:72)
    Thread[stderr copier for remote agent on WORKING-HOST,5,main] (RUNNABLE)
      java.io.FileInputStream.readBytes(Native Method)
      java.io.FileInputStream.read(FileInputStream.java:177)
      hudson.util.StreamCopyThread.run(StreamCopyThread.java:27)
    Thread[Channel reader thread: WORKING-HOST,5,main] (RUNNABLE)
      java.io.FileInputStream.readBytes(Native Method)
      java.io.FileInputStream.read(FileInputStream.java:194)
      java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
      java.io.BufferedInputStream.read(BufferedInputStream.java:235)
      java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2200)
      java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2490)
      java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2500)
      java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1267)
      java.io.ObjectInputStream.readObject(ObjectInputStream.java:339)
      hudson.remoting.Channel$ReaderThread.run(Channel.java:576)
    Thread[WinstoneObjectPoolMgmt,5,main] (TIMED_WAITING)
      java.lang.Thread.sleep(Native Method)
      winstone.ObjectPool.run(ObjectPool.java:103)
      java.lang.Thread.run(Thread.java:595)
     ****************************************************************
                     system info of client (sanitized)
System Properties
Name Value  
file.encoding ANSI_X3.4-1968
file.encoding.pkg sun.io
file.separator /
java.awt.graphicsenv sun.awt.X11GraphicsEnvironment
java.awt.printerjob sun.print.PSPrinterJob
java.class.path /NFS/PATH/to/Hudson-home/WEB-INF/slave.jar
java.class.version 50.0
java.endorsed.dirs /usr/java/jdk1.6.0_05/jre/lib/endorsed
java.ext.dirs /usr/java/jdk1.6.0_05/jre/lib/ext:/usr/java/packages/lib/ext
java.home /usr/java/jdk1.6.0_05/jre
java.io.tmpdir /tmp
java.library.path /usr/java/jdk1.6.0_05/jre/lib/i386/server:/usr/java/jdk1.6.0_05/jre/lib/i386:/usr/java/jdk1.6.0_05/jre/../lib/i386:/usr/local/lib:/usr/lib:/usr/java/packages/lib/i386:/lib:/usr/lib
java.runtime.name Java(TM) SE Runtime Environment
java.runtime.version 1.6.0_05-b13
java.specification.name Java Platform API Specification
java.specification.vendor Sun Microsystems Inc.
java.specification.version 1.6
java.vendor Sun Microsystems Inc.
java.vendor.url http://java.sun.com/
java.vendor.url.bug http://java.sun.com/cgi-bin/bugreport.cgi
java.version 1.6.0_05
java.vm.info mixed mode
java.vm.name Java HotSpot(TM) Server VM
java.vm.specification.name Java Virtual Machine Specification
java.vm.specification.vendor Sun Microsystems Inc.
java.vm.specification.version 1.0
java.vm.vendor Sun Microsystems Inc.
java.vm.version 10.0-b19
jna.platform.library.path /usr/lib:/lib
line.separator
os.arch i386
os.name Linux
os.version 2.6.17-1.2157_FC3asl.5smp
path.separator :
sun.arch.data.model 32
sun.boot.class.path /usr/java/jdk1.6.0_05/jre/lib/resources.jar:/usr/java/jdk1.6.0_05/jre/lib/rt.jar:/usr/java/jdk1.6.0_05/jre/lib/sunrsasign.jar:/usr/java/jdk1.6.0_05/jre/lib/jsse.jar:/usr/java/jdk1.6.0_05/jre/lib/jce.jar:/usr/java/jdk1.6.0_05/jre/lib/charsets.jar:/usr/java/jdk1.6.0_05/jre/classes
sun.boot.library.path /usr/java/jdk1.6.0_05/jre/lib/i386
sun.cpu.endian little
sun.cpu.isalist
sun.io.unicode.encoding UnicodeLittle
sun.java.launcher SUN_STANDARD
sun.jnu.encoding ANSI_X3.4-1968
sun.management.compiler HotSpot Tiered Compilers
sun.os.patch.level unknown
svnkit.ssh2.persistent false
svnkit.symlinks false
user.country US
user.dir /NFS/Huson/user/homedir
user.home /NFS/Huson/user/homedir
user.language en
user.name HudsonUser
user.timezone America/New_York

Environment Variables
Name Value
_ /usr/bin/java
ALREADY_SET_PATHS 1
HOME /NFS/Huson/user/homedir
HOST PROBLEM-HOST
LD_LIBRARY_PATH /usr/java/jdk1.6.0_05/jre/lib/i386/server:/usr/java/jdk1.6.0_05/jre/lib/i386:/usr/java/jdk1.6.0_05/jre/../lib/i386:/usr/local/lib:/usr/lib
LESS sMQwh10i
LOGNAME HudsonUser
MAIL /var/spool/mail/HudsonUser
MAILFILE /var/spool/mail/HudsonUser
MANPATH /NFS/Huson/user/homedir/system/man:/usr/local/man:/usr/X11R6/man:/usr/share/man:/usr/man:/usr/lib/perl5/man
METAMAIL_PAGER less
NLSPATH /usr/dt/lib/nls/msg/%L/%N.cat
OLDPWD /NFS/Huson/user/homedir
PAGER less
PATH /NFS/Huson/user/homedir/bin/linux:/NFS/Huson/user/homedir/bin/script:/NFS/Huson/user/homedir/bin:/net/dotfiles/bin:/usr/local/bin:/usr/X11R6/bin:/usr/bin:/bin:/sbin:/usr/sbin
PERLLIB
PGPPATH /NFS/Huson/user/homedir/.pgp
PRINTER prn
PWD /NFS/Huson/user/homedir
SHELL /bin/zsh
SHLVL 0
SSH_CLIENT ::ffff:10.4.1.103 34375 22
SSH_CONNECTION ::ffff:10.4.1.103 34375 ::ffff:10.4.2.134 22
TMPDIR /tmp
USER HudsonUser
XFILESEARCHPATH /usr/dt/app-defaults/%L/Dt

Thread Dump
pool-1-thread-52

"pool-1-thread-52" Id=198 RUNNABLE
        at sun.management.ThreadImpl.dumpThreads0(Native Method)
        at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:359)
        at hudson.Functions.getThreadInfos(Functions.java:635)
        at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:47)
        at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:43)
        at hudson.remoting.UserRequest.perform(UserRequest.java:69)
        at hudson.remoting.UserRequest.perform(UserRequest.java:23)
        at hudson.remoting.Request$2.run(Request.java:200)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

        Number of locked synchronizers = 1
        - java.util.concurrent.locks.ReentrantLock$NonfairSync@1f37d5


Channel reader thread: channel

"Channel reader thread: channel" Id=9 RUNNABLE (in native)
        at java.io.FileInputStream.readBytes(Native Method)
        at java.io.FileInputStream.read(FileInputStream.java:199)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
        -  locked java.io.BufferedInputStream@15e1229
        at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249)
        at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2542)
        at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2552)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
        at hudson.remoting.Channel$ReaderThread.run(Channel.java:574)


Signal Dispatcher

"Signal Dispatcher" Id=4 RUNNABLE


Finalizer

"Finalizer" Id=3 WAITING on java.lang.ref.ReferenceQueue$Lock@fa7ade
        at java.lang.Object.wait(Native Method)
        -  waiting on java.lang.ref.ReferenceQueue$Lock@fa7ade
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)


Reference Handler

"Reference Handler" Id=2 WAITING on java.lang.ref.Reference$Lock@98896c
        at java.lang.Object.wait(Native Method)
        -  waiting on java.lang.ref.Reference$Lock@98896c
        at java.lang.Object.wait(Object.java:485)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)


main

"main" Id=1 WAITING on hudson.remoting.Channel@664d41
        at java.lang.Object.wait(Native Method)
        -  waiting on hudson.remoting.Channel@664d41
        at java.lang.Object.wait(Object.java:485)
        at hudson.remoting.Channel.join(Channel.java:477)
        at hudson.remoting.Launcher.main(Launcher.java:69)
        at hudson.remoting.Launcher.main(Launcher.java:43)

     ****************************************************************

Kohsuke Kawaguchi writes:
 >
 > I suppose "java.net.NoRouteToHostException: No route to host" exception
 > is reported but the build hangs after that?
 >
 > When it hangs again, can you get us the thread dump on the server and
 > the slave? You can go to http://server/hudson/threadDump and
 > http://server/computer/YOURSLAVENAME/systemInfo to get those.
 >
 > Tom Cavin wrote:
 > > Hi Folks,
 > >
 > > I'm seeing a problem where hudson (v1.237) jobs hang until killed if there
 > > is a network problem during svn polling.
 > >
 > > The network I'm on is currently congested with the sporadic problem that
 > > arp packets sometimes collide causing ssh connections to occasionally get
 > > "no route to host" messages instead of connections.  A retry a few seconds
 > > later will be fine, but the momentary glitch does block the first attempt.
 > >
 > > My problem with Hudson is that I have a small job polling my svn repository
 > > every 5 minutes.  Most of the time, things go as I expect, but when a
 > > network glitch coincides with a hudson svn poll or a hudson svn update, I
 > > get the following error message and the build hangs.  The last time this
 > > happened, the normal 30 second job was at over 4 hours and counting when I
 > > killed the hudson master and restarted.
 > >
 > > What I'd like to see is a simple failure or abort, followed by a retry at
 > > a later time.
 > >
 > > Is this a known bug?
 > >
 > > Best Wishes,
 > >
 > >      --Tom
 > >
 > > Buildbot Hudson writes:
 > >  > See http://HUDSON_MASTER:8080/job/testBlame/393/changes
 > >  >
 > >  > ------------------------------------------
 > >  > started
 > >  > Building remotely on HUDSON_SLAVE_HOST
 > >  > Updating svn+ssh://REPOSITORY_HOST/REPOSITORY_PATH
 > >  > ERROR: Failed to update svn+ssh://REPOSITORY_HOST/REPOSITORY_PATH
 > >  > org.tmatesoft.svn.core.SVNException: svn: Cannot connect to 'svn+ssh://REPOSITORY_HOST': There was a problem while connecting to REPOSITORY_HOST:22
 > >  > at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:69)
 > >  > at org.tmatesoft.svn.core.internal.io.svn.SVNSSHSession.openConnection(SVNSSHSession.java:281)
 > >  > at org.tmatesoft.svn.core.internal.io.svn.SVNSSHSession.getConnection(SVNSSHSession.java:85)
 > >  > at org.tmatesoft.svn.core.internal.io.svn.SVNSSHConnector.open(SVNSSHConnector.java:74)
 > >  > at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.open(SVNConnection.java:60)
 > >  > at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.openConnection(SVNRepositoryImpl.java:974)
 > >  > at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.getDatedRevision(SVNRepositoryImpl.java:159)
 > >  > at org.tmatesoft.svn.core.wc.SVNBasicClient.getRevisionNumber(SVNBasicClient.java:346)
 > >  > at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:159)
 > >  > at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:387)
 > >  > at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:359)
 > >  > at hudson.FilePath$FileCallableWrapper.call(FilePath.java:1122)
 > >  > at hudson.remoting.UserRequest.perform(UserRequest.java:69)
 > >  > at hudson.remoting.UserRequest.perform(UserRequest.java:23)
 > >  > at hudson.remoting.Request$2.run(Request.java:200)
 > >  > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
 > >  > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
 > >  > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 > >  > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
 > >  > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
 > >  > at java.lang.Thread.run(Thread.java:619)
 > >  > Caused by: java.io.IOException: There was a problem while connecting to REPOSITORY_HOST:22
 > >  > at com.trilead.ssh2.Connection.connect(Connection.java:753)
 > >  > at com.trilead.ssh2.Connection.connect(Connection.java:544)
 > >  > at org.tmatesoft.svn.core.internal.io.svn.SVNSSHSession.openConnection(SVNSSHSession.java:241)
 > >  > ... 19 more
 > >  > Caused by: java.net.NoRouteToHostException: No route to host
 > >  > at java.net.PlainSocketImpl.socketConnect(Native Method)
 > >  > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
 > >  > at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
 > >  > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
 > >  > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
 > >  > at java.net.Socket.connect(Socket.java:519)
 > >  > at com.trilead.ssh2.transport.TransportManager.establishConnection(TransportManager.java:341)
 > >  > at com.trilead.ssh2.transport.TransportManager.initialize(TransportManager.java:449)
 > >  > at com.trilead.ssh2.Connection.connect(Connection.java:697)
 > >  > ... 21 more
 > >  > Recording fingerprints
 > >  >
 > >
 > > ---------------------------------------------------------------------
 > > To unsubscribe, e-mail: [hidden email]
 > > For additional commands, e-mail: [hidden email]
 > >
 > >
 >
 >
 > --
 > Kohsuke Kawaguchi
 > Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: build hang after temporary network failure

Kohsuke Kawaguchi
Administrator

Thanks. The stack trace suggests there's no hang, actually. The build
was marked as a failure, and no thread is executing that any more. I see
that you only have 1 executor for the PROBLEM-HOST, and that executor is
now available, waiting for the next job.

So when you say a build is hanging, I assume you still see a progress
bar in this executor, and the status ball for this build is still flashing?

Something doesn't add up here...


Tom Cavin wrote:

> Hi Kohsuke,
>
> It happened again, so here is the information requested.
>
> The server is actually running two jobs and using two executors.  The main
> production job is still running, but the minor test job (poll repository,
> update checkout, record file checksums and save in reference file,
> fingerprint and archive reference file, repeat ...) has failed and has tied
> up one of my executors.
>
> Information follows as:
>
>   failed Job console output (sanitized)
>      ****************************************************************
>     thread dump of server (sanitized)
>      ****************************************************************
>     system info of client (sanitized)
>      ****************************************************************
>
> Please let me know if you need more information.  Since the production job
> seems to be running and polling properly, I can probably leave the hung
> executor in its current state for a few hours.
>
> Best Wishes,
>
>      --Tom
>
>      ****************************************************************
>   failed Job console output (sanitized)
> started
> Building remotely on PROBLEM-HOST
> Updating svn+ssh://SVN_REPOSITORY_HOST/Path/to/Code
> ERROR: Failed to update svn+ssh://SVN_REPOSITORY_HOST/Path/to/Code
> org.tmatesoft.svn.core.SVNException: svn: Cannot connect to 'svn+ssh://SVN_REPOSITORY_HOST': There was a problem while connecting to SVN_REPOSITORY_HOST:22
> at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:69)
> at org.tmatesoft.svn.core.internal.io.svn.SVNSSHSession.openConnection(SVNSSHSession.java:281)
> at org.tmatesoft.svn.core.internal.io.svn.SVNSSHSession.getConnection(SVNSSHSession.java:85)
> at org.tmatesoft.svn.core.internal.io.svn.SVNSSHConnector.open(SVNSSHConnector.java:74)
> at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.open(SVNConnection.java:60)
> at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.openConnection(SVNRepositoryImpl.java:974)
> at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.getDatedRevision(SVNRepositoryImpl.java:159)
> at org.tmatesoft.svn.core.wc.SVNBasicClient.getRevisionNumber(SVNBasicClient.java:346)
> at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:159)
> at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:387)
> at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:359)
> at hudson.FilePath$FileCallableWrapper.call(FilePath.java:1122)
> at hudson.remoting.UserRequest.perform(UserRequest.java:69)
> at hudson.remoting.UserRequest.perform(UserRequest.java:23)
> at hudson.remoting.Request$2.run(Request.java:200)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: There was a problem while connecting to SVN_REPOSITORY_HOST:22
> at com.trilead.ssh2.Connection.connect(Connection.java:753)
> at com.trilead.ssh2.Connection.connect(Connection.java:544)
> at org.tmatesoft.svn.core.internal.io.svn.SVNSSHSession.openConnection(SVNSSHSession.java:241)
> ... 19 more
> Caused by: java.net.NoRouteToHostException: No route to host
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:519)
> at com.trilead.ssh2.transport.TransportManager.establishConnection(TransportManager.java:341)
> at com.trilead.ssh2.transport.TransportManager.initialize(TransportManager.java:449)
> at com.trilead.ssh2.Connection.connect(Connection.java:697)
> ... 21 more
> Recording fingerprints
> Sending e-mails to: Me@my-internal-email
> finished: FAILURE
>
>      ****************************************************************
>     thread dump of server (sanitized)
>   Thread Dump
>     Thread[RequestHandlerThread[#3],5,main] (WAITING)
>       java.lang.Object.wait(Native Method)
>       java.lang.Object.wait(Object.java:474)
>       winstone.RequestHandlerThread.run(RequestHandlerThread.java:216)
>       java.lang.Thread.run(Thread.java:595)
>     Thread[Executor #0 for UNRELATED-DEAD-HOST,5,main] (TIMED_WAITING)
>       java.lang.Object.wait(Native Method)
>       hudson.util.OneShotEvent.block(OneShotEvent.java:40)
>       hudson.model.Queue.pop(Queue.java:478)
>       hudson.model.Executor.run(Executor.java:72)
>     Thread[Hudson cron thread,5,main] (TIMED_WAITING)
>       java.lang.Object.wait(Native Method)
>       java.util.TimerThread.mainLoop(Timer.java:509)
>       java.util.TimerThread.run(Timer.java:462)
>     Thread[DestroyJavaVM,5,main] (RUNNABLE)
>     Thread[Executor #0 for PROBLEM-HOST,5,main] (WAITING)
>       java.lang.Object.wait(Native Method)
>       java.lang.Object.wait(Object.java:474)
>       hudson.util.OneShotEvent.block(OneShotEvent.java:28)
>       hudson.model.Queue.pop(Queue.java:476)
>       hudson.model.Executor.run(Executor.java:72)
>     Thread[RequestHandlerThread[#0],5,main] (WAITING)
>       java.lang.Object.wait(Native Method)
>       java.lang.Object.wait(Object.java:474)
>       winstone.RequestHandlerThread.run(RequestHandlerThread.java:216)
>       java.lang.Thread.run(Thread.java:595)
>     Thread[RequestHandlerThread[#1],5,main] (WAITING)
>       java.lang.Object.wait(Native Method)
>       java.lang.Object.wait(Object.java:474)
>       winstone.RequestHandlerThread.run(RequestHandlerThread.java:216)
>       java.lang.Thread.run(Thread.java:595)
>     Thread[stderr copier for remote agent on PROBLEM-HOST,5,main] (RUNNABLE)
>       java.io.FileInputStream.readBytes(Native Method)
>       java.io.FileInputStream.read(FileInputStream.java:177)
>       hudson.util.StreamCopyThread.run(StreamCopyThread.java:27)
>     Thread[RequestHandlerThread[#6],5,main] (RUNNABLE)
>       java.lang.Thread.dumpThreads(Native Method)
>       java.lang.Thread.getAllStackTraces(Thread.java:1434)
>       hudson.Functions.dumpAllThreads(Functions.java:630)
>       sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>       sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>       sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>       java.lang.reflect.Method.invoke(Method.java:585)
>       org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:259)
>       org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104)
>       org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
>       org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
>       org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
>       org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
>       hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:71)
>       org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61)
>       org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsIterator(ExpressionSupport.java:94)
>       org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:89)
>       org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
>       org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
>       org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
>       org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41)
>       org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
>       org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
>       org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
>       org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)
>       org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
>       org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
>       org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
>       org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
>       org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
>       org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
>       org.apache.commons.jelly.tags.core.IfTag.doTag(IfTag.java:42)
>       org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
>       org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
>       org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
>       org.apache.commons.jelly.tags.core.JellyTag.doTag(JellyTag.java:45)
>       org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
>       org.apache.commons.jelly.impl.DynamicTag.doTag(DynamicTag.java:80)
>       org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
>       org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
>       org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
>       org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
>       org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
>       org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
>       org.apache.commons.jelly.impl.StaticTag.doTag(StaticTag.java:65)
>       org.apache.commons.jelly.impl.StaticTagScript.run(StaticTagScript.java:112)
>       org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
>       org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
>       org.apache.commons.jelly.impl.StaticTag.doTag(StaticTag.java:65)
>       org.apache.commons.jelly.impl.StaticTagScript.run(StaticTagScript.java:112)
>       org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
>       org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
>       org.apache.commons.jelly.impl.StaticTag.doTag(StaticTag.java:65)
>       org.apache.commons.jelly.impl.StaticTagScript.run(StaticTagScript.java:112)
>       org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
>       org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
>       org.apache.commons.jelly.impl.StaticTag.doTag(StaticTag.java:65)
>       org.apache.commons.jelly.impl.StaticTagScript.run(StaticTagScript.java:112)
>       org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
>       org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
>       org.apache.commons.jelly.impl.StaticTag.doTag(StaticTag.java:65)
>       org.apache.commons.jelly.impl.StaticTagScript.run(StaticTagScript.java:112)
>       org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
>       org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
>       org.apache.commons.jelly.tags.core.JellyTag.doTag(JellyTag.java:45)
>       org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
>       org.apache.commons.jelly.impl.DynamicTag.doTag(DynamicTag.java:80)
>       org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
>       org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
>       org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
>       org.apache.commons.jelly.tags.core.JellyTag.doTag(JellyTag.java:45)
>       org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
>       org.kohsuke.stapler.jelly.JellyClassTearOff.invokeScript(JellyClassTearOff.java:78)
>       org.kohsuke.stapler.MetaClass$3.dispatch(MetaClass.java:125)
>       org.kohsuke.stapler.Stapler.invoke(Stapler.java:440)
>       org.kohsuke.stapler.Stapler.invoke(Stapler.java:361)
>       org.kohsuke.stapler.Stapler.service(Stapler.java:121)
>       javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
>       winstone.ServletConfiguration.execute(ServletConfiguration.java:249)
>       winstone.RequestDispatcher.forward(RequestDispatcher.java:335)
>       winstone.RequestDispatcher.doFilter(RequestDispatcher.java:378)
>       hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:53)
>       hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:45)
>       winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
>       winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
>       hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:52)
>       hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:28)
>       hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:55)
>       org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)
>       hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:55)
>       org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
>       hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:55)
>       org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
>       hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:55)
>       org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
>       hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:55)
>       org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
>       hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:55)
>       org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
>       hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:42)
>       hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:55)
>       hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:44)
>       hudson.security.HudsonFilter.doFilter(HudsonFilter.java:85)
>       winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
>       winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
>       winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
>       winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:244)
>       winstone.RequestHandlerThread.run(RequestHandlerThread.java:150)
>       java.lang.Thread.run(Thread.java:595)
>     Thread[WinstoneHostConfigurationMgmt:default,5,main] (TIMED_WAITING)
>       java.lang.Thread.sleep(Native Method)
>       winstone.HostConfiguration.run(HostConfiguration.java:176)
>       java.lang.Thread.run(Thread.java:595)
>     Thread[process reaper,5,main] (RUNNABLE)
>       java.lang.UNIXProcess.waitForProcessExit(Native Method)
>       java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
>       java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
>     Thread[Reference Handler,10,system] (WAITING)
>       java.lang.Object.wait(Native Method)
>       java.lang.Object.wait(Object.java:474)
>       java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>     Thread[Finalizer,8,system] (WAITING)
>       java.lang.Object.wait(Native Method)
>       java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>       java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>       java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>     Thread[Thread-2,5,main] (TIMED_WAITING)
>       java.lang.Object.wait(Native Method)
>       hudson.model.ViewJob$ReloadThread.getNext(ViewJob.java:136)
>       hudson.model.ViewJob$ReloadThread.run(ViewJob.java:152)
>     Thread[Channel reader thread: PROBLEM-HOST,5,main] (RUNNABLE)
>       java.io.FileInputStream.readBytes(Native Method)
>       java.io.FileInputStream.read(FileInputStream.java:194)
>       java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>       java.io.BufferedInputStream.read(BufferedInputStream.java:235)
>       java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2200)
>       java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2490)
>       java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2500)
>       java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1267)
>       java.io.ObjectInputStream.readObject(ObjectInputStream.java:339)
>       hudson.remoting.Channel$ReaderThread.run(Channel.java:576)
>     Thread[RequestHandlerThread[#5],5,main] (WAITING)
>       java.lang.Object.wait(Native Method)
>       java.lang.Object.wait(Object.java:474)
>       winstone.RequestHandlerThread.run(RequestHandlerThread.java:216)
>       java.lang.Thread.run(Thread.java:595)
>     Thread[process reaper,5,main] (RUNNABLE)
>       java.lang.UNIXProcess.waitForProcessExit(Native Method)
>       java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
>       java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
>     Thread[ConnectorThread:[ajp13-8009],5,main] (RUNNABLE)
>       java.net.PlainSocketImpl.socketAccept(Native Method)
>       java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>       java.net.ServerSocket.implAccept(ServerSocket.java:450)
>       java.net.ServerSocket.accept(ServerSocket.java:421)
>       winstone.ajp13.Ajp13Listener.run(Ajp13Listener.java:111)
>       java.lang.Thread.run(Thread.java:595)
>     Thread[RequestHandlerThread[#4],5,main] (WAITING)
>       java.lang.Object.wait(Native Method)
>       java.lang.Object.wait(Object.java:474)
>       winstone.RequestHandlerThread.run(RequestHandlerThread.java:216)
>       java.lang.Thread.run(Thread.java:595)
>     Thread[pool-2-thread-301,5,main] (TIMED_WAITING)
>       sun.misc.Unsafe.park(Native Method)
>       java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
>       java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:772)
>       java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1087)
>       java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
>       java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
>       java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
>       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
>       java.lang.Thread.run(Thread.java:595)
>     Thread[TCP slave agent listener port=0,5,main] (RUNNABLE)
>       java.net.PlainSocketImpl.socketAccept(Native Method)
>       java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>       java.net.ServerSocket.implAccept(ServerSocket.java:450)
>       java.net.ServerSocket.accept(ServerSocket.java:421)
>       hudson.TcpSlaveAgentListener.run(TcpSlaveAgentListener.java:82)
>     Thread[ConnectorThread:[http-8080],5,main] (RUNNABLE)
>       java.net.PlainSocketImpl.socketAccept(Native Method)
>       java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>       java.net.ServerSocket.implAccept(ServerSocket.java:450)
>       java.net.ServerSocket.accept(ServerSocket.java:421)
>       winstone.HttpListener.run(HttpListener.java:127)
>       java.lang.Thread.run(Thread.java:595)
>     Thread[RequestHandlerThread[#7],5,main] (WAITING)
>       java.lang.Object.wait(Native Method)
>       java.lang.Object.wait(Object.java:474)
>       winstone.RequestHandlerThread.run(RequestHandlerThread.java:216)
>       java.lang.Thread.run(Thread.java:595)
>     Thread[LauncherControlThread[ControlPort=-1],5,main] (TIMED_WAITING)
>       java.lang.Thread.sleep(Native Method)
>       winstone.Launcher.run(Launcher.java:279)
>       java.lang.Thread.run(Thread.java:595)
>     Thread[Signal Dispatcher,10,system] (RUNNABLE)
>     Thread[RequestHandlerThread[#2],5,main] (WAITING)
>       java.lang.Object.wait(Native Method)
>       java.lang.Object.wait(Object.java:474)
>       winstone.RequestHandlerThread.run(RequestHandlerThread.java:216)
>       java.lang.Thread.run(Thread.java:595)
>     Thread[Executor #0 for WORKING-HOST,5,main] (WAITING)
>       java.lang.Object.wait(Native Method)
>       java.lang.Object.wait(Object.java:474)
>       hudson.util.OneShotEvent.block(OneShotEvent.java:28)
>       hudson.model.Queue.pop(Queue.java:476)
>       hudson.model.Executor.run(Executor.java:72)
>     Thread[stderr copier for remote agent on WORKING-HOST,5,main] (RUNNABLE)
>       java.io.FileInputStream.readBytes(Native Method)
>       java.io.FileInputStream.read(FileInputStream.java:177)
>       hudson.util.StreamCopyThread.run(StreamCopyThread.java:27)
>     Thread[Channel reader thread: WORKING-HOST,5,main] (RUNNABLE)
>       java.io.FileInputStream.readBytes(Native Method)
>       java.io.FileInputStream.read(FileInputStream.java:194)
>       java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>       java.io.BufferedInputStream.read(BufferedInputStream.java:235)
>       java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2200)
>       java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2490)
>       java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2500)
>       java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1267)
>       java.io.ObjectInputStream.readObject(ObjectInputStream.java:339)
>       hudson.remoting.Channel$ReaderThread.run(Channel.java:576)
>     Thread[WinstoneObjectPoolMgmt,5,main] (TIMED_WAITING)
>       java.lang.Thread.sleep(Native Method)
>       winstone.ObjectPool.run(ObjectPool.java:103)
>       java.lang.Thread.run(Thread.java:595)
>      ****************************************************************
>     system info of client (sanitized)
> System Properties
> Name Value  
> file.encoding ANSI_X3.4-1968
> file.encoding.pkg sun.io
> file.separator /
> java.awt.graphicsenv sun.awt.X11GraphicsEnvironment
> java.awt.printerjob sun.print.PSPrinterJob
> java.class.path /NFS/PATH/to/Hudson-home/WEB-INF/slave.jar
> java.class.version 50.0
> java.endorsed.dirs /usr/java/jdk1.6.0_05/jre/lib/endorsed
> java.ext.dirs /usr/java/jdk1.6.0_05/jre/lib/ext:/usr/java/packages/lib/ext
> java.home /usr/java/jdk1.6.0_05/jre
> java.io.tmpdir /tmp
> java.library.path /usr/java/jdk1.6.0_05/jre/lib/i386/server:/usr/java/jdk1.6.0_05/jre/lib/i386:/usr/java/jdk1.6.0_05/jre/../lib/i386:/usr/local/lib:/usr/lib:/usr/java/packages/lib/i386:/lib:/usr/lib
> java.runtime.name Java(TM) SE Runtime Environment
> java.runtime.version 1.6.0_05-b13
> java.specification.name Java Platform API Specification
> java.specification.vendor Sun Microsystems Inc.
> java.specification.version 1.6
> java.vendor Sun Microsystems Inc.
> java.vendor.url http://java.sun.com/
> java.vendor.url.bug http://java.sun.com/cgi-bin/bugreport.cgi
> java.version 1.6.0_05
> java.vm.info mixed mode
> java.vm.name Java HotSpot(TM) Server VM
> java.vm.specification.name Java Virtual Machine Specification
> java.vm.specification.vendor Sun Microsystems Inc.
> java.vm.specification.version 1.0
> java.vm.vendor Sun Microsystems Inc.
> java.vm.version 10.0-b19
> jna.platform.library.path /usr/lib:/lib
> line.separator
> os.arch i386
> os.name Linux
> os.version 2.6.17-1.2157_FC3asl.5smp
> path.separator :
> sun.arch.data.model 32
> sun.boot.class.path /usr/java/jdk1.6.0_05/jre/lib/resources.jar:/usr/java/jdk1.6.0_05/jre/lib/rt.jar:/usr/java/jdk1.6.0_05/jre/lib/sunrsasign.jar:/usr/java/jdk1.6.0_05/jre/lib/jsse.jar:/usr/java/jdk1.6.0_05/jre/lib/jce.jar:/usr/java/jdk1.6.0_05/jre/lib/charsets.jar:/usr/java/jdk1.6.0_05/jre/classes
> sun.boot.library.path /usr/java/jdk1.6.0_05/jre/lib/i386
> sun.cpu.endian little
> sun.cpu.isalist
> sun.io.unicode.encoding UnicodeLittle
> sun.java.launcher SUN_STANDARD
> sun.jnu.encoding ANSI_X3.4-1968
> sun.management.compiler HotSpot Tiered Compilers
> sun.os.patch.level unknown
> svnkit.ssh2.persistent false
> svnkit.symlinks false
> user.country US
> user.dir /NFS/Huson/user/homedir
> user.home /NFS/Huson/user/homedir
> user.language en
> user.name HudsonUser
> user.timezone America/New_York
>
> Environment Variables
> Name Value
> _ /usr/bin/java
> ALREADY_SET_PATHS 1
> HOME /NFS/Huson/user/homedir
> HOST PROBLEM-HOST
> LD_LIBRARY_PATH /usr/java/jdk1.6.0_05/jre/lib/i386/server:/usr/java/jdk1.6.0_05/jre/lib/i386:/usr/java/jdk1.6.0_05/jre/../lib/i386:/usr/local/lib:/usr/lib
> LESS sMQwh10i
> LOGNAME HudsonUser
> MAIL /var/spool/mail/HudsonUser
> MAILFILE /var/spool/mail/HudsonUser
> MANPATH /NFS/Huson/user/homedir/system/man:/usr/local/man:/usr/X11R6/man:/usr/share/man:/usr/man:/usr/lib/perl5/man
> METAMAIL_PAGER less
> NLSPATH /usr/dt/lib/nls/msg/%L/%N.cat
> OLDPWD /NFS/Huson/user/homedir
> PAGER less
> PATH /NFS/Huson/user/homedir/bin/linux:/NFS/Huson/user/homedir/bin/script:/NFS/Huson/user/homedir/bin:/net/dotfiles/bin:/usr/local/bin:/usr/X11R6/bin:/usr/bin:/bin:/sbin:/usr/sbin
> PERLLIB
> PGPPATH /NFS/Huson/user/homedir/.pgp
> PRINTER prn
> PWD /NFS/Huson/user/homedir
> SHELL /bin/zsh
> SHLVL 0
> SSH_CLIENT ::ffff:10.4.1.103 34375 22
> SSH_CONNECTION ::ffff:10.4.1.103 34375 ::ffff:10.4.2.134 22
> TMPDIR /tmp
> USER HudsonUser
> XFILESEARCHPATH /usr/dt/app-defaults/%L/Dt
>
> Thread Dump
> pool-1-thread-52
>
> "pool-1-thread-52" Id=198 RUNNABLE
> at sun.management.ThreadImpl.dumpThreads0(Native Method)
> at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:359)
> at hudson.Functions.getThreadInfos(Functions.java:635)
> at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:47)
> at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:43)
> at hudson.remoting.UserRequest.perform(UserRequest.java:69)
> at hudson.remoting.UserRequest.perform(UserRequest.java:23)
> at hudson.remoting.Request$2.run(Request.java:200)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
>
> Number of locked synchronizers = 1
> - java.util.concurrent.locks.ReentrantLock$NonfairSync@1f37d5
>
>
> Channel reader thread: channel
>
> "Channel reader thread: channel" Id=9 RUNNABLE (in native)
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:199)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
> -  locked java.io.BufferedInputStream@15e1229
> at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249)
> at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2542)
> at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2552)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
> at hudson.remoting.Channel$ReaderThread.run(Channel.java:574)
>
>
> Signal Dispatcher
>
> "Signal Dispatcher" Id=4 RUNNABLE
>
>
> Finalizer
>
> "Finalizer" Id=3 WAITING on java.lang.ref.ReferenceQueue$Lock@fa7ade
> at java.lang.Object.wait(Native Method)
> -  waiting on java.lang.ref.ReferenceQueue$Lock@fa7ade
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
>
> Reference Handler
>
> "Reference Handler" Id=2 WAITING on java.lang.ref.Reference$Lock@98896c
> at java.lang.Object.wait(Native Method)
> -  waiting on java.lang.ref.Reference$Lock@98896c
> at java.lang.Object.wait(Object.java:485)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>
>
> main
>
> "main" Id=1 WAITING on hudson.remoting.Channel@664d41
> at java.lang.Object.wait(Native Method)
> -  waiting on hudson.remoting.Channel@664d41
> at java.lang.Object.wait(Object.java:485)
> at hudson.remoting.Channel.join(Channel.java:477)
> at hudson.remoting.Launcher.main(Launcher.java:69)
> at hudson.remoting.Launcher.main(Launcher.java:43)
>
>      ****************************************************************
>
> Kohsuke Kawaguchi writes:
>  >
>  > I suppose "java.net.NoRouteToHostException: No route to host" exception
>  > is reported but the build hangs after that?
>  >
>  > When it hangs again, can you get us the thread dump on the server and
>  > the slave? You can go to http://server/hudson/threadDump and
>  > http://server/computer/YOURSLAVENAME/systemInfo to get those.
>  >
>  > Tom Cavin wrote:
>  > > Hi Folks,
>  > >
>  > > I'm seeing a problem where hudson (v1.237) jobs hang until killed if there
>  > > is a network problem during svn polling.
>  > >
>  > > The network I'm on is currently congested with the sporadic problem that
>  > > arp packets sometimes collide causing ssh connections to occasionally get
>  > > "no route to host" messages instead of connections.  A retry a few seconds
>  > > later will be fine, but the momentary glitch does block the first attempt.
>  > >
>  > > My problem with Hudson is that I have a small job polling my svn repository
>  > > every 5 minutes.  Most of the time, things go as I expect, but when a
>  > > network glitch coincides with a hudson svn poll or a hudson svn update, I
>  > > get the following error message and the build hangs.  The last time this
>  > > happened, the normal 30 second job was at over 4 hours and counting when I
>  > > killed the hudson master and restarted.
>  > >
>  > > What I'd like to see is a simple failure or abort, followed by a retry at
>  > > a later time.
>  > >
>  > > Is this a known bug?
>  > >
>  > > Best Wishes,
>  > >
>  > >      --Tom
>  > >
>  > > Buildbot Hudson writes:
>  > >  > See http://HUDSON_MASTER:8080/job/testBlame/393/changes
>  > >  >
>  > >  > ------------------------------------------
>  > >  > started
>  > >  > Building remotely on HUDSON_SLAVE_HOST
>  > >  > Updating svn+ssh://REPOSITORY_HOST/REPOSITORY_PATH
>  > >  > ERROR: Failed to update svn+ssh://REPOSITORY_HOST/REPOSITORY_PATH
>  > >  > org.tmatesoft.svn.core.SVNException: svn: Cannot connect to 'svn+ssh://REPOSITORY_HOST': There was a problem while connecting to REPOSITORY_HOST:22
>  > >  > at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:69)
>  > >  > at org.tmatesoft.svn.core.internal.io.svn.SVNSSHSession.openConnection(SVNSSHSession.java:281)
>  > >  > at org.tmatesoft.svn.core.internal.io.svn.SVNSSHSession.getConnection(SVNSSHSession.java:85)
>  > >  > at org.tmatesoft.svn.core.internal.io.svn.SVNSSHConnector.open(SVNSSHConnector.java:74)
>  > >  > at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.open(SVNConnection.java:60)
>  > >  > at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.openConnection(SVNRepositoryImpl.java:974)
>  > >  > at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.getDatedRevision(SVNRepositoryImpl.java:159)
>  > >  > at org.tmatesoft.svn.core.wc.SVNBasicClient.getRevisionNumber(SVNBasicClient.java:346)
>  > >  > at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:159)
>  > >  > at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:387)
>  > >  > at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:359)
>  > >  > at hudson.FilePath$FileCallableWrapper.call(FilePath.java:1122)
>  > >  > at hudson.remoting.UserRequest.perform(UserRequest.java:69)
>  > >  > at hudson.remoting.UserRequest.perform(UserRequest.java:23)
>  > >  > at hudson.remoting.Request$2.run(Request.java:200)
>  > >  > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>  > >  > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>  > >  > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>  > >  > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>  > >  > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  > >  > at java.lang.Thread.run(Thread.java:619)
>  > >  > Caused by: java.io.IOException: There was a problem while connecting to REPOSITORY_HOST:22
>  > >  > at com.trilead.ssh2.Connection.connect(Connection.java:753)
>  > >  > at com.trilead.ssh2.Connection.connect(Connection.java:544)
>  > >  > at org.tmatesoft.svn.core.internal.io.svn.SVNSSHSession.openConnection(SVNSSHSession.java:241)
>  > >  > ... 19 more
>  > >  > Caused by: java.net.NoRouteToHostException: No route to host
>  > >  > at java.net.PlainSocketImpl.socketConnect(Native Method)
>  > >  > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
>  > >  > at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
>  > >  > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
>  > >  > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
>  > >  > at java.net.Socket.connect(Socket.java:519)
>  > >  > at com.trilead.ssh2.transport.TransportManager.establishConnection(TransportManager.java:341)
>  > >  > at com.trilead.ssh2.transport.TransportManager.initialize(TransportManager.java:449)
>  > >  > at com.trilead.ssh2.Connection.connect(Connection.java:697)
>  > >  > ... 21 more
>  > >  > Recording fingerprints
>  > >  >
>  > >
>  > > ---------------------------------------------------------------------
>  > > To unsubscribe, e-mail: [hidden email]
>  > > For additional commands, e-mail: [hidden email]
>  > >
>  > >
>  >
>  >
>  > --
>  > Kohsuke Kawaguchi
>  > Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

--
Kohsuke Kawaguchi
Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: build hang after temporary network failure

tec-3

Hi Kohsuke,

There were two slightly (or significantly) different things that occured.

The first problem that triggered this thread was a build that seemed to
fail to finish that was apparently due to a transient network failure
during svn polling.  In that case a build that normally takes less than 4
minutes was still claiming to be running after a few hours.

This last failure may or may not be the same thing, although it also seems
to have been triggered by the same type of transient network failure.  In
this case, the build failed (which is fine) but then Hudson stopped polling
the svn server and never started new builds even though many commits had
gone in.  (Polling is about every 5 minutes, our repository has averaged a
commit every 9 minutes for over a year, and the builder was idle for at
least a day.)

The two problems may be related because they both involved polling the svn
server, or they might be completely unrelated.  From a build process point
of view, the same cause (transient network failure) triggered a similar
Hudson failure (no new builds after commits).

I'll continue to watch for errors and hangs, and submit further stack
traces when or if they occur.  If you have any further things for me to
check, please let me know.

Best Wishes,

     --Tom

Kohsuke Kawaguchi writes:
 >
 > Thanks. The stack trace suggests there's no hang, actually. The build
 > was marked as a failure, and no thread is executing that any more. I see
 > that you only have 1 executor for the PROBLEM-HOST, and that executor is
 > now available, waiting for the next job.
 >
 > So when you say a build is hanging, I assume you still see a progress
 > bar in this executor, and the status ball for this build is still flashing?
 >
 > Something doesn't add up here...
 >
 >
 > Tom Cavin wrote:
 > > Hi Kohsuke,
 > >
 > > It happened again, so here is the information requested.
 > >
 > > The server is actually running two jobs and using two executors.  The main
 > > production job is still running, but the minor test job (poll repository,
 > > update checkout, record file checksums and save in reference file,
 > > fingerprint and archive reference file, repeat ...) has failed and has tied
 > > up one of my executors.
 > >
 > > Information follows as:
 > >
 > >   failed Job console output (sanitized)
 > >      ****************************************************************
 > >     thread dump of server (sanitized)
 > >      ****************************************************************
 > >     system info of client (sanitized)
 > >      ****************************************************************
 > >
 > > Please let me know if you need more information.  Since the production job
 > > seems to be running and polling properly, I can probably leave the hung
 > > executor in its current state for a few hours.
 > >
 > > Best Wishes,
 > >
 > >      --Tom
 > >
 > >      ****************************************************************
 > >   failed Job console output (sanitized)
 > > started
 > > Building remotely on PROBLEM-HOST
 > > Updating svn+ssh://SVN_REPOSITORY_HOST/Path/to/Code
 > > ERROR: Failed to update svn+ssh://SVN_REPOSITORY_HOST/Path/to/Code
 > > org.tmatesoft.svn.core.SVNException: svn: Cannot connect to 'svn+ssh://SVN_REPOSITORY_HOST': There was a problem while connecting to SVN_REPOSITORY_HOST:22
 > > at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:69)
 > > at org.tmatesoft.svn.core.internal.io.svn.SVNSSHSession.openConnection(SVNSSHSession.java:281)
 > > at org.tmatesoft.svn.core.internal.io.svn.SVNSSHSession.getConnection(SVNSSHSession.java:85)
 > > at org.tmatesoft.svn.core.internal.io.svn.SVNSSHConnector.open(SVNSSHConnector.java:74)
 > > at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.open(SVNConnection.java:60)
 > > at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.openConnection(SVNRepositoryImpl.java:974)
 > > at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.getDatedRevision(SVNRepositoryImpl.java:159)
 > > at org.tmatesoft.svn.core.wc.SVNBasicClient.getRevisionNumber(SVNBasicClient.java:346)
 > > at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:159)
 > > at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:387)
 > > at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:359)
 > > at hudson.FilePath$FileCallableWrapper.call(FilePath.java:1122)
 > > at hudson.remoting.UserRequest.perform(UserRequest.java:69)
 > > at hudson.remoting.UserRequest.perform(UserRequest.java:23)
 > > at hudson.remoting.Request$2.run(Request.java:200)
 > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
 > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
 > > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
 > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
 > > at java.lang.Thread.run(Thread.java:619)
 > > Caused by: java.io.IOException: There was a problem while connecting to SVN_REPOSITORY_HOST:22
 > > at com.trilead.ssh2.Connection.connect(Connection.java:753)
 > > at com.trilead.ssh2.Connection.connect(Connection.java:544)
 > > at org.tmatesoft.svn.core.internal.io.svn.SVNSSHSession.openConnection(SVNSSHSession.java:241)
 > > ... 19 more
 > > Caused by: java.net.NoRouteToHostException: No route to host
 > > at java.net.PlainSocketImpl.socketConnect(Native Method)
 > > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
 > > at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
 > > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
 > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
 > > at java.net.Socket.connect(Socket.java:519)
 > > at com.trilead.ssh2.transport.TransportManager.establishConnection(TransportManager.java:341)
 > > at com.trilead.ssh2.transport.TransportManager.initialize(TransportManager.java:449)
 > > at com.trilead.ssh2.Connection.connect(Connection.java:697)
 > > ... 21 more
 > > Recording fingerprints
 > > Sending e-mails to: Me@my-internal-email
 > > finished: FAILURE
 > >
 > >      ****************************************************************
 > >     thread dump of server (sanitized)
 > >   Thread Dump
 > >     Thread[RequestHandlerThread[#3],5,main] (WAITING)
 > >       java.lang.Object.wait(Native Method)
 > >       java.lang.Object.wait(Object.java:474)
 > >       winstone.RequestHandlerThread.run(RequestHandlerThread.java:216)
 > >       java.lang.Thread.run(Thread.java:595)
 > >     Thread[Executor #0 for UNRELATED-DEAD-HOST,5,main] (TIMED_WAITING)
 > >       java.lang.Object.wait(Native Method)
 > >       hudson.util.OneShotEvent.block(OneShotEvent.java:40)
 > >       hudson.model.Queue.pop(Queue.java:478)
 > >       hudson.model.Executor.run(Executor.java:72)
 > >     Thread[Hudson cron thread,5,main] (TIMED_WAITING)
 > >       java.lang.Object.wait(Native Method)
 > >       java.util.TimerThread.mainLoop(Timer.java:509)
 > >       java.util.TimerThread.run(Timer.java:462)
 > >     Thread[DestroyJavaVM,5,main] (RUNNABLE)
 > >     Thread[Executor #0 for PROBLEM-HOST,5,main] (WAITING)
 > >       java.lang.Object.wait(Native Method)
 > >       java.lang.Object.wait(Object.java:474)
 > >       hudson.util.OneShotEvent.block(OneShotEvent.java:28)
 > >       hudson.model.Queue.pop(Queue.java:476)
 > >       hudson.model.Executor.run(Executor.java:72)
 > >     Thread[RequestHandlerThread[#0],5,main] (WAITING)
 > >       java.lang.Object.wait(Native Method)
 > >       java.lang.Object.wait(Object.java:474)
 > >       winstone.RequestHandlerThread.run(RequestHandlerThread.java:216)
 > >       java.lang.Thread.run(Thread.java:595)
 > >     Thread[RequestHandlerThread[#1],5,main] (WAITING)
 > >       java.lang.Object.wait(Native Method)
 > >       java.lang.Object.wait(Object.java:474)
 > >       winstone.RequestHandlerThread.run(RequestHandlerThread.java:216)
 > >       java.lang.Thread.run(Thread.java:595)
 > >     Thread[stderr copier for remote agent on PROBLEM-HOST,5,main] (RUNNABLE)
 > >       java.io.FileInputStream.readBytes(Native Method)
 > >       java.io.FileInputStream.read(FileInputStream.java:177)
 > >       hudson.util.StreamCopyThread.run(StreamCopyThread.java:27)
 > >     Thread[RequestHandlerThread[#6],5,main] (RUNNABLE)
 > >       java.lang.Thread.dumpThreads(Native Method)
 > >       java.lang.Thread.getAllStackTraces(Thread.java:1434)
 > >       hudson.Functions.dumpAllThreads(Functions.java:630)
 > >       sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 > >       sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 > >       sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 > >       java.lang.reflect.Method.invoke(Method.java:585)
 > >       org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:259)
 > >       org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104)
 > >       org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
 > >       org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
 > >       org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
 > >       org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
 > >       hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:71)
 > >       org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61)
 > >       org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsIterator(ExpressionSupport.java:94)
 > >       org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:89)
 > >       org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
 > >       org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
 > >       org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
 > >       org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41)
 > >       org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
 > >       org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
 > >       org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
 > >       org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)
 > >       org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
 > >       org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
 > >       org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
 > >       org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
 > >       org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
 > >       org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
 > >       org.apache.commons.jelly.tags.core.IfTag.doTag(IfTag.java:42)
 > >       org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
 > >       org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
 > >       org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
 > >       org.apache.commons.jelly.tags.core.JellyTag.doTag(JellyTag.java:45)
 > >       org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
 > >       org.apache.commons.jelly.impl.DynamicTag.doTag(DynamicTag.java:80)
 > >       org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
 > >       org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
 > >       org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
 > >       org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
 > >       org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
 > >       org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
 > >       org.apache.commons.jelly.impl.StaticTag.doTag(StaticTag.java:65)
 > >       org.apache.commons.jelly.impl.StaticTagScript.run(StaticTagScript.java:112)
 > >       org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
 > >       org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
 > >       org.apache.commons.jelly.impl.StaticTag.doTag(StaticTag.java:65)
 > >       org.apache.commons.jelly.impl.StaticTagScript.run(StaticTagScript.java:112)
 > >       org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
 > >       org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
 > >       org.apache.commons.jelly.impl.StaticTag.doTag(StaticTag.java:65)
 > >       org.apache.commons.jelly.impl.StaticTagScript.run(StaticTagScript.java:112)
 > >       org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
 > >       org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
 > >       org.apache.commons.jelly.impl.StaticTag.doTag(StaticTag.java:65)
 > >       org.apache.commons.jelly.impl.StaticTagScript.run(StaticTagScript.java:112)
 > >       org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
 > >       org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
 > >       org.apache.commons.jelly.impl.StaticTag.doTag(StaticTag.java:65)
 > >       org.apache.commons.jelly.impl.StaticTagScript.run(StaticTagScript.java:112)
 > >       org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
 > >       org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
 > >       org.apache.commons.jelly.tags.core.JellyTag.doTag(JellyTag.java:45)
 > >       org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
 > >       org.apache.commons.jelly.impl.DynamicTag.doTag(DynamicTag.java:80)
 > >       org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
 > >       org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
 > >       org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:186)
 > >       org.apache.commons.jelly.tags.core.JellyTag.doTag(JellyTag.java:45)
 > >       org.apache.commons.jelly.impl.TagScript.run(TagScript.java:262)
 > >       org.kohsuke.stapler.jelly.JellyClassTearOff.invokeScript(JellyClassTearOff.java:78)
 > >       org.kohsuke.stapler.MetaClass$3.dispatch(MetaClass.java:125)
 > >       org.kohsuke.stapler.Stapler.invoke(Stapler.java:440)
 > >       org.kohsuke.stapler.Stapler.invoke(Stapler.java:361)
 > >       org.kohsuke.stapler.Stapler.service(Stapler.java:121)
 > >       javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
 > >       winstone.ServletConfiguration.execute(ServletConfiguration.java:249)
 > >       winstone.RequestDispatcher.forward(RequestDispatcher.java:335)
 > >       winstone.RequestDispatcher.doFilter(RequestDispatcher.java:378)
 > >       hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:53)
 > >       hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:45)
 > >       winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
 > >       winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
 > >       hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:52)
 > >       hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:28)
 > >       hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:55)
 > >       org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)
 > >       hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:55)
 > >       org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
 > >       hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:55)
 > >       org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
 > >       hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:55)
 > >       org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
 > >       hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:55)
 > >       org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
 > >       hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:55)
 > >       org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
 > >       hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:42)
 > >       hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:55)
 > >       hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:44)
 > >       hudson.security.HudsonFilter.doFilter(HudsonFilter.java:85)
 > >       winstone.FilterConfiguration.execute(FilterConfiguration.java:195)
 > >       winstone.RequestDispatcher.doFilter(RequestDispatcher.java:368)
 > >       winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
 > >       winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:244)
 > >       winstone.RequestHandlerThread.run(RequestHandlerThread.java:150)
 > >       java.lang.Thread.run(Thread.java:595)
 > >     Thread[WinstoneHostConfigurationMgmt:default,5,main] (TIMED_WAITING)
 > >       java.lang.Thread.sleep(Native Method)
 > >       winstone.HostConfiguration.run(HostConfiguration.java:176)
 > >       java.lang.Thread.run(Thread.java:595)
 > >     Thread[process reaper,5,main] (RUNNABLE)
 > >       java.lang.UNIXProcess.waitForProcessExit(Native Method)
 > >       java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
 > >       java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
 > >     Thread[Reference Handler,10,system] (WAITING)
 > >       java.lang.Object.wait(Native Method)
 > >       java.lang.Object.wait(Object.java:474)
 > >       java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
 > >     Thread[Finalizer,8,system] (WAITING)
 > >       java.lang.Object.wait(Native Method)
 > >       java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
 > >       java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
 > >       java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
 > >     Thread[Thread-2,5,main] (TIMED_WAITING)
 > >       java.lang.Object.wait(Native Method)
 > >       hudson.model.ViewJob$ReloadThread.getNext(ViewJob.java:136)
 > >       hudson.model.ViewJob$ReloadThread.run(ViewJob.java:152)
 > >     Thread[Channel reader thread: PROBLEM-HOST,5,main] (RUNNABLE)
 > >       java.io.FileInputStream.readBytes(Native Method)
 > >       java.io.FileInputStream.read(FileInputStream.java:194)
 > >       java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
 > >       java.io.BufferedInputStream.read(BufferedInputStream.java:235)
 > >       java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2200)
 > >       java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2490)
 > >       java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2500)
 > >       java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1267)
 > >       java.io.ObjectInputStream.readObject(ObjectInputStream.java:339)
 > >       hudson.remoting.Channel$ReaderThread.run(Channel.java:576)
 > >     Thread[RequestHandlerThread[#5],5,main] (WAITING)
 > >       java.lang.Object.wait(Native Method)
 > >       java.lang.Object.wait(Object.java:474)
 > >       winstone.RequestHandlerThread.run(RequestHandlerThread.java:216)
 > >       java.lang.Thread.run(Thread.java:595)
 > >     Thread[process reaper,5,main] (RUNNABLE)
 > >       java.lang.UNIXProcess.waitForProcessExit(Native Method)
 > >       java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
 > >       java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
 > >     Thread[ConnectorThread:[ajp13-8009],5,main] (RUNNABLE)
 > >       java.net.PlainSocketImpl.socketAccept(Native Method)
 > >       java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
 > >       java.net.ServerSocket.implAccept(ServerSocket.java:450)
 > >       java.net.ServerSocket.accept(ServerSocket.java:421)
 > >       winstone.ajp13.Ajp13Listener.run(Ajp13Listener.java:111)
 > >       java.lang.Thread.run(Thread.java:595)
 > >     Thread[RequestHandlerThread[#4],5,main] (WAITING)
 > >       java.lang.Object.wait(Native Method)
 > >       java.lang.Object.wait(Object.java:474)
 > >       winstone.RequestHandlerThread.run(RequestHandlerThread.java:216)
 > >       java.lang.Thread.run(Thread.java:595)
 > >     Thread[pool-2-thread-301,5,main] (TIMED_WAITING)
 > >       sun.misc.Unsafe.park(Native Method)
 > >       java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 > >       java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:772)
 > >       java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1087)
 > >       java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 > >       java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 > >       java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 > >       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 > >       java.lang.Thread.run(Thread.java:595)
 > >     Thread[TCP slave agent listener port=0,5,main] (RUNNABLE)
 > >       java.net.PlainSocketImpl.socketAccept(Native Method)
 > >       java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
 > >       java.net.ServerSocket.implAccept(ServerSocket.java:450)
 > >       java.net.ServerSocket.accept(ServerSocket.java:421)
 > >       hudson.TcpSlaveAgentListener.run(TcpSlaveAgentListener.java:82)
 > >     Thread[ConnectorThread:[http-8080],5,main] (RUNNABLE)
 > >       java.net.PlainSocketImpl.socketAccept(Native Method)
 > >       java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
 > >       java.net.ServerSocket.implAccept(ServerSocket.java:450)
 > >       java.net.ServerSocket.accept(ServerSocket.java:421)
 > >       winstone.HttpListener.run(HttpListener.java:127)
 > >       java.lang.Thread.run(Thread.java:595)
 > >     Thread[RequestHandlerThread[#7],5,main] (WAITING)
 > >       java.lang.Object.wait(Native Method)
 > >       java.lang.Object.wait(Object.java:474)
 > >       winstone.RequestHandlerThread.run(RequestHandlerThread.java:216)
 > >       java.lang.Thread.run(Thread.java:595)
 > >     Thread[LauncherControlThread[ControlPort=-1],5,main] (TIMED_WAITING)
 > >       java.lang.Thread.sleep(Native Method)
 > >       winstone.Launcher.run(Launcher.java:279)
 > >       java.lang.Thread.run(Thread.java:595)
 > >     Thread[Signal Dispatcher,10,system] (RUNNABLE)
 > >     Thread[RequestHandlerThread[#2],5,main] (WAITING)
 > >       java.lang.Object.wait(Native Method)
 > >       java.lang.Object.wait(Object.java:474)
 > >       winstone.RequestHandlerThread.run(RequestHandlerThread.java:216)
 > >       java.lang.Thread.run(Thread.java:595)
 > >     Thread[Executor #0 for WORKING-HOST,5,main] (WAITING)
 > >       java.lang.Object.wait(Native Method)
 > >       java.lang.Object.wait(Object.java:474)
 > >       hudson.util.OneShotEvent.block(OneShotEvent.java:28)
 > >       hudson.model.Queue.pop(Queue.java:476)
 > >       hudson.model.Executor.run(Executor.java:72)
 > >     Thread[stderr copier for remote agent on WORKING-HOST,5,main] (RUNNABLE)
 > >       java.io.FileInputStream.readBytes(Native Method)
 > >       java.io.FileInputStream.read(FileInputStream.java:177)
 > >       hudson.util.StreamCopyThread.run(StreamCopyThread.java:27)
 > >     Thread[Channel reader thread: WORKING-HOST,5,main] (RUNNABLE)
 > >       java.io.FileInputStream.readBytes(Native Method)
 > >       java.io.FileInputStream.read(FileInputStream.java:194)
 > >       java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
 > >       java.io.BufferedInputStream.read(BufferedInputStream.java:235)
 > >       java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2200)
 > >       java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2490)
 > >       java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2500)
 > >       java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1267)
 > >       java.io.ObjectInputStream.readObject(ObjectInputStream.java:339)
 > >       hudson.remoting.Channel$ReaderThread.run(Channel.java:576)
 > >     Thread[WinstoneObjectPoolMgmt,5,main] (TIMED_WAITING)
 > >       java.lang.Thread.sleep(Native Method)
 > >       winstone.ObjectPool.run(ObjectPool.java:103)
 > >       java.lang.Thread.run(Thread.java:595)
 > >      ****************************************************************
 > >     system info of client (sanitized)
 > > System Properties
 > > Name Value  
 > > file.encoding ANSI_X3.4-1968
 > > file.encoding.pkg sun.io
 > > file.separator /
 > > java.awt.graphicsenv sun.awt.X11GraphicsEnvironment
 > > java.awt.printerjob sun.print.PSPrinterJob
 > > java.class.path /NFS/PATH/to/Hudson-home/WEB-INF/slave.jar
 > > java.class.version 50.0
 > > java.endorsed.dirs /usr/java/jdk1.6.0_05/jre/lib/endorsed
 > > java.ext.dirs /usr/java/jdk1.6.0_05/jre/lib/ext:/usr/java/packages/lib/ext
 > > java.home /usr/java/jdk1.6.0_05/jre
 > > java.io.tmpdir /tmp
 > > java.library.path /usr/java/jdk1.6.0_05/jre/lib/i386/server:/usr/java/jdk1.6.0_05/jre/lib/i386:/usr/java/jdk1.6.0_05/jre/../lib/i386:/usr/local/lib:/usr/lib:/usr/java/packages/lib/i386:/lib:/usr/lib
 > > java.runtime.name Java(TM) SE Runtime Environment
 > > java.runtime.version 1.6.0_05-b13
 > > java.specification.name Java Platform API Specification
 > > java.specification.vendor Sun Microsystems Inc.
 > > java.specification.version 1.6
 > > java.vendor Sun Microsystems Inc.
 > > java.vendor.url http://java.sun.com/
 > > java.vendor.url.bug http://java.sun.com/cgi-bin/bugreport.cgi
 > > java.version 1.6.0_05
 > > java.vm.info mixed mode
 > > java.vm.name Java HotSpot(TM) Server VM
 > > java.vm.specification.name Java Virtual Machine Specification
 > > java.vm.specification.vendor Sun Microsystems Inc.
 > > java.vm.specification.version 1.0
 > > java.vm.vendor Sun Microsystems Inc.
 > > java.vm.version 10.0-b19
 > > jna.platform.library.path /usr/lib:/lib
 > > line.separator
 > > os.arch i386
 > > os.name Linux
 > > os.version 2.6.17-1.2157_FC3asl.5smp
 > > path.separator :
 > > sun.arch.data.model 32
 > > sun.boot.class.path /usr/java/jdk1.6.0_05/jre/lib/resources.jar:/usr/java/jdk1.6.0_05/jre/lib/rt.jar:/usr/java/jdk1.6.0_05/jre/lib/sunrsasign.jar:/usr/java/jdk1.6.0_05/jre/lib/jsse.jar:/usr/java/jdk1.6.0_05/jre/lib/jce.jar:/usr/java/jdk1.6.0_05/jre/lib/charsets.jar:/usr/java/jdk1.6.0_05/jre/classes
 > > sun.boot.library.path /usr/java/jdk1.6.0_05/jre/lib/i386
 > > sun.cpu.endian little
 > > sun.cpu.isalist
 > > sun.io.unicode.encoding UnicodeLittle
 > > sun.java.launcher SUN_STANDARD
 > > sun.jnu.encoding ANSI_X3.4-1968
 > > sun.management.compiler HotSpot Tiered Compilers
 > > sun.os.patch.level unknown
 > > svnkit.ssh2.persistent false
 > > svnkit.symlinks false
 > > user.country US
 > > user.dir /NFS/Huson/user/homedir
 > > user.home /NFS/Huson/user/homedir
 > > user.language en
 > > user.name HudsonUser
 > > user.timezone America/New_York
 > >
 > > Environment Variables
 > > Name Value
 > > _ /usr/bin/java
 > > ALREADY_SET_PATHS 1
 > > HOME /NFS/Huson/user/homedir
 > > HOST PROBLEM-HOST
 > > LD_LIBRARY_PATH /usr/java/jdk1.6.0_05/jre/lib/i386/server:/usr/java/jdk1.6.0_05/jre/lib/i386:/usr/java/jdk1.6.0_05/jre/../lib/i386:/usr/local/lib:/usr/lib
 > > LESS sMQwh10i
 > > LOGNAME HudsonUser
 > > MAIL /var/spool/mail/HudsonUser
 > > MAILFILE /var/spool/mail/HudsonUser
 > > MANPATH /NFS/Huson/user/homedir/system/man:/usr/local/man:/usr/X11R6/man:/usr/share/man:/usr/man:/usr/lib/perl5/man
 > > METAMAIL_PAGER less
 > > NLSPATH /usr/dt/lib/nls/msg/%L/%N.cat
 > > OLDPWD /NFS/Huson/user/homedir
 > > PAGER less
 > > PATH /NFS/Huson/user/homedir/bin/linux:/NFS/Huson/user/homedir/bin/script:/NFS/Huson/user/homedir/bin:/net/dotfiles/bin:/usr/local/bin:/usr/X11R6/bin:/usr/bin:/bin:/sbin:/usr/sbin
 > > PERLLIB
 > > PGPPATH /NFS/Huson/user/homedir/.pgp
 > > PRINTER prn
 > > PWD /NFS/Huson/user/homedir
 > > SHELL /bin/zsh
 > > SHLVL 0
 > > SSH_CLIENT ::ffff:10.4.1.103 34375 22
 > > SSH_CONNECTION ::ffff:10.4.1.103 34375 ::ffff:10.4.2.134 22
 > > TMPDIR /tmp
 > > USER HudsonUser
 > > XFILESEARCHPATH /usr/dt/app-defaults/%L/Dt
 > >
 > > Thread Dump
 > > pool-1-thread-52
 > >
 > > "pool-1-thread-52" Id=198 RUNNABLE
 > > at sun.management.ThreadImpl.dumpThreads0(Native Method)
 > > at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:359)
 > > at hudson.Functions.getThreadInfos(Functions.java:635)
 > > at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:47)
 > > at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:43)
 > > at hudson.remoting.UserRequest.perform(UserRequest.java:69)
 > > at hudson.remoting.UserRequest.perform(UserRequest.java:23)
 > > at hudson.remoting.Request$2.run(Request.java:200)
 > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
 > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
 > > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
 > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
 > > at java.lang.Thread.run(Thread.java:619)
 > >
 > > Number of locked synchronizers = 1
 > > - java.util.concurrent.locks.ReentrantLock$NonfairSync@1f37d5
 > >
 > >
 > > Channel reader thread: channel
 > >
 > > "Channel reader thread: channel" Id=9 RUNNABLE (in native)
 > > at java.io.FileInputStream.readBytes(Native Method)
 > > at java.io.FileInputStream.read(FileInputStream.java:199)
 > > at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
 > > at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
 > > -  locked java.io.BufferedInputStream@15e1229
 > > at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249)
 > > at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2542)
 > > at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2552)
 > > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
 > > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
 > > at hudson.remoting.Channel$ReaderThread.run(Channel.java:574)
 > >
 > >
 > > Signal Dispatcher
 > >
 > > "Signal Dispatcher" Id=4 RUNNABLE
 > >
 > >
 > > Finalizer
 > >
 > > "Finalizer" Id=3 WAITING on java.lang.ref.ReferenceQueue$Lock@fa7ade
 > > at java.lang.Object.wait(Native Method)
 > > -  waiting on java.lang.ref.ReferenceQueue$Lock@fa7ade
 > > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
 > > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
 > > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
 > >
 > >
 > > Reference Handler
 > >
 > > "Reference Handler" Id=2 WAITING on java.lang.ref.Reference$Lock@98896c
 > > at java.lang.Object.wait(Native Method)
 > > -  waiting on java.lang.ref.Reference$Lock@98896c
 > > at java.lang.Object.wait(Object.java:485)
 > > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
 > >
 > >
 > > main
 > >
 > > "main" Id=1 WAITING on hudson.remoting.Channel@664d41
 > > at java.lang.Object.wait(Native Method)
 > > -  waiting on hudson.remoting.Channel@664d41
 > > at java.lang.Object.wait(Object.java:485)
 > > at hudson.remoting.Channel.join(Channel.java:477)
 > > at hudson.remoting.Launcher.main(Launcher.java:69)
 > > at hudson.remoting.Launcher.main(Launcher.java:43)
 > >
 > >      ****************************************************************
 > >
 > > Kohsuke Kawaguchi writes:
 > >  >
 > >  > I suppose "java.net.NoRouteToHostException: No route to host" exception
 > >  > is reported but the build hangs after that?
 > >  >
 > >  > When it hangs again, can you get us the thread dump on the server and
 > >  > the slave? You can go to http://server/hudson/threadDump and
 > >  > http://server/computer/YOURSLAVENAME/systemInfo to get those.
 > >  >
 > >  > Tom Cavin wrote:
 > >  > > Hi Folks,
 > >  > >
 > >  > > I'm seeing a problem where hudson (v1.237) jobs hang until killed if there
 > >  > > is a network problem during svn polling.
 > >  > >
 > >  > > The network I'm on is currently congested with the sporadic problem that
 > >  > > arp packets sometimes collide causing ssh connections to occasionally get
 > >  > > "no route to host" messages instead of connections.  A retry a few seconds
 > >  > > later will be fine, but the momentary glitch does block the first attempt.
 > >  > >
 > >  > > My problem with Hudson is that I have a small job polling my svn repository
 > >  > > every 5 minutes.  Most of the time, things go as I expect, but when a
 > >  > > network glitch coincides with a hudson svn poll or a hudson svn update, I
 > >  > > get the following error message and the build hangs.  The last time this
 > >  > > happened, the normal 30 second job was at over 4 hours and counting when I
 > >  > > killed the hudson master and restarted.
 > >  > >
 > >  > > What I'd like to see is a simple failure or abort, followed by a retry at
 > >  > > a later time.
 > >  > >
 > >  > > Is this a known bug?
 > >  > >
 > >  > > Best Wishes,
 > >  > >
 > >  > >      --Tom
 > >  > >
 > >  > > Buildbot Hudson writes:
 > >  > >  > See http://HUDSON_MASTER:8080/job/testBlame/393/changes
 > >  > >  >
 > >  > >  > ------------------------------------------
 > >  > >  > started
 > >  > >  > Building remotely on HUDSON_SLAVE_HOST
 > >  > >  > Updating svn+ssh://REPOSITORY_HOST/REPOSITORY_PATH
 > >  > >  > ERROR: Failed to update svn+ssh://REPOSITORY_HOST/REPOSITORY_PATH
 > >  > >  > org.tmatesoft.svn.core.SVNException: svn: Cannot connect to 'svn+ssh://REPOSITORY_HOST': There was a problem while connecting to REPOSITORY_HOST:22
 > >  > >  > at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:69)
 > >  > >  > at org.tmatesoft.svn.core.internal.io.svn.SVNSSHSession.openConnection(SVNSSHSession.java:281)
 > >  > >  > at org.tmatesoft.svn.core.internal.io.svn.SVNSSHSession.getConnection(SVNSSHSession.java:85)
 > >  > >  > at org.tmatesoft.svn.core.internal.io.svn.SVNSSHConnector.open(SVNSSHConnector.java:74)
 > >  > >  > at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.open(SVNConnection.java:60)
 > >  > >  > at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.openConnection(SVNRepositoryImpl.java:974)
 > >  > >  > at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.getDatedRevision(SVNRepositoryImpl.java:159)
 > >  > >  > at org.tmatesoft.svn.core.wc.SVNBasicClient.getRevisionNumber(SVNBasicClient.java:346)
 > >  > >  > at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:159)
 > >  > >  > at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:387)
 > >  > >  > at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:359)
 > >  > >  > at hudson.FilePath$FileCallableWrapper.call(FilePath.java:1122)
 > >  > >  > at hudson.remoting.UserRequest.perform(UserRequest.java:69)
 > >  > >  > at hudson.remoting.UserRequest.perform(UserRequest.java:23)
 > >  > >  > at hudson.remoting.Request$2.run(Request.java:200)
 > >  > >  > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
 > >  > >  > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
 > >  > >  > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 > >  > >  > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
 > >  > >  > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
 > >  > >  > at java.lang.Thread.run(Thread.java:619)
 > >  > >  > Caused by: java.io.IOException: There was a problem while connecting to REPOSITORY_HOST:22
 > >  > >  > at com.trilead.ssh2.Connection.connect(Connection.java:753)
 > >  > >  > at com.trilead.ssh2.Connection.connect(Connection.java:544)
 > >  > >  > at org.tmatesoft.svn.core.internal.io.svn.SVNSSHSession.openConnection(SVNSSHSession.java:241)
 > >  > >  > ... 19 more
 > >  > >  > Caused by: java.net.NoRouteToHostException: No route to host
 > >  > >  > at java.net.PlainSocketImpl.socketConnect(Native Method)
 > >  > >  > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
 > >  > >  > at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
 > >  > >  > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
 > >  > >  > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
 > >  > >  > at java.net.Socket.connect(Socket.java:519)
 > >  > >  > at com.trilead.ssh2.transport.TransportManager.establishConnection(TransportManager.java:341)
 > >  > >  > at com.trilead.ssh2.transport.TransportManager.initialize(TransportManager.java:449)
 > >  > >  > at com.trilead.ssh2.Connection.connect(Connection.java:697)
 > >  > >  > ... 21 more
 > >  > >  > Recording fingerprints
 > >  > >  >
 > >  > >
 > >  > > ---------------------------------------------------------------------
 > >  > > To unsubscribe, e-mail: [hidden email]
 > >  > > For additional commands, e-mail: [hidden email]
 > >  > >
 > >  > >
 > >  >
 > >  >
 > >  > --
 > >  > Kohsuke Kawaguchi
 > >  > Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/
 > >
 > > ---------------------------------------------------------------------
 > > To unsubscribe, e-mail: [hidden email]
 > > For additional commands, e-mail: [hidden email]
 > >
 > >
 >
 >
 > --
 > Kohsuke Kawaguchi
 > Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]