occasional exception from junit call

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

occasional exception from junit call

jeremy mordkoff
I use scripted pipelines with a library. The master and the slaves are running in ubuntu 16.04 docker containers. 

About a week ago we started getting occasional failures from this call: 

junit testResults: ".build/ub18_debug/artifacts/systemtest/*xml"

There are about 60 junit xml files in that directory. 

Since then I have tried updating jenkins to 2.251 (including a container restart), which warned me about a place where a function marked 
NONCPS was calling CPS code, so I fixed that and restarted all of the slaves. But the problem persists. I would estimate it happens 30% 
of the time. 

For now I have wrapped that call in a try block that sets the build result to UNSTABLE if the call fails. But not having the test results 
available is messing up all of our reporting tooks. I also tried reordering some code thinking that maybe the previous step had caused the 
problem but the problem stayed with the junit call. 

I am willing to try just about anything to get past this issue. Any suggestions at all are welcome.





The Jenkins console for that job reports

[Pipeline] junit
Recording test results
java.io.IOException: Unexpected termination of the channel
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline
java.io.EOFException
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2353)
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2822)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:301)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
at hudson.remoting.Command.readFrom(Command.java:140)
at hudson.remoting.Command.readFrom(Command.java:126)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to j7-ssh2
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
at hudson.remoting.Request.call(Request.java:202)
at hudson.remoting.Channel.call(Channel.java:954)
at hudson.FilePath.act(FilePath.java:1069)
at hudson.FilePath.act(FilePath.java:1058)
at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:114)
at hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:137)
at hudson.tasks.junit.JUnitResultArchiver.parseAndAttach(JUnitResultArchiver.java:167)
at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:52)
at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:25)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused: hudson.remoting.RequestAbortedException
at hudson.remoting.Request.abort(Request.java:340)
at hudson.remoting.Channel.terminate(Channel.java:1038)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:94)
Finished: FAILURE



The slave's log reports 

Aug 11, 2020 4:19:21 AM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
Aug 12, 2020 7:55:32 AM hudson.slaves.ChannelPinger$1 onDead
INFO: Ping failed. Terminating the channel channel.
java.io.IOException: Broken pipe
        at java.io.FileOutputStream.writeBytes(Native Method)
        at java.io.FileOutputStream.write(FileOutputStream.java:313)
        at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83)
        at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89)
        at hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62)
        at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)
        at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
        at hudson.remoting.Channel.send(Channel.java:766)
        at hudson.remoting.Request.callAsync(Request.java:237)
        at hudson.remoting.Channel.callAsync(Channel.java:1032)
        at hudson.remoting.PingThread.ping(PingThread.java:110)
        at hudson.remoting.PingThread.run(PingThread.java:90)

Aug 12, 2020 7:55:32 AM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@1dd89b20
java.io.IOException: Broken pipe




and the master's logs include 


Agent JVM has not reported exit code. Is it still running?
[08/12/20 03:53:02] [SSH] Connection closed.
ERROR: Connection terminated
java.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2353)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2822)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:301)
        at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
        at hudson.remoting.Command.readFrom(Command.java:142)
        at hudson.remoting.Command.readFrom(Command.java:128)
        at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: java.io.IOException: Unexpected termination of the channel
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
        
        
        
FYI....Some time stamps are UTC and some are EDT 

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/0d78cf9e-e859-4a50-82cf-9e1906b9650cn%40googlegroups.com.