Jenkins pipeline hangs when pod deleted

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Jenkins pipeline hangs when pod deleted

Miquel Catany
Hi,

We are using GCP preemtible instances to deploy pod agents with Kubernetes Plugin. We want to be able to retry the pipeline if the node/pod is evicted, but our main problem is that when the pipeline is executing sh commands and the node/pod is evicted/deleted,  the pipeline instead of failing/aborted is hung undefinitely. This problem was supposed to be fixed in https://issues.jenkins.io/browse/JENKINS-49707 and https://issues.jenkins.io/browse/JENKINS-59340 but it doesn't seem to work in our use case. The only solution to this is to add a timeout (which is useless for us) or abort it manually. Any insights on this?

I attach our current Jenkins file and the job results:


Thanks,

Miquel

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/fc9edf1c-7f8e-4102-b797-750ff08b379fn%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Jenkins pipeline hangs when pod deleted

Miquel Catany
Cannot contact ci-worker-dind-jsvbh: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@13b5e548:JNLP4-connect connection from 10.251.13.10/10.251.13.10:48846": Remote call on JNLP4-connect connection from 10.251.13.10/10.251.13.10:48846 failed. The channel is closing down or has closed down

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/69055f5a-6f7d-4545-82c7-00bc871c7e27n%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Jenkins pipeline hangs when pod deleted

Miquel Catany
```
def worker_pipeline(){
 try{
   node("xxxxxxxx") {
      stage('Checkout SCM'){
      git branch: 'xxxx',
        credentialsId: 'xxx',
        url: 'https://github.com/xxxxx/xxxx.git'
      }
     container('xxxxxxxx'){
      stage('Run Builds'){
       "base_image": {
         sh 'gcloud builds submit --config .ci/builds/base_image_cloudbuild.yaml'
       },
       "assets_dependencies": {
         sh 'gcloud builds submit --config .ci/builds/assets_dependencies_cloudbuild.yaml'
       }
      }
     }
   }
 }
 catch(err)
 {
  if ( "${err}" == "hudson.remoting.RequestAbortedException: java.nio.channels.ClosedChannelException" ){
   echo "The Pipeline has failed because of a Connection fail, so retry"
   worker_pipeline()
  }  
  else{
   echo "${err}"
   echo "The pipeline has failed, exit"
   throw new Exception("Pipeline has failed, exit") 
  }
 }
}
// Call main function
worker_pipeline()
```

On Thursday, November 19, 2020 at 9:17:41 AM UTC+1 Miquel Catany wrote:
Cannot contact ci-worker-dind-jsvbh: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@13b5e548:JNLP4-connect connection from 10.251.13.10/10.251.13.10:48846": Remote call on JNLP4-connect connection from 10.251.13.10/10.251.13.10:48846 failed. The channel is closing down or has closed down

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/c7ef3e33-2d28-4c9c-9f42-fd310294793dn%40googlegroups.com.