Dynamic closures called in parallel

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Dynamic closures called in parallel

Mirek S
Hi!

I'm using the following Pipeline script:

testruns = [:]
for (name in ["one", "two", "three"]) {
  print("Adding ${name} to testruns")
  testruns[name] = { print("Staring phase ${name}"); sleep 10; print("Finished phase ${name}")}
}

pipeline {
  agent { label 'master' }
  stages {
    stage("Parallelism") {
      steps {
        script {
          parallel testruns
        }
      }
    }
  }
}

I'm expecting to run those three dynamically created closures in parallel. But it appears that all closures were created using only the last field ("three"). Console output:

Started by user ...
[Pipeline] echo
Adding one to testruns
[Pipeline] echo
Adding two to testruns
[Pipeline] echo
Adding three to testruns
[Pipeline] node
Running on master in ...
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Parallelism)
[Pipeline] script
[Pipeline] {
[Pipeline] parallel
[Pipeline] [one] { (Branch: one)
[Pipeline] [two] { (Branch: two)
[Pipeline] [three] { (Branch: three)
[Pipeline] [one] echo
[one] Staring phase three
[Pipeline] [one] sleep
[one] Sleeping for 10 sec
[Pipeline] [two] echo
[two] Staring phase three
[Pipeline] [two] sleep
[two] Sleeping for 10 sec
[Pipeline] [three] echo
[three] Staring phase three
[Pipeline] [three] sleep
[three] Sleeping for 10 sec
[Pipeline] [one] echo
[one] Finished phase three
[Pipeline] [one] }
[Pipeline] [two] echo
[two] Finished phase three
[Pipeline] [two] }
[Pipeline] [three] echo
[three] Finished phase three
[Pipeline] [three] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS


Please note that all phases are saying "three":

[two] Finished phase three

I was thinking that maybe there's a need for some kind of splat operator on argument for "parallel", but pipeline parallel example doesn't use it (and when I tried I got java.lang.UnsupportedOperationException)

What am I not seeing here? Why the closures are not created properly?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/9fe68722-9d88-4077-89ec-65aa786912aa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Dynamic closures called in parallel

Mirek S
Huh... Only now I noticed that my issue is mentioned in the pipeline parallel example. You can ignore this topic :)

//the dummy parameter is for preventing mutation of the parameter before the execution of the closure.
//we have to assign it outside the closure or it will run the job multiple times with the same parameter "4"
//and jenkins will unite them into a single run of the job

W dniu środa, 9 sierpnia 2017 17:17:53 UTC+2 użytkownik Mirek S napisał:
Hi!

I'm using the following Pipeline script:

testruns = [:]
for (name in ["one", "two", "three"]) {
  print("Adding ${name} to testruns")
  testruns[name] = { print("Staring phase ${name}"); sleep 10; print("Finished phase ${name}")}
}

pipeline {
  agent { label 'master' }
  stages {
    stage("Parallelism") {
      steps {
        script {
          parallel testruns
        }
      }
    }
  }
}

I'm expecting to run those three dynamically created closures in parallel. But it appears that all closures were created using only the last field ("three"). Console output:

Started by user ...
[Pipeline] echo
Adding one to testruns
[Pipeline] echo
Adding two to testruns
[Pipeline] echo
Adding three to testruns
[Pipeline] node
Running on master in ...
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Parallelism)
[Pipeline] script
[Pipeline] {
[Pipeline] parallel
[Pipeline] [one] { (Branch: one)
[Pipeline] [two] { (Branch: two)
[Pipeline] [three] { (Branch: three)
[Pipeline] [one] echo
[one] Staring phase three
[Pipeline] [one] sleep
[one] Sleeping for 10 sec
[Pipeline] [two] echo
[two] Staring phase three
[Pipeline] [two] sleep
[two] Sleeping for 10 sec
[Pipeline] [three] echo
[three] Staring phase three
[Pipeline] [three] sleep
[three] Sleeping for 10 sec
[Pipeline] [one] echo
[one] Finished phase three
[Pipeline] [one] }
[Pipeline] [two] echo
[two] Finished phase three
[Pipeline] [two] }
[Pipeline] [three] echo
[three] Finished phase three
[Pipeline] [three] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS


Please note that all phases are saying "three":

[two] Finished phase three

I was thinking that maybe there's a need for some kind of splat operator on argument for "parallel", but <a href="https://jenkins.io/doc/pipeline/examples/#jobs-in-parallel" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fjenkins.io%2Fdoc%2Fpipeline%2Fexamples%2F%23jobs-in-parallel\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF7lZWBatcxehpuatlerqS2zcdqBA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fjenkins.io%2Fdoc%2Fpipeline%2Fexamples%2F%23jobs-in-parallel\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF7lZWBatcxehpuatlerqS2zcdqBA&#39;;return true;">pipeline parallel example doesn't use it (and when I tried I got java.lang.UnsupportedOperationException)

What am I not seeing here? Why the closures are not created properly?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1418333c-4b96-4644-9b62-33d9264aad92%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Dynamic closures called in parallel

Jenkins users mailing list
In reply to this post by Mirek S

Hi,

your problem is  actually addressed in the example.
The closure is evalutated only during the parallel call. There $name is constant.
Put a
def xname  = name
inside your loop and use xname in the closures.
 (in the example it is  the def index=i line).

Björn


Am Mittwoch, 9. August 2017 17:17:53 UTC+2 schrieb Mirek S:
Hi!

I'm using the following Pipeline script:

testruns = [:]
for (name in ["one", "two", "three"]) {
  print("Adding ${name} to testruns")
  testruns[name] = { print("Staring phase ${name}"); sleep 10; print("Finished phase ${name}")}
}

pipeline {
  agent { label 'master' }
  stages {
    stage("Parallelism") {
      steps {
        script {
          parallel testruns
        }
      }
    }
  }
}

I'm expecting to run those three dynamically created closures in parallel. But it appears that all closures were created using only the last field ("three"). Console output:

Started by user ...
[Pipeline] echo
Adding one to testruns
[Pipeline] echo
Adding two to testruns
[Pipeline] echo
Adding three to testruns
[Pipeline] node
Running on master in ...
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Parallelism)
[Pipeline] script
[Pipeline] {
[Pipeline] parallel
[Pipeline] [one] { (Branch: one)
[Pipeline] [two] { (Branch: two)
[Pipeline] [three] { (Branch: three)
[Pipeline] [one] echo
[one] Staring phase three
[Pipeline] [one] sleep
[one] Sleeping for 10 sec
[Pipeline] [two] echo
[two] Staring phase three
[Pipeline] [two] sleep
[two] Sleeping for 10 sec
[Pipeline] [three] echo
[three] Staring phase three
[Pipeline] [three] sleep
[three] Sleeping for 10 sec
[Pipeline] [one] echo
[one] Finished phase three
[Pipeline] [one] }
[Pipeline] [two] echo
[two] Finished phase three
[Pipeline] [two] }
[Pipeline] [three] echo
[three] Finished phase three
[Pipeline] [three] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS


Please note that all phases are saying "three":

[two] Finished phase three

I was thinking that maybe there's a need for some kind of splat operator on argument for "parallel", but <a href="https://jenkins.io/doc/pipeline/examples/#jobs-in-parallel" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fjenkins.io%2Fdoc%2Fpipeline%2Fexamples%2F%23jobs-in-parallel\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF7lZWBatcxehpuatlerqS2zcdqBA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fjenkins.io%2Fdoc%2Fpipeline%2Fexamples%2F%23jobs-in-parallel\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF7lZWBatcxehpuatlerqS2zcdqBA&#39;;return true;">pipeline parallel example doesn't use it (and when I tried I got java.lang.UnsupportedOperationException)

What am I not seeing here? Why the closures are not created properly?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/f1a88248-913f-43ac-8930-2dcbbe4b854f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Loading...