Throttling parallel step

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Throttling parallel step

Mirek S
Hi!

I'm using something like this to run tests in parallel:

stage('Test') {
  steps
{
    script
{
      testing_closures
= [one: { print("staring one"); sleep 10; print("finishing one") },
                          two
: { print("staring two"); sleep 10; print("finishing two") },
                          three
: { print("staring three"); sleep 10; print("finishing three") },
                          four
: { print("staring four"); sleep 10; print("finishing four") },
                          five
: { print("staring five"); sleep 10; print("finishing five") },
                          six
: { print("staring six"); sleep 10; print("finishing six") }]
      parallel
(testing_closures)
   
}
 
}
}


The main goal is to throttle those closures - I don't want for all six of them to run concurrently - only 3 at a time.

I was thinking about using nodes for this - i.e. wrapping each closure in node{} block:

one: { node { print("staring one"); sleep 10; print("finishing one") } }

Works OK as long as I use master node and limit the executors to 4 (1 for main job, 3 for the node{} concurrent steps). 

Unfortunately I need master node executors to be available for other jobs, so I cannot limit them.

So - how do I achieve my goal? Is there a way to create local node similar to master, but not master?

Or is there another way to throttle parallel step? 

The only solution I came up with is to create 3 lockable resources of the same label and lock them in each
of the closures. But in reality my closures are already locking resources (for test-machines, there's 7 of them, 
but I want one build to use only up to 3 of them), so I would have to have:
  lock(label: 'test-host') {
   
lock (label: 'test-throttler') {
     
...code...
   
}
 
}
Looks dirty, increases maintenance nightmare and it's adding unnecessary resources (which we already have too much) that do
not reflect the "real, physical" resource.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/de512d65-2b99-4f85-8a32-69c20cfcbc1e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Throttling parallel step

Mirek S
The only solution I came up with is to create 3 lockable resources of the same label and lock them in each
of the closures. 

Now that I think of it - it wouldn't work anyway - another build of this job would not be able to reach the test-throttler resources, so it would wait for them anyway... 

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/fed443ff-3256-4a91-98c5-8154d01c6051%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Loading...