How to prevent concurrent execution of jobs on the same node, but allow a few parallel steps execution?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

How to prevent concurrent execution of jobs on the same node, but allow a few parallel steps execution?

vitaly_il

I have multibranch job, whole pipeline runs on certain node. 

Currently I have a slaves with single executor, so multiple jobs cannot run on the same node. I'd like to use parallel steps inside my pipeline, so I need to increase number of executors. 

What is the best way to prevent concurrent job execution? 

Such configuration was trivial with Jenkins freestyle job and Throttling plugin - I simple used "Maximum Concurrent Builds Per Node". But in pipeline it seems that I cannot use "throttle" for whole pipeline, I need to use it for each step. Is there more elegant solution?

TIA, Vitaly

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/cab8c3e6-694c-4329-bc08-6414d4e318a5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How to prevent concurrent execution of jobs on the same node, but allow a few parallel steps execution?

Den Slobozhanskyi
Just to clarify - does it mean that Throttling plugin throttles not the job as a whole, but each step in parallel? My assumption was that throttling is only for the job, and nested threads have nothing to do with throttle. By your words it looks like setting throttling to "1" for the job reduce the possible parallel threads execution to a single thread at a time as well, is that true?

If this is not true, you just gather all the jobs under one throttling category and set the concurrent builds limit to 1.

BR,
Denys

среда, 30 января 2019 г., 14:17:15 UTC+1 пользователь Vitaly Karasik написал:

I have multibranch job, whole pipeline runs on certain node. 

Currently I have a slaves with single executor, so multiple jobs cannot run on the same node. I'd like to use parallel steps inside my pipeline, so I need to increase number of executors. 

What is the best way to prevent concurrent job execution? 

Such configuration was trivial with Jenkins freestyle job and Throttling plugin - I simple used "Maximum Concurrent Builds Per Node". But in pipeline it seems that I cannot use "throttle" for whole pipeline, I need to use it for each step. Is there more elegant solution?

TIA, Vitaly

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/08107dc8-9bb4-4979-89c3-302a00183bc1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How to prevent concurrent execution of jobs on the same node, but allow a few parallel steps execution?

vitaly_il
I'd like to mark *whole* pipeline job as throttling resource; and unlike freestyle jobs, Throttleling plugin does't know to do that. I.e. I cannot use "throttle" in pipeline options.

On Wednesday, January 30, 2019 at 7:47:20 PM UTC+2, Den Slobozhanskyi wrote:
Just to clarify - does it mean that Throttling plugin throttles not the job as a whole, but each step in parallel? My assumption was that throttling is only for the job, and nested threads have nothing to do with throttle. By your words it looks like setting throttling to "1" for the job reduce the possible parallel threads execution to a single thread at a time as well, is that true?

If this is not true, you just gather all the jobs under one throttling category and set the concurrent builds limit to 1.

BR,
Denys

среда, 30 января 2019 г., 14:17:15 UTC+1 пользователь Vitaly Karasik написал:

I have multibranch job, whole pipeline runs on certain node. 

Currently I have a slaves with single executor, so multiple jobs cannot run on the same node. I'd like to use parallel steps inside my pipeline, so I need to increase number of executors. 

What is the best way to prevent concurrent job execution? 

Such configuration was trivial with Jenkins freestyle job and Throttling plugin - I simple used "Maximum Concurrent Builds Per Node". But in pipeline it seems that I cannot use "throttle" for whole pipeline, I need to use it for each step. Is there more elegant solution?

TIA, Vitaly

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d6b76faf-f9e6-4960-8dc5-ab79c6c6ac0f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How to prevent concurrent execution of jobs on the same node, but allow a few parallel steps execution?

Daniel Butler
You could use the lockable resources plugin (v2.2+):

Set up a single resource for each node and give them all a common label

Then your Jenkinsfile would be:

lock(label: "common-label", variable: "LOCKED_NODE"){
    node(env.LOCKED_NODE){
        //build project
    }
}

Regards,
Daniel.

On Thu, Jan 31, 2019 at 7:48 AM Vitaly Karasik <[hidden email]> wrote:
I'd like to mark *whole* pipeline job as throttling resource; and unlike freestyle jobs, Throttleling plugin does't know to do that. I.e. I cannot use "throttle" in pipeline options.

On Wednesday, January 30, 2019 at 7:47:20 PM UTC+2, Den Slobozhanskyi wrote:
Just to clarify - does it mean that Throttling plugin throttles not the job as a whole, but each step in parallel? My assumption was that throttling is only for the job, and nested threads have nothing to do with throttle. By your words it looks like setting throttling to "1" for the job reduce the possible parallel threads execution to a single thread at a time as well, is that true?

If this is not true, you just gather all the jobs under one throttling category and set the concurrent builds limit to 1.

BR,
Denys

среда, 30 января 2019 г., 14:17:15 UTC+1 пользователь Vitaly Karasik написал:

I have multibranch job, whole pipeline runs on certain node. 

Currently I have a slaves with single executor, so multiple jobs cannot run on the same node. I'd like to use parallel steps inside my pipeline, so I need to increase number of executors. 

What is the best way to prevent concurrent job execution? 

Such configuration was trivial with Jenkins freestyle job and Throttling plugin - I simple used "Maximum Concurrent Builds Per Node". But in pipeline it seems that I cannot use "throttle" for whole pipeline, I need to use it for each step. Is there more elegant solution?

TIA, Vitaly

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d6b76faf-f9e6-4960-8dc5-ab79c6c6ac0f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO37hbgQQWsggy8VdYxnn6RhWmYP%3DVBh4A%3D8JxO-etH6iOSpTg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How to prevent concurrent execution of jobs on the same node, but allow a few parallel steps execution?

vitaly_il
>Set up a single resource for each node and give them all a common label
Thank you - I   thought about using lockable resources, but didn't know that I can mark each node as separate resource. 
I'll check it.

On Thursday, January 31, 2019 at 11:43:30 AM UTC+2, Daniel Butler wrote:
You could use the lockable resources plugin (v2.2+):

Set up a single resource for each node and give them all a common label

Then your Jenkinsfile would be:

lock(label: "common-label", variable: "LOCKED_NODE"){
    node(env.LOCKED_NODE){
        //build project
    }
}

Regards,
Daniel.

On Thu, Jan 31, 2019 at 7:48 AM Vitaly Karasik <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="4u-_Zlr6FgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">linu...@...> wrote:
I'd like to mark *whole* pipeline job as throttling resource; and unlike freestyle jobs, Throttleling plugin does't know to do that. I.e. I cannot use "throttle" in pipeline options.

On Wednesday, January 30, 2019 at 7:47:20 PM UTC+2, Den Slobozhanskyi wrote:
Just to clarify - does it mean that Throttling plugin throttles not the job as a whole, but each step in parallel? My assumption was that throttling is only for the job, and nested threads have nothing to do with throttle. By your words it looks like setting throttling to "1" for the job reduce the possible parallel threads execution to a single thread at a time as well, is that true?

If this is not true, you just gather all the jobs under one throttling category and set the concurrent builds limit to 1.

BR,
Denys

среда, 30 января 2019 г., 14:17:15 UTC+1 пользователь Vitaly Karasik написал:

I have multibranch job, whole pipeline runs on certain node. 

Currently I have a slaves with single executor, so multiple jobs cannot run on the same node. I'd like to use parallel steps inside my pipeline, so I need to increase number of executors. 

What is the best way to prevent concurrent job execution? 

Such configuration was trivial with Jenkins freestyle job and Throttling plugin - I simple used "Maximum Concurrent Builds Per Node". But in pipeline it seems that I cannot use "throttle" for whole pipeline, I need to use it for each step. Is there more elegant solution?

TIA, Vitaly

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="4u-_Zlr6FgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/d6b76faf-f9e6-4960-8dc5-ab79c6c6ac0f%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/d6b76faf-f9e6-4960-8dc5-ab79c6c6ac0f%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/d6b76faf-f9e6-4960-8dc5-ab79c6c6ac0f%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/d6b76faf-f9e6-4960-8dc5-ab79c6c6ac0f%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/01769347-83f1-4b9c-8a98-204d1a0b5ca2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How to prevent concurrent execution of jobs on the same node, but allow a few parallel steps execution?

Den Slobozhanskyi
In reply to this post by vitaly_il

<a imageanchor="1" href="about:invalid#zClosurez" style="margin-left: 1em; margin-right: 1em;">Screenshot from 2019-01-31 12-35-45.png

what plugin do you use precisely? I run jenkins 2.150.1 and this throttling plugin https://github.com/jenkinsci/throttle-concurrent-builds-plugin/blob/master/README.md . It clearly gives me an option to throttle a pipeline job in job configurations (yep, 100% pipeline job type)

четверг, 31 января 2019 г., 8:48:14 UTC+1 пользователь Vitaly Karasik написал:
I'd like to mark *whole* pipeline job as throttling resource; and unlike freestyle jobs, Throttleling plugin does't know to do that. I.e. I cannot use "throttle" in pipeline options.

On Wednesday, January 30, 2019 at 7:47:20 PM UTC+2, Den Slobozhanskyi wrote:
Just to clarify - does it mean that Throttling plugin throttles not the job as a whole, but each step in parallel? My assumption was that throttling is only for the job, and nested threads have nothing to do with throttle. By your words it looks like setting throttling to "1" for the job reduce the possible parallel threads execution to a single thread at a time as well, is that true?

If this is not true, you just gather all the jobs under one throttling category and set the concurrent builds limit to 1.

BR,
Denys

среда, 30 января 2019 г., 14:17:15 UTC+1 пользователь Vitaly Karasik написал:

I have multibranch job, whole pipeline runs on certain node. 

Currently I have a slaves with single executor, so multiple jobs cannot run on the same node. I'd like to use parallel steps inside my pipeline, so I need to increase number of executors. 

What is the best way to prevent concurrent job execution? 

Such configuration was trivial with Jenkins freestyle job and Throttling plugin - I simple used "Maximum Concurrent Builds Per Node". But in pipeline it seems that I cannot use "throttle" for whole pipeline, I need to use it for each step. Is there more elegant solution?

TIA, Vitaly

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/5ed835bb-43fb-434d-9b5c-15d48e47b040%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How to prevent concurrent execution of jobs on the same node, but allow a few parallel steps execution?

vitaly_il
My job is Multibranch job, for some reason such job, unlike 'regulal' pipeline job, doesn't provide this checkbox.

On Thursday, January 31, 2019 at 3:25:53 PM UTC+2, Den Slobozhanskyi wrote:

Screenshot from 2019-01-31 12-35-45.png

what plugin do you use precisely? I run jenkins 2.150.1 and this throttling plugin <a href="https://github.com/jenkinsci/throttle-concurrent-builds-plugin/blob/master/README.md" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Fthrottle-concurrent-builds-plugin%2Fblob%2Fmaster%2FREADME.md\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHBJwTWq_yCit3NK4PPoa0uUcqsJw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Fthrottle-concurrent-builds-plugin%2Fblob%2Fmaster%2FREADME.md\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHBJwTWq_yCit3NK4PPoa0uUcqsJw&#39;;return true;">https://github.com/jenkinsci/throttle-concurrent-builds-plugin/blob/master/README.md . It clearly gives me an option to throttle a pipeline job in job configurations (yep, 100% pipeline job type)

четверг, 31 января 2019 г., 8:48:14 UTC+1 пользователь Vitaly Karasik написал:
I'd like to mark *whole* pipeline job as throttling resource; and unlike freestyle jobs, Throttleling plugin does't know to do that. I.e. I cannot use "throttle" in pipeline options.

On Wednesday, January 30, 2019 at 7:47:20 PM UTC+2, Den Slobozhanskyi wrote:
Just to clarify - does it mean that Throttling plugin throttles not the job as a whole, but each step in parallel? My assumption was that throttling is only for the job, and nested threads have nothing to do with throttle. By your words it looks like setting throttling to "1" for the job reduce the possible parallel threads execution to a single thread at a time as well, is that true?

If this is not true, you just gather all the jobs under one throttling category and set the concurrent builds limit to 1.

BR,
Denys

среда, 30 января 2019 г., 14:17:15 UTC+1 пользователь Vitaly Karasik написал:

I have multibranch job, whole pipeline runs on certain node. 

Currently I have a slaves with single executor, so multiple jobs cannot run on the same node. I'd like to use parallel steps inside my pipeline, so I need to increase number of executors. 

What is the best way to prevent concurrent job execution? 

Such configuration was trivial with Jenkins freestyle job and Throttling plugin - I simple used "Maximum Concurrent Builds Per Node". But in pipeline it seems that I cannot use "throttle" for whole pipeline, I need to use it for each step. Is there more elegant solution?

TIA, Vitaly

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/c1350fac-3c06-4c57-9017-a19c8f776054%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How to prevent concurrent execution of jobs on the same node, but allow a few parallel steps execution?

vitaly_il
In reply to this post by Daniel Butler
I started to play with the lockable resources plugin. 
I'd like to prevent parallel execution of my pipeline (multibranch pipeline) on the same Jenkins slave, so I tried to use "$NODE_NAME" as lockable resource, like this:

pipeline {
  agent { label 'slave-parallel' }
  options {
    lock ("${NODE_NAME}")
   }

stages() 
{
......
}

But unfortunately $NODE_NAME isn't available on this step. Is there simple workaround for thaT? 

On Thursday, January 31, 2019 at 11:43:30 AM UTC+2, Daniel Butler wrote:
You could use the lockable resources plugin (v2.2+):

Set up a single resource for each node and give them all a common label

Then your Jenkinsfile would be:

lock(label: "common-label", variable: "LOCKED_NODE"){
    node(env.LOCKED_NODE){
        //build project
    }
}

Regards,
Daniel.

On Thu, Jan 31, 2019 at 7:48 AM Vitaly Karasik <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="4u-_Zlr6FgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">linu...@...> wrote:
I'd like to mark *whole* pipeline job as throttling resource; and unlike freestyle jobs, Throttleling plugin does't know to do that. I.e. I cannot use "throttle" in pipeline options.

On Wednesday, January 30, 2019 at 7:47:20 PM UTC+2, Den Slobozhanskyi wrote:
Just to clarify - does it mean that Throttling plugin throttles not the job as a whole, but each step in parallel? My assumption was that throttling is only for the job, and nested threads have nothing to do with throttle. By your words it looks like setting throttling to "1" for the job reduce the possible parallel threads execution to a single thread at a time as well, is that true?

If this is not true, you just gather all the jobs under one throttling category and set the concurrent builds limit to 1.

BR,
Denys

среда, 30 января 2019 г., 14:17:15 UTC+1 пользователь Vitaly Karasik написал:

I have multibranch job, whole pipeline runs on certain node. 

Currently I have a slaves with single executor, so multiple jobs cannot run on the same node. I'd like to use parallel steps inside my pipeline, so I need to increase number of executors. 

What is the best way to prevent concurrent job execution? 

Such configuration was trivial with Jenkins freestyle job and Throttling plugin - I simple used "Maximum Concurrent Builds Per Node". But in pipeline it seems that I cannot use "throttle" for whole pipeline, I need to use it for each step. Is there more elegant solution?

TIA, Vitaly

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="4u-_Zlr6FgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/d6b76faf-f9e6-4960-8dc5-ab79c6c6ac0f%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/d6b76faf-f9e6-4960-8dc5-ab79c6c6ac0f%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/d6b76faf-f9e6-4960-8dc5-ab79c6c6ac0f%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/d6b76faf-f9e6-4960-8dc5-ab79c6c6ac0f%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d7a4d843-79ba-4b46-aaf5-66fe45c00e34%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How to prevent concurrent execution of jobs on the same node, but allow a few parallel steps execution?

Brian Ray
Can you put your lock step inside a steps block? We still use Imperative Pipeline so I'm less familiar with Declarative Pipeline. But it seems like inside a step block that NODE_NAME should resolve correctly, because--I think--the block is running in a node context.

On Monday, February 11, 2019 at 5:33:44 AM UTC-8, Vitaly Karasik wrote:
I started to play with the lockable resources plugin. 
I'd like to prevent parallel execution of my pipeline (multibranch pipeline) on the same Jenkins slave, so I tried to use "$NODE_NAME" as lockable resource, like this:

pipeline {
  agent { label 'slave-parallel' }
  options {
    lock ("${NODE_NAME}")
   }

stages() 
{
......
}

But unfortunately $NODE_NAME isn't available on this step. Is there simple workaround for thaT? 

On Thursday, January 31, 2019 at 11:43:30 AM UTC+2, Daniel Butler wrote:
You could use the lockable resources plugin (v2.2+):

Set up a single resource for each node and give them all a common label

Then your Jenkinsfile would be:

lock(label: "common-label", variable: "LOCKED_NODE"){
    node(env.LOCKED_NODE){
        //build project
    }
}

Regards,
Daniel.

On Thu, Jan 31, 2019 at 7:48 AM Vitaly Karasik <[hidden email]> wrote:
I'd like to mark *whole* pipeline job as throttling resource; and unlike freestyle jobs, Throttleling plugin does't know to do that. I.e. I cannot use "throttle" in pipeline options.

On Wednesday, January 30, 2019 at 7:47:20 PM UTC+2, Den Slobozhanskyi wrote:
Just to clarify - does it mean that Throttling plugin throttles not the job as a whole, but each step in parallel? My assumption was that throttling is only for the job, and nested threads have nothing to do with throttle. By your words it looks like setting throttling to "1" for the job reduce the possible parallel threads execution to a single thread at a time as well, is that true?

If this is not true, you just gather all the jobs under one throttling category and set the concurrent builds limit to 1.

BR,
Denys

среда, 30 января 2019 г., 14:17:15 UTC+1 пользователь Vitaly Karasik написал:

I have multibranch job, whole pipeline runs on certain node. 

Currently I have a slaves with single executor, so multiple jobs cannot run on the same node. I'd like to use parallel steps inside my pipeline, so I need to increase number of executors. 

What is the best way to prevent concurrent job execution? 

Such configuration was trivial with Jenkins freestyle job and Throttling plugin - I simple used "Maximum Concurrent Builds Per Node". But in pipeline it seems that I cannot use "throttle" for whole pipeline, I need to use it for each step. Is there more elegant solution?

TIA, Vitaly

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/d6b76faf-f9e6-4960-8dc5-ab79c6c6ac0f%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/d6b76faf-f9e6-4960-8dc5-ab79c6c6ac0f%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/d6b76faf-f9e6-4960-8dc5-ab79c6c6ac0f%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/d6b76faf-f9e6-4960-8dc5-ab79c6c6ac0f%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/7eabfb51-9587-41c8-a40e-7fdbc9f92416%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: How to prevent concurrent execution of jobs on the same node, but allow a few parallel steps execution?

Brian Ray
In reply to this post by vitaly_il
Can you put your lock step inside a step block? We still use Imperative Pipeline so I'm less familiar with Declarative Pipeline. But it seems like inside a step block that NODE_NAME should resolve correctly, because--I think--the block is running in a node context.

On Monday, February 11, 2019 at 5:33:44 AM UTC-8, Vitaly Karasik wrote:
I started to play with the lockable resources plugin. 
I'd like to prevent parallel execution of my pipeline (multibranch pipeline) on the same Jenkins slave, so I tried to use "$NODE_NAME" as lockable resource, like this:

pipeline {
  agent { label 'slave-parallel' }
  options {
    lock ("${NODE_NAME}")
   }

stages() 
{
......
}

But unfortunately $NODE_NAME isn't available on this step. Is there simple workaround for thaT? 

On Thursday, January 31, 2019 at 11:43:30 AM UTC+2, Daniel Butler wrote:
You could use the lockable resources plugin (v2.2+):

Set up a single resource for each node and give them all a common label

Then your Jenkinsfile would be:

lock(label: "common-label", variable: "LOCKED_NODE"){
    node(env.LOCKED_NODE){
        //build project
    }
}

Regards,
Daniel.

On Thu, Jan 31, 2019 at 7:48 AM Vitaly Karasik <[hidden email]> wrote:
I'd like to mark *whole* pipeline job as throttling resource; and unlike freestyle jobs, Throttleling plugin does't know to do that. I.e. I cannot use "throttle" in pipeline options.

On Wednesday, January 30, 2019 at 7:47:20 PM UTC+2, Den Slobozhanskyi wrote:
Just to clarify - does it mean that Throttling plugin throttles not the job as a whole, but each step in parallel? My assumption was that throttling is only for the job, and nested threads have nothing to do with throttle. By your words it looks like setting throttling to "1" for the job reduce the possible parallel threads execution to a single thread at a time as well, is that true?

If this is not true, you just gather all the jobs under one throttling category and set the concurrent builds limit to 1.

BR,
Denys

среда, 30 января 2019 г., 14:17:15 UTC+1 пользователь Vitaly Karasik написал:

I have multibranch job, whole pipeline runs on certain node. 

Currently I have a slaves with single executor, so multiple jobs cannot run on the same node. I'd like to use parallel steps inside my pipeline, so I need to increase number of executors. 

What is the best way to prevent concurrent job execution? 

Such configuration was trivial with Jenkins freestyle job and Throttling plugin - I simple used "Maximum Concurrent Builds Per Node". But in pipeline it seems that I cannot use "throttle" for whole pipeline, I need to use it for each step. Is there more elegant solution?

TIA, Vitaly

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/d6b76faf-f9e6-4960-8dc5-ab79c6c6ac0f%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/d6b76faf-f9e6-4960-8dc5-ab79c6c6ac0f%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/d6b76faf-f9e6-4960-8dc5-ab79c6c6ac0f%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/d6b76faf-f9e6-4960-8dc5-ab79c6c6ac0f%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d54117cb-3251-4cca-9f98-50b8887fc2de%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.