Locking multiple stages with NODE_NAME based lock

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Locking multiple stages with NODE_NAME based lock

Dusan Nikolov
Hi, 
I'm trying to figure out how/if it's possible to create a lock that will guard multiple stages of a pipeline execution, based on the NODE_NAME on which they are supposed to execute.

I tried putting the lock in the options block of the surrounding stage, but this throws an error because options block in a stage is executed/evaluated before agent block.
Please see the example pipeline below for reference:

pipeline {
agent none
stages {
stage ('main') {
parallel {
stage ('server') {
agent { label 'server' }
// options { lock(NODE_NAME) }
stages {
stage ('all') {
options { lock("${NODE_NAME}") } // this also seems to fail
steps {
script {
echo "${NODE_NAME}"
sleep(10)
}
}
}
}
}
stage ('client') {
agent { label 'client' }
stages {
stage ('all') {
options { lock("${NODE_NAME}") }
steps {
script {
echo "${NODE_NAME}"
sleep(10)
}
}
}
}
}
}
}
}
}

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/6d488fc9-333b-4d30-93f0-4fa59c7769fe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Locking multiple stages with NODE_NAME based lock

Brian Ray
I just made a suggestion on a similar thread. Does lock work inside a steps block? Or a script block? It seems like it should, because it is a step that does something imperatively.

On Thursday, March 7, 2019 at 3:58:59 PM UTC-8, Dusan Nikolov wrote:
Hi, 
I'm trying to figure out how/if it's possible to create a lock that will guard multiple stages of a pipeline execution, based on the NODE_NAME on which they are supposed to execute.

I tried putting the lock in the options block of the surrounding stage, but this throws an error because options block in a stage is executed/evaluated before agent block.
Please see the example pipeline below for reference:

pipeline {
agent none
stages {
stage ('main') {
parallel {
stage ('server') {
agent { label 'server' }
// options { lock(NODE_NAME) }
stages {
stage ('all') {
options { lock("${NODE_NAME}") } // this also seems to fail
steps {
script {
echo "${NODE_NAME}"
sleep(10)
}
}
}
}
}
stage ('client') {
agent { label 'client' }
stages {
stage ('all') {
options { lock("${NODE_NAME}") }
steps {
script {
echo "${NODE_NAME}"
sleep(10)
}
}
}
}
}
}
}
}
}

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/0e87d972-12b5-4b51-ae2d-7f269f417b5a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Locking multiple stages with NODE_NAME based lock

Dusan Nikolov
Hi Brian,

I think lock works within steps and script block, however this doesn't help me. I need bigger to lock a node across several sequential stages to prevent it from being freed.

This is the use case:
Node1 and Node2 start working their parts of a stage in parallel. Parallel stages is used to, well parallelize workflow. End of first parallel block is a synchronization point (end of setup phase).
After this, nodes go into the second parallel block, where they do the testing (testing involves both nodes as a unit).

Nodes can finish phase1 at different speeds, and due to this Jenkins makes the node that is waiting on the other one available for other jobs (including a queued run of this same job). I want to prevent it from being available.

I hope I explained my use case more closely now.

On Sun, Mar 24, 2019, 10:11 AM Brian Ray <[hidden email]> wrote:
I just made a suggestion on a similar thread. Does lock work inside a steps block? Or a script block? It seems like it should, because it is a step that does something imperatively.

On Thursday, March 7, 2019 at 3:58:59 PM UTC-8, Dusan Nikolov wrote:
Hi, 
I'm trying to figure out how/if it's possible to create a lock that will guard multiple stages of a pipeline execution, based on the NODE_NAME on which they are supposed to execute.

I tried putting the lock in the options block of the surrounding stage, but this throws an error because options block in a stage is executed/evaluated before agent block.
Please see the example pipeline below for reference:

pipeline {
agent none
stages {
stage ('main') {
parallel {
stage ('server') {
agent { label 'server' }
// options { lock(NODE_NAME) }
stages {
stage ('all') {
options { lock("${NODE_NAME}") } // this also seems to fail
steps {
script {
echo "${NODE_NAME}"
sleep(10)
}
}
}
}
}
stage ('client') {
agent { label 'client' }
stages {
stage ('all') {
options { lock("${NODE_NAME}") }
steps {
script {
echo "${NODE_NAME}"
sleep(10)
}
}
}
}
}
}
}
}
}

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/Jlhy1hxe92M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/0e87d972-12b5-4b51-ae2d-7f269f417b5a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAFVs2_B8bR%3DHUqsR8Tzy21v8WBg49VVJfTSpZnD-txPrzAU5ng%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Locking multiple stages with NODE_NAME based lock

Brian Ray
Now I understand. I had not considered locking a particular node for re-use in the same job. There must be some way to pin the node even if it is freed up. I will try to think of a solution.

On Wednesday, April 3, 2019 at 7:05:34 PM UTC-7, Dusan Nikolov wrote:
Hi Brian,

I think lock works within steps and script block, however this doesn't help me. I need bigger to lock a node across several sequential stages to prevent it from being freed.

This is the use case:
Node1 and Node2 start working their parts of a stage in parallel. Parallel stages is used to, well parallelize workflow. End of first parallel block is a synchronization point (end of setup phase).
After this, nodes go into the second parallel block, where they do the testing (testing involves both nodes as a unit).

Nodes can finish phase1 at different speeds, and due to this Jenkins makes the node that is waiting on the other one available for other jobs (including a queued run of this same job). I want to prevent it from being available.

I hope I explained my use case more closely now.

On Sun, Mar 24, 2019, 10:11 AM Brian Ray <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="VqJhBhryBgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">be_...@...> wrote:
I just made a suggestion <a href="https://groups.google.com/d/msg/jenkinsci-users/41bKAIfrPAw/zpuyg__7AAAJ" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msg/jenkinsci-users/41bKAIfrPAw/zpuyg__7AAAJ&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msg/jenkinsci-users/41bKAIfrPAw/zpuyg__7AAAJ&#39;;return true;">on a similar thread. Does lock work inside a steps block? Or a script block? It seems like it should, because it is a step that does something imperatively.

On Thursday, March 7, 2019 at 3:58:59 PM UTC-8, Dusan Nikolov wrote:
Hi, 
I'm trying to figure out how/if it's possible to create a lock that will guard multiple stages of a pipeline execution, based on the NODE_NAME on which they are supposed to execute.

I tried putting the lock in the options block of the surrounding stage, but this throws an error because options block in a stage is executed/evaluated before agent block.
Please see the example pipeline below for reference:

pipeline {
agent none
stages {
stage ('main') {
parallel {
stage ('server') {
agent { label 'server' }
// options { lock(NODE_NAME) }
stages {
stage ('all') {
options { lock("${NODE_NAME}") } // this also seems to fail
steps {
script {
echo "${NODE_NAME}"
sleep(10)
}
}
}
}
}
stage ('client') {
agent { label 'client' }
stages {
stage ('all') {
options { lock("${NODE_NAME}") }
steps {
script {
echo "${NODE_NAME}"
sleep(10)
}
}
}
}
}
}
}
}
}

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit <a href="https://groups.google.com/d/topic/jenkinsci-users/Jlhy1hxe92M/unsubscribe" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/topic/jenkinsci-users/Jlhy1hxe92M/unsubscribe&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/topic/jenkinsci-users/Jlhy1hxe92M/unsubscribe&#39;;return true;">https://groups.google.com/d/topic/jenkinsci-users/Jlhy1hxe92M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to <a href="javascript:" rel="nofollow" target="_blank" gdf-obfuscated-mailto="VqJhBhryBgAJ" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jenkins...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/0e87d972-12b5-4b51-ae2d-7f269f417b5a%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/0e87d972-12b5-4b51-ae2d-7f269f417b5a%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/0e87d972-12b5-4b51-ae2d-7f269f417b5a%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/0e87d972-12b5-4b51-ae2d-7f269f417b5a%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d1103e8c-d6e0-4dc2-b6dd-c70786a18317%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.