Using the same label multiple times in pipeline

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Using the same label multiple times in pipeline

Daniel Becroft
Hi,
I'm experimenting with the scripted pipeline, and have a question about the use of node('') with a label. If I have the following scenario:

node('A') { }
node('B') { }
node('A') { // Which node will be used here? }

Is there any guarantee that the second node() step for "A" will hit the same slave as the first one, or will it use whichever slave is now available? I'd like for the first node() to allocate one from the pool of slaves with the label of "A", but then the second one to somehow reuse the first one (there might be some cleanup tasks, etc that need to happen on the original node).

Cheers,
Daniel B.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAJM0iSTyKhk_jcdTrRgkB80mCWi7qn9ANPJMz-XOyfZb_7xmMA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Using the same label multiple times in pipeline

Robert Hales
If you have multiple nodes with a label of A, you will need to declare a variable to hold the name of the specific node you ended up on in the first "node" declaration, and then use that node in the third one: 

def allocatedNode

node('A') { allocatedNode = $NODE_NAME } //NODE_NAME is an environment variable provided by Jenkins
node('B') {}
node($NODE_NAME) {}


HOWEVER, be aware that this isn't always a safe thing to do, depending on what your goal is. There is no guarantee that the workspace that was used in the first node block will still be there when the third block gets there. Even if it is, there is no guarantee that the third block will use the original workspace. It probably will most of the time, but not guaranteed. Depending on what you are doing, you may need to allocate a custom workspace (still not a guaranteed solution), move to a separate directory, stash/unstash the things you need between nodes, or some other solution. 

On Sunday, October 29, 2017 at 8:51:02 PM UTC-6, Daniel Becroft wrote:
Hi,
I'm experimenting with the scripted pipeline, and have a question about the use of node('') with a label. If I have the following scenario:

node('A') { }
node('B') { }
node('A') { // Which node will be used here? }

Is there any guarantee that the second node() step for "A" will hit the same slave as the first one, or will it use whichever slave is now available? I'd like for the first node() to allocate one from the pool of slaves with the label of "A", but then the second one to somehow reuse the first one (there might be some cleanup tasks, etc that need to happen on the original node).

Cheers,
Daniel B.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/f43a48c1-15bd-4feb-9a91-3e818a3cf268%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

RE: Using the same label multiple times in pipeline

danielsimonbutler

Couple of approaches:

You could use the External Workspace Manager plugin: https://jenkins.io/doc/pipeline/steps/external-workspace-manager/

 

Or as Robert suggested use the ws(){} block, this should give a guarantee on the workspace but you’ll need to confirm that you’re not about to fill up your node’s disks with old workspaces:

Something along the lines of:
def sharedWorkspace = “/shared-workspace/${JOB_NAME}/${BUILD_NUMBER}”

node(‘A’){

    ws(sharedWorkspace){

        //do stuff

    }

}

node(‘B’){}

node(‘A’){

    ws(sharedWorkspace){

        //do more stuff

    }

}

 

 

From: [hidden email]
Sent: 30 October 2017 03:04
To: [hidden email]
Subject: Re: Using the same label multiple times in pipeline

 

If you have multiple nodes with a label of A, you will need to declare a variable to hold the name of the specific node you ended up on in the first "node" declaration, and then use that node in the third one: 

 

def allocatedNode

 

node('A') { allocatedNode = $NODE_NAME } //NODE_NAME is an environment variable provided by Jenkins

node('B') {}

node($NODE_NAME) {}

 

 

HOWEVER, be aware that this isn't always a safe thing to do, depending on what your goal is. There is no guarantee that the workspace that was used in the first node block will still be there when the third block gets there. Even if it is, there is no guarantee that the third block will use the original workspace. It probably will most of the time, but not guaranteed. Depending on what you are doing, you may need to allocate a custom workspace (still not a guaranteed solution), move to a separate directory, stash/unstash the things you need between nodes, or some other solution. 

On Sunday, October 29, 2017 at 8:51:02 PM UTC-6, Daniel Becroft wrote:

Hi,

I'm experimenting with the scripted pipeline, and have a question about the use of node('') with a label. If I have the following scenario:

 

node('A') { }

node('B') { }

node('A') { // Which node will be used here? }

 

Is there any guarantee that the second node() step for "A" will hit the same slave as the first one, or will it use whichever slave is now available? I'd like for the first node() to allocate one from the pool of slaves with the label of "A", but then the second one to somehow reuse the first one (there might be some cleanup tasks, etc that need to happen on the original node).

 

Cheers,

Daniel B.

 

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/f43a48c1-15bd-4feb-9a91-3e818a3cf268%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

 

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/59f6eb23.c8e1500a.573c1.f1ae%40mx.google.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Using the same label multiple times in pipeline

Robert Hales
The ws{} block will not give a guarantee that you will get the same workspace. If, for example, 2 jobs run at the same time, they cannot use the workspace concurrently. The second one to request the workspace will get one with a @2 on the end. You would have to make sure multiple jobs can't run at the same time, or use resource locking to prevent concurrent use of the workspace.  If you don't care about concurrent use, then skip the ws{} and use dir{} instead. That will always use the same directory and doesn't care about multiple jobs accessing it. 


On Monday, October 30, 2017 at 3:05:03 AM UTC-6, Daniel Butler wrote:

Couple of approaches:

You could use the External Workspace Manager plugin: <a href="https://jenkins.io/doc/pipeline/steps/external-workspace-manager/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fjenkins.io%2Fdoc%2Fpipeline%2Fsteps%2Fexternal-workspace-manager%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH-RhFdRrgWRZk3z9kDAIMCI1x4Tw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fjenkins.io%2Fdoc%2Fpipeline%2Fsteps%2Fexternal-workspace-manager%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH-RhFdRrgWRZk3z9kDAIMCI1x4Tw&#39;;return true;">https://jenkins.io/doc/pipeline/steps/external-workspace-manager/

 

Or as Robert suggested use the ws(){} block, this should give a guarantee on the workspace but you’ll need to confirm that you’re not about to fill up your node’s disks with old workspaces:

Something along the lines of:
def sharedWorkspace = “/shared-workspace/${JOB_NAME}/${BUILD_NUMBER}”

node(‘A’){

    ws(sharedWorkspace){

        //do stuff

    }

}

node(‘B’){}

node(‘A’){

    ws(sharedWorkspace){

        //do more stuff

    }

}

 

 

From: <a href="javascript:" target="_blank" gdf-obfuscated-mailto="0rMB7NqtAgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">Robert Hales
Sent: 30 October 2017 03:04
To: <a href="javascript:" target="_blank" gdf-obfuscated-mailto="0rMB7NqtAgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">Jenkins Users
Subject: Re: Using the same label multiple times in pipeline

 

If you have multiple nodes with a label of A, you will need to declare a variable to hold the name of the specific node you ended up on in the first "node" declaration, and then use that node in the third one: 

 

def allocatedNode

 

node('A') { allocatedNode = $NODE_NAME } //NODE_NAME is an environment variable provided by Jenkins

node('B') {}

node($NODE_NAME) {}

 

 

HOWEVER, be aware that this isn't always a safe thing to do, depending on what your goal is. There is no guarantee that the workspace that was used in the first node block will still be there when the third block gets there. Even if it is, there is no guarantee that the third block will use the original workspace. It probably will most of the time, but not guaranteed. Depending on what you are doing, you may need to allocate a custom workspace (still not a guaranteed solution), move to a separate directory, stash/unstash the things you need between nodes, or some other solution. 

On Sunday, October 29, 2017 at 8:51:02 PM UTC-6, Daniel Becroft wrote:

Hi,

I'm experimenting with the scripted pipeline, and have a question about the use of node('') with a label. If I have the following scenario:

 

node('A') { }

node('B') { }

node('A') { // Which node will be used here? }

 

Is there any guarantee that the second node() step for "A" will hit the same slave as the first one, or will it use whichever slave is now available? I'd like for the first node() to allocate one from the pool of slaves with the label of "A", but then the second one to somehow reuse the first one (there might be some cleanup tasks, etc that need to happen on the original node).

 

Cheers,

Daniel B.

 

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="0rMB7NqtAgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/f43a48c1-15bd-4feb-9a91-3e818a3cf268%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/f43a48c1-15bd-4feb-9a91-3e818a3cf268%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/f43a48c1-15bd-4feb-9a91-3e818a3cf268%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/f43a48c1-15bd-4feb-9a91-3e818a3cf268%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

 

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/60f2e5ee-df59-4785-a7ed-2e5a656aa861%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Using the same label multiple times in pipeline

Daniel Becroft
Thanks for the reply, Robert and Daniel. The actual workspace that I get is not an issue - I just need to get back to that agent in order to do some cleanup. 

My scenario is:
node('A') => create and initialize a database, and start it on the port. Database is outside of the workspace, in a build# folder
node('B') => run the integration tests against that database
node('A') => stop and remove the database

This currently works as I only have a single node with a label of "a", but we might be adding more.

Thanks again.

On Tue, Oct 31, 2017 at 12:51 AM Robert Hales <[hidden email]> wrote:
The ws{} block will not give a guarantee that you will get the same workspace. If, for example, 2 jobs run at the same time, they cannot use the workspace concurrently. The second one to request the workspace will get one with a @2 on the end. You would have to make sure multiple jobs can't run at the same time, or use resource locking to prevent concurrent use of the workspace.  If you don't care about concurrent use, then skip the ws{} and use dir{} instead. That will always use the same directory and doesn't care about multiple jobs accessing it. 


On Monday, October 30, 2017 at 3:05:03 AM UTC-6, Daniel Butler wrote:

Couple of approaches:

You could use the External Workspace Manager plugin: https://jenkins.io/doc/pipeline/steps/external-workspace-manager/

 

Or as Robert suggested use the ws(){} block, this should give a guarantee on the workspace but you’ll need to confirm that you’re not about to fill up your node’s disks with old workspaces:

Something along the lines of:
def sharedWorkspace = “/shared-workspace/${JOB_NAME}/${BUILD_NUMBER}”

node(‘A’){

    ws(sharedWorkspace){

        //do stuff

    }

}

node(‘B’){}

node(‘A’){

    ws(sharedWorkspace){

        //do more stuff

    }

}

 

 

From: Robert Hales
Sent: 30 October 2017 03:04
To: Jenkins Users
Subject: Re: Using the same label multiple times in pipeline

 

If you have multiple nodes with a label of A, you will need to declare a variable to hold the name of the specific node you ended up on in the first "node" declaration, and then use that node in the third one: 

 

def allocatedNode

 

node('A') { allocatedNode = $NODE_NAME } //NODE_NAME is an environment variable provided by Jenkins

node('B') {}

node($NODE_NAME) {}

 

 

HOWEVER, be aware that this isn't always a safe thing to do, depending on what your goal is. There is no guarantee that the workspace that was used in the first node block will still be there when the third block gets there. Even if it is, there is no guarantee that the third block will use the original workspace. It probably will most of the time, but not guaranteed. Depending on what you are doing, you may need to allocate a custom workspace (still not a guaranteed solution), move to a separate directory, stash/unstash the things you need between nodes, or some other solution. 

On Sunday, October 29, 2017 at 8:51:02 PM UTC-6, Daniel Becroft wrote:

Hi,

I'm experimenting with the scripted pipeline, and have a question about the use of node('') with a label. If I have the following scenario:

 

node('A') { }

node('B') { }

node('A') { // Which node will be used here? }

 

Is there any guarantee that the second node() step for "A" will hit the same slave as the first one, or will it use whichever slave is now available? I'd like for the first node() to allocate one from the pool of slaves with the label of "A", but then the second one to somehow reuse the first one (there might be some cleanup tasks, etc that need to happen on the original node).

 

Cheers,

Daniel B.

 

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/60f2e5ee-df59-4785-a7ed-2e5a656aa861%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAJM0iSRjoYmb27JUoNvmLLxj2yVW-onjhk7Zf1jwcCdSKKtSbg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Using the same label multiple times in pipeline

Andreas Tscharner-2
In reply to this post by Daniel Becroft
On 30.10.2017 03:50, Daniel Becroft wrote:

> Hi,
> I'm experimenting with the scripted pipeline, and have a question about
> the use of node('') with a label. If I have the following scenario:
>
> node('A') { }
> node('B') { }
> node('A') { // Which node will be used here? }
>
> Is there any guarantee that the second node() step for "A" will hit the
> same slave as the first one, or will it use whichever slave is now
> available? I'd like for the first node() to allocate one from the pool
> of slaves with the label of "A", but then the second one to somehow
> reuse the first one (there might be some cleanup tasks, etc that need to
> happen on the original node).

I would try something like this:

node('A') {
   ...
   ...
   node('B') {
     ...
     ...
   }
   ...
   ...
}

HTH and best regards
        Andreas
--
Andreas Tscharner                             [hidden email]

Gordon's Law:
If you think you have the solution, the question was poorly phrased.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/2ca95dc7-20f3-385e-bf87-c88dd8888ef5%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Using the same label multiple times in pipeline

Lee Meador-3
You can use 'stash' to solve this problem since you can't count on the same workspace in part 3 as in part 1 nor, if it is the same workspace can you count on it being unchanged in the interim. You can get confused by trying it and maybe you do get it this time but not next time. Further, depending on whether one build agent has the both labels, A and B, all three parts could happen in the same workspace.

node('A') {
    ... some stuff that creates the files you want to use in part 3 (perhaps load stuff from source code repo and build everything)
     stash 'stuff'
 }
node('B') { }
node('A') { 
    unstash 'stuff'
     .... use those files
 }

You need to understand the implications of what stash does. Basically it takes all files in the workspace (unless you specify specific ones) and stores them compressed on the master. the unstash retrieves them and puts them in the workspace. Perhaps you want to delete the workspace content before unstashing or maybe move to a subfolder.

On Thu, Nov 2, 2017 at 10:14 AM, Andreas Tscharner <[hidden email]> wrote:
On 30.10.2017 03:50, Daniel Becroft wrote:
Hi,
I'm experimenting with the scripted pipeline, and have a question about the use of node('') with a label. If I have the following scenario:

node('A') { }
node('B') { }
node('A') { // Which node will be used here? }

Is there any guarantee that the second node() step for "A" will hit the same slave as the first one, or will it use whichever slave is now available? I'd like for the first node() to allocate one from the pool of slaves with the label of "A", but then the second one to somehow reuse the first one (there might be some cleanup tasks, etc that need to happen on the original node).

I would try something like this:

node('A') {
  ...
  ...
  node('B') {
    ...
    ...
  }
  ...
  ...
}

HTH and best regards
        Andreas
--
Andreas Tscharner                             [hidden email]

Gordon's Law:
If you think you have the solution, the question was poorly phrased.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/2ca95dc7-20f3-385e-bf87-c88dd8888ef5%40gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
-- Lee Meador
Sent from gmail. My real email address is lee AT leemeador.com

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAA4vtDQtkcOO0cZ%2BOW8gU68uA_RqT1cacnsYXjg1E0bnOJ7nMA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.