Which agents are used when a pipeline calls multiple downstream freestyle projects

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Which agents are used when a pipeline calls multiple downstream freestyle projects

Tony P
Hi,

We have multiple pipelines for which each will run 3+ downstream freestyle projects.

This process is defined by an automation tool that creates and initiates Jenkins builds, so using other methodologies is out :-). The problem I have is that I need all the freestyle projects to run on the same agent, because the first couple of projects clone repos and other projects then do the build or deploy - so obviously it all needs to happen in exactly the same place.

I also want to initiate this process using agent/slave labels rather than agent/slave names as we will have a group of agents to handle the multiple sometimes duplicate pipeline builds.

I am beginning to think that even though I am using labels Jenkins will always run all the downstream projects on the same agent. Trouble is I am having trouble finding any conclusive documentation that says I am right or I am wrong - I have spent an awful lot of time in the company of Mr Google trying to figure this out. If each of the downstream projects could potentially run on any available agent using the same label then I have a big problem and my next task is to figure out how to get them to all use the same agent.

Can someone tell me whether I am correct that Jenkins will run all downstream projects on the same agent and/or where do I look for this level of documentation ?

Thanks in advance

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/e806f3a6-02c8-4fbd-b574-d29fb8dee547%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Which agents are used when a pipeline calls multiple downstream freestyle projects

stephenconnolly
I think you would need to write a plugin that enforces your "same node" constraint.

It would not be general utility as eg enough people are doing builds on ephemeral agents and you'd end up with stuck jobs waiting for a free executor on the same "ephemeral" node that will never return.

But I can see it being of utility for some people. Probably a JobProperty that does the enforcing and some other coordination token to identify which jobs you need to be on the same node as (plus a timeout if you are stuck for too long waiting on that node)
On Sat 18 Mar 2017 at 10:28, Tony P <[hidden email]> wrote:
Hi,

We have multiple pipelines for which each will run 3+ downstream freestyle projects.

This process is defined by an automation tool that creates and initiates Jenkins builds, so using other methodologies is out :-). The problem I have is that I need all the freestyle projects to run on the same agent, because the first couple of projects clone repos and other projects then do the build or deploy - so obviously it all needs to happen in exactly the same place.

I also want to initiate this process using agent/slave labels rather than agent/slave names as we will have a group of agents to handle the multiple sometimes duplicate pipeline builds.

I am beginning to think that even though I am using labels Jenkins will always run all the downstream projects on the same agent. Trouble is I am having trouble finding any conclusive documentation that says I am right or I am wrong - I have spent an awful lot of time in the company of Mr Google trying to figure this out. If each of the downstream projects could potentially run on any available agent using the same label then I have a big problem and my next task is to figure out how to get them to all use the same agent.

Can someone tell me whether I am correct that Jenkins will run all downstream projects on the same agent and/or where do I look for this level of documentation ?

Thanks in advance

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/e806f3a6-02c8-4fbd-b574-d29fb8dee547%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Sent from my phone

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CA%2BnPnMzREdXYwpgUSPCf2k%3D8_Ny8U9y%2BTezygSkMqxA-_E1Dew%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Which agents are used when a pipeline calls multiple downstream freestyle projects

Tony P
Does this mean I am correct that the default behavior is for downstream projects to run on any agents that have the same label ? This was perhaps the big thing I was trying to find out.

Also our builds are done via "build job: " but I am struggling to find out how to force it to use a particular agent.

My plan has been kick off the first downstream project, find out which agent it used and then force the remaining downstream projects to use that agent. Try as I may and I tried so many ideas, I can't find out how to determine what agent that downstream agent used. It seems next to impossible ?

Thanks again

On Sunday, March 19, 2017 at 12:27:03 AM UTC+13, Stephen Connolly wrote:
I think you would need to write a plugin that enforces your "same node" constraint.

It would not be general utility as eg enough people are doing builds on ephemeral agents and you'd end up with stuck jobs waiting for a free executor on the same "ephemeral" node that will never return.

But I can see it being of utility for some people. Probably a JobProperty that does the enforcing and some other coordination token to identify which jobs you need to be on the same node as (plus a timeout if you are stuck for too long waiting on that node)
On Sat 18 Mar 2017 at 10:28, Tony P <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="2lDDzyLMAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">ton...@...> wrote:
Hi,

We have multiple pipelines for which each will run 3+ downstream freestyle projects.

This process is defined by an automation tool that creates and initiates Jenkins builds, so using other methodologies is out :-). The problem I have is that I need all the freestyle projects to run on the same agent, because the first couple of projects clone repos and other projects then do the build or deploy - so obviously it all needs to happen in exactly the same place.

I also want to initiate this process using agent/slave labels rather than agent/slave names as we will have a group of agents to handle the multiple sometimes duplicate pipeline builds.

I am beginning to think that even though I am using labels Jenkins will always run all the downstream projects on the same agent. Trouble is I am having trouble finding any conclusive documentation that says I am right or I am wrong - I have spent an awful lot of time in the company of Mr Google trying to figure this out. If each of the downstream projects could potentially run on any available agent using the same label then I have a big problem and my next task is to figure out how to get them to all use the same agent.

Can someone tell me whether I am correct that Jenkins will run all downstream projects on the same agent and/or where do I look for this level of documentation ?

Thanks in advance

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="2lDDzyLMAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/e806f3a6-02c8-4fbd-b574-d29fb8dee547%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/e806f3a6-02c8-4fbd-b574-d29fb8dee547%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/e806f3a6-02c8-4fbd-b574-d29fb8dee547%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/e806f3a6-02c8-4fbd-b574-d29fb8dee547%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.
--
Sent from my phone

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/52b89bf8-4615-40cb-affd-51ab29792f00%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Which agents are used when a pipeline calls multiple downstream freestyle projects

stephenconnolly
Downstream jobs will run on any agent matching the label they are configured for, with preference given to running on the agent that they most recently ran on *if idle*.

Upstream jobs have no affect - unless they are still running at the time - in which case they may force a *different* agent if they are occupying all available slots.
On Sat 18 Mar 2017 at 19:22, Tony P <[hidden email]> wrote:
Does this mean I am correct that the default behavior is for downstream projects to run on any agents that have the same label ? This was perhaps the big thing I was trying to find out.

Also our builds are done via "build job: " but I am struggling to find out how to force it to use a particular agent.

My plan has been kick off the first downstream project, find out which agent it used and then force the remaining downstream projects to use that agent. Try as I may and I tried so many ideas, I can't find out how to determine what agent that downstream agent used. It seems next to impossible ?

Thanks again


On Sunday, March 19, 2017 at 12:27:03 AM UTC+13, Stephen Connolly wrote:
I think you would need to write a plugin that enforces your "same node" constraint.

It would not be general utility as eg enough people are doing builds on ephemeral agents and you'd end up with stuck jobs waiting for a free executor on the same "ephemeral" node that will never return.

But I can see it being of utility for some people. Probably a JobProperty that does the enforcing and some other coordination token to identify which jobs you need to be on the same node as (plus a timeout if you are stuck for too long waiting on that node)
On Sat 18 Mar 2017 at 10:28, Tony P <[hidden email]> wrote:
Hi,

We have multiple pipelines for which each will run 3+ downstream freestyle projects.

This process is defined by an automation tool that creates and initiates Jenkins builds, so using other methodologies is out :-). The problem I have is that I need all the freestyle projects to run on the same agent, because the first couple of projects clone repos and other projects then do the build or deploy - so obviously it all needs to happen in exactly the same place.

I also want to initiate this process using agent/slave labels rather than agent/slave names as we will have a group of agents to handle the multiple sometimes duplicate pipeline builds.

I am beginning to think that even though I am using labels Jenkins will always run all the downstream projects on the same agent. Trouble is I am having trouble finding any conclusive documentation that says I am right or I am wrong - I have spent an awful lot of time in the company of Mr Google trying to figure this out. If each of the downstream projects could potentially run on any available agent using the same label then I have a big problem and my next task is to figure out how to get them to all use the same agent.

Can someone tell me whether I am correct that Jenkins will run all downstream projects on the same agent and/or where do I look for this level of documentation ?

Thanks in advance

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
--
Sent from my phone

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/52b89bf8-4615-40cb-affd-51ab29792f00%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Sent from my phone

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CA%2BnPnMzFi1wJnD7R-BAcQd%2Bj7e9nCAgLQy24e2QGkt9DaXSAng%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Which agents are used when a pipeline calls multiple downstream freestyle projects

Tony P
Found the solution wrap the whole thing in a node() block and everything within the node block will run on the same node

Cheers

On Sunday, March 19, 2017 at 10:47:45 AM UTC+13, Stephen Connolly wrote:
Downstream jobs will run on any agent matching the label they are configured for, with preference given to running on the agent that they most recently ran on *if idle*.

Upstream jobs have no affect - unless they are still running at the time - in which case they may force a *different* agent if they are occupying all available slots.
On Sat 18 Mar 2017 at 19:22, Tony P <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="bJir3RmiBwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">ton...@...> wrote:
Does this mean I am correct that the default behavior is for downstream projects to run on any agents that have the same label ? This was perhaps the big thing I was trying to find out.

Also our builds are done via "build job: " but I am struggling to find out how to force it to use a particular agent.

My plan has been kick off the first downstream project, find out which agent it used and then force the remaining downstream projects to use that agent. Try as I may and I tried so many ideas, I can't find out how to determine what agent that downstream agent used. It seems next to impossible ?

Thanks again


On Sunday, March 19, 2017 at 12:27:03 AM UTC+13, Stephen Connolly wrote:
I think you would need to write a plugin that enforces your "same node" constraint.

It would not be general utility as eg enough people are doing builds on ephemeral agents and you'd end up with stuck jobs waiting for a free executor on the same "ephemeral" node that will never return.

But I can see it being of utility for some people. Probably a JobProperty that does the enforcing and some other coordination token to identify which jobs you need to be on the same node as (plus a timeout if you are stuck for too long waiting on that node)
On Sat 18 Mar 2017 at 10:28, Tony P <[hidden email]> wrote:
Hi,

We have multiple pipelines for which each will run 3+ downstream freestyle projects.

This process is defined by an automation tool that creates and initiates Jenkins builds, so using other methodologies is out :-). The problem I have is that I need all the freestyle projects to run on the same agent, because the first couple of projects clone repos and other projects then do the build or deploy - so obviously it all needs to happen in exactly the same place.

I also want to initiate this process using agent/slave labels rather than agent/slave names as we will have a group of agents to handle the multiple sometimes duplicate pipeline builds.

I am beginning to think that even though I am using labels Jenkins will always run all the downstream projects on the same agent. Trouble is I am having trouble finding any conclusive documentation that says I am right or I am wrong - I have spent an awful lot of time in the company of Mr Google trying to figure this out. If each of the downstream projects could potentially run on any available agent using the same label then I have a big problem and my next task is to figure out how to get them to all use the same agent.

Can someone tell me whether I am correct that Jenkins will run all downstream projects on the same agent and/or where do I look for this level of documentation ?

Thanks in advance

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/e806f3a6-02c8-4fbd-b574-d29fb8dee547%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/e806f3a6-02c8-4fbd-b574-d29fb8dee547%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/e806f3a6-02c8-4fbd-b574-d29fb8dee547%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/e806f3a6-02c8-4fbd-b574-d29fb8dee547%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.
--
Sent from my phone

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="bJir3RmiBwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/52b89bf8-4615-40cb-affd-51ab29792f00%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/52b89bf8-4615-40cb-affd-51ab29792f00%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/52b89bf8-4615-40cb-affd-51ab29792f00%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/52b89bf8-4615-40cb-affd-51ab29792f00%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.
--
Sent from my phone

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/a6aedb7f-a48b-4359-9319-13aa7a4b893d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Which agents are used when a pipeline calls multiple downstream freestyle projects

stephenconnolly
I thought you were launching freestyle jobs from a pipeline (where this will not work)?

If I'd known toyou were doing all in the same pipeline I could have told you this sooner...

On Sun 19 Mar 2017 at 16:21, Tony P <[hidden email]> wrote:
Found the solution wrap the whole thing in a node() block and everything within the node block will run on the same node

Cheers


On Sunday, March 19, 2017 at 10:47:45 AM UTC+13, Stephen Connolly wrote:
Downstream jobs will run on any agent matching the label they are configured for, with preference given to running on the agent that they most recently ran on *if idle*.

Upstream jobs have no affect - unless they are still running at the time - in which case they may force a *different* agent if they are occupying all available slots.
On Sat 18 Mar 2017 at 19:22, Tony P <[hidden email]> wrote:
Does this mean I am correct that the default behavior is for downstream projects to run on any agents that have the same label ? This was perhaps the big thing I was trying to find out.

Also our builds are done via "build job: " but I am struggling to find out how to force it to use a particular agent.

My plan has been kick off the first downstream project, find out which agent it used and then force the remaining downstream projects to use that agent. Try as I may and I tried so many ideas, I can't find out how to determine what agent that downstream agent used. It seems next to impossible ?

Thanks again


On Sunday, March 19, 2017 at 12:27:03 AM UTC+13, Stephen Connolly wrote:
I think you would need to write a plugin that enforces your "same node" constraint.

It would not be general utility as eg enough people are doing builds on ephemeral agents and you'd end up with stuck jobs waiting for a free executor on the same "ephemeral" node that will never return.

But I can see it being of utility for some people. Probably a JobProperty that does the enforcing and some other coordination token to identify which jobs you need to be on the same node as (plus a timeout if you are stuck for too long waiting on that node)
On Sat 18 Mar 2017 at 10:28, Tony P <[hidden email]> wrote:
Hi,

We have multiple pipelines for which each will run 3+ downstream freestyle projects.

This process is defined by an automation tool that creates and initiates Jenkins builds, so using other methodologies is out :-). The problem I have is that I need all the freestyle projects to run on the same agent, because the first couple of projects clone repos and other projects then do the build or deploy - so obviously it all needs to happen in exactly the same place.

I also want to initiate this process using agent/slave labels rather than agent/slave names as we will have a group of agents to handle the multiple sometimes duplicate pipeline builds.

I am beginning to think that even though I am using labels Jenkins will always run all the downstream projects on the same agent. Trouble is I am having trouble finding any conclusive documentation that says I am right or I am wrong - I have spent an awful lot of time in the company of Mr Google trying to figure this out. If each of the downstream projects could potentially run on any available agent using the same label then I have a big problem and my next task is to figure out how to get them to all use the same agent.

Can someone tell me whether I am correct that Jenkins will run all downstream projects on the same agent and/or where do I look for this level of documentation ?

Thanks in advance

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
--
Sent from my phone

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
--
Sent from my phone

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/a6aedb7f-a48b-4359-9319-13aa7a4b893d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Sent from my phone

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CA%2BnPnMwW%2B1ptc4mGvgUaMsnaTk8ChtHpjEaSKQkaa%3D3N5ePPXQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.