Run pipeline flyweight executors off of master

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Run pipeline flyweight executors off of master

Eli White
We follow the Jenkins configuration best practices and have no executors on our master node and force everything to run on our agents. 

We are starting to work with pipeline jobs and are worried that bad Jenkinsfiles could cause problems on our master. 
How can we force the flyweight jobs to run on a designated machine *other* than master? 

How do other people handle this?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Run pipeline flyweight executors off of master

Craig Rodrigues
In your pipeline, the node parameter can take an argument of which node to run on.

See this example:
https://github.com/freebsd/freebsd-ci/blob/master/scripts/build/build-test.groovy#L100

In my job, I defined BUILD_NODE with the NodeLabel Parameter Plugin ( https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin ).

--
Craig



On Mon, Jun 13, 2016 at 2:03 PM, Eli White <[hidden email]> wrote:
We follow the Jenkins configuration best practices and have no executors on our master node and force everything to run on our agents. 

We are starting to work with pipeline jobs and are worried that bad Jenkinsfiles could cause problems on our master. 
How can we force the flyweight jobs to run on a designated machine *other* than master? 

How do other people handle this?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAG%3DrPVdn8jc2EUH77KYYxw0ghcsgo_eCt8rycn20oP5xuqEeAg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Run pipeline flyweight executors off of master

Eli White
My understanding is that Jenkinsfile execution runs as a flyweight node on master, but then uses heavyweight nodes on the given node label to execute.

Per this file: https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md

Why are there two executors consumed by one Pipeline build?

  • Every Pipeline build itself runs on the master, using a flyweight executor — an uncounted slot that is assumed to not take any significant computational power.
  • This executor represents the actual Groovy script, which is almost always idle, waiting for a step to complete.
  • Flyweight executors are always available.

On Monday, June 13, 2016 at 2:11:19 PM UTC-7, Craig Rodrigues wrote:
In your pipeline, the node parameter can take an argument of which node to run on.

See this example:
<a href="https://github.com/freebsd/freebsd-ci/blob/master/scripts/build/build-test.groovy#L100" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Ffreebsd%2Ffreebsd-ci%2Fblob%2Fmaster%2Fscripts%2Fbuild%2Fbuild-test.groovy%23L100\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHvncLQwvcdrPDkJa0qDGRvGlfEKA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Ffreebsd%2Ffreebsd-ci%2Fblob%2Fmaster%2Fscripts%2Fbuild%2Fbuild-test.groovy%23L100\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHvncLQwvcdrPDkJa0qDGRvGlfEKA&#39;;return true;">https://github.com/freebsd/freebsd-ci/blob/master/scripts/build/build-test.groovy#L100

In my job, I defined BUILD_NODE with the NodeLabel Parameter Plugin ( <a href="https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwiki.jenkins-ci.org%2Fdisplay%2FJENKINS%2FNodeLabel%2BParameter%2BPlugin\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF2fSi7uK_SN1cTtnx2k4G4WuKuyw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwiki.jenkins-ci.org%2Fdisplay%2FJENKINS%2FNodeLabel%2BParameter%2BPlugin\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF2fSi7uK_SN1cTtnx2k4G4WuKuyw&#39;;return true;">https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin ).

--
Craig



On Mon, Jun 13, 2016 at 2:03 PM, Eli White <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="Hpb9XdDkAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">e...@...> wrote:
We follow the Jenkins configuration best practices and have no executors on our master node and force everything to run on our agents. 

We are starting to work with pipeline jobs and are worried that bad Jenkinsfiles could cause problems on our master. 
How can we force the flyweight jobs to run on a designated machine *other* than master? 

How do other people handle this?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="Hpb9XdDkAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/a41c28a9-fc93-459f-b990-87cb1210cb64%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Run pipeline flyweight executors off of master

Thomas Zoratto
Hello, 

I’m not sure one can do what you want.

Out of curiosity, what’s the use case for this ?


Le 13 juin 2016 à 23:16, Eli White <[hidden email]> a écrit :

My understanding is that Jenkinsfile execution runs as a flyweight node on master, but then uses heavyweight nodes on the given node label to execute.


Why are there two executors consumed by one Pipeline build?

  • Every Pipeline build itself runs on the master, using a flyweight executor — an uncounted slot that is assumed to not take any significant computational power.
  • This executor represents the actual Groovy script, which is almost always idle, waiting for a step to complete.
  • Flyweight executors are always available.

On Monday, June 13, 2016 at 2:11:19 PM UTC-7, Craig Rodrigues wrote:
In your pipeline, the node parameter can take an argument of which node to run on.

See this example:
<a href="https://github.com/freebsd/freebsd-ci/blob/master/scripts/build/build-test.groovy#L100" target="_blank" rel="nofollow" onmousedown="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Ffreebsd%2Ffreebsd-ci%2Fblob%2Fmaster%2Fscripts%2Fbuild%2Fbuild-test.groovy%23L100\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHvncLQwvcdrPDkJa0qDGRvGlfEKA';return true;" onclick="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Ffreebsd%2Ffreebsd-ci%2Fblob%2Fmaster%2Fscripts%2Fbuild%2Fbuild-test.groovy%23L100\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHvncLQwvcdrPDkJa0qDGRvGlfEKA';return true;" class="">https://github.com/freebsd/freebsd-ci/blob/master/scripts/build/build-test.groovy#L100

In my job, I defined BUILD_NODE with the NodeLabel Parameter Plugin ( <a href="https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin" target="_blank" rel="nofollow" onmousedown="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fwiki.jenkins-ci.org%2Fdisplay%2FJENKINS%2FNodeLabel%2BParameter%2BPlugin\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF2fSi7uK_SN1cTtnx2k4G4WuKuyw';return true;" onclick="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fwiki.jenkins-ci.org%2Fdisplay%2FJENKINS%2FNodeLabel%2BParameter%2BPlugin\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF2fSi7uK_SN1cTtnx2k4G4WuKuyw';return true;" class="">https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin ).

--
Craig



On Mon, Jun 13, 2016 at 2:03 PM, Eli White <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="Hpb9XdDkAwAJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;" class="">e...@...> wrote:
We follow the Jenkins configuration best practices and have no executors on our master node and force everything to run on our agents. 

We are starting to work with pipeline jobs and are worried that bad Jenkinsfiles could cause problems on our master. 
How can we force the flyweight jobs to run on a designated machine *other* than master? 

How do other people handle this?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="Hpb9XdDkAwAJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;" class="">jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter';return true;" onclick="this.href='https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter';return true;" class="">https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;" class="">https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/a41c28a9-fc93-459f-b990-87cb1210cb64%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/9823286F-055A-4C0B-8DD7-D62B87BCB5B6%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Run pipeline flyweight executors off of master

Eli White
By not running anything on master we don't have to worry about any sort of failure related to misconfiguration of jobs, or job related failures. For example, OOM, out of disk space, pausing when 'input' is run, etc. 

If a slave goes down you have 1 machine down. If master goes down, all CI for our company goes down. The need for us to protect and make behavior guarantees of the master node, we want it to do only what is absolutely necessary and keep user code from running on it. 

On Monday, June 13, 2016 at 2:19:56 PM UTC-7, Thomas Zoratto wrote:
Hello, 

I’m not sure one can do what you want.

Out of curiosity, what’s the use case for this ?


Le 13 juin 2016 à 23:16, Eli White <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="OSztiEjlAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">e...@...> a écrit :

My understanding is that Jenkinsfile execution runs as a flyweight node on master, but then uses heavyweight nodes on the given node label to execute.

Per this file: <a href="https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Fpipeline-plugin%2Fblob%2Fmaster%2FTUTORIAL.md\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmRd5uoaEGwemnrAEu1_3Yf_1bbQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Fpipeline-plugin%2Fblob%2Fmaster%2FTUTORIAL.md\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmRd5uoaEGwemnrAEu1_3Yf_1bbQ&#39;;return true;">https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md

Why are there two executors consumed by one Pipeline build?

  • Every Pipeline build itself runs on the master, using a flyweight executor — an uncounted slot that is assumed to not take any significant computational power.
  • This executor represents the actual Groovy script, which is almost always idle, waiting for a step to complete.
  • Flyweight executors are always available.

On Monday, June 13, 2016 at 2:11:19 PM UTC-7, Craig Rodrigues wrote:
In your pipeline, the node parameter can take an argument of which node to run on.

See this example:
<a href="https://github.com/freebsd/freebsd-ci/blob/master/scripts/build/build-test.groovy#L100" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Ffreebsd%2Ffreebsd-ci%2Fblob%2Fmaster%2Fscripts%2Fbuild%2Fbuild-test.groovy%23L100\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHvncLQwvcdrPDkJa0qDGRvGlfEKA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Ffreebsd%2Ffreebsd-ci%2Fblob%2Fmaster%2Fscripts%2Fbuild%2Fbuild-test.groovy%23L100\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHvncLQwvcdrPDkJa0qDGRvGlfEKA&#39;;return true;">https://github.com/freebsd/freebsd-ci/blob/master/scripts/build/build-test.groovy#L100

In my job, I defined BUILD_NODE with the NodeLabel Parameter Plugin ( <a href="https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwiki.jenkins-ci.org%2Fdisplay%2FJENKINS%2FNodeLabel%2BParameter%2BPlugin\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF2fSi7uK_SN1cTtnx2k4G4WuKuyw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwiki.jenkins-ci.org%2Fdisplay%2FJENKINS%2FNodeLabel%2BParameter%2BPlugin\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF2fSi7uK_SN1cTtnx2k4G4WuKuyw&#39;;return true;">https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin ).

--
Craig



On Mon, Jun 13, 2016 at 2:03 PM, Eli White <[hidden email]> wrote:
We follow the Jenkins configuration best practices and have no executors on our master node and force everything to run on our agents. 

We are starting to work with pipeline jobs and are worried that bad Jenkinsfiles could cause problems on our master. 
How can we force the flyweight jobs to run on a designated machine *other* than master? 

How do other people handle this?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="OSztiEjlAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/a41c28a9-fc93-459f-b990-87cb1210cb64%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/a41c28a9-fc93-459f-b990-87cb1210cb64%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/a41c28a9-fc93-459f-b990-87cb1210cb64%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/a41c28a9-fc93-459f-b990-87cb1210cb64%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1fdcf5dd-5f64-4ef4-83a0-d503fb70c371%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Run pipeline flyweight executors off of master

Sverre Moe-2
I don't think you are supposed to do any real work in a flyweight executor.
Steps need to be within a node{} which will allocate a heavyweight executor. I use the flyweight executors only to trigger downstream builds.

tirsdag 14. juni 2016 02.47.40 UTC+2 skrev Eli White følgende:
By not running anything on master we don't have to worry about any sort of failure related to misconfiguration of jobs, or job related failures. For example, OOM, out of disk space, pausing when 'input' is run, etc. 

If a slave goes down you have 1 machine down. If master goes down, all CI for our company goes down. The need for us to protect and make behavior guarantees of the master node, we want it to do only what is absolutely necessary and keep user code from running on it. 

On Monday, June 13, 2016 at 2:19:56 PM UTC-7, Thomas Zoratto wrote:
Hello, 

I’m not sure one can do what you want.

Out of curiosity, what’s the use case for this ?


Le 13 juin 2016 à 23:16, Eli White <[hidden email]> a écrit :

My understanding is that Jenkinsfile execution runs as a flyweight node on master, but then uses heavyweight nodes on the given node label to execute.

Per this file: <a href="https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Fpipeline-plugin%2Fblob%2Fmaster%2FTUTORIAL.md\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmRd5uoaEGwemnrAEu1_3Yf_1bbQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Fpipeline-plugin%2Fblob%2Fmaster%2FTUTORIAL.md\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmRd5uoaEGwemnrAEu1_3Yf_1bbQ&#39;;return true;">https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md

Why are there two executors consumed by one Pipeline build?

  • Every Pipeline build itself runs on the master, using a flyweight executor — an uncounted slot that is assumed to not take any significant computational power.
  • This executor represents the actual Groovy script, which is almost always idle, waiting for a step to complete.
  • Flyweight executors are always available.

On Monday, June 13, 2016 at 2:11:19 PM UTC-7, Craig Rodrigues wrote:
In your pipeline, the node parameter can take an argument of which node to run on.

See this example:
<a href="https://github.com/freebsd/freebsd-ci/blob/master/scripts/build/build-test.groovy#L100" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Ffreebsd%2Ffreebsd-ci%2Fblob%2Fmaster%2Fscripts%2Fbuild%2Fbuild-test.groovy%23L100\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHvncLQwvcdrPDkJa0qDGRvGlfEKA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Ffreebsd%2Ffreebsd-ci%2Fblob%2Fmaster%2Fscripts%2Fbuild%2Fbuild-test.groovy%23L100\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHvncLQwvcdrPDkJa0qDGRvGlfEKA&#39;;return true;">https://github.com/freebsd/freebsd-ci/blob/master/scripts/build/build-test.groovy#L100

In my job, I defined BUILD_NODE with the NodeLabel Parameter Plugin ( <a href="https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwiki.jenkins-ci.org%2Fdisplay%2FJENKINS%2FNodeLabel%2BParameter%2BPlugin\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF2fSi7uK_SN1cTtnx2k4G4WuKuyw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwiki.jenkins-ci.org%2Fdisplay%2FJENKINS%2FNodeLabel%2BParameter%2BPlugin\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF2fSi7uK_SN1cTtnx2k4G4WuKuyw&#39;;return true;">https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin ).

--
Craig



On Mon, Jun 13, 2016 at 2:03 PM, Eli White <[hidden email]> wrote:
We follow the Jenkins configuration best practices and have no executors on our master node and force everything to run on our agents. 

We are starting to work with pipeline jobs and are worried that bad Jenkinsfiles could cause problems on our master. 
How can we force the flyweight jobs to run on a designated machine *other* than master? 

How do other people handle this?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/a41c28a9-fc93-459f-b990-87cb1210cb64%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/a41c28a9-fc93-459f-b990-87cb1210cb64%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/a41c28a9-fc93-459f-b990-87cb1210cb64%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/a41c28a9-fc93-459f-b990-87cb1210cb64%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/37051bc7-a153-4317-85c4-f2bb9ef90ad7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Run pipeline flyweight executors off of master

Eli White
Agreed, you are not supposed to do any real work in a flyweight node. 

However, in an organization where teams' repos contain Jenkinsfiles that they wrote, we can't guarantee the safety and correctness of those files and their following of best practices. 

We want to protect the master node against whatever user error we feasibly can. Messing up jenkins files seems like a really easy thing to do. 

It sounds like this ability might not exist. I'll open a JIRA for it.

On Monday, June 13, 2016 at 11:29:40 PM UTC-7, Sverre Moe wrote:
I don't think you are supposed to do any real work in a flyweight executor.
Steps need to be within a node{} which will allocate a heavyweight executor. I use the flyweight executors only to trigger downstream builds.

tirsdag 14. juni 2016 02.47.40 UTC+2 skrev Eli White følgende:
By not running anything on master we don't have to worry about any sort of failure related to misconfiguration of jobs, or job related failures. For example, OOM, out of disk space, pausing when 'input' is run, etc. 

If a slave goes down you have 1 machine down. If master goes down, all CI for our company goes down. The need for us to protect and make behavior guarantees of the master node, we want it to do only what is absolutely necessary and keep user code from running on it. 

On Monday, June 13, 2016 at 2:19:56 PM UTC-7, Thomas Zoratto wrote:
Hello, 

I’m not sure one can do what you want.

Out of curiosity, what’s the use case for this ?


Le 13 juin 2016 à 23:16, Eli White <[hidden email]> a écrit :

My understanding is that Jenkinsfile execution runs as a flyweight node on master, but then uses heavyweight nodes on the given node label to execute.

Per this file: <a href="https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Fpipeline-plugin%2Fblob%2Fmaster%2FTUTORIAL.md\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmRd5uoaEGwemnrAEu1_3Yf_1bbQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Fpipeline-plugin%2Fblob%2Fmaster%2FTUTORIAL.md\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmRd5uoaEGwemnrAEu1_3Yf_1bbQ&#39;;return true;">https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md

Why are there two executors consumed by one Pipeline build?

  • Every Pipeline build itself runs on the master, using a flyweight executor — an uncounted slot that is assumed to not take any significant computational power.
  • This executor represents the actual Groovy script, which is almost always idle, waiting for a step to complete.
  • Flyweight executors are always available.

On Monday, June 13, 2016 at 2:11:19 PM UTC-7, Craig Rodrigues wrote:
In your pipeline, the node parameter can take an argument of which node to run on.

See this example:
<a href="https://github.com/freebsd/freebsd-ci/blob/master/scripts/build/build-test.groovy#L100" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Ffreebsd%2Ffreebsd-ci%2Fblob%2Fmaster%2Fscripts%2Fbuild%2Fbuild-test.groovy%23L100\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHvncLQwvcdrPDkJa0qDGRvGlfEKA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Ffreebsd%2Ffreebsd-ci%2Fblob%2Fmaster%2Fscripts%2Fbuild%2Fbuild-test.groovy%23L100\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHvncLQwvcdrPDkJa0qDGRvGlfEKA&#39;;return true;">https://github.com/freebsd/freebsd-ci/blob/master/scripts/build/build-test.groovy#L100

In my job, I defined BUILD_NODE with the NodeLabel Parameter Plugin ( <a href="https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwiki.jenkins-ci.org%2Fdisplay%2FJENKINS%2FNodeLabel%2BParameter%2BPlugin\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF2fSi7uK_SN1cTtnx2k4G4WuKuyw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwiki.jenkins-ci.org%2Fdisplay%2FJENKINS%2FNodeLabel%2BParameter%2BPlugin\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF2fSi7uK_SN1cTtnx2k4G4WuKuyw&#39;;return true;">https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin ).

--
Craig



On Mon, Jun 13, 2016 at 2:03 PM, Eli White <[hidden email]> wrote:
We follow the Jenkins configuration best practices and have no executors on our master node and force everything to run on our agents. 

We are starting to work with pipeline jobs and are worried that bad Jenkinsfiles could cause problems on our master. 
How can we force the flyweight jobs to run on a designated machine *other* than master? 

How do other people handle this?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/a41c28a9-fc93-459f-b990-87cb1210cb64%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/a41c28a9-fc93-459f-b990-87cb1210cb64%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/a41c28a9-fc93-459f-b990-87cb1210cb64%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/a41c28a9-fc93-459f-b990-87cb1210cb64%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/cf57ee6d-6228-4ea0-8ee5-d8c909659ed6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Run pipeline flyweight executors off of master

Sverre Moe-2
Jenkins already does some of that for you by running the pipeline script in a sandbox. Certain methods/functions in the Jenkinsfile must be approved by a Jenkins administrator.

tirsdag 14. juni 2016 08.43.00 UTC+2 skrev Eli White følgende:
Agreed, you are not supposed to do any real work in a flyweight node. 

However, in an organization where teams' repos contain Jenkinsfiles that they wrote, we can't guarantee the safety and correctness of those files and their following of best practices. 

We want to protect the master node against whatever user error we feasibly can. Messing up jenkins files seems like a really easy thing to do. 

It sounds like this ability might not exist. I'll open a JIRA for it.

On Monday, June 13, 2016 at 11:29:40 PM UTC-7, Sverre Moe wrote:
I don't think you are supposed to do any real work in a flyweight executor.
Steps need to be within a node{} which will allocate a heavyweight executor. I use the flyweight executors only to trigger downstream builds.

tirsdag 14. juni 2016 02.47.40 UTC+2 skrev Eli White følgende:
By not running anything on master we don't have to worry about any sort of failure related to misconfiguration of jobs, or job related failures. For example, OOM, out of disk space, pausing when 'input' is run, etc. 

If a slave goes down you have 1 machine down. If master goes down, all CI for our company goes down. The need for us to protect and make behavior guarantees of the master node, we want it to do only what is absolutely necessary and keep user code from running on it. 

On Monday, June 13, 2016 at 2:19:56 PM UTC-7, Thomas Zoratto wrote:
Hello, 

I’m not sure one can do what you want.

Out of curiosity, what’s the use case for this ?


Le 13 juin 2016 à 23:16, Eli White <[hidden email]> a écrit :

My understanding is that Jenkinsfile execution runs as a flyweight node on master, but then uses heavyweight nodes on the given node label to execute.

Per this file: <a href="https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Fpipeline-plugin%2Fblob%2Fmaster%2FTUTORIAL.md\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmRd5uoaEGwemnrAEu1_3Yf_1bbQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Fpipeline-plugin%2Fblob%2Fmaster%2FTUTORIAL.md\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFmRd5uoaEGwemnrAEu1_3Yf_1bbQ&#39;;return true;">https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md

Why are there two executors consumed by one Pipeline build?

  • Every Pipeline build itself runs on the master, using a flyweight executor — an uncounted slot that is assumed to not take any significant computational power.
  • This executor represents the actual Groovy script, which is almost always idle, waiting for a step to complete.
  • Flyweight executors are always available.

On Monday, June 13, 2016 at 2:11:19 PM UTC-7, Craig Rodrigues wrote:
In your pipeline, the node parameter can take an argument of which node to run on.

See this example:
<a href="https://github.com/freebsd/freebsd-ci/blob/master/scripts/build/build-test.groovy#L100" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Ffreebsd%2Ffreebsd-ci%2Fblob%2Fmaster%2Fscripts%2Fbuild%2Fbuild-test.groovy%23L100\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHvncLQwvcdrPDkJa0qDGRvGlfEKA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Ffreebsd%2Ffreebsd-ci%2Fblob%2Fmaster%2Fscripts%2Fbuild%2Fbuild-test.groovy%23L100\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHvncLQwvcdrPDkJa0qDGRvGlfEKA&#39;;return true;">https://github.com/freebsd/freebsd-ci/blob/master/scripts/build/build-test.groovy#L100

In my job, I defined BUILD_NODE with the NodeLabel Parameter Plugin ( <a href="https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwiki.jenkins-ci.org%2Fdisplay%2FJENKINS%2FNodeLabel%2BParameter%2BPlugin\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF2fSi7uK_SN1cTtnx2k4G4WuKuyw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwiki.jenkins-ci.org%2Fdisplay%2FJENKINS%2FNodeLabel%2BParameter%2BPlugin\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF2fSi7uK_SN1cTtnx2k4G4WuKuyw&#39;;return true;">https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin ).

--
Craig



On Mon, Jun 13, 2016 at 2:03 PM, Eli White <[hidden email]> wrote:
We follow the Jenkins configuration best practices and have no executors on our master node and force everything to run on our agents. 

We are starting to work with pipeline jobs and are worried that bad Jenkinsfiles could cause problems on our master. 
How can we force the flyweight jobs to run on a designated machine *other* than master? 

How do other people handle this?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/d4ee67f4-ab75-4a1e-9883-69453bdd656b%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/a41c28a9-fc93-459f-b990-87cb1210cb64%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/a41c28a9-fc93-459f-b990-87cb1210cb64%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/a41c28a9-fc93-459f-b990-87cb1210cb64%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/a41c28a9-fc93-459f-b990-87cb1210cb64%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d4d3e045-e7dd-4b60-9baa-bcdbbf8ea529%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Run pipeline flyweight executors off of master

Kevin Phillips
In reply to this post by Eli White
In my opinion, even more problematic than possibly running unsafe code in a flyweight executor on the master is the fact that these flyweight pipeline threads perform a full checkout of the target repository - at least when using Git as the SCM. So if one has a relatively large code repository containing product code - say several gigabytes - and they want to store their Jenkinsfile with the code so they can leverage the MultiBranch options in Jenkins Pipeline, they end up with many copies of their entire codebase being checked out on the master, possibly causing the master to run out of disk space. Add to this the fact that these "temporary" workspaces created on the master by the flyweight tasks are not automatically deleted when not in use and there is no way to purge them from the Jenkins dashboard and you are just asking for problems. This isn't a case of whether this will become a problem as much as it is an issue of when it will become a problem.

In my opinion, Jenkinsfile's stored in SCM should be checked out on an agent as a typical freestyle job would do, and a default "node" allocated by the executor that launches the script contained therein for the node on which it is already running on. Then the rest of the Jenkinsfile could be processed in place without requiring any sort of flyweight executor at all, and avoiding the issues with having multiple checkouts of the same code spread across a build farm. Also, I think this would be inline with how some other more modern continuous integration servers work like travis-ci, so it's not without precidence.

On Monday, June 13, 2016 at 6:03:20 PM UTC-3, Eli White wrote:
We follow the Jenkins configuration best practices and have no executors on our master node and force everything to run on our agents. 

We are starting to work with pipeline jobs and are worried that bad Jenkinsfiles could cause problems on our master. 
How can we force the flyweight jobs to run on a designated machine *other* than master? 

How do other people handle this?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/2e7225fd-1017-4e66-ab9e-8802ed24c1a3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Run pipeline flyweight executors off of master

Joshua Noble
i would have to agree, this is definitely an issue. I'm currently using Amazon EFS as the storage backend for my Jenkins master, who doesn't have any executors. The flyweight executors checking out repositories is definitely a concern of mine. EFS will automatically scale so storage isn't a huge issue for me, but duplication and clone times are. (for larger repos)

On Wednesday, August 10, 2016 at 2:31:45 PM UTC-4, Kevin Phillips wrote:
In my opinion, even more problematic than possibly running unsafe code in a flyweight executor on the master is the fact that these flyweight pipeline threads perform a full checkout of the target repository - at least when using Git as the SCM. So if one has a relatively large code repository containing product code - say several gigabytes - and they want to store their Jenkinsfile with the code so they can leverage the MultiBranch options in Jenkins Pipeline, they end up with many copies of their entire codebase being checked out on the master, possibly causing the master to run out of disk space. Add to this the fact that these "temporary" workspaces created on the master by the flyweight tasks are not automatically deleted when not in use and there is no way to purge them from the Jenkins dashboard and you are just asking for problems. This isn't a case of whether this will become a problem as much as it is an issue of when it will become a problem.

In my opinion, Jenkinsfile's stored in SCM should be checked out on an agent as a typical freestyle job would do, and a default "node" allocated by the executor that launches the script contained therein for the node on which it is already running on. Then the rest of the Jenkinsfile could be processed in place without requiring any sort of flyweight executor at all, and avoiding the issues with having multiple checkouts of the same code spread across a build farm. Also, I think this would be inline with how some other more modern continuous integration servers work like travis-ci, so it's not without precidence.

On Monday, June 13, 2016 at 6:03:20 PM UTC-3, Eli White wrote:
We follow the Jenkins configuration best practices and have no executors on our master node and force everything to run on our agents. 

We are starting to work with pipeline jobs and are worried that bad Jenkinsfiles could cause problems on our master. 
How can we force the flyweight jobs to run on a designated machine *other* than master? 

How do other people handle this?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d6b297d9-0a17-42ea-a608-45f324b33caf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.