Quantcast

How to properly use parallels in pipelines

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

How to properly use parallels in pipelines

ctoverend
parallel (
    'windows': {
        stage('windows tests') {
            parallel (
                    
stage('some_tests_1') {
                        parallel (
                            some_tests_1, # is a chunk of tests
                        )
                     }
                    
stage('some_tests_2') {
                        parallel (
                            some_tests_2, 
 # is a chunk of tests
                        )
                    }
                    stage('some_tests_3') {
                        parallel (
                            some_tests_3 
 # is a chunk of tests
                        )
                    }
            )
    }
    'linux': ...same as above

Currently we call flows calling flows and the above displays how it would look in a pipeline.

from: https://github.com/jenkinsci/pipeline-examples/blob/master/docs/BEST_PRACTICES.md

Nested parallel blocks can lead to swamping your available executors, as each execution of the first parallel block calls multiple executions of the second parallel block, and so on. In general, think carefully about your parallelism and your available executors when using parallel.

With this in mind we could

parallel (
    'windows': {
        stage('windows tests') {
            parallel (

                some_tests_1, # is a chunk of tests
               
some_tests_2,  # is a chunk of tests
               
some_tests_3  # is a chunk of tests
            )
        }
    }
    'linux': ...same as above

Also from 
https://github.com/jenkinsci/pipeline-examples/blob/master/docs/BEST_PRACTICES.md

Don’t put stages in parallel blocks - that just goes weird, breaking a lot of logic in the Stage View and elsewhere. Save yourself the pain - don't do it!

So how do you display this work flow properly if we should not use stages in parallel?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d33e1ba8-5d60-4ecd-a509-5ac1234a97a5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to properly use parallels in pipelines

Nick Le Mouton
I've been playing around with this as well, there doesn't seem to be a lot of documentation around using parallel in declarative pipelines, but this seems to work for us:

    stage('Code analysis') {

      steps
{

        parallel
(

          lint
: {

           
//run lint

         
},

          phpmd
: {

           
//run phpmd

         
},

          phpcs
: {

           
//run phpcs

         
}

       
)

     
}

     

 


Or if you're using older scripted pipelines then it'd be something like:


    stage('Code analysis') {
      parallel
(
        phpmd
: {
         
//run phpmd
       
},
        phpcs
: {
         
//run phpcs
       
},
        phploc
: {
         
//run phploc
       
},
     
)
   
}

I hope this helps

Nick Le Mouton



On Thursday, March 16, 2017 at 4:34:22 AM UTC+13, Chris Overend wrote:
parallel (
    'windows': {
        stage('windows tests') {
            parallel (
                    
stage('some_tests_1') {
                        parallel (
                            some_tests_1, # is a chunk of tests
                        )
                     }
                    
stage('some_tests_2') {
                        parallel (
                            some_tests_2, 
 # is a chunk of tests
                        )
                    }
                    stage('some_tests_3') {
                        parallel (
                            some_tests_3 
 # is a chunk of tests
                        )
                    }
            )
    }
    'linux': ...same as above

Currently we call flows calling flows and the above displays how it would look in a pipeline.

from: <a href="https://github.com/jenkinsci/pipeline-examples/blob/master/docs/BEST_PRACTICES.md" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Fpipeline-examples%2Fblob%2Fmaster%2Fdocs%2FBEST_PRACTICES.md\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEeSU7lqUdUmS2e46lFHPP19ZOCAA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Fpipeline-examples%2Fblob%2Fmaster%2Fdocs%2FBEST_PRACTICES.md\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEeSU7lqUdUmS2e46lFHPP19ZOCAA&#39;;return true;">https://github.com/jenkinsci/pipeline-examples/blob/master/docs/BEST_PRACTICES.md

Nested parallel blocks can lead to swamping your available executors, as each execution of the first parallel block calls multiple executions of the second parallel block, and so on. In general, think carefully about your parallelism and your available executors when using parallel.

With this in mind we could

parallel (
    'windows': {
        stage('windows tests') {
            parallel (

                some_tests_1, # is a chunk of tests
               
some_tests_2,  # is a chunk of tests
               
some_tests_3  # is a chunk of tests
            )
        }
    }
    'linux': ...same as above

Also from 
<a href="https://github.com/jenkinsci/pipeline-examples/blob/master/docs/BEST_PRACTICES.md" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Fpipeline-examples%2Fblob%2Fmaster%2Fdocs%2FBEST_PRACTICES.md\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEeSU7lqUdUmS2e46lFHPP19ZOCAA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Fpipeline-examples%2Fblob%2Fmaster%2Fdocs%2FBEST_PRACTICES.md\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEeSU7lqUdUmS2e46lFHPP19ZOCAA&#39;;return true;">https://github.com/jenkinsci/pipeline-examples/blob/master/docs/BEST_PRACTICES.md

Don’t put stages in parallel blocks - that just goes weird, breaking a lot of logic in the Stage View and elsewhere. Save yourself the pain - don't do it!

So how do you display this work flow properly if we should not use stages in parallel?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/89ecb14f-fb3f-4414-97a0-73e6127ac56e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Loading...