Run same Jenkins job on different repositories in parallel using Declarative Pipeline

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Run same Jenkins job on different repositories in parallel using Declarative Pipeline

Roman O
My goal is to run the same job check-single-repo on multiple repositories in parallel.

Below pipeline doesn't seem to achieve the goal:

    pipeline {
        agent any
        options
{
            ansiColor
('xterm')                    
       
}
        parameters
{
            extendedChoice description
: '', multiSelectDelimiter: ',', name: 'REPO_NAMES', quoteValue: false, saveJSONParameterToFile: false, type: 'PT_CHECKBOX', value: 'repo1,repo2,repo3', visibleItemCount: 3
       
}
   
        stages
{
            stage
('Prepare repos to run on') {
               
                steps
{
                    script
{  
                        repo_names
= params.REPO_NAMES.split(',')
                       
def single_repo_jobs = [:]
                       
for (repo_name in repo_names) {
                            println repo_name
                            single_repo_jobs
[repo_name] = {  
                                    stage
(repo_name) {    
                                        catchError
{
                                            build job
: 'check-single-repo',
                                                    parameters
:
                                                           
[                                                    
                                                                   
string(name:'REPO_NAME', value: repo_name)
                                                           
] }
                                       
}
                                   
}
                       
}
                        println single_repo_jobs
                        parallel single_repo_jobs                        
                   
}                
               
}
                           
           
}
       
}
   
}


Its output:

    ...
    repo1
    repo2
    repo3
   
{repo1=org.jenkinsci.plugins.workflow.cps.CpsClosure2@3a396959, repo2=org.jenkinsci.plugins.workflow.cps.CpsClosure2@1a4b5000, repo3=org.jenkinsci.plugins.workflow.cps.CpsClosure2@1d034ac}
   
[Pipeline] parallel
   
[Pipeline] { (Branch: repo1)
   
[Pipeline] { (Branch: repo2)
   
[Pipeline] { (Branch: repo3)
   
[Pipeline] stage
   
[Pipeline] { (repo3)
   
[Pipeline] stage
   
[Pipeline] { (repo3)
   
[Pipeline] stage
   
[Pipeline] { (repo3)
   
[Pipeline] catchError
   
[Pipeline] {
   
[Pipeline] catchError
   
[Pipeline] {
   
[Pipeline] catchError
   
[Pipeline] {
   
[Pipeline] build (Building check-single-repo)
   
Scheduling project: check-single-repo
   
[Pipeline] build (Building check-single-repo)
   
Scheduling project: check-single-repo
   
[Pipeline] build (Building check-single-repo)
   
Scheduling project: check-single-repo
   
   
Starting building: check-single-repo #230
   
Starting building: check-single-repo #230
   
Starting building: check-single-repo #230
   
   
[Pipeline] }
    check
-single-repo #230 repo3 completed with status UNSTABLE (propagate: false to ignore)
   
[Pipeline] }
    check
-single-repo #230 repo3 completed with status UNSTABLE (propagate: false to ignore)
   
[Pipeline] }
    check
-single-repo #230 repo3 completed with status UNSTABLE (propagate: false to ignore)
   
...


As it invokes only one pipeline #230 on repo3


How to run the same Jenkins job on different repositories in parallel using Declarative Pipeline?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/3aff4bde-6bbc-4c12-bc76-52320a01fecao%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Run same Jenkins job on different repositories in parallel using Declarative Pipeline

Roman O
The following code worked for me after following the example from docs:

pipeline { agent any options { ansiColor('xterm') } parameters { extendedChoice description: '', multiSelectDelimiter: ',', name: 'REPO_NAMES', quoteValue: false, saveJSONParameterToFile: false, type: 'PT_CHECKBOX', value: 'repo1,repo2,repo3', visibleItemCount: 3 } stages { stage('Prepare repos to run on') { steps { script { def repo_names = params.REPO_NAMES.split(',') def single_repo_jobs = repo_names.collectEntries { ["${it}" : { catchError { stage(it) { build job: 'check-single-repo', parameters: [ string(name:'REPO_NAME', value: "${it}") ] } } } ] } parallel single_repo_jobs } }


On Thursday, June 11, 2020 at 4:49:15 PM UTC+3, Roman O wrote:
My goal is to run the same job check-single-repo on multiple repositories in parallel.

Below pipeline doesn't seem to achieve the goal:

    pipeline {
        agent any
        options
{
            ansiColor
('xterm')                    
       
}
        parameters
{
            extendedChoice description
: '', multiSelectDelimiter: ',', name: 'REPO_NAMES', quoteValue: false, saveJSONParameterToFile: false, type: 'PT_CHECKBOX', value: 'repo1,repo2,repo3', visibleItemCount: 3
       
}
   
        stages
{
            stage
('Prepare repos to run on') {
               
                steps
{
                    script
{  
                        repo_names
= params.REPO_NAMES.split(',')
                       
def single_repo_jobs = [:]
                       
for (repo_name in repo_names) {
                            println repo_name
                            single_repo_jobs
[repo_name] = {  
                                    stage
(repo_name) {    
                                        catchError
{
                                            build job
: 'check-single-repo',
                                                    parameters
:
                                                           
[                                                    
                                                                   
string(name:'REPO_NAME', value: repo_name)
                                                           
] }
                                       
}
                                   
}
                       
}
                        println single_repo_jobs
                        parallel single_repo_jobs                        
                   
}                
               
}
                           
           
}
       
}
   
}


Its output:

    ...
    repo1
    repo2
    repo3
   
{repo1=org.jenkinsci.plugins.workflow.cps.CpsClosure2@3a396959, repo2=org.jenkinsci.plugins.workflow.cps.CpsClosure2@1a4b5000, repo3=org.jenkinsci.plugins.workflow.cps.CpsClosure2@1d034ac}
   
[Pipeline] parallel
   
[Pipeline] { (Branch: repo1)
   
[Pipeline] { (Branch: repo2)
   
[Pipeline] { (Branch: repo3)
   
[Pipeline] stage
   
[Pipeline] { (repo3)
   
[Pipeline] stage
   
[Pipeline] { (repo3)
   
[Pipeline] stage
   
[Pipeline] { (repo3)
   
[Pipeline] catchError
   
[Pipeline] {
   
[Pipeline] catchError
   
[Pipeline] {
   
[Pipeline] catchError
   
[Pipeline] {
   
[Pipeline] build (Building check-single-repo)
   
Scheduling project: check-single-repo
   
[Pipeline] build (Building check-single-repo)
   
Scheduling project: check-single-repo
   
[Pipeline] build (Building check-single-repo)
   
Scheduling project: check-single-repo
   
   
Starting building: check-single-repo #230
   
Starting building: check-single-repo #230
   
Starting building: check-single-repo #230
   
   
[Pipeline] }
    check
-single-repo #230 repo3 completed with status UNSTABLE (propagate: false to ignore)
   
[Pipeline] }
    check
-single-repo #230 repo3 completed with status UNSTABLE (propagate: false to ignore)
   
[Pipeline] }
    check
-single-repo #230 repo3 completed with status UNSTABLE (propagate: false to ignore)
   
...


As it invokes only one pipeline #230 on repo3


How to run the same Jenkins job on different repositories in parallel using Declarative Pipeline?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/507c0997-e718-4f7a-ab5d-05b598f3e736o%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Run same Jenkins job on different repositories in parallel using Declarative Pipeline

Roman O
In reply to this post by Roman O
The following code worked for me after following the example from docs:

    pipeline {
        agent any
        options
{
            ansiColor
('xterm')                    
       
}
        parameters
{
            extendedChoice description
: '', multiSelectDelimiter: ',', name: 'REPO_NAMES', quoteValue: false, saveJSONParameterToFile: false, type: 'PT_CHECKBOX', value: 'repo1,repo2,repo3', visibleItemCount: 3
       
}
   
        stages
{
            stage
('Prepare repos to run on') {
               
                steps
{
                    script
{  
                       
def repo_names = params.REPO_NAMES.split(',')
                       
def single_repo_jobs = repo_names.collectEntries {
                           
["${it}" :
                               
{  
                                    catchError
{                                            
                                            stage
(it) {  
                                                build job
: 'check-single-repo',
                                                        parameters
:
                                                               
[                                                    
                                                                       
string(name:'REPO_NAME', value: "${it}")
                                                               
]
                                           
}                                            
                                   
}
                               
}                        
                           
]
                       
}                    
                        parallel single_repo_jobs        
                   
}                
               
}
                           
           
}
       
}
   
}




On Thursday, June 11, 2020 at 4:49:15 PM UTC+3, Roman O wrote:
My goal is to run the same job check-single-repo on multiple repositories in parallel.

Below pipeline doesn't seem to achieve the goal:

    pipeline {
        agent any
        options
{
            ansiColor
('xterm')                    
       
}
        parameters
{
            extendedChoice description
: '', multiSelectDelimiter: ',', name: 'REPO_NAMES', quoteValue: false, saveJSONParameterToFile: false, type: 'PT_CHECKBOX', value: 'repo1,repo2,repo3', visibleItemCount: 3
       
}
   
        stages
{
            stage
('Prepare repos to run on') {
               
                steps
{
                    script
{  
                        repo_names
= params.REPO_NAMES.split(',')
                       
def single_repo_jobs = [:]
                       
for (repo_name in repo_names) {
                            println repo_name
                            single_repo_jobs
[repo_name] = {  
                                    stage
(repo_name) {    
                                        catchError
{
                                            build job
: 'check-single-repo',
                                                    parameters
:
                                                           
[                                                    
                                                                   
string(name:'REPO_NAME', value: repo_name)
                                                           
] }
                                       
}
                                   
}
                       
}
                        println single_repo_jobs
                        parallel single_repo_jobs                        
                   
}                
               
}
                           
           
}
       
}
   
}


Its output:

    ...
    repo1
    repo2
    repo3
   
{repo1=org.jenkinsci.plugins.workflow.cps.CpsClosure2@3a396959, repo2=org.jenkinsci.plugins.workflow.cps.CpsClosure2@1a4b5000, repo3=org.jenkinsci.plugins.workflow.cps.CpsClosure2@1d034ac}
   
[Pipeline] parallel
   
[Pipeline] { (Branch: repo1)
   
[Pipeline] { (Branch: repo2)
   
[Pipeline] { (Branch: repo3)
   
[Pipeline] stage
   
[Pipeline] { (repo3)
   
[Pipeline] stage
   
[Pipeline] { (repo3)
   
[Pipeline] stage
   
[Pipeline] { (repo3)
   
[Pipeline] catchError
   
[Pipeline] {
   
[Pipeline] catchError
   
[Pipeline] {
   
[Pipeline] catchError
   
[Pipeline] {
   
[Pipeline] build (Building check-single-repo)
   
Scheduling project: check-single-repo
   
[Pipeline] build (Building check-single-repo)
   
Scheduling project: check-single-repo
   
[Pipeline] build (Building check-single-repo)
   
Scheduling project: check-single-repo
   
   
Starting building: check-single-repo #230
   
Starting building: check-single-repo #230
   
Starting building: check-single-repo #230
   
   
[Pipeline] }
    check
-single-repo #230 repo3 completed with status UNSTABLE (propagate: false to ignore)
   
[Pipeline] }
    check
-single-repo #230 repo3 completed with status UNSTABLE (propagate: false to ignore)
   
[Pipeline] }
    check
-single-repo #230 repo3 completed with status UNSTABLE (propagate: false to ignore)
   
...


As it invokes only one pipeline #230 on repo3


How to run the same Jenkins job on different repositories in parallel using Declarative Pipeline?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d094696f-9984-4e0e-9189-66c06aee8dado%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Run same Jenkins job on different repositories in parallel using Declarative Pipeline

Devin Nusbaum
The problem in your original code is that the stages you create (more specifically the closures for the stages) reference the mutable repo_name variable directly, and so after you create all the stages in the loop, repo_name’s value is the final repo in repo_names, so when the stages are executed they all use the same value of repo_name. 

You need to modify the script so that the stages do not refer to the loop variable directly, usually by creating a copy of the loop variable, but approaches like using collectEntries or passing the loop variable as an argument to a closure that creates another closure would also fix the issue.




On Jun 11, 2020, at 10:00, Roman O <[hidden email]> wrote:

The following code worked for me after following the example from docs:

    pipeline {
        agent any 
        options 
{
            ansiColor
('xterm')                    
        
}
        parameters 
{
            extendedChoice description
: '', multiSelectDelimiter: ',', name: 'REPO_NAMES', quoteValue: false,saveJSONParameterToFile: false, type: 'PT_CHECKBOX', value: 'repo1,repo2,repo3', visibleItemCount: 3
        
}
    
        stages 
{
            stage
('Prepare repos to run on') {
                
                steps 
{ 
                    script 
{  
                        
def repo_names = params.REPO_NAMES.split(',')
                        
def single_repo_jobs = repo_names.collectEntries {
                            
["${it}" : 
                                
{  
                                    catchError 
{                                            
                                            stage
(it) {   
                                                build job
: 'check-single-repo',
                                                        parameters
:
                                                                
[                                                    
                                                                        
string(name:'REPO_NAME', value: "${it}")
                                                                
] 
                                            
}                                            
                                    
}
                                
}                        
                            
]
                        
}                    
                        parallel single_repo_jobs        
                    
}                
                
}
                           
            
}
        
}
    
}




On Thursday, June 11, 2020 at 4:49:15 PM UTC+3, Roman O wrote:
My goal is to run the same job check-single-repo on multiple repositories in parallel.

Below pipeline doesn't seem to achieve the goal:

    pipeline {
        agent any 
        options 
{
            ansiColor
('xterm')                    
        
}
        parameters 
{
            extendedChoice description
: '', multiSelectDelimiter: ',', name: 'REPO_NAMES', quoteValue: false,saveJSONParameterToFile: false, type: 'PT_CHECKBOX', value: 'repo1,repo2,repo3', visibleItemCount: 3
        
}
    
        stages 
{
            stage
('Prepare repos to run on') {
                
                steps 
{ 
                    script 
{  
                        repo_names 
= params.REPO_NAMES.split(',')
                        
def single_repo_jobs = [:]
                        
for (repo_name in repo_names) {
                            println repo_name
                            single_repo_jobs
[repo_name] = {  
                                    stage
(repo_name) {    
                                        catchError 
{
                                            build job
: 'check-single-repo',
                                                    parameters
:
                                                            
[                                                    
                                                                    
string(name:'REPO_NAME', value: repo_name)
                                                            
] }
                                        
}
                                    
}
                        
}
                        println single_repo_jobs
                        parallel single_repo_jobs                        
                    
}                
                
}
                           
            
}
        
}
    
}


Its output:

    ...
    repo1
    repo2
    repo3
    
{repo1=org.jenkinsci.plugins.workflow.cps.CpsClosure2@3a396959, repo2=org.jenkinsci.plugins.workflow.cps.CpsClosure2@1a4b5000, repo3=org.jenkinsci.plugins.workflow.cps.CpsClosure2@1d034ac}
    
[Pipeline] parallel
    
[Pipeline] { (Branch: repo1)
    
[Pipeline] { (Branch: repo2)
    
[Pipeline] { (Branch: repo3)
    
[Pipeline] stage
    
[Pipeline] { (repo3)
    
[Pipeline] stage
    
[Pipeline] { (repo3)
    
[Pipeline] stage
    
[Pipeline] { (repo3)
    
[Pipeline] catchError
    
[Pipeline] {
    
[Pipeline] catchError
    
[Pipeline] {
    
[Pipeline] catchError
    
[Pipeline] {
    
[Pipeline] build (Building check-single-repo)
    
Scheduling project: check-single-repo
    
[Pipeline] build (Building check-single-repo)
    
Scheduling project: check-single-repo
    
[Pipeline] build (Building check-single-repo)
    
Scheduling project: check-single-repo
    
    
Starting building: check-single-repo #230
    
Starting building: check-single-repo #230
    
Starting building: check-single-repo #230
    
    
[Pipeline] }
    check
-single-repo #230 repo3 completed with status UNSTABLE (propagate: false to ignore)
    
[Pipeline] }
    check
-single-repo #230 repo3 completed with status UNSTABLE (propagate: false to ignore)
    
[Pipeline] }
    check
-single-repo #230 repo3 completed with status UNSTABLE (propagate: false to ignore)
    
...


As it invokes only one pipeline #230 on repo3


How to run the same Jenkins job on different repositories in parallel using Declarative Pipeline?

-- 
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d094696f-9984-4e0e-9189-66c06aee8dado%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/6232FF71-658F-4FB1-8EEC-7C3BE3D3CEE1%40cloudbees.com.
Reply | Threaded
Open this post in threaded view
|

Re: Run same Jenkins job on different repositories in parallel using Declarative Pipeline

David Riemens
In reply to this post by Roman O
Hi,
not sure if your problem is now solved, so ..
Try changing your code to use:

w pipeline doesn't seem to achieve the goal:

    pipeline {
        agent any
        options {
            ansiColor('xterm')                    
        }
        parameters {
            extendedChoice description: '', multiSelectDelimiter: ',', name: 'REPO_NAMES', quoteValue: false, saveJSONParameterToFile: false, type: 'PT_CHECKBOX', value: 'repo1,repo2,repo3', visibleItemCount: 3
        }
   
        stages {
            stage('Prepare repos to run on') {
               
                steps {
                    script {  
                        repo_names = params.REPO_NAMES.split(',')
                        def single_repo_jobs = [:]
                        for (repo_name in repo_names) {
                            def theRepoName = "$repo_name"                    // This assignment is important
                            println theRepoName  
                            single_repo_jobs[ theRepoName ] = {  
                                    stage( theRepoName ) {    
                                        catchError {
                                            build job: 'check-single-repo',
                                                    parameters:
                                                            [                                                    
                                                                    string(name:'REPO_NAME', value:  theRepoName )
                                                            ] }
                                        }
                                    }
                        }
                        println single_repo_jobs
                        parallel single_repo_jobs                        
                    }                
                }
                           
            }
        }
    }

I keep forgetting the exact reason for this extra assignment needed but do know it's needed. I'm sure it's
properly explained and documented, but remains non-intuitive.
gr M


On Thursday, June 11, 2020, 04:54:51 PM GMT+2, Roman O <[hidden email]> wrote:


The following code worked for me after following the example from docs:

pipeline { agent any options { ansiColor('xterm') } parameters { extendedChoice description: '', multiSelectDelimiter: ',', name: 'REPO_NAMES', quoteValue: false, saveJSONParameterToFile: false, type: 'PT_CHECKBOX', value: 'repo1,repo2,repo3', visibleItemCount: 3 } stages { stage('Prepare repos to run on') { steps { script { def repo_names = params.REPO_NAMES.split(',') def single_repo_jobs = repo_names.collectEntries { ["${it}" : { catchError { stage(it) { build job: 'check-single-repo', parameters: [ string(name:'REPO_NAME', value: "${it}") ] } } } ] } parallel single_repo_jobs } }


On Thursday, June 11, 2020 at 4:49:15 PM UTC+3, Roman O wrote:
My goal is to run the same job check-single-repo on multiple repositories in parallel.

Below pipeline doesn't seem to achieve the goal:

    pipeline {
        agent any
        options
{
            ansiColor
('xterm')                    
       
}
        parameters
{
            extendedChoice description
: '', multiSelectDelimiter: ',', name: 'REPO_NAMES', quoteValue: false, saveJSONParameterToFile: false, type: 'PT_CHECKBOX', value: 'repo1,repo2,repo3', visibleItemCount: 3
       
}
   
        stages
{
            stage
('Prepare repos to run on') {
               
                steps
{
                    script
{  
                        repo_names
= params.REPO_NAMES.split(',')
                       
def single_repo_jobs = [:]
                       
for (repo_name in repo_names) {
                            println repo_name
                            single_repo_jobs
[repo_name] = {  
                                    stage
(repo_name) {    
                                        catchError
{
                                            build job
: 'check-single-repo',
                                                    parameters
:
                                                           
[                                                    
                                                                   
string(name:'REPO_NAME', value: repo_name)
                                                           
] }
                                       
}
                                   
}
                       
}
                        println single_repo_jobs
                        parallel single_repo_jobs                        
                   
}                
               
}
                           
           
}
       
}
   
}


Its output:

    ...
    repo1
    repo2
    repo3
   
{repo1=org.jenkinsci.plugins.w orkflow.cps.CpsClosure2@ 3a396959, repo2=org.jenkinsci.plugins.wo rkflow.cps.CpsClosure2@ 1a4b5000, repo3=org.jenkinsci.plugins.wo rkflow.cps.CpsClosure2@1d034ac }
   
[Pipeline] parallel
   
[Pipeline] { (Branch: repo1)
   
[Pipeline] { (Branch: repo2)
   
[Pipeline] { (Branch: repo3)
   
[Pipeline] stage
   
[Pipeline] { (repo3)
   
[Pipeline] stage
   
[Pipeline] { (repo3)
   
[Pipeline] stage
   
[Pipeline] { (repo3)
   
[Pipeline] catchError
   
[Pipeline] {
   
[Pipeline] catchError
   
[Pipeline] {
   
[Pipeline] catchError
   
[Pipeline] {
   
[Pipeline] build (Building check-single-repo)
   
Scheduling project: check-single-repo
   
[Pipeline] build (Building check-single-repo)
   
Scheduling project: check-single-repo
   
[Pipeline] build (Building check-single-repo)
   
Scheduling project: check-single-repo
   
   
Starting building: check-single-repo #230
   
Starting building: check-single-repo #230
   
Starting building: check-single-repo #230
   
   
[Pipeline] }
    check
-single-repo #230 repo3 completed with status UNSTABLE (propagate: false to ignore)
   
[Pipeline] }
    check
-single-repo #230 repo3 completed with status UNSTABLE (propagate: false to ignore)
   
[Pipeline] }
    check
-single-repo #230 repo3 completed with status UNSTABLE (propagate: false to ignore)
   
...


As it invokes only one pipeline #230 on repo3


How to run the same Jenkins job on different repositories in parallel using Declarative Pipeline?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/507c0997-e718-4f7a-ab5d-05b598f3e736o%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/831150088.126722.1592380519804%40mail.yahoo.com.