trying to get Jenkins pipeline to run across multiple nodes in parallel

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

trying to get Jenkins pipeline to run across multiple nodes in parallel

Stephen DiMilla
I'm new to using pipelines.
I'm trying to execute the same commands against a number of nodes in parallel
Ideally I will pass a list of machines to run the same code against. Below I am 
hardcoding the machine names (machine1 and machine2) for the purposes of this example.

I found the following in Jenkins Pipeline examples:
----------------------------
def labels = ['precise', 'trusty'] // labels for Jenkins node types we will build on
def builders = [:]
for (x in labels) {
    def label = x // Need to bind the label variable before the closure - can't do 'for (label in labels)'

    // Create a map to pass in to the 'parallel' step so we can fire all the builds at once
    builders[label] = {
      node(label) {
        // build steps that should happen on all nodes go here
      }
    }
}

parallel builders
----------------------------
So I adapted it to be the following:
----------------------------
pipeline {
stages {
stage('Stage1') {
steps {
def labels = ['machine1','machine2'] // labels for Jenkins node types we will build on
def builders = [:]
for (x in labels) {
def label = x // Need to bind the label variable before the closure - can't do 'for (label in labels)'

// Create a map to pass in to the 'parallel' step so we can fire all the builds at once
builders[label] = {
node(label) {
sh '''
echo "hostname:`hostname`"
echo "whoami:`whoami`"

'''
}
}
}
parallel builders
}
}
}
}
----------------------------

But when I use it I get the following error:
Running in Durability level: MAX_SURVIVABILITY
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: 5: Expected a step @ line 5, column 17.
                   def labels = ['machine1','machine2'] // labels for Jenkins node types we will build on
                   ^

WorkflowScript: 6: Expected a step @ line 6, column 17.
                   def builders = [:]
                   ^

WorkflowScript: 7: Expected a step @ line 7, column 17.
                   for (x in labels) {
                   ^

General error during semantic analysis: There's no @DataBoundConstructor on any constructor of class org.jenkinsci.plugins.workflow.cps.steps.ParallelStep

org.kohsuke.stapler.NoStaplerConstructorException: There's no @DataBoundConstructor on any constructor of class org.jenkinsci.plugins.workflow.cps.steps.ParallelStep


----------------------------

I can't figure a way around this.
So I was googling around and found this code that does work


pipeline {
agent none
stages {
stage('stage1') {
failFast true
parallel {
stage('S1') {
agent {
label "machine1"
}
steps {
sh '''
echo "hostname: `hostname`"
echo "whoami:`whoami`"

'''
}
}
stage('S2') {
agent {
label "machine2"
}
steps {
sh '''
echo "hostname: `hostname`"
echo "whoami:`whoami`"

'''
}
}
}

}
}
}

But it's not dynamic, I need to have a stage per machine. Does anyone have a suggestion
about how I can achieve what I have in my original code?
I want to dynamically decide what nodes to run the code on.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/b2ae506c-023e-4604-896e-50b4eac6d817%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: trying to get Jenkins pipeline to run across multiple nodes in parallel

Mark Waite-2
As far as I can tell, dynamically defining the number of parallel stages will require that you use the scripted Pipeline syntax rather than the declarative Pipeline syntax.

There is a stackoverflow article that shows dynamically defined steps being used in a declarative Pipeline.  It uses either a Pipeline shared library or a snippet of scripted Pipeline at the end of the declarative Pipeline definition to generate the steps.

On Fri, Jul 12, 2019 at 6:04 AM Stephen DiMilla <[hidden email]> wrote:
I'm new to using pipelines.
I'm trying to execute the same commands against a number of nodes in parallel
Ideally I will pass a list of machines to run the same code against. Below I am 
hardcoding the machine names (machine1 and machine2) for the purposes of this example.

I found the following in Jenkins Pipeline examples:
----------------------------
def labels = ['precise', 'trusty'] // labels for Jenkins node types we will build on
def builders = [:]
for (x in labels) {
    def label = x // Need to bind the label variable before the closure - can't do 'for (label in labels)'

    // Create a map to pass in to the 'parallel' step so we can fire all the builds at once
    builders[label] = {
      node(label) {
        // build steps that should happen on all nodes go here
      }
    }
}

parallel builders
----------------------------
So I adapted it to be the following:
----------------------------
pipeline {
stages {
stage('Stage1') {
steps {
def labels = ['machine1','machine2'] // labels for Jenkins node types we will build on
def builders = [:]
for (x in labels) {
def label = x // Need to bind the label variable before the closure - can't do 'for (label in labels)'

// Create a map to pass in to the 'parallel' step so we can fire all the builds at once
builders[label] = {
node(label) {
sh '''
echo "hostname:`hostname`"
echo "whoami:`whoami`"

'''
}
}
}
parallel builders
}
}
}
}
----------------------------

But when I use it I get the following error:
Running in Durability level: MAX_SURVIVABILITY
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: 5: Expected a step @ line 5, column 17.
                   def labels = ['machine1','machine2'] // labels for Jenkins node types we will build on
                   ^

WorkflowScript: 6: Expected a step @ line 6, column 17.
                   def builders = [:]
                   ^

WorkflowScript: 7: Expected a step @ line 7, column 17.
                   for (x in labels) {
                   ^

General error during semantic analysis: There's no @DataBoundConstructor on any constructor of class org.jenkinsci.plugins.workflow.cps.steps.ParallelStep

org.kohsuke.stapler.NoStaplerConstructorException: There's no @DataBoundConstructor on any constructor of class org.jenkinsci.plugins.workflow.cps.steps.ParallelStep


----------------------------

I can't figure a way around this.
So I was googling around and found this code that does work


pipeline {
agent none
stages {
stage('stage1') {
failFast true
parallel {
stage('S1') {
agent {
label "machine1"
}
steps {
sh '''
echo "hostname: `hostname`"
echo "whoami:`whoami`"

'''
}
}
stage('S2') {
agent {
label "machine2"
}
steps {
sh '''
echo "hostname: `hostname`"
echo "whoami:`whoami`"

'''
}
}
}

}
}
}

But it's not dynamic, I need to have a stage per machine. Does anyone have a suggestion
about how I can achieve what I have in my original code?
I want to dynamically decide what nodes to run the code on.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/b2ae506c-023e-4604-896e-50b4eac6d817%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Thanks!
Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtHHOtUuej6vYzDhKbauCXq1w7qehM7UbMod4zGkfYYY9g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: trying to get Jenkins pipeline to run across multiple nodes in parallel

Ivan Fernandez Calvo
In reply to this post by Stephen DiMilla
to use groovy in dleclarative you have to use the `script` step to warp your code
 
pipeline {
  stages {
    stage('Stage1') {
      steps {
        script { // YOU NEED TO USE THE SCRIPT BLOCK
          def labels = ['machine1', 'machine2'] // labels for Jenkins node types we will build on
          def builders = [: ]
          for (x in labels) {
            def label = x // Need to bind the label variable before the closure - can't do 'for (label in labels)'

            // Create a map to pass in to the 'parallel' step so we can fire all the builds at once
            builders[label] = {
              node(label) {
                sh ''
                '
                echo "hostname:`hostname`"
                echo "whoami:`whoami`"

                ''
                '
              }
            }
          }
          parallel builders
        }
      }
    }
  }
}

or define a function then call it, I personally like it much is cleaner

pipeline {
  stages {
    stage('Stage1') {
      steps {
        parallelTasks()
      }
    }
  }
}

def parallelTasks() {
  def labels = ['machine1', 'machine2'] // labels for Jenkins node types we will build on
  def builders = [: ]
  for (x in labels) {
    def label = x // Need to bind the label variable before the closure - can't do 'for (label in labels)'

    // Create a map to pass in to the 'parallel' step so we can fire all the builds at once
    builders[label] = {
      node(label) {
        sh ''
        '
        echo "hostname:`hostname`"
        echo "whoami:`whoami`"

        ''
        '
      }
    }
  }
  parallel builders
}

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/c0e081d0-668a-4eb3-b538-be3ada207078%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.