Help with NotSerializableException in pipeline

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Help with NotSerializableException in pipeline

Slava Dubrovskiy
Hi.

I use a special algorithm to pre-sort the steps for parallel start.
Here is my test pipeline:

#!groovy

def behatList =['AutoSuiteSet_0', 'AutoSuiteSet_1', 'AutoSuiteSet_2', 'AutoSuiteSet_3', 'AutoSuiteSet_4', 'AutoSuiteSet_5']
def suitesStat=[AutoSuiteSet_0:0, AutoSuiteSet_1:1, AutoSuiteSet_2:2, AutoSuiteSet_3:3, AutoSuiteSet_4:4, AutoSuiteSet_5:5]

stage
("test") {
    node
('master') {
        behatList2
=sortSuites(behatList, suitesStat)
        echo
"SUITES2=${behatList2}"
   
}
}

@NonCPS
def sortSuites(suites, suites_time){
    timeLimit
= suites_time.values().max()
   
def suitesMap= [:]
   
for(s in suites){
        x
=suites_time.find{ artifact -> artifact.key == s}
       
if(x){
            suitesMap
.put(x.key, x.value)
       
}else{
            suitesMap
.put(s, timeLimit)
       
}
   
}
    tasks
= [suitesMap]
    timeLimit
= suitesMap.values().max()
   
while(canSplit()) {
        tasks
= tasks.collect { t ->
           
if(checkLimit(t)){
                t
= splitTo2(t)
           
}
            t
       
}.flatten()
   
}
    tasks
.sort { a, b -> b.values().sum() <=> a.values().sum() }
    tasks
= tasks.collect { t -> t.keySet()}
   
return tasks
}


@NonCPS
def checkLimit(t) {
   
if(t.values().sum()>timeLimit && t.size()>1){
       
return true
   
}else{
       
return false
   
}
}

@NonCPS
def canSplit() {
 
for(t in tasks) {
     
if(checkLimit(t)){
         
return true
     
}
 
}  
 
return false
}


@NonCPS
def splitTo2(int_map) {
    A
=[:]
    B
=[:]
   
for(n in int_map.sort{it.value}) {
       
if (A.size() < B.size()) {
            A
.put(n.key, n.value)
       
}else{
            B
.put(n.key, n.value)
       
}
   
}
   
return [A, B]
}



If I run it, I get the error:
an exception which occurred:
 
in field delegate
 
in field closures
 
in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@7fd2cde1
Caused: java.io.NotSerializableException: java.util.LinkedHashMap$Entry
 at org
.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860)
 at org
.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
 at org
.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
 at org
.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOu
...

Please, help me what is wrong?
All methods under @NonCPS directive.

--
WBR,
Slava.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/e7a79c56-512f-4706-ab65-9a347966abb3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help with NotSerializableException in pipeline

Michael Pailloncy
It seems like there are some variables used out of their scope and some misused Groovy methods.

Here is a working version, IIUC your pipeline :-)

#!groovy

def behatList =['AutoSuiteSet_0', 'AutoSuiteSet_1', 'AutoSuiteSet_2', 'AutoSuiteSet_3', 'AutoSuiteSet_4', 'AutoSuiteSet_5']
def suitesStat=[AutoSuiteSet_0:0, AutoSuiteSet_1:1, AutoSuiteSet_2:2, AutoSuiteSet_3:3, AutoSuiteSet_4:4, AutoSuiteSet_5:5]

stage("test") {
    node('master') {
        behatList2=sortSuites(behatList, suitesStat)
        echo "SUITES2=${behatList2}"
    }
}

@NonCPS
def sortSuites(suites, suites_time){
    def timeLimit = suites_time.values().max()
    def suitesMap= [:]
    for(s in suites){
        def x = suites_time.find{ artifact -> artifact.key == s}
        if(x){
            suitesMap.put(x.key, x.value)
        }else{
            suitesMap.put(s, timeLimit)
        }
    }
    def tasks = [suitesMap]
    timeLimit = suitesMap.values().max()
    while(canSplit(tasks, timeLimit)) {
        tasks = tasks.collect { t ->
            if(checkLimit(t, timeLimit)){
                t = splitTo2(t)
            }
            t
        }.flatten()
    }
    tasks.sort { a, b -> b.values().sum() <=> a.values().sum() }
    // tasks = tasks.collect { t -> t.keySet()} // not working, multiple elements are collected here
    tasks = tasks.collectMany { t -> t.keySet()}
    return tasks
}


@NonCPS
def checkLimit(t, timeLimit) {
    if(t.values().sum()>timeLimit && t.size()>1){
        return true
    }else{
        return false
    }
}

@NonCPS
def canSplit(tasks, timeLimit) {
  for(t in tasks) {
      if(checkLimit(t, timeLimit)){
          return true
      }
  }  
  return false
}


@NonCPS
def splitTo2(int_map) {
    A=[:]
    B=[:]

    // int_map.sort{it.value} not working 
    // for(n in int_map.sort{it.value}) {
    for(n in int_map) {
        if (A.size() < B.size()) {
            A.put(n.key, n.value)
        }else{
            B.put(n.key, n.value)
        }
    }
    return [A, B]
}


However, what do you want to achieve exactly ? One of the main advantage of Pipeline is its resumability after restart. But if you use too many @NonCPS, you avoid this feature and this could lead to unexpected behaviour sometimes.

Hopefully it helps :-)


2017-08-04 19:17 GMT+02:00 Slava Dubrovskiy <[hidden email]>:
Hi.

I use a special algorithm to pre-sort the steps for parallel start.
Here is my test pipeline:

#!groovy

def behatList =['AutoSuiteSet_0', 'AutoSuiteSet_1', 'AutoSuiteSet_2', 'AutoSuiteSet_3', 'AutoSuiteSet_4', 'AutoSuiteSet_5']
def suitesStat=[AutoSuiteSet_0:0, AutoSuiteSet_1:1, AutoSuiteSet_2:2, AutoSuiteSet_3:3, AutoSuiteSet_4:4, AutoSuiteSet_5:5]

stage
("test") {
    node
('master') {
        behatList2
=sortSuites(behatList, suitesStat)
        echo
"SUITES2=${behatList2}"
   
}
}

@NonCPS
def sortSuites(suites, suites_time){
    timeLimit
= suites_time.values().max()
   
def suitesMap= [:]
   
for(s in suites){
        x
=suites_time.find{ artifact -> artifact.key == s}
       
if(x){
            suitesMap
.put(x.key, x.value)
       
}else{
            suitesMap
.put(s, timeLimit)
       
}
   
}
    tasks
= [suitesMap]
    timeLimit
= suitesMap.values().max()
   
while(canSplit()) {
        tasks
= tasks.collect { t ->
           
if(checkLimit(t)){
                t
= splitTo2(t)
           
}
            t
       
}.flatten()
   
}
    tasks
.sort { a, b -> b.values().sum() <=> a.values().sum() }
    tasks
= tasks.collect { t -> t.keySet()}
   
return tasks
}


@NonCPS
def checkLimit(t) {
   
if(t.values().sum()>timeLimit && t.size()>1){
       
return true
   
}else{
       
return false
   
}
}

@NonCPS
def canSplit() {
 
for(t in tasks) {
     
if(checkLimit(t)){
         
return true
     
}
 
}  
 
return false
}


@NonCPS
def splitTo2(int_map) {
    A
=[:]
    B
=[:]
   
for(n in int_map.sort{it.value}) {
       
if (A.size() < B.size()) {
            A
.put(n.key, n.value)
       
}else{
            B
.put(n.key, n.value)
       
}
   
}
   
return [A, B]
}



If I run it, I get the error:
an exception which occurred:
 
in field delegate
 
in field closures
 
in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@7fd2cde1
Caused: java.io.NotSerializableException: java.util.LinkedHashMap$Entry
 at org
.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860)
 at org
.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
 at org
.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
 at org
.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOu
...

Please, help me what is wrong?
All methods under @NonCPS directive.

--
WBR,
Slava.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/e7a79c56-512f-4706-ab65-9a347966abb3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAPO77c0_uu7GwQGL3GNGoyh-D878mrbiMj-1fUZefgN0bRPmsA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help with NotSerializableException in pipeline

jerome-2
In reply to this post by Slava Dubrovskiy
Maybe not relevent, but I would declare the function above/before the main stage. Maybe this won't change anything. Also is the suitesStat properly defined when declared? [AutoSuiteSet_0:0 ...]

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/196f9dc8-3445-4fca-8030-475caaec4191%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help with NotSerializableException in pipeline

Slava Dubrovskiy
In reply to this post by Michael Pailloncy

Hi All,

Thank you for answers. Finally I found correct way. Only map.sort was necessary move under @NonCPS function.
My work pipeline looks:


def behatList =['AutoSuiteSet_0', 'AutoSuiteSet_1', 'AutoSuiteSet_2', 'AutoSuiteSet_3', 'AutoSuiteSet_4', 'AutoSuiteSet_5']
def suitesStat=[AutoSuiteSet_0:0, AutoSuiteSet_1:1, AutoSuiteSet_2:2, AutoSuiteSet_3:3, AutoSuiteSet_4:4, AutoSuiteSet_5:5]

stage("test") {
    node('master') {
        behatList2=sortSuites(behatList, suitesStat)
        echo "SUITES2=${behatList2}"
    }
}

def sortSuites(suites, suites_time){
    def suitesMap = [:]
    if (suites_time.size()>0){
        timeLimit = suites_time.values().max()
        for(s in suites){
          x = suites_time.find{ artifact -> artifact.key == s}
          if(x){
            suitesMap.put(x.key, x.value)
          }else{
            suitesMap.put(s, timeLimit)
          }
        }
    }else{
        suitesList = suites.collect { t -> [t] }
        return suitesList
    }
    tasks = divideList(suitesMap)
    tasks = sortListInMap(tasks)
    tasksList = tasks.collect { t -> t.keySet()}
    return tasksList
}

def divideList(inputMap){
    inputMap=sortMap(inputMap)
    maxValue = inputMap.values().max()
    def outputSet = []
    def outputMap = [:]
    for (inputElement in inputMap) {
       possibleMap = outputMap + [inputElement]
       possibleMapSum = possibleMap.values().sum()
       if (possibleMapSum < maxValue) {
           outputMap = possibleMap
       }else if(possibleMapSum == maxValue) {
           outputSet.add(possibleMap)
           outputMap = [:]
       }else if(possibleMapSum > maxValue) {
           outputSet.add(outputMap)
           outputMap = [:]
           outputMap.put(inputElement.key, inputElement.value)
       }
    }
    if (outputMap) {
       outputSet.add(outputMap)
    }
    return outputSet
}

@NonCPS
def sortMap(map) {
  map.sort{a, b -> b.value <=> a.value }
}

@NonCPS
def sortListInMap(map) {
    map.sort { a, b -> b.values().sum() <=> a.values().sum() }
}




06.08.2017 17:47, Michael Pailloncy пишет:
It seems like there are some variables used out of their scope and some misused Groovy methods.

Here is a working version, IIUC your pipeline :-)

#!groovy

def behatList =['AutoSuiteSet_0', 'AutoSuiteSet_1', 'AutoSuiteSet_2', 'AutoSuiteSet_3', 'AutoSuiteSet_4', 'AutoSuiteSet_5']
def suitesStat=[AutoSuiteSet_0:0, AutoSuiteSet_1:1, AutoSuiteSet_2:2, AutoSuiteSet_3:3, AutoSuiteSet_4:4, AutoSuiteSet_5:5]

stage("test") {
    node('master') {
        behatList2=sortSuites(behatList, suitesStat)
        echo "SUITES2=${behatList2}"
    }
}

@NonCPS
def sortSuites(suites, suites_time){
    def timeLimit = suites_time.values().max()
    def suitesMap= [:]
    for(s in suites){
        def x = suites_time.find{ artifact -> artifact.key == s}
        if(x){
            suitesMap.put(x.key, x.value)
        }else{
            suitesMap.put(s, timeLimit)
        }
    }
    def tasks = [suitesMap]
    timeLimit = suitesMap.values().max()
    while(canSplit(tasks, timeLimit)) {
        tasks = tasks.collect { t ->
            if(checkLimit(t, timeLimit)){
                t = splitTo2(t)
            }
            t
        }.flatten()
    }
    tasks.sort { a, b -> b.values().sum() <=> a.values().sum() }
    // tasks = tasks.collect { t -> t.keySet()} // not working, multiple elements are collected here
    tasks = tasks.collectMany { t -> t.keySet()}
    return tasks
}


@NonCPS
def checkLimit(t, timeLimit) {
    if(t.values().sum()>timeLimit && t.size()>1){
        return true
    }else{
        return false
    }
}

@NonCPS
def canSplit(tasks, timeLimit) {
  for(t in tasks) {
      if(checkLimit(t, timeLimit)){
          return true
      }
  }  
  return false
}


@NonCPS
def splitTo2(int_map) {
    A=[:]
    B=[:]

    // int_map.sort{it.value} not working 
    // for(n in int_map.sort{it.value}) {
    for(n in int_map) {
        if (A.size() < B.size()) {
            A.put(n.key, n.value)
        }else{
            B.put(n.key, n.value)
        }
    }
    return [A, B]
}


However, what do you want to achieve exactly ? One of the main advantage of Pipeline is its resumability after restart. But if you use too many @NonCPS, you avoid this feature and this could lead to unexpected behaviour sometimes.

Hopefully it helps :-)


2017-08-04 19:17 GMT+02:00 Slava Dubrovskiy <[hidden email]>:
Hi.

I use a special algorithm to pre-sort the steps for parallel start.
Here is my test pipeline:

#!groovy

def behatList =['AutoSuiteSet_0', 'AutoSuiteSet_1', 'AutoSuiteSet_2', 'AutoSuiteSet_3', 'AutoSuiteSet_4', 'AutoSuiteSet_5']
def suitesStat=[AutoSuiteSet_0:0, AutoSuiteSet_1:1, AutoSuiteSet_2:2, AutoSuiteSet_3:3, AutoSuiteSet_4:4, AutoSuiteSet_5:5]

stage
("test") {
    node
('master') {
        behatList2
=sortSuites(behatList, suitesStat)
        echo
"SUITES2=${behatList2}"
   
}
}

@NonCPS
def sortSuites(suites, suites_time){
    timeLimit
= suites_time.values().max()
   
def suitesMap= [:]
   
for(s in suites){
        x
=suites_time.find{ artifact -> artifact.key == s}
       
if(x){
            suitesMap
.put(x.key, x.value)
       
}else{
            suitesMap
.put(s, timeLimit)
       
}
   
}
    tasks
= [suitesMap]
    timeLimit
= suitesMap.values().max()
   
while(canSplit()) {
        tasks
= tasks.collect { t ->
           
if(checkLimit(t)){
                t
= splitTo2(t)
           
}
            t
       
}.flatten()
   
}
    tasks
.sort { a, b -> b.values().sum() <=> a.values().sum() }
    tasks
= tasks.collect { t -> t.keySet()}
   
return tasks
}


@NonCPS
def checkLimit(t) {
   
if(t.values().sum()>timeLimit && t.size()>1){
       
return true
   
}else{
       
return false
   
}
}

@NonCPS
def canSplit() {
 
for(t in tasks) {
     
if(checkLimit(t)){
         
return true
     
}
 
}  
 
return false
}


@NonCPS
def splitTo2(int_map) {
    A
=[:]
    B
=[:]
   
for(n in int_map.sort{it.value}) {
       
if (A.size() < B.size()) {
            A
.put(n.key, n.value)
       
}else{
            B
.put(n.key, n.value)
       
}
   
}
   
return [A, B]
}



If I run it, I get the error:
an exception which occurred:
 
in field delegate
 
in field closures
 
in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@7fd2cde1
Caused: java.io.NotSerializableException: java.util.LinkedHashMap$Entry
 at org
.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860)
 at org
.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
 at org
.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
 at org
.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOu
...

Please, help me what is wrong?
All methods under @NonCPS directive.

--
WBR,
Slava.
--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/e7a79c56-512f-4706-ab65-9a347966abb3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAPO77c0_uu7GwQGL3GNGoyh-D878mrbiMj-1fUZefgN0bRPmsA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

-- 
WBD,
Viacheslav Dubrovskyi

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/19b4ba24-82ea-a925-12c5-a82528816e21%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help with NotSerializableException in pipeline

Slava Dubrovskiy
In reply to this post by jerome-2

Hi

> Also is the suitesStat properly defined when declared?

Yes. Also I check size of map in function.


07.08.2017 15:41, [hidden email] пишет:
Maybe not relevent, but I would declare the function above/before the main stage. Maybe this won't change anything. Also is the suitesStat properly defined when declared? [AutoSuiteSet_0:0 ...]
--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/196f9dc8-3445-4fca-8030-475caaec4191%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
WBD,
Viacheslav Dubrovskyi

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/eb7a4350-938c-9a12-b24a-9f0cfab24143%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to group and sort tests and run in parallel without Parallel Test Executor Plugin

Slava Dubrovskiy
In reply to this post by Michael Pailloncy

Hi Michael,

We have list of tests which I would like run in parallel. The Parallel Test Executor Plugin doesn't work in our case, because we don't use junit and the list of tests (behatList) is result of script work. I can't run all tests on one node because it will take about 16 hours.

So I save results of previous builds to groovy file as map and load it before (suitesStat) . Then I group and sort tests (sortSuites(suites, suites_time) in previous post). And then I use this list for create parallel run.


def behatList =['AutoSuiteSet_0', 'AutoSuiteSet_1', 'AutoSuiteSet_2', 'AutoSuiteSet_3', 'AutoSuiteSet_4', 'AutoSuiteSet_5']
def suitesStat=[AutoSuiteSet_0:0, AutoSuiteSet_1:1, AutoSuiteSet_2:2, AutoSuiteSet_3:3, AutoSuiteSet_4:4, AutoSuiteSet_5:5]
behatList2=sortSuites(behatList, suitesStat)

stage("test") {
    behatList=sortSuites(behatList, suitesStat)
    def enviroments_b = [failFast: failFirstError]
    for (int j = 0; j < behatList.size() ; j++) {
        int index_b=j
        enviroments_b["TestEnv behat=${behatList[index_b]}"] = {
          node('behat') {
             for (int j2 = 0; j2 < behatList[index_b].size() ; j2++) {
                int index_b2=j2
                sh "./run_test ${behatList[index_b][index_b2]}"
             }
          }
        }
    }
    parallel enviroments_b
}
...
        


06.08.2017 17:47, Michael Pailloncy пишет:
However, what do you want to achieve exactly ? 

-- 
WBD,
Viacheslav Dubrovskyi

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/bad445be-212a-6f0b-7001-0c9212761be1%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to group and sort tests and run in parallel without Parallel Test Executor Plugin

Michael Pailloncy
Hi, thanks for the full explanation of your use case :-)
How do you keep the Groovy file containing your previous builds result ? inside an agent's workspace ?

2017-08-09 14:52 GMT+02:00 Viacheslav Dubrovskyi <[hidden email]>:

Hi Michael,

We have list of tests which I would like run in parallel. The Parallel Test Executor Plugin doesn't work in our case, because we don't use junit and the list of tests (behatList) is result of script work. I can't run all tests on one node because it will take about 16 hours.

So I save results of previous builds to groovy file as map and load it before (suitesStat) . Then I group and sort tests (sortSuites(suites, suites_time) in previous post). And then I use this list for create parallel run.


def behatList =['AutoSuiteSet_0', 'AutoSuiteSet_1', 'AutoSuiteSet_2', 'AutoSuiteSet_3', 'AutoSuiteSet_4', 'AutoSuiteSet_5']
def suitesStat=[AutoSuiteSet_0:0, AutoSuiteSet_1:1, AutoSuiteSet_2:2, AutoSuiteSet_3:3, AutoSuiteSet_4:4, AutoSuiteSet_5:5]
behatList2=sortSuites(behatList, suitesStat)

stage("test") {
    behatList=sortSuites(behatList, suitesStat)
    def enviroments_b = [failFast: failFirstError]
    for (int j = 0; j < behatList.size() ; j++) {
        int index_b=j
        enviroments_b["TestEnv behat=${behatList[index_b]}"] = {
          node('behat') {
             for (int j2 = 0; j2 < behatList[index_b].size() ; j2++) {
                int index_b2=j2
                sh "./run_test ${behatList[index_b][index_b2]}"
             }
          }
        }
    }
    parallel enviroments_b
}
...
        


06.08.2017 17:47, Michael Pailloncy пишет:
However, what do you want to achieve exactly ? 

-- 
WBD,
Viacheslav Dubrovskyi

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/bad445be-212a-6f0b-7001-0c9212761be1%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAPO77c3TeEWZEsxNHXb-y0nRg%3DZ5%2BVnveqVcZuiRBj7GkttjBQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to group and sort tests and run in parallel without Parallel Test Executor Plugin

Slava Dubrovskiy
Hi

15.08.2017 03:11, Michael Pailloncy пишет:
Hi, thanks for the full explanation of your use case :-)
How do you keep the Groovy file containing your previous builds result ? inside an agent's workspace ?

No, as artifact:
def stat_file = "suite_statics.groovy"
for (t2 in behatList) {
    suitesStat.put(behatList[t][t2], duration_s)
}
writeFile file: "stats/${stat_file}", text: "suitesStat=${suitesStat}\nreturn this;"

and then save stats/${stat_file} as artifact



2017-08-09 14:52 GMT+02:00 Viacheslav Dubrovskyi <[hidden email]>:

Hi Michael,

We have list of tests which I would like run in parallel. The Parallel Test Executor Plugin doesn't work in our case, because we don't use junit and the list of tests (behatList) is result of script work. I can't run all tests on one node because it will take about 16 hours.

So I save results of previous builds to groovy file as map and load it before (suitesStat) . Then I group and sort tests (sortSuites(suites, suites_time) in previous post). And then I use this list for create parallel run.


def behatList =['AutoSuiteSet_0', 'AutoSuiteSet_1', 'AutoSuiteSet_2', 'AutoSuiteSet_3', 'AutoSuiteSet_4', 'AutoSuiteSet_5']
def suitesStat=[AutoSuiteSet_0:0, AutoSuiteSet_1:1, AutoSuiteSet_2:2, AutoSuiteSet_3:3, AutoSuiteSet_4:4, AutoSuiteSet_5:5]
behatList2=sortSuites(behatList, suitesStat)

stage("test") {
    behatList=sortSuites(behatList, suitesStat)
    def enviroments_b = [failFast: failFirstError]
    for (int j = 0; j < behatList.size() ; j++) {
        int index_b=j
        enviroments_b["TestEnv behat=${behatList[index_b]}"] = {
          node('behat') {
             for (int j2 = 0; j2 < behatList[index_b].size() ; j2++) {
                int index_b2=j2
                sh "./run_test ${behatList[index_b][index_b2]}"
             }
          }
        }
    }
    parallel enviroments_b
}
...
        


06.08.2017 17:47, Michael Pailloncy пишет:
However, what do you want to achieve exactly ? 

-- 
WBD,
Viacheslav Dubrovskyi
--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/bad445be-212a-6f0b-7001-0c9212761be1%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAPO77c3TeEWZEsxNHXb-y0nRg%3DZ5%2BVnveqVcZuiRBj7GkttjBQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

-- 
WBD,
Viacheslav Dubrovskyi

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/50260078-77cc-c360-ef06-b5de745a565d%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
Loading...