In jenkins test harness (only) node() {} commands crash the executor

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

In jenkins test harness (only) node() {} commands crash the executor

Michael Carter
Last of the thorns in my side before I release this code internally today.   All my tests via pipeline suddenly stopped working a week ago.   One thing I did try was having a few minute pause at the beginning of this command so I could log in via the http console.   Then tired creating the job via the UI and same results.  As soon as the node part start it crashes the executor and all jobs just spin.  

Any echo commands and such on the job run fine.   As long as there is no node() {} command

Script running
node
() {
  echo
"hello"
}


 
44.247 [jobname-enableTest #1] [Pipeline] Start of Pipeline
 
47.466 [jobname-enableTest #1] [Pipeline] node
 
49.717 [id=47] SEVERE hudson.model.Executor#finish1: Executor threw an exception
java
.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.sameThreadExecutor()Lcom/google/common/util/concurrent/ListeningExecutorService;
 at org
.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:90)
 at org
.jenkinsci.plugins.workflow.cps.CpsFlowExecution.runInCpsVmThread(CpsFlowExecution.java:900)
 at org
.jenkinsci.plugins.workflow.cps.CpsBodyInvoker.start(CpsBodyInvoker.java:153)
 at org
.jenkinsci.plugins.workflow.cps.CpsBodyInvoker.start(CpsBodyInvoker.java:56)
 at org
.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable.run(ExecutorStepExecution.java:844)
 at hudson
.model.ResourceController.execute(ResourceController.java:97)
 at hudson
.model.Executor.run(Executor.java:428)

Not sure what I did to cause the problem but I'm sure this is just some library I'm missing as it seems to be the root of all my issues so far.   Just not sure where to look or where to start.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/3ac8d5d6-bf08-40d5-ad44-8dceeca1eaa3o%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: In jenkins test harness (only) node() {} commands crash the executor

Jesse Glick-4
On Wed, Jul 15, 2020 at 7:42 AM Michael Carter
<[hidden email]> wrote:
> java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.sameThreadExecutor()Lcom/google/common/util/concurrent/ListeningExecutorService;
>  at org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:90)
>  …
>
> Not sure what I did to cause the problem

Somehow pulled in an incompatible version of `guava`. Jenkins bundles
11.x, and without a lot of advanced acrobatics¹ you cannot override
this. If your plugin bundles some third-party libraries, they may be
depending on a newer version of Guava and Maven picks that version for
tests, breaking everything. Check

mvn dependency:tree

and use an `<exclusion>` as necessary. Unless the third-party library
actually _requires_ newer Guava features, in which case you have
problems.


Also, did your tests stop working with no code changes? If `master`
was blue/green until now, check whether your `Jenkinsfile` is using,
say,

buildPlugin(configurations: recommendedConfigurations())

which is nondeterministic. If tests broke inside a PR (I would
recommend a PR for all nontrivial changes, even if you are a sole
maintainer, just for trackability and CI), link to that from any post
to the mailing list.


¹Read: https://www.jenkins.io/doc/developer/plugin-development/dependencies-and-class-loading/#pluginfirstclassloader-and-its-discontents

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1SG%3D%2BuPiiT3LURfW88oSBZU%3D3X09YCLi6ia4XwFuED3A%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: In jenkins test harness (only) node() {} commands crash the executor

Michael Carter
Tried excluding the 27.1-jre version out of one of the apis I'm using so it can pick up version 27.1-jre of guava from the war.   Didn't work.

Tried adding directly:

        <dependency>
         
<groupId>com.google.guava</groupId>
         
<artifactId>guava</artifactId>
         
<version>27.1-jre</version>
         
<scope>test</scope>
       
</dependency>


Didn't work either.   I think the thread is crashing.  Any way to turn on more debugging?   Doing -X on maven doesn't put the jenkins harness into debug mode or anything.

As for did the test stop working without code change.   Not to that test but I as doing a lot of work in other areas before re-testing pipeline so probably something I did... problem is backtracking what I added from it last working is near impossible at this point.

On Wednesday, July 15, 2020 at 8:13:55 AM UTC-4, Jesse Glick wrote:
On Wed, Jul 15, 2020 at 7:42 AM Michael Carter
<<a href="javascript:" target="_blank" gdf-obfuscated-mailto="kpzUVK1NAgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">mikeyca...@...> wrote:
> java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.sameThreadExecutor()Lcom/google/common/util/concurrent/ListeningExecutorService;
>  at org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:90)
>  …
>
> Not sure what I did to cause the problem

Somehow pulled in an incompatible version of `guava`. Jenkins bundles
11.x, and without a lot of advanced acrobatics¹ you cannot override
this. If your plugin bundles some third-party libraries, they may be
depending on a newer version of Guava and Maven picks that version for
tests, breaking everything. Check

mvn dependency:tree

and use an `<exclusion>` as necessary. Unless the third-party library
actually _requires_ newer Guava features, in which case you have
problems.


Also, did your tests stop working with no code changes? If `master`
was blue/green until now, check whether your `Jenkinsfile` is using,
say,

buildPlugin(configurations: recommendedConfigurations())

which is nondeterministic. If tests broke inside a PR (I would
recommend a PR for all nontrivial changes, even if you are a sole
maintainer, just for trackability and CI), link to that from any post
to the mailing list.


¹Read: <a href="https://www.jenkins.io/doc/developer/plugin-development/dependencies-and-class-loading/#pluginfirstclassloader-and-its-discontents" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.jenkins.io%2Fdoc%2Fdeveloper%2Fplugin-development%2Fdependencies-and-class-loading%2F%23pluginfirstclassloader-and-its-discontents\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFRGS1t6WZzCh2CF7UC8Qne_nqfPw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.jenkins.io%2Fdoc%2Fdeveloper%2Fplugin-development%2Fdependencies-and-class-loading%2F%23pluginfirstclassloader-and-its-discontents\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFRGS1t6WZzCh2CF7UC8Qne_nqfPw&#39;;return true;">https://www.jenkins.io/doc/developer/plugin-development/dependencies-and-class-loading/#pluginfirstclassloader-and-its-discontents

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/67873b52-020b-4d56-b3c7-17522c1a57b8o%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: In jenkins test harness (only) node() {} commands crash the executor

Jesse Glick-4
On Wed, Jul 15, 2020 at 9:17 AM Michael Carter
<[hidden email]> wrote:
> so it can pick up version 27.1-jre of guava from the war.

From `jenkins.war`? As mentioned, that bundles 11, not 27. Keep
editing `pom.xml` until

$ mvn dependency:tree | fgrep guava
[INFO] |  +- com.google.guava:guava:jar:11.0.1:provided

> problem is backtracking what I added from it last working is near impossible at this point.

git bisect

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr0yGuc%2ByC2oCe1Z_HOd9tj0gnx16866dUXwX2YLwNFynA%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: In jenkins test harness (only) node() {} commands crash the executor

Michael Carter
+- org.jenkins-ci.main:jenkins-core:jar:2.235.1:provided
|  +- com.google.guava:guava:jar:27.1-jre:provided

Guess that's my problem... somehow when I upgraded the pom.xml to pull the 2.235.1/2.222.4 it must have broke then.

At least this gives me a place to start looking.

On Wednesday, July 15, 2020 at 1:42:56 PM UTC-4, Jesse Glick wrote:
On Wed, Jul 15, 2020 at 9:17 AM Michael Carter
<<a href="javascript:" target="_blank" gdf-obfuscated-mailto="28FMbmV6AwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">mikeyca...@...> wrote:
> so it can pick up version 27.1-jre of guava from the war.

From `jenkins.war`? As mentioned, that bundles 11, not 27. Keep
editing `pom.xml` until

$ mvn dependency:tree | fgrep guava
[INFO] |  +- com.google.guava:guava:jar:11.0.1:provided

> problem is backtracking what I added from it last working is near impossible at this point.

git bisect

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/a3f4739f-f9a1-40ed-bc5b-903a14c2beaao%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: In jenkins test harness (only) node() {} commands crash the executor

Michael Carter
... and that did the trick:

        <dependency>
         
<groupId>com.google.guava</groupId>
         
<artifactId>guava</artifactId>
         
<version>11.0.1</version>
         
<scope>test</scope>
       
</dependency>


Output:
  62.511 [jobname-PipelineManualFailTest #1] [Pipeline] node
 
62.763 [jobname-PipelineManualFailTest #1] Running on Jenkins in ....workspace/jobname
 
62.964 [jobname-PipelineManualFailTest #1] [Pipeline] {
 
63.568 [jobname-PipelineManualFailTest #1] [Pipeline] stage
 
63.668 [jobname-PipelineManualFailTest #1] [Pipeline] { (Example Build)
 
64.121 [jobname-PipelineManualFailTest #1] [Pipeline] sh


Guess that's what had me all turned around when I saw the guava jar coming from the jenkins-core I just assumed it was the correct one I should be using.

Thanks for all the help everyone!

On Wednesday, July 15, 2020 at 2:05:15 PM UTC-4, Michael Carter wrote:
+- org.jenkins-ci.main:jenkins-core:jar:2.235.1:provided
|  +- com.google.guava:guava:jar:27.1-jre:provided

Guess that's my problem... somehow when I upgraded the pom.xml to pull the 2.235.1/2.222.4 it must have broke then.

At least this gives me a place to start looking.

On Wednesday, July 15, 2020 at 1:42:56 PM UTC-4, Jesse Glick wrote:
On Wed, Jul 15, 2020 at 9:17 AM Michael Carter
<[hidden email]> wrote:
> so it can pick up version 27.1-jre of guava from the war.

From `jenkins.war`? As mentioned, that bundles 11, not 27. Keep
editing `pom.xml` until

$ mvn dependency:tree | fgrep guava
[INFO] |  +- com.google.guava:guava:jar:11.0.1:provided

> problem is backtracking what I added from it last working is near impossible at this point.

git bisect

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/c2bc8121-ae41-404c-98f0-3c4f67171877o%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: In jenkins test harness (only) node() {} commands crash the executor

Jesse Glick-4
On Wed, Jul 15, 2020 at 2:09 PM Michael Carter
<[hidden email]> wrote:
> ... and that did the trick:
>
>         <dependency>
>           <groupId>com.google.guava</groupId>
>           <artifactId>guava</artifactId>
>           <version>11.0.1</version>
>           <scope>test</scope>
>         </dependency>

Not recommended. Better to find out which other dependency was pulling
it a newer version of Guava and add an `<exclusion>` to suppress that.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1Jd6dp0MfL%3Dogiv0CJgzokhgfb0iXsXutrG9MqoP9YJg%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: In jenkins test harness (only) node() {} commands crash the executor

Michael Carter
jenkins-core:jar appears to be the one... 

On Wednesday, July 15, 2020 at 3:00:23 PM UTC-4, Jesse Glick wrote:
On Wed, Jul 15, 2020 at 2:09 PM Michael Carter
<<a href="javascript:" target="_blank" gdf-obfuscated-mailto="ys3qbJ9-AwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">mikeyca...@...> wrote:
> ... and that did the trick:
>
>         <dependency>
>           <groupId>com.google.guava</groupId>
>           <artifactId>guava</artifactId>
>           <version>11.0.1</version>
>           <scope>test</scope>
>         </dependency>

Not recommended. Better to find out which other dependency was pulling
it a newer version of Guava and add an `<exclusion>` to suppress that.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/af72e673-8f91-4f7b-9f7e-288f7bbec467o%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: In jenkins test harness (only) node() {} commands crash the executor

Michael Carter
So my ignorance might be showing but I think I tracked down where the 27.1-jre is coming from as if I do an exclude on jenkins-core:jar and pipeline-model-definition it goes away... well all versions of gauva go away.

I'm guessing it's part of this?

      <jenkins.version>2.235.1</jenkins.version>
      <jenkins.bom>bom-2.235.x</jenkins.bom>

         
<dependency>
           
<groupId>io.jenkins.tools.bom</groupId>
           
<artifactId>${jenkins.bom}</artifactId>
           
<version>${jenkins.bomver}</version>
           
<scope>import</scope>
           
<type>pom</type>
         
</dependency>




On Wednesday, July 15, 2020 at 4:09:23 PM UTC-4, Michael Carter wrote:
jenkins-core:jar appears to be the one... 

On Wednesday, July 15, 2020 at 3:00:23 PM UTC-4, Jesse Glick wrote:
On Wed, Jul 15, 2020 at 2:09 PM Michael Carter
<[hidden email]> wrote:
> ... and that did the trick:
>
>         <dependency>
>           <groupId>com.google.guava</groupId>
>           <artifactId>guava</artifactId>
>           <version>11.0.1</version>
>           <scope>test</scope>
>         </dependency>

Not recommended. Better to find out which other dependency was pulling
it a newer version of Guava and add an `<exclusion>` to suppress that.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/ab24f231-539c-492c-9de7-df1a044d0d6bo%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: In jenkins test harness (only) node() {} commands crash the executor

Jesse Glick-4
On Wed, Jul 15, 2020 at 4:25 PM Michael Carter
<[hidden email]> wrote:
> if I do an exclude on jenkins-core:jar and pipeline-model-definition it goes away... well all versions of gauva go away.

Try excluding only from `pipeline-model-definition`.

Without seeing your POM it is hard to help here, this is all just
general advice.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr0rhqHL4BneDQV4YPp5vDvjZ%3DgeYePg7UhC4G2bKSZx9Q%40mail.gmail.com.