gitSCM failed after updates

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

gitSCM failed after updates

Giles
I'm running Jenkins 2.164.3 on a Windows server.  It's been running well for several months.  Every Friday evening I do all the Jenkins plugin updates.  After this past Friday's updates (details below), all our jobs are broken - apparently because the GitSCM checkout is broken.  Our Jenkinsfiles are all stored in a git repo, so Jenkins attempts to pull a new version before running the job.  This now fails.  I created a test Jenkinsfile (editing directly in Jenkins rather than in the git repo) with this code in it:

        def scmVars = checkout(
            [
                $class: 'GitSCM',
                branches: [[name: '*/' + branch ]],
                doGenerateSubmoduleConfigurations: false,
                extensions: [],
                submoduleCfg: [],
                userRemoteConfigs: [
                    [
                        credentialsId: jenkinsCred,
                        url: jenkinsRepo,
                    ]
                ]
            ]
        );

(Assume the "jenkinsCred" and "jenkinsRepo" values are correct.)  The failure looks like this:

[Pipeline] {
[Pipeline] checkout
using credential 29d83033-ebf6-4811-9c45-b0aadec775c2
 > C:\cygwin64\bin\git.exe rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > C:\cygwin64\bin\git.exe config remote.origin.url [hidden email]:*/*.git # timeout=10
Fetching upstream changes from [hidden email]:*/*.git
 > C:\cygwin64\bin\git.exe --version # timeout=10
using GIT_SSH to set credentials Read-only access to the "Jenkins" repository at github.com/*.
 > C:\cygwin64\bin\git.exe fetch --tags --force --progress [hidden email]:*/*.git +refs/heads/*:refs/remotes/origin/* # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from [hidden email]:*/*.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: hudson.plugins.git.GitException: Command "C:\cygwin64\bin\git.exe fetch --tags --force --progress [hidden email]:*/*.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: cannot run D:\Jenkins\workspace\DEBUG_07_git_Checkout@tmp\jenkins-gitclient-ssh1436722960383118000.bat: No such file or directory
fatal: unable to fork

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2318)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1905)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:81)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:488)
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:892)
	... 11 more
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: Error fetching remote repo 'origin'
Finished: FAILURE

This is essentially identical to the failure we're seeing for all jobs. I assume it's normal behaviour that GitSCM generates and
tries to run a random-named bat file? It's unclear to me if the bat file isn't created or is unreadable, although Jenkins is running
as SYSTEM and has generally been able to read and write everything.

Does anyone have any thoughts on what may have gone wrong?

-----

The updates made Friday night:

>>> Artifactory 3.2.2 -> 3.2.4  [ security warning ]                           

>>> Pipeline: API 2.34 -> 2.35                                                 
>>> Pipeline: Basic Steps 2.16 -> 2.18                                         
>>> Pipeline: Declarative 1.3.8 -> 1.3.9                                       
>>> Pipeline: Declarative Extension Points API 1.3.8 -> 1.3.9                  
>>> Pipeline: Groovy 2.69 -> 2.70                                              
>>> Pipeline: Model API 1.3.8 -> 1.3.9                                                                      
>>> Pipeline: Nodes and Processes 2.30 -> 2.31                                                               
>>> Pipeline: SCM Step 2.7 -> 2.9                                                        
>>> Pipeline: Stage Tags Metadata 1.3.8 -> 1.3.9                               
>>> Pipeline: Step API 2.19 -> 2.20                                                      
>>> Slack Notification 2.23 -> 2.24
I've since rolled back all of the "Pipeline: ..." changes.  The problem didn't change.

About 40 Cygwin updates (I didn't enumerate those.)

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/2d7c0927-22ba-4443-8121-0867169df831%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: gitSCM failed after updates

Mark Waite-2
Are you quite sure that your configuration did not change in about the same time as that upgrade?

The log file indicates that you're running command line git as provided by Cygwin.  Is that intentional?  Has it always been that way?  

I don't test the Jenkins git plugin with Cygwin.  I test with Git for Windows.  I'm glad cygwin git works, but don't intend to apply effort to make it work.

I don't see anything suspicious or troubling in the list of updates.  As far as I can tell, none of them should have changed that behavior of the git plugin.

On Tue, Jun 11, 2019 at 1:54 PM Giles <[hidden email]> wrote:
I'm running Jenkins 2.164.3 on a Windows server.  It's been running well for several months.  Every Friday evening I do all the Jenkins plugin updates.  After this past Friday's updates (details below), all our jobs are broken - apparently because the GitSCM checkout is broken.  Our Jenkinsfiles are all stored in a git repo, so Jenkins attempts to pull a new version before running the job.  This now fails.  I created a test Jenkinsfile (editing directly in Jenkins rather than in the git repo) with this code in it:

        def scmVars = checkout(
            [
                $class: 'GitSCM',
                branches: [[name: '*/' + branch ]],
                doGenerateSubmoduleConfigurations: false,
                extensions: [],
                submoduleCfg: [],
                userRemoteConfigs: [
                    [
                        credentialsId: jenkinsCred,
                        url: jenkinsRepo,
                    ]
                ]
            ]
        );

(Assume the "jenkinsCred" and "jenkinsRepo" values are correct.)  The failure looks like this:

[Pipeline] {
[Pipeline] checkout
using credential 29d83033-ebf6-4811-9c45-b0aadec775c2
 > C:\cygwin64\bin\git.exe rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > C:\cygwin64\bin\git.exe config remote.origin.url [hidden email]:*/*.git # timeout=10
Fetching upstream changes from [hidden email]:*/*.git
 > C:\cygwin64\bin\git.exe --version # timeout=10
using GIT_SSH to set credentials Read-only access to the "Jenkins" repository at github.com/*.
 > C:\cygwin64\bin\git.exe fetch --tags --force --progress [hidden email]:*/*.git +refs/heads/*:refs/remotes/origin/* # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from [hidden email]:*/*.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: hudson.plugins.git.GitException: Command "C:\cygwin64\bin\git.exe fetch --tags --force --progress [hidden email]:*/*.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: cannot run D:\Jenkins\workspace\DEBUG_07_git_Checkout@tmp\jenkins-gitclient-ssh1436722960383118000.bat: No such file or directory
fatal: unable to fork

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2318)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1905)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:81)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:488)
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:892)
	... 11 more
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: Error fetching remote repo 'origin'
Finished: FAILURE

This is essentially identical to the failure we're seeing for all jobs. I assume it's normal behaviour that GitSCM generates and
tries to run a random-named bat file? It's unclear to me if the bat file isn't created or is unreadable, although Jenkins is running
as SYSTEM and has generally been able to read and write everything.

Does anyone have any thoughts on what may have gone wrong?

-----

The updates made Friday night:

>>> Artifactory 3.2.2 -> 3.2.4  [ security warning ]                           

>>> Pipeline: API 2.34 -> 2.35                                                 
>>> Pipeline: Basic Steps 2.16 -> 2.18                                         
>>> Pipeline: Declarative 1.3.8 -> 1.3.9                                       
>>> Pipeline: Declarative Extension Points API 1.3.8 -> 1.3.9                  
>>> Pipeline: Groovy 2.69 -> 2.70                                              
>>> Pipeline: Model API 1.3.8 -> 1.3.9                                                                      
>>> Pipeline: Nodes and Processes 2.30 -> 2.31                                                               
>>> Pipeline: SCM Step 2.7 -> 2.9                                                        
>>> Pipeline: Stage Tags Metadata 1.3.8 -> 1.3.9                               
>>> Pipeline: Step API 2.19 -> 2.20                                                      
>>> Slack Notification 2.23 -> 2.24
I've since rolled back all of the "Pipeline: ..." changes.  The problem didn't change.

About 40 Cygwin updates (I didn't enumerate those.)

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/2d7c0927-22ba-4443-8121-0867169df831%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Thanks!
Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtEygQMpku8DGpOa%3DhFp7mS5E5TUaVNxjJwoG4RC0BM%3DfQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: gitSCM failed after updates

Giles
Nailed it in one.

I was sure it was a Jenkins issue ... but I'd upgraded Cygwin at the same time, and that probably included their version of git.  Which yes, is my default git.  I changed "Manage Jenkins" -> "Global Tool Configuration" -> git to the just-installed "Git for Windows", and ... fixed.  I may have to make some changes in the Jenkinsfiles themselves, but it seems clear now that the latest Cygwin git broke something, and that Git for Windows fixed it ...

This was crisis-level breakage for my department - I can't thank you enough.

On Tuesday, 11 June 2019 16:36:38 UTC-4, Mark Waite wrote:
Are you quite sure that your configuration did not change in about the same time as that upgrade?

The log file indicates that you're running command line git as provided by Cygwin.  Is that intentional?  Has it always been that way?  

I don't test the Jenkins git plugin with Cygwin.  I test with Git for Windows.  I'm glad cygwin git works, but don't intend to apply effort to make it work.

I don't see anything suspicious or troubling in the list of updates.  As far as I can tell, none of them should have changed that behavior of the git plugin.

On Tue, Jun 11, 2019 at 1:54 PM Giles <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="64vl15WgBgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">gile...@...> wrote:
I'm running Jenkins 2.164.3 on a Windows server.  It's been running well for several months.  Every Friday evening I do all the Jenkins plugin updates.  After this past Friday's updates (details below), all our jobs are broken - apparently because the GitSCM checkout is broken.  Our Jenkinsfiles are all stored in a git repo, so Jenkins attempts to pull a new version before running the job.  This now fails.  I created a test Jenkinsfile (editing directly in Jenkins rather than in the git repo) with this code in it:

        def scmVars = checkout(
            [
                $class: 'GitSCM',
                branches: [[name: '*/' + branch ]],
                doGenerateSubmoduleConfigurations: false,
                extensions: [],
                submoduleCfg: [],
                userRemoteConfigs: [
                    [
                        credentialsId: jenkinsCred,
                        url: jenkinsRepo,
                    ]
                ]
            ]
        );

(Assume the "jenkinsCred" and "jenkinsRepo" values are correct.)  The failure looks like this:

[Pipeline] {
[Pipeline] checkout
using credential 29d83033-ebf6-4811-9c45-b0aadec775c2
 > C:\cygwin64\bin\git.exe rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > C:\cygwin64\bin\git.exe config remote.origin.url [hidden email]:*/*.git # timeout=10
Fetching upstream changes from [hidden email]:*/*.git
 > C:\cygwin64\bin\git.exe --version # timeout=10
using GIT_SSH to set credentials Read-only access to the "Jenkins" repository at <a href="http://github.com/*" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fgithub.com%2F*\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNENYQ-MeB318WtyMw3be6vUrVzp2g&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fgithub.com%2F*\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNENYQ-MeB318WtyMw3be6vUrVzp2g&#39;;return true;">github.com/*.
 > C:\cygwin64\bin\git.exe fetch --tags --force --progress [hidden email]:*/*.git +refs/heads/*:refs/remotes/origin/* # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from [hidden email]:*/*.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: hudson.plugins.git.GitException: Command "C:\cygwin64\bin\git.exe fetch --tags --force --progress [hidden email]:*/*.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: cannot run D:\Jenkins\workspace\DEBUG_07_git_Checkout@tmp\jenkins-gitclient-ssh1436722960383118000.bat: No such file or directory
fatal: unable to fork

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2318)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1905)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:81)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:488)
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:892)
	... 11 more
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: Error fetching remote repo 'origin'
Finished: FAILURE

This is essentially identical to the failure we're seeing for all jobs. I assume it's normal behaviour that GitSCM generates and
tries to run a random-named bat file? It's unclear to me if the bat file isn't created or is unreadable, although Jenkins is running
as SYSTEM and has generally been able to read and write everything.

Does anyone have any thoughts on what may have gone wrong?

-----

The updates made Friday night:

>>> Artifactory 3.2.2 -> 3.2.4  [ security warning ]                           

>>> Pipeline: API 2.34 -> 2.35                                                 
>>> Pipeline: Basic Steps 2.16 -> 2.18                                         
>>> Pipeline: Declarative 1.3.8 -> 1.3.9                                       
>>> Pipeline: Declarative Extension Points API 1.3.8 -> 1.3.9                  
>>> Pipeline: Groovy 2.69 -> 2.70                                              
>>> Pipeline: Model API 1.3.8 -> 1.3.9                                                                      
>>> Pipeline: Nodes and Processes 2.30 -> 2.31                                                               
>>> Pipeline: SCM Step 2.7 -> 2.9                                                        
>>> Pipeline: Stage Tags Metadata 1.3.8 -> 1.3.9                               
>>> Pipeline: Step API 2.19 -> 2.20                                                      
>>> Slack Notification 2.23 -> 2.24
I've since rolled back all of the "Pipeline: ..." changes.  The problem didn't change.

About 40 Cygwin updates (I didn't enumerate those.)

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="64vl15WgBgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jenkins...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/2d7c0927-22ba-4443-8121-0867169df831%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/2d7c0927-22ba-4443-8121-0867169df831%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/2d7c0927-22ba-4443-8121-0867169df831%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/2d7c0927-22ba-4443-8121-0867169df831%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.


--
Thanks!
Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/3c977d6b-a180-402b-a5f7-bb4825ba2217%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: gitSCM failed after updates

Mark Waite-2
I'm glad that helped.  In the past, we've seen cases where a new version of ssh surprises the credential integration that is used by the git plugin.  I don't think your case is related to that, but I did see that Git for Windows 2.22.0 now includes OpenSSH 8.0.  I need to add that version into my test environment.  I'm not aware of any breakage with OpenSSH 8.0, but we've been surprised in the past.

On Tue, Jun 11, 2019 at 3:09 PM Giles <[hidden email]> wrote:
Nailed it in one.

I was sure it was a Jenkins issue ... but I'd upgraded Cygwin at the same time, and that probably included their version of git.  Which yes, is my default git.  I changed "Manage Jenkins" -> "Global Tool Configuration" -> git to the just-installed "Git for Windows", and ... fixed.  I may have to make some changes in the Jenkinsfiles themselves, but it seems clear now that the latest Cygwin git broke something, and that Git for Windows fixed it ...

This was crisis-level breakage for my department - I can't thank you enough.

On Tuesday, 11 June 2019 16:36:38 UTC-4, Mark Waite wrote:
Are you quite sure that your configuration did not change in about the same time as that upgrade?

The log file indicates that you're running command line git as provided by Cygwin.  Is that intentional?  Has it always been that way?  

I don't test the Jenkins git plugin with Cygwin.  I test with Git for Windows.  I'm glad cygwin git works, but don't intend to apply effort to make it work.

I don't see anything suspicious or troubling in the list of updates.  As far as I can tell, none of them should have changed that behavior of the git plugin.

On Tue, Jun 11, 2019 at 1:54 PM Giles <[hidden email]> wrote:
I'm running Jenkins 2.164.3 on a Windows server.  It's been running well for several months.  Every Friday evening I do all the Jenkins plugin updates.  After this past Friday's updates (details below), all our jobs are broken - apparently because the GitSCM checkout is broken.  Our Jenkinsfiles are all stored in a git repo, so Jenkins attempts to pull a new version before running the job.  This now fails.  I created a test Jenkinsfile (editing directly in Jenkins rather than in the git repo) with this code in it:

        def scmVars = checkout(
            [
                $class: 'GitSCM',
                branches: [[name: '*/' + branch ]],
                doGenerateSubmoduleConfigurations: false,
                extensions: [],
                submoduleCfg: [],
                userRemoteConfigs: [
                    [
                        credentialsId: jenkinsCred,
                        url: jenkinsRepo,
                    ]
                ]
            ]
        );

(Assume the "jenkinsCred" and "jenkinsRepo" values are correct.)  The failure looks like this:

[Pipeline] {
[Pipeline] checkout
using credential 29d83033-ebf6-4811-9c45-b0aadec775c2
 > C:\cygwin64\bin\git.exe rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > C:\cygwin64\bin\git.exe config remote.origin.url [hidden email]:*/*.git # timeout=10
Fetching upstream changes from [hidden email]:*/*.git
 > C:\cygwin64\bin\git.exe --version # timeout=10
using GIT_SSH to set credentials Read-only access to the "Jenkins" repository at github.com/*.
 > C:\cygwin64\bin\git.exe fetch --tags --force --progress [hidden email]:*/*.git +refs/heads/*:refs/remotes/origin/* # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from [hidden email]:*/*.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: hudson.plugins.git.GitException: Command "C:\cygwin64\bin\git.exe fetch --tags --force --progress [hidden email]:*/*.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: cannot run D:\Jenkins\workspace\DEBUG_07_git_Checkout@tmp\jenkins-gitclient-ssh1436722960383118000.bat: No such file or directory
fatal: unable to fork

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2318)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1905)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:81)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:488)
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:892)
	... 11 more
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: Error fetching remote repo 'origin'
Finished: FAILURE

This is essentially identical to the failure we're seeing for all jobs. I assume it's normal behaviour that GitSCM generates and
tries to run a random-named bat file? It's unclear to me if the bat file isn't created or is unreadable, although Jenkins is running
as SYSTEM and has generally been able to read and write everything.

Does anyone have any thoughts on what may have gone wrong?

-----

The updates made Friday night:

>>> Artifactory 3.2.2 -> 3.2.4  [ security warning ]                           

>>> Pipeline: API 2.34 -> 2.35                                                 
>>> Pipeline: Basic Steps 2.16 -> 2.18                                         
>>> Pipeline: Declarative 1.3.8 -> 1.3.9                                       
>>> Pipeline: Declarative Extension Points API 1.3.8 -> 1.3.9                  
>>> Pipeline: Groovy 2.69 -> 2.70                                              
>>> Pipeline: Model API 1.3.8 -> 1.3.9                                                                      
>>> Pipeline: Nodes and Processes 2.30 -> 2.31                                                               
>>> Pipeline: SCM Step 2.7 -> 2.9                                                        
>>> Pipeline: Stage Tags Metadata 1.3.8 -> 1.3.9                               
>>> Pipeline: Step API 2.19 -> 2.20                                                      
>>> Slack Notification 2.23 -> 2.24
I've since rolled back all of the "Pipeline: ..." changes.  The problem didn't change.

About 40 Cygwin updates (I didn't enumerate those.)

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/2d7c0927-22ba-4443-8121-0867169df831%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Thanks!
Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/3c977d6b-a180-402b-a5f7-bb4825ba2217%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Thanks!
Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtFezd_R29FWSTc0y6prFDbicpe0DDTj9Bvng%3DPFdgpTuA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: gitSCM failed after updates

Giles
Unfortunately, I appear to have gone from the frying pan to the fire.  I ran one deploy that hadn't worked previously and it did work, and I immediately responded with an "it's working" message.  And that single deploy still works.  But multiple others (all based on the same code, with only varying parameters) don't, all failing with this:

Started by user Giles
java.lang.NoSuchMethodError: org.eclipse.jgit.lib.Repository.getRef(Ljava/lang/String;)Lorg/eclipse/jgit/lib/Ref;
	at jenkins.plugins.git.GitSCMFileSystem$1.invoke(GitSCMFileSystem.java:117)
	at jenkins.plugins.git.GitSCMFileSystem$1.invoke(GitSCMFileSystem.java:114)
	at jenkins.plugins.git.GitSCMFileSystem$3.invoke(GitSCMFileSystem.java:193)
	at org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.withRepository(AbstractGitAPIImpl.java:29)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.withRepository(CliGitAPIImpl.java:81)
	at jenkins.plugins.git.GitSCMFileSystem.invoke(GitSCMFileSystem.java:189)
	at jenkins.plugins.git.GitSCMFileSystem.<init>(GitSCMFileSystem.java:114)
	at jenkins.plugins.git.GitSCMFileSystem$BuilderImpl.build(GitSCMFileSystem.java:353)
	at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:198)
	at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:174)
	at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:108)
	at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:67)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:293)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
Finished: FAILURE

This at least is a known error: it appears to happen when there's a newer version of git, or a misalignment of the git version and the git plugin.  I had initially installed the very new version of Git for Windows 2.22.0.  I downgraded that to 2.17.1.2, and eventually went to 2.21.0 - all resulting in the same failure.  Any thoughts?  It certainly doesn't seem to be the Git version, and the one working pipeline muddies the water further.

On Tuesday, 11 June 2019 17:12:11 UTC-4, Mark Waite wrote:
I'm glad that helped.  In the past, we've seen cases where a new version of ssh surprises the credential integration that is used by the git plugin.  I don't think your case is related to that, but I did see that Git for Windows 2.22.0 now includes OpenSSH 8.0.  I need to add that version into my test environment.  I'm not aware of any breakage with OpenSSH 8.0, but we've been surprised in the past.

On Tue, Jun 11, 2019 at 3:09 PM Giles <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="ztQ-2IaiBgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">gile...@...> wrote:
Nailed it in one.

I was sure it was a Jenkins issue ... but I'd upgraded Cygwin at the same time, and that probably included their version of git.  Which yes, is my default git.  I changed "Manage Jenkins" -> "Global Tool Configuration" -> git to the just-installed "Git for Windows", and ... fixed.  I may have to make some changes in the Jenkinsfiles themselves, but it seems clear now that the latest Cygwin git broke something, and that Git for Windows fixed it ...

This was crisis-level breakage for my department - I can't thank you enough.

On Tuesday, 11 June 2019 16:36:38 UTC-4, Mark Waite wrote:
Are you quite sure that your configuration did not change in about the same time as that upgrade?

The log file indicates that you're running command line git as provided by Cygwin.  Is that intentional?  Has it always been that way?  

I don't test the Jenkins git plugin with Cygwin.  I test with Git for Windows.  I'm glad cygwin git works, but don't intend to apply effort to make it work.

I don't see anything suspicious or troubling in the list of updates.  As far as I can tell, none of them should have changed that behavior of the git plugin.

On Tue, Jun 11, 2019 at 1:54 PM Giles <[hidden email]> wrote:
I'm running Jenkins 2.164.3 on a Windows server.  It's been running well for several months.  Every Friday evening I do all the Jenkins plugin updates.  After this past Friday's updates (details below), all our jobs are broken - apparently because the GitSCM checkout is broken.  Our Jenkinsfiles are all stored in a git repo, so Jenkins attempts to pull a new version before running the job.  This now fails.  I created a test Jenkinsfile (editing directly in Jenkins rather than in the git repo) with this code in it:

        def scmVars = checkout(
            [
                $class: 'GitSCM',
                branches: [[name: '*/' + branch ]],
                doGenerateSubmoduleConfigurations: false,
                extensions: [],
                submoduleCfg: [],
                userRemoteConfigs: [
                    [
                        credentialsId: jenkinsCred,
                        url: jenkinsRepo,
                    ]
                ]
            ]
        );

(Assume the "jenkinsCred" and "jenkinsRepo" values are correct.)  The failure looks like this:

[Pipeline] {
[Pipeline] checkout
using credential 29d83033-ebf6-4811-9c45-b0aadec775c2
 > C:\cygwin64\bin\git.exe rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > C:\cygwin64\bin\git.exe config remote.origin.url [hidden email]:*/*.git # timeout=10
Fetching upstream changes from [hidden email]:*/*.git
 > C:\cygwin64\bin\git.exe --version # timeout=10
using GIT_SSH to set credentials Read-only access to the "Jenkins" repository at <a href="http://github.com/*" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fgithub.com%2F*\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNENYQ-MeB318WtyMw3be6vUrVzp2g&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fgithub.com%2F*\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNENYQ-MeB318WtyMw3be6vUrVzp2g&#39;;return true;">github.com/*.
 > C:\cygwin64\bin\git.exe fetch --tags --force --progress [hidden email]:*/*.git +refs/heads/*:refs/remotes/origin/* # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from [hidden email]:*/*.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: hudson.plugins.git.GitException: Command "C:\cygwin64\bin\git.exe fetch --tags --force --progress [hidden email]:*/*.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: cannot run D:\Jenkins\workspace\DEBUG_07_git_Checkout@tmp\jenkins-gitclient-ssh1436722960383118000.bat: No such file or directory
fatal: unable to fork

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2318)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1905)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:81)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:488)
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:892)
	... 11 more
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: Error fetching remote repo 'origin'
Finished: FAILURE

This is essentially identical to the failure we're seeing for all jobs. I assume it's normal behaviour that GitSCM generates and
tries to run a random-named bat file? It's unclear to me if the bat file isn't created or is unreadable, although Jenkins is running
as SYSTEM and has generally been able to read and write everything.

Does anyone have any thoughts on what may have gone wrong?

-----

The updates made Friday night:

>>> Artifactory 3.2.2 -> 3.2.4  [ security warning ]                           

>>> Pipeline: API 2.34 -> 2.35                                                 
>>> Pipeline: Basic Steps 2.16 -> 2.18                                         
>>> Pipeline: Declarative 1.3.8 -> 1.3.9                                       
>>> Pipeline: Declarative Extension Points API 1.3.8 -> 1.3.9                  
>>> Pipeline: Groovy 2.69 -> 2.70                                              
>>> Pipeline: Model API 1.3.8 -> 1.3.9                                                                      
>>> Pipeline: Nodes and Processes 2.30 -> 2.31                                                               
>>> Pipeline: SCM Step 2.7 -> 2.9                                                        
>>> Pipeline: Stage Tags Metadata 1.3.8 -> 1.3.9                               
>>> Pipeline: Step API 2.19 -> 2.20                                                      
>>> Slack Notification 2.23 -> 2.24
I've since rolled back all of the "Pipeline: ..." changes.  The problem didn't change.

About 40 Cygwin updates (I didn't enumerate those.)

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/18b9de6c-9c98-455e-b149-9f43b2e4011e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: gitSCM failed after updates

Mark Waite-2


On Tue, Jun 11, 2019 at 4:53 PM Giles <[hidden email]> wrote:
Unfortunately, I appear to have gone from the frying pan to the fire.  I ran one deploy that hadn't worked previously and it did work, and I immediately responded with an "it's working" message.  And that single deploy still works.  But multiple others (all based on the same code, with only varying parameters) don't, all failing with this:

Started by user Giles
java.lang.NoSuchMethodError: org.eclipse.jgit.lib.Repository.getRef(Ljava/lang/String;)Lorg/eclipse/jgit/lib/Ref;
	at jenkins.plugins.git.GitSCMFileSystem$1.invoke(GitSCMFileSystem.java:117)
That may indicate that something has installed a newer version of JGit than JGit 4.5 which is bundled with git client plugin 2.7.x.  JGit 4.5 and prior versions provided the getRef(String) API.  Newer JGit versions (like 4.9, 5.1, 5.2, and 5.3) have deprecated or stopped delivering the getRef(String) API.

Can you confirm that the git client plugin version you're running is 2.7.x and not one of the pre-release 3.0 versions?  If it is not, then you likely need to install the most recent 2.7 version.

Can you confirm that the git plugin version you're running is 3.9.x and not one of the pre-release 4.0 versions?  I expect that is the case since pre-release versions of git plugin 4.0.0 generally do not refer to getRef(String).

That message is unrelated to the version of command line git installed on the master.  That message is related to an internal mismatch between the JGit API that is available in the running Jenkins process and the JGit API that the git plugin requires.  The git plugin is expecting to find JGit 4.5 in the Jenkins process.  It does not seem to be there in this case.

This at least is a known error: it appears to happen when there's a newer version of git, or a misalignment of the git version and the git plugin.  I had initially installed the very new version of Git for Windows 2.22.0.  I downgraded that to 2.17.1.2, and eventually went to 2.21.0 - all resulting in the same failure.  Any thoughts?  It certainly doesn't seem to be the Git version, and the one working pipeline muddies the water further.

On Tuesday, 11 June 2019 17:12:11 UTC-4, Mark Waite wrote:
I'm glad that helped.  In the past, we've seen cases where a new version of ssh surprises the credential integration that is used by the git plugin.  I don't think your case is related to that, but I did see that Git for Windows 2.22.0 now includes OpenSSH 8.0.  I need to add that version into my test environment.  I'm not aware of any breakage with OpenSSH 8.0, but we've been surprised in the past.

On Tue, Jun 11, 2019 at 3:09 PM Giles <[hidden email]> wrote:
Nailed it in one.

I was sure it was a Jenkins issue ... but I'd upgraded Cygwin at the same time, and that probably included their version of git.  Which yes, is my default git.  I changed "Manage Jenkins" -> "Global Tool Configuration" -> git to the just-installed "Git for Windows", and ... fixed.  I may have to make some changes in the Jenkinsfiles themselves, but it seems clear now that the latest Cygwin git broke something, and that Git for Windows fixed it ...

This was crisis-level breakage for my department - I can't thank you enough.

On Tuesday, 11 June 2019 16:36:38 UTC-4, Mark Waite wrote:
Are you quite sure that your configuration did not change in about the same time as that upgrade?

The log file indicates that you're running command line git as provided by Cygwin.  Is that intentional?  Has it always been that way?  

I don't test the Jenkins git plugin with Cygwin.  I test with Git for Windows.  I'm glad cygwin git works, but don't intend to apply effort to make it work.

I don't see anything suspicious or troubling in the list of updates.  As far as I can tell, none of them should have changed that behavior of the git plugin.

On Tue, Jun 11, 2019 at 1:54 PM Giles <[hidden email]> wrote:
I'm running Jenkins 2.164.3 on a Windows server.  It's been running well for several months.  Every Friday evening I do all the Jenkins plugin updates.  After this past Friday's updates (details below), all our jobs are broken - apparently because the GitSCM checkout is broken.  Our Jenkinsfiles are all stored in a git repo, so Jenkins attempts to pull a new version before running the job.  This now fails.  I created a test Jenkinsfile (editing directly in Jenkins rather than in the git repo) with this code in it:

        def scmVars = checkout(
            [
                $class: 'GitSCM',
                branches: [[name: '*/' + branch ]],
                doGenerateSubmoduleConfigurations: false,
                extensions: [],
                submoduleCfg: [],
                userRemoteConfigs: [
                    [
                        credentialsId: jenkinsCred,
                        url: jenkinsRepo,
                    ]
                ]
            ]
        );

(Assume the "jenkinsCred" and "jenkinsRepo" values are correct.)  The failure looks like this:

[Pipeline] {
[Pipeline] checkout
using credential 29d83033-ebf6-4811-9c45-b0aadec775c2
 > C:\cygwin64\bin\git.exe rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > C:\cygwin64\bin\git.exe config remote.origin.url [hidden email]:*/*.git # timeout=10
Fetching upstream changes from [hidden email]:*/*.git
 > C:\cygwin64\bin\git.exe --version # timeout=10
using GIT_SSH to set credentials Read-only access to the "Jenkins" repository at github.com/*.
 > C:\cygwin64\bin\git.exe fetch --tags --force --progress [hidden email]:*/*.git +refs/heads/*:refs/remotes/origin/* # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from [hidden email]:*/*.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: hudson.plugins.git.GitException: Command "C:\cygwin64\bin\git.exe fetch --tags --force --progress [hidden email]:*/*.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: cannot run D:\Jenkins\workspace\DEBUG_07_git_Checkout@tmp\jenkins-gitclient-ssh1436722960383118000.bat: No such file or directory
fatal: unable to fork

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2318)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1905)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:81)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:488)
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:892)
	... 11 more
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: Error fetching remote repo 'origin'
Finished: FAILURE

This is essentially identical to the failure we're seeing for all jobs. I assume it's normal behaviour that GitSCM generates and
tries to run a random-named bat file? It's unclear to me if the bat file isn't created or is unreadable, although Jenkins is running
as SYSTEM and has generally been able to read and write everything.

Does anyone have any thoughts on what may have gone wrong?

-----

The updates made Friday night:

>>> Artifactory 3.2.2 -> 3.2.4  [ security warning ]                           

>>> Pipeline: API 2.34 -> 2.35                                                 
>>> Pipeline: Basic Steps 2.16 -> 2.18                                         
>>> Pipeline: Declarative 1.3.8 -> 1.3.9                                       
>>> Pipeline: Declarative Extension Points API 1.3.8 -> 1.3.9                  
>>> Pipeline: Groovy 2.69 -> 2.70                                              
>>> Pipeline: Model API 1.3.8 -> 1.3.9                                                                      
>>> Pipeline: Nodes and Processes 2.30 -> 2.31                                                               
>>> Pipeline: SCM Step 2.7 -> 2.9                                                        
>>> Pipeline: Stage Tags Metadata 1.3.8 -> 1.3.9                               
>>> Pipeline: Step API 2.19 -> 2.20                                                      
>>> Slack Notification 2.23 -> 2.24
I've since rolled back all of the "Pipeline: ..." changes.  The problem didn't change.

About 40 Cygwin updates (I didn't enumerate those.)

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/18b9de6c-9c98-455e-b149-9f43b2e4011e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Thanks!
Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtEfNCbPbx85enK_y7gD2B%3D%3DcbeTE%3DVtEMLZLuSJz7MeTg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: gitSCM failed after updates

Giles
Hi Mark.

Correct again:

- Git plugin 3.10.0, now downgraded to 3.9.1
- Git client plugin 3.0.0-rc, now downgraded to 2.7.6

Deploys do now appear to all be working (this time I waited until we'd done a fair bit of testing before reporting ...).

I upgraded the above plugins to 4.0.0-rc and 3.0.0-rc back in February because there were security advisories: I don't usually upgrade major versions immediately (especially not RC), instead waiting for them to have time to stabilize.

These downgrades means that I'm now getting a fair bit of messaging from Jenkins about things I should upgrade or change:

    Dependency errors:

    Some plugins could not be loaded due to unsatisfied dependencies. Fix these issues and restart Jenkins to restore the functionality provided by these plugins.

    Git Parameter Plug-In version 0.9.10
    Jenkins Git plugin version 3.9.1 is older than required. To fix, install version 3.9.3 or later.
    GitHub Branch Source Plugin version 2.5.3
    Jenkins Git plugin version 3.9.1 is older than required. To fix, install version 3.9.3 or later.

    Warnings have been published for the following currently installed components.
    Git plugin 3.9.1
    CSRF vulnerability

Given that everything is working, I take it I should just ignore these for now?  Or possibly remove "Git Parameter Plug-In" and "GitHub Branch Source Plugin" as we apparently don't use them at all ...

Once again, thank you very much for your help.


On Tuesday, 11 June 2019 19:02:56 UTC-4, Mark Waite wrote:


On Tue, Jun 11, 2019 at 4:53 PM Giles <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="7-AdlZGoBgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">gile...@...> wrote:
Unfortunately, I appear to have gone from the frying pan to the fire.  I ran one deploy that hadn't worked previously and it did work, and I immediately responded with an "it's working" message.  And that single deploy still works.  But multiple others (all based on the same code, with only varying parameters) don't, all failing with this:

Started by user Giles
java.lang.NoSuchMethodError: org.eclipse.jgit.lib.Repository.getRef(Ljava/lang/String;)Lorg/eclipse/jgit/lib/Ref;
	at jenkins.plugins.git.GitSCMFileSystem$1.invoke(GitSCMFileSystem.java:117)
That may indicate that something has installed a newer version of JGit than JGit 4.5 which is bundled with git client plugin 2.7.x.  JGit 4.5 and prior versions provided the getRef(String) API.  Newer JGit versions (like 4.9, 5.1, 5.2, and 5.3) have deprecated or stopped delivering the getRef(String) API.

Can you confirm that the git client plugin version you're running is 2.7.x and not one of the pre-release 3.0 versions?  If it is not, then you likely need to install the most recent 2.7 version.

Can you confirm that the git plugin version you're running is 3.9.x and not one of the pre-release 4.0 versions?  I expect that is the case since pre-release versions of git plugin 4.0.0 generally do not refer to getRef(String).

That message is unrelated to the version of command line git installed on the master.  That message is related to an internal mismatch between the JGit API that is available in the running Jenkins process and the JGit API that the git plugin requires.  The git plugin is expecting to find JGit 4.5 in the Jenkins process.  It does not seem to be there in this case.

This at least is a known error: it appears to happen when there's a newer version of git, or a misalignment of the git version and the git plugin.  I had initially installed the very new version of Git for Windows 2.22.0.  I downgraded that to 2.17.1.2, and eventually went to 2.21.0 - all resulting in the same failure.  Any thoughts?  It certainly doesn't seem to be the Git version, and the one working pipeline muddies the water further.

On Tuesday, 11 June 2019 17:12:11 UTC-4, Mark Waite wrote:
I'm glad that helped.  In the past, we've seen cases where a new version of ssh surprises the credential integration that is used by the git plugin.  I don't think your case is related to that, but I did see that Git for Windows 2.22.0 now includes OpenSSH 8.0.  I need to add that version into my test environment.  I'm not aware of any breakage with OpenSSH 8.0, but we've been surprised in the past.

On Tue, Jun 11, 2019 at 3:09 PM Giles <[hidden email]> wrote:
Nailed it in one.

I was sure it was a Jenkins issue ... but I'd upgraded Cygwin at the same time, and that probably included their version of git.  Which yes, is my default git.  I changed "Manage Jenkins" -> "Global Tool Configuration" -> git to the just-installed "Git for Windows", and ... fixed.  I may have to make some changes in the Jenkinsfiles themselves, but it seems clear now that the latest Cygwin git broke something, and that Git for Windows fixed it ...

This was crisis-level breakage for my department - I can't thank you enough.

On Tuesday, 11 June 2019 16:36:38 UTC-4, Mark Waite wrote:
Are you quite sure that your configuration did not change in about the same time as that upgrade?

The log file indicates that you're running command line git as provided by Cygwin.  Is that intentional?  Has it always been that way?  

I don't test the Jenkins git plugin with Cygwin.  I test with Git for Windows.  I'm glad cygwin git works, but don't intend to apply effort to make it work.

I don't see anything suspicious or troubling in the list of updates.  As far as I can tell, none of them should have changed that behavior of the git plugin.

On Tue, Jun 11, 2019 at 1:54 PM Giles <[hidden email]> wrote:
I'm running Jenkins 2.164.3 on a Windows server.  It's been running well for several months.  Every Friday evening I do all the Jenkins plugin updates.  After this past Friday's updates (details below), all our jobs are broken - apparently because the GitSCM checkout is broken.  Our Jenkinsfiles are all stored in a git repo, so Jenkins attempts to pull a new version before running the job.  This now fails.  I created a test Jenkinsfile (editing directly in Jenkins rather than in the git repo) with this code in it:

        def scmVars = checkout(
            [
                $class: 'GitSCM',
                branches: [[name: '*/' + branch ]],
                doGenerateSubmoduleConfigurations: false,
                extensions: [],
                submoduleCfg: [],
                userRemoteConfigs: [
                    [
                        credentialsId: jenkinsCred,
                        url: jenkinsRepo,
                    ]
                ]
            ]
        );

(Assume the "jenkinsCred" and "jenkinsRepo" values are correct.)  The failure looks like this:

[Pipeline] {
[Pipeline] checkout
using credential 29d83033-ebf6-4811-9c45-b0aadec775c2
 > C:\cygwin64\bin\git.exe rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > C:\cygwin64\bin\git.exe config remote.origin.url [hidden email]:*/*.git # timeout=10
Fetching upstream changes from [hidden email]:*/*.git
 > C:\cygwin64\bin\git.exe --version # timeout=10
using GIT_SSH to set credentials Read-only access to the "Jenkins" repository at <a href="http://github.com/*" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fgithub.com%2F*\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNENYQ-MeB318WtyMw3be6vUrVzp2g&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fgithub.com%2F*\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNENYQ-MeB318WtyMw3be6vUrVzp2g&#39;;return true;">github.com/*.
 > C:\cygwin64\bin\git.exe fetch --tags --force --progress [hidden email]:*/*.git +refs/heads/*:refs/remotes/origin/* # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from [hidden email]:*/*.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: hudson.plugins.git.GitException: Command "C:\cygwin64\bin\git.exe fetch --tags --force --progress [hidden email]:*/*.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: cannot run D:\Jenkins\workspace\DEBUG_07_git_Checkout@tmp\jenkins-gitclient-ssh1436722960383118000.bat: No such file or directory
fatal: unable to fork

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2318)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1905)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:81)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:488)
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:892)
	... 11 more
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: Error fetching remote repo 'origin'
Finished: FAILURE

This is essentially identical to the failure we're seeing for all jobs. I assume it's normal behaviour that GitSCM generates and
tries to run a random-named bat file? It's unclear to me if the bat file isn't created or is unreadable, although Jenkins is running
as SYSTEM and has generally been able to read and write everything.

Does anyone have any thoughts on what may have gone wrong?

-----

The updates made Friday night:

>>> Artifactory 3.2.2 -> 3.2.4  [ security warning ]                           

>>> Pipeline: API 2.34 -> 2.35                                                 
>>> Pipeline: Basic Steps 2.16 -> 2.18                                         
>>> Pipeline: Declarative 1.3.8 -> 1.3.9                                       
>>> Pipeline: Declarative Extension Points API 1.3.8 -> 1.3.9                  
>>> Pipeline: Groovy 2.69 -> 2.70                                              
>>> Pipeline: Model API 1.3.8 -> 1.3.9                                                                      
>>> Pipeline: Nodes and Processes 2.30 -> 2.31                                                               
>>> Pipeline: SCM Step 2.7 -> 2.9                                                        
>>> Pipeline: Stage Tags Metadata 1.3.8 -> 1.3.9                               
>>> Pipeline: Step API 2.19 -> 2.20                                                      
>>> Slack Notification 2.23 -> 2.24
I've since rolled back all of the "Pipeline: ..." changes.  The problem didn't change.

About 40 Cygwin updates (I didn't enumerate those.)

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="7-AdlZGoBgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jenkins...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/18b9de6c-9c98-455e-b149-9f43b2e4011e%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/18b9de6c-9c98-455e-b149-9f43b2e4011e%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/18b9de6c-9c98-455e-b149-9f43b2e4011e%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/18b9de6c-9c98-455e-b149-9f43b2e4011e%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.


--
Thanks!
Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/2769d445-024a-4ffa-a593-b37958466581%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: gitSCM failed after updates

Mark Waite-2


On Wed, Jun 12, 2019 at 9:52 AM Giles <[hidden email]> wrote:
Hi Mark.

Correct again:

- Git plugin 3.10.0, now downgraded to 3.9.1

Git plugin 3.10.0 is a good choice.  Upgrade to it.
 
- Git client plugin 3.0.0-rc, now downgraded to 2.7.6


Git client plugin 3.0.0-rc is a bad choice.  Don't upgrade (yet) to any git client plugin newer than 2.7.x
 
Deploys do now appear to all be working (this time I waited until we'd done a fair bit of testing before reporting ...).

I upgraded the above plugins to 4.0.0-rc and 3.0.0-rc back in February because there were security advisories: I don't usually upgrade major versions immediately (especially not RC), instead waiting for them to have time to stabilize.


I think that you may have misread the security advisory.  No security advisory has been issued that would require installation of a release candidate.  Those two releases (git plugin 4.0.0-rc and git client plugin 3.0.0-rc) have been removed from the update center.  They have serious compatibility problems which have been resolved in later pre-releases of the plugins.
 
These downgrades means that I'm now getting a fair bit of messaging from Jenkins about things I should upgrade or change:

    Dependency errors:

    Some plugins could not be loaded due to unsatisfied dependencies. Fix these issues and restart Jenkins to restore the functionality provided by these plugins.

    Git Parameter Plug-In version 0.9.10
    Jenkins Git plugin version 3.9.1 is older than required. To fix, install version 3.9.3 or later.
    GitHub Branch Source Plugin version 2.5.3
    Jenkins Git plugin version 3.9.1 is older than required. To fix, install version 3.9.3 or later.

    Warnings have been published for the following currently installed components.
    Git plugin 3.9.1
    CSRF vulnerability

Given that everything is working, I take it I should just ignore these for now?  Or possibly remove "Git Parameter Plug-In" and "GitHub Branch Source Plugin" as we apparently don't use them at all ...

Once again, thank you very much for your help.


On Tuesday, 11 June 2019 19:02:56 UTC-4, Mark Waite wrote:


On Tue, Jun 11, 2019 at 4:53 PM Giles <[hidden email]> wrote:
Unfortunately, I appear to have gone from the frying pan to the fire.  I ran one deploy that hadn't worked previously and it did work, and I immediately responded with an "it's working" message.  And that single deploy still works.  But multiple others (all based on the same code, with only varying parameters) don't, all failing with this:

Started by user Giles
java.lang.NoSuchMethodError: org.eclipse.jgit.lib.Repository.getRef(Ljava/lang/String;)Lorg/eclipse/jgit/lib/Ref;
	at jenkins.plugins.git.GitSCMFileSystem$1.invoke(GitSCMFileSystem.java:117)
That may indicate that something has installed a newer version of JGit than JGit 4.5 which is bundled with git client plugin 2.7.x.  JGit 4.5 and prior versions provided the getRef(String) API.  Newer JGit versions (like 4.9, 5.1, 5.2, and 5.3) have deprecated or stopped delivering the getRef(String) API.

Can you confirm that the git client plugin version you're running is 2.7.x and not one of the pre-release 3.0 versions?  If it is not, then you likely need to install the most recent 2.7 version.

Can you confirm that the git plugin version you're running is 3.9.x and not one of the pre-release 4.0 versions?  I expect that is the case since pre-release versions of git plugin 4.0.0 generally do not refer to getRef(String).

That message is unrelated to the version of command line git installed on the master.  That message is related to an internal mismatch between the JGit API that is available in the running Jenkins process and the JGit API that the git plugin requires.  The git plugin is expecting to find JGit 4.5 in the Jenkins process.  It does not seem to be there in this case.

This at least is a known error: it appears to happen when there's a newer version of git, or a misalignment of the git version and the git plugin.  I had initially installed the very new version of Git for Windows 2.22.0.  I downgraded that to 2.17.1.2, and eventually went to 2.21.0 - all resulting in the same failure.  Any thoughts?  It certainly doesn't seem to be the Git version, and the one working pipeline muddies the water further.

On Tuesday, 11 June 2019 17:12:11 UTC-4, Mark Waite wrote:
I'm glad that helped.  In the past, we've seen cases where a new version of ssh surprises the credential integration that is used by the git plugin.  I don't think your case is related to that, but I did see that Git for Windows 2.22.0 now includes OpenSSH 8.0.  I need to add that version into my test environment.  I'm not aware of any breakage with OpenSSH 8.0, but we've been surprised in the past.

On Tue, Jun 11, 2019 at 3:09 PM Giles <[hidden email]> wrote:
Nailed it in one.

I was sure it was a Jenkins issue ... but I'd upgraded Cygwin at the same time, and that probably included their version of git.  Which yes, is my default git.  I changed "Manage Jenkins" -> "Global Tool Configuration" -> git to the just-installed "Git for Windows", and ... fixed.  I may have to make some changes in the Jenkinsfiles themselves, but it seems clear now that the latest Cygwin git broke something, and that Git for Windows fixed it ...

This was crisis-level breakage for my department - I can't thank you enough.

On Tuesday, 11 June 2019 16:36:38 UTC-4, Mark Waite wrote:
Are you quite sure that your configuration did not change in about the same time as that upgrade?

The log file indicates that you're running command line git as provided by Cygwin.  Is that intentional?  Has it always been that way?  

I don't test the Jenkins git plugin with Cygwin.  I test with Git for Windows.  I'm glad cygwin git works, but don't intend to apply effort to make it work.

I don't see anything suspicious or troubling in the list of updates.  As far as I can tell, none of them should have changed that behavior of the git plugin.

On Tue, Jun 11, 2019 at 1:54 PM Giles <[hidden email]> wrote:
I'm running Jenkins 2.164.3 on a Windows server.  It's been running well for several months.  Every Friday evening I do all the Jenkins plugin updates.  After this past Friday's updates (details below), all our jobs are broken - apparently because the GitSCM checkout is broken.  Our Jenkinsfiles are all stored in a git repo, so Jenkins attempts to pull a new version before running the job.  This now fails.  I created a test Jenkinsfile (editing directly in Jenkins rather than in the git repo) with this code in it:

        def scmVars = checkout(
            [
                $class: 'GitSCM',
                branches: [[name: '*/' + branch ]],
                doGenerateSubmoduleConfigurations: false,
                extensions: [],
                submoduleCfg: [],
                userRemoteConfigs: [
                    [
                        credentialsId: jenkinsCred,
                        url: jenkinsRepo,
                    ]
                ]
            ]
        );

(Assume the "jenkinsCred" and "jenkinsRepo" values are correct.)  The failure looks like this:

[Pipeline] {
[Pipeline] checkout
using credential 29d83033-ebf6-4811-9c45-b0aadec775c2
 > C:\cygwin64\bin\git.exe rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > C:\cygwin64\bin\git.exe config remote.origin.url [hidden email]:*/*.git # timeout=10
Fetching upstream changes from [hidden email]:*/*.git
 > C:\cygwin64\bin\git.exe --version # timeout=10
using GIT_SSH to set credentials Read-only access to the "Jenkins" repository at github.com/*.
 > C:\cygwin64\bin\git.exe fetch --tags --force --progress [hidden email]:*/*.git +refs/heads/*:refs/remotes/origin/* # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from [hidden email]:*/*.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: hudson.plugins.git.GitException: Command "C:\cygwin64\bin\git.exe fetch --tags --force --progress [hidden email]:*/*.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: cannot run D:\Jenkins\workspace\DEBUG_07_git_Checkout@tmp\jenkins-gitclient-ssh1436722960383118000.bat: No such file or directory
fatal: unable to fork

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2318)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1905)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:81)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:488)
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:892)
	... 11 more
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: Error fetching remote repo 'origin'
Finished: FAILURE

This is essentially identical to the failure we're seeing for all jobs. I assume it's normal behaviour that GitSCM generates and
tries to run a random-named bat file? It's unclear to me if the bat file isn't created or is unreadable, although Jenkins is running
as SYSTEM and has generally been able to read and write everything.

Does anyone have any thoughts on what may have gone wrong?

-----

The updates made Friday night:

>>> Artifactory 3.2.2 -> 3.2.4  [ security warning ]                           

>>> Pipeline: API 2.34 -> 2.35                                                 
>>> Pipeline: Basic Steps 2.16 -> 2.18                                         
>>> Pipeline: Declarative 1.3.8 -> 1.3.9                                       
>>> Pipeline: Declarative Extension Points API 1.3.8 -> 1.3.9                  
>>> Pipeline: Groovy 2.69 -> 2.70                                              
>>> Pipeline: Model API 1.3.8 -> 1.3.9                                                                      
>>> Pipeline: Nodes and Processes 2.30 -> 2.31                                                               
>>> Pipeline: SCM Step 2.7 -> 2.9                                                        
>>> Pipeline: Stage Tags Metadata 1.3.8 -> 1.3.9                               
>>> Pipeline: Step API 2.19 -> 2.20                                                      
>>> Slack Notification 2.23 -> 2.24
I've since rolled back all of the "Pipeline: ..." changes.  The problem didn't change.

About 40 Cygwin updates (I didn't enumerate those.)

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/18b9de6c-9c98-455e-b149-9f43b2e4011e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Thanks!
Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/2769d445-024a-4ffa-a593-b37958466581%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Thanks!
Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtH%2BVCzfbKvHF6f-yRtJ%2BRhCNrWdYZAK6HqpK_xttLZVvg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.