fatal: git index-pack failed

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

fatal: git index-pack failed

Kevin Stevens
I'm getting a fatal error reported when doing a repository scan (Scan Multibranch Pipeline Now) which was previously working.
I've run the git commands manually from the command line (copied and pasted) and they appear to work correctly, so it seems like a problem with the Jenkins git client plugin.

The scan log is below (repository name and account information modified for security reasons).
I'm not clear if this is a GIT or a Jenkins git client problem. I've cleared the Jenkins workspace and updated to the latest Jenkins plugins (running on Ubuntu 18.04).
Is anyone able to offer help in debugging this issue please?


[Tue Jul 02 09:48:00 BST 2019] Starting branch indexing...
 > git --version # timeout=10
using GIT_ASKPASS to set credentials Jenkins Bitbucket User
 > git ls-remote --symref [hidden email]:my-company-name/my-repository-name.git # timeout=10
 > git rev-parse --is-inside-work-tree # timeout=10
Setting origin to [hidden email]:my-company-name/my-repository-name.git
 > git config remote.origin.url [hidden email]:my-company-name/my-repository-name.git # timeout=10
Fetching & pruning origin...
Listing remote references...
 > git config --get remote.origin.url # timeout=10
 > git --version # timeout=10
using GIT_ASKPASS to set credentials Jenkins Bitbucket User
 > git ls-remote -h [hidden email]:my-company-name/my-repository-name.git # timeout=10
Fetching upstream changes from origin
 > git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials Jenkins Bitbucket User
 > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune
ERROR: [Tue Jul 02 09:48:10 BST 2019] Could not fetch branches from source 498872a6-3888-419f-9d2f-a2baa4520968
hudson.plugins.git.GitException: Command "git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune" returned status code 128:
stdout: 
stderr: remote: Counting objects: 114, done.        
remote: Compressing objects:   0% (1/114)           
remote: Compressing objects:   1% (2/114)           
remote: Compressing objects:   2% (3/114)           
remote: Compressing objects:   3% (4/114)           
remote: Compressing objects:   4% (5/114)           
remote: Compressing objects:   5% (6/114)           
remote: Compressing objects:   6% (7/114)           
remote: Compressing objects:   7% (8/114)           
remote: Compressing objects:   8% (10/114)           
remote: Compressing objects:   9% (11/114)           
remote: Compressing objects:  10% (12/114)           
remote: Compressing objects:  11% (13/114)           
remote: Compressing objects:  12% (14/114)           
remote: Compressing objects:  13% (15/114)           
remote: Compressing objects:  14% (16/114)           
remote: Compressing objects:  15% (18/114)           
remote: Compressing objects:  16% (19/114)           
remote: Compressing objects:  17% (20/114)           
remote: Compressing objects:  18% (21/114)           
remote: Compressing objects:  19% (22/114)           
remote: Compressing objects:  20% (23/114)           
remote: Compressing objects:  21% (24/114)           
remote: Compressing objects:  22% (26/114)           
remote: Compressing objects:  23% (27/114)           
remote: Compressing objects:  24% (28/114)           
remote: Compressing objects:  25% (29/114)           
remote: Compressing objects:  26% (30/114)           
remote: Compressing objects:  27% (31/114)           
remote: Compressing objects:  28% (32/114)           
remote: Compressing objects:  29% (34/114)           
remote: Compressing objects:  30% (35/114)           
remote: Compressing objects:  31% (36/114)           
remote: Compressing objects:  32% (37/114)           
remote: Compressing objects:  33% (38/114)           
remote: Compressing objects:  34% (39/114)           
remote: Compressing objects:  35% (40/114)           
remote: Compressing objects:  36% (42/114)           
remote: Compressing objects:  37% (43/114)           
remote: Compressing objects:  38% (44/114)           
remote: Compressing objects:  39% (45/114)           
remote: Compressing objects:  40% (46/114)           
remote: Compressing objects:  41% (47/114)           
remote: Compressing objects:  42% (48/114)           
remote: Compressing objects:  43% (50/114)           
remote: Compressing objects:  44% (51/114)           
remote: Compressing objects:  45% (52/114)           
remote: Compressing objects:  46% (53/114)           
remote: Compressing objects:  47% (54/114)           
remote: Compressing objects:  48% (55/114)           
remote: Compressing objects:  49% (56/114)           
remote: Compressing objects:  50% (57/114)           
remote: Compressing objects:  51% (59/114)           
remote: Compressing objects:  52% (60/114)           
remote: Compressing objects:  53% (61/114)           
remote: Compressing objects:  54% (62/114)           
remote: Compressing objects:  55% (63/114)           
remote: Compressing objects:  56% (64/114)           
remote: Compressing objects:  57% (65/114)           
remote: Compressing objects:  58% (67/114)           
remote: Compressing objects:  59% (68/114)           
remote: Compressing objects:  60% (69/114)           
remote: Compressing objects:  61% (70/114)           
remote: Compressing objects:  62% (71/114)           
remote: Compressing objects:  63% (72/114)           
remote: Compressing objects:  64% (73/114)           
remote: Compressing objects:  65% (75/114)           
remote: Compressing objects:  66% (76/114)           
remote: Compressing objects:  67% (77/114)           
remote: Compressing objects:  68% (78/114)           
remote: Compressing objects:  69% (79/114)           
remote: Compressing objects:  70% (80/114)           
remote: Compressing objects:  71% (81/114)           
remote: Compressing objects:  72% (83/114)           
remote: Compressing objects:  73% (84/114)           
remote: Compressing objects:  74% (85/114)           
remote: Compressing objects:  75% (86/114)           
remote: Compressing objects:  76% (87/114)           
remote: Compressing objects:  77% (88/114)           
remote: Compressing objects:  78% (89/114)           
remote: Compressing objects:  79% (91/114)           
remote: Compressing objects:  80% (92/114)           
remote: Compressing objects:  81% (93/114)           
remote: Compressing objects:  82% (94/114)           
remote: Compressing objects:  83% (95/114)           
remote: Compressing objects:  84% (96/114)           
remote: Compressing objects:  85% (97/114)           
remote: Compressing objects:  86% (99/114)           
remote: Compressing objects:  87% (100/114)           
remote: Compressing objects:  88% (101/114)           
remote: Compressing objects:  89% (102/114)           
remote: Compressing objects:  90% (103/114)           
remote: Compressing objects:  91% (104/114)           
remote: Compressing objects:  92% (105/114)           
remote: Compressing objects:  93% (107/114)           
remote: Compressing objects:  94% (108/114)           
remote: Compressing objects:  95% (109/114)           
remote: Compressing objects:  96% (110/114)           
remote: Compressing objects:  97% (111/114)           
remote: Compressing objects:  98% (112/114)           
remote: Compressing objects:  99% (113/114)           
remote: Compressing objects: 100% (114/114)           
remote: Compressing objects: 100% (114/114), done.        
Receiving objects:   0% (1/114)   
Receiving objects:   1% (2/114)   
Receiving objects:   2% (3/114)   
Receiving objects:   3% (4/114)   
Receiving objects:   4% (5/114)   
Receiving objects:   5% (6/114)   
Receiving objects:   6% (7/114)   
Receiving objects:   7% (8/114)   
Receiving objects:   8% (10/114)   
Receiving objects:   9% (11/114)   
Receiving objects:  10% (12/114)   
Receiving objects:  11% (13/114)   
Receiving objects:  12% (14/114)   
Receiving objects:  13% (15/114)   
Receiving objects:  14% (16/114)   
error: object file .git/objects/21/a28795e9c6fa55f026860e0c2f0b08d1b31611 is empty
fatal: cannot read existing object info 21a28795e9c6fa55f026860e0c2f0b08d1b31611
fatal: index-pack failed

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:72)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:442)
at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:575)
at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:556)
at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:367)
at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:556)
at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283)
at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:634)
at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:164)
at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
[Tue Jul 02 09:48:10 BST 2019] Finished branch indexing. Indexing took 10 sec
FATAL: Failed to recompute children of my-repository-name
hudson.plugins.git.GitException: Command "git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune" returned status code 128:
stdout: 
stderr: remote: Counting objects: 114, done.        
remote: Compressing objects:   0% (1/114)           
remote: Compressing objects:   1% (2/114)           
remote: Compressing objects:   2% (3/114)           
remote: Compressing objects:   3% (4/114)           
remote: Compressing objects:   4% (5/114)           
remote: Compressing objects:   5% (6/114)           
remote: Compressing objects:   6% (7/114)           
remote: Compressing objects:   7% (8/114)           
remote: Compressing objects:   8% (10/114)           
remote: Compressing objects:   9% (11/114)           
remote: Compressing objects:  10% (12/114)           
remote: Compressing objects:  11% (13/114)           
remote: Compressing objects:  12% (14/114)           
remote: Compressing objects:  13% (15/114)           
remote: Compressing objects:  14% (16/114)           
remote: Compressing objects:  15% (18/114)           
remote: Compressing objects:  16% (19/114)           
remote: Compressing objects:  17% (20/114)           
remote: Compressing objects:  18% (21/114)           
remote: Compressing objects:  19% (22/114)           
remote: Compressing objects:  20% (23/114)           
remote: Compressing objects:  21% (24/114)           
remote: Compressing objects:  22% (26/114)           
remote: Compressing objects:  23% (27/114)           
remote: Compressing objects:  24% (28/114)           
remote: Compressing objects:  25% (29/114)           
remote: Compressing objects:  26% (30/114)           
remote: Compressing objects:  27% (31/114)           
remote: Compressing objects:  28% (32/114)           
remote: Compressing objects:  29% (34/114)           
remote: Compressing objects:  30% (35/114)           
remote: Compressing objects:  31% (36/114)           
remote: Compressing objects:  32% (37/114)           
remote: Compressing objects:  33% (38/114)           
remote: Compressing objects:  34% (39/114)           
remote: Compressing objects:  35% (40/114)           
remote: Compressing objects:  36% (42/114)           
remote: Compressing objects:  37% (43/114)           
remote: Compressing objects:  38% (44/114)           
remote: Compressing objects:  39% (45/114)           
remote: Compressing objects:  40% (46/114)           
remote: Compressing objects:  41% (47/114)           
remote: Compressing objects:  42% (48/114)           
remote: Compressing objects:  43% (50/114)           
remote: Compressing objects:  44% (51/114)           
remote: Compressing objects:  45% (52/114)           
remote: Compressing objects:  46% (53/114)           
remote: Compressing objects:  47% (54/114)           
remote: Compressing objects:  48% (55/114)           
remote: Compressing objects:  49% (56/114)           
remote: Compressing objects:  50% (57/114)           
remote: Compressing objects:  51% (59/114)           
remote: Compressing objects:  52% (60/114)           
remote: Compressing objects:  53% (61/114)           
remote: Compressing objects:  54% (62/114)           
remote: Compressing objects:  55% (63/114)           
remote: Compressing objects:  56% (64/114)           
remote: Compressing objects:  57% (65/114)           
remote: Compressing objects:  58% (67/114)           
remote: Compressing objects:  59% (68/114)           
remote: Compressing objects:  60% (69/114)           
remote: Compressing objects:  61% (70/114)           
remote: Compressing objects:  62% (71/114)           
remote: Compressing objects:  63% (72/114)           
remote: Compressing objects:  64% (73/114)           
remote: Compressing objects:  65% (75/114)           
remote: Compressing objects:  66% (76/114)           
remote: Compressing objects:  67% (77/114)           
remote: Compressing objects:  68% (78/114)           
remote: Compressing objects:  69% (79/114)           
remote: Compressing objects:  70% (80/114)           
remote: Compressing objects:  71% (81/114)           
remote: Compressing objects:  72% (83/114)           
remote: Compressing objects:  73% (84/114)           
remote: Compressing objects:  74% (85/114)           
remote: Compressing objects:  75% (86/114)           
remote: Compressing objects:  76% (87/114)           
remote: Compressing objects:  77% (88/114)           
remote: Compressing objects:  78% (89/114)           
remote: Compressing objects:  79% (91/114)           
remote: Compressing objects:  80% (92/114)           
remote: Compressing objects:  81% (93/114)           
remote: Compressing objects:  82% (94/114)           
remote: Compressing objects:  83% (95/114)           
remote: Compressing objects:  84% (96/114)           
remote: Compressing objects:  85% (97/114)           
remote: Compressing objects:  86% (99/114)           
remote: Compressing objects:  87% (100/114)           
remote: Compressing objects:  88% (101/114)           
remote: Compressing objects:  89% (102/114)           
remote: Compressing objects:  90% (103/114)           
remote: Compressing objects:  91% (104/114)           
remote: Compressing objects:  92% (105/114)           
remote: Compressing objects:  93% (107/114)           
remote: Compressing objects:  94% (108/114)           
remote: Compressing objects:  95% (109/114)           
remote: Compressing objects:  96% (110/114)           
remote: Compressing objects:  97% (111/114)           
remote: Compressing objects:  98% (112/114)           
remote: Compressing objects:  99% (113/114)           
remote: Compressing objects: 100% (114/114)           
remote: Compressing objects: 100% (114/114), done.        
Receiving objects:   0% (1/114)   
Receiving objects:   1% (2/114)   
Receiving objects:   2% (3/114)   
Receiving objects:   3% (4/114)   
Receiving objects:   4% (5/114)   
Receiving objects:   5% (6/114)   
Receiving objects:   6% (7/114)   
Receiving objects:   7% (8/114)   
Receiving objects:   8% (10/114)   
Receiving objects:   9% (11/114)   
Receiving objects:  10% (12/114)   
Receiving objects:  11% (13/114)   
Receiving objects:  12% (14/114)   
Receiving objects:  13% (15/114)   
Receiving objects:  14% (16/114)   
error: object file .git/objects/21/a28795e9c6fa55f026860e0c2f0b08d1b31611 is empty
fatal: cannot read existing object info 21a28795e9c6fa55f026860e0c2f0b08d1b31611
fatal: index-pack failed

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:72)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:442)
at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:575)
at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:556)
at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:367)
at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:556)
at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283)
at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:634)
at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:164)
at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Finished: FAILURE

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/b6a89f4c-7b2e-441d-ba26-cf46a762b0af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: fatal: git index-pack failed

Mark Waite-2
The repository in the workspace on the agent running that build is probably damaged.  Wipe the workspace on the agent and run the job again.

If wiping the workspace and running the job again does not resolve it, you may also be encountering a timeout while cloning the repository.  Increase the timeout for the repository clone from its default of 10 minutes to something large enough that it will allow you to clone the repository.

The amount of output you're showing hints that the repository is large or the network connection between the agent and the upstream cloned repository is slow.  In either of those cases, you probably want to reduce the clone time and the disc space use by applying one or more of the techniques described in "Git in the Large".  Those techniques include (1) reference repositories on the agent, (2) narrow refspecs, (3) shallow clones, and (4) sparse checkouts.

If this is a Pipeline repository and the failure is during the initial clone of the repository, then you may also need to use lightweight checkout to only checkout the Jenkinsfile rather than the entire repository.  Alternately for Pipelines, if your git provider is GitHub, Bitbucket, or Gitea, you can significantly improve performance by using those branch source plugins to manage the Pipeline instead of relying on low-level command line git calls.

Reference repositories are usually the most effective technique to reduce data transfer time and disc space use for large git repositories.  Allan Burdajewicz of CloudBees wrote a great article on reference repositories at https://support.cloudbees.com/hc/en-us/articles/115001728812-Using-a-Git-reference-repository .

On Tue, Jul 2, 2019 at 5:51 AM Kevin Stevens <[hidden email]> wrote:
I'm getting a fatal error reported when doing a repository scan (Scan Multibranch Pipeline Now) which was previously working.
I've run the git commands manually from the command line (copied and pasted) and they appear to work correctly, so it seems like a problem with the Jenkins git client plugin.

The scan log is below (repository name and account information modified for security reasons).
I'm not clear if this is a GIT or a Jenkins git client problem. I've cleared the Jenkins workspace and updated to the latest Jenkins plugins (running on Ubuntu 18.04).
Is anyone able to offer help in debugging this issue please?


[Tue Jul 02 09:48:00 BST 2019] Starting branch indexing...
 > git --version # timeout=10
using GIT_ASKPASS to set credentials Jenkins Bitbucket User
 > git ls-remote --symref [hidden email]:my-company-name/my-repository-name.git # timeout=10
 > git rev-parse --is-inside-work-tree # timeout=10
Setting origin to [hidden email]:my-company-name/my-repository-name.git
 > git config remote.origin.url [hidden email]:my-company-name/my-repository-name.git # timeout=10
Fetching & pruning origin...
Listing remote references...
 > git config --get remote.origin.url # timeout=10
 > git --version # timeout=10
using GIT_ASKPASS to set credentials Jenkins Bitbucket User
 > git ls-remote -h [hidden email]:my-company-name/my-repository-name.git # timeout=10
Fetching upstream changes from origin
 > git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials Jenkins Bitbucket User
 > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune
ERROR: [Tue Jul 02 09:48:10 BST 2019] Could not fetch branches from source 498872a6-3888-419f-9d2f-a2baa4520968
hudson.plugins.git.GitException: Command "git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune" returned status code 128:
stdout: 
stderr: remote: Counting objects: 114, done.        
remote: Compressing objects:   0% (1/114)           
remote: Compressing objects:   1% (2/114)           
remote: Compressing objects:   2% (3/114)           
remote: Compressing objects:   3% (4/114)           
remote: Compressing objects:   4% (5/114)           
remote: Compressing objects:   5% (6/114)           
remote: Compressing objects:   6% (7/114)           
remote: Compressing objects:   7% (8/114)           
remote: Compressing objects:   8% (10/114)           
remote: Compressing objects:   9% (11/114)           
remote: Compressing objects:  10% (12/114)           
remote: Compressing objects:  11% (13/114)           
remote: Compressing objects:  12% (14/114)           
remote: Compressing objects:  13% (15/114)           
remote: Compressing objects:  14% (16/114)           
remote: Compressing objects:  15% (18/114)           
remote: Compressing objects:  16% (19/114)           
remote: Compressing objects:  17% (20/114)           
remote: Compressing objects:  18% (21/114)           
remote: Compressing objects:  19% (22/114)           
remote: Compressing objects:  20% (23/114)           
remote: Compressing objects:  21% (24/114)           
remote: Compressing objects:  22% (26/114)           
remote: Compressing objects:  23% (27/114)           
remote: Compressing objects:  24% (28/114)           
remote: Compressing objects:  25% (29/114)           
remote: Compressing objects:  26% (30/114)           
remote: Compressing objects:  27% (31/114)           
remote: Compressing objects:  28% (32/114)           
remote: Compressing objects:  29% (34/114)           
remote: Compressing objects:  30% (35/114)           
remote: Compressing objects:  31% (36/114)           
remote: Compressing objects:  32% (37/114)           
remote: Compressing objects:  33% (38/114)           
remote: Compressing objects:  34% (39/114)           
remote: Compressing objects:  35% (40/114)           
remote: Compressing objects:  36% (42/114)           
remote: Compressing objects:  37% (43/114)           
remote: Compressing objects:  38% (44/114)           
remote: Compressing objects:  39% (45/114)           
remote: Compressing objects:  40% (46/114)           
remote: Compressing objects:  41% (47/114)           
remote: Compressing objects:  42% (48/114)           
remote: Compressing objects:  43% (50/114)           
remote: Compressing objects:  44% (51/114)           
remote: Compressing objects:  45% (52/114)           
remote: Compressing objects:  46% (53/114)           
remote: Compressing objects:  47% (54/114)           
remote: Compressing objects:  48% (55/114)           
remote: Compressing objects:  49% (56/114)           
remote: Compressing objects:  50% (57/114)           
remote: Compressing objects:  51% (59/114)           
remote: Compressing objects:  52% (60/114)           
remote: Compressing objects:  53% (61/114)           
remote: Compressing objects:  54% (62/114)           
remote: Compressing objects:  55% (63/114)           
remote: Compressing objects:  56% (64/114)           
remote: Compressing objects:  57% (65/114)           
remote: Compressing objects:  58% (67/114)           
remote: Compressing objects:  59% (68/114)           
remote: Compressing objects:  60% (69/114)           
remote: Compressing objects:  61% (70/114)           
remote: Compressing objects:  62% (71/114)           
remote: Compressing objects:  63% (72/114)           
remote: Compressing objects:  64% (73/114)           
remote: Compressing objects:  65% (75/114)           
remote: Compressing objects:  66% (76/114)           
remote: Compressing objects:  67% (77/114)           
remote: Compressing objects:  68% (78/114)           
remote: Compressing objects:  69% (79/114)           
remote: Compressing objects:  70% (80/114)           
remote: Compressing objects:  71% (81/114)           
remote: Compressing objects:  72% (83/114)           
remote: Compressing objects:  73% (84/114)           
remote: Compressing objects:  74% (85/114)           
remote: Compressing objects:  75% (86/114)           
remote: Compressing objects:  76% (87/114)           
remote: Compressing objects:  77% (88/114)           
remote: Compressing objects:  78% (89/114)           
remote: Compressing objects:  79% (91/114)           
remote: Compressing objects:  80% (92/114)           
remote: Compressing objects:  81% (93/114)           
remote: Compressing objects:  82% (94/114)           
remote: Compressing objects:  83% (95/114)           
remote: Compressing objects:  84% (96/114)           
remote: Compressing objects:  85% (97/114)           
remote: Compressing objects:  86% (99/114)           
remote: Compressing objects:  87% (100/114)           
remote: Compressing objects:  88% (101/114)           
remote: Compressing objects:  89% (102/114)           
remote: Compressing objects:  90% (103/114)           
remote: Compressing objects:  91% (104/114)           
remote: Compressing objects:  92% (105/114)           
remote: Compressing objects:  93% (107/114)           
remote: Compressing objects:  94% (108/114)           
remote: Compressing objects:  95% (109/114)           
remote: Compressing objects:  96% (110/114)           
remote: Compressing objects:  97% (111/114)           
remote: Compressing objects:  98% (112/114)           
remote: Compressing objects:  99% (113/114)           
remote: Compressing objects: 100% (114/114)           
remote: Compressing objects: 100% (114/114), done.        
Receiving objects:   0% (1/114)   
Receiving objects:   1% (2/114)   
Receiving objects:   2% (3/114)   
Receiving objects:   3% (4/114)   
Receiving objects:   4% (5/114)   
Receiving objects:   5% (6/114)   
Receiving objects:   6% (7/114)   
Receiving objects:   7% (8/114)   
Receiving objects:   8% (10/114)   
Receiving objects:   9% (11/114)   
Receiving objects:  10% (12/114)   
Receiving objects:  11% (13/114)   
Receiving objects:  12% (14/114)   
Receiving objects:  13% (15/114)   
Receiving objects:  14% (16/114)   
error: object file .git/objects/21/a28795e9c6fa55f026860e0c2f0b08d1b31611 is empty
fatal: cannot read existing object info 21a28795e9c6fa55f026860e0c2f0b08d1b31611
fatal: index-pack failed

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:72)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:442)
at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:575)
at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:556)
at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:367)
at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:556)
at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283)
at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:634)
at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:164)
at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
[Tue Jul 02 09:48:10 BST 2019] Finished branch indexing. Indexing took 10 sec
FATAL: Failed to recompute children of my-repository-name
hudson.plugins.git.GitException: Command "git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune" returned status code 128:
stdout: 
stderr: remote: Counting objects: 114, done.        
remote: Compressing objects:   0% (1/114)           
remote: Compressing objects:   1% (2/114)           
remote: Compressing objects:   2% (3/114)           
remote: Compressing objects:   3% (4/114)           
remote: Compressing objects:   4% (5/114)           
remote: Compressing objects:   5% (6/114)           
remote: Compressing objects:   6% (7/114)           
remote: Compressing objects:   7% (8/114)           
remote: Compressing objects:   8% (10/114)           
remote: Compressing objects:   9% (11/114)           
remote: Compressing objects:  10% (12/114)           
remote: Compressing objects:  11% (13/114)           
remote: Compressing objects:  12% (14/114)           
remote: Compressing objects:  13% (15/114)           
remote: Compressing objects:  14% (16/114)           
remote: Compressing objects:  15% (18/114)           
remote: Compressing objects:  16% (19/114)           
remote: Compressing objects:  17% (20/114)           
remote: Compressing objects:  18% (21/114)           
remote: Compressing objects:  19% (22/114)           
remote: Compressing objects:  20% (23/114)           
remote: Compressing objects:  21% (24/114)           
remote: Compressing objects:  22% (26/114)           
remote: Compressing objects:  23% (27/114)           
remote: Compressing objects:  24% (28/114)           
remote: Compressing objects:  25% (29/114)           
remote: Compressing objects:  26% (30/114)           
remote: Compressing objects:  27% (31/114)           
remote: Compressing objects:  28% (32/114)           
remote: Compressing objects:  29% (34/114)           
remote: Compressing objects:  30% (35/114)           
remote: Compressing objects:  31% (36/114)           
remote: Compressing objects:  32% (37/114)           
remote: Compressing objects:  33% (38/114)           
remote: Compressing objects:  34% (39/114)           
remote: Compressing objects:  35% (40/114)           
remote: Compressing objects:  36% (42/114)           
remote: Compressing objects:  37% (43/114)           
remote: Compressing objects:  38% (44/114)           
remote: Compressing objects:  39% (45/114)           
remote: Compressing objects:  40% (46/114)           
remote: Compressing objects:  41% (47/114)           
remote: Compressing objects:  42% (48/114)           
remote: Compressing objects:  43% (50/114)           
remote: Compressing objects:  44% (51/114)           
remote: Compressing objects:  45% (52/114)           
remote: Compressing objects:  46% (53/114)           
remote: Compressing objects:  47% (54/114)           
remote: Compressing objects:  48% (55/114)           
remote: Compressing objects:  49% (56/114)           
remote: Compressing objects:  50% (57/114)           
remote: Compressing objects:  51% (59/114)           
remote: Compressing objects:  52% (60/114)           
remote: Compressing objects:  53% (61/114)           
remote: Compressing objects:  54% (62/114)           
remote: Compressing objects:  55% (63/114)           
remote: Compressing objects:  56% (64/114)           
remote: Compressing objects:  57% (65/114)           
remote: Compressing objects:  58% (67/114)           
remote: Compressing objects:  59% (68/114)           
remote: Compressing objects:  60% (69/114)           
remote: Compressing objects:  61% (70/114)           
remote: Compressing objects:  62% (71/114)           
remote: Compressing objects:  63% (72/114)           
remote: Compressing objects:  64% (73/114)           
remote: Compressing objects:  65% (75/114)           
remote: Compressing objects:  66% (76/114)           
remote: Compressing objects:  67% (77/114)           
remote: Compressing objects:  68% (78/114)           
remote: Compressing objects:  69% (79/114)           
remote: Compressing objects:  70% (80/114)           
remote: Compressing objects:  71% (81/114)           
remote: Compressing objects:  72% (83/114)           
remote: Compressing objects:  73% (84/114)           
remote: Compressing objects:  74% (85/114)           
remote: Compressing objects:  75% (86/114)           
remote: Compressing objects:  76% (87/114)           
remote: Compressing objects:  77% (88/114)           
remote: Compressing objects:  78% (89/114)           
remote: Compressing objects:  79% (91/114)           
remote: Compressing objects:  80% (92/114)           
remote: Compressing objects:  81% (93/114)           
remote: Compressing objects:  82% (94/114)           
remote: Compressing objects:  83% (95/114)           
remote: Compressing objects:  84% (96/114)           
remote: Compressing objects:  85% (97/114)           
remote: Compressing objects:  86% (99/114)           
remote: Compressing objects:  87% (100/114)           
remote: Compressing objects:  88% (101/114)           
remote: Compressing objects:  89% (102/114)           
remote: Compressing objects:  90% (103/114)           
remote: Compressing objects:  91% (104/114)           
remote: Compressing objects:  92% (105/114)           
remote: Compressing objects:  93% (107/114)           
remote: Compressing objects:  94% (108/114)           
remote: Compressing objects:  95% (109/114)           
remote: Compressing objects:  96% (110/114)           
remote: Compressing objects:  97% (111/114)           
remote: Compressing objects:  98% (112/114)           
remote: Compressing objects:  99% (113/114)           
remote: Compressing objects: 100% (114/114)           
remote: Compressing objects: 100% (114/114), done.        
Receiving objects:   0% (1/114)   
Receiving objects:   1% (2/114)   
Receiving objects:   2% (3/114)   
Receiving objects:   3% (4/114)   
Receiving objects:   4% (5/114)   
Receiving objects:   5% (6/114)   
Receiving objects:   6% (7/114)   
Receiving objects:   7% (8/114)   
Receiving objects:   8% (10/114)   
Receiving objects:   9% (11/114)   
Receiving objects:  10% (12/114)   
Receiving objects:  11% (13/114)   
Receiving objects:  12% (14/114)   
Receiving objects:  13% (15/114)   
Receiving objects:  14% (16/114)   
error: object file .git/objects/21/a28795e9c6fa55f026860e0c2f0b08d1b31611 is empty
fatal: cannot read existing object info 21a28795e9c6fa55f026860e0c2f0b08d1b31611
fatal: index-pack failed

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:72)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:442)
at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:575)
at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:556)
at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:367)
at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:556)
at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283)
at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:634)
at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:164)
at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Finished: FAILURE

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/b6a89f4c-7b2e-441d-ba26-cf46a762b0af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Thanks!
Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtH0ivhBiPTaWa0B1BeT_9sYaXiNKzDcJShLU_%3DvWP%3DNWA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: fatal: git index-pack failed

Mark Waite-2
Sorry, I didn't read your description thoroughly enough.  You said:

I'm getting a fatal error reported when doing a repository scan (Scan Multibranch Pipeline Now) which was previously working.  

The command line git log output indicates that your repository is large enough that it may be reaching the default 10 minute clone timeout.  Refer to https://issues.jenkins-ci.org/browse/JENKINS-38973 for a discussion of possible workarounds.

On Tue, Jul 2, 2019 at 6:31 AM Mark Waite <[hidden email]> wrote:
The repository in the workspace on the agent running that build is probably damaged.  Wipe the workspace on the agent and run the job again.

If wiping the workspace and running the job again does not resolve it, you may also be encountering a timeout while cloning the repository.  Increase the timeout for the repository clone from its default of 10 minutes to something large enough that it will allow you to clone the repository.

The amount of output you're showing hints that the repository is large or the network connection between the agent and the upstream cloned repository is slow.  In either of those cases, you probably want to reduce the clone time and the disc space use by applying one or more of the techniques described in "Git in the Large".  Those techniques include (1) reference repositories on the agent, (2) narrow refspecs, (3) shallow clones, and (4) sparse checkouts.

If this is a Pipeline repository and the failure is during the initial clone of the repository, then you may also need to use lightweight checkout to only checkout the Jenkinsfile rather than the entire repository.  Alternately for Pipelines, if your git provider is GitHub, Bitbucket, or Gitea, you can significantly improve performance by using those branch source plugins to manage the Pipeline instead of relying on low-level command line git calls.

Reference repositories are usually the most effective technique to reduce data transfer time and disc space use for large git repositories.  Allan Burdajewicz of CloudBees wrote a great article on reference repositories at https://support.cloudbees.com/hc/en-us/articles/115001728812-Using-a-Git-reference-repository .

On Tue, Jul 2, 2019 at 5:51 AM Kevin Stevens <[hidden email]> wrote:
I'm getting a fatal error reported when doing a repository scan (Scan Multibranch Pipeline Now) which was previously working.
I've run the git commands manually from the command line (copied and pasted) and they appear to work correctly, so it seems like a problem with the Jenkins git client plugin.

The scan log is below (repository name and account information modified for security reasons).
I'm not clear if this is a GIT or a Jenkins git client problem. I've cleared the Jenkins workspace and updated to the latest Jenkins plugins (running on Ubuntu 18.04).
Is anyone able to offer help in debugging this issue please?


[Tue Jul 02 09:48:00 BST 2019] Starting branch indexing...
 > git --version # timeout=10
using GIT_ASKPASS to set credentials Jenkins Bitbucket User
 > git ls-remote --symref [hidden email]:my-company-name/my-repository-name.git # timeout=10
 > git rev-parse --is-inside-work-tree # timeout=10
Setting origin to [hidden email]:my-company-name/my-repository-name.git
 > git config remote.origin.url [hidden email]:my-company-name/my-repository-name.git # timeout=10
Fetching & pruning origin...
Listing remote references...
 > git config --get remote.origin.url # timeout=10
 > git --version # timeout=10
using GIT_ASKPASS to set credentials Jenkins Bitbucket User
 > git ls-remote -h [hidden email]:my-company-name/my-repository-name.git # timeout=10
Fetching upstream changes from origin
 > git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials Jenkins Bitbucket User
 > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune
ERROR: [Tue Jul 02 09:48:10 BST 2019] Could not fetch branches from source 498872a6-3888-419f-9d2f-a2baa4520968
hudson.plugins.git.GitException: Command "git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune" returned status code 128:
stdout: 
stderr: remote: Counting objects: 114, done.        
remote: Compressing objects:   0% (1/114)           
remote: Compressing objects:   1% (2/114)           
remote: Compressing objects:   2% (3/114)           
remote: Compressing objects:   3% (4/114)           
remote: Compressing objects:   4% (5/114)           
remote: Compressing objects:   5% (6/114)           
remote: Compressing objects:   6% (7/114)           
remote: Compressing objects:   7% (8/114)           
remote: Compressing objects:   8% (10/114)           
remote: Compressing objects:   9% (11/114)           
remote: Compressing objects:  10% (12/114)           
remote: Compressing objects:  11% (13/114)           
remote: Compressing objects:  12% (14/114)           
remote: Compressing objects:  13% (15/114)           
remote: Compressing objects:  14% (16/114)           
remote: Compressing objects:  15% (18/114)           
remote: Compressing objects:  16% (19/114)           
remote: Compressing objects:  17% (20/114)           
remote: Compressing objects:  18% (21/114)           
remote: Compressing objects:  19% (22/114)           
remote: Compressing objects:  20% (23/114)           
remote: Compressing objects:  21% (24/114)           
remote: Compressing objects:  22% (26/114)           
remote: Compressing objects:  23% (27/114)           
remote: Compressing objects:  24% (28/114)           
remote: Compressing objects:  25% (29/114)           
remote: Compressing objects:  26% (30/114)           
remote: Compressing objects:  27% (31/114)           
remote: Compressing objects:  28% (32/114)           
remote: Compressing objects:  29% (34/114)           
remote: Compressing objects:  30% (35/114)           
remote: Compressing objects:  31% (36/114)           
remote: Compressing objects:  32% (37/114)           
remote: Compressing objects:  33% (38/114)           
remote: Compressing objects:  34% (39/114)           
remote: Compressing objects:  35% (40/114)           
remote: Compressing objects:  36% (42/114)           
remote: Compressing objects:  37% (43/114)           
remote: Compressing objects:  38% (44/114)           
remote: Compressing objects:  39% (45/114)           
remote: Compressing objects:  40% (46/114)           
remote: Compressing objects:  41% (47/114)           
remote: Compressing objects:  42% (48/114)           
remote: Compressing objects:  43% (50/114)           
remote: Compressing objects:  44% (51/114)           
remote: Compressing objects:  45% (52/114)           
remote: Compressing objects:  46% (53/114)           
remote: Compressing objects:  47% (54/114)           
remote: Compressing objects:  48% (55/114)           
remote: Compressing objects:  49% (56/114)           
remote: Compressing objects:  50% (57/114)           
remote: Compressing objects:  51% (59/114)           
remote: Compressing objects:  52% (60/114)           
remote: Compressing objects:  53% (61/114)           
remote: Compressing objects:  54% (62/114)           
remote: Compressing objects:  55% (63/114)           
remote: Compressing objects:  56% (64/114)           
remote: Compressing objects:  57% (65/114)           
remote: Compressing objects:  58% (67/114)           
remote: Compressing objects:  59% (68/114)           
remote: Compressing objects:  60% (69/114)           
remote: Compressing objects:  61% (70/114)           
remote: Compressing objects:  62% (71/114)           
remote: Compressing objects:  63% (72/114)           
remote: Compressing objects:  64% (73/114)           
remote: Compressing objects:  65% (75/114)           
remote: Compressing objects:  66% (76/114)           
remote: Compressing objects:  67% (77/114)           
remote: Compressing objects:  68% (78/114)           
remote: Compressing objects:  69% (79/114)           
remote: Compressing objects:  70% (80/114)           
remote: Compressing objects:  71% (81/114)           
remote: Compressing objects:  72% (83/114)           
remote: Compressing objects:  73% (84/114)           
remote: Compressing objects:  74% (85/114)           
remote: Compressing objects:  75% (86/114)           
remote: Compressing objects:  76% (87/114)           
remote: Compressing objects:  77% (88/114)           
remote: Compressing objects:  78% (89/114)           
remote: Compressing objects:  79% (91/114)           
remote: Compressing objects:  80% (92/114)           
remote: Compressing objects:  81% (93/114)           
remote: Compressing objects:  82% (94/114)           
remote: Compressing objects:  83% (95/114)           
remote: Compressing objects:  84% (96/114)           
remote: Compressing objects:  85% (97/114)           
remote: Compressing objects:  86% (99/114)           
remote: Compressing objects:  87% (100/114)           
remote: Compressing objects:  88% (101/114)           
remote: Compressing objects:  89% (102/114)           
remote: Compressing objects:  90% (103/114)           
remote: Compressing objects:  91% (104/114)           
remote: Compressing objects:  92% (105/114)           
remote: Compressing objects:  93% (107/114)           
remote: Compressing objects:  94% (108/114)           
remote: Compressing objects:  95% (109/114)           
remote: Compressing objects:  96% (110/114)           
remote: Compressing objects:  97% (111/114)           
remote: Compressing objects:  98% (112/114)           
remote: Compressing objects:  99% (113/114)           
remote: Compressing objects: 100% (114/114)           
remote: Compressing objects: 100% (114/114), done.        
Receiving objects:   0% (1/114)   
Receiving objects:   1% (2/114)   
Receiving objects:   2% (3/114)   
Receiving objects:   3% (4/114)   
Receiving objects:   4% (5/114)   
Receiving objects:   5% (6/114)   
Receiving objects:   6% (7/114)   
Receiving objects:   7% (8/114)   
Receiving objects:   8% (10/114)   
Receiving objects:   9% (11/114)   
Receiving objects:  10% (12/114)   
Receiving objects:  11% (13/114)   
Receiving objects:  12% (14/114)   
Receiving objects:  13% (15/114)   
Receiving objects:  14% (16/114)   
error: object file .git/objects/21/a28795e9c6fa55f026860e0c2f0b08d1b31611 is empty
fatal: cannot read existing object info 21a28795e9c6fa55f026860e0c2f0b08d1b31611
fatal: index-pack failed

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:72)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:442)
at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:575)
at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:556)
at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:367)
at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:556)
at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283)
at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:634)
at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:164)
at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
[Tue Jul 02 09:48:10 BST 2019] Finished branch indexing. Indexing took 10 sec
FATAL: Failed to recompute children of my-repository-name
hudson.plugins.git.GitException: Command "git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune" returned status code 128:
stdout: 
stderr: remote: Counting objects: 114, done.        
remote: Compressing objects:   0% (1/114)           
remote: Compressing objects:   1% (2/114)           
remote: Compressing objects:   2% (3/114)           
remote: Compressing objects:   3% (4/114)           
remote: Compressing objects:   4% (5/114)           
remote: Compressing objects:   5% (6/114)           
remote: Compressing objects:   6% (7/114)           
remote: Compressing objects:   7% (8/114)           
remote: Compressing objects:   8% (10/114)           
remote: Compressing objects:   9% (11/114)           
remote: Compressing objects:  10% (12/114)           
remote: Compressing objects:  11% (13/114)           
remote: Compressing objects:  12% (14/114)           
remote: Compressing objects:  13% (15/114)           
remote: Compressing objects:  14% (16/114)           
remote: Compressing objects:  15% (18/114)           
remote: Compressing objects:  16% (19/114)           
remote: Compressing objects:  17% (20/114)           
remote: Compressing objects:  18% (21/114)           
remote: Compressing objects:  19% (22/114)           
remote: Compressing objects:  20% (23/114)           
remote: Compressing objects:  21% (24/114)           
remote: Compressing objects:  22% (26/114)           
remote: Compressing objects:  23% (27/114)           
remote: Compressing objects:  24% (28/114)           
remote: Compressing objects:  25% (29/114)           
remote: Compressing objects:  26% (30/114)           
remote: Compressing objects:  27% (31/114)           
remote: Compressing objects:  28% (32/114)           
remote: Compressing objects:  29% (34/114)           
remote: Compressing objects:  30% (35/114)           
remote: Compressing objects:  31% (36/114)           
remote: Compressing objects:  32% (37/114)           
remote: Compressing objects:  33% (38/114)           
remote: Compressing objects:  34% (39/114)           
remote: Compressing objects:  35% (40/114)           
remote: Compressing objects:  36% (42/114)           
remote: Compressing objects:  37% (43/114)           
remote: Compressing objects:  38% (44/114)           
remote: Compressing objects:  39% (45/114)           
remote: Compressing objects:  40% (46/114)           
remote: Compressing objects:  41% (47/114)           
remote: Compressing objects:  42% (48/114)           
remote: Compressing objects:  43% (50/114)           
remote: Compressing objects:  44% (51/114)           
remote: Compressing objects:  45% (52/114)           
remote: Compressing objects:  46% (53/114)           
remote: Compressing objects:  47% (54/114)           
remote: Compressing objects:  48% (55/114)           
remote: Compressing objects:  49% (56/114)           
remote: Compressing objects:  50% (57/114)           
remote: Compressing objects:  51% (59/114)           
remote: Compressing objects:  52% (60/114)           
remote: Compressing objects:  53% (61/114)           
remote: Compressing objects:  54% (62/114)           
remote: Compressing objects:  55% (63/114)           
remote: Compressing objects:  56% (64/114)           
remote: Compressing objects:  57% (65/114)           
remote: Compressing objects:  58% (67/114)           
remote: Compressing objects:  59% (68/114)           
remote: Compressing objects:  60% (69/114)           
remote: Compressing objects:  61% (70/114)           
remote: Compressing objects:  62% (71/114)           
remote: Compressing objects:  63% (72/114)           
remote: Compressing objects:  64% (73/114)           
remote: Compressing objects:  65% (75/114)           
remote: Compressing objects:  66% (76/114)           
remote: Compressing objects:  67% (77/114)           
remote: Compressing objects:  68% (78/114)           
remote: Compressing objects:  69% (79/114)           
remote: Compressing objects:  70% (80/114)           
remote: Compressing objects:  71% (81/114)           
remote: Compressing objects:  72% (83/114)           
remote: Compressing objects:  73% (84/114)           
remote: Compressing objects:  74% (85/114)           
remote: Compressing objects:  75% (86/114)           
remote: Compressing objects:  76% (87/114)           
remote: Compressing objects:  77% (88/114)           
remote: Compressing objects:  78% (89/114)           
remote: Compressing objects:  79% (91/114)           
remote: Compressing objects:  80% (92/114)           
remote: Compressing objects:  81% (93/114)           
remote: Compressing objects:  82% (94/114)           
remote: Compressing objects:  83% (95/114)           
remote: Compressing objects:  84% (96/114)           
remote: Compressing objects:  85% (97/114)           
remote: Compressing objects:  86% (99/114)           
remote: Compressing objects:  87% (100/114)           
remote: Compressing objects:  88% (101/114)           
remote: Compressing objects:  89% (102/114)           
remote: Compressing objects:  90% (103/114)           
remote: Compressing objects:  91% (104/114)           
remote: Compressing objects:  92% (105/114)           
remote: Compressing objects:  93% (107/114)           
remote: Compressing objects:  94% (108/114)           
remote: Compressing objects:  95% (109/114)           
remote: Compressing objects:  96% (110/114)           
remote: Compressing objects:  97% (111/114)           
remote: Compressing objects:  98% (112/114)           
remote: Compressing objects:  99% (113/114)           
remote: Compressing objects: 100% (114/114)           
remote: Compressing objects: 100% (114/114), done.        
Receiving objects:   0% (1/114)   
Receiving objects:   1% (2/114)   
Receiving objects:   2% (3/114)   
Receiving objects:   3% (4/114)   
Receiving objects:   4% (5/114)   
Receiving objects:   5% (6/114)   
Receiving objects:   6% (7/114)   
Receiving objects:   7% (8/114)   
Receiving objects:   8% (10/114)   
Receiving objects:   9% (11/114)   
Receiving objects:  10% (12/114)   
Receiving objects:  11% (13/114)   
Receiving objects:  12% (14/114)   
Receiving objects:  13% (15/114)   
Receiving objects:  14% (16/114)   
error: object file .git/objects/21/a28795e9c6fa55f026860e0c2f0b08d1b31611 is empty
fatal: cannot read existing object info 21a28795e9c6fa55f026860e0c2f0b08d1b31611
fatal: index-pack failed

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:72)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:442)
at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:575)
at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:556)
at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:367)
at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:556)
at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283)
at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:634)
at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:164)
at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Finished: FAILURE

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/b6a89f4c-7b2e-441d-ba26-cf46a762b0af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Thanks!
Mark Waite


--
Thanks!
Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtGU4_fn5cteSsvcZ-R0V0D1K6dvkYRKmeUHLOC4au-PiQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: fatal: git index-pack failed

Kevin Stevens
Thanks for the responses and the issue link Mark, I've increased the Git timeout value as described, and confirmed in the log that the 30 minute timeout is in effect, but I'm still seeing the same failure (i.e. fatal error in Scan Multibranch Pipeline Now).
I was pretty confident that there's not a timeout issue as the failure occurs in a matter of seconds, so I'm going to try the reference repo approach which seems like it should work, but it would appear to be an issue within the git client parsing to me. I'll confirm my outcome.

Regards, Kevin

On Tuesday, 2 July 2019 13:37:41 UTC+1, Mark Waite wrote:
Sorry, I didn't read your description thoroughly enough.  You said:

I'm getting a fatal error reported when doing a repository scan (Scan Multibranch Pipeline Now) which was previously working.  

The command line git log output indicates that your repository is large enough that it may be reaching the default 10 minute clone timeout.  Refer to <a href="https://issues.jenkins-ci.org/browse/JENKINS-38973" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fissues.jenkins-ci.org%2Fbrowse%2FJENKINS-38973\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGT0rAgy_bxeoGUnnPvVTq0wYfZ9g&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fissues.jenkins-ci.org%2Fbrowse%2FJENKINS-38973\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGT0rAgy_bxeoGUnnPvVTq0wYfZ9g&#39;;return true;">https://issues.jenkins-ci.org/browse/JENKINS-38973 for a discussion of possible workarounds.

On Tue, Jul 2, 2019 at 6:31 AM Mark Waite <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="GpagSbmjAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">mark.e...@...> wrote:
The repository in the workspace on the agent running that build is probably damaged.  Wipe the workspace on the agent and run the job again.

If wiping the workspace and running the job again does not resolve it, you may also be encountering a timeout while cloning the repository.  Increase the timeout for the repository clone from its default of 10 minutes to something large enough that it will allow you to clone the repository.

The amount of output you're showing hints that the repository is large or the network connection between the agent and the upstream cloned repository is slow.  In either of those cases, you probably want to reduce the clone time and the disc space use by applying one or more of the techniques described in "<a href="https://youtu.be/jBGFjFc6Jf8?t=6434" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://youtu.be/jBGFjFc6Jf8?t\x3d6434&#39;;return true;" onclick="this.href=&#39;https://youtu.be/jBGFjFc6Jf8?t\x3d6434&#39;;return true;">Git in the Large".  Those techniques include (1) reference repositories on the agent, (2) narrow refspecs, (3) shallow clones, and (4) sparse checkouts.

If this is a Pipeline repository and the failure is during the initial clone of the repository, then you may also need to use lightweight checkout to only checkout the Jenkinsfile rather than the entire repository.  Alternately for Pipelines, if your git provider is GitHub, Bitbucket, or Gitea, you can significantly improve performance by using those branch source plugins to manage the Pipeline instead of relying on low-level command line git calls.

Reference repositories are usually the most effective technique to reduce data transfer time and disc space use for large git repositories.  Allan Burdajewicz of CloudBees wrote a great article on reference repositories at <a href="https://support.cloudbees.com/hc/en-us/articles/115001728812-Using-a-Git-reference-repository" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F115001728812-Using-a-Git-reference-repository\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE84i-XL-Vvi7bZyZMHxuYI3PIVkQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F115001728812-Using-a-Git-reference-repository\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE84i-XL-Vvi7bZyZMHxuYI3PIVkQ&#39;;return true;">https://support.cloudbees.com/hc/en-us/articles/115001728812-Using-a-Git-reference-repository .

On Tue, Jul 2, 2019 at 5:51 AM Kevin Stevens <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="GpagSbmjAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">kevs...@...> wrote:
I'm getting a fatal error reported when doing a repository scan (Scan Multibranch Pipeline Now) which was previously working.
I've run the git commands manually from the command line (copied and pasted) and they appear to work correctly, so it seems like a problem with the Jenkins git client plugin.

The scan log is below (repository name and account information modified for security reasons).
I'm not clear if this is a GIT or a Jenkins git client problem. I've cleared the Jenkins workspace and updated to the latest Jenkins plugins (running on Ubuntu 18.04).
Is anyone able to offer help in debugging this issue please?


[Tue Jul 02 09:48:00 BST 2019] Starting branch indexing...
 > git --version # timeout=10
using GIT_ASKPASS to set credentials Jenkins Bitbucket User
 > git ls-remote --symref [hidden email]:my-company-name/my-repository-name.git # timeout=10
 > git rev-parse --is-inside-work-tree # timeout=10
Setting origin to [hidden email]:my-company-name/my-repository-name.git
 > git config remote.origin.url [hidden email]:my-company-name/my-repository-name.git # timeout=10
Fetching & pruning origin...
Listing remote references...
 > git config --get remote.origin.url # timeout=10
 > git --version # timeout=10
using GIT_ASKPASS to set credentials Jenkins Bitbucket User
 > git ls-remote -h [hidden email]:my-company-name/my-repository-name.git # timeout=10
Fetching upstream changes from origin
 > git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials Jenkins Bitbucket User
 > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune
ERROR: [Tue Jul 02 09:48:10 BST 2019] Could not fetch branches from source 498872a6-3888-419f-9d2f-a2baa4520968
hudson.plugins.git.GitException: Command "git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune" returned status code 128:
stdout: 
stderr: remote: Counting objects: 114, done.        
remote: Compressing objects:   0% (1/114)           
remote: Compressing objects:   1% (2/114)           
remote: Compressing objects:   2% (3/114)           
remote: Compressing objects:   3% (4/114)           
remote: Compressing objects:   4% (5/114)           
remote: Compressing objects:   5% (6/114)           
remote: Compressing objects:   6% (7/114)           
remote: Compressing objects:   7% (8/114)           
remote: Compressing objects:   8% (10/114)           
remote: Compressing objects:   9% (11/114)           
remote: Compressing objects:  10% (12/114)           
remote: Compressing objects:  11% (13/114)           
remote: Compressing objects:  12% (14/114)           
remote: Compressing objects:  13% (15/114)           
remote: Compressing objects:  14% (16/114)           
remote: Compressing objects:  15% (18/114)           
remote: Compressing objects:  16% (19/114)           
remote: Compressing objects:  17% (20/114)           
remote: Compressing objects:  18% (21/114)           
remote: Compressing objects:  19% (22/114)           
remote: Compressing objects:  20% (23/114)           
remote: Compressing objects:  21% (24/114)           
remote: Compressing objects:  22% (26/114)           
remote: Compressing objects:  23% (27/114)           
remote: Compressing objects:  24% (28/114)           
remote: Compressing objects:  25% (29/114)           
remote: Compressing objects:  26% (30/114)           
remote: Compressing objects:  27% (31/114)           
remote: Compressing objects:  28% (32/114)           
remote: Compressing objects:  29% (34/114)           
remote: Compressing objects:  30% (35/114)           
remote: Compressing objects:  31% (36/114)           
remote: Compressing objects:  32% (37/114)           
remote: Compressing objects:  33% (38/114)           
remote: Compressing objects:  34% (39/114)           
remote: Compressing objects:  35% (40/114)           
remote: Compressing objects:  36% (42/114)           
remote: Compressing objects:  37% (43/114)           
remote: Compressing objects:  38% (44/114)           
remote: Compressing objects:  39% (45/114)           
remote: Compressing objects:  40% (46/114)           
remote: Compressing objects:  41% (47/114)           
remote: Compressing objects:  42% (48/114)           
remote: Compressing objects:  43% (50/114)           
remote: Compressing objects:  44% (51/114)           
remote: Compressing objects:  45% (52/114)           
remote: Compressing objects:  46% (53/114)           
remote: Compressing objects:  47% (54/114)           
remote: Compressing objects:  48% (55/114)           
remote: Compressing objects:  49% (56/114)           
remote: Compressing objects:  50% (57/114)           
remote: Compressing objects:  51% (59/114)           
remote: Compressing objects:  52% (60/114)           
remote: Compressing objects:  53% (61/114)           
remote: Compressing objects:  54% (62/114)           
remote: Compressing objects:  55% (63/114)           
remote: Compressing objects:  56% (64/114)           
remote: Compressing objects:  57% (65/114)           
remote: Compressing objects:  58% (67/114)           
remote: Compressing objects:  59% (68/114)           
remote: Compressing objects:  60% (69/114)           
remote: Compressing objects:  61% (70/114)           
remote: Compressing objects:  62% (71/114)           
remote: Compressing objects:  63% (72/114)           
remote: Compressing objects:  64% (73/114)           
remote: Compressing objects:  65% (75/114)           
remote: Compressing objects:  66% (76/114)           
remote: Compressing objects:  67% (77/114)           
remote: Compressing objects:  68% (78/114)           
remote: Compressing objects:  69% (79/114)           
remote: Compressing objects:  70% (80/114)           
remote: Compressing objects:  71% (81/114)           
remote: Compressing objects:  72% (83/114)           
remote: Compressing objects:  73% (84/114)           
remote: Compressing objects:  74% (85/114)           
remote: Compressing objects:  75% (86/114)           
remote: Compressing objects:  76% (87/114)           
remote: Compressing objects:  77% (88/114)           
remote: Compressing objects:  78% (89/114)           
remote: Compressing objects:  79% (91/114)           
remote: Compressing objects:  80% (92/114)           
remote: Compressing objects:  81% (93/114)           
remote: Compressing objects:  82% (94/114)           
remote: Compressing objects:  83% (95/114)           
remote: Compressing objects:  84% (96/114)           
remote: Compressing objects:  85% (97/114)           
remote: Compressing objects:  86% (99/114)           
remote: Compressing objects:  87% (100/114)           
remote: Compressing objects:  88% (101/114)           
remote: Compressing objects:  89% (102/114)           
remote: Compressing objects:  90% (103/114)           
remote: Compressing objects:  91% (104/114)           
remote: Compressing objects:  92% (105/114)           
remote: Compressing objects:  93% (107/114)           
remote: Compressing objects:  94% (108/114)           
remote: Compressing objects:  95% (109/114)           
remote: Compressing objects:  96% (110/114)           
remote: Compressing objects:  97% (111/114)           
remote: Compressing objects:  98% (112/114)           
remote: Compressing objects:  99% (113/114)           
remote: Compressing objects: 100% (114/114)           
remote: Compressing objects: 100% (114/114), done.        
Receiving objects:   0% (1/114)   
Receiving objects:   1% (2/114)   
Receiving objects:   2% (3/114)   
Receiving objects:   3% (4/114)   
Receiving objects:   4% (5/114)   
Receiving objects:   5% (6/114)   
Receiving objects:   6% (7/114)   
Receiving objects:   7% (8/114)   
Receiving objects:   8% (10/114)   
Receiving objects:   9% (11/114)   
Receiving objects:  10% (12/114)   
Receiving objects:  11% (13/114)   
Receiving objects:  12% (14/114)   
Receiving objects:  13% (15/114)   
Receiving objects:  14% (16/114)   
error: object file .git/objects/21/a28795e9c6fa55f026860e0c2f0b08d1b31611 is empty
fatal: cannot read existing object info 21a28795e9c6fa55f026860e0c2f0b08d1b31611
fatal: index-pack failed

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:72)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:442)
at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:575)
at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:556)
at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:367)
at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:556)
at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283)
at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:634)
at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:164)
at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
[Tue Jul 02 09:48:10 BST 2019] Finished branch indexing. Indexing took 10 sec
FATAL: Failed to recompute children of my-repository-name
hudson.plugins.git.GitException: Command "git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune" returned status code 128:
stdout: 
stderr: remote: Counting objects: 114, done.        
remote: Compressing objects:   0% (1/114)           
remote: Compressing objects:   1% (2/114)           
remote: Compressing objects:   2% (3/114)           
remote: Compressing objects:   3% (4/114)           
remote: Compressing objects:   4% (5/114)           
remote: Compressing objects:   5% (6/114)           
remote: Compressing objects:   6% (7/114)           
remote: Compressing objects:   7% (8/114)           
remote: Compressing objects:   8% (10/114)           
remote: Compressing objects:   9% (11/114)           
remote: Compressing objects:  10% (12/114)           
remote: Compressing objects:  11% (13/114)           
remote: Compressing objects:  12% (14/114)           
remote: Compressing objects:  13% (15/114)           
remote: Compressing objects:  14% (16/114)           
remote: Compressing objects:  15% (18/114)           
remote: Compressing objects:  16% (19/114)           
remote: Compressing objects:  17% (20/114)           
remote: Compressing objects:  18% (21/114)           
remote: Compressing objects:  19% (22/114)           
remote: Compressing objects:  20% (23/114)           
remote: Compressing objects:  21% (24/114)           
remote: Compressing objects:  22% (26/114)           
remote: Compressing objects:  23% (27/114)           
remote: Compressing objects:  24% (28/114)           
remote: Compressing objects:  25% (29/114)           
remote: Compressing objects:  26% (30/114)           
remote: Compressing objects:  27% (31/114)           
remote: Compressing objects:  28% (32/114)           
remote: Compressing objects:  29% (34/114)           
remote: Compressing objects:  30% (35/114)           
remote: Compressing objects:  31% (36/114)           
remote: Compressing objects:  32% (37/114)           
remote: Compressing objects:  33% (38/114)           
remote: Compressing objects:  34% (39/114)           
remote: Compressing objects:  35% (40/114)           
remote: Compressing objects:  36% (42/114)           
remote: Compressing objects:  37% (43/114)           
remote: Compressing objects:  38% (44/114)           
remote: Compressing objects:  39% (45/114)           
remote: Compressing objects:  40% (46/114)           
remote: Compressing objects:  41% (47/114)           
remote: Compressing objects:  42% (48/114)           
remote: Compressing objects:  43% (50/114)           
remote: Compressing objects:  44% (51/114)           
remote: Compressing objects:  45% (52/114)           
remote: Compressing objects:  46% (53/114)           
remote: Compressing objects:  47% (54/114)           
remote: Compressing objects:  48% (55/114)           
remote: Compressing objects:  49% (56/114)           
remote: Compressing objects:  50% (57/114)           
remote: Compressing objects:  51% (59/114)           
remote: Compressing objects:  52% (60/114)           
remote: Compressing objects:  53% (61/114)           
remote: Compressing objects:  54% (62/114)           
remote: Compressing objects:  55% (63/114)           
remote: Compressing objects:  56% (64/114)           
remote: Compressing objects:  57% (65/114)           
remote: Compressing objects:  58% (67/114)           
remote: Compressing objects:  59% (68/114)           
remote: Compressing objects:  60% (69/114)           
remote: Compressing objects:  61% (70/114)           
remote: Compressing objects:  62% (71/114)           
remote: Compressing objects:  63% (72/114)           
remote: Compressing objects:  64% (73/114)           
remote: Compressing objects:  65% (75/114)           
remote: Compressing objects:  66% (76/114)           
remote: Compressing objects:  67% (77/114)           
remote: Compressing objects:  68% (78/114)           
remote: Compressing objects:  69% (79/114)           
remote: Compressing objects:  70% (80/114)           
remote: Compressing objects:  71% (81/114)           
remote: Compressing objects:  72% (83/114)           
remote: Compressing objects:  73% (84/114)           
remote: Compressing objects:  74% (85/114)           
remote: Compressing objects:  75% (86/114)           
remote: Compressing objects:  76% (87/114)           
remote: Compressing objects:  77% (88/114)           
remote: Compressing objects:  78% (89/114)           
remote: Compressing objects:  79% (91/114)           
remote: Compressing objects:  80% (92/114)           
remote: Compressing objects:  81% (93/114)           
remote: Compressing objects:  82% (94/114)           
remote: Compressing objects:  83% (95/114)           
remote: Compressing objects:  84% (96/114)           
remote: Compressing objects:  85% (97/114)           
remote: Compressing objects:  86% (99/114)           
remote: Compressing objects:  87% (100/114)           
remote: Compressing objects:  88% (101/114)           
remote: Compressing objects:  89% (102/114)           
remote: Compressing objects:  90% (103/114)           
remote: Compressing objects:  91% (104/114)           
remote: Compressing objects:  92% (105/114)           
remote: Compressing objects:  93% (107/114)           
remote: Compressing objects:  94% (108/114)           
remote: Compressing objects:  95% (109/114)           
remote: Compressing objects:  96% (110/114)           
remote: Compressing objects:  97% (111/114)           
remote: Compressing objects:  98% (112/114)           
remote: Compressing objects:  99% (113/114)           
remote: Compressing objects: 100% (114/114)           
remote: Compressing objects: 100% (114/114), done.        
Receiving objects:   0% (1/114)   
Receiving objects:   1% (2/114)   
Receiving objects:   2% (3/114)   
Receiving objects:   3% (4/114)   
Receiving objects:   4% (5/114)   
Receiving objects:   5% (6/114)   
Receiving objects:   6% (7/114)   
Receiving objects:   7% (8/114)   
Receiving objects:   8% (10/114)   
Receiving objects:   9% (11/114)   
Receiving objects:  10% (12/114)   
Receiving objects:  11% (13/114)   
Receiving objects:  12% (14/114)   
Receiving objects:  13% (15/114)   
Receiving objects:  14% (16/114)   
error: object file .git/objects/21/a28795e9c6fa55f026860e0c2f0b08d1b31611 is empty
fatal: cannot read existing object info 21a28795e9c6fa55f026860e0c2f0b08d1b31611
fatal: index-pack failed

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:72)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:442)
at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:575)
at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:556)
at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:367)
at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:556)
at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283)
at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:634)
at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:164)
at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Finished: FAILURE

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="GpagSbmjAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jenkins...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/b6a89f4c-7b2e-441d-ba26-cf46a762b0af%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/b6a89f4c-7b2e-441d-ba26-cf46a762b0af%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/b6a89f4c-7b2e-441d-ba26-cf46a762b0af%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/b6a89f4c-7b2e-441d-ba26-cf46a762b0af%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.


--
Thanks!
Mark Waite


--
Thanks!
Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/183327b0-b243-4c31-bbca-8a843651492a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: fatal: git index-pack failed

Kevin Stevens
Hi Mark - whilst looking for a place to stash my reference repo I discovered that Jenkins is keeping a cache of git checkouts in the /var/lib/jenkins/cache folder, and by deleting these files the multibranch pipeline scan now succeeds, so there was obviously something corrupt in that cache (possibly caused by a recent instability issue with the Jenkins server hardware), so I've not implemented a reference repo.
Thanks for your help ;-)

Kevin

On Tuesday, 2 July 2019 15:25:41 UTC+1, Kevin Stevens wrote:
Thanks for the responses and the issue link Mark, I've increased the Git timeout value as described, and confirmed in the log that the 30 minute timeout is in effect, but I'm still seeing the same failure (i.e. fatal error in Scan Multibranch Pipeline Now).
I was pretty confident that there's not a timeout issue as the failure occurs in a matter of seconds, so I'm going to try the reference repo approach which seems like it should work, but it would appear to be an issue within the git client parsing to me. I'll confirm my outcome.

Regards, Kevin

On Tuesday, 2 July 2019 13:37:41 UTC+1, Mark Waite wrote:
Sorry, I didn't read your description thoroughly enough.  You said:

I'm getting a fatal error reported when doing a repository scan (Scan Multibranch Pipeline Now) which was previously working.  

The command line git log output indicates that your repository is large enough that it may be reaching the default 10 minute clone timeout.  Refer to <a href="https://issues.jenkins-ci.org/browse/JENKINS-38973" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fissues.jenkins-ci.org%2Fbrowse%2FJENKINS-38973\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGT0rAgy_bxeoGUnnPvVTq0wYfZ9g&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fissues.jenkins-ci.org%2Fbrowse%2FJENKINS-38973\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGT0rAgy_bxeoGUnnPvVTq0wYfZ9g&#39;;return true;">https://issues.jenkins-ci.org/browse/JENKINS-38973 for a discussion of possible workarounds.

On Tue, Jul 2, 2019 at 6:31 AM Mark Waite <[hidden email]> wrote:
The repository in the workspace on the agent running that build is probably damaged.  Wipe the workspace on the agent and run the job again.

If wiping the workspace and running the job again does not resolve it, you may also be encountering a timeout while cloning the repository.  Increase the timeout for the repository clone from its default of 10 minutes to something large enough that it will allow you to clone the repository.

The amount of output you're showing hints that the repository is large or the network connection between the agent and the upstream cloned repository is slow.  In either of those cases, you probably want to reduce the clone time and the disc space use by applying one or more of the techniques described in "<a href="https://youtu.be/jBGFjFc6Jf8?t=6434" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://youtu.be/jBGFjFc6Jf8?t\x3d6434&#39;;return true;" onclick="this.href=&#39;https://youtu.be/jBGFjFc6Jf8?t\x3d6434&#39;;return true;">Git in the Large".  Those techniques include (1) reference repositories on the agent, (2) narrow refspecs, (3) shallow clones, and (4) sparse checkouts.

If this is a Pipeline repository and the failure is during the initial clone of the repository, then you may also need to use lightweight checkout to only checkout the Jenkinsfile rather than the entire repository.  Alternately for Pipelines, if your git provider is GitHub, Bitbucket, or Gitea, you can significantly improve performance by using those branch source plugins to manage the Pipeline instead of relying on low-level command line git calls.

Reference repositories are usually the most effective technique to reduce data transfer time and disc space use for large git repositories.  Allan Burdajewicz of CloudBees wrote a great article on reference repositories at <a href="https://support.cloudbees.com/hc/en-us/articles/115001728812-Using-a-Git-reference-repository" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F115001728812-Using-a-Git-reference-repository\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE84i-XL-Vvi7bZyZMHxuYI3PIVkQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F115001728812-Using-a-Git-reference-repository\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE84i-XL-Vvi7bZyZMHxuYI3PIVkQ&#39;;return true;">https://support.cloudbees.com/hc/en-us/articles/115001728812-Using-a-Git-reference-repository .

On Tue, Jul 2, 2019 at 5:51 AM Kevin Stevens <[hidden email]> wrote:
I'm getting a fatal error reported when doing a repository scan (Scan Multibranch Pipeline Now) which was previously working.
I've run the git commands manually from the command line (copied and pasted) and they appear to work correctly, so it seems like a problem with the Jenkins git client plugin.

The scan log is below (repository name and account information modified for security reasons).
I'm not clear if this is a GIT or a Jenkins git client problem. I've cleared the Jenkins workspace and updated to the latest Jenkins plugins (running on Ubuntu 18.04).
Is anyone able to offer help in debugging this issue please?


[Tue Jul 02 09:48:00 BST 2019] Starting branch indexing...
 > git --version # timeout=10
using GIT_ASKPASS to set credentials Jenkins Bitbucket User
 > git ls-remote --symref [hidden email]:my-company-name/my-repository-name.git # timeout=10
 > git rev-parse --is-inside-work-tree # timeout=10
Setting origin to [hidden email]:my-company-name/my-repository-name.git
 > git config remote.origin.url [hidden email]:my-company-name/my-repository-name.git # timeout=10
Fetching & pruning origin...
Listing remote references...
 > git config --get remote.origin.url # timeout=10
 > git --version # timeout=10
using GIT_ASKPASS to set credentials Jenkins Bitbucket User
 > git ls-remote -h [hidden email]:my-company-name/my-repository-name.git # timeout=10
Fetching upstream changes from origin
 > git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials Jenkins Bitbucket User
 > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune
ERROR: [Tue Jul 02 09:48:10 BST 2019] Could not fetch branches from source 498872a6-3888-419f-9d2f-a2baa4520968
hudson.plugins.git.GitException: Command "git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune" returned status code 128:
stdout: 
stderr: remote: Counting objects: 114, done.        
remote: Compressing objects:   0% (1/114)           
remote: Compressing objects:   1% (2/114)           
remote: Compressing objects:   2% (3/114)           
remote: Compressing objects:   3% (4/114)           
remote: Compressing objects:   4% (5/114)           
remote: Compressing objects:   5% (6/114)           
remote: Compressing objects:   6% (7/114)           
remote: Compressing objects:   7% (8/114)           
remote: Compressing objects:   8% (10/114)           
remote: Compressing objects:   9% (11/114)           
remote: Compressing objects:  10% (12/114)           
remote: Compressing objects:  11% (13/114)           
remote: Compressing objects:  12% (14/114)           
remote: Compressing objects:  13% (15/114)           
remote: Compressing objects:  14% (16/114)           
remote: Compressing objects:  15% (18/114)           
remote: Compressing objects:  16% (19/114)           
remote: Compressing objects:  17% (20/114)           
remote: Compressing objects:  18% (21/114)           
remote: Compressing objects:  19% (22/114)           
remote: Compressing objects:  20% (23/114)           
remote: Compressing objects:  21% (24/114)           
remote: Compressing objects:  22% (26/114)           
remote: Compressing objects:  23% (27/114)           
remote: Compressing objects:  24% (28/114)           
remote: Compressing objects:  25% (29/114)           
remote: Compressing objects:  26% (30/114)           
remote: Compressing objects:  27% (31/114)           
remote: Compressing objects:  28% (32/114)           
remote: Compressing objects:  29% (34/114)           
remote: Compressing objects:  30% (35/114)           
remote: Compressing objects:  31% (36/114)           
remote: Compressing objects:  32% (37/114)           
remote: Compressing objects:  33% (38/114)           
remote: Compressing objects:  34% (39/114)           
remote: Compressing objects:  35% (40/114)           
remote: Compressing objects:  36% (42/114)           
remote: Compressing objects:  37% (43/114)           
remote: Compressing objects:  38% (44/114)           
remote: Compressing objects:  39% (45/114)           
remote: Compressing objects:  40% (46/114)           
remote: Compressing objects:  41% (47/114)           
remote: Compressing objects:  42% (48/114)           
remote: Compressing objects:  43% (50/114)           
remote: Compressing objects:  44% (51/114)           
remote: Compressing objects:  45% (52/114)           
remote: Compressing objects:  46% (53/114)           
remote: Compressing objects:  47% (54/114)           
remote: Compressing objects:  48% (55/114)           
remote: Compressing objects:  49% (56/114)           
remote: Compressing objects:  50% (57/114)           
remote: Compressing objects:  51% (59/114)           
remote: Compressing objects:  52% (60/114)           
remote: Compressing objects:  53% (61/114)           
remote: Compressing objects:  54% (62/114)           
remote: Compressing objects:  55% (63/114)           
remote: Compressing objects:  56% (64/114)           
remote: Compressing objects:  57% (65/114)           
remote: Compressing objects:  58% (67/114)           
remote: Compressing objects:  59% (68/114)           
remote: Compressing objects:  60% (69/114)           
remote: Compressing objects:  61% (70/114)           
remote: Compressing objects:  62% (71/114)           
remote: Compressing objects:  63% (72/114)           
remote: Compressing objects:  64% (73/114)           
remote: Compressing objects:  65% (75/114)           
remote: Compressing objects:  66% (76/114)           
remote: Compressing objects:  67% (77/114)           
remote: Compressing objects:  68% (78/114)           
remote: Compressing objects:  69% (79/114)           
remote: Compressing objects:  70% (80/114)           
remote: Compressing objects:  71% (81/114)           
remote: Compressing objects:  72% (83/114)           
remote: Compressing objects:  73% (84/114)           
remote: Compressing objects:  74% (85/114)           
remote: Compressing objects:  75% (86/114)           
remote: Compressing objects:  76% (87/114)           
remote: Compressing objects:  77% (88/114)           
remote: Compressing objects:  78% (89/114)           
remote: Compressing objects:  79% (91/114)           
remote: Compressing objects:  80% (92/114)           
remote: Compressing objects:  81% (93/114)           
remote: Compressing objects:  82% (94/114)           
remote: Compressing objects:  83% (95/114)           
remote: Compressing objects:  84% (96/114)           
remote: Compressing objects:  85% (97/114)           
remote: Compressing objects:  86% (99/114)           
remote: Compressing objects:  87% (100/114)           
remote: Compressing objects:  88% (101/114)           
remote: Compressing objects:  89% (102/114)           
remote: Compressing objects:  90% (103/114)           
remote: Compressing objects:  91% (104/114)           
remote: Compressing objects:  92% (105/114)           
remote: Compressing objects:  93% (107/114)           
remote: Compressing objects:  94% (108/114)           
remote: Compressing objects:  95% (109/114)           
remote: Compressing objects:  96% (110/114)           
remote: Compressing objects:  97% (111/114)           
remote: Compressing objects:  98% (112/114)           
remote: Compressing objects:  99% (113/114)           
remote: Compressing objects: 100% (114/114)           
remote: Compressing objects: 100% (114/114), done.        
Receiving objects:   0% (1/114)   
Receiving objects:   1% (2/114)   
Receiving objects:   2% (3/114)   
Receiving objects:   3% (4/114)   
Receiving objects:   4% (5/114)   
Receiving objects:   5% (6/114)   
Receiving objects:   6% (7/114)   
Receiving objects:   7% (8/114)   
Receiving objects:   8% (10/114)   
Receiving objects:   9% (11/114)   
Receiving objects:  10% (12/114)   
Receiving objects:  11% (13/114)   
Receiving objects:  12% (14/114)   
Receiving objects:  13% (15/114)   
Receiving objects:  14% (16/114)   
error: object file .git/objects/21/a28795e9c6fa55f026860e0c2f0b08d1b31611 is empty
fatal: cannot read existing object info 21a28795e9c6fa55f026860e0c2f0b08d1b31611
fatal: index-pack failed

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:72)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:442)
at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:575)
at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:556)
at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:367)
at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:556)
at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283)
at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:634)
at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:164)
at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
[Tue Jul 02 09:48:10 BST 2019] Finished branch indexing. Indexing took 10 sec
FATAL: Failed to recompute children of my-repository-name
hudson.plugins.git.GitException: Command "git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune" returned status code 128:
stdout: 
stderr: remote: Counting objects: 114, done.        
remote: Compressing objects:   0% (1/114)           
remote: Compressing objects:   1% (2/114)           
remote: Compressing objects:   2% (3/114)           
remote: Compressing objects:   3% (4/114)           
remote: Compressing objects:   4% (5/114)           
remote: Compressing objects:   5% (6/114)           
remote: Compressing objects:   6% (7/114)           
remote: Compressing objects:   7% (8/114)           
remote: Compressing objects:   8% (10/114)           
remote: Compressing objects:   9% (11/114)           
remote: Compressing objects:  10% (12/114)           
remote: Compressing objects:  11% (13/114)           
remote: Compressing objects:  12% (14/114)           
remote: Compressing objects:  13% (15/114)           
remote: Compressing objects:  14% (16/114)           
remote: Compressing objects:  15% (18/114)           
remote: Compressing objects:  16% (19/114)           
remote: Compressing objects:  17% (20/114)           
remote: Compressing objects:  18% (21/114)           
remote: Compressing objects:  19% (22/114)           
remote: Compressing objects:  20% (23/114)           
remote: Compressing objects:  21% (24/114)           
remote: Compressing objects:  22% (26/114)           
remote: Compressing objects:  23% (27/114)           
remote: Compressing objects:  24% (28/114)           
remote: Compressing objects:  25% (29/114)           
remote: Compressing objects:  26% (30/114)           
remote: Compressing objects:  27% (31/114)           
remote: Compressing objects:  28% (32/114)           
remote: Compressing objects:  29% (34/114)           
remote: Compressing objects:  30% (35/114)           
remote: Compressing objects:  31% (36/114)           
remote: Compressing objects:  32% (37/114)           
remote: Compressing objects:  33% (38/114)           
remote: Compressing objects:  34% (39/114)           
remote: Compressing objects:  35% (40/114)           
remote: Compressing objects:  36% (42/114)           
remote: Compressing objects:  37% (43/114)           
remote: Compressing objects:  38% (44/114)           
remote: Compressing objects:  39% (45/114)           
remote: Compressing objects:  40% (46/114)           
remote: Compressing objects:  41% (47/114)           
remote: Compressing objects:  42% (48/114)           
remote: Compressing objects:  43% (50/114)           
remote: Compressing objects:  44% (51/114)           
remote: Compressing objects:  45% (52/114)           
remote: Compressing objects:  46% (53/114)           
remote: Compressing objects:  47% (54/114)           
remote: Compressing objects:  48% (55/114)           
remote: Compressing objects:  49% (56/114)           
remote: Compressing objects:  50% (57/114)           
remote: Compressing objects:  51% (59/114)           
remote: Compressing objects:  52% (60/114)           
remote: Compressing objects:  53% (61/114)           
remote: Compressing objects:  54% (62/114)           
remote: Compressing objects:  55% (63/114)           
remote: Compressing objects:  56% (64/114)           
remote: Compressing objects:  57% (65/114)           
remote: Compressing objects:  58% (67/114)           
remote: Compressing objects:  59% (68/114)           
remote: Compressing objects:  60% (69/114)           
remote: Compressing objects:  61% (70/114)           
remote: Compressing objects:  62% (71/114)           
remote: Compressing objects:  63% (72/114)           
remote: Compressing objects:  64% (73/114)           
remote: Compressing objects:  65% (75/114)           
remote: Compressing objects:  66% (76/114)           
remote: Compressing objects:  67% (77/114)           
remote: Compressing objects:  68% (78/114)           
remote: Compressing objects:  69% (79/114)           
remote: Compressing objects:  70% (80/114)           
remote: Compressing objects:  71% (81/114)           
remote: Compressing objects:  72% (83/114)           
remote: Compressing objects:  73% (84/114)           
remote: Compressing objects:  74% (85/114)           
remote: Compressing objects:  75% (86/114)           
remote: Compressing objects:  76% (87/114)           
remote: Compressing objects:  77% (88/114)           
remote: Compressing objects:  78% (89/114)           
remote: Compressing objects:  79% (91/114)           
remote: Compressing objects:  80% (92/114)           
remote: Compressing objects:  81% (93/114)           
remote: Compressing objects:  82% (94/114)           
remote: Compressing objects:  83% (95/114)           
remote: Compressing objects:  84% (96/114)           
remote: Compressing objects:  85% (97/114)           
remote: Compressing objects:  86% (99/114)           
remote: Compressing objects:  87% (100/114)           
remote: Compressing objects:  88% (101/114)           
remote: Compressing objects:  89% (102/114)           
remote: Compressing objects:  90% (103/114)           
remote: Compressing objects:  91% (104/114)           
remote: Compressing objects:  92% (105/114)           
remote: Compressing objects:  93% (107/114)           
remote: Compressing objects:  94% (108/114)           
remote: Compressing objects:  95% (109/114)           
remote: Compressing objects:  96% (110/114)           
remote: Compressing objects:  97% (111/114)           
remote: Compressing objects:  98% (112/114)           
remote: Compressing objects:  99% (113/114)           
remote: Compressing objects: 100% (114/114)           
remote: Compressing objects: 100% (114/114), done.        
Receiving objects:   0% (1/114)   
Receiving objects:   1% (2/114)   
Receiving objects:   2% (3/114)   
Receiving objects:   3% (4/114)   
Receiving objects:   4% (5/114)   
Receiving objects:   5% (6/114)   
Receiving objects:   6% (7/114)   
Receiving objects:   7% (8/114)   
Receiving objects:   8% (10/114)   
Receiving objects:   9% (11/114)   
Receiving objects:  10% (12/114)   
Receiving objects:  11% (13/114)   
Receiving objects:  12% (14/114)   
Receiving objects:  13% (15/114)   
Receiving objects:  14% (16/114)   
error: object file .git/objects/21/a28795e9c6fa55f026860e0c2f0b08d1b31611 is empty
fatal: cannot read existing object info 21a28795e9c6fa55f026860e0c2f0b08d1b31611
fatal: index-pack failed

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:72)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:442)
at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:575)
at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:556)
at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:367)
at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:556)
at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283)
at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:634)
at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:164)
at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Finished: FAILURE

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/b6a89f4c-7b2e-441d-ba26-cf46a762b0af%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/b6a89f4c-7b2e-441d-ba26-cf46a762b0af%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/b6a89f4c-7b2e-441d-ba26-cf46a762b0af%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/b6a89f4c-7b2e-441d-ba26-cf46a762b0af%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.


--
Thanks!
Mark Waite


--
Thanks!
Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/b30a6241-1427-46bf-b8a6-34a42b939608%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: fatal: git index-pack failed

Mark Waite-2
That's great news.  A git repository that needs more than 10 minutes to clone is a really good candidate for reference repositories and all the other techniques suggested in the "Git in the Large" video.  The first preference with repositories that large is to reduce their size by removing large files from the repository.  Unfortunately, most organizations can't do that.  A previous organization where I worked had a 25 GB repository that just kept growing due to large binary checkins.  Git large file storage (LFS) is another lifesaver to avoid creating large repositories.  Good luck!

On Tue, Jul 2, 2019 at 8:49 AM Kevin Stevens <[hidden email]> wrote:
Hi Mark - whilst looking for a place to stash my reference repo I discovered that Jenkins is keeping a cache of git checkouts in the /var/lib/jenkins/cache folder, and by deleting these files the multibranch pipeline scan now succeeds, so there was obviously something corrupt in that cache (possibly caused by a recent instability issue with the Jenkins server hardware), so I've not implemented a reference repo.
Thanks for your help ;-)

Kevin

On Tuesday, 2 July 2019 15:25:41 UTC+1, Kevin Stevens wrote:
Thanks for the responses and the issue link Mark, I've increased the Git timeout value as described, and confirmed in the log that the 30 minute timeout is in effect, but I'm still seeing the same failure (i.e. fatal error in Scan Multibranch Pipeline Now).
I was pretty confident that there's not a timeout issue as the failure occurs in a matter of seconds, so I'm going to try the reference repo approach which seems like it should work, but it would appear to be an issue within the git client parsing to me. I'll confirm my outcome.

Regards, Kevin

On Tuesday, 2 July 2019 13:37:41 UTC+1, Mark Waite wrote:
Sorry, I didn't read your description thoroughly enough.  You said:

I'm getting a fatal error reported when doing a repository scan (Scan Multibranch Pipeline Now) which was previously working.  

The command line git log output indicates that your repository is large enough that it may be reaching the default 10 minute clone timeout.  Refer to https://issues.jenkins-ci.org/browse/JENKINS-38973 for a discussion of possible workarounds.

On Tue, Jul 2, 2019 at 6:31 AM Mark Waite <[hidden email]> wrote:
The repository in the workspace on the agent running that build is probably damaged.  Wipe the workspace on the agent and run the job again.

If wiping the workspace and running the job again does not resolve it, you may also be encountering a timeout while cloning the repository.  Increase the timeout for the repository clone from its default of 10 minutes to something large enough that it will allow you to clone the repository.

The amount of output you're showing hints that the repository is large or the network connection between the agent and the upstream cloned repository is slow.  In either of those cases, you probably want to reduce the clone time and the disc space use by applying one or more of the techniques described in "Git in the Large".  Those techniques include (1) reference repositories on the agent, (2) narrow refspecs, (3) shallow clones, and (4) sparse checkouts.

If this is a Pipeline repository and the failure is during the initial clone of the repository, then you may also need to use lightweight checkout to only checkout the Jenkinsfile rather than the entire repository.  Alternately for Pipelines, if your git provider is GitHub, Bitbucket, or Gitea, you can significantly improve performance by using those branch source plugins to manage the Pipeline instead of relying on low-level command line git calls.

Reference repositories are usually the most effective technique to reduce data transfer time and disc space use for large git repositories.  Allan Burdajewicz of CloudBees wrote a great article on reference repositories at https://support.cloudbees.com/hc/en-us/articles/115001728812-Using-a-Git-reference-repository .

On Tue, Jul 2, 2019 at 5:51 AM Kevin Stevens <[hidden email]> wrote:
I'm getting a fatal error reported when doing a repository scan (Scan Multibranch Pipeline Now) which was previously working.
I've run the git commands manually from the command line (copied and pasted) and they appear to work correctly, so it seems like a problem with the Jenkins git client plugin.

The scan log is below (repository name and account information modified for security reasons).
I'm not clear if this is a GIT or a Jenkins git client problem. I've cleared the Jenkins workspace and updated to the latest Jenkins plugins (running on Ubuntu 18.04).
Is anyone able to offer help in debugging this issue please?


[Tue Jul 02 09:48:00 BST 2019] Starting branch indexing...
 > git --version # timeout=10
using GIT_ASKPASS to set credentials Jenkins Bitbucket User
 > git ls-remote --symref [hidden email]:my-company-name/my-repository-name.git # timeout=10
 > git rev-parse --is-inside-work-tree # timeout=10
Setting origin to [hidden email]:my-company-name/my-repository-name.git
 > git config remote.origin.url [hidden email]:my-company-name/my-repository-name.git # timeout=10
Fetching & pruning origin...
Listing remote references...
 > git config --get remote.origin.url # timeout=10
 > git --version # timeout=10
using GIT_ASKPASS to set credentials Jenkins Bitbucket User
 > git ls-remote -h [hidden email]:my-company-name/my-repository-name.git # timeout=10
Fetching upstream changes from origin
 > git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials Jenkins Bitbucket User
 > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune
ERROR: [Tue Jul 02 09:48:10 BST 2019] Could not fetch branches from source 498872a6-3888-419f-9d2f-a2baa4520968
hudson.plugins.git.GitException: Command "git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune" returned status code 128:
stdout: 
stderr: remote: Counting objects: 114, done.        
remote: Compressing objects:   0% (1/114)           
remote: Compressing objects:   1% (2/114)           
remote: Compressing objects:   2% (3/114)           
remote: Compressing objects:   3% (4/114)           
remote: Compressing objects:   4% (5/114)           
remote: Compressing objects:   5% (6/114)           
remote: Compressing objects:   6% (7/114)           
remote: Compressing objects:   7% (8/114)           
remote: Compressing objects:   8% (10/114)           
remote: Compressing objects:   9% (11/114)           
remote: Compressing objects:  10% (12/114)           
remote: Compressing objects:  11% (13/114)           
remote: Compressing objects:  12% (14/114)           
remote: Compressing objects:  13% (15/114)           
remote: Compressing objects:  14% (16/114)           
remote: Compressing objects:  15% (18/114)           
remote: Compressing objects:  16% (19/114)           
remote: Compressing objects:  17% (20/114)           
remote: Compressing objects:  18% (21/114)           
remote: Compressing objects:  19% (22/114)           
remote: Compressing objects:  20% (23/114)           
remote: Compressing objects:  21% (24/114)           
remote: Compressing objects:  22% (26/114)           
remote: Compressing objects:  23% (27/114)           
remote: Compressing objects:  24% (28/114)           
remote: Compressing objects:  25% (29/114)           
remote: Compressing objects:  26% (30/114)           
remote: Compressing objects:  27% (31/114)           
remote: Compressing objects:  28% (32/114)           
remote: Compressing objects:  29% (34/114)           
remote: Compressing objects:  30% (35/114)           
remote: Compressing objects:  31% (36/114)           
remote: Compressing objects:  32% (37/114)           
remote: Compressing objects:  33% (38/114)           
remote: Compressing objects:  34% (39/114)           
remote: Compressing objects:  35% (40/114)           
remote: Compressing objects:  36% (42/114)           
remote: Compressing objects:  37% (43/114)           
remote: Compressing objects:  38% (44/114)           
remote: Compressing objects:  39% (45/114)           
remote: Compressing objects:  40% (46/114)           
remote: Compressing objects:  41% (47/114)           
remote: Compressing objects:  42% (48/114)           
remote: Compressing objects:  43% (50/114)           
remote: Compressing objects:  44% (51/114)           
remote: Compressing objects:  45% (52/114)           
remote: Compressing objects:  46% (53/114)           
remote: Compressing objects:  47% (54/114)           
remote: Compressing objects:  48% (55/114)           
remote: Compressing objects:  49% (56/114)           
remote: Compressing objects:  50% (57/114)           
remote: Compressing objects:  51% (59/114)           
remote: Compressing objects:  52% (60/114)           
remote: Compressing objects:  53% (61/114)           
remote: Compressing objects:  54% (62/114)           
remote: Compressing objects:  55% (63/114)           
remote: Compressing objects:  56% (64/114)           
remote: Compressing objects:  57% (65/114)           
remote: Compressing objects:  58% (67/114)           
remote: Compressing objects:  59% (68/114)           
remote: Compressing objects:  60% (69/114)           
remote: Compressing objects:  61% (70/114)           
remote: Compressing objects:  62% (71/114)           
remote: Compressing objects:  63% (72/114)           
remote: Compressing objects:  64% (73/114)           
remote: Compressing objects:  65% (75/114)           
remote: Compressing objects:  66% (76/114)           
remote: Compressing objects:  67% (77/114)           
remote: Compressing objects:  68% (78/114)           
remote: Compressing objects:  69% (79/114)           
remote: Compressing objects:  70% (80/114)           
remote: Compressing objects:  71% (81/114)           
remote: Compressing objects:  72% (83/114)           
remote: Compressing objects:  73% (84/114)           
remote: Compressing objects:  74% (85/114)           
remote: Compressing objects:  75% (86/114)           
remote: Compressing objects:  76% (87/114)           
remote: Compressing objects:  77% (88/114)           
remote: Compressing objects:  78% (89/114)           
remote: Compressing objects:  79% (91/114)           
remote: Compressing objects:  80% (92/114)           
remote: Compressing objects:  81% (93/114)           
remote: Compressing objects:  82% (94/114)           
remote: Compressing objects:  83% (95/114)           
remote: Compressing objects:  84% (96/114)           
remote: Compressing objects:  85% (97/114)           
remote: Compressing objects:  86% (99/114)           
remote: Compressing objects:  87% (100/114)           
remote: Compressing objects:  88% (101/114)           
remote: Compressing objects:  89% (102/114)           
remote: Compressing objects:  90% (103/114)           
remote: Compressing objects:  91% (104/114)           
remote: Compressing objects:  92% (105/114)           
remote: Compressing objects:  93% (107/114)           
remote: Compressing objects:  94% (108/114)           
remote: Compressing objects:  95% (109/114)           
remote: Compressing objects:  96% (110/114)           
remote: Compressing objects:  97% (111/114)           
remote: Compressing objects:  98% (112/114)           
remote: Compressing objects:  99% (113/114)           
remote: Compressing objects: 100% (114/114)           
remote: Compressing objects: 100% (114/114), done.        
Receiving objects:   0% (1/114)   
Receiving objects:   1% (2/114)   
Receiving objects:   2% (3/114)   
Receiving objects:   3% (4/114)   
Receiving objects:   4% (5/114)   
Receiving objects:   5% (6/114)   
Receiving objects:   6% (7/114)   
Receiving objects:   7% (8/114)   
Receiving objects:   8% (10/114)   
Receiving objects:   9% (11/114)   
Receiving objects:  10% (12/114)   
Receiving objects:  11% (13/114)   
Receiving objects:  12% (14/114)   
Receiving objects:  13% (15/114)   
Receiving objects:  14% (16/114)   
error: object file .git/objects/21/a28795e9c6fa55f026860e0c2f0b08d1b31611 is empty
fatal: cannot read existing object info 21a28795e9c6fa55f026860e0c2f0b08d1b31611
fatal: index-pack failed

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:72)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:442)
at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:575)
at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:556)
at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:367)
at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:556)
at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283)
at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:634)
at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:164)
at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
[Tue Jul 02 09:48:10 BST 2019] Finished branch indexing. Indexing took 10 sec
FATAL: Failed to recompute children of my-repository-name
hudson.plugins.git.GitException: Command "git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune" returned status code 128:
stdout: 
stderr: remote: Counting objects: 114, done.        
remote: Compressing objects:   0% (1/114)           
remote: Compressing objects:   1% (2/114)           
remote: Compressing objects:   2% (3/114)           
remote: Compressing objects:   3% (4/114)           
remote: Compressing objects:   4% (5/114)           
remote: Compressing objects:   5% (6/114)           
remote: Compressing objects:   6% (7/114)           
remote: Compressing objects:   7% (8/114)           
remote: Compressing objects:   8% (10/114)           
remote: Compressing objects:   9% (11/114)           
remote: Compressing objects:  10% (12/114)           
remote: Compressing objects:  11% (13/114)           
remote: Compressing objects:  12% (14/114)           
remote: Compressing objects:  13% (15/114)           
remote: Compressing objects:  14% (16/114)           
remote: Compressing objects:  15% (18/114)           
remote: Compressing objects:  16% (19/114)           
remote: Compressing objects:  17% (20/114)           
remote: Compressing objects:  18% (21/114)           
remote: Compressing objects:  19% (22/114)           
remote: Compressing objects:  20% (23/114)           
remote: Compressing objects:  21% (24/114)           
remote: Compressing objects:  22% (26/114)           
remote: Compressing objects:  23% (27/114)           
remote: Compressing objects:  24% (28/114)           
remote: Compressing objects:  25% (29/114)           
remote: Compressing objects:  26% (30/114)           
remote: Compressing objects:  27% (31/114)           
remote: Compressing objects:  28% (32/114)           
remote: Compressing objects:  29% (34/114)           
remote: Compressing objects:  30% (35/114)           
remote: Compressing objects:  31% (36/114)           
remote: Compressing objects:  32% (37/114)           
remote: Compressing objects:  33% (38/114)           
remote: Compressing objects:  34% (39/114)           
remote: Compressing objects:  35% (40/114)           
remote: Compressing objects:  36% (42/114)           
remote: Compressing objects:  37% (43/114)           
remote: Compressing objects:  38% (44/114)           
remote: Compressing objects:  39% (45/114)           
remote: Compressing objects:  40% (46/114)           
remote: Compressing objects:  41% (47/114)           
remote: Compressing objects:  42% (48/114)           
remote: Compressing objects:  43% (50/114)           
remote: Compressing objects:  44% (51/114)           
remote: Compressing objects:  45% (52/114)           
remote: Compressing objects:  46% (53/114)           
remote: Compressing objects:  47% (54/114)           
remote: Compressing objects:  48% (55/114)           
remote: Compressing objects:  49% (56/114)           
remote: Compressing objects:  50% (57/114)           
remote: Compressing objects:  51% (59/114)           
remote: Compressing objects:  52% (60/114)           
remote: Compressing objects:  53% (61/114)           
remote: Compressing objects:  54% (62/114)           
remote: Compressing objects:  55% (63/114)           
remote: Compressing objects:  56% (64/114)           
remote: Compressing objects:  57% (65/114)           
remote: Compressing objects:  58% (67/114)           
remote: Compressing objects:  59% (68/114)           
remote: Compressing objects:  60% (69/114)           
remote: Compressing objects:  61% (70/114)           
remote: Compressing objects:  62% (71/114)           
remote: Compressing objects:  63% (72/114)           
remote: Compressing objects:  64% (73/114)           
remote: Compressing objects:  65% (75/114)           
remote: Compressing objects:  66% (76/114)           
remote: Compressing objects:  67% (77/114)           
remote: Compressing objects:  68% (78/114)           
remote: Compressing objects:  69% (79/114)           
remote: Compressing objects:  70% (80/114)           
remote: Compressing objects:  71% (81/114)           
remote: Compressing objects:  72% (83/114)           
remote: Compressing objects:  73% (84/114)           
remote: Compressing objects:  74% (85/114)           
remote: Compressing objects:  75% (86/114)           
remote: Compressing objects:  76% (87/114)           
remote: Compressing objects:  77% (88/114)           
remote: Compressing objects:  78% (89/114)           
remote: Compressing objects:  79% (91/114)           
remote: Compressing objects:  80% (92/114)           
remote: Compressing objects:  81% (93/114)           
remote: Compressing objects:  82% (94/114)           
remote: Compressing objects:  83% (95/114)           
remote: Compressing objects:  84% (96/114)           
remote: Compressing objects:  85% (97/114)           
remote: Compressing objects:  86% (99/114)           
remote: Compressing objects:  87% (100/114)           
remote: Compressing objects:  88% (101/114)           
remote: Compressing objects:  89% (102/114)           
remote: Compressing objects:  90% (103/114)           
remote: Compressing objects:  91% (104/114)           
remote: Compressing objects:  92% (105/114)           
remote: Compressing objects:  93% (107/114)           
remote: Compressing objects:  94% (108/114)           
remote: Compressing objects:  95% (109/114)           
remote: Compressing objects:  96% (110/114)           
remote: Compressing objects:  97% (111/114)           
remote: Compressing objects:  98% (112/114)           
remote: Compressing objects:  99% (113/114)           
remote: Compressing objects: 100% (114/114)           
remote: Compressing objects: 100% (114/114), done.        
Receiving objects:   0% (1/114)   
Receiving objects:   1% (2/114)   
Receiving objects:   2% (3/114)   
Receiving objects:   3% (4/114)   
Receiving objects:   4% (5/114)   
Receiving objects:   5% (6/114)   
Receiving objects:   6% (7/114)   
Receiving objects:   7% (8/114)   
Receiving objects:   8% (10/114)   
Receiving objects:   9% (11/114)   
Receiving objects:  10% (12/114)   
Receiving objects:  11% (13/114)   
Receiving objects:  12% (14/114)   
Receiving objects:  13% (15/114)   
Receiving objects:  14% (16/114)   
error: object file .git/objects/21/a28795e9c6fa55f026860e0c2f0b08d1b31611 is empty
fatal: cannot read existing object info 21a28795e9c6fa55f026860e0c2f0b08d1b31611
fatal: index-pack failed

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:72)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:442)
at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:575)
at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:556)
at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:367)
at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:556)
at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283)
at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:634)
at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:164)
at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Finished: FAILURE

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/b6a89f4c-7b2e-441d-ba26-cf46a762b0af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Thanks!
Mark Waite


--
Thanks!
Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/b30a6241-1427-46bf-b8a6-34a42b939608%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Thanks!
Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtG538t%2BAJLTa8KTheRWKjK%3DhoCs-Y0Z5k1uh0cDgY1hUw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: fatal: git index-pack failed

Hormoz Zarnani
On Tuesday, July 2, 2019 at 8:38:46 AM UTC-7, Mark Waite wrote:

> That's great news.  A git repository that needs more than 10 minutes to clone is a really good candidate for reference repositories and all the other techniques suggested in the "Git in the Large" video.  The first preference with repositories that large is to reduce their size by removing large files from the repository.  Unfortunately, most organizations can't do that.  A previous organization where I worked had a 25 GB repository that just kept growing due to large binary checkins.  Git large file storage (LFS) is another lifesaver to avoid creating large repositories.  Good luck!
>
>
> On Tue, Jul 2, 2019 at 8:49 AM Kevin Stevens <[hidden email]> wrote:
>
> Hi Mark - whilst looking for a place to stash my reference repo I discovered that Jenkins is keeping a cache of git checkouts in the /var/lib/jenkins/cache folder, and by deleting these files the multibranch pipeline scan now succeeds, so there was obviously something corrupt in that cache (possibly caused by a recent instability issue with the Jenkins server hardware), so I've not implemented a reference repo.
> Thanks for your help ;-)
>
>
> Kevin
>
> On Tuesday, 2 July 2019 15:25:41 UTC+1, Kevin Stevens  wrote:
> Thanks for the responses and the issue link Mark, I've increased the Git timeout value as described, and confirmed in the log that the 30 minute timeout is in effect, but I'm still seeing the same failure (i.e. fatal error in Scan Multibranch Pipeline Now).
> I was pretty confident that there's not a timeout issue as the failure occurs in a matter of seconds, so I'm going to try the reference repo approach which seems like it should work, but it would appear to be an issue within the git client parsing to me. I'll confirm my outcome.
>
>
> Regards, Kevin
>
> On Tuesday, 2 July 2019 13:37:41 UTC+1, Mark Waite  wrote:
> Sorry, I didn't read your description thoroughly enough.  You said:
>
> I'm getting a fatal error reported when doing a repository scan (Scan Multibranch Pipeline Now) which was previously working.  
>
>
>
> The command line git log output indicates that your repository is large enough that it may be reaching the default 10 minute clone timeout.  Refer to https://issues.jenkins-ci.org/browse/JENKINS-38973 for a discussion of possible workarounds.
>
>
> On Tue, Jul 2, 2019 at 6:31 AM Mark Waite <[hidden email]> wrote:
>
> The repository in the workspace on the agent running that build is probably damaged.  Wipe the workspace on the agent and run the job again.
>
>
> If wiping the workspace and running the job again does not resolve it, you may also be encountering a timeout while cloning the repository.  Increase the timeout for the repository clone from its default of 10 minutes to something large enough that it will allow you to clone the repository.
>
>
>
> The amount of output you're showing hints that the repository is large or the network connection between the agent and the upstream cloned repository is slow.  In either of those cases, you probably want to reduce the clone time and the disc space use by applying one or more of the techniques described in "Git in the Large".  Those techniques include (1) reference repositories on the agent, (2) narrow refspecs, (3) shallow clones, and (4) sparse checkouts.
>
>
> If this is a Pipeline repository and the failure is during the initial clone of the repository, then you may also need to use lightweight checkout to only checkout the Jenkinsfile rather than the entire repository.  Alternately for Pipelines, if your git provider is GitHub, Bitbucket, or Gitea, you can significantly improve performance by using those branch source plugins to manage the Pipeline instead of relying on low-level command line git calls.
>
> Reference repositories are usually the most effective technique to reduce data transfer time and disc space use for large git repositories.  Allan Burdajewicz of CloudBees wrote a great article on reference repositories at https://support.cloudbees.com/hc/en-us/articles/115001728812-Using-a-Git-reference-repository .
>
>
> On Tue, Jul 2, 2019 at 5:51 AM Kevin Stevens <[hidden email]> wrote:
>
> I'm getting a fatal error reported when doing a repository scan (Scan Multibranch Pipeline Now) which was previously working.
> I've run the git commands manually from the command line (copied and pasted) and they appear to work correctly, so it seems like a problem with the Jenkins git client plugin.
>
>
> The scan log is below (repository name and account information modified for security reasons).
> I'm not clear if this is a GIT or a Jenkins git client problem. I've cleared the Jenkins workspace and updated to the latest Jenkins plugins (running on Ubuntu 18.04).
> Is anyone able to offer help in debugging this issue please?
>
>
>
>
>
> [Tue Jul 02 09:48:00 BST 2019] Starting branch indexing...
>  > git --version # timeout=10
> using GIT_ASKPASS to set credentials Jenkins Bitbucket User
>  > git ls-remote --symref [hidden email]:my-company-name/my-repository-name.git # timeout=10
>  > git rev-parse --is-inside-work-tree # timeout=10
> Setting origin to [hidden email]:my-company-name/my-repository-name.git
>  > git config remote.origin.url [hidden email]:my-company-name/my-repository-name.git # timeout=10
> Fetching & pruning origin...
> Listing remote references...
>  > git config --get remote.origin.url # timeout=10
>  > git --version # timeout=10
> using GIT_ASKPASS to set credentials Jenkins Bitbucket User
>  > git ls-remote -h [hidden email]:my-company-name/my-repository-name.git # timeout=10
> Fetching upstream changes from origin
>  > git config --get remote.origin.url # timeout=10
> using GIT_ASKPASS to set credentials Jenkins Bitbucket User
>  > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune
> ERROR: [Tue Jul 02 09:48:10 BST 2019] Could not fetch branches from source 498872a6-3888-419f-9d2f-a2baa4520968
> hudson.plugins.git.GitException: Command "git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune" returned status code 128:
> stdout: 
> stderr: remote: Counting objects: 114, done.        
> remote: Compressing objects:   0% (1/114)           
> remote: Compressing objects:   1% (2/114)           
> remote: Compressing objects:   2% (3/114)           
> remote: Compressing objects:   3% (4/114)           
> remote: Compressing objects:   4% (5/114)           
> remote: Compressing objects:   5% (6/114)           
> remote: Compressing objects:   6% (7/114)           
> remote: Compressing objects:   7% (8/114)           
> remote: Compressing objects:   8% (10/114)           
> remote: Compressing objects:   9% (11/114)           
> remote: Compressing objects:  10% (12/114)           
> remote: Compressing objects:  11% (13/114)           
> remote: Compressing objects:  12% (14/114)           
> remote: Compressing objects:  13% (15/114)           
> remote: Compressing objects:  14% (16/114)           
> remote: Compressing objects:  15% (18/114)           
> remote: Compressing objects:  16% (19/114)           
> remote: Compressing objects:  17% (20/114)           
> remote: Compressing objects:  18% (21/114)           
> remote: Compressing objects:  19% (22/114)           
> remote: Compressing objects:  20% (23/114)           
> remote: Compressing objects:  21% (24/114)           
> remote: Compressing objects:  22% (26/114)           
> remote: Compressing objects:  23% (27/114)           
> remote: Compressing objects:  24% (28/114)           
> remote: Compressing objects:  25% (29/114)           
> remote: Compressing objects:  26% (30/114)           
> remote: Compressing objects:  27% (31/114)           
> remote: Compressing objects:  28% (32/114)           
> remote: Compressing objects:  29% (34/114)           
> remote: Compressing objects:  30% (35/114)           
> remote: Compressing objects:  31% (36/114)           
> remote: Compressing objects:  32% (37/114)           
> remote: Compressing objects:  33% (38/114)           
> remote: Compressing objects:  34% (39/114)           
> remote: Compressing objects:  35% (40/114)           
> remote: Compressing objects:  36% (42/114)           
> remote: Compressing objects:  37% (43/114)           
> remote: Compressing objects:  38% (44/114)           
> remote: Compressing objects:  39% (45/114)           
> remote: Compressing objects:  40% (46/114)           
> remote: Compressing objects:  41% (47/114)           
> remote: Compressing objects:  42% (48/114)           
> remote: Compressing objects:  43% (50/114)           
> remote: Compressing objects:  44% (51/114)           
> remote: Compressing objects:  45% (52/114)           
> remote: Compressing objects:  46% (53/114)           
> remote: Compressing objects:  47% (54/114)           
> remote: Compressing objects:  48% (55/114)           
> remote: Compressing objects:  49% (56/114)           
> remote: Compressing objects:  50% (57/114)           
> remote: Compressing objects:  51% (59/114)           
> remote: Compressing objects:  52% (60/114)           
> remote: Compressing objects:  53% (61/114)           
> remote: Compressing objects:  54% (62/114)           
> remote: Compressing objects:  55% (63/114)           
> remote: Compressing objects:  56% (64/114)           
> remote: Compressing objects:  57% (65/114)           
> remote: Compressing objects:  58% (67/114)           
> remote: Compressing objects:  59% (68/114)           
> remote: Compressing objects:  60% (69/114)           
> remote: Compressing objects:  61% (70/114)           
> remote: Compressing objects:  62% (71/114)           
> remote: Compressing objects:  63% (72/114)           
> remote: Compressing objects:  64% (73/114)           
> remote: Compressing objects:  65% (75/114)           
> remote: Compressing objects:  66% (76/114)           
> remote: Compressing objects:  67% (77/114)           
> remote: Compressing objects:  68% (78/114)           
> remote: Compressing objects:  69% (79/114)           
> remote: Compressing objects:  70% (80/114)           
> remote: Compressing objects:  71% (81/114)           
> remote: Compressing objects:  72% (83/114)           
> remote: Compressing objects:  73% (84/114)           
> remote: Compressing objects:  74% (85/114)           
> remote: Compressing objects:  75% (86/114)           
> remote: Compressing objects:  76% (87/114)           
> remote: Compressing objects:  77% (88/114)           
> remote: Compressing objects:  78% (89/114)           
> remote: Compressing objects:  79% (91/114)           
> remote: Compressing objects:  80% (92/114)           
> remote: Compressing objects:  81% (93/114)           
> remote: Compressing objects:  82% (94/114)           
> remote: Compressing objects:  83% (95/114)           
> remote: Compressing objects:  84% (96/114)           
> remote: Compressing objects:  85% (97/114)           
> remote: Compressing objects:  86% (99/114)           
> remote: Compressing objects:  87% (100/114)           
> remote: Compressing objects:  88% (101/114)           
> remote: Compressing objects:  89% (102/114)           
> remote: Compressing objects:  90% (103/114)           
> remote: Compressing objects:  91% (104/114)           
> remote: Compressing objects:  92% (105/114)           
> remote: Compressing objects:  93% (107/114)           
> remote: Compressing objects:  94% (108/114)           
> remote: Compressing objects:  95% (109/114)           
> remote: Compressing objects:  96% (110/114)           
> remote: Compressing objects:  97% (111/114)           
> remote: Compressing objects:  98% (112/114)           
> remote: Compressing objects:  99% (113/114)           
> remote: Compressing objects: 100% (114/114)           
> remote: Compressing objects: 100% (114/114), done.        
> Receiving objects:   0% (1/114)   
> Receiving objects:   1% (2/114)   
> Receiving objects:   2% (3/114)   
> Receiving objects:   3% (4/114)   
> Receiving objects:   4% (5/114)   
> Receiving objects:   5% (6/114)   
> Receiving objects:   6% (7/114)   
> Receiving objects:   7% (8/114)   
> Receiving objects:   8% (10/114)   
> Receiving objects:   9% (11/114)   
> Receiving objects:  10% (12/114)   
> Receiving objects:  11% (13/114)   
> Receiving objects:  12% (14/114)   
> Receiving objects:  13% (15/114)   
> Receiving objects:  14% (16/114)   
> error: object file .git/objects/21/a28795e9c6fa55f026860e0c2f0b08d1b31611 is empty
> fatal: cannot read existing object info 21a28795e9c6fa55f026860e0c2f0b08d1b31611
> fatal: index-pack failed
>
>
> at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
> at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761)
> at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:72)
> at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:442)
> at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:575)
> at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:556)
> at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:367)
> at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:556)
> at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
> at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283)
> at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:634)
> at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
> at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:164)
> at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025)
> at hudson.model.ResourceController.execute(ResourceController.java:97)
> at hudson.model.Executor.run(Executor.java:429)
> [Tue Jul 02 09:48:10 BST 2019] Finished branch indexing. Indexing took 10 sec
> FATAL: Failed to recompute children of my-repository-name
> hudson.plugins.git.GitException: Command "git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune" returned status code 128:
> stdout: 
> stderr: remote: Counting objects: 114, done.        
> remote: Compressing objects:   0% (1/114)           
> remote: Compressing objects:   1% (2/114)           
> remote: Compressing objects:   2% (3/114)           
> remote: Compressing objects:   3% (4/114)           
> remote: Compressing objects:   4% (5/114)           
> remote: Compressing objects:   5% (6/114)           
> remote: Compressing objects:   6% (7/114)           
> remote: Compressing objects:   7% (8/114)           
> remote: Compressing objects:   8% (10/114)           
> remote: Compressing objects:   9% (11/114)           
> remote: Compressing objects:  10% (12/114)           
> remote: Compressing objects:  11% (13/114)           
> remote: Compressing objects:  12% (14/114)           
> remote: Compressing objects:  13% (15/114)           
> remote: Compressing objects:  14% (16/114)           
> remote: Compressing objects:  15% (18/114)           
> remote: Compressing objects:  16% (19/114)           
> remote: Compressing objects:  17% (20/114)           
> remote: Compressing objects:  18% (21/114)           
> remote: Compressing objects:  19% (22/114)           
> remote: Compressing objects:  20% (23/114)           
> remote: Compressing objects:  21% (24/114)           
> remote: Compressing objects:  22% (26/114)           
> remote: Compressing objects:  23% (27/114)           
> remote: Compressing objects:  24% (28/114)           
> remote: Compressing objects:  25% (29/114)           
> remote: Compressing objects:  26% (30/114)           
> remote: Compressing objects:  27% (31/114)           
> remote: Compressing objects:  28% (32/114)           
> remote: Compressing objects:  29% (34/114)           
> remote: Compressing objects:  30% (35/114)           
> remote: Compressing objects:  31% (36/114)           
> remote: Compressing objects:  32% (37/114)           
> remote: Compressing objects:  33% (38/114)           
> remote: Compressing objects:  34% (39/114)           
> remote: Compressing objects:  35% (40/114)           
> remote: Compressing objects:  36% (42/114)           
> remote: Compressing objects:  37% (43/114)           
> remote: Compressing objects:  38% (44/114)           
> remote: Compressing objects:  39% (45/114)           
> remote: Compressing objects:  40% (46/114)           
> remote: Compressing objects:  41% (47/114)           
> remote: Compressing objects:  42% (48/114)           
> remote: Compressing objects:  43% (50/114)           
> remote: Compressing objects:  44% (51/114)           
> remote: Compressing objects:  45% (52/114)           
> remote: Compressing objects:  46% (53/114)           
> remote: Compressing objects:  47% (54/114)           
> remote: Compressing objects:  48% (55/114)           
> remote: Compressing objects:  49% (56/114)           
> remote: Compressing objects:  50% (57/114)           
> remote: Compressing objects:  51% (59/114)           
> remote: Compressing objects:  52% (60/114)           
> remote: Compressing objects:  53% (61/114)           
> remote: Compressing objects:  54% (62/114)           
> remote: Compressing objects:  55% (63/114)           
> remote: Compressing objects:  56% (64/114)           
> remote: Compressing objects:  57% (65/114)           
> remote: Compressing objects:  58% (67/114)           
> remote: Compressing objects:  59% (68/114)           
> remote: Compressing objects:  60% (69/114)           
> remote: Compressing objects:  61% (70/114)           
> remote: Compressing objects:  62% (71/114)           
> remote: Compressing objects:  63% (72/114)           
> remote: Compressing objects:  64% (73/114)           
> remote: Compressing objects:  65% (75/114)           
> remote: Compressing objects:  66% (76/114)           
> remote: Compressing objects:  67% (77/114)           
> remote: Compressing objects:  68% (78/114)           
> remote: Compressing objects:  69% (79/114)           
> remote: Compressing objects:  70% (80/114)           
> remote: Compressing objects:  71% (81/114)           
> remote: Compressing objects:  72% (83/114)           
> remote: Compressing objects:  73% (84/114)           
> remote: Compressing objects:  74% (85/114)           
> remote: Compressing objects:  75% (86/114)           
> remote: Compressing objects:  76% (87/114)           
> remote: Compressing objects:  77% (88/114)           
> remote: Compressing objects:  78% (89/114)           
> remote: Compressing objects:  79% (91/114)           
> remote: Compressing objects:  80% (92/114)           
> remote: Compressing objects:  81% (93/114)           
> remote: Compressing objects:  82% (94/114)           
> remote: Compressing objects:  83% (95/114)           
> remote: Compressing objects:  84% (96/114)           
> remote: Compressing objects:  85% (97/114)           
> remote: Compressing objects:  86% (99/114)           
> remote: Compressing objects:  87% (100/114)           
> remote: Compressing objects:  88% (101/114)           
> remote: Compressing objects:  89% (102/114)           
> remote: Compressing objects:  90% (103/114)           
> remote: Compressing objects:  91% (104/114)           
> remote: Compressing objects:  92% (105/114)           
> remote: Compressing objects:  93% (107/114)           
> remote: Compressing objects:  94% (108/114)           
> remote: Compressing objects:  95% (109/114)           
> remote: Compressing objects:  96% (110/114)           
> remote: Compressing objects:  97% (111/114)           
> remote: Compressing objects:  98% (112/114)           
> remote: Compressing objects:  99% (113/114)           
> remote: Compressing objects: 100% (114/114)           
> remote: Compressing objects: 100% (114/114), done.        
> Receiving objects:   0% (1/114)   
> Receiving objects:   1% (2/114)   
> Receiving objects:   2% (3/114)   
> Receiving objects:   3% (4/114)   
> Receiving objects:   4% (5/114)   
> Receiving objects:   5% (6/114)   
> Receiving objects:   6% (7/114)   
> Receiving objects:   7% (8/114)   
> Receiving objects:   8% (10/114)   
> Receiving objects:   9% (11/114)   
> Receiving objects:  10% (12/114)   
> Receiving objects:  11% (13/114)   
> Receiving objects:  12% (14/114)   
> Receiving objects:  13% (15/114)   
> Receiving objects:  14% (16/114)   
> error: object file .git/objects/21/a28795e9c6fa55f026860e0c2f0b08d1b31611 is empty
> fatal: cannot read existing object info 21a28795e9c6fa55f026860e0c2f0b08d1b31611
> fatal: index-pack failed
>
>
> at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
> at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761)
> at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:72)
> at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:442)
> at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:575)
> at jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:556)
> at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:367)
> at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:556)
> at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
> at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283)
> at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:634)
> at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
> at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:164)
> at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025)
> at hudson.model.ResourceController.execute(ResourceController.java:97)
> at hudson.model.Executor.run(Executor.java:429)
> Finished: FAILURE
>
>
>
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
>
> To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
>
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/b6a89f4c-7b2e-441d-ba26-cf46a762b0af%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
>
>
> Thanks!
> Mark Waite
>
>
>
> --
>
>
> Thanks!
> Mark Waite
>
>
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
>
> To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
>
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/b30a6241-1427-46bf-b8a6-34a42b939608%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
>
>
> Thanks!
> Mark Waite
Hi.   I am in a similar situation where I have several very large Git repositories, and I want to build multibranch pipelines for each of them, and my reference repositories don’t seem to be used for scanning/indexing.   My issue can’t be solved by a timeout increase.  My issue is storage.  I see in one of the pipelines, which was small enough to be cloned on my Jenkins master, that the reference repository is used for the checkouts in the per-branch workspaces.   But my master  node simply doesn’t have enough writable storage to fetch all objects of more than a dozen very large Git repositories.

Mark Waite, if this is something you know how to fix but prefer to delegate, I’m happy to volunteer.  I’m new to writing Jenkins plugins, but I should be able to help if I get some direction.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/e970d393-e3e6-4add-acd0-8d57144dbc8c%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: fatal: git index-pack failed

Mark Waite-2


On Thu, Jan 16, 2020 at 7:41 PM Hormoz Zarnani <[hidden email]> wrote:

Hi.   I am in a similar situation where I have several very large Git repositories, and I want to build multibranch pipelines for each of them, and my reference repositories don’t seem to be used for scanning/indexing.   My issue can’t be solved by a timeout increase.  My issue is storage.  I see in one of the pipelines, which was small enough to be cloned on my Jenkins master, that the reference repository is used for the checkouts in the per-branch workspaces.   But my master  node simply doesn’t have enough writable storage to fetch all objects of more than a dozen very large Git repositories.

Mark Waite, if this is something you know how to fix but prefer to delegate, I’m happy to volunteer.  I’m new to writing Jenkins plugins, but I should be able to help if I get some direction.


Thanks for the offer!  Unfortunately, I don't know how to fix it.  I haven't attempted to duplicate the bug myself, so I can't confirm with confidence that it is a bug.

You're welcome to download the git plugin source code, create an automated test which shows the issue, then add the changes which fix the test.

If it is not using a reference repository for branch scanning, then that may indicate the branch scanning code is not passing the setting for the reference repository to the git client plugin.  An automated test that confirms the reference repository is not used when scanning the repository will show the problem through a test and will provide a basis for a fix.

In terms of financial cost, you'll probably find it cheaper and faster to purchase additional disc space for your master than to find and fix the issue.  Attempting to save disc space is frequently much more expensive than the price of the disc drives.  I know that's not the answer you were hoping to hear, but considering the price of disc space and the cost of development time, I suspect it is realistic.

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/e970d393-e3e6-4add-acd0-8d57144dbc8c%40googlegroups.com.


--
Thanks!
Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtFwi3%3DwqWYL8HY0RQRfw4bz50XvWJEhBX-OtULEwPAXdw%40mail.gmail.com.