Download the git repo and use it for the downstream job

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Download the git repo and use it for the downstream job

sravan
Hi All,

My git repo is really big and I am running job parallel on two different machine.Sometimes I see an issue with downloading from git . I get this error "failed on download from git" Error.

ERROR: Timeout after 10 minutes
ERROR: Error cloning remote repo 'origin'
returned status code 128:
stdout: 
stderr: remote: Enumerating objects: 522, done.        
remote: Counting objects:   0% (1/522)        
remote: Counting objects:   1% (6/522)        

Could anyone help me with this please

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d0eafbb9-4e96-41ac-b026-368fb3d8450ao%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Download the git repo and use it for the downstream job

Nitish Goel
Have you tried git lfs?
If you have large files, then git lfs is your solution.

Thanks,
Nitish Goel

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/8b7627f7-350d-476f-b231-dff5cac2c584o%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Download the git repo and use it for the downstream job

Gianluca
In reply to this post by sravan
Hi,
it's hard to help with so little context.
For big what you mean? what is the actual size of the repository?
Git .. exactly where is the remote? GitHub, Gitlab, other?
The error is intermittent or constant? (i.e. could a simple retry do the trick?)


On Wednesday, 1 July 2020 20:03:12 UTC+1, sravan wrote:
Hi All,

My git repo is really big and I am running job parallel on two different machine.Sometimes I see an issue with downloading from git . I get this error "failed on download from git" Error.

ERROR: Timeout after 10 minutes
ERROR: Error cloning remote repo 'origin'
returned status code 128:
stdout: 
stderr: remote: Enumerating objects: 522, done.        
remote: Counting objects:   0% (1/522)        
remote: Counting objects:   1% (6/522)        

Could anyone help me with this please

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/113403d9-abd0-4d73-9ec0-6b684e893315o%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Download the git repo and use it for the downstream job

Mark Waite-2
You might watch https://youtu.be/jBGFjFc6Jf8?t=6434 and https://youtu.be/TsWkZLLU-s4?t=138 in case they contain hints that might help in your case.

Those videos suggest that you might:
  • Use a reference repository on the agent machine to reduce data transfer between the git server and the agent
  • Narrow the refspec in the job definition to request only the items you need
  • Use a shallow clone to request less history of the repository
You might also consider placing the git server "nearer" to the agents on the network.

You might also assess the capabilities of the git server as suggested by Gianluca.  If it is heavily loaded while providing the data, then it may need better I/O services or more processor power or more network bandwidth.

You might also confirm that the git repository on the git server is not needing a garbage collection run.

Many opportunities to tune and improve at many different locations.

Mark Waite

On Wed, Jul 1, 2020 at 1:11 PM Gianluca <[hidden email]> wrote:
Hi,
it's hard to help with so little context.
For big what you mean? what is the actual size of the repository?
Git .. exactly where is the remote? GitHub, Gitlab, other?
The error is intermittent or constant? (i.e. could a simple retry do the trick?)


On Wednesday, 1 July 2020 20:03:12 UTC+1, sravan wrote:
Hi All,

My git repo is really big and I am running job parallel on two different machine.Sometimes I see an issue with downloading from git . I get this error "failed on download from git" Error.

ERROR: Timeout after 10 minutes
ERROR: Error cloning remote repo 'origin'
returned status code 128:
stdout: 
stderr: remote: Enumerating objects: 522, done.        
remote: Counting objects:   0% (1/522)        
remote: Counting objects:   1% (6/522)        

Could anyone help me with this please

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/113403d9-abd0-4d73-9ec0-6b684e893315o%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtHiVOnNbDA5HtKw33EEWpYJsmcZJ3L%3D3mbzoWVE9qiLdw%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Download the git repo and use it for the downstream job

Ivan Fernandez Calvo
In reply to this post by sravan
we have a similar use case, where we need to checkout a repo and download a bunch of dependencies, then we have to share this with several Agents. Because stash and unstash will kill the performance of our Jenkins instance we use a custom step similar to stash/unstash that uses a GCP bucket to store the repo and the dependencies, it adds a little time to make the checkout because we have to upload everything to the bucket, but increases a lot the next checkouts+dependencies.
If you only need the source code, you could use a reference repo. basically your Agents have the code (or part of it) before they make the checkout.

El miércoles, 1 de julio de 2020, 21:03:12 (UTC+2), sravan escribió:
Hi All,

My git repo is really big and I am running job parallel on two different machine.Sometimes I see an issue with downloading from git . I get this error "failed on download from git" Error.

ERROR: Timeout after 10 minutes
ERROR: Error cloning remote repo 'origin'
returned status code 128:
stdout: 
stderr: remote: Enumerating objects: 522, done.        
remote: Counting objects:   0% (1/522)        
remote: Counting objects:   1% (6/522)        

Could anyone help me with this please

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/ac11119b-3aba-4930-b008-109606854b7do%40googlegroups.com.