Jenkins + Kubernetes - Cache .m2 for jobs

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Jenkins + Kubernetes - Cache .m2 for jobs

Enguerrand A
Hello everyone,

I use the Kubernetes plugin for Jenkins to launch all Jenkins jobs in a Kubernetes cluster. For better performance, I want to use a persistent disk for the cache of different job Maven with insulation of it, ie a disk space per job (to separate the cache)

Do you have ideas for automatic solutions with the Kubernetes plugin to have cache on the .m2 (or another type of cache) with a different disk by jobs ?

Thank you in advance !

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/de71d26c-115e-4679-af8e-0151fbb3b8c9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Jenkins + Kubernetes - Cache .m2 for jobs

Carlos Sanchez
you can mount a PV or PVC but depending on what backend you use that will prevent concurrent executions

On Tue, Jul 10, 2018 at 3:05 PM Enguerrand A <[hidden email]> wrote:
Hello everyone,

I use the Kubernetes plugin for Jenkins to launch all Jenkins jobs in a Kubernetes cluster. For better performance, I want to use a persistent disk for the cache of different job Maven with insulation of it, ie a disk space per job (to separate the cache)

Do you have ideas for automatic solutions with the Kubernetes plugin to have cache on the .m2 (or another type of cache) with a different disk by jobs ?

Thank you in advance !

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/de71d26c-115e-4679-af8e-0151fbb3b8c9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CALHFn6P%3DrDHFv1tJ1XFw1sHW4X0uVW4QX5DpARqNxQ_wUJ%3DYaA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Jenkins + Kubernetes - Cache .m2 for jobs

Enguerrand A
The problem is that I would like to automate the installation of the PV or PVC without needing to specify it in the Jenkinsfile but at least I can automate the creation of PVC or PV for each job to avoid manual operation.

I saw that automatically a PVC or PV could be mounted for the workspace Jenkins, but it raises concerns if 2 build on the same job runs on the same shared workspace (build can inflict another by removing files for example)

Le mardi 10 juillet 2018 15:35:17 UTC+2, Carlos Sanchez a écrit :
you can mount a PV or PVC but depending on what backend you use that will prevent concurrent executions

On Tue, Jul 10, 2018 at 3:05 PM Enguerrand A <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="smVbth00BwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">allamel.e...@thalesaleniaspace.com> wrote:
Hello everyone,

I use the Kubernetes plugin for Jenkins to launch all Jenkins jobs in a Kubernetes cluster. For better performance, I want to use a persistent disk for the cache of different job Maven with insulation of it, ie a disk space per job (to separate the cache)

Do you have ideas for automatic solutions with the Kubernetes plugin to have cache on the .m2 (or another type of cache) with a different disk by jobs ?

Thank you in advance !

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="smVbth00BwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/de71d26c-115e-4679-af8e-0151fbb3b8c9%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/de71d26c-115e-4679-af8e-0151fbb3b8c9%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/de71d26c-115e-4679-af8e-0151fbb3b8c9%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/de71d26c-115e-4679-af8e-0151fbb3b8c9%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/cb81d993-885d-460d-b006-22b474a813e1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Jenkins + Kubernetes - Cache .m2 for jobs

Tristan FAURE
Hello !
we have the same issue

what we would like is the same behavior as : https://docs.gitlab.com/ee/ci/caching/

I have seen this example : https://github.com/jenkinsci/kubernetes-plugin/blob/master/examples/maven-with-cache.groovy
but if several jobs run in parallel I don't want them to modify the same volumes and I would like to not create volumes before the jobs

I have seen the dynamic claim options for workspace but data are common for a run and not a job does it ?

our idea currently is creating a volume claim with a name ${jobid} or {folderId} and set a K8S cron job that delete frequently but we don't find a way to set this by default (we tried in raw yaml option but it seems it only support kind : Pod)

Regards

Le mercredi 11 juillet 2018 09:48:23 UTC+2, Enguerrand A a écrit :
The problem is that I would like to automate the installation of the PV or PVC without needing to specify it in the Jenkinsfile but at least I can automate the creation of PVC or PV for each job to avoid manual operation.

I saw that automatically a PVC or PV could be mounted for the workspace Jenkins, but it raises concerns if 2 build on the same job runs on the same shared workspace (build can inflict another by removing files for example)

Le mardi 10 juillet 2018 15:35:17 UTC+2, Carlos Sanchez a écrit :
you can mount a PV or PVC but depending on what backend you use that will prevent concurrent executions

On Tue, Jul 10, 2018 at 3:05 PM Enguerrand A <allamel.e...@thalesaleniaspace.com> wrote:
Hello everyone,

I use the Kubernetes plugin for Jenkins to launch all Jenkins jobs in a Kubernetes cluster. For better performance, I want to use a persistent disk for the cache of different job Maven with insulation of it, ie a disk space per job (to separate the cache)

Do you have ideas for automatic solutions with the Kubernetes plugin to have cache on the .m2 (or another type of cache) with a different disk by jobs ?

Thank you in advance !

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/de71d26c-115e-4679-af8e-0151fbb3b8c9%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/de71d26c-115e-4679-af8e-0151fbb3b8c9%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/de71d26c-115e-4679-af8e-0151fbb3b8c9%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/de71d26c-115e-4679-af8e-0151fbb3b8c9%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/58617773-5901-4f03-97e3-4bfae081b207%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Jenkins + Kubernetes - Cache .m2 for jobs

Liejun Tao
In reply to this post by Enguerrand A
This is my solution for a little different purpose: use a .m2 cache for common libs.
The idea is to make a docker image with the most used artifacts(external dependencies) in .m2 folder.
At the launch of the Agent, copy them to workspace's .m2 folder from the image.

This should be adapted to fit your needs.
I assume you are not going to run concurrent builds for a job to avoid concurrent write, if you want a job has a dedicated cache.
At the end of a build, create a docker image based on the existing .m2 content and push back for next time use.

If you are going to run concurrently builds, this idea of a sharing cache is invalid, I think.

On Tue, Jul 10, 2018 at 8:05 AM Enguerrand A <[hidden email]> wrote:
Hello everyone,

I use the Kubernetes plugin for Jenkins to launch all Jenkins jobs in a Kubernetes cluster. For better performance, I want to use a persistent disk for the cache of different job Maven with insulation of it, ie a disk space per job (to separate the cache)

Do you have ideas for automatic solutions with the Kubernetes plugin to have cache on the .m2 (or another type of cache) with a different disk by jobs ?

Thank you in advance !

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/de71d26c-115e-4679-af8e-0151fbb3b8c9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Thanks,
Baibai

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAMRm5jbRhhtF5S4bzzT%3Du7La4sggNW2c-kEAjVMaqBeBK6cHPg%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Jenkins + Kubernetes - Cache .m2 for jobs

Tristan FAURE
Hello
Thank you for the hint ! I think it could help some projects !
but it will make some docker images very heavy and I would be interested to have jenkins team opinion about getting the equivalent of https://docs.gitlab.com/ee/ci/caching/

Le jeu. 26 mars 2020 à 06:10, Liejun Tao <[hidden email]> a écrit :
This is my solution for a little different purpose: use a .m2 cache for common libs.
The idea is to make a docker image with the most used artifacts(external dependencies) in .m2 folder.
At the launch of the Agent, copy them to workspace's .m2 folder from the image.

This should be adapted to fit your needs.
I assume you are not going to run concurrent builds for a job to avoid concurrent write, if you want a job has a dedicated cache.
At the end of a build, create a docker image based on the existing .m2 content and push back for next time use.

If you are going to run concurrently builds, this idea of a sharing cache is invalid, I think.

On Tue, Jul 10, 2018 at 8:05 AM Enguerrand A <[hidden email]> wrote:
Hello everyone,

I use the Kubernetes plugin for Jenkins to launch all Jenkins jobs in a Kubernetes cluster. For better performance, I want to use a persistent disk for the cache of different job Maven with insulation of it, ie a disk space per job (to separate the cache)

Do you have ideas for automatic solutions with the Kubernetes plugin to have cache on the .m2 (or another type of cache) with a different disk by jobs ?

Thank you in advance !

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/de71d26c-115e-4679-af8e-0151fbb3b8c9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Thanks,
Baibai

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/X61BK83LHLU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAMRm5jbRhhtF5S4bzzT%3Du7La4sggNW2c-kEAjVMaqBeBK6cHPg%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CA%2BtQ8YP2MvN4zf3uvkWpAsbOrhbbqhufQAHRBsNy2nuE%2BpTe0g%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Jenkins + Kubernetes - Cache .m2 for jobs

Tristan FAURE
seems like there is an issue about it:  https://issues.jenkins-ci.org/browse/JENKINS-42422

Le jeu. 26 mars 2020 à 08:07, Tristan FAURE <[hidden email]> a écrit :
Hello
Thank you for the hint ! I think it could help some projects !
but it will make some docker images very heavy and I would be interested to have jenkins team opinion about getting the equivalent of https://docs.gitlab.com/ee/ci/caching/

Le jeu. 26 mars 2020 à 06:10, Liejun Tao <[hidden email]> a écrit :
This is my solution for a little different purpose: use a .m2 cache for common libs.
The idea is to make a docker image with the most used artifacts(external dependencies) in .m2 folder.
At the launch of the Agent, copy them to workspace's .m2 folder from the image.

This should be adapted to fit your needs.
I assume you are not going to run concurrent builds for a job to avoid concurrent write, if you want a job has a dedicated cache.
At the end of a build, create a docker image based on the existing .m2 content and push back for next time use.

If you are going to run concurrently builds, this idea of a sharing cache is invalid, I think.

On Tue, Jul 10, 2018 at 8:05 AM Enguerrand A <[hidden email]> wrote:
Hello everyone,

I use the Kubernetes plugin for Jenkins to launch all Jenkins jobs in a Kubernetes cluster. For better performance, I want to use a persistent disk for the cache of different job Maven with insulation of it, ie a disk space per job (to separate the cache)

Do you have ideas for automatic solutions with the Kubernetes plugin to have cache on the .m2 (or another type of cache) with a different disk by jobs ?

Thank you in advance !

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/de71d26c-115e-4679-af8e-0151fbb3b8c9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Thanks,
Baibai

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/X61BK83LHLU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAMRm5jbRhhtF5S4bzzT%3Du7La4sggNW2c-kEAjVMaqBeBK6cHPg%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CA%2BtQ8YN41OAVo_u6kzGm5xiMxWHurVgpn%2BX3pGVZ18gjY%2B49cA%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Jenkins + Kubernetes - Cache .m2 for jobs

James Nord-2
you can massively speed up the build (even without caching) by having a maven repository proxy/cache nexus etc) in the same cluster (or at least on a high bandwidth low latency link). even when you have things local to the container the remote maven repo  will still be contacted and depending on your setup can still slow the build.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1e1af14d-7190-487e-bf2f-a006b724eee4%40googlegroups.com.