Project with several Git repositories

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Project with several Git repositories

Sébastien Hinderer
Dear all,

Say a project has two repositories: one for its source code and one for
its test suite.

How would you do CI for such a project?

I mean, I did try to define several repositories but it seems that
Jenkins only clones the most recently modified rather than cloning all
of them side-by-side. iS there another way to achieve this?

Many thanks in advance,

Sébastien.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/20180502152621.GA7844%40prajna.paris.inria.fr.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Project with several Git repositories

Vivekanand S V
I do that using https://wiki.jenkins.io/display/JENKINS/Multiple+SCMs+Plugin

This plugin should help you.

On Thu, 3 May 2018 at 10:46 Sébastien Hinderer <[hidden email]> wrote:
Dear all,

Say a project has two repositories: one for its source code and one for
its test suite.

How would you do CI for such a project?

I mean, I did try to define several repositories but it seems that
Jenkins only clones the most recently modified rather than cloning all
of them side-by-side. iS there another way to achieve this?

Many thanks in advance,

Sébastien.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/20180502152621.GA7844%40prajna.paris.inria.fr.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CADijPE-bouZ9fjaJPQkCPJe1npcc5ULR7%2B4t-EPPDVhbutfB2w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Project with several Git repositories

Vivekanand S V
Okay, I just saw its deprecated :( ... the page points to Pipeline Plugin though, do check it out. I don't have experience using that.

On Thu, 3 May 2018 at 11:23 Vivekanand S V <[hidden email]> wrote:
I do that using https://wiki.jenkins.io/display/JENKINS/Multiple+SCMs+Plugin

This plugin should help you.

On Thu, 3 May 2018 at 10:46 Sébastien Hinderer <[hidden email]> wrote:
Dear all,

Say a project has two repositories: one for its source code and one for
its test suite.

How would you do CI for such a project?

I mean, I did try to define several repositories but it seems that
Jenkins only clones the most recently modified rather than cloning all
of them side-by-side. iS there another way to achieve this?

Many thanks in advance,

Sébastien.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/20180502152621.GA7844%40prajna.paris.inria.fr.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CADijPE9vjoP9AnN0UwX7hRbOxhyq3z7Znu3UvKYgJxmS9yAWHg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Project with several Git repositories

Sébastien Hinderer
OK many thanks for your response! Will definitely have a look to it!

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/20180503064453.GA11264%40prajna.paris.inria.fr.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Project with several Git repositories

Andrei Gheorghiu
You could use git hooks on any number of repositories to start the (same) Jenkins project.

On Thursday, May 3, 2018 at 9:45:05 AM UTC+3, Sébastien Hinderer wrote:
OK many thanks for your response! Will definitely have a look to it!

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/3088f10b-85f6-4514-b4ce-78c49b1ab5d2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Project with several Git repositories

Sébastien Hinderer
Dear Andrei,

Many thanks for your suggestion!

Andrei Gheorghiu (2018/05/03 05:15 -0700):
> You could use git hooks on any number of repositories to start the (same)
> Jenkins project.

Yeah, that's actually the solution I am playing with currently.

So just to make sure we are talking about the same thing, that means
checking the "None" radio button in the souce code managemnet area, then
check the "run builds remotely (from scripts)" checkbox and then,
finally, in the commands used to build, clone all the required
repositories.

Is that what you have in mind?

Many thanks in advance,

Sébastien.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/20180503133809.GA14922%40prajna.paris.inria.fr.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Project with several Git repositories

jmellor
On Thu, 2018-05-03 at 15:38 +0200, Sébastien Hinderer wrote:
> . . .
>
> So just to make sure we are talking about the same thing, that means
> checking the "None" radio button in the souce code managemnet area,
> then
> check the "run builds remotely (from scripts)" checkbox and then,
> finally, in the commands used to build, clone all the required
> repositories.

I have a bit of a different situation, where I want to use a common
Jenkinsfile across a large number of git repos.  Duplicating the
Jenkinsfile everywhere is a very poor solution, so I checkout the
project containing the Jenkinsfile, delete the .git directory in the
WORKSPACE, and then use the scm directives in the Jenkinsfile to
checkout the real project on top.  Its far from ideal, but for some
reason it looks like this very common scenario is not supported.

If you have a better way to do this...

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1525357081.24903.6.camel%40esentire.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Project with several Git repositories

Sébastien Hinderer
Hi John,

John Mellor (2018/05/03 14:18 +0000):

> On Thu, 2018-05-03 at 15:38 +0200, Sébastien Hinderer wrote:
> > . . .
> >
> > So just to make sure we are talking about the same thing, that means
> > checking the "None" radio button in the souce code managemnet area,
> > then
> > check the "run builds remotely (from scripts)" checkbox and then,
> > finally, in the commands used to build, clone all the required
> > repositories.
>
> I have a bit of a different situation, where I want to use a common
> Jenkinsfile across a large number of git repos.  Duplicating the
> Jenkinsfile everywhere is a very poor solution, so I checkout the
> project containing the Jenkinsfile, delete the .git directory in the
> WORKSPACE, and then use the scm directives in the Jenkinsfile to
> checkout the real project on top.  Its far from ideal, but for some
> reason it looks like this very common scenario is not supported.
>
> If you have a better way to do this...

No, no better alternative, sorry. I am not using pipelines yet, allas.

Sébastien.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/20180503142017.GA15942%40prajna.paris.inria.fr.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Project with several Git repositories

Mark Waite-2
In reply to this post by jmellor


On Thu, May 3, 2018 at 8:18 AM John Mellor <[hidden email]> wrote:
On Thu, 2018-05-03 at 15:38 +0200, Sébastien Hinderer wrote:
> . . .
>
> So just to make sure we are talking about the same thing, that means
> checking the "None" radio button in the souce code managemnet area,
> then
> check the "run builds remotely (from scripts)" checkbox and then,
> finally, in the commands used to build, clone all the required
> repositories.

I have a bit of a different situation, where I want to use a common
Jenkinsfile across a large number of git repos.  Duplicating the
Jenkinsfile everywhere is a very poor solution, so I checkout the
project containing the Jenkinsfile, delete the .git directory in the
WORKSPACE, and then use the scm directives in the Jenkinsfile to
checkout the real project on top.  Its far from ideal, but for some
reason it looks like this very common scenario is not supported.

If you have a better way to do this...


It seems like you're describing a very good use case for a Pipeline shared library.  Each repository and each branch then has a very simple Jenkinsfile which loads the shared library and calls it.  That's the technique used for Jenkins plugin development.

Mark Waite
 
--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1525357081.24903.6.camel%40esentire.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtHRwd6WYAK-XXpJqhRkfZ6L3mArtXaHOxRSf%3DwPukGz3Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Project with several Git repositories

jmellor
On Thu, 2018-05-03 at 16:26 +0000, Mark Waite wrote:
. . .It seems like you're describing a very good use case for a Pipeline shared library.  Each repository and each branch then has a very simple Jenkinsfile which loads the shared library and calls it.  That's the technique used for Jenkins plugin development

Correct me if I am mistaken, but that would imply that I would still need to insert an identical Jenkinsfile into 155 projects in order to actually use the @library. That's not improving anything. I want to specify the Jenkinsfile from a different git repo and not alter all the other 155 repos, and there is no means of doing this as the pipeline plugins are currently written. That should be a base use-case, and it seems to have been missed. As a result, every project build that is changed over to pipelines needs to be modified for very questionable reasons.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1525371033.24903.14.camel%40esentire.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Project with several Git repositories

Mark Waite-2
No, you're not mistaken.  You'd need to insert a Jenkinsfile into each repository to describe how that repository is built.

I don't think the use case you're describing should be a base use-case.

Mark Waite

On Thu, May 3, 2018 at 12:11 PM John Mellor <[hidden email]> wrote:
On Thu, 2018-05-03 at 16:26 +0000, Mark Waite wrote:
. . .It seems like you're describing a very good use case for a Pipeline shared library.  Each repository and each branch then has a very simple Jenkinsfile which loads the shared library and calls it.  That's the technique used for Jenkins plugin development

Correct me if I am mistaken, but that would imply that I would still need to insert an identical Jenkinsfile into 155 projects in order to actually use the @library. That's not improving anything. I want to specify the Jenkinsfile from a different git repo and not alter all the other 155 repos, and there is no means of doing this as the pipeline plugins are currently written. That should be a base use-case, and it seems to have been missed. As a result, every project build that is changed over to pipelines needs to be modified for very questionable reasons.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1525371033.24903.14.camel%40esentire.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtG7mbOrUTMDDrjity31-S5C6n8%2B_kPS09LJwOtH3pma_w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Project with several Git repositories

Sam Beckwith III
In reply to this post by jmellor
I second this request but not for the reasons given by Mr. John Mellor. Instead I request this because our team does not know or understand Jenkins nor the infrastructure available to it, and should not be allowed to make changes to the jenkins process without devops oversight. Putting the Jenkinsfile in a separate git repo allows devops to review that which concerns their ability to service the team (Jenkins), and keeping the Jenkinsfile out of the development team's repos allows the dev team to focus on what they know and need to progress on.

Currently we use Job DSL plugin for our Jenkins jobs however there is the desire to use pipelines, especially multi-branch pipelines, yet we cannot delegate the Jenkins build process to a Jenkinsfile in the microservices' repo. I see how that would be useful in many cases but I also see how it is not useful for teams like ourselves.

Would it be possible to setup a multi-branch pipeline where the Jenkinsfile is provided outside the target repo's source? I ask this after already looking several times for hours into this over the past year.



On Thursday, May 3, 2018 at 1:11:03 PM UTC-5, John Mellor wrote:
On Thu, 2018-05-03 at 16:26 +0000, Mark Waite wrote:
. . .It seems like you're describing a very good use case for a Pipeline shared library.  Each repository and each branch then has a very simple Jenkinsfile which loads the shared library and calls it.  That's the technique used for <a href="https://github.com/jenkinsci/git-client-plugin/blob/master/Jenkinsfile" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Fgit-client-plugin%2Fblob%2Fmaster%2FJenkinsfile\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEgK8TvQ1JxPECs0RyzsRFQmguxrQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Fgit-client-plugin%2Fblob%2Fmaster%2FJenkinsfile\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEgK8TvQ1JxPECs0RyzsRFQmguxrQ&#39;;return true;"> Jenkins plugin development

Correct me if I am mistaken, but that would imply that I would still need to insert an identical Jenkinsfile into 155 projects in order to actually use the @library. That's not improving anything. I want to specify the Jenkinsfile from a different git repo and not alter all the other 155 repos, and there is no means of doing this as the pipeline plugins are currently written. That should be a base use-case, and it seems to have been missed. As a result, every project build that is changed over to pipelines needs to be modified for very questionable reasons.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/a4880e9f-3f9f-496a-a746-e281cd104570%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Project with several Git repositories

Victor Martinez
Maybe it sounds a bit crazy, but what about using git submodules. The jenkinsfile logic is outside the target repo's source as your initial requirement, although my experience with git submodules is not really something I'd say I enjoyed it... but it might be something to explore if it helps.

Besides of that, It looks like your requirement matches quite well with one of the enterprise plugins of CloudBees:
- https://go.cloudbees.com/docs/cloudbees-documentation/cje-user-guide/index.html#pipeline-custom-factories

Cheers

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/28478f04-a514-4a4f-b005-3dd3cced918f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Project with several Git repositories

Martin d'Anjou
You can write a Jenkinsfile that creates a clone of any git repository (calling out to Git with sh step for example), hence Jenkinsfiles can be central for however many repositories you have. No need for 100's of copies of the same Jenkinsfile in 100's of git repositories. You can create and manipulate Jenkins Jobs using the REST API from a loop on the command line. For Java projects, you can use this nice library called jenkins-rest to talk to the Jenkins REST endpoints. If your devops is python/curl based or other, see Remote access API.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/f01ab29f-4ae5-4ba9-beb9-4744ac16d82d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.