Quantcast

question about the matrix projects

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

question about the matrix projects

Tyrael
Hi.

I'm setting up a distributed builds for a project, and I'm stucked
with a problem.
It is a C project, and we have a pretty big test suite, so I thought
that it would be the best to split the compiling and running the test
suite in two jobs: job A compiles the project, then kicks off job B
which executes the test suite.
I've used the "Use custom workspace" option so that my job B can use
the artifacts(binaries, tests, etc.) from job A, but everything worked
like a charm.

After this, I converted my job A into a matrix project, and created a
few axis' (x86 vs amd64, linux vs windows, just the usual stuff), but
I'm not sure how to proceed from here.
I would like to execute the job B (testsuite) for all configureation.
As far as I can see, I can't use the "Use custom workspace" as the
workspace contains the label used for each configuration, so that's a
dynamic value, which I can't define in my job configuration.
I also looked into the Copy Artifact Plugin, and judging from the
help, it is somehow aware of the upstream matrix jobs, but the help is
a little bit cryptic, and while I tried a few different setup, but I
wasn't able to solve my problem.

I think that maybe I have to set up job B as a parameterized build,
where the parameter is the label and through that the upstream job A
can pass his active configuration label to job B, and somehow I can
use that for pinning the job B to that label and copying the artifact
from there or using that workspace directly.

Can someone point me to the right direction? Is this a rare setup, or
are there any better alternatives?
Of course I could easily fix this problem, if I merge the job A and B,
but I would like to save that as a last resort.

thanks!

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: question about the matrix projects

Dirk Haun
On Thu, Oct 13, 2011 at 11:46 PM, Ferenc Kovacs <[hidden email]> wrote:

> I would like to execute the job B (testsuite) for all configureation.
> As far as I can see, I can't use the "Use custom workspace" as the
> workspace contains the label used for each configuration, so that's a
> dynamic value, which I can't define in my job configuration.

So you want to run a number of different executables (from job A) on
only one checkout of your testsuite?

I don't think there's an easy way to do that (at least not with the
Subversion plugin). You would want to be able to check out the test
project into a central location somewhere outside of the workspace,
but you can't really specify a random path.

Hmm ... Never tried this, but maybe symlinks would let you do that?
I.e. create symlinks in your workspace that point to the central
location.

If that doesn't work, do what we did: Check out the test project
multiple times, i.e. once for each active job in the matrix. Of course
that's a no-go if you have a really large testsuite ...

HTH
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: question about the matrix projects

Tyrael


On Fri, Oct 14, 2011 at 4:13 PM, Dirk Haun <[hidden email]> wrote:
On Thu, Oct 13, 2011 at 11:46 PM, Ferenc Kovacs <[hidden email]> wrote:

> I would like to execute the job B (testsuite) for all configureation.
> As far as I can see, I can't use the "Use custom workspace" as the
> workspace contains the label used for each configuration, so that's a
> dynamic value, which I can't define in my job configuration.

So you want to run a number of different executables (from job A) on
only one checkout of your testsuite?

I don't mind having a different checkout/workspace for my testsuite build, but I can't execute my testsuite build without the binaries/libs created in the upstream build.
For accessing that, I could use the Copy Artifact Plugin or using the "Use custom workspace" option, and both of them would work for "normal" (non-matrix) jobs.

For running my testsuite on each slave, I created my testsuite job as a matrix job also, and trigger that from my upstream matrix job through "Build other projects".
But I can't figure out how could I specify that each testsuite configuration job should use the appropriate working directory to either Copy the artifacts or use the parent's working directory.
So for example my upstream project will have
build-job/architecture/amd64/linux-distribution/debian/os-family/linux/os-version/6.0/ 
as the workspace directory and my testsuite job will use
test-job/architecture/amd64/linux-distribution/debian/os-family/linux/os-version/6.0/ 
as it's workspace, but amd64, debian, linux and 6.0 are the labels of my matrix job for that specific configuration*, so they are dinamic, I can't specify that literally in my testsuite matrix job's configuration.

Even if I could do that, there is still a slight problem:
If I have more than one slaves which satisfy a given label combination, there is a chance, that my test suite configuration job will be executed on a different slave than it was used for that configuration for the upstream matrix job, hence the binaries wouldn't be there.** ***

So I think that I have to copy my artifacts from each configuration build to some know central location (Copy To Slave Plugin: "Copy files back to the job's workspace on the master node" for example) and always copy from the master for each testsuite configuration job.

Having that in place, I can simply have my Testsuite job as a matrix build with the same axis' as the upstream and copy the artifacts in my build steps from the master using the labels or my working directory to match each testsuite configuration build with the appropriate directory on master.

If somebody sees a better alternative, please let me know.
After thinking this through, I think it would be useful, if there would be possible to pin a downstream matrix build to it's upstream matrix build.
That would mean that:
- there would be a checkbox to use/copy the axis' from the upstream matrix automatically
- there would be a checkbox which would pin the execution of the downstream configuration jobs to the same node where the same configuration job was executed for the upstream job.
- not so important: maybe there would be a checkbox which would allow to execute the downstream configuration job in the same workspace as it was used for the same configuration job in the upstream job.

what do you think?


** jenkins seems to prefer using the same node for the same job, don't know how that works with the matrix jobs: http://groups.google.com/group/jenkinsci-users/browse_thread/thread/c21114e496d9680c/e4badad5ae0b139f?lnk=gst&q=Random+slave+for#anchor_e4badad5ae0b139f
*** there is a plugin for triggering a build on a specific node/label, I think it isn't available in the LTS yet: https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: question about the matrix projects

Sami Tikka
Every now and then people want to use the same workspace for multiple jobs. I myself cannot see how that can be safe, unless people can somehow make sure only one of those jobs is running at a time. And when you add slaves to the mix, things get ugly.

The most Jenkins-like way to solve your problem is to configure artifact archiving in the job that builds binaries and libraries. "Archiving" means that Jenkins copies the binaries and libraries to the master and makes the files available and downloadable from the build page. 

Then you configure the testing job to use the copy artifact plugin or wget or curl to download the binaries and libraries from the artifacts of the above job. 

This way each of the jobs can execute independently, in their own workspace, they can even run at the same time. They can even run on different slaves!

As an added benefit, the binaries and libraries get stored to someplace more permanent. Jenkins doesn't consider job workspaces to be permanent storage. Sometimes it even wipes the workspaces to clean things up.

-- Sami

Ferenc Kovacs <[hidden email]> kirjoitti 15.10.2011 kello 21.57:



On Fri, Oct 14, 2011 at 4:13 PM, Dirk Haun <[hidden email]> wrote:
On Thu, Oct 13, 2011 at 11:46 PM, Ferenc Kovacs <[hidden email]> wrote:

> I would like to execute the job B (testsuite) for all configureation.
> As far as I can see, I can't use the "Use custom workspace" as the
> workspace contains the label used for each configuration, so that's a
> dynamic value, which I can't define in my job configuration.

So you want to run a number of different executables (from job A) on
only one checkout of your testsuite?

I don't mind having a different checkout/workspace for my testsuite build, but I can't execute my testsuite build without the binaries/libs created in the upstream build.
For accessing that, I could use the Copy Artifact Plugin or using the "Use custom workspace" option, and both of them would work for "normal" (non-matrix) jobs.

For running my testsuite on each slave, I created my testsuite job as a matrix job also, and trigger that from my upstream matrix job through "Build other projects".
But I can't figure out how could I specify that each testsuite configuration job should use the appropriate working directory to either Copy the artifacts or use the parent's working directory.
So for example my upstream project will have
build-job/architecture/amd64/linux-distribution/debian/os-family/linux/os-version/6.0/ 
as the workspace directory and my testsuite job will use
test-job/architecture/amd64/linux-distribution/debian/os-family/linux/os-version/6.0/ 
as it's workspace, but amd64, debian, linux and 6.0 are the labels of my matrix job for that specific configuration*, so they are dinamic, I can't specify that literally in my testsuite matrix job's configuration.

Even if I could do that, there is still a slight problem:
If I have more than one slaves which satisfy a given label combination, there is a chance, that my test suite configuration job will be executed on a different slave than it was used for that configuration for the upstream matrix job, hence the binaries wouldn't be there.** ***

So I think that I have to copy my artifacts from each configuration build to some know central location (Copy To Slave Plugin: "Copy files back to the job's workspace on the master node" for example) and always copy from the master for each testsuite configuration job.

Having that in place, I can simply have my Testsuite job as a matrix build with the same axis' as the upstream and copy the artifacts in my build steps from the master using the labels or my working directory to match each testsuite configuration build with the appropriate directory on master.

If somebody sees a better alternative, please let me know.
After thinking this through, I think it would be useful, if there would be possible to pin a downstream matrix build to it's upstream matrix build.
That would mean that:
- there would be a checkbox to use/copy the axis' from the upstream matrix automatically
- there would be a checkbox which would pin the execution of the downstream configuration jobs to the same node where the same configuration job was executed for the upstream job.
- not so important: maybe there would be a checkbox which would allow to execute the downstream configuration job in the same workspace as it was used for the same configuration job in the upstream job.

what do you think?


** jenkins seems to prefer using the same node for the same job, don't know how that works with the matrix jobs: http://groups.google.com/group/jenkinsci-users/browse_thread/thread/c21114e496d9680c/e4badad5ae0b139f?lnk=gst&q=Random+slave+for#anchor_e4badad5ae0b139f
*** there is a plugin for triggering a build on a specific node/label, I think it isn't available in the LTS yet: https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: question about the matrix projects

Tyrael
Thanks for the reply.
Yeah, sharing workspaces was just a shortcut (but it is possible to restrict concurrent builds, so it can be safe), and as you can see, I also mentioned that copying artifacts seems the right way to go(thanks for the info about archiving the artifacts).
In the meantime I found out why was I having problems with the copy artifacts plugin:
For passing params the Anonymous user needs job read permission, and it seems that adding that only for the upstream project via Role Strategy Plugin isn't enough. :/

But I got it working at least!

My second mail was more about telling what I found and emphasizing the fact that basic stuff gets more complicated if one works with matrix jobs. 

On Sat, Oct 15, 2011 at 11:18 PM, Sami Tikka <[hidden email]> wrote:
Every now and then people want to use the same workspace for multiple jobs. I myself cannot see how that can be safe, unless people can somehow make sure only one of those jobs is running at a time. And when you add slaves to the mix, things get ugly.

The most Jenkins-like way to solve your problem is to configure artifact archiving in the job that builds binaries and libraries. "Archiving" means that Jenkins copies the binaries and libraries to the master and makes the files available and downloadable from the build page. 

Then you configure the testing job to use the copy artifact plugin or wget or curl to download the binaries and libraries from the artifacts of the above job. 

This way each of the jobs can execute independently, in their own workspace, they can even run at the same time. They can even run on different slaves!

As an added benefit, the binaries and libraries get stored to someplace more permanent. Jenkins doesn't consider job workspaces to be permanent storage. Sometimes it even wipes the workspaces to clean things up.

-- Sami

Ferenc Kovacs <[hidden email]> kirjoitti 15.10.2011 kello 21.57:



On Fri, Oct 14, 2011 at 4:13 PM, Dirk Haun <[hidden email]> wrote:
On Thu, Oct 13, 2011 at 11:46 PM, Ferenc Kovacs <[hidden email]> wrote:

> I would like to execute the job B (testsuite) for all configureation.
> As far as I can see, I can't use the "Use custom workspace" as the
> workspace contains the label used for each configuration, so that's a
> dynamic value, which I can't define in my job configuration.

So you want to run a number of different executables (from job A) on
only one checkout of your testsuite?

I don't mind having a different checkout/workspace for my testsuite build, but I can't execute my testsuite build without the binaries/libs created in the upstream build.
For accessing that, I could use the Copy Artifact Plugin or using the "Use custom workspace" option, and both of them would work for "normal" (non-matrix) jobs.

For running my testsuite on each slave, I created my testsuite job as a matrix job also, and trigger that from my upstream matrix job through "Build other projects".
But I can't figure out how could I specify that each testsuite configuration job should use the appropriate working directory to either Copy the artifacts or use the parent's working directory.
So for example my upstream project will have
build-job/architecture/amd64/linux-distribution/debian/os-family/linux/os-version/6.0/ 
as the workspace directory and my testsuite job will use
test-job/architecture/amd64/linux-distribution/debian/os-family/linux/os-version/6.0/ 
as it's workspace, but amd64, debian, linux and 6.0 are the labels of my matrix job for that specific configuration*, so they are dinamic, I can't specify that literally in my testsuite matrix job's configuration.

Even if I could do that, there is still a slight problem:
If I have more than one slaves which satisfy a given label combination, there is a chance, that my test suite configuration job will be executed on a different slave than it was used for that configuration for the upstream matrix job, hence the binaries wouldn't be there.** ***

So I think that I have to copy my artifacts from each configuration build to some know central location (Copy To Slave Plugin: "Copy files back to the job's workspace on the master node" for example) and always copy from the master for each testsuite configuration job.

Having that in place, I can simply have my Testsuite job as a matrix build with the same axis' as the upstream and copy the artifacts in my build steps from the master using the labels or my working directory to match each testsuite configuration build with the appropriate directory on master.

If somebody sees a better alternative, please let me know.
After thinking this through, I think it would be useful, if there would be possible to pin a downstream matrix build to it's upstream matrix build.
That would mean that:
- there would be a checkbox to use/copy the axis' from the upstream matrix automatically
- there would be a checkbox which would pin the execution of the downstream configuration jobs to the same node where the same configuration job was executed for the upstream job.
- not so important: maybe there would be a checkbox which would allow to execute the downstream configuration job in the same workspace as it was used for the same configuration job in the upstream job.

what do you think?


** jenkins seems to prefer using the same node for the same job, don't know how that works with the matrix jobs: http://groups.google.com/group/jenkinsci-users/browse_thread/thread/c21114e496d9680c/e4badad5ae0b139f?lnk=gst&q=Random+slave+for#anchor_e4badad5ae0b139f
*** there is a plugin for triggering a build on a specific node/label, I think it isn't available in the LTS yet: https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu



--
Ferenc Kovács
@Tyr43l - http://tyrael.hu
Loading...