[Issue 682] New - Share workspace between jobs

classic Classic list List threaded Threaded
31 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[Issue 682] New - Share workspace between jobs

mmatula
https://hudson.dev.java.net/issues/show_bug.cgi?id=682
                 Issue #|682
                 Summary|Share workspace between jobs
               Component|hudson
                 Version|current
                Platform|All
              OS/Version|All
                     URL|
                  Status|NEW
       Status whiteboard|
                Keywords|
              Resolution|
              Issue type|DEFECT
                Priority|P3
            Subcomponent|other
             Assigned to|issues@hudson
             Reported by|mmatula






------- Additional comments from [hidden email] Fri Jul 27 13:26:19 +0000 2007 -------
Seems that the usual way how jobs work on hudson is that there is one job for
building a project and then if I want to have a subsequent job to run all unit
tests, that job needs to download artifacts produced by that build job from
Hudson. As a consequence, people usually add separate targets to their build
scripts for running unit tests from hudson which download the build artifacts
from the network.
I think it would be nice if things could work the way how users would do it when
running tests locally - i.e. use usual targets for building and then simply run
a target for unit tests in the same workspace. This could be achieved by
providing the subsequent jobs an exact snapshot of the workspace at the time
when the parent job finished running.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 682] Share workspace between jobs

mmatula
https://hudson.dev.java.net/issues/show_bug.cgi?id=682



User mmatula changed the following:

                What    |Old value                 |New value
================================================================================
              Issue type|DEFECT                    |ENHANCEMENT
--------------------------------------------------------------------------------




------- Additional comments from [hidden email] Fri Jul 27 13:28:05 +0000 2007 -------
Oops, this an enhancement request, not a defect...

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 682] Share workspace between jobs

kohsuke-djn
In reply to this post by mmatula
https://hudson.dev.java.net/issues/show_bug.cgi?id=682






------- Additional comments from [hidden email] Fri Jul 27 16:42:47 +0000 2007 -------
The problem with doing this is that then you won't be able to run the main build
and tests in parallel, so it increases the turn-around time.

What's the difference between setting up two jobs with the same workspace, as
opposed to setting up one job that does both the build and the test?

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 682] Share workspace between jobs

mmatula
In reply to this post by mmatula
https://hudson.dev.java.net/issues/show_bug.cgi?id=682






------- Additional comments from [hidden email] Fri Jul 27 16:59:44 +0000 2007 -------
I see. I was not sure how it is implemented - i.e. whether different slaves
access the same physical workspace. I thought each slave checks out its own
workspace for a given job in which case tests and a new build could still run in
parallel.
I thought the difference between running it as a single job would be exactly
that possibility to start another build before tests (and possibly other
subsequent jobs such as coverage, findbugs, javadoc generation, etc.) finished
running.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 682] Share workspace between jobs

kohsuke-djn
In reply to this post by mmatula
https://hudson.dev.java.net/issues/show_bug.cgi?id=682






------- Additional comments from [hidden email] Fri Jul 27 18:42:22 +0000 2007 -------
Every slave uses a workspace on local file system, so different slaves get
different workspaces.

The problem I was trying to point out in sharing a workspace is that if a new
build starts while a test run is in progress, it will most likely mess up. For
example, a build might try to overwrite the file that a test is using. So in
general, I don't see how you can reliably run multiple tasks over the same
workspace in parallel.

Or maybe what you are really saying is a slightly different model, where the
execution will go like:


 ws #0   <--- build #N ---><--- test #N --->
 ws #1                     <--- build #N+1 ---><---test #N+1 --->

?

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 682] Share workspace between jobs

gradopado-2
In reply to this post by mmatula
https://hudson.dev.java.net/issues/show_bug.cgi?id=682






------- Additional comments from [hidden email] Fri Jul 27 18:56:22 +0000 2007 -------
Have a look at Buildbot, where each build is composed of several build steps.
Steps are tied to one another, no overlapping.

The first one is mostly cvs-checkout and then you are free to do in steps what
you think is necessary. Each step has its own result and derective how to
proceed on WARNING and ERROR cases. This concept is nice.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 682] Share workspace between jobs

mmatula
In reply to this post by mmatula
https://hudson.dev.java.net/issues/show_bug.cgi?id=682






------- Additional comments from [hidden email] Fri Jul 27 19:56:31 +0000 2007 -------
Kohsuke, what I would like is to be able to configure my job in such a way that
it could assume the workspace is in the same state as it was after it's parent
job. So, if some job is triggered by other job, then I thought it would be good
if there was a way for this triggered job to inherit the workspace of that
triggering job. So that people would not have to do additional "magic" specific
to hudson (and maintain additional hudson-specific targets) in their build
scripts. Hudson does know that a given job is triggered by another job and most
likely it is going to use the artifacts that other job produces. So I thought it
would be nice if those could be provided to these subsequent jobs automatically.

As a user that does not know much about hudson internals, I cannot suggest how
to implement it. I thought one way (the basic one) it could be done is what you
showed in your ASCII diagram. Maybe other possible way could be to have a
mechanism to push a snapshot of the workspace from a job to other jobs it
triggers (potentially to other machines if those jobs run on other systems). I
guess this would be least limiting when it comes to parallel execution.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 682] Share workspace between jobs

kohsuke-djn
In reply to this post by mmatula
https://hudson.dev.java.net/issues/show_bug.cgi?id=682






------- Additional comments from [hidden email] Tue Sep  4 06:02:29 +0000 2007 -------
A note to myself --- recent introduction of Resource interface can be used as a
lock mechanism for multiple jobs to use the same workspace.

This is convenient for some use cases, like occasionally running lengthy-task
(like "mvn site") where normally a quick CI build runs.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 682] Share workspace between jobs

akostadinov
In reply to this post by mmatula
https://hudson.dev.java.net/issues/show_bug.cgi?id=682






------- Additional comments from [hidden email] Fri Sep 21 07:48:13 +0000 2007 -------
I want to mention the case where workspace is on NFS.

By many of the jobs we lose most of the time checking out the sources so being
able to share workspace saves considerable amount time.

That is in the case of a multiple configuration project or the job can be
separated into few tasks.

I think that by being able to mark arbitrary jobs as sequential that will work
for us.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 682] Share workspace between jobs

akostadinov
In reply to this post by mmatula
https://hudson.dev.java.net/issues/show_bug.cgi?id=682



User akostadinov changed the following:

                What    |Old value                 |New value
================================================================================
                      CC|''                        |'akostadinov'
--------------------------------------------------------------------------------




------- Additional comments from [hidden email] Fri Sep 21 07:49:40 +0000 2007 -------
adding myself to the cc

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 682] Share workspace between jobs

kohsuke-djn
In reply to this post by mmatula
https://hudson.dev.java.net/issues/show_bug.cgi?id=682



User kohsuke changed the following:

                What    |Old value                 |New value
================================================================================
                Priority|P3                        |P2
--------------------------------------------------------------------------------




------- Additional comments from [hidden email] Sat Oct 20 00:56:43 +0000 2007 -------
this is a commonly asked feature, and this is from my colleague, so bumping up
the priority a bit.

See another recent discussion at
http://www.nabble.com/Efficiently-using-Hudson-tf4649823.html

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 682] Share workspace between jobs

kohsuke-djn
In reply to this post by mmatula
https://hudson.dev.java.net/issues/show_bug.cgi?id=682






------- Additional comments from [hidden email] Tue Dec 18 02:03:29 +0000 2007 -------
Another recent discussion about this:
http://www.nabble.com/Multiple-jobs-per-project---to14341510.html

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 682] Share workspace between jobs

Kohsuke Kawaguchi
Administrator
In reply to this post by mmatula
https://hudson.dev.java.net/issues/show_bug.cgi?id=682






------- Additional comments from [hidden email] Fri Jun 27 17:57:09 +0000 2008 -------
*** Issue 398 has been marked as a duplicate of this issue. ***

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 682] Share workspace between jobs

hoshposh
In reply to this post by mmatula
https://hudson.dev.java.net/issues/show_bug.cgi?id=682



User hoshposh changed the following:

                What    |Old value                 |New value
================================================================================
                      CC|'akostadinov'             |'akostadinov,hoshposh'
--------------------------------------------------------------------------------




------- Additional comments from [hidden email] Sat Jun 28 05:42:00 +0000 2008 -------
Adding myself on the CC list for this issue.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 682] Share workspace between jobs

lloydchang
In reply to this post by mmatula
https://hudson.dev.java.net/issues/show_bug.cgi?id=682



User lloydchang changed the following:

                What    |Old value                 |New value
================================================================================
                      CC|'akostadinov,hoshposh'    |'akostadinov,hoshposh,lloy
                        |                          |dchang'
--------------------------------------------------------------------------------




------- Additional comments from [hidden email] Sat Jun 28 16:17:12 +0000 2008 -------
Adding myself as a cc

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 682] Share workspace between jobs

jameslivingston
In reply to this post by mmatula
https://hudson.dev.java.net/issues/show_bug.cgi?id=682



User jameslivingston changed the following:

                What    |Old value                 |New value
================================================================================
                      CC|'akostadinov,hoshposh,lloy|'akostadinov,hoshposh,jame
                        |dchang'                   |slivingston,lloydchang'
--------------------------------------------------------------------------------




------- Additional comments from [hidden email] Wed Jul 30 00:19:16 +0000 2008 -------
adding myself as cc

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 682] Share workspace between jobs

pgweiss
In reply to this post by mmatula
https://hudson.dev.java.net/issues/show_bug.cgi?id=682






------- Additional comments from [hidden email] Mon Aug 25 20:53:50 +0000 2008 -------
What would be sufficient for me would not necessarily be the workspace of the
job that triggers me, but rather, the build artifacts.  Even better would be if
I could pass the build artifact URL's as parameters to the triggered job.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 682] Share workspace between jobs

fhoare
In reply to this post by mmatula
https://hudson.dev.java.net/issues/show_bug.cgi?id=682



User fhoare changed the following:

                What    |Old value                 |New value
================================================================================
                      CC|'akostadinov,hoshposh,jame|'akostadinov,fhoare,hoshpo
                        |slivingston,lloydchang'   |sh,jameslivingston,lloydch
                        |                          |ang'
--------------------------------------------------------------------------------




------- Additional comments from [hidden email] Wed Nov  5 10:25:03 +0000 2008 -------
adding myself as cc

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 682] Share workspace between jobs

skaze
In reply to this post by mmatula
https://hudson.dev.java.net/issues/show_bug.cgi?id=682



User skaze changed the following:

                What    |Old value                 |New value
================================================================================
                      CC|'akostadinov,fhoare,hoshpo|'akostadinov,fhoare,hoshpo
                        |sh,jameslivingston,lloydch|sh,jameslivingston,lloydch
                        |ang'                      |ang,skaze'
--------------------------------------------------------------------------------




------- Additional comments from [hidden email] Fri Nov 21 11:15:26 +0000 2008 -------
adding myself to cc

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 682] Share workspace between jobs

skaze
In reply to this post by mmatula
https://hudson.dev.java.net/issues/show_bug.cgi?id=682






------- Additional comments from [hidden email] Fri Nov 21 11:59:43 +0000 2008 -------
i would to describe another scenario where having the ability to share a
workspace between jobs would be useful.

Note i want to keep using the built-in Maven 2 project type and not have to drop
to a 'free-style software project' and use shell scripting to get this job done.

Simply put I need to run maven more than once on the project to finish my build.
The build needs to do the following:

* Run a full build and generate results for all projects (i.e. "mvn install -fae
-Dmaven.test.failure.ignore=true"), yes ignore all failures.

* Run some data collection mojos across the now fully built project hierarchy
and then produce some reports (i.e. "mvn site -fae
-Dmaven.test.failure.ignore=true site:deploy")

Firstly a normal build is run and a number of code tools (checkstyle, pmd,
clover, cobertura, findbugs, javadoc, jxr) do their thing, these are my own
modified versions of the plugins that have all been changed to decouple their
analysis functionality from their reporting functionality. The reason we have
decoupled analysis from reporting is for the next stage.

The second invocation of mvn runs our custom plugins again (their bound to site
and pre-site phases), but this time in 'aggregate' mode. These 'aggregate' mojos
have the job of looking through the module hierarchy and aggregating (pulling up
and merging) the various result files they find (e.g. checkstyle-results.xml) up
the hierarchy, producing new analysis files as they go (i.e. for the parent pom
projects). The result of this is that at every level of the project hierarchy
one can see the aggregated results from JXR, javadoc, checkstyle, pmd, clover,
tests, etc.

Note, this is very different to the standard maven reporting plugin's aggregate
feature (i.e. <javadoc><aggregate>=true), when one uses these standard plugins
in aggregate mode only the top-most project gets the aggregated report, all the
other modules in the hierarchy do not generate any report at all.

Once the pre-site phase has run, the standard site plugin kicks. This in turn
runs all our reporting mojos and we get some rather lovely fully aggregated
multi-tier reports for javadoc, jxr, checkstyle, et al.

Was of doing this kind of thing with Hudson.

1) Prefered option? - Be able to define multiple build actions, i.e. dont chain
jobs but chain multiple build commands. a bit like the batch task functionality.
Thus you are in the same workspace as its the same job. Note this would need
some kind of 'continue if failed' functionality. I can see how this stretches
the hudson job model a bit so may not be viable for design reasons.

2) Call batch task on same project when main build finishes - this is what i'm
trying to do at the moment but unlike the 'Build other projects' post-build
action, the 'Invoke batch tasks of other projects' action does not have a
'Trigger even if the build is unstable' option which would allow us to only call
the batch task if the main build was successful (i.e. if the first mvn run fails
i do not want to call the batch task).

3) use seprate jobs, one that does the initial mvn build and then another, that
is downstream from that, shartes its workspace, and does the data aggregation
and site building. I think this one fits most easily into the hudson model...

Comments welcomed...

John

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

12