Newbie question: overlapping builds

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Newbie question: overlapping builds

tec-3

Hi Folks,

I'd like to configure a job so that it runs only when there has been a
change since the last build, and it also runs when a change has been made
even if another build is already running

The first part is easily handled by svn polling, however I don't know how
to get a second job started when there is already one in process.

My reason for asking is this situation.  The product is fairly mature and
most builds are fine.  The builds take about an hour and svn commits can
happen at any time.  Developers already know from their pre-checkin tests
that their changes work (for them anyway) but they don't always know if
their changes will conflict with something someone else did in another part
of the code.  Near code freeze, developers have to stay around until their
commits pass the integration tests, and if they have to wait until a prior
build finishes before their build will starts, that added delay is a
problem.

The example I found on the wiki about "build 10 - test 10", "build 11 -
test 11" being split up into "build 10", "build 11", "build 12", "build
13", ... and "idle", "test 10", "test 11", "test 12", ... is close to what
I want, except that I'd rather have "build 11" start before "build 10" has
finished.

Is this currently doable with Hudson?

Best Wishes,

     --Tom


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Newbie question: overlapping builds

Matt Tucker-4
Tom Cavin wrote:

> I'd like to configure a job so that it runs only when there has been a
> change since the last build, and it also runs when a change has been
> made even if another build is already running
>
> The first part is easily handled by svn polling, however I don't know
> how to get a second job started when there is already one in process.

Manage Hudson->Configure Executors-># of executors, put in a number
greater than one.

Is that what you mean?

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Newbie question: overlapping builds

Adam Purkiss
That is part of it, however I suspect what is required is not possible and in fact dangerous as you are changing files that are being used. Rather then a single job this might be better as a number of jobs each time shifted by say 15 mins that checks for updates. It does mean you loose some traceability though... so might not be ideal for what you want.




> Date: Thu, 3 Jul 2008 16:43:18 -0700

> From: [hidden email]
> To: [hidden email]
> CC: [hidden email]
> Subject: RE: Newbie question: overlapping builds
>
> Tom Cavin wrote:
>
> > I'd like to configure a job so that it runs only when there has been a
> > change since the last build, and it also runs when a change has been
> > made even if another build is already running
> >
> > The first part is easily handled by svn polling, however I don't know
> > how to get a second job started when there is already one in process.
>
> Manage Hudson->Configure Executors-># of executors, put in a number
> greater than one.
>
> Is that what you mean?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


Are you ready for Windows Live Messenger Beta 8.5 ? Get the latest for free today!
Reply | Threaded
Open this post in threaded view
|

RE: Newbie question: overlapping builds

tec-3

Hi Adam,

How would this be dangerous?

I do understand that having two jobs running on the same host in the same
workspace is a disaster waiting to happen, but that's not what I'm trying
to do.

What I have in mind is this scenario:

     8:47 developer commits a change to svn.
     9:00 Hudson polls svn, detects change, starts job 1 on host 1
     9:12 developer commits a change to svn. (with bug)
     9:15 Hudson polls svn, detects change, starts job 2 on host 2
     9:25 developer commits a change to svn. (fixing bug)
     9:30 Hudson polls svn, detects change and queues job 3
     9:42 developer commits a change to svn.
     9:45 Hudson polls svn, detects change and queues job 4
     9:55 host 1 job 1 completes successfully.
     9:56 Hudson drops job 3 and starts job 4 on host 1
          (host 1 updates changes from jobs 1 through 4)
    10:10 host 2 job 2 fails, Hudson sends failure notice.
    10:51 host 1 job 4 succeeds, Hudson sends recovery notice.

In this scenario the jobs are run independently and the central host keeps
track of the state of the build status by global build number for this job.

Does this seem reasonable?  Is this possible with the current version of
Hudson?  And if yes to both, how do I configure this?

Best Wishes,

     --Tom

Adam Purkiss writes:
 > That is part of it, however I suspect what is required is not possible
 > and in fact dangerous as you are changing files that are being
 > used. Rather then a single job this might be better as a number of jobs
 > each time shifted by say 15 mins that checks for updates. It does mean
 > you loose some traceability though... so might not be ideal for what you
 > want.
 >
 > Date: Thu, 3 Jul 2008 16:43:18 -0700
 > From: [hidden email]
 > To: [hidden email]
 > CC: [hidden email]
 > Subject: RE: Newbie question: overlapping builds
 >
 > > Tom Cavin wrote:
 > > > I'd like to configure a job so that it runs only when there has been
 > > > a change since the last build, and it also runs when a change has
 > > > been made even if another build is already running
 > > >
 > > > The first part is easily handled by svn polling, however I don't
 > > > know how to get a second job started when there is already one in
 > > > process.
 > >
 > > Manage Hudson->Configure Executors-># of executors, put in a number
 > > greater than one.
 > >
 > > Is that what you mean?
 >
 > ---------------------------------------------------------------------
 > To unsubscribe, e-mail: [hidden email]
 > For additional commands, e-mail: [hidden email]
 > _________________________________________________________________
 > Are you ready for Windows Live Messenger Beta 8.5 ? Get the latest for
 > free today!

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Newbie question: overlapping builds

Adam Purkiss
 Ah I thought you meant only Job 1 existed so the second run could possibly overwrite parts of the workspace used in job 1 thus making the results of job 1 invalid. (for example Junit test results from 1 could be overwritten by job 2 before job 1 finishes)
 
But what you are saying is that you want a master job to report back everything but have X jobs  do the work.
 
I cannot think of a way to do exactly what you want, only to have separate jobs time shifted and report back in their own jobs. You might be able to hack something together by having a script to parse you separate jobs and then make a fake hudson job that the results are written to which brings them together but Hudson as far as I am aware cannot do what you want.






> From: [hidden email]

> Date: Mon, 7 Jul 2008 14:08:53 -0400
> To: [hidden email]
> Subject: RE: Newbie question: overlapping builds
>
>
> Hi Adam,
>
> How would this be dangerous?
>
> I do understand that having two jobs running on the same host in the same
> workspace is a disaster waiting to happen, but that's not what I'm trying
> to do.
>
> What I have in mind is this scenario:
>
> 8:47 developer commits a change to svn.
> 9:00 Hudson polls svn, detects change, starts job 1 on host 1
> 9:12 developer commits a change to svn. (with bug)
> 9:15 Hudson polls svn, detects change, starts job 2 on host 2
> 9:25 developer commits a change to svn. (fixing bug)
> 9:30 Hudson polls svn, detects change and queues job 3
> 9:42 developer commits a change to svn.
> 9:45 Hudson polls svn, detects change and queues job 4
> 9:55 host 1 job 1 completes successfully.
> 9:56 Hudson drops job 3 and starts job 4 on host 1
> (host 1 updates changes from jobs 1 through 4)
> 10:10 host 2 job 2 fails, Hudson sends failure notice.
> 10:51 host 1 job 4 succeeds, Hudson sends recovery notice.
>
> In this scenario the jobs are run independently and the central host keeps
> track of the state of the build status by global build number for this job.
>
> Does this seem reasonable? Is this possible with the current version of
> Hudson? And if yes to both, how do I configure this?
>
> Best Wishes,
>
> --Tom
>
> Adam Purkiss writes:
> > That is part of it, however I suspect what is required is not possible
> > and in fact dangerous as you are changing files that are being
> > used. Rather then a single job this might be better as a number of jobs
> > each time shifted by say 15 mins that checks for updates. It does mean
> > you loose some traceability though... so might not be ideal for what you
> > want.
> >
> > Date: Thu, 3 Jul 2008 16:43:18 -0700
> > From: [hidden email]
> > To: [hidden email]
> > CC: [hidden email]
> > Subject: RE: Newbie question: overlapping builds
> >
> > > Tom Cavin wrote:
> > > > I'd like to configure a job so that it runs only when there has been
> > > > a change since the last build, and it also runs when a change has
> > > > been made even if another build is already running
> > > >
> > > > The first part is easily handled by svn polling, however I don't
> > > > know how to get a second job started when there is already one in
> > > > process.
> > >
> > > Manage Hudson->Configure Executors-># of executors, put in a number
> > > greater than one.
> > >
> > > Is that what you mean?
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> > _________________________________________________________________
> > Are you ready for Windows Live Messenger Beta 8.5 ? Get the latest for
> > free today!
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>



Reply | Threaded
Open this post in threaded view
|

RE: Newbie question: overlapping builds

tec-3

Hi Adam,

Many thanks.

Your suggestion of multiple time-skewed jobs seems like it would get me
most of what I want.  The only potential problem is wasted builds when
there is a gap in the commit flow and one of the builders picks up the same
svn version as is currently building on another builder.  That issue might
be addressed by a locking mechanism, but it would have to work in such a
way that a locked job isn't started rather than being aborted.

The hudson status collector job also sounds promising.  Would that be a job
that is downstream of two or more independent jobs?  As any of the upstream
jobs finishes, the collector job would be triggered and would pick up and
display the status of the most recently completed upstream job.  This could
get confused by a later job failing earlier, but again some type of
flagging by version or job number would let this stay sane.  (If job N
reports its results after job N+1, the status display would still reflect
the results from job N+1.)

My general desire in this regard is to get some inexpensive parallelization
in the builds.  I'm working with big projects that take a lot of cycles to
build and even more cycles to test.  I also have cycles available on
multiple hosts.  What I want is to configure Hudson to let me make more
efficient use of my hosts and reduce the time between the moment developers
commit their changes and the moment they know whether their changes worked.

Back on my original idea of allowing overlaping jobs on different hosts:
does that sound generally useful enough to warrant a Hudson feature
request?

Best Wishes,

     --Tom

Adam Purkiss writes:
 >  Ah I thought you meant only Job 1 existed so the second run could
 > possibly overwrite parts of the workspace used in job 1 thus making the
 > results of job 1 invalid. (for example Junit test results from 1 could
 > be overwritten by job 2 before job 1 finishes)
 >  
 > But what you are saying is that you want a master job to report back
 > everything but have X jobs do the work.
 >  
 > I cannot think of a way to do exactly what you want, only to have
 > separate jobs time shifted and report back in their own jobs. You might
 > be able to hack something together by having a script to parse your
 > separate jobs and then make a fake hudson job that the results are
 > written to which brings them together but Hudson as far as I am aware
 > cannot do what you want.
 >
 >
 > Tom Cavin writes:
 > >
 > > Hi Adam,
 > >
 > > How would this be dangerous?
 > >
 > > I do understand that having two jobs running on the same host in the same
 > > workspace is a disaster waiting to happen, but that's not what I'm trying
 > > to do.
 > >
 > > What I have in mind is this scenario:
 > >
 > >      8:47 developer commits a change to svn.
 > >      9:00 Hudson polls svn, detects change, starts job 1 on host 1
 > >      9:12 developer commits a change to svn. (with bug)
 > >      9:15 Hudson polls svn, detects change, starts job 2 on host 2
 > >      9:25 developer commits a change to svn. (fixing bug)
 > >      9:30 Hudson polls svn, detects change and queues job 3
 > >      9:42 developer commits a change to svn.
 > >      9:45 Hudson polls svn, detects change and queues job 4
 > >      9:55 host 1 job 1 completes successfully.
 > >      9:56 Hudson drops job 3 and starts job 4 on host 1
 > >  (host 1 updates changes from jobs 1 through 4)
 > >     10:10 host 2 job 2 fails, Hudson sends failure notice.
 > >     10:51 host 1 job 4 succeeds, Hudson sends recovery notice.
 > >
 > > In this scenario the jobs are run independently and the central host keeps
 > > track of the state of the build status by global build number for this job.
 > >
 > > Does this seem reasonable?  Is this possible with the current version of
 > > Hudson?  And if yes to both, how do I configure this?
 > >
 > > Best Wishes,
 > >
 > >      --Tom
 > >
 > > Adam Purkiss writes:
 > >  > That is part of it, however I suspect what is required is not possible
 > >  > and in fact dangerous as you are changing files that are being
 > >  > used. Rather then a single job this might be better as a number of jobs
 > >  > each time shifted by say 15 mins that checks for updates. It does mean
 > >  > you loose some traceability though... so might not be ideal for what you
 > >  > want.
 > >  >
 > >  > Date: Thu, 3 Jul 2008 16:43:18 -0700
 > >  > From: [hidden email]
 > >  > To: [hidden email]
 > >  > CC: [hidden email]
 > >  > Subject: RE: Newbie question: overlapping builds
 > >  >
 > >  > > Tom Cavin wrote:
 > >  > > > I'd like to configure a job so that it runs only when there has been
 > >  > > > a change since the last build, and it also runs when a change has
 > >  > > > been made even if another build is already running
 > >  > > >
 > >  > > > The first part is easily handled by svn polling, however I don't
 > >  > > > know how to get a second job started when there is already one in
 > >  > > > process.
 > >  > >
 > >  > > Manage Hudson->Configure Executors-># of executors, put in a number
 > >  > > greater than one.
 > >  > >
 > >  > > Is that what you mean?
 > >  >
 > >  > ---------------------------------------------------------------------

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Newbie question: overlapping builds

Adam Purkiss
If it is something you want then I would say yes <insert comment about doing it yourself>  ;)
 
The other thing you could of course do is to have two different jobs if possible. The full set of tests which runs as often as possible and a cut down version which gives developers a sense that they have committed working code. You make the cut down version reduced enough to give a good warm feeling while taking a lot less time.
 
Of course if the developers cannot do anything till the tests finish then maybe the tests should be run locally before committing ?? IIRC that was part of the original email






> From: [hidden email]

> Date: Mon, 7 Jul 2008 15:41:31 -0400
> To: [hidden email]
> Subject: RE: Newbie question: overlapping builds
>
>
> Hi Adam,
>
> Many thanks.
>
> Your suggestion of multiple time-skewed jobs seems like it would get me
> most of what I want. The only potential problem is wasted builds when
> there is a gap in the commit flow and one of the builders picks up the same
> svn version as is currently building on another builder. That issue might
> be addressed by a locking mechanism, but it would have to work in such a
> way that a locked job isn't started rather than being aborted.
>
> The hudson status collector job also sounds promising. Would that be a job
> that is downstream of two or more independent jobs? As any of the upstream
> jobs finishes, the collector job would be triggered and would pick up and
> display the status of the most recently completed upstream job. This could
> get confused by a later job failing earlier, but again some type of
> flagging by version or job number would let this stay sane. (If job N
> reports its results after job N+1, the status display would still reflect
> the results from job N+1.)
>
> My general desire in this regard is to get some inexpensive parallelization
> in the builds. I'm working with big projects that take a lot of cycles to
> build and even more cycles to test. I also have cycles available on
> multiple hosts. What I want is to configure Hudson to let me make more
> efficient use of my hosts and reduce the time between the moment developers
> commit their changes and the moment they know whether their changes worked.
>
> Back on my original idea of allowing overlaping jobs on different hosts:
> does that sound generally useful enough to warrant a Hudson feature
> request?
>
> Best Wishes,
>
> --Tom
>
> Adam Purkiss writes:
> > Ah I thought you meant only Job 1 existed so the second run could
> > possibly overwrite parts of the workspace used in job 1 thus making the
> > results of job 1 invalid. (for example Junit test results from 1 could
> > be overwritten by job 2 before job 1 finishes)
> >
> > But what you are saying is that you want a master job to report back
> > everything but have X jobs do the work.
> >
> > I cannot think of a way to do exactly what you want, only to have
> > separate jobs time shifted and report back in their own jobs. You might
> > be able to hack something together by having a script to parse your
> > separate jobs and then make a fake hudson job that the results are
> > written to which brings them together but Hudson as far as I am aware
> > cannot do what you want.
> >
> >
> > Tom Cavin writes:
> > >
> > > Hi Adam,
> > >
> > > How would this be dangerous?
> > >
> > > I do understand that having two jobs running on the same host in the same
> > > workspace is a disaster waiting to happen, but that's not what I'm trying
> > > to do.
> > >
> > > What I have in mind is this scenario:
> > >
> > > 8:47 developer commits a change to svn.
> > > 9:00 Hudson polls svn, detects change, starts job 1 on host 1
> > > 9:12 developer commits a change to svn. (with bug)
> > > 9:15 Hudson polls svn, detects change, starts job 2 on host 2
> > > 9:25 developer commits a change to svn. (fixing bug)
> > > 9:30 Hudson polls svn, detects change and queues job 3
> > > 9:42 developer commits a change to svn.
> > > 9:45 Hudson polls svn, detects change and queues job 4
> > > 9:55 host 1 job 1 completes successfully.
> > > 9:56 Hudson drops job 3 and starts job 4 on host 1
> > > (host 1 updates changes from jobs 1 through 4)
> > > 10:10 host 2 job 2 fails, Hudson sends failure notice.
> > > 10:51 host 1 job 4 succeeds, Hudson sends recovery notice.
> > >
> > > In this scenario the jobs are run independently and the central host keeps
> > > track of the state of the build status by global build number for this job.
> > >
> > > Does this seem reasonable? Is this possible with the current version of
> > > Hudson? And if yes to both, how do I configure this?
> > >
> > > Best Wishes,
> > >
> > > --Tom
> > >
> > > Adam Purkiss writes:
> > > > That is part of it, however I suspect what is required is not possible
> > > > and in fact dangerous as you are changing files that are being
> > > > used. Rather then a single job this might be better as a number of jobs
> > > > each time shifted by say 15 mins that checks for updates. It does mean
> > > > you loose some traceability though... so might not be ideal for what you
> > > > want.
> > > >
> > > > Date: Thu, 3 Jul 2008 16:43:18 -0700
> > > > From: [hidden email]
> > > > To: [hidden email]
> > > > CC: [hidden email]
> > > > Subject: RE: Newbie question: overlapping builds
> > > >
> > > > > Tom Cavin wrote:
> > > > > > I'd like to configure a job so that it runs only when there has been
> > > > > > a change since the last build, and it also runs when a change has
> > > > > > been made even if another build is already running
> > > > > >
> > > > > > The first part is easily handled by svn polling, however I don't
> > > > > > know how to get a second job started when there is already one in
> > > > > > process.
> > > > >
> > > > > Manage Hudson->Configure Executors-># of executors, put in a number
> > > > > greater than one.
> > > > >
> > > > > Is that what you mean?
> > > >
> > > > ---------------------------------------------------------------------
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>