Custom build order

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Custom build order

Matt Tucker-4
Custom build order

I've got a bunch of projects in Hudson now, and I'm using the Ivy plugin to cause projects to rebuild when upstream projects are rebuilt. What I'm seeing now is that a lot of projects get rebuilt multiple times because of a scenario like this:

        C -> depends on -> B -> depends on -> A
        C -> depends on -> A

There are much more complicated examples, but this is a pretty simple one.  What happens is that A rebuilds, which triggers rebuilds of B,C. Then C rebuilds first, after which B rebuilds, which triggers another rebuild of C.

I could alleviate this somewhat by controlling what order projects are put into the build queue, but what I really want is some kind of hook into the ordering of the build queue, as well as control over when projects are pulled out of it.  For instance, if a space comes available, I'd like to have the queue choose B instead of C, and if B is building and a space becomes available, I'd like to have it block the start of the C build until B finishes.

It doesn't look like there's any provision currently for custom build ordering and handling. Am I missing something?  If not, is there any thought to provide for this kind of functionality?


Reply | Threaded
Open this post in threaded view
|

AW: Custom build order

Martin Ficker
Custom build order
Hello,
 
I use Ivy&Hudson myself and have got the same issue. But things are getting better:
 
If A Project triggers multiple builds it does so in dependency order (in your example a would trigger B first  then C). If you have got a single executor this reduces the time of builds.
This had a bug before and should work correct since 1.228.
I' testing a solution locally that's quite simple:
I changed Project.isBuildBlocked to check if any upstream project is still in the queue. I could publish the patch, but since it could lead to builds waiting forever in some envvironments it can not
be commited at the moment.
I asked for a more plugable Build Queue that would allow plugins to controle the order, but there hasn't been much response yet.
 
Best regards,
 
Martin
 


Von: Matt Tucker [mailto:[hidden email]]
Gesendet: Dienstag, 24. Juni 2008 03:28
An: [hidden email]
Betreff: Custom build order

I've got a bunch of projects in Hudson now, and I'm using the Ivy plugin to cause projects to rebuild when upstream projects are rebuilt. What I'm seeing now is that a lot of projects get rebuilt multiple times because of a scenario like this:

        C -> depends on -> B -> depends on -> A
        C -> depends on -> A

There are much more complicated examples, but this is a pretty simple one.  What happens is that A rebuilds, which triggers rebuilds of B,C. Then C rebuilds first, after which B rebuilds, which triggers another rebuild of C.

I could alleviate this somewhat by controlling what order projects are put into the build queue, but what I really want is some kind of hook into the ordering of the build queue, as well as control over when projects are pulled out of it.  For instance, if a space comes available, I'd like to have the queue choose B instead of C, and if B is building and a space becomes available, I'd like to have it block the start of the C build until B finishes.

It doesn't look like there's any provision currently for custom build ordering and handling. Am I missing something?  If not, is there any thought to provide for this kind of functionality?


Reply | Threaded
Open this post in threaded view
|

RE: AW: Custom build order

Matt Tucker-4
Custom build order
Certainly for the simple case it ends up building in the correct order, as long as you only have one build queue.
 
But we actually have multiple build queues, and we don't always have the simple case.  There are situations where a project depends on another through multiple paths, and a number of situations that, even with a single queue, would I believe lead to a project rebuilding unnecessarily several times.
 
I could possibly see the Project.isBuildBlocked helping, but I'm curious about the concern that there are situations where a build could wait forever in some environments.  What's the repro case for that?


From: Martin Eigenbrodt [mailto:[hidden email]]
Sent: Monday, June 23, 2008 11:46 PM
To: [hidden email]
Subject: AW: Custom build order

Hello,
 
I use Ivy&Hudson myself and have got the same issue. But things are getting better:
 
If A Project triggers multiple builds it does so in dependency order (in your example a would trigger B first  then C). If you have got a single executor this reduces the time of builds.
This had a bug before and should work correct since 1.228.
I' testing a solution locally that's quite simple:
I changed Project.isBuildBlocked to check if any upstream project is still in the queue. I could publish the patch, but since it could lead to builds waiting forever in some envvironments it can not
be commited at the moment.
I asked for a more plugable Build Queue that would allow plugins to controle the order, but there hasn't been much response yet.
 
Best regards,
 
Martin
 


Von: Matt Tucker [mailto:[hidden email]]
Gesendet: Dienstag, 24. Juni 2008 03:28
An: [hidden email]
Betreff: Custom build order

I've got a bunch of projects in Hudson now, and I'm using the Ivy plugin to cause projects to rebuild when upstream projects are rebuilt. What I'm seeing now is that a lot of projects get rebuilt multiple times because of a scenario like this:

        C -> depends on -> B -> depends on -> A
        C -> depends on -> A

There are much more complicated examples, but this is a pretty simple one.  What happens is that A rebuilds, which triggers rebuilds of B,C. Then C rebuilds first, after which B rebuilds, which triggers another rebuild of C.

I could alleviate this somewhat by controlling what order projects are put into the build queue, but what I really want is some kind of hook into the ordering of the build queue, as well as control over when projects are pulled out of it.  For instance, if a space comes available, I'd like to have the queue choose B instead of C, and if B is building and a space becomes available, I'd like to have it block the start of the C build until B finishes.

It doesn't look like there's any provision currently for custom build ordering and handling. Am I missing something?  If not, is there any thought to provide for this kind of functionality?


Reply | Threaded
Open this post in threaded view
|

AW: AW: Custom build order

Martin Ficker
Custom build order
Hi
If   A -> B -> C and you keep triggering builds for A , C will never be build. It is always blocked by the  build for A. One could argue this is the correct behaviour since C relies on the latest build of B and A... If your Builds are triggered infrequent or are quite fast this
woun't be a issue. I'll publish the patch as I find time.


Von: Matt Tucker [mailto:[hidden email]]
Gesendet: Dienstag, 24. Juni 2008 08:52
An: [hidden email]
Betreff: RE: AW: Custom build order

Certainly for the simple case it ends up building in the correct order, as long as you only have one build queue.
 
But we actually have multiple build queues, and we don't always have the simple case.  There are situations where a project depends on another through multiple paths, and a number of situations that, even with a single queue, would I believe lead to a project rebuilding unnecessarily several times.
 
I could possibly see the Project.isBuildBlocked helping, but I'm curious about the concern that there are situations where a build could wait forever in some environments.  What's the repro case for that?


From: Martin Eigenbrodt [mailto:[hidden email]]
Sent: Monday, June 23, 2008 11:46 PM
To: [hidden email]
Subject: AW: Custom build order

Hello,
 
I use Ivy&Hudson myself and have got the same issue. But things are getting better:
 
If A Project triggers multiple builds it does so in dependency order (in your example a would trigger B first  then C). If you have got a single executor this reduces the time of builds.
This had a bug before and should work correct since 1.228.
I' testing a solution locally that's quite simple:
I changed Project.isBuildBlocked to check if any upstream project is still in the queue. I could publish the patch, but since it could lead to builds waiting forever in some envvironments it can not
be commited at the moment.
I asked for a more plugable Build Queue that would allow plugins to controle the order, but there hasn't been much response yet.
 
Best regards,
 
Martin
 


Von: Matt Tucker [mailto:[hidden email]]
Gesendet: Dienstag, 24. Juni 2008 03:28
An: [hidden email]
Betreff: Custom build order

I've got a bunch of projects in Hudson now, and I'm using the Ivy plugin to cause projects to rebuild when upstream projects are rebuilt. What I'm seeing now is that a lot of projects get rebuilt multiple times because of a scenario like this:

        C -> depends on -> B -> depends on -> A
        C -> depends on -> A

There are much more complicated examples, but this is a pretty simple one.  What happens is that A rebuilds, which triggers rebuilds of B,C. Then C rebuilds first, after which B rebuilds, which triggers another rebuild of C.

I could alleviate this somewhat by controlling what order projects are put into the build queue, but what I really want is some kind of hook into the ordering of the build queue, as well as control over when projects are pulled out of it.  For instance, if a space comes available, I'd like to have the queue choose B instead of C, and if B is building and a space becomes available, I'd like to have it block the start of the C build until B finishes.

It doesn't look like there's any provision currently for custom build ordering and handling. Am I missing something?  If not, is there any thought to provide for this kind of functionality?


Reply | Threaded
Open this post in threaded view
|

AW: AW: Custom build order

Martin Ficker
Custom build order
submitted Patch as #1938
https://hudson.dev.java.net/issues/show_bug.cgi?id=1938


Von: Martin Eigenbrodt [mailto:[hidden email]]
Gesendet: Dienstag, 24. Juni 2008 09:26
An: [hidden email]
Betreff: AW: AW: Custom build order

Hi
If   A -> B -> C and you keep triggering builds for A , C will never be build. It is always blocked by the  build for A. One could argue this is the correct behaviour since C relies on the latest build of B and A... If your Builds are triggered infrequent or are quite fast this
woun't be a issue. I'll publish the patch as I find time.


Von: Matt Tucker [mailto:[hidden email]]
Gesendet: Dienstag, 24. Juni 2008 08:52
An: [hidden email]
Betreff: RE: AW: Custom build order

Certainly for the simple case it ends up building in the correct order, as long as you only have one build queue.
 
But we actually have multiple build queues, and we don't always have the simple case.  There are situations where a project depends on another through multiple paths, and a number of situations that, even with a single queue, would I believe lead to a project rebuilding unnecessarily several times.
 
I could possibly see the Project.isBuildBlocked helping, but I'm curious about the concern that there are situations where a build could wait forever in some environments.  What's the repro case for that?


From: Martin Eigenbrodt [mailto:[hidden email]]
Sent: Monday, June 23, 2008 11:46 PM
To: [hidden email]
Subject: AW: Custom build order

Hello,
 
I use Ivy&Hudson myself and have got the same issue. But things are getting better:
 
If A Project triggers multiple builds it does so in dependency order (in your example a would trigger B first  then C). If you have got a single executor this reduces the time of builds.
This had a bug before and should work correct since 1.228.
I' testing a solution locally that's quite simple:
I changed Project.isBuildBlocked to check if any upstream project is still in the queue. I could publish the patch, but since it could lead to builds waiting forever in some envvironments it can not
be commited at the moment.
I asked for a more plugable Build Queue that would allow plugins to controle the order, but there hasn't been much response yet.
 
Best regards,
 
Martin
 


Von: Matt Tucker [mailto:[hidden email]]
Gesendet: Dienstag, 24. Juni 2008 03:28
An: [hidden email]
Betreff: Custom build order

I've got a bunch of projects in Hudson now, and I'm using the Ivy plugin to cause projects to rebuild when upstream projects are rebuilt. What I'm seeing now is that a lot of projects get rebuilt multiple times because of a scenario like this:

        C -> depends on -> B -> depends on -> A
        C -> depends on -> A

There are much more complicated examples, but this is a pretty simple one.  What happens is that A rebuilds, which triggers rebuilds of B,C. Then C rebuilds first, after which B rebuilds, which triggers another rebuild of C.

I could alleviate this somewhat by controlling what order projects are put into the build queue, but what I really want is some kind of hook into the ordering of the build queue, as well as control over when projects are pulled out of it.  For instance, if a space comes available, I'd like to have the queue choose B instead of C, and if B is building and a space becomes available, I'd like to have it block the start of the C build until B finishes.

It doesn't look like there's any provision currently for custom build ordering and handling. Am I missing something?  If not, is there any thought to provide for this kind of functionality?