m2 builds: sometimes some modules aren't triggered

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

m2 builds: sometimes some modules aren't triggered

Rhett Sutphin
Hi,

I have an alpha m2 build which is occasionally not building every  
module.  I haven't been able to figure out what causes it, but  
occasionally modules low in the dependency hierarchy just don't get  
triggered.  The summary of the build lists all the modules correctly,  
but a couple of them will say "(none)" instead of having a link to  
the individual module's build page.  I can file a bug, but since it's  
pretty random, I thought I should send a message to the list first.  
Has anyone else seen this behavior?

Details:

This has happened twice so far, both on the same job.  The first  
time, I was running 1.106 and the second (today) I'm running 1.114.  
I have three m2 jobs in this instance, but the one that's breaking  
has no upstream dependencies on the other two.  Both of the times it  
failed to trigger everything, the build itself was started by a  
subversion change (as are nearly all the builds for this module).  
Forcing a build after the incomplete builds has worked each time.

The dependency structure for the affected modules is like this:  (B  
and L) => C => W.  That is, B and L have no dependencies on each  
other (they are triggered in parallel).  C depends on both of them.  
W depends on C, B, and L (all directly).

I've noticed that C is sometimes triggered by B and sometimes by L.  
I presume Hudson is trying to satisfy all the dependencies before  
triggering C -- maybe there's a race condition in there if they  
finish too close together (or something).  That's one theory.  The  
other is that L, C, and W are all upstream dependencies of other  
modules in another job.  I believe that the downstream components get  
triggered as the build continues, so I wonder if that has something  
to do with it.

Rhett

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: m2 builds: sometimes some modules aren't triggered

Kohsuke Kawaguchi
Administrator
Rhett Sutphin wrote:
> I have an alpha m2 build which is occasionally not building every  
> module.  I haven't been able to figure out what causes it, but  
> occasionally modules low in the dependency hierarchy just don't get  
> triggered.  The summary of the build lists all the modules correctly,  
> but a couple of them will say "(none)" instead of having a link to  
> the individual module's build page.  I can file a bug, but since it's  
> pretty random, I thought I should send a message to the list first.  
> Has anyone else seen this behavior?

I think I saw this happen for a few times, too. So there's something
going on here.

If you run "hudson.maven.MavenBuild.debug=true;" from the scripting
console, it will cause Hudson to report extra information about when
Hudson decides to trigger another module, and when not to.

If this switch is enabled, when you hit the problem the next time, you
can look at the console output of upstream modules (those modules that
are supposed to trigger the missing build) and that should tell us why
it's not working as expected.


> This has happened twice so far, both on the same job.  The first  
> time, I was running 1.106 and the second (today) I'm running 1.114.  
> I have three m2 jobs in this instance, but the one that's breaking  
> has no upstream dependencies on the other two.  Both of the times it  
> failed to trigger everything, the build itself was started by a  
> subversion change (as are nearly all the builds for this module).  
> Forcing a build after the incomplete builds has worked each time.
>
> The dependency structure for the affected modules is like this:  (B  
> and L) => C => W.  That is, B and L have no dependencies on each  
> other (they are triggered in parallel).  C depends on both of them.  
> W depends on C, B, and L (all directly).
>
> I've noticed that C is sometimes triggered by B and sometimes by L.  
> I presume Hudson is trying to satisfy all the dependencies before  
> triggering C -- maybe there's a race condition in there if they  
> finish too close together (or something).
Yes, Hudson makes sure it doesn't trigger C twice, and yes, it's
possible that there's some race condition.

 > That's one theory.  The
> other is that L, C, and W are all upstream dependencies of other  
> modules in another job.  I believe that the downstream components get  
> triggered as the build continues, so I wonder if that has something  
> to do with it.

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: m2 builds: sometimes some modules aren't triggered

Rhett Sutphin
On Jul 10, 2007, at 9:59 AM, Kohsuke Kawaguchi wrote:

> Rhett Sutphin wrote:
>> I have an alpha m2 build which is occasionally not building every  
>> module.  I haven't been able to figure out what causes it, but  
>> occasionally modules low in the dependency hierarchy just don't  
>> get  triggered.  The summary of the build lists all the modules  
>> correctly,  but a couple of them will say "(none)" instead of  
>> having a link to  the individual module's build page.  I can file  
>> a bug, but since it's  pretty random, I thought I should send a  
>> message to the list first.   Has anyone else seen this behavior?
>
> I think I saw this happen for a few times, too. So there's  
> something going on here.
>
> If you run "hudson.maven.MavenBuild.debug=true;" from the scripting  
> console, it will cause Hudson to report extra information about  
> when Hudson decides to trigger another module, and when not to.
>
> If this switch is enabled, when you hit the problem the next time,  
> you can look at the console output of upstream modules (those  
> modules that are supposed to trigger the missing build) and that  
> should tell us why it's not working as expected.

Will do.  Thanks.

Rhett

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: m2 builds: sometimes some modules aren't triggered

Rhett Sutphin
On Jul 10, 2007, at 10:14 AM, Rhett Sutphin wrote:

> On Jul 10, 2007, at 9:59 AM, Kohsuke Kawaguchi wrote:
>> Rhett Sutphin wrote:
>>> I have an alpha m2 build which is occasionally not building  
>>> every  module.  I haven't been able to figure out what causes it,  
>>> but  occasionally modules low in the dependency hierarchy just  
>>> don't get  triggered.  The summary of the build lists all the  
>>> modules correctly,  but a couple of them will say "(none)"  
>>> instead of having a link to  the individual module's build page.  
>>> I can file a bug, but since it's  pretty random, I thought I  
>>> should send a message to the list first.   Has anyone else seen  
>>> this behavior?
>>
>> I think I saw this happen for a few times, too. So there's  
>> something going on here.
>>
>> If you run "hudson.maven.MavenBuild.debug=true;" from the  
>> scripting console, it will cause Hudson to report extra  
>> information about when Hudson decides to trigger another module,  
>> and when not to.
>>
>> If this switch is enabled, when you hit the problem the next time,  
>> you can look at the console output of upstream modules (those  
>> modules that are supposed to trigger the missing build) and that  
>> should tell us why it's not working as expected.

This happened again just now and I got the debug output.  You'll  
recall that the dependency structure is like this: (B and L) => C =>  
W.  These modules are all parts of a library called ctms-commons; B =  
ctms-commons-base, L = lang, C = core, and W = web.

Here's the end of the log for L (ctms-commons-lang):

[INFO]  
------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO]  
------------------------------------------------------------------------
[INFO] Total time: 51 seconds
[INFO] Finished at: Wed Jul 18 11:09:49 CDT 2007
[INFO] Final Memory: 11M/40M
[INFO]  
------------------------------------------------------------------------
<snipped outside dependents that are never triggered here>
Considering whether to trigger hudson.maven.MavenModule@ef553d[CTMS  
Commons/gov.nih.nci.cabig.ctms:ctms-commons-core] or not
  -> No, because another upstream hudson.maven.MavenModule@7286ca
[CTMS Commons/gov.nih.nci.cabig.ctms:ctms-commons-base] for  
hudson.maven.MavenModule@ef553d[CTMS Commons/
gov.nih.nci.cabig.ctms:ctms-commons-core] is building
Considering whether to trigger hudson.maven.MavenModule@724da1[CTMS  
Commons/gov.nih.nci.cabig.ctms:ctms-commons-web] or not
  -> No, because there's a longer dependency path
finished: SUCCESS

And here's B (ctms-commons-base):

[INFO]  
------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO]  
------------------------------------------------------------------------
[INFO] Total time: 42 seconds
[INFO] Finished at: Wed Jul 18 11:09:33 CDT 2007
[INFO] Final Memory: 11M/33M
[INFO]  
------------------------------------------------------------------------
Considering whether to trigger hudson.maven.MavenModule@ef553d[CTMS  
Commons/gov.nih.nci.cabig.ctms:ctms-commons-core] or not
  -> No, because another upstream hudson.maven.MavenModule@c51ef0
[CTMS Commons/gov.nih.nci.cabig.ctms:ctms-commons-lang] for  
hudson.maven.MavenModule@ef553d[CTMS Commons/
gov.nih.nci.cabig.ctms:ctms-commons-core] is building
Considering whether to trigger hudson.maven.MavenModule@724da1[CTMS  
Commons/gov.nih.nci.cabig.ctms:ctms-commons-web] or not
  -> No, because there's a longer dependency path
finished: SUCCESS

So it appears it is a race condition of some kind -- each build  
thinks the other is still in progress, although B finishes 16 seconds  
before L.

I've filed a bug for this:  https://hudson.dev.java.net/issues/ 
show_bug.cgi?id=647

Rhett

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: m2 builds: sometimes some modules aren't triggered

Nigel Magnay
I've just had a thought - I see this a fair bit, and we have builds that sometimes run for a reasonable length of time - sometimes a build is triggered half-way through a build that is currently in progress. Does the scheduler suspend builds while builds are in progress ?

On 18/07/07, Rhett Sutphin <[hidden email]> wrote:
On Jul 10, 2007, at 10:14 AM, Rhett Sutphin wrote:

> On Jul 10, 2007, at 9:59 AM, Kohsuke Kawaguchi wrote:
>> Rhett Sutphin wrote:
>>> I have an alpha m2 build which is occasionally not building
>>> every  module.  I haven't been able to figure out what causes it,
>>> but  occasionally modules low in the dependency hierarchy just
>>> don't get  triggered.  The summary of the build lists all the
>>> modules correctly,  but a couple of them will say "(none)"
>>> instead of having a link to  the individual module's build page.
>>> I can file a bug, but since it's  pretty random, I thought I
>>> should send a message to the list first.   Has anyone else seen
>>> this behavior?
>>
>> I think I saw this happen for a few times, too. So there's
>> something going on here.
>>
>> If you run "hudson.maven.MavenBuild.debug=true;" from the
>> scripting console, it will cause Hudson to report extra
>> information about when Hudson decides to trigger another module,
>> and when not to.
>>
>> If this switch is enabled, when you hit the problem the next time,
>> you can look at the console output of upstream modules (those
>> modules that are supposed to trigger the missing build) and that
>> should tell us why it's not working as expected.

This happened again just now and I got the debug output.  You'll
recall that the dependency structure is like this: (B and L) => C =>
W.  These modules are all parts of a library called ctms-commons; B =
ctms-commons-base, L = lang, C = core, and W = web.

Here's the end of the log for L (ctms-commons-lang):

[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 51 seconds
[INFO] Finished at: Wed Jul 18 11:09:49 CDT 2007
[INFO] Final Memory: 11M/40M
[INFO]
------------------------------------------------------------------------
<snipped outside dependents that are never triggered here>
Considering whether to trigger hudson.maven.MavenModule@ef553d[CTMS
Commons/gov.nih.nci.cabig.ctms:ctms-commons-core] or not
  -> No, because another upstream hudson.maven.MavenModule@7286ca
[CTMS Commons/gov.nih.nci.cabig.ctms:ctms-commons-base] for
hudson.maven.MavenModule@ef553d [CTMS Commons/
gov.nih.nci.cabig.ctms:ctms-commons-core] is building
Considering whether to trigger hudson.maven.MavenModule@724da1[CTMS
Commons/gov.nih.nci.cabig.ctms:ctms-commons-web] or not
  -> No, because there's a longer dependency path
finished: SUCCESS

And here's B (ctms-commons-base):

[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 42 seconds
[INFO] Finished at: Wed Jul 18 11:09:33 CDT 2007
[INFO] Final Memory: 11M/33M
[INFO]
------------------------------------------------------------------------
Considering whether to trigger hudson.maven.MavenModule@ef553d[CTMS
Commons/gov.nih.nci.cabig.ctms:ctms-commons-core] or not
  -> No, because another upstream hudson.maven.MavenModule@c51ef0
[CTMS Commons/gov.nih.nci.cabig.ctms:ctms-commons-lang] for
hudson.maven.MavenModule@ef553d[CTMS Commons/
gov.nih.nci.cabig.ctms:ctms-commons-core] is building
Considering whether to trigger hudson.maven.MavenModule@724da1[CTMS
Commons/gov.nih.nci.cabig.ctms:ctms-commons-web] or not
  -> No, because there's a longer dependency path
finished: SUCCESS

So it appears it is a race condition of some kind -- each build
thinks the other is still in progress, although B finishes 16 seconds
before L.

I've filed a bug for this:  https://hudson.dev.java.net/issues/
show_bug.cgi?id=647

Rhett

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: m2 builds: sometimes some modules aren't triggered

Kohsuke Kawaguchi
Administrator
Nigel Magnay wrote:
> I've just had a thought - I see this a fair bit, and we have builds that
> sometimes run for a reasonable length of time - sometimes a build is
> triggered half-way through a build that is currently in progress. Does the
> scheduler suspend builds while builds are in progress ?

I'm not sure if I'm completely following your question, but Hudson makes
sure that no two builds of the same module will run at the same time.

That is, if two builds are run in parallel that works on the same
workspace, you'll see all kinds of weird problems. Hudson doesn't do that.

>
> On 18/07/07, Rhett Sutphin <[hidden email]> wrote:
>>
>> On Jul 10, 2007, at 10:14 AM, Rhett Sutphin wrote:
>> > On Jul 10, 2007, at 9:59 AM, Kohsuke Kawaguchi wrote:
>> >> Rhett Sutphin wrote:
>> >>> I have an alpha m2 build which is occasionally not building
>> >>> every  module.  I haven't been able to figure out what causes it,
>> >>> but  occasionally modules low in the dependency hierarchy just
>> >>> don't get  triggered.  The summary of the build lists all the
>> >>> modules correctly,  but a couple of them will say "(none)"
>> >>> instead of having a link to  the individual module's build page.
>> >>> I can file a bug, but since it's  pretty random, I thought I
>> >>> should send a message to the list first.   Has anyone else seen
>> >>> this behavior?
>> >>
>> >> I think I saw this happen for a few times, too. So there's
>> >> something going on here.
>> >>
>> >> If you run "hudson.maven.MavenBuild.debug=true;" from the
>> >> scripting console, it will cause Hudson to report extra
>> >> information about when Hudson decides to trigger another module,
>> >> and when not to.
>> >>
>> >> If this switch is enabled, when you hit the problem the next time,
>> >> you can look at the console output of upstream modules (those
>> >> modules that are supposed to trigger the missing build) and that
>> >> should tell us why it's not working as expected.
>>
>> This happened again just now and I got the debug output.  You'll
>> recall that the dependency structure is like this: (B and L) => C =>
>> W.  These modules are all parts of a library called ctms-commons; B =
>> ctms-commons-base, L = lang, C = core, and W = web.
>>
>> Here's the end of the log for L (ctms-commons-lang):
>>
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] BUILD SUCCESSFUL
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Total time: 51 seconds
>> [INFO] Finished at: Wed Jul 18 11:09:49 CDT 2007
>> [INFO] Final Memory: 11M/40M
>> [INFO]
>> ------------------------------------------------------------------------
>> <snipped outside dependents that are never triggered here>
>> Considering whether to trigger hudson.maven.MavenModule@ef553d[CTMS
>> Commons/gov.nih.nci.cabig.ctms:ctms-commons-core] or not
>>   -> No, because another upstream hudson.maven.MavenModule@7286ca
>> [CTMS Commons/gov.nih.nci.cabig.ctms:ctms-commons-base] for
>> hudson.maven.MavenModule@ef553d[CTMS Commons/
>> gov.nih.nci.cabig.ctms:ctms-commons-core] is building
>> Considering whether to trigger hudson.maven.MavenModule@724da1[CTMS
>> Commons/gov.nih.nci.cabig.ctms:ctms-commons-web] or not
>>   -> No, because there's a longer dependency path
>> finished: SUCCESS
>>
>> And here's B (ctms-commons-base):
>>
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] BUILD SUCCESSFUL
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Total time: 42 seconds
>> [INFO] Finished at: Wed Jul 18 11:09:33 CDT 2007
>> [INFO] Final Memory: 11M/33M
>> [INFO]
>> ------------------------------------------------------------------------
>> Considering whether to trigger hudson.maven.MavenModule@ef553d[CTMS
>> Commons/gov.nih.nci.cabig.ctms:ctms-commons-core] or not
>>   -> No, because another upstream hudson.maven.MavenModule@c51ef0
>> [CTMS Commons/gov.nih.nci.cabig.ctms:ctms-commons-lang] for
>> hudson.maven.MavenModule@ef553d[CTMS Commons/
>> gov.nih.nci.cabig.ctms:ctms-commons-core] is building
>> Considering whether to trigger hudson.maven.MavenModule@724da1[CTMS
>> Commons/gov.nih.nci.cabig.ctms:ctms-commons-web] or not
>>   -> No, because there's a longer dependency path
>> finished: SUCCESS
>>
>> So it appears it is a race condition of some kind -- each build
>> thinks the other is still in progress, although B finishes 16 seconds
>> before L.
>>
>> I've filed a bug for this:  https://hudson.dev.java.net/issues/
>> show_bug.cgi?id=647
>>
>> Rhett
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: m2 builds: sometimes some modules aren't triggered

Kohsuke Kawaguchi
Administrator
In reply to this post by Rhett Sutphin
Rhett Sutphin wrote:
> So it appears it is a race condition of some kind -- each build  
> thinks the other is still in progress, although B finishes 16 seconds  
> before L.

Thanks. This is very useful. I'm bit uncomfortable with this 16 seconds
difference, but you are right that if both B and L are built at the same
time time and they are each considering to trigger downstream modules at
the same time, then it will have this exact problem.

It's bit like playing a doubles tennis game and two players each assume
that the other one will handle the ball, only to see the ball go right
through them.

> I've filed a bug for this:  https://hudson.dev.java.net/issues/ 
> show_bug.cgi?id=647

Thanks. Now the tricky part is to figure out how to fix this...


--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment