New approach for SCM polling

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

New approach for SCM polling

Jean-Baptiste Quenot
Dear developers,

I'd like to provide a new feature regarding SCM polling that
would allow the polling to:

* be synchronous, only one thread is involved and polls every
  project in turn
* respect the dependencies expressed between the projects

This is to resolve my #1 concern with Hudson already discussed at
http://www.nabble.com/Commit-spanning-multiple-interdependant-projects-tf3870814.html#a10966678

I wrote a test case to reproduce the current situation, see
hudson/extras/tester/src/test/java/hudson/tasks/SubversionPollingTest.java

Would you mind to review the patch filed under issue #806, in
order to make sure I'm going into the right direction.  The UI is
not connected to this new feature yet.

https://hudson.dev.java.net/issues/show_bug.cgi?id=806

Thanks in advance,
--
     Jean-Baptiste Quenot
aka  John Banana   Qwerty
http://caraldi.com/jbq/

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: New approach for SCM polling

Jean-Baptiste Quenot
I'm realizing  DependencyRunner is completely wrong.   Do you have
an idea how  to get all Hudson projects in  the right dependencies
order, ie the least dependent first?
--
     Jean-Baptiste Quenot
aka  John Banana   Qwerty
http://caraldi.com/jbq/

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: New approach for SCM polling

Kohsuke Kawaguchi
Administrator
In reply to this post by Jean-Baptiste Quenot
Jean-Baptiste Quenot wrote:
> Dear developers,
>
> I'd like to provide a new feature regarding SCM polling that
> would allow the polling to:
>
> * be synchronous, only one thread is involved and polls every
>   project in turn

Polling is carried out by a dedicated Executor, and the number of
threads is already configurable from the system config screen. So you
can just set it to 1 to achieve this effect.


> * respect the dependencies expressed between the projects
>
> This is to resolve my #1 concern with Hudson already discussed at
> http://www.nabble.com/Commit-spanning-multiple-interdependant-projects-tf3870814.html#a10966678

I'm not sure how doing polling in one thread would help you resolve the
issue above. I could be wrong, but it seems like what really matters is
to do the polling of all the projects that share a repository in one
batch, and then based on the dependency information decide what build to
schedule.

I guess even then there's still a difficulty in eliminating the
tentative build failures, as similar issue can still happen across
repositories.



> I wrote a test case to reproduce the current situation, see
> hudson/extras/tester/src/test/java/hudson/tasks/SubversionPollingTest.java
>
> Would you mind to review the patch filed under issue #806, in
> order to make sure I'm going into the right direction.  The UI is
> not connected to this new feature yet.
>
> https://hudson.dev.java.net/issues/show_bug.cgi?id=806

Will do.

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: New approach for SCM polling

Kohsuke Kawaguchi
Administrator
In reply to this post by Jean-Baptiste Quenot
Jean-Baptiste Quenot wrote:
> I'm realizing  DependencyRunner is completely wrong.   Do you have
> an idea how  to get all Hudson projects in  the right dependencies
> order, ie the least dependent first?

You'd need to do topological sort over
AbstractProject.getUpstreamProject() by DFS.

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: New approach for SCM polling

Jean-Baptiste Quenot
In reply to this post by Kohsuke Kawaguchi
* Kohsuke Kawaguchi:

> I'm not  sure how  doing polling  in one  thread would  help you
> resolve the  issue above. I  could be wrong,  but it  seems like
> what really  matters is to  do the  polling of all  the projects
> that share  a repository  in one  batch, and  then based  on the
> dependency information decide what build to schedule.

There's no need to limit this  feature to projects within the same
repository, as you can have  dependencies expressed in Hudson that
span across repositories.

The important  point here is  to trigger  the builds in  the right
order.  So  if the polling is  done in the order  of dependencies,
builds will be scheduled in the right order.

Yesterday I deployed this feature to my production site, and until
now  it behaves  correctly, polling  is done  in the  right order.
Today I'll be taking care of using the Executor thread for polling
instead of hijacking the Cron thread for this.

I think it sounds reasonable  to commit this work-in-progress once
this is  done.  After  that I'll  be able to  wire the  UI screens
accordingly.  BTW does  it make sense to provide  a global crontab
spec setting  in the  Hudson configuration  screen if  user checks
"synchronous polling"?
--
     Jean-Baptiste Quenot
aka  John Banana   Qwerty
http://caraldi.com/jbq/

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: New approach for SCM polling

Jean-Baptiste Quenot
BTW would you mind to comment if it's OK to change "protected void
run()"  to "public  void run()"  in Trigger?   I wouldn't  want to
break the API without notice.
--
     Jean-Baptiste Quenot
aka  John Banana   Qwerty
http://caraldi.com/jbq/

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: New approach for SCM polling

Kohsuke Kawaguchi
Administrator
In reply to this post by Jean-Baptiste Quenot

I'd really appreciate if you can create a branch and commit this change
there, just so that it doesn't interact with the release cycles.

I'm looking forward to seeing the commit.

(And sorry about the delay in Trigger.run())

Jean-Baptiste Quenot wrote:

> * Kohsuke Kawaguchi:
>
>> I'm not  sure how  doing polling  in one  thread would  help you
>> resolve the  issue above. I  could be wrong,  but it  seems like
>> what really  matters is to  do the  polling of all  the projects
>> that share  a repository  in one  batch, and  then based  on the
>> dependency information decide what build to schedule.
>
> There's no need to limit this  feature to projects within the same
> repository, as you can have  dependencies expressed in Hudson that
> span across repositories.
>
> The important  point here is  to trigger  the builds in  the right
> order.  So  if the polling is  done in the order  of dependencies,
> builds will be scheduled in the right order.
>
> Yesterday I deployed this feature to my production site, and until
> now  it behaves  correctly, polling  is done  in the  right order.
> Today I'll be taking care of using the Executor thread for polling
> instead of hijacking the Cron thread for this.
>
> I think it sounds reasonable  to commit this work-in-progress once
> this is  done.  After  that I'll  be able to  wire the  UI screens
> accordingly.  BTW does  it make sense to provide  a global crontab
> spec setting  in the  Hudson configuration  screen if  user checks
> "synchronous polling"?

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: New approach for SCM polling

Jean-Baptiste Quenot
* Kohsuke Kawaguchi:

> I'd really appreciate if you can create a branch and commit this
> change there, just so that  it doesn't interact with the release
> cycles.

Well,   if  it   was   a  big   incompatible   change,  why   not.
But  this   change  should  be  completely   backwards  compatible
(synchronousPolling setting defaults to false).

And there's  the eternal problem of  CVS.  Creating a branch  is a
real PITA.

What about switching  to Mercurial for the  development of Hudson?
I  see that  you are  familiar with  this tool,  so we  could move
forward and use it?  At least branching would not be necessary.
--
     Jean-Baptiste Quenot
aka  John Banana   Qwerty
http://caraldi.com/jbq/

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: New approach for SCM polling

Kohsuke Kawaguchi
Administrator
Jean-Baptiste Quenot wrote:
> * Kohsuke Kawaguchi:
>
>> I'd really appreciate if you can create a branch and commit this
>> change there, just so that  it doesn't interact with the release
>> cycles.
>
> Well,   if  it   was   a  big   incompatible   change,  why   not.
> But  this   change  should  be  completely   backwards  compatible
> (synchronousPolling setting defaults to false).

You know the nature of the change more than I do, but my concern was
mainly bugs introduced by a big change, even if it's meant to be
compatible. If you feel that the impact to the existing codepath is
minimal, then that would make me feel better.

> And there's  the eternal problem of  CVS.  Creating a branch  is a
> real PITA.

True. But I thought it's not too much of a pain as long as changes are
only on small number of files and it has a short lifespan, which seems
to be the case this time.

> What about switching  to Mercurial for the  development of Hudson?
> I  see that  you are  familiar with  this tool,  so we  could move
> forward and use it?  At least branching would not be necessary.

I've spent some time on mercurial, and my personal opinion right now is
that it is even bigger PITA. Mercurial would have been helpful if you
were not a committer and doesn't plan to push back changes to Hudson,
but in this case it wouldn't have really helped.

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment