Hudson and Dependency Management with Freestyle Projects

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Hudson and Dependency Management with Freestyle Projects

matt.dimich-2

We used to use the “Build Other  Projects” trigger after our utility projects to create in essence, a “chain” of dependencies that would get fired off every time an upstream project was built.  Our build chain became too big however, causing forced builds of projects that didn’t need to build.  I’ve been searching for a while on how Hudson could do this, but I haven’t found anything yet.  Has anyone requested or talked about making that “Build Other Projects” feature into a sort of dependency managing feature?  What we are looking for is if an upstream project changes, it would notify the downstream projects they should build, but only if they “have” to.  Then they would kick off a recursive similar process for each of their downstream dependencies.  Likewise, if a downstream project changed, it could look upstream first to make sure no upstream projects have changed.  Once they have completed building, then the current project would build.

 

Has anyone talked about this yet?  Is this something that could be put in the plans?

 

Thanks!

Matt

 

Software Engineer

Thomson Reuters

 

Reply | Threaded
Open this post in threaded view
|

Re: Hudson and Dependency Management with Freestyle Projects

Jon Schewe
[hidden email] wrote:

We used to use the “Build Other  Projects” trigger after our utility projects to create in essence, a “chain” of dependencies that would get fired off every time an upstream project was built.  Our build chain became too big however, causing forced builds of projects that didn’t need to build.  I’ve been searching for a while on how Hudson could do this, but I haven’t found anything yet.  Has anyone requested or talked about making that “Build Other Projects” feature into a sort of dependency managing feature?  What we are looking for is if an upstream project changes, it would notify the downstream projects they should build, but only if they “have” to.  Then they would kick off a recursive similar process for each of their downstream dependencies.  Likewise, if a downstream project changed, it could look upstream first to make sure no upstream projects have changed.  Once they have completed building, then the current project would build.

 

Has anyone talked about this yet?  Is this something that could be put in the plans?

I would think that you could enhance the build target of the downstream projects to do the check against the upstream job and just return success if a build is not required.

-- 
Jon Schewe | http://mtu.net/~jpschewe
If you see an attachment named signature.asc, this is my digital
signature. See http://www.gnupg.org for more information.

For I am convinced that neither death nor life, neither angels nor
demons, neither the present nor the future, nor any powers,
neither height nor depth, nor anything else in all creation, will
be able to separate us from the love of God that is in Christ
Jesus our Lord. - Romans 8:38-39
Reply | Threaded
Open this post in threaded view
|

Re: Hudson and Dependency Management with Freestyle Projects

Sami Tikka
In reply to this post by matt.dimich-2
I'm a bit confused. If a projects depends on other projects, I think
it is a very good idea to build it every time any one of its
dependencies is built.

You have projects that are triggering other projects but they do not
have a dependency relationship? How would Hudson be able to determine
when should a downstream project to be triggered?

If you need multiple projects all to finish building before triggering
a downstream build, the new Join plugin can do it for you.

Or are you actually trying to solve a different problem? Is your
problem maybe that builds are taking a long time? When you have a
large and complex project tree, any commit in a project near the root
of the tree causes a lot of builds to percolate down the tree. Active
development (== frequent commits) near the root would cause Hudson to
be busy for a long time doing builds that are not maybe all very
important.

We also have a lot of projects and the graph of all projects is
getting pretty complex. When someone commits a small change to our
testing tools project, it causes all the other projects to build and
runs all the tests. It is true this seems very "heavy", but we believe
a full build and test cycle is needed after every commit, no matter
how small it is. Just try to keep the builds and tests as light as
possible.

-- Sami

2009/7/7  <[hidden email]>:

> We used to use the “Build Other  Projects” trigger after our utility
> projects to create in essence, a “chain” of dependencies that would get
> fired off every time an upstream project was built.  Our build chain became
> too big however, causing forced builds of projects that didn’t need to
> build.  I’ve been searching for a while on how Hudson could do this, but I
> haven’t found anything yet.  Has anyone requested or talked about making
> that “Build Other Projects” feature into a sort of dependency managing
> feature?  What we are looking for is if an upstream project changes, it
> would notify the downstream projects they should build, but only if they
> “have” to.  Then they would kick off a recursive similar process for each of
> their downstream dependencies.  Likewise, if a downstream project changed,
> it could look upstream first to make sure no upstream projects have
> changed.  Once they have completed building, then the current project would
> build.
>
>
>
> Has anyone talked about this yet?  Is this something that could be put in
> the plans?
>
>
>
> Thanks!
>
> Matt
>
>
>
> Software Engineer
>
> Thomson Reuters
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Re: Hudson and Dependency Management with Freestyle Projects

matt.dimich-2
In reply to this post by Jon Schewe

Ok.  It sounds like this hasn’t been done before, but your suggestion sounds like what I’m looking for. 

 

Thanks for the info.

 

--------

Matt Dimich

 

From: Jon Schewe [mailto:[hidden email]]
Sent: Tuesday, July 07, 2009 2:40 PM
To: [hidden email]
Subject: Re: Hudson and Dependency Management with Freestyle Projects

 

[hidden email] wrote:

We used to use the “Build Other  Projects” trigger after our utility projects to create in essence, a “chain” of dependencies that would get fired off every time an upstream project was built.  Our build chain became too big however, causing forced builds of projects that didn’t need to build.  I’ve been searching for a while on how Hudson could do this, but I haven’t found anything yet.  Has anyone requested or talked about making that “Build Other Projects” feature into a sort of dependency managing feature?  What we are looking for is if an upstream project changes, it would notify the downstream projects they should build, but only if they “have” to.  Then they would kick off a recursive similar process for each of their downstream dependencies.  Likewise, if a downstream project changed, it could look upstream first to make sure no upstream projects have changed.  Once they have completed building, then the current project would build.

 

Has anyone talked about this yet?  Is this something that could be put in the plans?

I would think that you could enhance the build target of the downstream projects to do the check against the upstream job and just return success if a build is not required.


-- 
Jon Schewe | http://mtu.net/~jpschewe
If you see an attachment named signature.asc, this is my digital
signature. See http://www.gnupg.org for more information.
 
For I am convinced that neither death nor life, neither angels nor
demons, neither the present nor the future, nor any powers,
neither height nor depth, nor anything else in all creation, will
be able to separate us from the love of God that is in Christ
Jesus our Lord. - Romans 8:38-39
Reply | Threaded
Open this post in threaded view
|

RE: Re: Hudson and Dependency Management with Freestyle Projects

matt.dimich-2
In reply to this post by Sami Tikka
You are exactly correct that the problem we are trying to solve is builds that take too long.  We have two versions of our build, one is a monolithic giant that takes a very long time because it builds everything.  When we started using Hudson, we broke that build up and made all of the utility jars into their own projects.  I understand the view that you take about every commit should be tested, and I agree to a point, but for us, each commit should be tested by that project, not necessarily by the projects that use that particular jar.  Basically, if only jar 1 changes, but jars 2-10 use jar 1, we don't want to build jars 2-10 just because jar 1 changed.  If jars 2-10 change, we don't want to rebuild jar 1.  We have a lot of projects on our instance of Hudson and so keeping excessive builds down is important.

Thanks for the thoughts!

--------
Matt Dimich


-----Original Message-----
From: Sami Tikka [mailto:[hidden email]]
Sent: Tuesday, July 07, 2009 3:01 PM
To: [hidden email]
Subject: Re: Hudson and Dependency Management with Freestyle Projects

I'm a bit confused. If a projects depends on other projects, I think
it is a very good idea to build it every time any one of its
dependencies is built.

You have projects that are triggering other projects but they do not
have a dependency relationship? How would Hudson be able to determine
when should a downstream project to be triggered?

If you need multiple projects all to finish building before triggering
a downstream build, the new Join plugin can do it for you.

Or are you actually trying to solve a different problem? Is your
problem maybe that builds are taking a long time? When you have a
large and complex project tree, any commit in a project near the root
of the tree causes a lot of builds to percolate down the tree. Active
development (== frequent commits) near the root would cause Hudson to
be busy for a long time doing builds that are not maybe all very
important.

We also have a lot of projects and the graph of all projects is
getting pretty complex. When someone commits a small change to our
testing tools project, it causes all the other projects to build and
runs all the tests. It is true this seems very "heavy", but we believe
a full build and test cycle is needed after every commit, no matter
how small it is. Just try to keep the builds and tests as light as
possible.

-- Sami

2009/7/7  <[hidden email]>:

> We used to use the "Build Other  Projects" trigger after our utility
> projects to create in essence, a "chain" of dependencies that would get
> fired off every time an upstream project was built.  Our build chain became
> too big however, causing forced builds of projects that didn't need to
> build.  I've been searching for a while on how Hudson could do this, but I
> haven't found anything yet.  Has anyone requested or talked about making
> that "Build Other Projects" feature into a sort of dependency managing
> feature?  What we are looking for is if an upstream project changes, it
> would notify the downstream projects they should build, but only if they
> "have" to.  Then they would kick off a recursive similar process for each of
> their downstream dependencies.  Likewise, if a downstream project changed,
> it could look upstream first to make sure no upstream projects have
> changed.  Once they have completed building, then the current project would
> build.
>
>
>
> Has anyone talked about this yet?  Is this something that could be put in
> the plans?
>
>
>
> Thanks!
>
> Matt
>
>
>
> Software Engineer
>
> Thomson Reuters
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

AW: RE: Re: Hudson and Dependency Management with Freestyle Projects

Jan.Materne
Dont forget: if jar-1 changes it could break the jar-n projects because of
API changes (e.g. removed deprecated methods which are used by them).
See: http://gump.apache.org/status.html#Recent+success+stories

Jan

>-----Ursprüngliche Nachricht-----
>Von: [hidden email]
>[mailto:[hidden email]]
>Gesendet: Mittwoch, 8. Juli 2009 15:00
>An: [hidden email]
>Betreff: RE: Re: Hudson and Dependency Management with
>Freestyle Projects
>
>You are exactly correct that the problem we are trying to
>solve is builds that take too long.  We have two versions of
>our build, one is a monolithic giant that takes a very long
>time because it builds everything.  When we started using
>Hudson, we broke that build up and made all of the utility
>jars into their own projects.  I understand the view that you
>take about every commit should be tested, and I agree to a
>point, but for us, each commit should be tested by that
>project, not necessarily by the projects that use that
>particular jar.  Basically, if only jar 1 changes, but jars
>2-10 use jar 1, we don't want to build jars 2-10 just because
>jar 1 changed.  If jars 2-10 change, we don't want to rebuild
>jar 1.  We have a lot of projects on our instance of Hudson
>and so keeping excessive builds down is important.
>
>Thanks for the thoughts!
>
>--------
>Matt Dimich
>
>
>-----Original Message-----
>From: Sami Tikka [mailto:[hidden email]]
>Sent: Tuesday, July 07, 2009 3:01 PM
>To: [hidden email]
>Subject: Re: Hudson and Dependency Management with Freestyle Projects
>
>I'm a bit confused. If a projects depends on other projects, I think
>it is a very good idea to build it every time any one of its
>dependencies is built.
>
>You have projects that are triggering other projects but they do not
>have a dependency relationship? How would Hudson be able to determine
>when should a downstream project to be triggered?
>
>If you need multiple projects all to finish building before triggering
>a downstream build, the new Join plugin can do it for you.
>
>Or are you actually trying to solve a different problem? Is your
>problem maybe that builds are taking a long time? When you have a
>large and complex project tree, any commit in a project near the root
>of the tree causes a lot of builds to percolate down the tree. Active
>development (== frequent commits) near the root would cause Hudson to
>be busy for a long time doing builds that are not maybe all very
>important.
>
>We also have a lot of projects and the graph of all projects is
>getting pretty complex. When someone commits a small change to our
>testing tools project, it causes all the other projects to build and
>runs all the tests. It is true this seems very "heavy", but we believe
>a full build and test cycle is needed after every commit, no matter
>how small it is. Just try to keep the builds and tests as light as
>possible.
>
>-- Sami
>
>2009/7/7  <[hidden email]>:
>> We used to use the "Build Other  Projects" trigger after our utility
>> projects to create in essence, a "chain" of dependencies
>that would get
>> fired off every time an upstream project was built.  Our
>build chain became
>> too big however, causing forced builds of projects that
>didn't need to
>> build.  I've been searching for a while on how Hudson could
>do this, but I
>> haven't found anything yet.  Has anyone requested or talked
>about making
>> that "Build Other Projects" feature into a sort of
>dependency managing
>> feature?  What we are looking for is if an upstream project
>changes, it
>> would notify the downstream projects they should build, but
>only if they
>> "have" to.  Then they would kick off a recursive similar
>process for each of
>> their downstream dependencies.  Likewise, if a downstream
>project changed,
>> it could look upstream first to make sure no upstream projects have
>> changed.  Once they have completed building, then the
>current project would
>> build.
>>
>>
>>
>> Has anyone talked about this yet?  Is this something that
>could be put in
>> the plans?
>>
>>
>>
>> Thanks!
>>
>> Matt
>>
>>
>>
>> Software Engineer
>>
>> Thomson Reuters
>>
>>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: [hidden email]
>For additional commands, e-mail: [hidden email]
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: [hidden email]
>For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Re: Hudson and Dependency Management with Freestyle Projects

Paul Sander-2
In reply to this post by matt.dimich-2
There's a lot of merit to the unit vs. integration testing argument.  A good approach is to employ a promotion system in which each build of a given project is given a quality assessment.  The dependency graph among projects then rely on the quality assessments.  A build is said to be promoted when its quality assessment increases.  A project is built whenever a checkin is made within that project, or when one of its prerequisites changes due to promotion.  This provides the necessary hooks in which projects can be built and unit tested without interfering with dependent projects.  Integration testing need be done only when a dependent project builds due to a promotion of its prerequisites.

There exists a promotion plug-in for Hudson, but I haven't looked into its capabilities yet.  If it doesn't have these capabilities then someone should consider adding them.  This method is quite efficient and has a proven track record.

> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]]
> Sent: Wednesday, July 08, 2009 6:00 AM
> To: [hidden email]
> Subject: RE: Re: Hudson and Dependency Management with
> Freestyle Projects
>
> You are exactly correct that the problem we are trying to
> solve is builds that take too long.  We have two versions of
> our build, one is a monolithic giant that takes a very long
> time because it builds everything.  When we started using
> Hudson, we broke that build up and made all of the utility
> jars into their own projects.  I understand the view that you
> take about every commit should be tested, and I agree to a
> point, but for us, each commit should be tested by that
> project, not necessarily by the projects that use that
> particular jar.  Basically, if only jar 1 changes, but jars
> 2-10 use jar 1, we don't want to build jars 2-10 just because
> jar 1 changed.  If jars 2-10 change, we don't want to rebuild
> jar 1.  We have a lot of projects on our instance of Hudson
> and so keeping excessive builds down is important.
>
> Thanks for the thoughts!
>
> --------
> Matt Dimich
>
>
> -----Original Message-----
> From: Sami Tikka [mailto:[hidden email]]
> Sent: Tuesday, July 07, 2009 3:01 PM
> To: [hidden email]
> Subject: Re: Hudson and Dependency Management with Freestyle Projects
>
> I'm a bit confused. If a projects depends on other projects, I think
> it is a very good idea to build it every time any one of its
> dependencies is built.
>
> You have projects that are triggering other projects but they do not
> have a dependency relationship? How would Hudson be able to determine
> when should a downstream project to be triggered?
>
> If you need multiple projects all to finish building before triggering
> a downstream build, the new Join plugin can do it for you.
>
> Or are you actually trying to solve a different problem? Is your
> problem maybe that builds are taking a long time? When you have a
> large and complex project tree, any commit in a project near the root
> of the tree causes a lot of builds to percolate down the tree. Active
> development (== frequent commits) near the root would cause Hudson to
> be busy for a long time doing builds that are not maybe all very
> important.
>
> We also have a lot of projects and the graph of all projects is
> getting pretty complex. When someone commits a small change to our
> testing tools project, it causes all the other projects to build and
> runs all the tests. It is true this seems very "heavy", but we believe
> a full build and test cycle is needed after every commit, no matter
> how small it is. Just try to keep the builds and tests as light as
> possible.
>
> -- Sami
>
> 2009/7/7  <[hidden email]>:
> > We used to use the "Build Other  Projects" trigger after our utility
> > projects to create in essence, a "chain" of dependencies
> that would get
> > fired off every time an upstream project was built.  Our
> build chain became
> > too big however, causing forced builds of projects that
> didn't need to
> > build.  I've been searching for a while on how Hudson could
> do this, but I
> > haven't found anything yet.  Has anyone requested or talked
> about making
> > that "Build Other Projects" feature into a sort of
> dependency managing
> > feature?  What we are looking for is if an upstream project
> changes, it
> > would notify the downstream projects they should build, but
> only if they
> > "have" to.  Then they would kick off a recursive similar
> process for each of
> > their downstream dependencies.  Likewise, if a downstream
> project changed,
> > it could look upstream first to make sure no upstream projects have
> > changed.  Once they have completed building, then the
> current project would
> > build.
> >
> >
> >
> > Has anyone talked about this yet?  Is this something that
> could be put in
> > the plans?
> >
> >
> >
> > Thanks!
> >
> > Matt
> >
> >
> >
> > Software Engineer
> >
> > Thomson Reuters
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]