Breaking up a monolithic build

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Breaking up a monolithic build

Peter Kahn
How can I break my build into several pieces that feed artifacts to later stages of the build?  (if I need to RTFM please let me know what keywords I should be looking for)


5 artifacts, runs unit tests, static analysis and integration tests.   Sometimes, I just need one artifact and other times I just want to re-run one part of the build.  I'd like to break the build up by artifact and lifecycle phase.  The artifact split, is mine and straight forward, but I'd like some advice on the lifecycle phase split.

I'd like to have a hudson build that compiles/packages and another that runs the integration tests.  Then, I'd like have a 3rd end-to-end build that runs the compile/package and feeds the results to the integration test.

This would let me package when I like and run integration tests when I like or have them flow together.   I'd imagine that I'd want the integration-test build to be parameterized and have those params fed by the end-to-end.


Is this possible?  Am I approaching this problem in the right way?

Thanks for the help.

--
Peter Kahn
[hidden email]
pkahnpie1@AIM
http://www.google.com/profiles/citizenkahn
Awareness - Intention - Action
Reply | Threaded
Open this post in threaded view
|

RE: Breaking up a monolithic build

EJ Ciramella-2
Heh - we just went through a "deconstruction" effort here and finally drove the last nail into the coffin that was a monolith build.
 
We had one structure generating libraries and deployable units and all kinds of things - maybe 20 - 25 different build artifacts.
 
When using maven, I think the real goal is a high degree of modularity.  If that's what you're after, if you have a project that generates 5 artifacts, maybe you should think if some of those could be stripped out:
 
1 - do we NEED to build these 5 things with each pass?
2 - are there different concerns that are overlapping and should be broken apart (maybe a templates project, jar artifact, war artifact and then server configuration layer project that consumes the war - so you could support jboss/weblogic/websphere/etc.)
 
That being said, with Hudson, you can chain things together.  So in the most basic use case, you'd have a ci build that monitors your source control system for a change in a given project.  That would build and report success then fire off the sonar build of the same module.  At that point, you could also have it kick off the integration test -> smoke -> load ->etc testing.
 
When building an artifact - at what point can you say to yourself, "yes, I could happliy release this untested" and I'm guessing most likely the answer would be never.  When would you really want to bypass the integration tests?  I think that an artifact should not be made available unless it's passed all the tests - then and only then, generate the artifact (RC1/RC2/etc...).
 
I personally am using hudson to set up "stages" - I've been describing them as ribbons across a finish line.  First, you need to run through the "compile" ribbon, then the "CI" ribbon, etc with the final one being the "release" ribbon.  You aren't finished (can't release) unless you've crossed each stage successfully.  Most times, there's no sense in trying to cross the "smoke test" line prior to getting across "unit test" line successfully.

________________________________

From: Peter Kahn [mailto:[hidden email]]
Sent: Thu 6/25/2009 12:34 PM
To: [hidden email]
Subject: Breaking up a monolithic build


How can I break my build into several pieces that feed artifacts to later stages of the build?  (if I need to RTFM please let me know what keywords I should be looking for)


5 artifacts, runs unit tests, static analysis and integration tests.   Sometimes, I just need one artifact and other times I just want to re-run one part of the build.  I'd like to break the build up by artifact and lifecycle phase.  The artifact split, is mine and straight forward, but I'd like some advice on the lifecycle phase split.

I'd like to have a hudson build that compiles/packages and another that runs the integration tests.  Then, I'd like have a 3rd end-to-end build that runs the compile/package and feeds the results to the integration test.

This would let me package when I like and run integration tests when I like or have them flow together.   I'd imagine that I'd want the integration-test build to be parameterized and have those params fed by the end-to-end.


Is this possible?  Am I approaching this problem in the right way?

Thanks for the help.

--
Peter Kahn
[hidden email]
pkahnpie1@AIM
http://www.google.com/profiles/citizenkahn
Awareness - Intention - Action


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Breaking up a monolithic build

stormcloud-3
In reply to this post by Peter Kahn
Maybe I'm not quite understanding your situation, but what I'd do is:

1) Work out what tasks you require (clean, build, build-test, run-test etc).

2) Split your monolithic project into several smaller ones. For example you might have a low level project that takes care of generic, low-level utility classes, another one might look after a UI.

Each project should implement each of the tasks you defined in the previous step; even it it does nothing.

3) use Hudson's upstream/downstream project feature to link them together in the right order. Upstream projects are defined in a project's "Build triggers | build after other projects are built" section. Downstream projects are defined in the "Post build actions | build other project" actions.


The important thing is that every task should be as self contained as possible; Hudson should be set up to clean, compile, deploy, and test each component before it starts the next one.




2009/6/25 Peter Kahn <[hidden email]>
How can I break my build into several pieces that feed artifacts to later stages of the build?  (if I need to RTFM please let me know what keywords I should be looking for)


5 artifacts, runs unit tests, static analysis and integration tests.   Sometimes, I just need one artifact and other times I just want to re-run one part of the build.  I'd like to break the build up by artifact and lifecycle phase.  The artifact split, is mine and straight forward, but I'd like some advice on the lifecycle phase split.

I'd like to have a hudson build that compiles/packages and another that runs the integration tests.  Then, I'd like have a 3rd end-to-end build that runs the compile/package and feeds the results to the integration test.

This would let me package when I like and run integration tests when I like or have them flow together.   I'd imagine that I'd want the integration-test build to be parameterized and have those params fed by the end-to-end.


Is this possible?  Am I approaching this problem in the right way?

Thanks for the help.

--
Peter Kahn
[hidden email]
pkahnpie1@AIM
http://www.google.com/profiles/citizenkahn
Awareness - Intention - Action

Reply | Threaded
Open this post in threaded view
|

Re: Breaking up a monolithic build

stephenconnolly
given that your build system was set up by Peter Reilly, it is
basically a maven2 build written in ant with binary dependencies
stored in svn as Peter does not like downloading the Internet every
tome you do a build.

modularizing some of the ant builds that were not written by Peter r
is a much harder task

On Thursday, June 25, 2009, stormcloud <[hidden email]> wrote:

> Maybe I'm not quite understanding your situation, but what I'd do is:
>
> 1) Work out what tasks you require (clean, build, build-test, run-test etc).
>
> 2) Split your monolithic project into several smaller ones. For example you might have a low level project that takes care of generic, low-level utility classes, another one might look after a UI.
>
> Each project should implement each of the tasks you defined in the previous step; even it it does nothing.
>
> 3) use Hudson's upstream/downstream project feature to link them together in the right order. Upstream projects are defined in a project's "Build triggers | build after other projects are built" section. Downstream projects are defined in the "Post build actions | build other project" actions.
>
>
> The important thing is that every task should be as self contained as possible; Hudson should be set up to clean, compile, deploy, and test each component before it starts the next one.
>
>
>
>
>
> 2009/6/25 Peter Kahn <[hidden email]>
>
> How can I break my build into several pieces that feed artifacts to later stages of the build?  (if I need to RTFM please let me know what keywords I should be looking for)
>
>
> 5 artifacts, runs unit tests, static analysis and integration tests.   Sometimes, I just need one artifact and other times I just want to re-run one part of the build.  I'd like to break the build up by artifact and lifecycle phase.  The artifact split, is mine and straight forward, but I'd like some advice on the lifecycle phase split.
>
> I'd like to have a hudson build that compiles/packages and another that runs the integration tests.  Then, I'd like have a 3rd end-to-end build that runs the compile/package and feeds the results to the integration test.
>
> This would let me package when I like and run integration tests when I like or have them flow together.   I'd imagine that I'd want the integration-test build to be parameterized and have those params fed by the end-to-end.
>
>
> Is this possible?  Am I approaching this problem in the right way?
>
> Thanks for the help.
> --
> Peter Kahn
> [hidden email]
> pkahnpie1@AIM
> http://www.google.com/profiles/citizenkahn
> Awareness - Intention - Action
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Breaking up a monolithic build

stormcloud-3
It's certainly true that starting with a well designed build system helps. That said, what we've got has evolved as needs have changed (we've added integration tests and split apart some of the larger projects).

Peter's decision to "not download the internet" every time we kick off a build is something I'd strongly defend - the build server would be unacceptably if we were to go reload everything in SVN each time

2009/6/25 Stephen Connolly <[hidden email]>
given that your build system was set up by Peter Reilly, it is
basically a maven2 build written in ant with binary dependencies
stored in svn as Peter does not like downloading the Internet every
tome you do a build.

modularizing some of the ant builds that were not written by Peter r
is a much harder task

On Thursday, June 25, 2009, stormcloud <[hidden email]> wrote:
> Maybe I'm not quite understanding your situation, but what I'd do is:
>
> 1) Work out what tasks you require (clean, build, build-test, run-test etc).
>
> 2) Split your monolithic project into several smaller ones. For example you might have a low level project that takes care of generic, low-level utility classes, another one might look after a UI.
>
> Each project should implement each of the tasks you defined in the previous step; even it it does nothing.
>
> 3) use Hudson's upstream/downstream project feature to link them together in the right order. Upstream projects are defined in a project's "Build triggers | build after other projects are built" section. Downstream projects are defined in the "Post build actions | build other project" actions.
>
>
> The important thing is that every task should be as self contained as possible; Hudson should be set up to clean, compile, deploy, and test each component before it starts the next one.
>
>
>
>
>
> 2009/6/25 Peter Kahn <[hidden email]>
>
> How can I break my build into several pieces that feed artifacts to later stages of the build?  (if I need to RTFM please let me know what keywords I should be looking for)
>
>
> 5 artifacts, runs unit tests, static analysis and integration tests.   Sometimes, I just need one artifact and other times I just want to re-run one part of the build.  I'd like to break the build up by artifact and lifecycle phase.  The artifact split, is mine and straight forward, but I'd like some advice on the lifecycle phase split.
>
> I'd like to have a hudson build that compiles/packages and another that runs the integration tests.  Then, I'd like have a 3rd end-to-end build that runs the compile/package and feeds the results to the integration test.
>
> This would let me package when I like and run integration tests when I like or have them flow together.   I'd imagine that I'd want the integration-test build to be parameterized and have those params fed by the end-to-end.
>
>
> Is this possible?  Am I approaching this problem in the right way?
>
> Thanks for the help.
> --
> Peter Kahn
> [hidden email]
> pkahnpie1@AIM
> http://www.google.com/profiles/citizenkahn
> Awareness - Intention - Action
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Breaking up a monolithic build

stephenconnolly
2009/6/26 stormcloud <[hidden email]>:
> It's certainly true that starting with a well designed build system helps.
> That said, what we've got has evolved as needs have changed (we've added
> integration tests and split apart some of the larger projects).
>
> Peter's decision to "not download the internet" every time we kick off a
> build is something I'd strongly defend - the build server would be
> unacceptably if we were to go reload everything in SVN each time
>

Actually, that's the whole point of the m2 local repository... it's a
cache of the internet so that it need only be downloaded once.

Storing the artifacts in SVN forces you to download all of them every
time you do a clean checkout.

Maven 2 has the rules of:
* download -SNAPSHOT versions every time
* download non -SNAPSHOT versions once and only once

So as long as your build is working off of released artifacts, they
only get downloaded the once.

;-)

> 2009/6/25 Stephen Connolly <[hidden email]>
>>
>> given that your build system was set up by Peter Reilly, it is
>> basically a maven2 build written in ant with binary dependencies
>> stored in svn as Peter does not like downloading the Internet every
>> tome you do a build.
>>
>> modularizing some of the ant builds that were not written by Peter r
>> is a much harder task
>>
>> On Thursday, June 25, 2009, stormcloud <[hidden email]>
>> wrote:
>> > Maybe I'm not quite understanding your situation, but what I'd do is:
>> >
>> > 1) Work out what tasks you require (clean, build, build-test, run-test
>> > etc).
>> >
>> > 2) Split your monolithic project into several smaller ones. For example
>> > you might have a low level project that takes care of generic, low-level
>> > utility classes, another one might look after a UI.
>> >
>> > Each project should implement each of the tasks you defined in the
>> > previous step; even it it does nothing.
>> >
>> > 3) use Hudson's upstream/downstream project feature to link them
>> > together in the right order. Upstream projects are defined in a project's
>> > "Build triggers | build after other projects are built" section. Downstream
>> > projects are defined in the "Post build actions | build other project"
>> > actions.
>> >
>> >
>> > The important thing is that every task should be as self contained as
>> > possible; Hudson should be set up to clean, compile, deploy, and test each
>> > component before it starts the next one.
>> >
>> >
>> >
>> >
>> >
>> > 2009/6/25 Peter Kahn <[hidden email]>
>> >
>> > How can I break my build into several pieces that feed artifacts to
>> > later stages of the build?  (if I need to RTFM please let me know what
>> > keywords I should be looking for)
>> >
>> >
>> > 5 artifacts, runs unit tests, static analysis and integration tests.
>> > Sometimes, I just need one artifact and other times I just want to re-run
>> > one part of the build.  I'd like to break the build up by artifact and
>> > lifecycle phase.  The artifact split, is mine and straight forward, but I'd
>> > like some advice on the lifecycle phase split.
>> >
>> > I'd like to have a hudson build that compiles/packages and another that
>> > runs the integration tests.  Then, I'd like have a 3rd end-to-end build that
>> > runs the compile/package and feeds the results to the integration test.
>> >
>> > This would let me package when I like and run integration tests when I
>> > like or have them flow together.   I'd imagine that I'd want the
>> > integration-test build to be parameterized and have those params fed by the
>> > end-to-end.
>> >
>> >
>> > Is this possible?  Am I approaching this problem in the right way?
>> >
>> > Thanks for the help.
>> > --
>> > Peter Kahn
>> > [hidden email]
>> > pkahnpie1@AIM
>> > http://www.google.com/profiles/citizenkahn
>> > Awareness - Intention - Action
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]