parallel maven multi-module build

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

parallel maven multi-module build

Ishaaq Chandy
Hi all,
We're using maven and hudson in our team quite successfully. Each developer has their own personal git repository and a personal hudson instance with a couple of slaves to run builds on. Ideally, they are expected to run a full build through their hudson instance to ensure that their code changes pass before they push their commits back to the central source control (where our central hudson instance will trigger a full build as well).

All this works perfectly except for one thing - the full build is taking over an hour and every day as more code and tests are added this is getting longer. This is not ideal as it is discouraging developers from rigourously running full builds - they have been tending to take short cuts.

So, I was trying to come up with a solution to this problem and I think I might have one. Our full build consists of about 20 maven submodules. Running maven in "skipTests" mode over the lot takes avout 4 minutes to complete. My idea was to run the build on hudson in two phases:

1. Run in skipTests mode. As each submodule is compiled and packaged deploy it into a personal repo (e.g. sonatype's Nexus repo)
2. After deploying each submodule the main skipTests build triggers a secondary build for that individual submodule in parallel, this parallel build will kick off immediately if there is a free slave executor. This job will be configured to pick up the required artifacts from the Nexus repository that was mentioned above. This implies that we can get the latest artifacts that were built in the skipTests step.

There are certain complexities here though:

1. I have to figure out how to get the skipTests build to trigger off a submodule build mid-flight, i.e. I don't want to have to wait for the skipTests build to complete all the modules before starting to kick off all the submodule builds (even if skipTests takes only 4 minutes to complete - this is four minutes during which the slaves would otherwise be sitting idle).
2. I have to figure out how to programmatically manage these submodule jobs. I.e. as new submodules get added to the project or older obsoleted submodules get removed I need corresponding submodule jobs to get added and deleted dynamically without requring manual configuration. I'll have to do this as part of the first phase somehow.

I realise this is not trivial work and am prepared to spend some time on this but was wondering - surely I am not the first person to think of this? Has anyone else investigated something like this? Any suggestions/pointers?