Case Studies?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Case Studies?

Mark_Melvin

Hello Fellow Users,

I was wondering if anyone had any time to quickly share how they use Hudson in the scope of an actual software development process that involves shipping multiple versions of a product, traceability, etc.  Or is anyone using it for this?  It could be that people using Hudson are just using it for internal projects, but I'd like to know more about how it can fit into an end-to-end software release process.  We are looking at Hudson as our CI server, but I still have some holes to fill in.  As I see it - building projects automatically is all well and good, but there are still a lot of other issues like tagging and releasing (promoting) of builds, archiving artifacts, automatic versioning, etc.

I did go through the mailing list archives as well as the case studies on the website.  I think our usage will be similar to Sven Reimers's case study, but it sounds like we'll need a binary repository outside of Hudson similar to Maven or Ivy.  This seems like a bit of duplicated infrastructure since Hudson can archive artifacts as well (and the fingerprint feature is cool too).  So, perhaps I can throw out some questions and people can either share their experiences, or choose to answer a few questions.  Any feedback would be very helpful indeed.
  • Do you use Hudson to automatically build product that you would "release", or simply for continuous integration (and manually build your "releases")?
  • If so, how do you "release" a Hudson build?
  • How many Hudson "projects" do you have set up?
  • Do you build from HEAD and then "promote" and tag a specific nightly build when you release, or do you manually "release" components by having developers tag them in SCM first, and always build releases from a tag?
  • Do you use some other form of dependency manager or binary repository like Maven or Ivy, a home-brew method, or do you rely on Hudson entirely?
  • Do you use Hudson artifacts as inputs to the build of other projects, or do you always build from tagged items (source or binary) in SCM?
  • When you do "release" something, what is your release process?  What do you archive?
  • How many Hudson builds do you keep around (and how big are your hard drives)? ;o)
  • Do you back up your Hudson workspace and artifacts?

Also, a general question I have is - if you needed to "go back in time", how would you reproduce a build from a few months ago using Hudson?  If everything is tagged in source control this seems easy enough to do, but what if you used "latest-version" binaries from either the Hudson archive, or an Ivy repository for instance?  It seems to me that you need some sort way to "tag" artifacts in the Hudson or Ivy repositories to be able to do this.  Alternatively, you just manually check in all of these binary dependencies into SCM and tag them as source - which is what we do now, but we want to get away from this because it is a highly manual process.

Anyway, I think that is about it.  As I said, any feedback at all would be very useful.  Thanks in advance for your time!  

Mark.
AMI Semiconductor - "Silicon Solutions for the Real World"
NOTICE: 
This electronic message contains information that may be confidential or privileged. The information is intended for the use of the individual or entity named above. If you are not the intended recipient, please be aware that any disclosure, copying, distribution or use of the contents of this information is prohibited. If you received this electronic message in error, please notify the sender and delete the copy you received.


	
	
	
	
Reply | Threaded
Open this post in threaded view
|

Re: Case Studies?

Kohsuke Kawaguchi-2
[hidden email] wrote:

> Hello Fellow Users,
>
> I was wondering if anyone had any time to quickly share how they use
> Hudson in the scope of an actual software development process that
> involves shipping multiple versions of a product, traceability, etc.  Or
> is anyone using it for this?  It could be that people using Hudson are
> just using it for internal projects, but I'd like to know more about how
> it can fit into an end-to-end software release process.  We are looking at
> Hudson as our CI server, but I still have some holes to fill in.  As I see
> it - building projects automatically is all well and good, but there are
> still a lot of other issues like tagging and releasing (promoting) of
> builds, archiving artifacts, automatic versioning, etc.
>
> I did go through the mailing list archives as well as the case studies on
> the website.  I think our usage will be similar to Sven Reimers's case
> study, but it sounds like we'll need a binary repository outside of Hudson
> similar to Maven or Ivy.  This seems like a bit of duplicated
> infrastructure since Hudson can archive artifacts as well (and the
> fingerprint feature is cool too).  So, perhaps I can throw out some
> questions and people can either share their experiences, or choose to
> answer a few questions.  Any feedback would be very helpful indeed.
>
> Do you use Hudson to automatically build product that you would "release",
> or simply for continuous integration (and manually build your "releases")?
On some of my jobs, a certain build of the continuous build becomes a
"release". Usually I have associated test job set up, so say when I
build msv #150, Hudson then run msv-test #92 or whatever.

When I want to release, I go back the history and make sure that msv
#150 had a good test result. If so, then I download the artifact from
msv #150 and post that to java.net, for example.

But in many case, like when I release Hudson, I check Hudson build of
Hudson just to verify that it looks good, then I manually create a
release build. I do this mainly because Maven (the build system I use to
build Hudson) needs an entirely separate procedure for producing release
artifacts.

In yet other cases, the developers look at Hudson builds (by typically
checking the results of the associated test jobs), then tag that build
(by using Hudson's "tag this build" command.) Then that tag is sent to
the release engineering team and they do the release build. We lose the
fingerprint trackability when we do this.

With Maven, I do want to eventually be able to release from Hudson ---
so say I decide to use hudson #2300 as 1.80 release, then I go to this
build and click "release" and Hudson will do everything. Just an idea at
this point, no implementation, though.

issue #141 tries to go into this space, I think.

> Also, a general question I have is - if you needed to "go back in time",
> how would you reproduce a build from a few months ago using Hudson?  If
> everything is tagged in source control this seems easy enough to do, but
> what if you used "latest-version" binaries from either the Hudson archive,
> or an Ivy repository for instance?

The first requirement is that you keep the archive. Say I'm now at
jaxb-ri #2000 and I want to get back to jaxb-ri #1500. I must have this
log, or all is lost. I usually mark key milestone builds as "keep this
build forever" so that the log won't get lost.

Then I can either look at the timestamp of #1500 to check out the exact
same workspace, or I can use "tag this build" command to retroactively
tag this build, then use that for a build.

JAXB RI is built by Ant, so all the dependency jars are a part of the
workspace. So reproducing a build is relatively easy.

There's also a feature in Hudson where you can keep all the dependency
build logs. So if your project relies on artifacts from another projects
in Hudson, then you can check that option so that all the logs of the
dependency builds are also kept.

I don't have experience with Ivy, so I can't answer that part of the
question, but having an Ivy plugin would probably allow Hudson to do
more helpful things to reproduce the build.

 > It seems to me that you need some sort
> way to "tag" artifacts in the Hudson or Ivy repositories to be able to do
> this.  Alternatively, you just manually check in all of these binary
> dependencies into SCM and tag them as source - which is what we do now,
> but we want to get away from this because it is a highly manual process.
>
> Anyway, I think that is about it.  As I said, any feedback at all would be
> very useful.  Thanks in advance for your time!

I hope others would chime in. I'm always curious how people are using it.

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Case Studies?

Wolfram Kroll-2
In reply to this post by Mark_Melvin
[hidden email] schrieb:

Hello Fellow Users,

Hi Mark,

[...]
  • Do you use Hudson to automatically build product that you would "release", or simply for continuous integration (and manually build your "releases")?
we build our releases with hudson, but then triggered manually.
  • If so, how do you "release" a Hudson build?
yes, we copy artifacts from the web site and I tag sources based on time stamp.
  • How many Hudson "projects" do you have set up?
20 or so
  • Do you build from HEAD and then "promote" and tag a specific nightly build when you release, or do you manually "release" components by having developers tag them in SCM first, and always build releases from a tag?
In most cases we cannot use hudson's tag facility. Because most of the projects were created before a build server was introduced, build scripts check out sources itself, build and test it, then tag it in cvs (when in  "release mode").
  • Do you use some other form of dependency manager or binary repository like Maven or Ivy, a home-brew method, or do you rely on Hudson entirely?
We have binaries in cvs. I don't like it, so I'm negotiating with project managers to get resources to introduce Ivy (or another dependency managing system).
  • Do you use Hudson artifacts as inputs to the build of other projects, or do you always build from tagged items (source or binary) in SCM?
No, currently we don't access Hudson's artifacts from scripts (only manually). We have to commit binaries to SCM. A dependency manager like Ivy would be a better bet.
We build from cvs HEAD and branches, and only occasionally from a tag.
  • When you do "release" something, what is your release process?  What do you archive?
We archive those binaries that go to customers. But because sources and build scripts are tagged we can reproduce builds from the not-so-distant history.
  • How many Hudson builds do you keep around (and how big are your hard drives)? ;o)
Only the latest artifacts and logs for 14 days. This could be reduced, because most broken CI builds get fixed in a day or two and nobody wants to know what was wrong last week. We do a "keep this build forever" for builds which were propagated (manually) to a release.
  • Do you back up your Hudson workspace and artifacts?
Thank you for the reminder... our build scripts have the most value (they are in cvs), our hudson config is not that complicated...

Also, a general question I have is - if you needed to "go back in time", how would you reproduce a build from a few months ago using Hudson?  If everything is tagged in source control this seems easy enough to do, but what if you used "latest-version" binaries from either the Hudson archive, or an Ivy repository for instance?  It seems to me that you need some sort way to "tag" artifacts in the Hudson or Ivy repositories to be able to do this.  Alternatively, you just manually check in all of these binary dependencies into SCM and tag them as source - which is what we do now, but we want to get away from this because it is a highly manual process.
Similar situation here.

Anyway, I think that is about it.  As I said, any feedback at all would be very useful.  Thanks in advance for your time! 

Thanks for sharing your thoughts!

Wolfram


Mark.
AMI Semiconductor - "Silicon Solutions for the Real World"
NOTICE: 
This electronic message contains information that may be confidential or privileged. The information is intended for the use of the individual or entity named above. If you are not the intended recipient, please be aware that any disclosure, copying, distribution or use of the contents of this information is prohibited. If you received this electronic message in error, please notify the sender and delete the copy you received.

  

Reply | Threaded
Open this post in threaded view
|

Re: Case Studies?

Mark_Melvin

Hi Wolfram and Kohsuke,

Thank you both for your reply!  That information is very valuable.  I have a couple of other things to add to the discussion.

Both of you mentioned a "keep this build forever" feature in Hudson.  Where is this feature?  I can't seem to find it anywhere.  That is something I would see us using as part of a release or build promotion operation.

Back to the whole "binaries in SCM" thing.  Yeah - we all seem to not feel so good about storing binaries in CVS/SVN, but is using something like Ivy really any better?  All it is *really* is, is another repository.  And it is arguably worse because you don't have the concept of "tagging".  You are tied to specifying versions.  So, if you want true, repeatable builds (binary, and timestamp-identical) you need to backup your Ivy repository as well.  It seems to me like we're just moving the problem somewhere else - unless I am missing something.  Although Ivy does admittedly give you some additional functionality like resolvers and latest version calculators, but you need to be very careful about this stuff if you want the capability of re-creating old builds.  I guess what also bothers me is maintaining two repositories.  And if you consider the Hudson archives as a repository - then you're talking three repositories.  I don't really have a solution for this problem, but I have been thinking about it quite a bit.  I have a few ideas, but I am not sure any one of them is better than the other:
  • Use Ivy in combination with Hudson, and write an Ivy publisher plugin for Hudson that publishes artifacts to an Ivy repository (something we are looking into right now).
  • Write a CVS publisher plugin for Hudson that publishes artifacts back to SCM.  Any tagging done from Hudson would have to take these into account as well.
  • Somehow, as part of your promotion of a build to release status (realizing it is a yet-to-be-implemented feature), have Hudson check in artifacts into SCM and tag them.  This is the same as above, but only puts artifacts you've "released" into CVS.  And admittedly is a little convoluted because you would have built with an artifact from the Hudson repository and are relying on an artifact in CVS for future build reproducibility.  I'm not sure how you would account for this path discrepancy.
  • Use the Hudson artifact repository as an Ivy-like repository.  Maybe you can introduce a way of "tagging" artifacts using fingerprints and permalinks, so things are reproducible.  But this starts to overlap with Ivy functionality and is probably not the way to go.

Anyway, we'll keep plugging away at this.  If anyone has any other ideas, please share!

Thanks,
Mark.
----------------------------------------------------------

AMI Semiconductor - "Silicon Solutions for the Real World"
NOTICE: 
This electronic message contains information that may be confidential or privileged. The information is intended for the use of the individual or entity named above. If you are not the intended recipient, please be aware that any disclosure, copying, distribution or use of the contents of this information is prohibited. If you received this electronic message in error, please notify the sender and delete the copy you received.


	
	
	
	
Reply | Threaded
Open this post in threaded view
|

Re: Case Studies?

Kohsuke Kawaguchi-2
[hidden email] wrote:
> Both of you mentioned a "keep this build forever" feature in Hudson. Where
> is this feature?  I can't seem to find it anywhere.  That is something I
> would see us using as part of a release or build promotion operation.

You should see the "keep this build forever" button when you go to a
build page, like http://myhost/hudson/job/MYPROJECT/lastBuild/

But someone saying "I can't find it" is usually a sign that the UI is
not correctly designed, so my queestion to you is where you'd expect to
see such action?


> Back to the whole "binaries in SCM" thing.  Yeah - we all seem to not feel
> so good about storing binaries in CVS/SVN, but is using something like Ivy
> really any better?  All it is *really* is, is another repository.  And it
> is arguably worse because you don't have the concept of "tagging".

Right.

The trade off is that on one hand you keep jars in SCM, which requires
you to constantly re-integrate newer bits, which is tedious even if you
automate some of it. On the other hand you can download jars from an
artifact repository like Ivy/Maven, and then it becomes harder to
reproduce a build (especially when you depend on snapshot builds.)

I don't know much about Ivy, but with Maven I'm wondering if Hudson can
help the situation. The native maven support allows Hudson to track
exactly what 'snapshot' jar you used for a build.

So when you later decide that you need to reproduce the build, then
knows exactly what jars are used. It can also keep all those artifacts
around.

So if someone I can convince Maven to use the set of jars that Hudson
knows, then ...


 > You
> are tied to specifying versions.  So, if you want true, repeatable builds
> (binary, and timestamp-identical) you need to backup your Ivy repository
> as well.  It seems to me like we're just moving the problem somewhere else
> - unless I am missing something.

You are trading one problem to another. Sometimes it makes sense,
sometimes it doesn't. But I don't think they are the same problem.

 > Although Ivy does admittedly give you

> some additional functionality like resolvers and latest version
> calculators, but you need to be very careful about this stuff if you want
> the capability of re-creating old builds.  I guess what also bothers me is
> maintaining two repositories.  And if you consider the Hudson archives as
> a repository - then you're talking three repositories.  I don't really
> have a solution for this problem, but I have been thinking about it quite
> a bit.  I have a few ideas, but I am not sure any one of them is better
> than the other:
>
> Use Ivy in combination with Hudson, and write an Ivy publisher plugin for
> Hudson that publishes artifacts to an Ivy repository (something we are
> looking into right now).
Yeah, again I don't know much about Ivy, but it sounded like it should
be possible to make Hudson an Ivy repository. Whether you need a custom
publisher to do this or not I don't know.

There has been some interest to this in the past. If you are going to
write a plugin for this, please consider doing so on
http://hudson.dev.java.net/

> Write a CVS publisher plugin for Hudson that publishes artifacts back to
> SCM.  Any tagging done from Hudson would have to take these into account
> as well.
> Somehow, as part of your promotion of a build to release status (realizing
> it is a yet-to-be-implemented feature), have Hudson check in artifacts
> into SCM and tag them.  This is the same as above, but only puts artifacts
> you've "released" into CVS.  And admittedly is a little convoluted because
> you would have built with an artifact from the Hudson repository and are
> relying on an artifact in CVS for future build reproducibility.  I'm not
> sure how you would account for this path discrepancy.
> Use the Hudson artifact repository as an Ivy-like repository.  Maybe you
> can introduce a way of "tagging" artifacts using fingerprints and
> permalinks, so things are reproducible.  But this starts to overlap with
> Ivy functionality and is probably not the way to go.
>
> Anyway, we'll keep plugging away at this.  If anyone has any other ideas,
> please share!
>
> Thanks,
> Mark.
> ----------------------------------------------------------
>
>
> AMI Semiconductor - "Silicon Solutions for the Real World"
> NOTICE:
> This electronic message contains information that may be confidential or privileged. The information is intended for the use of the individual or entity named above. If you are not the intended recipient, please be aware that any disclosure, copying, distribution or use of the contents of this information is prohibited. If you received this electronic message in error, please notify the sender and delete the copy you received.
>

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Case Studies?

Mark_Melvin

Kohsuke Kawaguchi <[hidden email]> wrote on 02/08/2007 01:54:11 PM:

> You should see the "keep this build forever" button when you go to a
> build page, like http://myhost/hudson/job/MYPROJECT/lastBuild/
>
> But someone saying "I can't find it" is usually a sign that the UI is
> not correctly designed, so my queestion to you is where you'd expect to
> see such action?
>

My mistake.  I had configured my test build without log rotation.  In that case, there is no button to "keep this build forever" because they are all kept forever.  I turned on log rotation and now I see it.  Thanks.

.
.
>
> Yeah, again I don't know much about Ivy, but it sounded like it should
> be possible to make Hudson an Ivy repository. Whether you need a custom
> publisher to do this or not I don't know.
>
> There has been some interest to this in the past. If you are going to
> write a plugin for this, please consider doing so on
> http://hudson.dev.java.net/
>

We'll definitely look at doing this on http://hudson.dev.java.net/ if we do decide to implement something.

Thanks,
Mark.
AMI Semiconductor - "Silicon Solutions for the Real World"
NOTICE: 
This electronic message contains information that may be confidential or privileged. The information is intended for the use of the individual or entity named above. If you are not the intended recipient, please be aware that any disclosure, copying, distribution or use of the contents of this information is prohibited. If you received this electronic message in error, please notify the sender and delete the copy you received.