Using Hudson in productive environment

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Using Hudson in productive environment

Sven Reimers
Hi all,

we have set up Hudson in a productive environment for testing purposes
(switching from Anthill). First thing I have to say is that Hudson
really fits our needs - really nice work.

Nevertheless we discovered some problems during configuration in our
environment. The environment consists of a large number of jobs (>400
- this is caused due to heavy modularization and library usage on a
binary basis).

1. Problems with svn polling: Actually there is no visual feedback if
polling does not work (misconfiguration, server unavailable -
whatever) - are there any plans to provide such a feedback - maybe by
putting a badge on the job status icon?

2. Splitting projects with views is quite nice, but it would also be
useful to have a batch edit mode, that applies the settings to a group
(maybe the same as the views) of jobs can administration much more
easy.

3. Limiting the number of subversion polls executed in parallel, is a
requirement we encountered on windows platform. If too many polls are
executed in parallel, the Windows Desktop Heap is not sufficient to
perform those, so that an exception is caused (out of heap in windows
not in java) - after that nothing that tries to start more than a
small process dies because of insufficient memory to access during
process creation. So maybe - as used with the build executors - a
limitation could be used.

4. Job-View: The view pattern is really nice, but with the large
number of projects a further refined view would  be helpful (treeview,
ajax?)

5. Make iconsize selectable - this can help reduce page length with a
large number of projects.

Keep up the good work

-Sven

P.S. Trying to setup a hudson project environment - so I may hack some
of those issues myself ;-)

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using Hudson in productive environment

Kohsuke Kawaguchi-2
Sven Reimers wrote:
> Hi all,
>
> we have set up Hudson in a productive environment for testing purposes
> (switching from Anthill). First thing I have to say is that Hudson
> really fits our needs - really nice work.

Thank you.

> Nevertheless we discovered some problems during configuration in our
> environment. The environment consists of a large number of jobs (>400
> - this is caused due to heavy modularization and library usage on a
> binary basis).

Cool. I have 150 jobs or so, but 400 is by far the biggest I've heard. I
wonder if you could talk some more about how you set things up --- I'm
trying to collect more "case study" like this, because I think it help
other people think about how they want to deploy it.

You can blog about it and I can link to it, or we can just talk
interactively like this and I can write it up, or whatever suits you.


> 1. Problems with svn polling: Actually there is no visual feedback if
> polling does not work (misconfiguration, server unavailable -
> whatever) - are there any plans to provide such a feedback - maybe by
> putting a badge on the job status icon?

I believe each project page with SVN polling gets a new item that shows
the log. See the attached screen shot.


> 2. Splitting projects with views is quite nice, but it would also be
> useful to have a batch edit mode, that applies the settings to a group
> (maybe the same as the views) of jobs can administration much more
> easy.

Hmm. I see what you mean. The other related idea was to have inheritance
of setting between jobs.

The tricky part is how to maintain a simple UI for simpler cases.
Inheritance might work, because we can keep the current UI unless
"inherit setting from another project" checkbox is left off.

Another approach is to have a better programatic API to Hudson, so that
you can write a little script to do changes like this. With 400 jobs,
you might need that a better scripting capability anyway.

Speaking of scripting, a recent version of Hudson has a Groovy scripting
console, and you can execute any code fragment inside Hudson, and this
might be a handy way of changing configuration in a batch.

Oh, and just so you know, all the configuration files are XML, so you
can also choose to edit those XML files on the disk and have Hudson
reload changes from the disk.


> 3. Limiting the number of subversion polls executed in parallel, is a
> requirement we encountered on windows platform. If too many polls are
> executed in parallel, the Windows Desktop Heap is not sufficient to
> perform those, so that an exception is caused (out of heap in windows
> not in java) - after that nothing that tries to start more than a
> small process dies because of insufficient memory to access during
> process creation. So maybe - as used with the build executors - a
> limitation could be used.

Yeah, we could do that. Can you file an issue for this?

But ultimately, for your environment, I recommend a push notification,
not polling. You can configure SVN repo to wget the
"http://foo.acme.org/hudson/job/myjob/build" URL as a post-commit
script, or have another machine do that when it receives e-mails.

I do that for my Hudson installation, and it makes everyone happy.


> 4. Job-View: The view pattern is really nice, but with the large
> number of projects a further refined view would  be helpful (treeview,
> ajax?)

I think we can make a view pluggable, so that people can write plugins
for showing views in various different formats they like.

I think that might enable interesting things, like organizing a tree
view based on POM inheritance (if yours are maven projects), etc. Just
the other day I was talking with my colleague that we need to be able to
see the aggregated test results of 8 or so jobs that we run for the
JAX-WS RI, and we a "pluggable view" would let you do that kind of things.

But tree view is certainly basic enough that we might want to support
out of the box. The challenging part would be the UI to edit the tree,
but we'll see.


> 5. Make iconsize selectable - this can help reduce page length with a
> large number of projects.

Good suggestion. Please file an issue for this.

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

svn-polling.png (52K) Download Attachment
smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using Hudson in productive environment

Tom Ball
Speaking of using Hudson for automated testing, can Hudson be enhanced
to support environment variables?  I need to set a DISPLAY variable, so
that a desktop application can be tested using a virtual frame buffer.
Headless AWT won't work because headless components aren't supported.

Tom

Kohsuke Kawaguchi wrote:

> Sven Reimers wrote:
>> Hi all,
>>
>> we have set up Hudson in a productive environment for testing purposes
>> (switching from Anthill). First thing I have to say is that Hudson
>> really fits our needs - really nice work.
>
> Thank you.
>
>> Nevertheless we discovered some problems during configuration in our
>> environment. The environment consists of a large number of jobs (>400
>> - this is caused due to heavy modularization and library usage on a
>> binary basis).
>
> Cool. I have 150 jobs or so, but 400 is by far the biggest I've heard. I
> wonder if you could talk some more about how you set things up --- I'm
> trying to collect more "case study" like this, because I think it help
> other people think about how they want to deploy it.
>
> You can blog about it and I can link to it, or we can just talk
> interactively like this and I can write it up, or whatever suits you.
>
>
>> 1. Problems with svn polling: Actually there is no visual feedback if
>> polling does not work (misconfiguration, server unavailable -
>> whatever) - are there any plans to provide such a feedback - maybe by
>> putting a badge on the job status icon?
>
> I believe each project page with SVN polling gets a new item that shows
> the log. See the attached screen shot.
>
>
>> 2. Splitting projects with views is quite nice, but it would also be
>> useful to have a batch edit mode, that applies the settings to a group
>> (maybe the same as the views) of jobs can administration much more
>> easy.
>
> Hmm. I see what you mean. The other related idea was to have inheritance
> of setting between jobs.
>
> The tricky part is how to maintain a simple UI for simpler cases.
> Inheritance might work, because we can keep the current UI unless
> "inherit setting from another project" checkbox is left off.
>
> Another approach is to have a better programatic API to Hudson, so that
> you can write a little script to do changes like this. With 400 jobs,
> you might need that a better scripting capability anyway.
>
> Speaking of scripting, a recent version of Hudson has a Groovy scripting
> console, and you can execute any code fragment inside Hudson, and this
> might be a handy way of changing configuration in a batch.
>
> Oh, and just so you know, all the configuration files are XML, so you
> can also choose to edit those XML files on the disk and have Hudson
> reload changes from the disk.
>
>
>> 3. Limiting the number of subversion polls executed in parallel, is a
>> requirement we encountered on windows platform. If too many polls are
>> executed in parallel, the Windows Desktop Heap is not sufficient to
>> perform those, so that an exception is caused (out of heap in windows
>> not in java) - after that nothing that tries to start more than a
>> small process dies because of insufficient memory to access during
>> process creation. So maybe - as used with the build executors - a
>> limitation could be used.
>
> Yeah, we could do that. Can you file an issue for this?
>
> But ultimately, for your environment, I recommend a push notification,
> not polling. You can configure SVN repo to wget the
> "http://foo.acme.org/hudson/job/myjob/build" URL as a post-commit
> script, or have another machine do that when it receives e-mails.
>
> I do that for my Hudson installation, and it makes everyone happy.
>
>
>> 4. Job-View: The view pattern is really nice, but with the large
>> number of projects a further refined view would  be helpful (treeview,
>> ajax?)
>
> I think we can make a view pluggable, so that people can write plugins
> for showing views in various different formats they like.
>
> I think that might enable interesting things, like organizing a tree
> view based on POM inheritance (if yours are maven projects), etc. Just
> the other day I was talking with my colleague that we need to be able to
> see the aggregated test results of 8 or so jobs that we run for the
> JAX-WS RI, and we a "pluggable view" would let you do that kind of things.
>
> But tree view is certainly basic enough that we might want to support
> out of the box. The challenging part would be the UI to edit the tree,
> but we'll see.
>
>
>> 5. Make iconsize selectable - this can help reduce page length with a
>> large number of projects.
>
> Good suggestion. Please file an issue for this.
>
>
> ------------------------------------------------------------------------
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using Hudson in productive environment

Sven Reimers
In reply to this post by Kohsuke Kawaguchi-2
On 11/13/06, Kohsuke Kawaguchi <[hidden email]> wrote:

> Sven Reimers wrote:
> > Hi all,
> >
> > we have set up Hudson in a productive environment for testing purposes
> > (switching from Anthill). First thing I have to say is that Hudson
> > really fits our needs - really nice work.
>
> Thank you.
>
> > Nevertheless we discovered some problems during configuration in our
> > environment. The environment consists of a large number of jobs (>400
> > - this is caused due to heavy modularization and library usage on a
> > binary basis).
>
> Cool. I have 150 jobs or so, but 400 is by far the biggest I've heard. I
> wonder if you could talk some more about how you set things up --- I'm
> trying to collect more "case study" like this, because I think it help
> other people think about how they want to deploy it.
>
Setting things up worked like a breeze. A colleague of mine created a
script to create the job configuration.xml files, some files that
contained the parts we could put into the view configuration.xml. All
this could easily be done, because our internal setup is quite well
organized and named, else it could be a bit more problematic...

If you want to know more just ask ;-)

> You can blog about it and I can link to it, or we can just talk
> interactively like this and I can write it up, or whatever suits you.
>
>
> > 1. Problems with svn polling: Actually there is no visual feedback if
> > polling does not work (misconfiguration, server unavailable -
> > whatever) - are there any plans to provide such a feedback - maybe by
> > putting a badge on the job status icon?
>
> I believe each project page with SVN polling gets a new item that shows
> the log. See the attached screen shot.
>
Yeah - but I have to look inside to determine if it worked out
correclty or if some problem occured. The idea was to put this into a
more visible place so that it is more obvoius that a build may not be
uptodate.

>
> > 2. Splitting projects with views is quite nice, but it would also be
> > useful to have a batch edit mode, that applies the settings to a group
> > (maybe the same as the views) of jobs can administration much more
> > easy.
>
> Hmm. I see what you mean. The other related idea was to have inheritance
> of setting between jobs.
>
> The tricky part is how to maintain a simple UI for simpler cases.
> Inheritance might work, because we can keep the current UI unless
> "inherit setting from another project" checkbox is left off.
>
> Another approach is to have a better programatic API to Hudson, so that
> you can write a little script to do changes like this. With 400 jobs,
> you might need that a better scripting capability anyway.
>
> Speaking of scripting, a recent version of Hudson has a Groovy scripting
> console, and you can execute any code fragment inside Hudson, and this
> might be a handy way of changing configuration in a batch.
>

Ok. I see the advantage of inline scripting Hudson - I will have to
think some more about how to exploit this for our use cases...

> Oh, and just so you know, all the configuration files are XML, so you
> can also choose to edit those XML files on the disk and have Hudson
> reload changes from the disk.
>
Yeah - this my favourite feature ;-)

>
> > 3. Limiting the number of subversion polls executed in parallel, is a
> > requirement we encountered on windows platform. If too many polls are
> > executed in parallel, the Windows Desktop Heap is not sufficient to
> > perform those, so that an exception is caused (out of heap in windows
> > not in java) - after that nothing that tries to start more than a
> > small process dies because of insufficient memory to access during
> > process creation. So maybe - as used with the build executors - a
> > limitation could be used.
>
> Yeah, we could do that. Can you file an issue for this?
>
> But ultimately, for your environment, I recommend a push notification,
> not polling. You can configure SVN repo to wget the
> "http://foo.acme.org/hudson/job/myjob/build" URL as a post-commit
> script, or have another machine do that when it receives e-mails.
>
> I do that for my Hudson installation, and it makes everyone happy.
>
I could possibly do this - but I have to put this in correct for all
the projects - not that easy anyway - but maybe we should do both ;-)

I will file an issue.

>
> > 4. Job-View: The view pattern is really nice, but with the large
> > number of projects a further refined view would  be helpful (treeview,
> > ajax?)
>
> I think we can make a view pluggable, so that people can write plugins
> for showing views in various different formats they like.
>
> I think that might enable interesting things, like organizing a tree
> view based on POM inheritance (if yours are maven projects), etc. Just
> the other day I was talking with my colleague that we need to be able to
> see the aggregated test results of 8 or so jobs that we run for the
> JAX-WS RI, and we a "pluggable view" would let you do that kind of things.
>
> But tree view is certainly basic enough that we might want to support
> out of the box. The challenging part would be the UI to edit the tree,
> but we'll see.
>
Ok. Sounds promising. This actually not a showstopper - but would be
nice in the long run.
>
> > 5. Make iconsize selectable - this can help reduce page length with a
> > large number of projects.
>
> Good suggestion. Please file an issue for this.
Ok. I'll do it.

Thanks for your quick response.

-Sven

>
> --
> Kohsuke Kawaguchi
> Sun Microsystems                   [hidden email]
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using Hudson in productive environment

Jesse Glick
In reply to this post by Tom Ball
Tom Ball wrote:
> I need to set a DISPLAY variable, so that a desktop application can
> be tested using a virtual frame buffer. Headless AWT won't work
> because headless components aren't supported.

FWIW, I handle this by using a wrapper script to start Hudson:

progdir=`dirname $0`
unset XAUTHORITY
Xvnc -localhost -geometry 1024x768 -depth 16 :69 &
echo $! > $progdir/Xvnc.pid
export DISPLAY=:69
# ...

and to stop:

# ...
progdir=`dirname $0`
if [ -f $progdir/Xvnc.pid ] ; then
     kill `head $progdir/Xvnc.pid`
     rm $progdir/Xvnc.pid
fi

Unfortunately multiple simultaneous GUI-mode tests can clobber one
another so I have restricted Hudson to one executor.

It would be nice if Hudson had direct support for setting up a private
display for each job (if requested), at least launching Xvnc/Xnest/X on
Linux (don't know about other OS's), with a unique display number for
each concurrent job. Getting it right even for one OS is tricky, however.

You can always ask Hudson to run a shell script which sets up a special
display and then runs Ant (or whatever your build tool is). To handle
concurrent jobs, you could use $BUILD_NUMBER to compute a display
number, though it would be nicer if Hudson defined $EXECUTOR_NUMBER.

-J.

--
[hidden email]  x22801  netbeans.org  ant.apache.org
       http://google.com/search?q=e%5E%28pi*i%29%2B1

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using Hudson in productive environment

Kohsuke Kawaguchi-2
Jesse Glick wrote:
> You can always ask Hudson to run a shell script which sets up a special
> display and then runs Ant (or whatever your build tool is). To handle
> concurrent jobs, you could use $BUILD_NUMBER to compute a display
> number, though it would be nicer if Hudson defined $EXECUTOR_NUMBER.

Yeah, we talked about that when you were in Santa Clara. My bad. I just
implemented it.

With that, I think I'm going to release 1.61.

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using Hudson in productive environment

Kohsuke Kawaguchi-2
In reply to this post by Jesse Glick
Jesse Glick wrote:

> Tom Ball wrote:
>> I need to set a DISPLAY variable, so that a desktop application can
>> be tested using a virtual frame buffer. Headless AWT won't work
>> because headless components aren't supported.
>
> FWIW, I handle this by using a wrapper script to start Hudson:
>
> progdir=`dirname $0`
> unset XAUTHORITY
> Xvnc -localhost -geometry 1024x768 -depth 16 :69 &
> echo $! > $progdir/Xvnc.pid
> export DISPLAY=:69
> # ...
>
> and to stop:
>
> # ...
> progdir=`dirname $0`
> if [ -f $progdir/Xvnc.pid ] ; then
>      kill `head $progdir/Xvnc.pid`
>      rm $progdir/Xvnc.pid
> fi
>
> Unfortunately multiple simultaneous GUI-mode tests can clobber one
> another so I have restricted Hudson to one executor.
>
> It would be nice if Hudson had direct support for setting up a private
> display for each job (if requested), at least launching Xvnc/Xnest/X on
> Linux (don't know about other OS's), with a unique display number for
> each concurrent job. Getting it right even for one OS is tricky, however.
I wonder if there's VNC implemented in Java. Then I think that would
make some nice plugin. Hudson might be able to even let people see the
screen from hudson web pages real time!

The other thing is to write a plugin just for a particular environment,
which is always easier. We need to make a few tweaks to core Hudson (for
example so that you can export DISPLAY variable to the build that runs),
but otherwise executing the prepare script and clean up script shouldn't
be that hard.

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using Hudson in productive environment

Jesse Glick
Kohsuke Kawaguchi wrote:
> I wonder if there's VNC implemented in Java.

AFAIK there is a Java client, i.e. you could display an applet with the
live build, though there is no Java server (the listener on the X port):

http://www.realvnc.com/javavncviewer.html

In practice I don't think anyone would want to look at the display while
it's being used, though. You might while *developing* a GUI test, but
probably not while running it on a continuous builder. It's more
important for there to be some functional X display - otherwise the test
dies.

NB tests do have a special feature that if the test run dies (or
deadlocks) it will somehow save a PNG screenshot for later diagnosis,
but I don't think this could be easily generalized to other test harnesses.

-J.

--
[hidden email]  x22801  netbeans.org  ant.apache.org
       http://google.com/search?q=e%5E%28pi*i%29%2B1

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using Hudson in productive environment

Kohsuke Kawaguchi-2
Jesse Glick wrote:

> Kohsuke Kawaguchi wrote:
>> I wonder if there's VNC implemented in Java.
>
> AFAIK there is a Java client, i.e. you could display an applet with the
> live build, though there is no Java server (the listener on the X port):
>
> http://www.realvnc.com/javavncviewer.html
>
> In practice I don't think anyone would want to look at the display while
> it's being used, though. You might while *developing* a GUI test, but
> probably not while running it on a continuous builder.
True, except I just thought it would be kinda cool to be able to see it
(regardless of its actual usefulness :-)

 > It's more
> important for there to be some functional X display - otherwise the test
> dies.

I guess another area where working X display could be useful is some of
the AJAX client testing --- they often have this JavaScript based unit
test for browsers.

Would you like to file an RFE for that? Better yet :-) would you be
interested in working on it?

> NB tests do have a special feature that if the test run dies (or
> deadlocks) it will somehow save a PNG screenshot for later diagnosis,
> but I don't think this could be easily generalized to other test harnesses.


--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using Hudson in productive environment

Kohsuke Kawaguchi-2
In reply to this post by Sven Reimers
Sven Reimers wrote:

>> Cool. I have 150 jobs or so, but 400 is by far the biggest I've heard. I
>> wonder if you could talk some more about how you set things up --- I'm
>> trying to collect more "case study" like this, because I think it help
>> other people think about how they want to deploy it.
>>
> Setting things up worked like a breeze. A colleague of mine created a
> script to create the job configuration.xml files, some files that
> contained the parts we could put into the view configuration.xml. All
> this could easily be done, because our internal setup is quite well
> organized and named, else it could be a bit more problematic...
>
> If you want to know more just ask ;-)
OK, so now I put my interviewer hat on and here it goes...

- what server do you use for Hudson? OS? CPU? memory? what container?
how much disk does it use?

- do you use master/slave mode? If so, how many slaves? How do you
manage them?

- what kind of projects do you build on Hudson? You said 400 jobs -- are
they mostly software builds? Or do have other kind of jobs, like tests?
Are they chained? Do you use Ant? Maven?

- how does you manage dependencies between your projects? How does an
artifact of one module is consumed by another?

- when it's time to ship your milestone or final release, how do you do
it? Does that happen completely outside Hudson?


all right, that should do. If you could answer those, I'd write it up
and post it to http://hudson.dev.java.net/


>> > 1. Problems with svn polling: Actually there is no visual feedback if
>> > polling does not work (misconfiguration, server unavailable -
>> > whatever) - are there any plans to provide such a feedback - maybe by
>> > putting a badge on the job status icon?
>>
>> I believe each project page with SVN polling gets a new item that shows
>> the log. See the attached screen shot.
>>
> Yeah - but I have to look inside to determine if it worked out
> correclty or if some problem occured. The idea was to put this into a
> more visible place so that it is more obvoius that a build may not be
> uptodate.
OK. That makes sense. So I guess where to put them would be a question.

Maybe I can have a special "job" where its "builds" are each polling
run. So if any of the polling of any of the jobs fail, you'd see a red
ball in this job.

Another idea is maybe just provide a RSS feed from logs. If polling
failures are logged at a proper level, the sysadmin only needs to look
at a suitably filtered RSS from logs.

Yet another idea is to change the icon of the clipboard when the polling
is failing. It might not be visible enough, though.


>> But ultimately, for your environment, I recommend a push notification,
>> not polling. You can configure SVN repo to wget the
>> "http://foo.acme.org/hudson/job/myjob/build" URL as a post-commit
>> script, or have another machine do that when it receives e-mails.
>>
>> I do that for my Hudson installation, and it makes everyone happy.
>>
> I could possibly do this - but I have to put this in correct for all
> the projects - not that easy anyway - but maybe we should do both ;-)

With a bit of scripting and organized module layout in the SVN repo, I
think it shouldn't be too hard to write a simple program that parses the
change notification and trigger the build of the right job.

I have perhaps 10 line shell script that does it for my environment.

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using Hudson in productive environment

Sven Reimers
On 11/16/06, Kohsuke Kawaguchi <[hidden email]> wrote:

> Sven Reimers wrote:
> >> Cool. I have 150 jobs or so, but 400 is by far the biggest I've heard. I
> >> wonder if you could talk some more about how you set things up --- I'm
> >> trying to collect more "case study" like this, because I think it help
> >> other people think about how they want to deploy it.
> >>
> > Setting things up worked like a breeze. A colleague of mine created a
> > script to create the job configuration.xml files, some files that
> > contained the parts we could put into the view configuration.xml. All
> > this could easily be done, because our internal setup is quite well
> > organized and named, else it could be a bit more problematic...
> >
> > If you want to know more just ask ;-)
>
> OK, so now I put my interviewer hat on and here it goes...
>
> - what server do you use for Hudson? OS? CPU? memory? what container?
> how much disk does it use?
>
4 Systems each as follows:
Windows2003 Server
CPU Xeon 3.4Ghz
Memory: 2GB
Container - Tomcat

All behind an Apache WebServer (Compression, LDAP-AccessControl)
> - do you use master/slave mode? If so, how many slaves? How do you
> manage them?
>
Actually no master and slave but we are planning to set this up in the
near future
> - what kind of projects do you build on Hudson? You said 400 jobs -- are
> they mostly software builds? Or do have other kind of jobs, like tests?
> Are they chained? Do you use Ant? Maven?
>
SoftwareBuilds, Installer distributed between 4 build machines. One
Machine does MDA based codegeneration (>400 jobs, Java), one is for
manually written Java Code (>200), one is designated for building
native (MS VisualStudio .NET based), last machine builds Innosetup
based installers. Ant is the default, using a centralized
'buildsystem' containing a lot of infrastructure hacks (somehow
comparable to Maven - as far as processing is concerned, e.g. fetching
of binaries, javadoc, crossreferenced sources (sorcerer),
FindBugs,...)

Tests are part of the builds as well as seperate jobs for large test
chunks (including database) - we do love the JUnit integration display
from Hudson.

We are currently investigating chaining (that is automatic building of
dependencies) - one solution is to set this up as a seperate Job, that
triggers the dependencies at a specified time interval, since
rebuilding all those libraries on a continous base is not the way to
go.

> - how does you manage dependencies between your projects? How does an
> artifact of one module is consumed by another?
Dependencies is tricky, in fact they are defined in a special file
that is used during the build process. We are considering to use Ivy
(remember I asked for Ivy as a base for dependency management inside
Hudson ;-))
We use a central binary repository (similar to Maven) to post our
builds to. Those are fetched from the repository to be used in
building other libraries as well.
>
> - when it's time to ship your milestone or final release, how do you do
> it? Does that happen completely outside Hudson?
>
The installers are the ultimate deployment unit. Since those are based
on binary builds (taken from the described repository), there is no
need for a special treatment at that point in time, but I consider to
use the promote build feature, to this on a pre library basis. Maybe
even using a Hudson Job to tag along a bunch of jobs that are tied
together.
>
> all right, that should do. If you could answer those, I'd write it up
> and post it to http://hudson.dev.java.net/
>
Thx. For more info - just contact me ;-)

>
> >> > 1. Problems with svn polling: Actually there is no visual feedback if
> >> > polling does not work (misconfiguration, server unavailable -
> >> > whatever) - are there any plans to provide such a feedback - maybe by
> >> > putting a badge on the job status icon?
> >>
> >> I believe each project page with SVN polling gets a new item that shows
> >> the log. See the attached screen shot.
> >>
> > Yeah - but I have to look inside to determine if it worked out
> > correclty or if some problem occured. The idea was to put this into a
> > more visible place so that it is more obvoius that a build may not be
> > uptodate.
>
> OK. That makes sense. So I guess where to put them would be a question.
>
> Maybe I can have a special "job" where its "builds" are each polling
> run. So if any of the polling of any of the jobs fail, you'd see a red
> ball in this job.
>
> Another idea is maybe just provide a RSS feed from logs. If polling
> failures are logged at a proper level, the sysadmin only needs to look
> at a suitably filtered RSS from logs.
>
> Yet another idea is to change the icon of the clipboard when the polling
> is failing. It might not be visible enough, though.
>
How about adding a badge to the overall build state of the job - only
occuring if polling fails? But rss is ok as well I think.

>
> >> But ultimately, for your environment, I recommend a push notification,
> >> not polling. You can configure SVN repo to wget the
> >> "http://foo.acme.org/hudson/job/myjob/build" URL as a post-commit
> >> script, or have another machine do that when it receives e-mails.
> >>
> >> I do that for my Hudson installation, and it makes everyone happy.
> >>
> > I could possibly do this - but I have to put this in correct for all
> > the projects - not that easy anyway - but maybe we should do both ;-)
>
> With a bit of scripting and organized module layout in the SVN repo, I
> think it shouldn't be too hard to write a simple program that parses the
> change notification and trigger the build of the right job.
>
> I have perhaps 10 line shell script that does it for my environment.
>
I will pass this on to my colleague ;-)
> --
> Kohsuke Kawaguchi
> Sun Microsystems                   [hidden email]
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Using Hudson in productive environment

Kohsuke Kawaguchi-2
Sven Reimers wrote:

> On 11/16/06, Kohsuke Kawaguchi <[hidden email]> wrote:
>> Sven Reimers wrote:
>> >> Cool. I have 150 jobs or so, but 400 is by far the biggest I've heard. I
>> >> wonder if you could talk some more about how you set things up --- I'm
>> >> trying to collect more "case study" like this, because I think it help
>> >> other people think about how they want to deploy it.
>> >>
>> > Setting things up worked like a breeze. A colleague of mine created a
>> > script to create the job configuration.xml files, some files that
>> > contained the parts we could put into the view configuration.xml. All
>> > this could easily be done, because our internal setup is quite well
>> > organized and named, else it could be a bit more problematic...
>> >
>> > If you want to know more just ask ;-)
>>
>> OK, so now I put my interviewer hat on and here it goes...
>>
>> - what server do you use for Hudson? OS? CPU? memory? what container?
>> how much disk does it use?
>>
> 4 Systems each as follows:
> Windows2003 Server
> CPU Xeon 3.4Ghz
> Memory: 2GB
> Container - Tomcat
>
> All behind an Apache WebServer (Compression, LDAP-AccessControl)
>> - do you use master/slave mode? If so, how many slaves? How do you
>> manage them?
>>
> Actually no master and slave but we are planning to set this up in the
> near future
>> - what kind of projects do you build on Hudson? You said 400 jobs -- are
>> they mostly software builds? Or do have other kind of jobs, like tests?
>> Are they chained? Do you use Ant? Maven?
>>
> SoftwareBuilds, Installer distributed between 4 build machines. One
> Machine does MDA based codegeneration (>400 jobs, Java), one is for
> manually written Java Code (>200), one is designated for building
> native (MS VisualStudio .NET based), last machine builds Innosetup
> based installers. Ant is the default, using a centralized
> 'buildsystem' containing a lot of infrastructure hacks (somehow
> comparable to Maven - as far as processing is concerned, e.g. fetching
> of binaries, javadoc, crossreferenced sources (sorcerer),
> FindBugs,...)
>
> Tests are part of the builds as well as seperate jobs for large test
> chunks (including database) - we do love the JUnit integration display
> from Hudson.
>
> We are currently investigating chaining (that is automatic building of
> dependencies) - one solution is to set this up as a seperate Job, that
> triggers the dependencies at a specified time interval, since
> rebuilding all those libraries on a continous base is not the way to
> go.
>
>> - how does you manage dependencies between your projects? How does an
>> artifact of one module is consumed by another?
> Dependencies is tricky, in fact they are defined in a special file
> that is used during the build process. We are considering to use Ivy
> (remember I asked for Ivy as a base for dependency management inside
> Hudson ;-))
> We use a central binary repository (similar to Maven) to post our
> builds to. Those are fetched from the repository to be used in
> building other libraries as well.
>>
>> - when it's time to ship your milestone or final release, how do you do
>> it? Does that happen completely outside Hudson?
>>
> The installers are the ultimate deployment unit. Since those are based
> on binary builds (taken from the described repository), there is no
> need for a special treatment at that point in time, but I consider to
> use the promote build feature, to this on a pre library basis. Maybe
> even using a Hudson Job to tag along a bunch of jobs that are tied
> together.
>>
>> all right, that should do. If you could answer those, I'd write it up
>> and post it to http://hudson.dev.java.net/
>>
> Thx. For more info - just contact me ;-)
I posted it on https://hudson.dev.java.net/case-study-1.html
Does that look reasonable to you?

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Loading...