surefire m2 problem

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

surefire m2 problem

Nigel Magnay
I've got several projects that keep failing (but not always) like this:

started
$ java -cp E:\XAMPP\xampp\tomcat\webapps\hudson\WEB-INF\lib\maven-agent-1.129.jar;c:\maven\boot\classworlds-1.1.jar hudson.maven.agent.Main c:\maven E:\XAMPP\xampp\tomcat\webapps\hudson\WEB-INF\lib\remoting-1.129.jar E:\XAMPP\xampp\tomcat\webapps\hudson\WEB-INF\lib\maven-interceptor-1.129.jar
channel started
[INFO] Scanning for projects...
[INFO] ----------------------------------------------------------------------------
[INFO] Building KES OWL - Config
[INFO] task-segment: [clean, install]
[INFO] ----------------------------------------------------------------------------
[INFO] [clean:clean]
[INFO] Deleting directory C:\Documents and Settings\tomcat\.hudson\jobs\JnJ\workspace\trunk\owl-config\owl-config\target
[INFO] Deleting directory C:\Documents and Settings\tomcat\.hudson\jobs\JnJ\workspace\trunk\owl-config\owl-config\target\classes
[INFO] Deleting directory C:\Documents and Settings\tomcat\.hudson\jobs\JnJ\workspace\trunk\owl-config\owl-config\target\test-classes
[HUDSON] Archiving C:\Documents and Settings\tomcat\.hudson\jobs\JnJ\workspace\trunk\owl-config\owl-config\pom.xml
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] The plugin 'org.apache.maven.plugins:maven-surefire-plugin' does not exist or no valid version could be found
[INFO] ------------------------------------------------------------------------
[INFO] For more information, run Maven with the -e switch
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1 second
[INFO] Finished at: Fri Aug 17 09:43:28 BST 2007
[INFO] Final Memory: 3M/5M
[INFO] ------------------------------------------------------------------------
Sending e-mails to: [hidden email]
finished: FAILURE


I've set hudson.maven.ProcessCache.MAX_AGE=0; to make sure processes aren't getting re-used (I think there is a problem if a plugin is used 2 times with different versions).

Any ideas why it might be doing this? I've just upgraded to maven 2.0.7, but seems to be the same..

Reply | Threaded
Open this post in threaded view
|

RE: surefire m2 problem

Nord, James-2
I had a very similar issue.  Our proxy had been updated and rebooted whilst maven was checking for an update to plugins, as it couldn't reach the site it marked them as not available.
 
Deleting the affected areas of the local mavan repository cache helped and the next build succeeded.
 
/James


From: Nigel Magnay [mailto:[hidden email]]
Sent: 17 August 2007 09:48
To: [hidden email]
Subject: surefire m2 problem

I've got several projects that keep failing (but not always) like this:

started
$ java -cp E:\XAMPP\xampp\tomcat\webapps\hudson\WEB-INF\lib\maven-agent-1.129.jar;c:\maven\boot\classworlds-1.1.jar hudson.maven.agent.Main c:\maven E:\XAMPP\xampp\tomcat\webapps\hudson\WEB-INF\lib\remoting-1.129.jar E:\XAMPP\xampp\tomcat\webapps\hudson\WEB-INF\lib\maven-interceptor-1.129.jar
channel started
[INFO] Scanning for projects...
[INFO] ----------------------------------------------------------------------------
[INFO] Building KES OWL - Config
[INFO] task-segment: [clean, install]
[INFO] ----------------------------------------------------------------------------
[INFO] [clean:clean]
[INFO] Deleting directory C:\Documents and Settings\tomcat\.hudson\jobs\JnJ\workspace\trunk\owl-config\owl-config\target
[INFO] Deleting directory C:\Documents and Settings\tomcat\.hudson\jobs\JnJ\workspace\trunk\owl-config\owl-config\target\classes
[INFO] Deleting directory C:\Documents and Settings\tomcat\.hudson\jobs\JnJ\workspace\trunk\owl-config\owl-config\target\test-classes
[HUDSON] Archiving C:\Documents and Settings\tomcat\.hudson\jobs\JnJ\workspace\trunk\owl-config\owl-config\pom.xml
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] The plugin 'org.apache.maven.plugins:maven-surefire-plugin' does not exist or no valid version could be found
[INFO] ------------------------------------------------------------------------
[INFO] For more information, run Maven with the -e switch
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1 second
[INFO] Finished at: Fri Aug 17 09:43:28 BST 2007
[INFO] Final Memory: 3M/5M
[INFO] ------------------------------------------------------------------------
Sending e-mails to: [hidden email]
finished: FAILURE


I've set hudson.maven.ProcessCache.MAX_AGE=0; to make sure processes aren't getting re-used (I think there is a problem if a plugin is used 2 times with different versions).

Any ideas why it might be doing this? I've just upgraded to maven 2.0.7, but seems to be the same..

*********************************************************************************************************
This e-mail is confidential, the property of NDS Ltd and intended for the addressee only. Any dissemination, copying or distribution of this message or any attachments by anyone other than the intended recipient is strictly prohibited. If you have received this message in error, please immediately notify the [hidden email] and destroy the original message. Messages sent to and from NDS may be monitored. NDS cannot guarantee any message delivery method is secure or error-free. Information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. We do not accept responsibility for any errors or omissions in this message and/or attachment that arise as a result of transmission. You should carry out your own virus checks before opening any attachment. Any views or opinions presented are solely those of the author and do not necessarily represent those of NDS.

To protect the environment please do not print this e-mail unless necessary.

NDS Limited Registered office: One Heathrow Boulevard, 286 Bath Road, West Drayton, Middlesex, UB7 0DQ, United Kingdom. A company registered in England and Wales Registered no. 3080780 VAT no. GB 603 8808 40-00
**********************************************************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: surefire m2 problem

Kohsuke Kawaguchi
Administrator
In reply to this post by Nigel Magnay

In the past I had this happened to me outside Hudson, when I was
manually running Maven from my shell.

I never really figured out what's causing this (and Maven certainly
doesn't help me diagnose the problem!), but I recall wiping out relevant
part of the local repository, like Nord says.

So I suspect this is not really a Hudson issue, but if anyone has any
more info, please chime in.

Nigel Magnay wrote:

> I've got several projects that keep failing (but not always) like this:
>
> started
> $ java -cp E:\XAMPP\xampp\tomcat\webapps\hudson\WEB-INF\lib\maven-agent-1.129.jar;c:\maven\boot\classworlds-1.1.jar
> hudson.maven.agent.Main c:\maven
> E:\XAMPP\xampp\tomcat\webapps\hudson\WEB-INF\lib\remoting-1.129.jar
> E:\XAMPP\xampp\tomcat\webapps\hudson\WEB-INF\lib\maven-interceptor-1.129.jar
> channel started
> [INFO] Scanning for projects...
> [INFO] ----------------------------------------------------------------------------
> [INFO] Building KES OWL - Config
> [INFO]    task-segment: [clean, install]
> [INFO] ----------------------------------------------------------------------------
> [INFO] [clean:clean]
> [INFO] Deleting directory C:\Documents and
> Settings\tomcat\.hudson\jobs\JnJ\workspace\trunk\owl-config\owl-config\target
> [INFO] Deleting directory C:\Documents and
> Settings\tomcat\.hudson\jobs\JnJ\workspace\trunk\owl-config\owl-config\target\classes
> [INFO] Deleting directory C:\Documents and
> Settings\tomcat\.hudson\jobs\JnJ\workspace\trunk\owl-config\owl-config\target\test-classes
> [HUDSON] Archiving C:\Documents and
> Settings\tomcat\.hudson\jobs\JnJ\workspace\trunk\owl-config\owl-config\pom.xml
> [INFO] ------------------------------------------------------------------------
> [ERROR] BUILD ERROR
> [INFO] ------------------------------------------------------------------------
> [INFO] The plugin 'org.apache.maven.plugins:maven-surefire-plugin'
> does not exist or no valid version could be found
> [INFO] ------------------------------------------------------------------------
> [INFO] For more information, run Maven with the -e switch
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 1 second
> [INFO] Finished at: Fri Aug 17 09:43:28 BST 2007
> [INFO] Final Memory: 3M/5M
> [INFO] ------------------------------------------------------------------------
> Sending e-mails to: [hidden email]
> finished: FAILURE
>
>
> I've set hudson.maven.ProcessCache.MAX_AGE=0; to make sure processes aren't
> getting re-used (I think there is a problem if a plugin is used 2 times with
> different versions).
>
> Any ideas why it might be doing this? I've just upgraded to maven 2.0.7, but
> seems to be the same..
>

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: Re: surefire m2 problem

tvworks
I have found that this can happen with any plugin, not just surefire,
when running in hudson and using the "-U" with maven.  In particular, if
two builds start at roughly the same time a both of them use "-U" it can
be reproduced regularly.  I should add that they need to start on the
same machine.

-----Original Message-----
From: Kohsuke Kawaguchi [mailto:[hidden email]]
Sent: Friday, August 17, 2007 6:38 PM
To: [hidden email]
Subject: Re: surefire m2 problem


In the past I had this happened to me outside Hudson, when I was
manually running Maven from my shell.

I never really figured out what's causing this (and Maven certainly
doesn't help me diagnose the problem!), but I recall wiping out relevant

part of the local repository, like Nord says.

So I suspect this is not really a Hudson issue, but if anyone has any
more info, please chime in.

Nigel Magnay wrote:
> I've got several projects that keep failing (but not always) like
this:
>
> started
> $ java -cp
E:\XAMPP\xampp\tomcat\webapps\hudson\WEB-INF\lib\maven-agent-1.129.jar;c
:\maven\boot\classworlds-1.1.jar
> hudson.maven.agent.Main c:\maven
> E:\XAMPP\xampp\tomcat\webapps\hudson\WEB-INF\lib\remoting-1.129.jar
>
E:\XAMPP\xampp\tomcat\webapps\hudson\WEB-INF\lib\maven-interceptor-1.129
.jar
> channel started
> [INFO] Scanning for projects...
> [INFO]
------------------------------------------------------------------------
----
> [INFO] Building KES OWL - Config
> [INFO]    task-segment: [clean, install]
> [INFO]
------------------------------------------------------------------------
----
> [INFO] [clean:clean]
> [INFO] Deleting directory C:\Documents and
>
Settings\tomcat\.hudson\jobs\JnJ\workspace\trunk\owl-config\owl-config\t
arget
> [INFO] Deleting directory C:\Documents and
>
Settings\tomcat\.hudson\jobs\JnJ\workspace\trunk\owl-config\owl-config\t
arget\classes
> [INFO] Deleting directory C:\Documents and
>
Settings\tomcat\.hudson\jobs\JnJ\workspace\trunk\owl-config\owl-config\t
arget\test-classes
> [HUDSON] Archiving C:\Documents and
>
Settings\tomcat\.hudson\jobs\JnJ\workspace\trunk\owl-config\owl-config\p
om.xml
> [INFO]
------------------------------------------------------------------------
> [ERROR] BUILD ERROR
> [INFO]
------------------------------------------------------------------------
> [INFO] The plugin 'org.apache.maven.plugins:maven-surefire-plugin'
> does not exist or no valid version could be found
> [INFO]
------------------------------------------------------------------------
> [INFO] For more information, run Maven with the -e switch
> [INFO]
------------------------------------------------------------------------
> [INFO] Total time: 1 second
> [INFO] Finished at: Fri Aug 17 09:43:28 BST 2007
> [INFO] Final Memory: 3M/5M
> [INFO]
------------------------------------------------------------------------
> Sending e-mails to: [hidden email]
> finished: FAILURE
>
>
> I've set hudson.maven.ProcessCache.MAX_AGE=0; to make sure processes
aren't
> getting re-used (I think there is a problem if a plugin is used 2
times with
> different versions).
>
> Any ideas why it might be doing this? I've just upgraded to maven
2.0.7, but
> seems to be the same..
>


--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Re: surefire m2 problem

Nigel Magnay
I've deleted my .m2/repository directory to see what happens. Actually I'd quite like to set hudson to do this nightly - one of the common failure modes of m2 projects has been transitive dependencies pulled from remote repositories that have since been removed from the poms - this means it all works for existing developers but not for anyone starting from scratch.

I have found that this can happen with any plugin, not just surefire,
when running in hudson and using the "-U" with maven.  In particular, if
two builds start at roughly the same time a both of them use "-U" it can
be reproduced regularly.  I should add that they need to start on the
same machine.

Isn't this quite a serious problem? Given that m2 relies on a local repository, and that repostiory isn't designed to be accessed concurrently as it's just a filesystem? (in fact I'm sure I've seen similar errors to that effect on the m2 list) ?

Reply | Threaded
Open this post in threaded view
|

Re: surefire m2 problem

Stephen Connolly-2
I have been thinking about this type of problem a little.

What I am thinking is that each project should have it's own local
repository that gets wiped for every build... but that would be a lot of
downloading...

The downloading is not so bad if you are running a mirror of
repo1.maven.org locally on your site... but not everyone can afford to
allocate 25GB+ of disk space for such.

Then other problems like publishing artifacts and version control
between artifacts got me thinking.

Hudson could do with being a proxying & caching maven repository.  It
could give each build a unique settings.xml which pointed to the proxy
url for that build...

For example Job "goo" might be given
http://yourhost/hudson/maven/goo/repository as it's repository URL while
Job "goo is a silly name for a project peter" would be given
http://yourhost/hudson/maven/goo%20is%20a%20silly%20name%20for%20a%20project%20peter/repository

Thus hudson can ensure that each job gets a consistent view of it's
artifacts (as of the start of the job)

Anyone who wants to pick up artifacts from hudson would just use
http://yourhost/hudson/maven/repository to get the released artifacts

Hudson would be able to clear out old -SNAPSHOT artifacts when the next
release is available

Hudson would be able to publish to the
http://yourhost/hudson/maven/repository repository if and only if the
build completes successfully, meaning that developers pulling artifacts
from http://yourhost/hudson/maven/repository will only get artifacts
that are good.

Now the real sweet-spot would be if we could have a split local
repository in maven whereby there is a read-only (to maven but not
hudson) on disk local repository and a read-write local repository.  
Thus we could keep disk usage down to a minimum and stop downloading as
copying...

OK, enough rambling for now!

-Stephen Connolly
Nigel Magnay wrote:

> I've deleted my .m2/repository directory to see what happens. Actually
> I'd quite like to set hudson to do this nightly - one of the common
> failure modes of m2 projects has been transitive dependencies pulled
> from remote repositories that have since been removed from the poms -
> this means it all works for existing developers but not for anyone
> starting from scratch.
>
>     I have found that this can happen with any plugin, not just surefire,
>     when running in hudson and using the "-U" with maven.  In
>     particular, if
>     two builds start at roughly the same time a both of them use "-U"
>     it can
>     be reproduced regularly.  I should add that they need to start on the
>     same machine.
>
>
> Isn't this quite a serious problem? Given that m2 relies on a local
> repository, and that repostiory isn't designed to be accessed
> concurrently as it's just a filesystem? (in fact I'm sure I've seen
> similar errors to that effect on the m2 list) ?
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: surefire m2 problem

Nigel Magnay
Hmm - I have a *lot* of modules - but actually it might be useful to discover which modules won't build properly in isolation, because they rely on a transitive dependency fetched by a different upstream project.

Part of the problem is that m2 seems not to be designed to have concurrent accesses to the repository. This seems a real shame - it would be nice to have some sort of local repository "manager" running in the background at all times, synchronizing access to the repository itself and being able to answer questions like "what are the transitive dependencies required for artifact 'X'" and caching the results. I think a large amount of execution time is spent parsing and reparsing XML descriptors...

It's also a pity m2 doesn't do parallel builds (like hudson does). My desktop has 8 cores in it.. 7 of which sit pretty much idle whilst a build goes on.

On 18/08/07, Stephen Connolly <[hidden email]> wrote:
I have been thinking about this type of problem a little.

What I am thinking is that each project should have it's own local
repository that gets wiped for every build... but that would be a lot of
downloading...

The downloading is not so bad if you are running a mirror of
repo1.maven.org locally on your site... but not everyone can afford to
allocate 25GB+ of disk space for such.

Then other problems like publishing artifacts and version control
between artifacts got me thinking.

Hudson could do with being a proxying & caching maven repository.  It
could give each build a unique settings.xml which pointed to the proxy
url for that build...

For example Job "goo" might be given
http://yourhost/hudson/maven/goo/repository as it's repository URL while
Job "goo is a silly name for a project peter" would be given
http://yourhost/hudson/maven/goo%20is%20a%20silly%20name%20for%20a%20project%20peter/repository

Thus hudson can ensure that each job gets a consistent view of it's
artifacts (as of the start of the job)

Anyone who wants to pick up artifacts from hudson would just use
http://yourhost/hudson/maven/repository to get the released artifacts

Hudson would be able to clear out old -SNAPSHOT artifacts when the next
release is available

Hudson would be able to publish to the
http://yourhost/hudson/maven/repository repository if and only if the
build completes successfully, meaning that developers pulling artifacts
from http://yourhost/hudson/maven/repository will only get artifacts
that are good.

Now the real sweet-spot would be if we could have a split local
repository in maven whereby there is a read-only (to maven but not
hudson) on disk local repository and a read-write local repository.
Thus we could keep disk usage down to a minimum and stop downloading as
copying...

OK, enough rambling for now!

-Stephen Connolly
Nigel Magnay wrote:

> I've deleted my .m2/repository directory to see what happens. Actually
> I'd quite like to set hudson to do this nightly - one of the common
> failure modes of m2 projects has been transitive dependencies pulled
> from remote repositories that have since been removed from the poms -
> this means it all works for existing developers but not for anyone
> starting from scratch.
>
>     I have found that this can happen with any plugin, not just surefire,
>     when running in hudson and using the "-U" with maven.  In
>     particular, if
>     two builds start at roughly the same time a both of them use "-U"
>     it can
>     be reproduced regularly.  I should add that they need to start on the
>     same machine.
>
>
> Isn't this quite a serious problem? Given that m2 relies on a local
> repository, and that repostiory isn't designed to be accessed
> concurrently as it's just a filesystem? (in fact I'm sure I've seen
> similar errors to that effect on the m2 list) ?
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

RE: Re: surefire m2 problem

tvworks
In reply to this post by Stephen Connolly-2
How about getting the excellent developers/contributors involved with
hudson to rewrite maven the "right" way...at least with a nice object
model that makes sense.  :)

I guess this is a good time to comment that I have been quite impressed
with design/implementation of hudson.  The code is quite good and
actually makes sense...something I have not been able to say about
maven.

-----Original Message-----
From: Stephen Connolly [mailto:[hidden email]]
Sent: Saturday, August 18, 2007 11:07 AM
To: [hidden email]
Subject: Re: surefire m2 problem

I have been thinking about this type of problem a little.

What I am thinking is that each project should have it's own local
repository that gets wiped for every build... but that would be a lot of

downloading...

The downloading is not so bad if you are running a mirror of
repo1.maven.org locally on your site... but not everyone can afford to
allocate 25GB+ of disk space for such.

Then other problems like publishing artifacts and version control
between artifacts got me thinking.

Hudson could do with being a proxying & caching maven repository.  It
could give each build a unique settings.xml which pointed to the proxy
url for that build...

For example Job "goo" might be given
http://yourhost/hudson/maven/goo/repository as it's repository URL while

Job "goo is a silly name for a project peter" would be given
http://yourhost/hudson/maven/goo%20is%20a%20silly%20name%20for%20a%20pro
ject%20peter/repository

Thus hudson can ensure that each job gets a consistent view of it's
artifacts (as of the start of the job)

Anyone who wants to pick up artifacts from hudson would just use
http://yourhost/hudson/maven/repository to get the released artifacts

Hudson would be able to clear out old -SNAPSHOT artifacts when the next
release is available

Hudson would be able to publish to the
http://yourhost/hudson/maven/repository repository if and only if the
build completes successfully, meaning that developers pulling artifacts
from http://yourhost/hudson/maven/repository will only get artifacts
that are good.

Now the real sweet-spot would be if we could have a split local
repository in maven whereby there is a read-only (to maven but not
hudson) on disk local repository and a read-write local repository.  
Thus we could keep disk usage down to a minimum and stop downloading as
copying...

OK, enough rambling for now!

-Stephen Connolly
Nigel Magnay wrote:
> I've deleted my .m2/repository directory to see what happens. Actually

> I'd quite like to set hudson to do this nightly - one of the common
> failure modes of m2 projects has been transitive dependencies pulled
> from remote repositories that have since been removed from the poms -
> this means it all works for existing developers but not for anyone
> starting from scratch.
>
>     I have found that this can happen with any plugin, not just
surefire,
>     when running in hudson and using the "-U" with maven.  In
>     particular, if
>     two builds start at roughly the same time a both of them use "-U"
>     it can
>     be reproduced regularly.  I should add that they need to start on
the
>     same machine.
>
>
> Isn't this quite a serious problem? Given that m2 relies on a local
> repository, and that repostiory isn't designed to be accessed
> concurrently as it's just a filesystem? (in fact I'm sure I've seen
> similar errors to that effect on the m2 list) ?
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Re: surefire m2 problem

Nigel Magnay
Don't tempt me :-S. If m2 was as good (and as rapidly improved) as hudson, half these things wouldn't be a problem.

Don't get me wrong, m2 is massively useful, but the development cycle is so slow - probably the problem being that it now 'mostly works'. But the multitude of stuff that you have to understand feels like a Not-Invented-Here syndrome gone mad ( e.g. plexus, modello). To me that puts a massive barrier-to-entry in the way of potential contributors - and I think it needs them. I ditched continuum for Hudson because of stagnation, and the same for switching archiva for proximity.

Perhaps the goal is to make it just complex enough, so you get to open your own ALM consultancy company to tell everyone how it's supposed to be used... :-/

On 18/08/07, Jason Chaffee <[hidden email]> wrote:
How about getting the excellent developers/contributors involved with
hudson to rewrite maven the "right" way...at least with a nice object
model that makes sense.  :)

I guess this is a good time to comment that I have been quite impressed
with design/implementation of hudson.  The code is quite good and
actually makes sense...something I have not been able to say about
maven.

-----Original Message-----
From: Stephen Connolly [mailto:[hidden email]]
Sent: Saturday, August 18, 2007 11:07 AM
To: [hidden email]
Subject: Re: surefire m2 problem

I have been thinking about this type of problem a little.

What I am thinking is that each project should have it's own local
repository that gets wiped for every build... but that would be a lot of

downloading...

The downloading is not so bad if you are running a mirror of
repo1.maven.org locally on your site... but not everyone can afford to
allocate 25GB+ of disk space for such.

Then other problems like publishing artifacts and version control
between artifacts got me thinking.

Hudson could do with being a proxying & caching maven repository.  It
could give each build a unique settings.xml which pointed to the proxy
url for that build...

For example Job "goo" might be given
http://yourhost/hudson/maven/goo/repository as it's repository URL while

Job "goo is a silly name for a project peter" would be given
http://yourhost/hudson/maven/goo%20is%20a%20silly%20name%20for%20a%20pro
ject%20peter/repository

Thus hudson can ensure that each job gets a consistent view of it's
artifacts (as of the start of the job)

Anyone who wants to pick up artifacts from hudson would just use
http://yourhost/hudson/maven/repository to get the released artifacts

Hudson would be able to clear out old -SNAPSHOT artifacts when the next
release is available

Hudson would be able to publish to the
http://yourhost/hudson/maven/repository repository if and only if the
build completes successfully, meaning that developers pulling artifacts
from http://yourhost/hudson/maven/repository will only get artifacts
that are good.

Now the real sweet-spot would be if we could have a split local
repository in maven whereby there is a read-only (to maven but not
hudson) on disk local repository and a read-write local repository.
Thus we could keep disk usage down to a minimum and stop downloading as
copying...

OK, enough rambling for now!

-Stephen Connolly
Nigel Magnay wrote:
> I've deleted my .m2/repository directory to see what happens. Actually

> I'd quite like to set hudson to do this nightly - one of the common
> failure modes of m2 projects has been transitive dependencies pulled
> from remote repositories that have since been removed from the poms -
> this means it all works for existing developers but not for anyone
> starting from scratch.
>
>     I have found that this can happen with any plugin, not just
surefire,
>     when running in hudson and using the "-U" with maven.  In
>     particular, if
>     two builds start at roughly the same time a both of them use "-U"
>     it can
>     be reproduced regularly.  I should add that they need to start on
the
>     same machine.
>
>
> Isn't this quite a serious problem? Given that m2 relies on a local
> repository, and that repostiory isn't designed to be accessed
> concurrently as it's just a filesystem? (in fact I'm sure I've seen
> similar errors to that effect on the m2 list) ?
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: surefire m2 problem

Kohsuke Kawaguchi
Administrator
In reply to this post by Stephen Connolly-2

I have thought about making Hudson a Maven repository, too. In fact the
restfulness of it should also allow Hudson to become a nice repository
browser, as well. I have a large number of slaves, and making every
slave download from remote repositories don't make much sense.

But the main problem for me is about picking the battle. Hudson has tons
of RFEs filed already, and so it doesn't seem wise for me to try to do
everything. (But now if you are interested in doing this, that would be
a whole different issue!)


I'm trying to think of cheap and practical way to make concurrent builds
work, but this is certainly a tricky problem. Ensuring a consistent view
  as of the build start time is particularly hard to do cheaply.

Maybe it's possible to do something fairly clever by replacing
ArtifactResolver.


Stephen Connolly wrote:

> I have been thinking about this type of problem a little.
>
> What I am thinking is that each project should have it's own local
> repository that gets wiped for every build... but that would be a lot of
> downloading...
>
> The downloading is not so bad if you are running a mirror of
> repo1.maven.org locally on your site... but not everyone can afford to
> allocate 25GB+ of disk space for such.
>
> Then other problems like publishing artifacts and version control
> between artifacts got me thinking.
>
> Hudson could do with being a proxying & caching maven repository.  It
> could give each build a unique settings.xml which pointed to the proxy
> url for that build...
>
> For example Job "goo" might be given
> http://yourhost/hudson/maven/goo/repository as it's repository URL while
> Job "goo is a silly name for a project peter" would be given
> http://yourhost/hudson/maven/goo%20is%20a%20silly%20name%20for%20a%20project%20peter/repository
>
> Thus hudson can ensure that each job gets a consistent view of it's
> artifacts (as of the start of the job)
>
> Anyone who wants to pick up artifacts from hudson would just use
> http://yourhost/hudson/maven/repository to get the released artifacts
>
> Hudson would be able to clear out old -SNAPSHOT artifacts when the next
> release is available
>
> Hudson would be able to publish to the
> http://yourhost/hudson/maven/repository repository if and only if the
> build completes successfully, meaning that developers pulling artifacts
> from http://yourhost/hudson/maven/repository will only get artifacts
> that are good.
>
> Now the real sweet-spot would be if we could have a split local
> repository in maven whereby there is a read-only (to maven but not
> hudson) on disk local repository and a read-write local repository.  
> Thus we could keep disk usage down to a minimum and stop downloading as
> copying...
>
> OK, enough rambling for now!
>
> -Stephen Connolly
> Nigel Magnay wrote:
>> I've deleted my .m2/repository directory to see what happens. Actually
>> I'd quite like to set hudson to do this nightly - one of the common
>> failure modes of m2 projects has been transitive dependencies pulled
>> from remote repositories that have since been removed from the poms -
>> this means it all works for existing developers but not for anyone
>> starting from scratch.
>>
>>     I have found that this can happen with any plugin, not just surefire,
>>     when running in hudson and using the "-U" with maven.  In
>>     particular, if
>>     two builds start at roughly the same time a both of them use "-U"
>>     it can
>>     be reproduced regularly.  I should add that they need to start on the
>>     same machine.
>>
>>
>> Isn't this quite a serious problem? Given that m2 relies on a local
>> repository, and that repostiory isn't designed to be accessed
>> concurrently as it's just a filesystem? (in fact I'm sure I've seen
>> similar errors to that effect on the m2 list) ?
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: surefire m2 problem

Nigel Magnay
In reply to this post by Kohsuke Kawaguchi
Hmm - I've zapped my repo, but I'm still getting it - I've just had a failure finding maven-compiler-plugin. This has to be an oddity as most of the other projects are compiling (and I assume those rely on the same plugin too).

I'll try adding -X to the maven options, see if it spits out anything useful :-S

On 18/08/07, Kohsuke Kawaguchi <[hidden email]> wrote:

In the past I had this happened to me outside Hudson, when I was
manually running Maven from my shell.

I never really figured out what's causing this (and Maven certainly
doesn't help me diagnose the problem!), but I recall wiping out relevant
part of the local repository, like Nord says.

So I suspect this is not really a Hudson issue, but if anyone has any
more info, please chime in.

Nigel Magnay wrote:
> I've got several projects that keep failing (but not always) like this:
>

> started
> $ java -cp E:\XAMPP\xampp\tomcat\webapps\hudson\WEB-INF\lib\maven-agent-1.129.jar;c:\maven\boot\classworlds-1.1.jar
> hudson.maven.agent.Main c:\maven
> E:\XAMPP\xampp\tomcat\webapps\hudson\WEB-INF\lib\remoting- 1.129.jar
> E:\XAMPP\xampp\tomcat\webapps\hudson\WEB-INF\lib\maven-interceptor-1.129.jar
> channel started
> [INFO] Scanning for projects...
> [INFO] ----------------------------------------------------------------------------
> [INFO] Building KES OWL - Config
> [INFO]    task-segment: [clean, install]
> [INFO] ----------------------------------------------------------------------------
> [INFO] [clean:clean]
> [INFO] Deleting directory C:\Documents and
> Settings\tomcat\.hudson\jobs\JnJ\workspace\trunk\owl-config\owl-config\target
> [INFO] Deleting directory C:\Documents and
> Settings\tomcat\.hudson\jobs\JnJ\workspace\trunk\owl-config\owl-config\target\classes
> [INFO] Deleting directory C:\Documents and
> Settings\tomcat\.hudson\jobs\JnJ\workspace\trunk\owl-config\owl-config\target\test-classes
> [HUDSON] Archiving C:\Documents and
> Settings\tomcat\.hudson\jobs\JnJ\workspace\trunk\owl-config\owl-config\pom.xml
> [INFO] ------------------------------------------------------------------------
> [ERROR] BUILD ERROR
> [INFO] ------------------------------------------------------------------------
> [INFO] The plugin ' org.apache.maven.plugins:maven-surefire-plugin'
> does not exist or no valid version could be found
> [INFO] ------------------------------------------------------------------------
> [INFO] For more information, run Maven with the -e switch
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 1 second
> [INFO] Finished at: Fri Aug 17 09:43:28 BST 2007
> [INFO] Final Memory: 3M/5M
> [INFO] ------------------------------------------------------------------------
> Sending e-mails to: [hidden email]
> finished: FAILURE
>
>
> I've set hudson.maven.ProcessCache.MAX_AGE=0; to make sure processes aren't
> getting re-used (I think there is a problem if a plugin is used 2 times with
> different versions).
>
> Any ideas why it might be doing this? I've just upgraded to maven 2.0.7, but
> seems to be the same..
>


--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]


Reply | Threaded
Open this post in threaded view
|

RE: Re: Re: surefire m2 problem

tvworks
In reply to this post by Nigel Magnay

I agree.  The sad thing is that maven could have all the features and functionality that it currently does with half the code, it is just written so poorly…I imagine that is why the turn around on bug fixes and features takes so long.

 


From: Nigel Magnay [mailto:[hidden email]]
Sent: Saturday, August 18, 2007 1:38 PM
To: [hidden email]
Subject: Re: Re: surefire m2 problem

 

Don't tempt me :-S. If m2 was as good (and as rapidly improved) as hudson, half these things wouldn't be a problem.

Don't get me wrong, m2 is massively useful, but the development cycle is so slow - probably the problem being that it now 'mostly works'. But the multitude of stuff that you have to understand feels like a Not-Invented-Here syndrome gone mad ( e.g. plexus, modello). To me that puts a massive barrier-to-entry in the way of potential contributors - and I think it needs them. I ditched continuum for Hudson because of stagnation, and the same for switching archiva for proximity.

Perhaps the goal is to make it just complex enough, so you get to open your own ALM consultancy company to tell everyone how it's supposed to be used... :-/

On 18/08/07, Jason Chaffee <[hidden email]> wrote:

How about getting the excellent developers/contributors involved with
hudson to rewrite maven the "right" way...at least with a nice object
model that makes sense.  :)

I guess this is a good time to comment that I have been quite impressed
with design/implementation of hudson.  The code is quite good and
actually makes sense...something I have not been able to say about
maven.

-----Original Message-----
From: Stephen Connolly [mailto:[hidden email]]
Sent: Saturday, August 18, 2007 11:07 AM
To: [hidden email]
Subject: Re: surefire m2 problem

I have been thinking about this type of problem a little.

What I am thinking is that each project should have it's own local
repository that gets wiped for every build... but that would be a lot of

downloading...

The downloading is not so bad if you are running a mirror of
repo1.maven.org locally on your site... but not everyone can afford to
allocate 25GB+ of disk space for such.

Then other problems like publishing artifacts and version control
between artifacts got me thinking.

Hudson could do with being a proxying & caching maven repository.  It
could give each build a unique settings.xml which pointed to the proxy
url for that build...

For example Job "goo" might be given
http://yourhost/hudson/maven/goo/repository as it's repository URL while

Job "goo is a silly name for a project peter" would be given
http://yourhost/hudson/maven/goo%20is%20a%20silly%20name%20for%20a%20pro
ject%20peter/repository

Thus hudson can ensure that each job gets a consistent view of it's
artifacts (as of the start of the job)

Anyone who wants to pick up artifacts from hudson would just use
http://yourhost/hudson/maven/repository to get the released artifacts

Hudson would be able to clear out old -SNAPSHOT artifacts when the next
release is available

Hudson would be able to publish to the
http://yourhost/hudson/maven/repository repository if and only if the
build completes successfully, meaning that developers pulling artifacts
from http://yourhost/hudson/maven/repository will only get artifacts
that are good.

Now the real sweet-spot would be if we could have a split local
repository in maven whereby there is a read-only (to maven but not
hudson) on disk local repository and a read-write local repository.
Thus we could keep disk usage down to a minimum and stop downloading as
copying...

OK, enough rambling for now!

-Stephen Connolly
Nigel Magnay wrote:
> I've deleted my .m2/repository directory to see what happens. Actually

> I'd quite like to set hudson to do this nightly - one of the common
> failure modes of m2 projects has been transitive dependencies pulled
> from remote repositories that have since been removed from the poms -
> this means it all works for existing developers but not for anyone
> starting from scratch.
>
>     I have found that this can happen with any plugin, not just
surefire,
>     when running in hudson and using the "-U" with maven.  In
>     particular, if
>     two builds start at roughly the same time a both of them use "-U"
>     it can
>     be reproduced regularly.  I should add that they need to start on
the
>     same machine.
>
>
> Isn't this quite a serious problem? Given that m2 relies on a local
> repository, and that repostiory isn't designed to be accessed
> concurrently as it's just a filesystem? (in fact I'm sure I've seen
> similar errors to that effect on the m2 list) ?
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

 

Reply | Threaded
Open this post in threaded view
|

Re: surefire m2 problem

Nigel Magnay
In reply to this post by Kohsuke Kawaguchi
I've started trying to dig through the m2 code to understand how it works. Unfortunately it's completely undocumented, so it's a bigger task than it ought to be simply to get to the starting line.

I think I understand some of what I see in the output now - I wonder if it's the right thing:

If I have 2 projects, each with 2 modules:

A
-- module A1
-- module A2

B
-- module B1
-- module B2

if B2 depends on module A1, then after A1 finishes, it triggers a build of B2, even though it's in a different project. If A2 depends on A1, and B2 on B1:

So if A and B start building at the same time, each with build #100, then
A1#100 starts
B1#100 starts
--
A1#100 succeeds. This triggers A2#100 and B2#100.
--
A2#100 succeeds.
B2#100 fails, because it relies on a change in B1#100, that hasn't finished building yet.
--
B1#100 succeeds, triggers B2#101
--
B2#100 now succeeds.

I think this is probably fine, but it's a little bit confusing to view, as 'build 100' for B could actually have modules of different versions (100 or 101 in this case).




On 19/08/07, Kohsuke Kawaguchi <[hidden email]> wrote:

I have thought about making Hudson a Maven repository, too. In fact the
restfulness of it should also allow Hudson to become a nice repository
browser, as well. I have a large number of slaves, and making every
slave download from remote repositories don't make much sense.

But the main problem for me is about picking the battle. Hudson has tons
of RFEs filed already, and so it doesn't seem wise for me to try to do
everything. (But now if you are interested in doing this, that would be
a whole different issue!)


I'm trying to think of cheap and practical way to make concurrent builds
work, but this is certainly a tricky problem. Ensuring a consistent view
  as of the build start time is particularly hard to do cheaply.

Maybe it's possible to do something fairly clever by replacing
ArtifactResolver.


Stephen Connolly wrote:

> I have been thinking about this type of problem a little.
>
> What I am thinking is that each project should have it's own local
> repository that gets wiped for every build... but that would be a lot of
> downloading...
>
> The downloading is not so bad if you are running a mirror of
> repo1.maven.org locally on your site... but not everyone can afford to
> allocate 25GB+ of disk space for such.
>
> Then other problems like publishing artifacts and version control
> between artifacts got me thinking.
>
> Hudson could do with being a proxying & caching maven repository.  It
> could give each build a unique settings.xml which pointed to the proxy
> url for that build...
>
> For example Job "goo" might be given
> http://yourhost/hudson/maven/goo/repository as it's repository URL while
> Job "goo is a silly name for a project peter" would be given
> http://yourhost/hudson/maven/goo%20is%20a%20silly%20name%20for%20a%20project%20peter/repository
>
> Thus hudson can ensure that each job gets a consistent view of it's
> artifacts (as of the start of the job)
>
> Anyone who wants to pick up artifacts from hudson would just use
> http://yourhost/hudson/maven/repository to get the released artifacts
>
> Hudson would be able to clear out old -SNAPSHOT artifacts when the next
> release is available
>
> Hudson would be able to publish to the
> http://yourhost/hudson/maven/repository repository if and only if the
> build completes successfully, meaning that developers pulling artifacts
> from http://yourhost/hudson/maven/repository will only get artifacts
> that are good.
>
> Now the real sweet-spot would be if we could have a split local
> repository in maven whereby there is a read-only (to maven but not

> hudson) on disk local repository and a read-write local repository.
> Thus we could keep disk usage down to a minimum and stop downloading as
> copying...
>
> OK, enough rambling for now!
>
> -Stephen Connolly
> Nigel Magnay wrote:
>> I've deleted my .m2/repository directory to see what happens. Actually
>> I'd quite like to set hudson to do this nightly - one of the common
>> failure modes of m2 projects has been transitive dependencies pulled
>> from remote repositories that have since been removed from the poms -
>> this means it all works for existing developers but not for anyone
>> starting from scratch.
>>
>>     I have found that this can happen with any plugin, not just surefire,
>>     when running in hudson and using the "-U" with maven.  In
>>     particular, if

>>     two builds start at roughly the same time a both of them use "-U"
>>     it can
>>     be reproduced regularly.  I should add that they need to start on the
>>     same machine.
>>
>>
>> Isn't this quite a serious problem? Given that m2 relies on a local
>> repository, and that repostiory isn't designed to be accessed
>> concurrently as it's just a filesystem? (in fact I'm sure I've seen
>> similar errors to that effect on the m2 list) ?
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: surefire m2 problem

Kohsuke Kawaguchi
Administrator
Nigel Magnay wrote:
> I've started trying to dig through the m2 code to understand how it works.
> Unfortunately it's completely undocumented, so it's a bigger task than it
> ought to be simply to get to the starting line.

If you can make a suggestion about what kind of document you'd like to
see, that would be helpful. I usually try to put enough javadoc in all
the contracts, but it sounds like I'm failing there.

Or maybe what you want is a high-level view of how Hudson does its
native m2 support.

> I think I understand some of what I see in the output now - I wonder if it's
> the right thing:
>
> If I have 2 projects, each with 2 modules:
>
> A
> -- module A1
> -- module A2
>
> B
> -- module B1
> -- module B2
>
> if B2 depends on module A1, then after A1 finishes, it triggers a build of
> B2, even though it's in a different project.
Right.

Oh, so you are talking about user document that explains how this stuff
works. Yeah, we need that. I wonder, now that you know how it works,
would you be interested in putting whatever you can in the Wiki page?


 > If A2 depends on A1, and B2 on
> B1:
>
> So if A and B start building at the same time, each with build #100, then
> A1#100 starts
> B1#100 starts
> --
> A1#100 succeeds. This triggers A2#100 and B2#100.

Hmm, this is actually not supposed to happen. Hudson should notice that
B1 #100 is in progress, and so it shouldn't trigger B2 #100.

Sounds like you found a bug in there. If you can run it with
"hudson.maven.MavenBuild.debug=true;" from your scripting console,
Hudson will print out more messages about why it does or doesn't trigger
new builds.

If you can run the experiment with this option on, and send the
information to me (system log should show when the build has finished),
that would be really useful.

> --
> A2#100 succeeds.
> B2#100 fails, because it relies on a change in B1#100, that hasn't finished
> building yet.


> --
> B1#100 succeeds, triggers B2#101
> --
> B2#100 now succeeds.
>
> I think this is probably fine, but it's a little bit confusing to view, as
> 'build 100' for B could actually have modules of different versions (100 or
> 101 in this case).
>
>
>
>
> On 19/08/07, Kohsuke Kawaguchi <[hidden email]> wrote:
>>
>>
>> I have thought about making Hudson a Maven repository, too. In fact the
>> restfulness of it should also allow Hudson to become a nice repository
>> browser, as well. I have a large number of slaves, and making every
>> slave download from remote repositories don't make much sense.
>>
>> But the main problem for me is about picking the battle. Hudson has tons
>> of RFEs filed already, and so it doesn't seem wise for me to try to do
>> everything. (But now if you are interested in doing this, that would be
>> a whole different issue!)
>>
>>
>> I'm trying to think of cheap and practical way to make concurrent builds
>> work, but this is certainly a tricky problem. Ensuring a consistent view
>>   as of the build start time is particularly hard to do cheaply.
>>
>> Maybe it's possible to do something fairly clever by replacing
>> ArtifactResolver.
>>
>>
>> Stephen Connolly wrote:
>> > I have been thinking about this type of problem a little.
>> >
>> > What I am thinking is that each project should have it's own local
>> > repository that gets wiped for every build... but that would be a lot of
>> > downloading...
>> >
>> > The downloading is not so bad if you are running a mirror of
>> > repo1.maven.org locally on your site... but not everyone can afford to
>> > allocate 25GB+ of disk space for such.
>> >
>> > Then other problems like publishing artifacts and version control
>> > between artifacts got me thinking.
>> >
>> > Hudson could do with being a proxying & caching maven repository.  It
>> > could give each build a unique settings.xml which pointed to the proxy
>> > url for that build...
>> >
>> > For example Job "goo" might be given
>> > http://yourhost/hudson/maven/goo/repository as it's repository URL while
>> > Job "goo is a silly name for a project peter" would be given
>> >
>> http://yourhost/hudson/maven/goo%20is%20a%20silly%20name%20for%20a%20project%20peter/repository
>> >
>> > Thus hudson can ensure that each job gets a consistent view of it's
>> > artifacts (as of the start of the job)
>> >
>> > Anyone who wants to pick up artifacts from hudson would just use
>> > http://yourhost/hudson/maven/repository to get the released artifacts
>> >
>> > Hudson would be able to clear out old -SNAPSHOT artifacts when the next
>> > release is available
>> >
>> > Hudson would be able to publish to the
>> > http://yourhost/hudson/maven/repository repository if and only if the
>> > build completes successfully, meaning that developers pulling artifacts
>> > from http://yourhost/hudson/maven/repository will only get artifacts
>> > that are good.
>> >
>> > Now the real sweet-spot would be if we could have a split local
>> > repository in maven whereby there is a read-only (to maven but not
>> > hudson) on disk local repository and a read-write local repository.
>> > Thus we could keep disk usage down to a minimum and stop downloading as
>> > copying...
>> >
>> > OK, enough rambling for now!
>> >
>> > -Stephen Connolly
>> > Nigel Magnay wrote:
>> >> I've deleted my .m2/repository directory to see what happens. Actually
>> >> I'd quite like to set hudson to do this nightly - one of the common
>> >> failure modes of m2 projects has been transitive dependencies pulled
>> >> from remote repositories that have since been removed from the poms -
>> >> this means it all works for existing developers but not for anyone
>> >> starting from scratch.
>> >>
>> >>     I have found that this can happen with any plugin, not just
>> surefire,
>> >>     when running in hudson and using the "-U" with maven.  In
>> >>     particular, if
>> >>     two builds start at roughly the same time a both of them use "-U"
>> >>     it can
>> >>     be reproduced regularly.  I should add that they need to start on
>> the
>> >>     same machine.
>> >>
>> >>
>> >> Isn't this quite a serious problem? Given that m2 relies on a local
>> >> repository, and that repostiory isn't designed to be accessed
>> >> concurrently as it's just a filesystem? (in fact I'm sure I've seen
>> >> similar errors to that effect on the m2 list) ?
>> >>
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: [hidden email]
>> > For additional commands, e-mail: [hidden email]
>> >
>> >
>>
>>
>> --
>> Kohsuke Kawaguchi
>> Sun Microsystems                   [hidden email]
>>
>>
>

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: surefire m2 problem

Nigel Magnay
Sorry - I actually meant the documentation in m2, not hudson :-))

It seems to me that there's a general maven2 problem that the local repository assumes that it's only accessed by one process at once, as it just uses the filesystem to store artifacts and metadata.

What would be nice would be the ability to either synchronize particular parts of the repository whilst updates are in-flight, or perhaps by instead of using a filesystem for the repository, using a service to get/put the artifacts (together with the synchronization).

I don't know how hard that would be, because I don't know how embedded into the way m2 works the current behaviour is - but it seems to me that 'update artifacts and metadata' really ought to be an atomic operation.

I'm pretty sure I've come across this problem just as an m2 user, with half-failed updates being brought down from a remote repo..


On 21/08/07, Kohsuke Kawaguchi <[hidden email]> wrote:
Nigel Magnay wrote:
> I've started trying to dig through the m2 code to understand how it works.
> Unfortunately it's completely undocumented, so it's a bigger task than it
> ought to be simply to get to the starting line.

If you can make a suggestion about what kind of document you'd like to
see, that would be helpful. I usually try to put enough javadoc in all
the contracts, but it sounds like I'm failing there.

Or maybe what you want is a high-level view of how Hudson does its
native m2 support.

> I think I understand some of what I see in the output now - I wonder if it's
> the right thing:
>
> If I have 2 projects, each with 2 modules:
>
> A
> -- module A1
> -- module A2
>
> B
> -- module B1
> -- module B2
>
> if B2 depends on module A1, then after A1 finishes, it triggers a build of
> B2, even though it's in a different project.

Right.

Oh, so you are talking about user document that explains how this stuff
works. Yeah, we need that. I wonder, now that you know how it works,
would you be interested in putting whatever you can in the Wiki page?


> If A2 depends on A1, and B2 on
> B1:
>
> So if A and B start building at the same time, each with build #100, then
> A1#100 starts
> B1#100 starts
> --
> A1#100 succeeds. This triggers A2#100 and B2#100.

Hmm, this is actually not supposed to happen. Hudson should notice that
B1 #100 is in progress, and so it shouldn't trigger B2 #100.

Sounds like you found a bug in there. If you can run it with
" hudson.maven.MavenBuild.debug=true;" from your scripting console,
Hudson will print out more messages about why it does or doesn't trigger
new builds.

If you can run the experiment with this option on, and send the
information to me (system log should show when the build has finished),
that would be really useful.

> --
> A2#100 succeeds.
> B2#100 fails, because it relies on a change in B1#100, that hasn't finished
> building yet.


> --
> B1#100 succeeds, triggers B2#101
> --
> B2#100 now succeeds.
>
> I think this is probably fine, but it's a little bit confusing to view, as
> 'build 100' for B could actually have modules of different versions (100 or

> 101 in this case).
>
>
>
>
> On 19/08/07, Kohsuke Kawaguchi <[hidden email]> wrote:
>>
>>
>> I have thought about making Hudson a Maven repository, too. In fact the
>> restfulness of it should also allow Hudson to become a nice repository
>> browser, as well. I have a large number of slaves, and making every
>> slave download from remote repositories don't make much sense.
>>
>> But the main problem for me is about picking the battle. Hudson has tons
>> of RFEs filed already, and so it doesn't seem wise for me to try to do
>> everything. (But now if you are interested in doing this, that would be
>> a whole different issue!)
>>
>>
>> I'm trying to think of cheap and practical way to make concurrent builds
>> work, but this is certainly a tricky problem. Ensuring a consistent view
>>   as of the build start time is particularly hard to do cheaply.
>>
>> Maybe it's possible to do something fairly clever by replacing
>> ArtifactResolver.
>>
>>
>> Stephen Connolly wrote:
>> > I have been thinking about this type of problem a little.
>> >
>> > What I am thinking is that each project should have it's own local
>> > repository that gets wiped for every build... but that would be a lot of
>> > downloading...
>> >
>> > The downloading is not so bad if you are running a mirror of
>> > repo1.maven.org locally on your site... but not everyone can afford to
>> > allocate 25GB+ of disk space for such.
>> >
>> > Then other problems like publishing artifacts and version control
>> > between artifacts got me thinking.
>> >
>> > Hudson could do with being a proxying & caching maven repository.  It
>> > could give each build a unique settings.xml which pointed to the proxy
>> > url for that build...
>> >

>> > For example Job "goo" might be given
>> > http://yourhost/hudson/maven/goo/repository as it's repository URL while
>> > Job "goo is a silly name for a project peter" would be given
>> >
>> http://yourhost/hudson/maven/goo%20is%20a%20silly%20name%20for%20a%20project%20peter/repository
>> >
>> > Thus hudson can ensure that each job gets a consistent view of it's
>> > artifacts (as of the start of the job)
>> >
>> > Anyone who wants to pick up artifacts from hudson would just use
>> > http://yourhost/hudson/maven/repository to get the released artifacts
>> >
>> > Hudson would be able to clear out old -SNAPSHOT artifacts when the next
>> > release is available
>> >
>> > Hudson would be able to publish to the
>> > http://yourhost/hudson/maven/repository repository if and only if the

>> > build completes successfully, meaning that developers pulling artifacts
>> > from http://yourhost/hudson/maven/repository will only get artifacts
>> > that are good.
>> >
>> > Now the real sweet-spot would be if we could have a split local
>> > repository in maven whereby there is a read-only (to maven but not
>> > hudson) on disk local repository and a read-write local repository.
>> > Thus we could keep disk usage down to a minimum and stop downloading as
>> > copying...
>> >
>> > OK, enough rambling for now!
>> >
>> > -Stephen Connolly
>> > Nigel Magnay wrote:
>> >> I've deleted my .m2/repository directory to see what happens. Actually
>> >> I'd quite like to set hudson to do this nightly - one of the common
>> >> failure modes of m2 projects has been transitive dependencies pulled
>> >> from remote repositories that have since been removed from the poms -
>> >> this means it all works for existing developers but not for anyone
>> >> starting from scratch.
>> >>
>> >>     I have found that this can happen with any plugin, not just
>> surefire,
>> >>     when running in hudson and using the "-U" with maven.  In
>> >>     particular, if
>> >>     two builds start at roughly the same time a both of them use "-U"
>> >>     it can
>> >>     be reproduced regularly.  I should add that they need to start on
>> the
>> >>     same machine.
>> >>
>> >>
>> >> Isn't this quite a serious problem? Given that m2 relies on a local

>> >> repository, and that repostiory isn't designed to be accessed
>> >> concurrently as it's just a filesystem? (in fact I'm sure I've seen
>> >> similar errors to that effect on the m2 list) ?
>> >>
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: [hidden email]
>> > For additional commands, e-mail: [hidden email]
>> >
>> >
>>
>>
>> --
>> Kohsuke Kawaguchi
>> Sun Microsystems                   [hidden email]
>>
>>
>


--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]