m2 multiproject builds with svn:externals

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

m2 multiproject builds with svn:externals

Nigel Magnay
This might be a bug, or it might be me not understanding something.

We have a reasonably complex m2 build, consisting of several individual m2 multiprojects, tied together by a root product directory, consisting of just a pom.xml and svn:external definitions (This is what the cargo project does which probably serves as a better example).

I've got it imported into hudson, and I can see all the pom projects. On my local machine, I make a change to one of the sub-module files; this change is detected and a build starts. I can see the project and externals being correctly fetched. I have one file in the change:

/trunk/csw/kms/kes/modules/kms-app-ui/src/main/webapp/WEB-INF/.... etc

in my product (top-level) project, I have the mapping of

kes http://repo/csw/trunk/csw/kms/kes

In the logfile, I see after the checkout that hudson mentions the externals for which no change was detected, and this (correctly) doesn't include the project that has changed.

However, after "Parsing POMs", all I get is "Triggering com.cswgroup:kes-trunk" (which is the top level project) - it doesn't try to rebuild the Job that has the file changed (and, consequently, all the jobs that have that as a dependency)

I'm not sure why this is - do I have to have a pre-built version before it will generate a build?






Reply | Threaded
Open this post in threaded view
|

Re: m2 multiproject builds with svn:externals

Kohsuke Kawaguchi
Administrator
Nigel Magnay wrote:
> However, after "Parsing POMs", all I get is "Triggering
> com.cswgroup:kes-trunk" (which is the top level project) - it doesn't try to
> rebuild the Job that has the file changed (and, consequently, all the jobs
> that have that as a dependency)

Currently, even if a change only happens in a small sub part of the
project, Hudson tries to build all the modules. There's obviously a room
for improvements in this, and you are welcome to file RFEs, but just to
explain the thinking behind this:

- being able to assign the same build numbers to all modules are very
useful and easier for humans. If Hudson only builds affected modules,
you'd quickly get into "I need #15 of module A and #17 of module B,
...", which is rather intractable.

- it's tricky for Hudson to correctly infer what modules need to be
rebuilt from a file change. For example, a module can use files outside
its directory for build, by specifying a custom resource, source
directory, or any arbitrary configuration.

And I just found out today that there was some bug in Hudson where it
gets confused as to what modules need to be built after what modules,
and I was just fixing them. So maybe the same bug was affecting you.

Can you try the snapshot build and see if the problem is fixed? Even if
that doesn't fix it, the snapshot build has additional probe points to
let us find out what Hudson is thinking.

> I'm not sure why this is - do I have to have a pre-built version before it
> will generate a build?



--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: m2 multiproject builds with svn:externals

Nigel Magnay
Snapshot install seems to do the same thing

Currently, even if a change only happens in a small sub part of the
project, Hudson tries to build all the modules. There's obviously a room
for improvements in this, and you are welcome to file RFEs, but just to
explain the thinking behind this:

- being able to assign the same build numbers to all modules are very
useful and easier for humans. If Hudson only builds affected modules,
you'd quickly get into "I need #15 of module A and #17 of module B,
...", which is rather intractable.

- it's tricky for Hudson to correctly infer what modules need to be
rebuilt from a file change. For example, a module can use files outside
its directory for build, by specifying a custom resource, source
directory, or any arbitrary configuration.

Hm - I wonder what things like continuum do. I'd assumed that they regarded a change in any file in any subdirectory of a module as a change in the module - would that be enough for most cases?

When we used continuum, we ended up with 2 builds - 1 build with all individual modules, and 1 build of 'everything'. The 1st was for quick feedback of 'this module has broken', and the 2nd for 'this whole project has broken'

And I just found out today that there was some bug in Hudson where it
gets confused as to what modules need to be built after what modules,
and I was just fixing them. So maybe the same bug was affecting you.

Can you try the snapshot build and see if the problem is fixed? Even if
that doesn't fix it, the snapshot build has additional probe points to
let us find out what Hudson is thinking.

> I'm not sure why this is - do I have to have a pre-built version before it
> will generate a build?



--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: m2 multiproject builds with svn:externals

Kohsuke Kawaguchi
Administrator
2007/4/13, Nigel Magnay <[hidden email]>:
> Snapshot install seems to do the same thing

Just to check, does the triggering of the top level module eventually
caused all the modules to be rebuilt? Or are you seeing that your
modules are not built at all?

> Hm - I wonder what things like continuum do. I'd assumed that they regarded
> a change in any file in any subdirectory of a module as a change in the
> module - would that be enough for most cases?

I think it's a reasonable layout that worth supporting for (but I know
it does not apply for some projects on my production system.)

Doing this also requires that Hudson does build things differently
depending on how a build is triggered; for example when you manually
click "build" link you'd like to see the full rebuild.

RFE would be appreciated.

--
Kohsuke Kawaguchi

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: m2 multiproject builds with svn:externals

Nigel Magnay
It's triggering, building the very top level project (which is just a POM), but nothing else.

On 13/04/07, Kohsuke Kawaguchi <[hidden email]> wrote:
2007/4/13, Nigel Magnay <[hidden email]>:
> Snapshot install seems to do the same thing

Just to check, does the triggering of the top level module eventually
caused all the modules to be rebuilt? Or are you seeing that your
modules are not built at all?

> Hm - I wonder what things like continuum do. I'd assumed that they regarded
> a change in any file in any subdirectory of a module as a change in the
> module - would that be enough for most cases?

I think it's a reasonable layout that worth supporting for (but I know
it does not apply for some projects on my production system.)

Doing this also requires that Hudson does build things differently
depending on how a build is triggered; for example when you manually
click "build" link you'd like to see the full rebuild.

RFE would be appreciated.

--
Kohsuke Kawaguchi

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: m2 multiproject builds with svn:externals

Kohsuke Kawaguchi
Administrator
At the end of the POM build, you are supposed to see further
triggering of child modules. But sounds like that's not happening.

Can you go to scripting console and enter
"hudson.maven.MavenBuild.debug=true;" ?

That should show us more info.


2007/4/13, Nigel Magnay <[hidden email]>:

> It's triggering, building the very top level project (which is just a POM),
> but nothing else.
>
>
> On 13/04/07, Kohsuke Kawaguchi < [hidden email]> wrote:
> >
> > 2007/4/13, Nigel Magnay < [hidden email]>:
> > > Snapshot install seems to do the same thing
> >
> > Just to check, does the triggering of the top level module eventually
> > caused all the modules to be rebuilt? Or are you seeing that your
> > modules are not built at all?
> >
> > > Hm - I wonder what things like continuum do. I'd assumed that they
> regarded
> > > a change in any file in any subdirectory of a module as a change in the
> > > module - would that be enough for most cases?
> >
> > I think it's a reasonable layout that worth supporting for (but I know
> > it does not apply for some projects on my production system.)
> >
> > Doing this also requires that Hudson does build things differently
> > depending on how a build is triggered; for example when you manually
> > click "build" link you'd like to see the full rebuild.
> >
> > RFE would be appreciated.
> >
> > --
> > Kohsuke Kawaguchi
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
>


--
Kohsuke Kawaguchi

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: m2 multiproject builds with svn:externals

Nigel Magnay
Ok - I've just updated to the latest snapshot, and re-created my project just to be sure.

Unfortunately, on the 1st update, I get a

FATAL: Unable to delete C:\Documents and Settings\tomcat\.hudson\jobs\KES\workspace\trunk

java.io.IOException: Unable to delete C:\Documents and Settings\tomcat\.hudson\jobs\KES\workspace\trunk
at hudson.Util.deleteFile(Util.java:140)
at hudson.Util.deleteContentsRecursive(Util.java:118)
at hudson.scm.SubversionSCM$1.invoke (SubversionSCM.java:297)
at hudson.scm.SubversionSCM$1.invoke(SubversionSCM.java:278)
at hudson.FilePath.act(FilePath.java:238)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:278)
at hudson.scm.SubversionSCM.checkout (SubversionSCM.java:236)
at hudson.model.AbstractProject.checkout(AbstractProject.java:306)
at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:152)
at hudson.model.AbstractBuild$AbstractRunner.run (AbstractBuild.java:118)
at hudson.model.Run.run(Run.java:559)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:130)
at hudson.model.Executor.run(Executor.java:61)
Process Explorer confirms that it's java that has the file open, so there's probably a hanging handle somewhere. If I restart tomcat, I get an error in the svn update log of

org.tmatesoft.svn.core.SVNException: svn: 'C:\Documents and Settings\tomcat\.hudson\jobs\KES\workspace\trunk' is not a working copy
svn: Cannot read from to 'C:\Documents and Settings\tomcat\.hudson\jobs\KES\workspace\trunk\.svn\format': path refers to directory or read access is denied
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:49)
at org.tmatesoft.svn.core.internal.wc.admin.SVNAdminAreaFactory.open(SVNAdminAreaFactory.java:132)
at org.tmatesoft.svn.core.internal.wc.admin.SVNWCAccess.doOpen (SVNWCAccess.java:332)
at org.tmatesoft.svn.core.internal.wc.admin.SVNWCAccess.open(SVNWCAccess.java:250)
at org.tmatesoft.svn.core.internal.wc.admin.SVNWCAccess.probeOpen(SVNWCAccess.java:268)
at org.tmatesoft.svn.core.wc.SVNWCClient.doInfo (SVNWCClient.java:1817)
at org.tmatesoft.svn.core.wc.SVNWCClient.doInfo(SVNWCClient.java:1754)
at org.tmatesoft.svn.core.wc.SVNWCClient.doInfo(SVNWCClient.java:2108)
at hudson.scm.SubversionSCM$2.invoke(SubversionSCM.java :405)
at hudson.scm.SubversionSCM$2.invoke(SubversionSCM.java:389)
at hudson.FilePath.act(FilePath.java:238)
at hudson.scm.SubversionSCM.buildRevisionMap(SubversionSCM.java:389)
at hudson.scm.SubversionSCM.pollChanges (SubversionSCM.java:473)
at hudson.model.AbstractProject.pollSCMChanges(AbstractProject.java:339)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:256)
at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java :273)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
(it's deleted everything bar the trunk directory). If I delete that, then it complains about workspace not being a working copy.. so it's got a bit unhappy :-S




On 13/04/07, Kohsuke Kawaguchi <[hidden email]> wrote:
At the end of the POM build, you are supposed to see further
triggering of child modules. But sounds like that's not happening.

Can you go to scripting console and enter
"hudson.maven.MavenBuild.debug=true ;" ?

That should show us more info.


2007/4/13, Nigel Magnay <[hidden email]>:

> It's triggering, building the very top level project (which is just a POM),
> but nothing else.
>
>
> On 13/04/07, Kohsuke Kawaguchi < [hidden email]> wrote:
> >
> > 2007/4/13, Nigel Magnay < [hidden email]>:
> > > Snapshot install seems to do the same thing
> >
> > Just to check, does the triggering of the top level module eventually
> > caused all the modules to be rebuilt? Or are you seeing that your
> > modules are not built at all?
> >
> > > Hm - I wonder what things like continuum do. I'd assumed that they
> regarded
> > > a change in any file in any subdirectory of a module as a change in the
> > > module - would that be enough for most cases?
> >
> > I think it's a reasonable layout that worth supporting for (but I know
> > it does not apply for some projects on my production system.)
> >
> > Doing this also requires that Hudson does build things differently
> > depending on how a build is triggered; for example when you manually
> > click "build" link you'd like to see the full rebuild.
> >
> > RFE would be appreciated.
> >
> > --
> > Kohsuke Kawaguchi
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
>


--
Kohsuke Kawaguchi

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: m2 multiproject builds with svn:externals

Nigel Magnay
In reply to this post by Kohsuke Kawaguchi
Part of the problem seems to be that the builds don't work if update isn't checked; so I've done that, but it's still not rebuilding everything and the logs don't say much more with hudson.maven.MavenBuild.debug=true .

I get

Apr 17, 2007 2:56:52 PM hudson.model.Run run
INFO: com.cswgroup:kes-trunk #9 main build action completed: SUCCESS
Apr 17, 2007 2:56:52 PM hudson.model.Run setResult
INFO: com.cswgroup:kes-trunk #9 : result is set to SUCCESS by hudson.model.Run.run(Run.java:559)
In the hudson log, and the build results give the following (it's only built com.cswgroup:kes-trunk, not each of the sub-projects)

started
Updating http://csw-00701.cswgroup.local/csw/products/kes/trunk
Fetching external item into 'C:\Documents and Settings\tomcat\.hudson\jobs\KES\workspace\trunk\commons'
External at revision -1
commons
At revision 2592
Fetching external item into 'C:\Documents and Settings\tomcat\.hudson\jobs\KES\workspace\trunk\integration-test'
External at revision -1
integration-test
At revision 2592
Fetching external item into 'C:\Documents and Settings\tomcat\.hudson\jobs\KES\workspace\trunk\kes'
External at revision -1
U kes/modules/kms-app-ui/src/main/webapp/resources/css/styles.css
kes/modules/kms-app-ui/src/main/webapp/resources/css
kes/modules/kms-app-ui/src/main/webapp/resources
kes/modules/kms-app-ui/src/main/webapp
kes/modules/kms-app-ui/src/main
kes/modules/kms-app-ui/src
kes/modules/kms-app-ui
kes/modules
kes
At revision 2592
Fetching external item into 'C:\Documents and Settings\tomcat\.hudson\jobs\KES\workspace\trunk\pom'
External at revision -1
pom
At revision 2592
Fetching external item into 'C:\Documents and Settings\tomcat\.hudson\jobs\KES\workspace\trunk\safeharbour'
External at revision -1
safeharbour
At revision 2592
Fetching external item into 'C:\Documents and Settings\tomcat\.hudson\jobs\KES\workspace\trunk\sample-publication'
External at revision -1
sample-publication
At revision 2592
At revision 2592
no change for http://csw-00701.cswgroup.local/csw/products/kes/trunk since the previous build
no change for http://csw-00701.cswgroup.local/csw/trunk/csw/commons since the previous build
no change for http://csw-00701.cswgroup.local/csw/packages/kms/kes-tests/trunk/integration since the previous build
no change for http://csw-00701.cswgroup.local/csw/pom/trunk since the previous build
no change for http://csw-00701.cswgroup.local/csw/trunk/csw/safeharbour since the previous build
no change for http://csw-00701.cswgroup.local/csw/packages/kms/kes-sample-data/trunk/workflow-guide since the previous build
Parsing POMs
Triggering com.cswgroup:kes-trunk
finished: SUCCESS




On 13/04/07, Kohsuke Kawaguchi <[hidden email]> wrote:
At the end of the POM build, you are supposed to see further
triggering of child modules. But sounds like that's not happening.

Can you go to scripting console and enter
"hudson.maven.MavenBuild.debug=true ;" ?

That should show us more info.


2007/4/13, Nigel Magnay <[hidden email]>:

> It's triggering, building the very top level project (which is just a POM),
> but nothing else.
>
>
> On 13/04/07, Kohsuke Kawaguchi < [hidden email]> wrote:
> >
> > 2007/4/13, Nigel Magnay < [hidden email]>:
> > > Snapshot install seems to do the same thing
> >
> > Just to check, does the triggering of the top level module eventually
> > caused all the modules to be rebuilt? Or are you seeing that your
> > modules are not built at all?
> >
> > > Hm - I wonder what things like continuum do. I'd assumed that they
> regarded
> > > a change in any file in any subdirectory of a module as a change in the
> > > module - would that be enough for most cases?
> >
> > I think it's a reasonable layout that worth supporting for (but I know
> > it does not apply for some projects on my production system.)
> >
> > Doing this also requires that Hudson does build things differently
> > depending on how a build is triggered; for example when you manually
> > click "build" link you'd like to see the full rebuild.
> >
> > RFE would be appreciated.
> >
> > --
> > Kohsuke Kawaguchi
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
>


--
Kohsuke Kawaguchi

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]