m2 projects : --non-recursive

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

m2 projects : --non-recursive

Nigel Magnay
Hello

Firstly I'd like to say hudson is really great! I've been bashing my head against continuum for too long, so it's a breath of fresh air!

One thing that I'd like to be able to do - I'm assuming that hudson executes m2 projects with --non-recursive. I'd like to be able to turn this off, as I have a 'root level build everything' project.

Also, is there a way of having multiple triggers? e.g. in continuum, we'd set a project to build every hour with 'install', and every night with 'install site site:deploy' parameters.

Nigel

Reply | Threaded
Open this post in threaded view
|

Re: m2 projects : --non-recursive

Rhett Sutphin
Hi,

> One thing that I'd like to be able to do - I'm assuming that hudson  
> executes m2 projects with --non-recursive. I'd like to be able to  
> turn this off, as I have a 'root level build everything' project.

You can do this with the freestyle build option -- if you don't want  
to build the modules separately in sequence, I don't think there's an  
advantage to using the native m2 build style.

> Also, is there a way of having multiple triggers? e.g. in  
> continuum, we'd set a project to build every hour with 'install',  
> and every night with 'install site site:deploy' parameters.

AFAICT, the only way to do this is with multiple projects.

Rhett

On Mar 6, 2007, at 5:24 AM, Nigel Magnay wrote:

> Hello
>
> Firstly I'd like to say hudson is really great! I've been bashing  
> my head against continuum for too long, so it's a breath of fresh air!
>
> One thing that I'd like to be able to do - I'm assuming that hudson  
> executes m2 projects with --non-recursive. I'd like to be able to  
> turn this off, as I have a 'root level build everything' project.
>
> Also, is there a way of having multiple triggers? e.g. in  
> continuum, we'd set a project to build every hour with 'install',  
> and every night with 'install site site:deploy' parameters.
>
> Nigel
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: m2 projects : --non-recursive

Kohsuke Kawaguchi-2
In reply to this post by Nigel Magnay
Nigel Magnay wrote:
> Hello
>
> Firstly I'd like to say hudson is really great! I've been bashing my head
> against continuum for too long, so it's a breath of fresh air!
>
> One thing that I'd like to be able to do - I'm assuming that hudson executes
> m2 projects with --non-recursive. I'd like to be able to turn this off, as I
> have a 'root level build everything' project.

Can you elaborate a bit more on what you mean by "root level build
everything" ?

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: m2 projects : --non-recursive

Nigel Magnay
Sure -

If I have a project:

 Proj (pom)
  |-- Module1 (jar)
  |-- Module2 (jar)

If I import into hudson, I'll get 3 projects. If I change a file in Module1, module1 will get rebuilt.

If I build the 'Proj' project, it'll do very little, just parse the pom.xml file and maybe install it to the local repository. This is because we run
m2 --non-recursive.

However, if I want to do a 'full rebuild on everything with a consistent SVN version', I'd like to be able to build Proj without the --non-recursive ( e.g. equivalent to just doing "mvn install" in that directory).

On 07/03/07, Kohsuke Kawaguchi <[hidden email]> wrote:
Nigel Magnay wrote:
> Hello
>
> Firstly I'd like to say hudson is really great! I've been bashing my head
> against continuum for too long, so it's a breath of fresh air!
>
> One thing that I'd like to be able to do - I'm assuming that hudson executes
> m2 projects with --non-recursive. I'd like to be able to turn this off, as I
> have a 'root level build everything' project.

Can you elaborate a bit more on what you mean by "root level build
everything" ?

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: m2 projects : --non-recursive

Tomasz Sterna-2
Dnia 08-03-2007, czw o godzinie 09:04 +0000, Nigel Magnay napisał(a):
> However, if I want to do a 'full rebuild on everything with a
> consistent SVN version', I'd like to be able to build Proj without the
> --non-recursive ( e.g. equivalent to just doing "mvn install" in that
> directory).

What is the rationale behind it? Why would you want to do that?


--
Tomasz Sterna
eo Networks

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: m2 projects : --non-recursive

Nigel Magnay
Firstly, because module level builds can only ever tell you a small part of the overall picture, and rebuilding just the 'declared' dependencies isn't ever necessarily enough (any project of any size will be able to show you cases where code using a shared API suddenly breaks for unexpected reasons).

Secondly, because of QA. Dev want module builds 'immediately'. Agile QA wants builds often - usually once a day. But what they really don't want in answer to the question 'what version of the software is this' is 'I don't know' or '3244 for module1, 3248 for module2, 3000 for module 3', etc. It's next to impossible to recreate, and otherwise without it forces you to go into a different mode of working when you start code freezing towards a release. Having a 'striped' version is essential in these cases.

On 08/03/07, Tomasz Sterna <[hidden email]> wrote:
Dnia 08-03-2007, czw o godzinie 09:04 +0000, Nigel Magnay napisał(a):
> However, if I want to do a 'full rebuild on everything with a
> consistent SVN version', I'd like to be able to build Proj without the
> --non-recursive ( e.g. equivalent to just doing "mvn install" in that
> directory).

What is the rationale behind it? Why would you want to do that?


--
Tomasz Sterna
eo Networks

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: m2 projects : --non-recursive

Kohsuke Kawaguchi-2
In reply to this post by Nigel Magnay
Nigel Magnay wrote:

> Sure -
>
> If I have a project:
>
>  Proj (pom)
>   |-- Module1 (jar)
>   |-- Module2 (jar)
>
> If I import into hudson, I'll get 3 projects. If I change a file in Module1,
> module1 will get rebuilt.
>
> If I build the 'Proj' project, it'll do very little, just parse the
> pom.xmlfile and maybe install it to the local repository. This is
> because we run
> m2 --non-recursive.
>
> However, if I want to do a 'full rebuild on everything with a consistent SVN
> version', I'd like to be able to build Proj without the --non-recursive (e.g.
> equivalent to just doing "mvn install" in that directory).
Ah, but what Hudson does after it builds 'Proj' module is to trigger the
builds of module1 and module2. If you look at the console output it'll
report this at the very end.

So essentially hudson does the equivalent of "mvn install" in a
recursive fashion.

Maybe you are saying that in your environment hudson doesn't seem to
trigger downstream builds?

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: m2 projects : --non-recursive

Kohsuke Kawaguchi-2
In reply to this post by Nigel Magnay
Nigel Magnay wrote:
> Secondly, because of QA. Dev want module builds 'immediately'. Agile QA
> wants builds often - usually once a day. But what they really don't want in
> answer to the question 'what version of the software is this' is 'I don't
> know' or '3244 for module1, 3248 for module2, 3000 for module 3', etc.

That's exactly why Hudson aligns build numbers among modules. When you
start #3244 of the Proj module, all other modules will get the same
build number.

(Or did I somehow implement smarter triggering algorithm that only
rebuilds the changed modules?)

I suspect we are really on the same page wrt the design of Hudson and
what you'd like to see. I suspect there's some bugs that cause Hudson to
behave differently from its design.

Perhaps it might be helpful if you can explain the actual behaviors that
you are seeing.

 > It's
> next to impossible to recreate, and otherwise without it forces you to go
> into a different mode of working when you start code freezing towards a
> release. Having a 'striped' version is essential in these cases.

>
> On 08/03/07, Tomasz Sterna <[hidden email]> wrote:
>>
>> Dnia 08-03-2007, czw o godzinie 09:04 +0000, Nigel Magnay napisał(a):
>> > However, if I want to do a 'full rebuild on everything with a
>> > consistent SVN version', I'd like to be able to build Proj without the
>> > --non-recursive ( e.g. equivalent to just doing "mvn install" in that
>> > directory).
>>
>> What is the rationale behind it? Why would you want to do that?
>>
>>
>> --
>> Tomasz Sterna
>> eo Networks
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: m2 projects : --non-recursive

Nigel Magnay
In reply to this post by Kohsuke Kawaguchi-2


Ah, but what Hudson does after it builds 'Proj' module is to trigger the
builds of module1 and module2. If you look at the console output it'll
report this at the very end.

So essentially hudson does the equivalent of "mvn install" in a
recursive fashion.

Maybe you are saying that in your environment hudson doesn't seem to
trigger downstream builds?

The main issue with "just" triggering is that It's all very well having the same logical build number, but unless that build number is tied to a *single* subversion revision number it's not useful.

What I need (and this isn't very complex) is the ability to stop hudson passing the '--non-recursive' parameter to mvn on building.

What would be even more useful would be the ability for each project item to be allowed to have more than one build parameter. So, for example, build every night with 'mvn install site site:deploy', and after every checkin with just 'mvn install'.


Reply | Threaded
Open this post in threaded view
|

many schedules per job [was Re: m2 projects : --non-recursive]

Tomasz Sterna-2
Dnia 09-03-2007, pią o godzinie 11:21 +0000, Nigel Magnay napisał(a):
> What would be even more useful would be the ability for each project
> item to be allowed to have more than one build parameter. So, for
> example, build every night with 'mvn install site site:deploy', and
> after every checkin with just 'mvn install'.

This is exactly what I proposed in
https://hudson.dev.java.net/issues/show_bug.cgi?id=346

--
Tomasz Sterna
eo Network

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: m2 projects : --non-recursive

Kohsuke Kawaguchi-2
In reply to this post by Nigel Magnay
Nigel Magnay wrote:

>>
>>
>>
>> Ah, but what Hudson does after it builds 'Proj' module is to trigger the
>> builds of module1 and module2. If you look at the console output it'll
>> report this at the very end.
>>
>> So essentially hudson does the equivalent of "mvn install" in a
>> recursive fashion.
>>
>> Maybe you are saying that in your environment hudson doesn't seem to
>> trigger downstream builds?
>>
>
> The main issue with "just" triggering is that It's all very well having the
> same logical build number, but unless that build number is tied to a
> *single* subversion revision number it's not useful.
All the builds with the same build number are tied to a single
subversion revision. Modules don't update their workspaces.

> What I need (and this isn't very complex) is the ability to stop hudson
> passing the '--non-recursive' parameter to mvn on building.

So I really think it already does what you are asking for. Whether Maven
does the recursion or Hudson does it for maven seems to be just a
technical detail to me.

The problem with letting Maven run recursion is that Hudson has to
somehow split up the build log for individual modules, and this kind of
interception is rather difficult to do in Maven.

You also won't be able to take advantages of multiple executors on the
same system if you have any.

> What would be even more useful would be the ability for each project item to
> be allowed to have more than one build parameter. So, for example, build
> every night with 'mvn install site site:deploy', and after every checkin
> with just 'mvn install'.

I think enough people are asking for this that I have to do something
about this.

The tricky part is to come up with a good UI.

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: m2 projects : --non-recursive

Tomasz Sterna-2
Dnia 09-03-2007, pią o godzinie 07:25 -0800, Kohsuke Kawaguchi
napisał(a):

> > What would be even more useful would be the ability for each project
> item to
> > be allowed to have more than one build parameter. So, for example,
> build
> > every night with 'mvn install site site:deploy', and after every
> checkin
> > with just 'mvn install'.
>
> I think enough people are asking for this that I have to do something
> about this.
>
> The tricky part is to come up with a good UI.

I'm thinking about something like this:


Builds
---------------------------------------------------------------------
   Triggers_________________________________________________________
    ( ) Build after other projects are built
    (*) Poll SCM
    ( ) Build periodically
    Schedule   | 0 * * * 6,7                                        |
               | */10 * * * 1-5                                     |
               |                                                    |
   Build_____________________________________________________________
    [ ] Execute shell
    [x] Execute Windows batch command
        Command |                                                   |
                |                                                   |
    [ ] Invoke top-level Ant targets
    [ ] Invoke top-level Maven targets
    [x] m2 build type
         Maven Version [ maven 1.0.2                              V ]
              Root POM [ pom.xml                                    ]
                 Goals [ package                                    ]
                                                         [ Delete ]

  [ Add new Build Trigger ]



Basically linking Build Triggers and Build sections together and making
it duplicatable with [ Add new Build Trigger ] button.

This way type of trigger could become radiobutton with no need for
duplicate schedule field (we could duplicate it by adding new trigger).

Maybe some options from Post-build Actions should be linked here also.


--
Tomasz Sterna                      E-Mail: [hidden email]
eo Networks Sp. z o.o.           JabberID: [hidden email]
ul. Jana Olbrachta 94, Warszawa   SkypeID: tomasz.sterna

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: m2 projects : --non-recursive

Kohsuke Kawaguchi-2
Tomasz Sterna wrote:

> Dnia 09-03-2007, pi�? o godzinie 07:25 -0800, Kohsuke Kawaguchi
> napisa�?(a):
>> > What would be even more useful would be the ability for each project
>> item to
>> > be allowed to have more than one build parameter. So, for example,
>> build
>> > every night with 'mvn install site site:deploy', and after every
>> checkin
>> > with just 'mvn install'.
>>
>> I think enough people are asking for this that I have to do something
>> about this.
>>
>> The tricky part is to come up with a good UI.
>
> I'm thinking about something like this:
>
>
> Builds
> ---------------------------------------------------------------------
>    Triggers_________________________________________________________
>     ( ) Build after other projects are built
>     (*) Poll SCM
>     ( ) Build periodically
>     Schedule   | 0 * * * 6,7                                        |
>                | */10 * * * 1-5                                     |
>                |                                                    |
>    Build_____________________________________________________________
>     [ ] Execute shell
>     [x] Execute Windows batch command
>         Command |                                                   |
>                 |                                                   |
>     [ ] Invoke top-level Ant targets
>     [ ] Invoke top-level Maven targets
>     [x] m2 build type
>          Maven Version [ maven 1.0.2                              V ]
>               Root POM [ pom.xml                                    ]
>                  Goals [ package                                    ]
>                                                          [ Delete ]
>
>   [ Add new Build Trigger ]
>
>
>
> Basically linking Build Triggers and Build sections together and making
> it duplicatable with [ Add new Build Trigger ] button.
>
> This way type of trigger could become radiobutton with no need for
> duplicate schedule field (we could duplicate it by adding new trigger).
Yeah, something along the line of this. I think triggers could be still
checkboxes, as it seems like allowing different triggers to share the
same build setting is useful.

I was thinking about giving each "build plan" (isn't that how it's
called by some other CI tools?) a separate config page. I think the
config page is getting bit too long to be comprehensible. Or maybe I can
still maintain it in a single page by folding plans that are not being
edited.

In either case I need to extend the UI taglibs and do some ground work
first. There's also a compatibility aspect and internal modeling to
think about.


> Maybe some options from Post-build Actions should be linked here also.


--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: m2 projects : --non-recursive

Nigel Magnay
In reply to this post by Kohsuke Kawaguchi-2

So I really think it already does what you are asking for. Whether Maven
does the recursion or Hudson does it for maven seems to be just a
technical detail to me.

The problem with letting hudson do it is the potential difference in results to doing 'mvn install' from the root level project. There's a few ways this could happen

1) If the checkin changed the actual project dependencies structure (the pom.xml files) - so the order you'll be building will instantly be wrong. And unless hudson can dynamically update the build order structures from this, there's a mode in which you can fail.
2) Just because the code hasn't changed doesn't mean the output artifact(s) won't have changed. A good example is something that doesn't change much, with a snapshot transitive dependency. If you don't rebuild it, you potentially don't pull down a new snapshot that might break something downstream.
3) The mere fact that it's different, and doesn't rebuild everything in the same way will be a deal breaker for QA; they'll simply refuse to take it unless the build is recreatable outside of hudson and they can build it in the same way it does.

I can do all this outside in a non-m2 project builder, but it seems a real shame to have m2 support that is fantastic for the 90% developer-make-sure-nothing-broken cases, but not for the 10% release-to-paranoid-QA.
 
 

The problem with letting Maven run recursion is that Hudson has to
somehow split up the build log for individual modules, and this kind of
interception is rather difficult to do in Maven.

Don't need the log file split up into the individual modules, is sufficient to just have the whole output.

You also won't be able to take advantages of multiple executors on the
same system if you have any.

Again, not important.
 

> What would be even more useful would be the ability for each project item to
> be allowed to have more than one build parameter. So, for example, build
> every night with 'mvn install site site:deploy', and after every checkin
> with just 'mvn install'.

I think enough people are asking for this that I have to do something
about this.

The tricky part is to come up with a good UI.

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]