build output summary?

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

build output summary?

rja84
Hi,

I came across Hudson just last week and am very excited about what I'm seeing.  I have several CVS hosted test projects running through it already and the reply I'm getting from my development team is extremely favorable.

One feature I'm looking for is something, or some way, of producing a summary of errors and/or warnings encountered during a build.  Right now, Hudson may return a build as "failed", but the only way I can see to find out the details of the failure is to go the the "Last build" listed under Permalinks and then go to the "Console Output".  I then have to manually scroll through that console output to find the error(s) that caused the failure.  With all the other nice things already built into Hudson, seems like something that produced a more usable output, maybe something with a header section containing the errors and warnings encountered, with links to the spot in the full output file (something like the "Brief Log/Full Log" found in Tinderbox) should already be available and maybe I'm just missing it.

Any ideas?

Thanks,
Rick
Reply | Threaded
Open this post in threaded view
|

Re: build output summary?

Jesse Glick
richard alanis wrote:
> One feature I'm looking for is something, or some way, of producing a
> summary of errors and/or warnings encountered during a build.  Right
> now, Hudson may return a build as "failed", but the only way I can
> see to find out the details of the failure is to go the the "Last
> build" listed under Permalinks and then go to the "Console Output".
> I then have to manually scroll through that console output to find
> the error(s) that caused the failure.

Sounds similar to

https://hudson.dev.java.net/issues/show_bug.cgi?id=261

-J.

--
[hidden email]  netbeans.org  ant.apache.org  hudson.dev.java.net
             http://google.com/search?q=e%5E%28pi*i%29%2B1

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: build output summary?

rja84
Thanks for the very prompt reply.  The subject of that id=261 is kind of related.  JGlick reports that their Firefox browser freezing up when trying to open either the HTML or raw text of a large console output.  There is also mention of perhaps just showing just the tail of the console output since often that will provide enough information at to why a build failed.   Only issue with showing just the tail is sometimes a project may have a need to force a build to run completely through and not terminate after the first error.

I think my group is more interested in a summary type "Brief Log" report that lists just the errors (and warnings) as hyperlinks to the section in the "Full Log" where the errors (and warnings) actually appear.  If no errors or warnings are show up during a build, the "Brief Log" could simply be empty.  I use the terms "Brief Log" and "Full Log" here simply because those are the names used in Tinderbox.



On 8/13/07, Jesse Glick <[hidden email]> wrote:
richard alanis wrote:
> One feature I'm looking for is something, or some way, of producing a
> summary of errors and/or warnings encountered during a build.  Right
> now, Hudson may return a build as "failed", but the only way I can
> see to find out the details of the failure is to go the the "Last
> build" listed under Permalinks and then go to the "Console Output".
> I then have to manually scroll through that console output to find
> the error(s) that caused the failure.

Sounds similar to

https://hudson.dev.java.net/issues/show_bug.cgi?id=261

-J.

--
[hidden email]  netbeans.org  ant.apache.org  hudson.dev.java.net
             http://google.com/search?q=e%5E%28pi*i%29%2B1

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: build output summary?

Jesse Glick
richard alanis wrote:
> [We are] more interested in a summary type "Brief Log" report that
> lists just the errors (and warnings) as hyperlinks to the section in
> the "Full Log" where the errors (and warnings) actually appear.

That could be useful, too, though you need some heuristic for deciding
what is an error. Showing warnings may not be desirable. At least in our
case, building hundreds of megs of sources, there are a bunch of
deprecation warnings and the like which do not halt the build but which
would obscure hard errors. Even if you want warnings too, you need to
decide which output lines are to be treated as errors or warnings,
rather than just some other informational messages.

-J.

--
[hidden email]  netbeans.org  ant.apache.org  hudson.dev.java.net
             http://google.com/search?q=e%5E%28pi*i%29%2B1

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: build output summary?

Marziou, Gael
> That could be useful, too, though you need some heuristic for
> deciding what is an error. Showing warnings may not be
> desirable.

I think it depends what tool you use for building, if you are using ant you can configure logging to output errors and warnings only in a separate file.
In such case you would see only errors and warning logged by the ant tasks not by the tools (e.g. javac) used by these tasks.

Gael

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: build output summary?

rja84
We're a c++ shop using 'automake'.  I have things configured in Hudson using the "Execute build" Build type.  What I execute is a simple shell script that contains what's needed to build our projects (set a few required environment variables then basically run 'autoreconf', 'configure', 'make', and 'make install').   I like seeing all the output captured, but if there are errors someplace along the line, I'd like to see a summary of those immediately at the top (or perhaps in some other output file that Hudson can immediately be directed to) rather then having to step through the entire console output.

Hope what I'm describing makes sense.  Just seems like error/warning parsing of the build output would be extremely useful and a very attractive feature in Hudson.  Prior to using Hudson, this is what my shell scripts would do.  Grep the output for certain strings that I would define as an error, ignore certain strings that look like errors, but really aren't.  Would do the same with warnings.  I could adjust this simple grep filtering on a per project basis.  Really simple stuff that was directed to a Summary file.  I'd email this summary to the interested develop team, and in the email was a link to the raw output file. 

As I've mentioned, Mozilla Tinderbox does something similar, but much, much more fancy then the simple txt files I used.  Was/am really hoping something similar is available in Hudson.  Or maybe it's a feature that can be considered for a future version... :-)

On 8/14/07, Marziou, Gael <[hidden email]> wrote:
> That could be useful, too, though you need some heuristic for
> deciding what is an error. Showing warnings may not be
> desirable.

I think it depends what tool you use for building, if you are using ant you can configure logging to output errors and warnings only in a separate file.
In such case you would see only errors and warning logged by the ant tasks not by the tools (e.g. javac) used by these tasks.

Gael

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: build output summary?

Jesse Glick
In reply to this post by Marziou, Gael
Marziou, Gael wrote:
> I think it depends what tool you use for building, if you are using
> ant you can configure logging to output errors and warnings only in a
> separate file.

Yes, though this would require a special parameter with a
Hudson-specific class name to be passed to Ant, which would not work for
people who use the "shell script" option and call the 'ant' binary
directly one or more times.

> In such case you would see only errors and warning logged by the ant
> tasks not by the tools (e.g. javac) used by these tasks.

In fact <javac> captures compiler warnings and errors from the javac
tool and reports them as Ant log messages at the appropriate severity level.

-J.

--
[hidden email]  netbeans.org  ant.apache.org  hudson.dev.java.net
             http://google.com/search?q=e%5E%28pi*i%29%2B1

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: build output summary?

Kohsuke Kawaguchi
Administrator
In reply to this post by rja84

I agree that this would be a nice feature --- I thought there's an issue
that better captures this (other than #261), but couldn't find any with
my cursory search right now.

The problem is, as Jesse mentioned, how to correctly detect problems.
Even for maven/ant, this is not really obvious (maybe there's some
conservative heuristics that's good enough?), but for shell/batch script
builds this is very hard. And if you are using automake, sounds like
that's yet another output format that we need to look for.

(I was hoping to do just this in the native m2 support, where Hudson can
exactly monitor which phase of the build is going on and capture output.)

Maybe the way to do this is for Hudson to define one more extension
point so that the plugins can contribute code that scans the console
output. Hudson can call them one by one and aggregate all the results in
a single view.

Maybe with a reasonable heuristics, this might be doable without
bombarding the user with million configuration options. For example, Ant
and Maven has a fairly distinctive header/footer that might allow a dumb
scanner to do a reasonably good job.



richard alanis wrote:

> Hi,
>
> I came across Hudson just last week and am very excited about what I'm
> seeing.  I have several CVS hosted test projects running through it already
> and the reply I'm getting from my development team is extremely favorable.
>
> One feature I'm looking for is something, or some way, of producing a
> summary of errors and/or warnings encountered during a build.  Right now,
> Hudson may return a build as "failed", but the only way I can see to find
> out the details of the failure is to go the the "Last build" listed under
> Permalinks and then go to the "Console Output".  I then have to manually
> scroll through that console output to find the error(s) that caused the
> failure.  With all the other nice things already built into Hudson, seems
> like something that produced a more usable output, maybe something with a
> header section containing the errors and warnings encountered, with links to
> the spot in the full output file (something like the "Brief Log/Full Log"
> found in Tinderbox) should already be available and maybe I'm just missing
> it.
>
> Any ideas?
>
> Thanks,
> Rick
>

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: build output summary?

Jesse Glick
Kohsuke Kawaguchi wrote:
> Maybe with a reasonable heuristics, this might be doable without
> bombarding the user with million configuration options.

And as a fallback, build failures in many projects can be diagnosed
readily just by looking at the last hundred or so lines of the log (my
suggestion in #261). This works so long as (1) the build is essentially
linear and stops at the first failure, which is true of most Ant
scripts, and make without -j, and (AFAIK) Maven; (2) log output from the
final step is not excessively long, which is likely to be true unless
you have e.g. one javac error mixed into a hundred deprecation warnings.

Of course, the root cause of a problem might not be diagnosable just
from the final error messages. But this is well beyond Hudson's ability
to help; relevant information in the log may not be formatted to look
like an error at all. For difficult cases, only a human looking at the
full log is going to work.

Summarizing compiler warnings might be a different case than displaying
build-breaking errors. Ideally some plug-in could scan the log for
warning messages logged from javac (or tools with similar output) and
create a summary of warning density for different areas of the code
base, maybe with neat charts of "hot" areas. One glitch is that
command-line javac won't report warnings unless asked to (although it
may say that there were some it did not report!), and will stop after
100 per invocation, so you don't get a very accurate picture. It is
possible to use JSR 199 to collect all (javac) warnings precisely, but
(unless you are using Maven) tricky to know what sources to parse with
what classpath to match the project's build scripts - the simple
approach of parsing every *.java in the workdir against every *.jar
would work for many projects but not all.

-J.

--
[hidden email]  netbeans.org  ant.apache.org  hudson.dev.java.net
             http://google.com/search?q=e%5E%28pi*i%29%2B1

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: build output summary?

rja84
Maybe I'm oversimplifying things, but for a c/c++ project "some conservative heuristics that's good enough?" to start with could be as simple as scanning the console output for " *** ", "Error 1", "Error 2", "Fatal", and "(S)".  Would bet that over 95% of c/c++ compiler errors will be identified by one of these markers.  Seems like starting at around 95% wouldn't be such a bad place to start from.  I'm not as familiar with the output generated by ant/maven/java based builds but would guess most of the compile errors they produce are going to be marked by a handful of markers as well.

Absolutely agree that for difficult cases only a human looking at the full log is going to be able to figure out what the real cause of the problem is.  I'm just trying give our developers an easier way of wading through what could easily be a log file that is several thousand lines long (just checked, the log file from one of our projects built last night was 4611 lines long...).  I know I'm stating the obvious here, but the easier you make something to use, the more likely it will be used.  I already have 12 seperate projects being processed through Hudson (don't think that's too bad considering I only heard about Hudson last Thursday... :-) and expect other development projects will be extremely anxious to jump on board as soon as they find out about it.  So far, the only feature I'm being asked for is a better way of error reporting.  I can always continue using my external 'find/ignore' scripts and email those after the build, but just seems like it would be soo slick to have it all contained within Hudson...

Thanks...

On 8/15/07, Jesse Glick <[hidden email]> wrote:
Kohsuke Kawaguchi wrote:
> Maybe with a reasonable heuristics, this might be doable without
> bombarding the user with million configuration options.

And as a fallback, build failures in many projects can be diagnosed
readily just by looking at the last hundred or so lines of the log (my
suggestion in #261). This works so long as (1) the build is essentially
linear and stops at the first failure, which is true of most Ant
scripts, and make without -j, and (AFAIK) Maven; (2) log output from the
final step is not excessively long, which is likely to be true unless
you have e.g. one javac error mixed into a hundred deprecation warnings.

Of course, the root cause of a problem might not be diagnosable just
from the final error messages. But this is well beyond Hudson's ability
to help; relevant information in the log may not be formatted to look
like an error at all. For difficult cases, only a human looking at the
full log is going to work.

Summarizing compiler warnings might be a different case than displaying
build-breaking errors. Ideally some plug-in could scan the log for
warning messages logged from javac (or tools with similar output) and
create a summary of warning density for different areas of the code
base, maybe with neat charts of "hot" areas. One glitch is that
command-line javac won't report warnings unless asked to (although it
may say that there were some it did not report!), and will stop after
100 per invocation, so you don't get a very accurate picture. It is
possible to use JSR 199 to collect all (javac) warnings precisely, but
(unless you are using Maven) tricky to know what sources to parse with
what classpath to match the project's build scripts - the simple
approach of parsing every *.java in the workdir against every *.jar
would work for many projects but not all.

-J.

--
[hidden email]   netbeans.org  ant.apache.org  hudson.dev.java.net
             http://google.com/search?q=e%5E%28pi*i%29%2B1

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: build output summary?

Kohsuke Kawaguchi
Administrator

If you can file an RFE for this and link to this thread in the archive,
that would be great.

richard alanis wrote:

> Maybe I'm oversimplifying things, but for a c/c++ project "some conservative
> heuristics that's good enough?" to start with could be as simple as scanning
> the console output for " *** ", "Error 1", "Error 2", "Fatal", and "(S)".
> Would bet that over 95% of c/c++ compiler errors will be identified by one
> of these markers.  Seems like starting at around 95% wouldn't be such a bad
> place to start from.  I'm not as familiar with the output generated by
> ant/maven/java based builds but would guess most of the compile errors they
> produce are going to be marked by a handful of markers as well.
>
> Absolutely agree that for difficult cases only a human looking at the full
> log is going to be able to figure out what the real cause of the problem
> is.  I'm just trying give our developers an easier way of wading through
> what could easily be a log file that is several thousand lines long (just
> checked, the log file from one of our projects built last night was 4611
> lines long...).  I know I'm stating the obvious here, but the easier you
> make something to use, the more likely it will be used.  I already have 12
> seperate projects being processed through Hudson (don't think that's too bad
> considering I only heard about Hudson last Thursday... :-) and expect other
> development projects will be extremely anxious to jump on board as soon as
> they find out about it.  So far, the only feature I'm being asked for is a
> better way of error reporting.  I can always continue using my external
> 'find/ignore' scripts and email those after the build, but just seems like
> it would be soo slick to have it all contained within Hudson...
>
> Thanks...
>
> On 8/15/07, Jesse Glick <[hidden email]> wrote:
>>
>> Kohsuke Kawaguchi wrote:
>> > Maybe with a reasonable heuristics, this might be doable without
>> > bombarding the user with million configuration options.
>>
>> And as a fallback, build failures in many projects can be diagnosed
>> readily just by looking at the last hundred or so lines of the log (my
>> suggestion in #261). This works so long as (1) the build is essentially
>> linear and stops at the first failure, which is true of most Ant
>> scripts, and make without -j, and (AFAIK) Maven; (2) log output from the
>> final step is not excessively long, which is likely to be true unless
>> you have e.g. one javac error mixed into a hundred deprecation warnings.
>>
>> Of course, the root cause of a problem might not be diagnosable just
>> from the final error messages. But this is well beyond Hudson's ability
>> to help; relevant information in the log may not be formatted to look
>> like an error at all. For difficult cases, only a human looking at the
>> full log is going to work.
>>
>> Summarizing compiler warnings might be a different case than displaying
>> build-breaking errors. Ideally some plug-in could scan the log for
>> warning messages logged from javac (or tools with similar output) and
>> create a summary of warning density for different areas of the code
>> base, maybe with neat charts of "hot" areas. One glitch is that
>> command-line javac won't report warnings unless asked to (although it
>> may say that there were some it did not report!), and will stop after
>> 100 per invocation, so you don't get a very accurate picture. It is
>> possible to use JSR 199 to collect all (javac) warnings precisely, but
>> (unless you are using Maven) tricky to know what sources to parse with
>> what classpath to match the project's build scripts - the simple
>> approach of parsing every *.java in the workdir against every *.jar
>> would work for many projects but not all.
>>
>> -J.
>>
>> --
>> [hidden email]  netbeans.org  ant.apache.org  hudson.dev.java.net
>>              http://google.com/search?q=e%5E%28pi*i%29%2B1
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: build output summary?

rja84
I'm kind of new to Hudson, java.net, and all this.  I believe I've just opened issue ID=748 about this enhancement request, and hope that's what you meant by an RFE.  Hopefully I submitted it correctly.  If not, please let me know and I'll correct.

Thanks...


On 8/19/07, Kohsuke Kawaguchi <[hidden email]> wrote:

If you can file an RFE for this and link to this thread in the archive,
that would be great.

richard alanis wrote:

> Maybe I'm oversimplifying things, but for a c/c++ project "some conservative
> heuristics that's good enough?" to start with could be as simple as scanning
> the console output for " *** ", "Error 1", "Error 2", "Fatal", and "(S)".
> Would bet that over 95% of c/c++ compiler errors will be identified by one
> of these markers.  Seems like starting at around 95% wouldn't be such a bad
> place to start from.  I'm not as familiar with the output generated by
> ant/maven/java based builds but would guess most of the compile errors they
> produce are going to be marked by a handful of markers as well.
>
> Absolutely agree that for difficult cases only a human looking at the full
> log is going to be able to figure out what the real cause of the problem
> is.  I'm just trying give our developers an easier way of wading through
> what could easily be a log file that is several thousand lines long (just
> checked, the log file from one of our projects built last night was 4611
> lines long...).  I know I'm stating the obvious here, but the easier you
> make something to use, the more likely it will be used.  I already have 12
> seperate projects being processed through Hudson (don't think that's too bad
> considering I only heard about Hudson last Thursday... :-) and expect other
> development projects will be extremely anxious to jump on board as soon as
> they find out about it.  So far, the only feature I'm being asked for is a
> better way of error reporting.  I can always continue using my external
> 'find/ignore' scripts and email those after the build, but just seems like
> it would be soo slick to have it all contained within Hudson...
>
> Thanks...
>
> On 8/15/07, Jesse Glick <[hidden email]> wrote:
>>
>> Kohsuke Kawaguchi wrote:
>> > Maybe with a reasonable heuristics, this might be doable without
>> > bombarding the user with million configuration options.
>>
>> And as a fallback, build failures in many projects can be diagnosed
>> readily just by looking at the last hundred or so lines of the log (my
>> suggestion in #261). This works so long as (1) the build is essentially
>> linear and stops at the first failure, which is true of most Ant
>> scripts, and make without -j, and (AFAIK) Maven; (2) log output from the
>> final step is not excessively long, which is likely to be true unless
>> you have e.g. one javac error mixed into a hundred deprecation warnings.
>>
>> Of course, the root cause of a problem might not be diagnosable just
>> from the final error messages. But this is well beyond Hudson's ability
>> to help; relevant information in the log may not be formatted to look
>> like an error at all. For difficult cases, only a human looking at the
>> full log is going to work.
>>
>> Summarizing compiler warnings might be a different case than displaying
>> build-breaking errors. Ideally some plug-in could scan the log for

>> warning messages logged from javac (or tools with similar output) and
>> create a summary of warning density for different areas of the code
>> base, maybe with neat charts of "hot" areas. One glitch is that
>> command-line javac won't report warnings unless asked to (although it
>> may say that there were some it did not report!), and will stop after
>> 100 per invocation, so you don't get a very accurate picture. It is
>> possible to use JSR 199 to collect all (javac) warnings precisely, but
>> (unless you are using Maven) tricky to know what sources to parse with
>> what classpath to match the project's build scripts - the simple
>> approach of parsing every *.java in the workdir against every *.jar
>> would work for many projects but not all.
>>
>> -J.
>>
>> --
>> [hidden email]  netbeans.org  ant.apache.org   hudson.dev.java.net
>>              http://google.com/search?q=e%5E%28pi*i%29%2B1
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>


--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: build output summary?

Kohsuke Kawaguchi
Administrator
richard alanis wrote:
> I'm kind of new to Hudson, java.net, and all this.  I believe I've just
> opened issue ID=748 about this enhancement request, and hope that's what you
> meant by an RFE.  Hopefully I submitted it correctly.  If not, please let me
> know and I'll correct.

Thanks. That was what I meant.

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment