More JUnit test report problems...

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

More JUnit test report problems...

Narve Sætre
Hi all,

I see several people are having problems with JUnit test reports. I've
read the postings, and we have a similar problem.

Before (vr 1.123) the problem was "Test reports were found but none of
them are new". Now (with 1.128) I don't get that message anymore, but
hudson doesn't find any tests in my report. The log says:

BUILD SUCCESSFUL

Total time: 2 minutes 42 seconds
Publishing Javadoc
Recording test results
finished: SUCCESS

There is a link to the test results, but it says "Latest Test Result
(no tests)". The test result is a valid xml file and using
<junitreport todir="testreports" > etc the html files are generated
perfectly. I currently use this target inside my ant-file run by hudson,
so I can view the test results in the workspace browser. This works but
is not optimal.

What can prevent hudson from successfully understanding my xml file?
Thanks for any help. I can post (parts of) the xml file if anybody wants
it, but it really is a standard junit xml file...


--
Narve Sætre
Partner / Machina Networks as / +47 41915331

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

narve.vcf (269 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: More JUnit test report problems...

Kohsuke Kawaguchi
Administrator

I'm looking at the code, but in all the code path, I seem to be handling
errors correctly. I run a few tests but I couldn't reproduce the issue.

If you can look at server's file system and find junitResult.xml from
jobs/YOURJOB/builds/TIMESTAMP/junitResult.xml and send it to me, that
would be great.

Also, does this involve any slaves?


Narve Sætre wrote:

> Hi all,
>
> I see several people are having problems with JUnit test reports. I've
> read the postings, and we have a similar problem.
>
> Before (vr 1.123) the problem was "Test reports were found but none of
> them are new". Now (with 1.128) I don't get that message anymore, but
> hudson doesn't find any tests in my report. The log says:
>
> BUILD SUCCESSFUL
>
> Total time: 2 minutes 42 seconds
> Publishing Javadoc
> Recording test results
> finished: SUCCESS
>
> There is a link to the test results, but it says "Latest Test Result
> (no tests)". The test result is a valid xml file and using
> <junitreport todir="testreports" > etc the html files are generated
> perfectly. I currently use this target inside my ant-file run by hudson,
> so I can view the test results in the workspace browser. This works but
> is not optimal.
>
> What can prevent hudson from successfully understanding my xml file?
> Thanks for any help. I can post (parts of) the xml file if anybody wants
> it, but it really is a standard junit xml file...
>
>
>
>
> ------------------------------------------------------------------------
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]


smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: More JUnit test report problems...

Narve Sætre
Thanks for helping out, I will provide more info:


The junitResult.xml file is empty:

<?xml version='1.0' encoding='UTF-8'?>
<result>
   <suites>
     <suite>
       <cases/>
     </suite>
   </suites>
</result>

System setup:

Hudson: hudson vr 1.128
Container: Glassfish
Java: Java HotSpot(TM) 64-Bit Server VM (build 1.6.0_01-b06, mixed mode)
OS: Linux build 2.6.20-15-generic #2 SMP Sun Apr 15 06:17:24 UTC 2007
x86_64 GNU/Linux. (Ubuntu)
Slaves: None


The junit xml file is 170k, here is a representative part:

<?xml version="1.0" encoding="UTF-8" ?>
<testsuites>
   <testsuite errors="0" failures="0" hostname="build" id="0"
name="GestaltTestSu
ite" package="no.machina.gestalt2" tests="132" time="92.655"
timestamp="2007-08-
14T12:23:31">
       <properties>
           ... lots of them
       </properties>

       <testcase
classname="no.machina.gestalt2.workflow.AutoCorrectForkJoin" nam
e="testCardinalityCounter" time="1.808" />

       <testcase classname="no.machina.gestalt2.workflow.ArchiveTest"
name="testSaveAndArchive" time="1.087" />
     ... lots of testcases

       <system-out><![CDATA[ ... lots of output, ditto for system-err
       </system-err>

   </testsuite>
</testsuites>


The file is server/testreports/TESTS-TestSuites.xml

in the config I can specify server/testreports/TESTS-TestSuites.xml or
server/testreports/*.xml, same result.

(If I use asdf.xml the web interface gives me a warning and the build
fails with "Recording test results. No test report files were found.
Configuration error?" so obviously the file is found. )


--
Narve Sætre
Partner / Machina Networks as / +47 41915331




Kohsuke Kawaguchi wrote:

>
> I'm looking at the code, but in all the code path, I seem to be handling
> errors correctly. I run a few tests but I couldn't reproduce the issue.
>
> If you can look at server's file system and find junitResult.xml from
> jobs/YOURJOB/builds/TIMESTAMP/junitResult.xml and send it to me, that
> would be great.
>
> Also, does this involve any slaves?
>
>
> Narve Sætre wrote:
>> Hi all,
>>
>> I see several people are having problems with JUnit test reports. I've
>> read the postings, and we have a similar problem.
>>
>> Before (vr 1.123) the problem was "Test reports were found but none of
>> them are new". Now (with 1.128) I don't get that message anymore, but
>> hudson doesn't find any tests in my report. The log says:
>>
>> BUILD SUCCESSFUL
>>
>> Total time: 2 minutes 42 seconds
>> Publishing Javadoc
>> Recording test results
>> finished: SUCCESS
>>
>> There is a link to the test results, but it says "Latest Test Result
>> (no tests)". The test result is a valid xml file and using
>> <junitreport todir="testreports" > etc the html files are generated
>> perfectly. I currently use this target inside my ant-file run by
>> hudson, so I can view the test results in the workspace browser. This
>> works but is not optimal.
>>
>> What can prevent hudson from successfully understanding my xml file?
>> Thanks for any help. I can post (parts of) the xml file if anybody
>> wants it, but it really is a standard junit xml file...
>>
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

narve.vcf (269 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: More JUnit test report problems...

Narve Sætre
In reply to this post by Kohsuke Kawaguchi

I just set up findbugs, cpd and pmd in the same project. All of them run
fine, generating xml reports and html reports nicely. However, in the
graph generated by hudson they all show -1 violations.

This problem seems to be too similar to the junit problem to happen by
chance. Could there be an xml parsing misconfiguration? Encoding error?
Namespace problems?

-n


Kohsuke Kawaguchi wrote:

>
> I'm looking at the code, but in all the code path, I seem to be handling
> errors correctly. I run a few tests but I couldn't reproduce the issue.
>
> If you can look at server's file system and find junitResult.xml from
> jobs/YOURJOB/builds/TIMESTAMP/junitResult.xml and send it to me, that
> would be great.
>
> Also, does this involve any slaves?
>
>
> Narve Sætre wrote:
>> Hi all,
>>
>> I see several people are having problems with JUnit test reports. I've
>> read the postings, and we have a similar problem.
>>
>> Before (vr 1.123) the problem was "Test reports were found but none of
>> them are new". Now (with 1.128) I don't get that message anymore, but
>> hudson doesn't find any tests in my report. The log says:
>>
>> BUILD SUCCESSFUL
>>
>> Total time: 2 minutes 42 seconds
>> Publishing Javadoc
>> Recording test results
>> finished: SUCCESS
>>
>> There is a link to the test results, but it says "Latest Test Result
>> (no tests)". The test result is a valid xml file and using
>> <junitreport todir="testreports" > etc the html files are generated
>> perfectly. I currently use this target inside my ant-file run by
>> hudson, so I can view the test results in the workspace browser. This
>> works but is not optimal.
>>
>> What can prevent hudson from successfully understanding my xml file?
>> Thanks for any help. I can post (parts of) the xml file if anybody
>> wants it, but it really is a standard junit xml file...
>>
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

narve.vcf (269 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: More JUnit test report problems...

Peter Reilly-2
The -1 just means that the xml filename pattern did not
match any files for that violation type.

(I know this is not great gui feedback - but it is a
little better that breaking the build)

Check the name of the xml report files. (for example
the checkstyle report may be named "raw.xml" from
the default ant task or "checkstyle.xml" from maven)
and enter for example **/raw.xml in the xml filename pattern
field for the checkstyle configuration line on the violations
configuration panel.

It is real important to enter the two stars - **,
otherwise the pattern will not match sub directories.

Peter


On 8/16/07, Narve Sætre <[hidden email]> wrote:

>
> I just set up findbugs, cpd and pmd in the same project. All of them run
> fine, generating xml reports and html reports nicely. However, in the
> graph generated by hudson they all show -1 violations.
>
> This problem seems to be too similar to the junit problem to happen by
> chance. Could there be an xml parsing misconfiguration? Encoding error?
> Namespace problems?
>
> -n
>
>
> Kohsuke Kawaguchi wrote:
> >
> > I'm looking at the code, but in all the code path, I seem to be handling
> > errors correctly. I run a few tests but I couldn't reproduce the issue.
> >
> > If you can look at server's file system and find junitResult.xml from
> > jobs/YOURJOB/builds/TIMESTAMP/junitResult.xml and send it to me, that
> > would be great.
> >
> > Also, does this involve any slaves?
> >
> >
> > Narve Sætre wrote:
> >> Hi all,
> >>
> >> I see several people are having problems with JUnit test reports. I've
> >> read the postings, and we have a similar problem.
> >>
> >> Before (vr 1.123) the problem was "Test reports were found but none of
> >> them are new". Now (with 1.128) I don't get that message anymore, but
> >> hudson doesn't find any tests in my report. The log says:
> >>
> >> BUILD SUCCESSFUL
> >>
> >> Total time: 2 minutes 42 seconds
> >> Publishing Javadoc
> >> Recording test results
> >> finished: SUCCESS
> >>
> >> There is a link to the test results, but it says "Latest Test Result
> >> (no tests)". The test result is a valid xml file and using
> >> <junitreport todir="testreports" > etc the html files are generated
> >> perfectly. I currently use this target inside my ant-file run by
> >> hudson, so I can view the test results in the workspace browser. This
> >> works but is not optimal.
> >>
> >> What can prevent hudson from successfully understanding my xml file?
> >> Thanks for any help. I can post (parts of) the xml file if anybody
> >> wants it, but it really is a standard junit xml file...
> >>
> >>
> >>
> >>
> >> ------------------------------------------------------------------------
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: More JUnit test report problems...

Narve Sætre

Yes, you were right, more or less. The file references were wrong, now
findbugs and pmd works. Cpd still doesn't work, but that probably
something else.

Remind me to triplecheck my configuration next time - doublechecking is
obviously not enough :)

But the unit tests though... :(

--
Narve Sætre
Partner / Machina Networks as / +47 41915331

Peter Reilly wrote:

> The -1 just means that the xml filename pattern did not
> match any files for that violation type.
>
> (I know this is not great gui feedback - but it is a
> little better that breaking the build)
>
> Check the name of the xml report files. (for example
> the checkstyle report may be named "raw.xml" from
> the default ant task or "checkstyle.xml" from maven)
> and enter for example **/raw.xml in the xml filename pattern
> field for the checkstyle configuration line on the violations
> configuration panel.
>
> It is real important to enter the two stars - **,
> otherwise the pattern will not match sub directories.
>
> Peter
>
>
> On 8/16/07, Narve Sætre <[hidden email]> wrote:
>> I just set up findbugs, cpd and pmd in the same project. All of them run
>> fine, generating xml reports and html reports nicely. However, in the
>> graph generated by hudson they all show -1 violations.
>>
>> This problem seems to be too similar to the junit problem to happen by
>> chance. Could there be an xml parsing misconfiguration? Encoding error?
>> Namespace problems?
>>
>> -n
>>
>>
>> Kohsuke Kawaguchi wrote:
>>> I'm looking at the code, but in all the code path, I seem to be handling
>>> errors correctly. I run a few tests but I couldn't reproduce the issue.
>>>
>>> If you can look at server's file system and find junitResult.xml from
>>> jobs/YOURJOB/builds/TIMESTAMP/junitResult.xml and send it to me, that
>>> would be great.
>>>
>>> Also, does this involve any slaves?
>>>
>>>
>>> Narve Sætre wrote:
>>>> Hi all,
>>>>
>>>> I see several people are having problems with JUnit test reports. I've
>>>> read the postings, and we have a similar problem.
>>>>
>>>> Before (vr 1.123) the problem was "Test reports were found but none of
>>>> them are new". Now (with 1.128) I don't get that message anymore, but
>>>> hudson doesn't find any tests in my report. The log says:
>>>>
>>>> BUILD SUCCESSFUL
>>>>
>>>> Total time: 2 minutes 42 seconds
>>>> Publishing Javadoc
>>>> Recording test results
>>>> finished: SUCCESS
>>>>
>>>> There is a link to the test results, but it says "Latest Test Result
>>>> (no tests)". The test result is a valid xml file and using
>>>> <junitreport todir="testreports" > etc the html files are generated
>>>> perfectly. I currently use this target inside my ant-file run by
>>>> hudson, so I can view the test results in the workspace browser. This
>>>> works but is not optimal.
>>>>
>>>> What can prevent hudson from successfully understanding my xml file?
>>>> Thanks for any help. I can post (parts of) the xml file if anybody
>>>> wants it, but it really is a standard junit xml file...
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

narve.vcf (269 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: More JUnit test report problems...

Kohsuke Kawaguchi
Administrator
In reply to this post by Narve Sætre
Narve Sætre wrote:

> The junit xml file is 170k, here is a representative part:
>
> <?xml version="1.0" encoding="UTF-8" ?>
> <testsuites>
>    <testsuite errors="0" failures="0" hostname="build" id="0"
> name="GestaltTestSu
> ite" package="no.machina.gestalt2" tests="132" time="92.655"
> timestamp="2007-08-
> 14T12:23:31">
>        <properties>
>            ... lots of them
>        </properties>
>
>        <testcase
> classname="no.machina.gestalt2.workflow.AutoCorrectForkJoin" nam
> e="testCardinalityCounter" time="1.808" />
>
>        <testcase classname="no.machina.gestalt2.workflow.ArchiveTest"
> name="testSaveAndArchive" time="1.087" />
>      ... lots of testcases
>
>        <system-out><![CDATA[ ... lots of output, ditto for system-err
>        </system-err>
>
>    </testsuite>
> </testsuites>
OK, this is interesting. The format is different from the ones that I
know of, which puts <testsuite> at the top level. What program did you
use to create this? Maybe something in Ant has changed? Or is this a
mode that I was just unaware of?

Anyway, now that I know what to look for, I'll fix this in 1.131.

> The file is server/testreports/TESTS-TestSuites.xml
>
> in the config I can specify server/testreports/TESTS-TestSuites.xml or
> server/testreports/*.xml, same result.
>
> (If I use asdf.xml the web interface gives me a warning and the build
> fails with "Recording test results. No test report files were found.
> Configuration error?" so obviously the file is found. )
>
>
>
>
> ------------------------------------------------------------------------
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]


smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: More JUnit test report problems...

Kohsuke Kawaguchi
Administrator
In reply to this post by Peter Reilly-2
Peter Reilly wrote:
> The -1 just means that the xml filename pattern did not
> match any files for that violation type.
>
> (I know this is not great gui feedback - but it is a
> little better that breaking the build)

I actually think it's better to mark the build as a failure if there's a
configuration error. You can also do a form-field validation to
forestall problems like this.

You can also print out the error message to logger during build if you
didn't find any match.

I think it's good for all of us if we look out for mistakes. It's good
for us because we won't have to respond to "why doesn't XYZ work?"
e-mails, and it's good for users because they'll be more productive.

A little effort to implement an error check pays off, IMHO.

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: More JUnit test report problems...

Erik Ramfelt
In reply to this post by Kohsuke Kawaguchi
On 8/21/07, Kohsuke Kawaguchi <[hidden email]> wrote:
> OK, this is interesting. The format is different from the ones that I
> know of, which puts <testsuite> at the top level. What program did you
> use to create this? Maybe something in Ant has changed? Or is this a
> mode that I was just unaware of?

This leads me to a question that I found when trying to convert nunit
test reports to junit reports. The NUnit XML report often contains
more than one test suite in each file, and as I can see from most
(all?) JUnit XML report that they only contains one test suite. Will
your change also support two test suites in one file?

Regards
//Erik

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: More JUnit test report problems...

Peter Reilly-2
In reply to this post by Kohsuke Kawaguchi
On 8/21/07, Kohsuke Kawaguchi <[hidden email]> wrote:

> Peter Reilly wrote:
> > The -1 just means that the xml filename pattern did not
> > match any files for that violation type.
> >
> > (I know this is not great gui feedback - but it is a
> > little better that breaking the build)
>
> I actually think it's better to mark the build as a failure if there's a
> configuration error. You can also do a form-field validation to
> forestall problems like this.
>
> You can also print out the error message to logger during build if you
> didn't find any match.
>
> I think it's good for all of us if we look out for mistakes. It's good
> for us because we won't have to respond to "why doesn't XYZ work?"
> e-mails, and it's good for users because they'll be more productive.
>
> A little effort to implement an error check pays off, IMHO.
On my list.
However, I did not want to break the build just because
someone mistyped a file name.

I have placed an error message in the result class for the
type - I am planning to have this error message in the
health report (at 0% a pretty stormy report).

Also, I will do the same validation as junit does on
the configuration screen.

Peter

>
> --
> Kohsuke Kawaguchi
> Sun Microsystems                   [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: More JUnit test report problems...

Narve Sætre
In reply to this post by Kohsuke Kawaguchi
>
> OK, this is interesting. The format is different from the ones that I
> know of, which puts <testsuite> at the top level. What program did you
> use to create this? Maybe something in Ant has changed? Or is this a
> mode that I was just unaware of?
>
> Anyway, now that I know what to look for, I'll fix this in 1.131.
>

Yes, it was an agregated view of all test suites run - we did some
post-processing on the resulting xml files. The resulting xml file was
similar to the one created by the junitreport ant task, so I assumed it
was similar to / compatible with the individual test suite xml files as
well.

I am now able to circumvent the problem by specifying the individual
test suites separately, and it works. Nice graphs at last :)

That means that the fix is no longer necessary (for us), but it is
probably a good idea to include it anyway.

Thanks for helping out! Hudson is a great product!

--
Narve Sætre
Partner / Machina Networks as / +47 41915331

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

narve.vcf (269 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: More JUnit test report problems...

Kohsuke Kawaguchi
Administrator
In reply to this post by Erik Ramfelt
2007/8/20, Erik Ramfelt <[hidden email]>:
> This leads me to a question that I found when trying to convert nunit
> test reports to junit reports. The NUnit XML report often contains
> more than one test suite in each file, and as I can see from most
> (all?) JUnit XML report that they only contains one test suite. Will
> your change also support two test suites in one file?

Yes. You can look at the change. I've already committed it.

Or maybe what you want to look at is the sample junit report file that
Narve sent, which would show you what XML you can create.

--
Kohsuke Kawaguchi

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: More JUnit test report problems...

Kohsuke Kawaguchi
Administrator
In reply to this post by Peter Reilly-2
2007/8/21, Peter Reilly <[hidden email]>:
> > A little effort to implement an error check pays off, IMHO.
> On my list.
> However, I did not want to break the build just because
> someone mistyped a file name.

OK, I respect your choice but that is different from how Hudson works
everywhere else. If you make a configuration error, it always fails a
build.

> I have placed an error message in the result class for the
> type - I am planning to have this error message in the
> health report (at 0% a pretty stormy report).
>
> Also, I will do the same validation as junit does on
> the configuration screen.

OK, sounds good.

--
Kohsuke Kawaguchi

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]