Test naming in Hudson 1.219

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Test naming in Hudson 1.219

Joshua McKinnon
Hi,

Last  Monday I upgraded from Hudson 1.200 to 1.219, and how Hudson
reads test names from JUnit xml reports seems to have changed. We have
set of JUnit tests that generate tests from xml files using a single
TestCase class and add them to testsuites. Basically, our test suite +
test name is always unique, but testcase+test name is not.

Say we have two JUnit test report files (grossly abbreviated for simplicity):
<testsuite errors="0" failures="0" name="com.some.TestA" tests="85"
time="14.482">
    ...
   <testcase classname="com.some.TestCase" name="test-part1"
time="0.4"></testcase>
   <testcase classname="com.some.TestCase" name="test-part2"
time="0.3"></testcase>
   ...
</testsuite>

<testsuite errors="0" failures="0" name="com.some.TestB" tests="85"
time="14.482">
    ...
   <testcase classname="com.some.TestCase" name="test-part1"
time="0.4"></testcase>
   <testcase classname="com.some.TestCase" name="test-part2"
time="0.3"></testcase>
   ...
</testsuite>

Briefly put, previously Hudson would have 4 tests as follows:
TestA.test-part1
TestA.test-part2
TestB.test-part1
TestB.test-part2
Now Hudson has:
TestCase.test-part1
TestCase.test-part2
TestCase.test-part1
TestCase.test-part2

We also suspect that, although Hudson has the same # of tests as
before, that the test report link for a given test name is
last-test-wins. I.e. if TestCase.test-part1 failed, but a separate
TestCase.test-part1 in another test suite later succeeded...the test
report link for both is a success message.

The original behavior is highly desirable with the way our
infrastructure is currently setup...should I file an issue, or is this
intentional and not likely to be reversed?


Joshua

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Test naming in Hudson 1.219

David Vrzalik-2
Yes, it has changed and it 's going to hurt us as well :(. See the last
comment on the issue 1233 for more informations...

On Monday 09 June 2008 16:07:37 Joshua McKinnon wrote:

> Hi,
>
> Last  Monday I upgraded from Hudson 1.200 to 1.219, and how Hudson
> reads test names from JUnit xml reports seems to have changed. We have
> set of JUnit tests that generate tests from xml files using a single
> TestCase class and add them to testsuites. Basically, our test suite +
> test name is always unique, but testcase+test name is not.
>
> Say we have two JUnit test report files (grossly abbreviated for
> simplicity): <testsuite errors="0" failures="0" name="com.some.TestA"
> tests="85" time="14.482">
>     ...
>    <testcase classname="com.some.TestCase" name="test-part1"
> time="0.4"></testcase>
>    <testcase classname="com.some.TestCase" name="test-part2"
> time="0.3"></testcase>
>    ...
> </testsuite>
>
> <testsuite errors="0" failures="0" name="com.some.TestB" tests="85"
> time="14.482">
>     ...
>    <testcase classname="com.some.TestCase" name="test-part1"
> time="0.4"></testcase>
>    <testcase classname="com.some.TestCase" name="test-part2"
> time="0.3"></testcase>
>    ...
> </testsuite>
>
> Briefly put, previously Hudson would have 4 tests as follows:
> TestA.test-part1
> TestA.test-part2
> TestB.test-part1
> TestB.test-part2
> Now Hudson has:
> TestCase.test-part1
> TestCase.test-part2
> TestCase.test-part1
> TestCase.test-part2
>
> We also suspect that, although Hudson has the same # of tests as
> before, that the test report link for a given test name is
> last-test-wins. I.e. if TestCase.test-part1 failed, but a separate
> TestCase.test-part1 in another test suite later succeeded...the test
> report link for both is a success message.
>
> The original behavior is highly desirable with the way our
> infrastructure is currently setup...should I file an issue, or is this
> intentional and not likely to be reversed?
>
>
> Joshua
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]

--
David Vrzalik

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Test naming in Hudson 1.219

Kohsuke Kawaguchi
Administrator
David Vrzalik wrote:
> Yes, it has changed and it 's going to hurt us as well :(. See the last
> comment on the issue 1233 for more informations...

OK, I should have known this better. Can someone file an issue for this?

>
> On Monday 09 June 2008 16:07:37 Joshua McKinnon wrote:
>> Hi,
>>
>> Last  Monday I upgraded from Hudson 1.200 to 1.219, and how Hudson
>> reads test names from JUnit xml reports seems to have changed. We have
>> set of JUnit tests that generate tests from xml files using a single
>> TestCase class and add them to testsuites. Basically, our test suite +
>> test name is always unique, but testcase+test name is not.
>>
>> Say we have two JUnit test report files (grossly abbreviated for
>> simplicity): <testsuite errors="0" failures="0" name="com.some.TestA"
>> tests="85" time="14.482">
>>     ...
>>    <testcase classname="com.some.TestCase" name="test-part1"
>> time="0.4"></testcase>
>>    <testcase classname="com.some.TestCase" name="test-part2"
>> time="0.3"></testcase>
>>    ...
>> </testsuite>
>>
>> <testsuite errors="0" failures="0" name="com.some.TestB" tests="85"
>> time="14.482">
>>     ...
>>    <testcase classname="com.some.TestCase" name="test-part1"
>> time="0.4"></testcase>
>>    <testcase classname="com.some.TestCase" name="test-part2"
>> time="0.3"></testcase>
>>    ...
>> </testsuite>
>>
>> Briefly put, previously Hudson would have 4 tests as follows:
>> TestA.test-part1
>> TestA.test-part2
>> TestB.test-part1
>> TestB.test-part2
>> Now Hudson has:
>> TestCase.test-part1
>> TestCase.test-part2
>> TestCase.test-part1
>> TestCase.test-part2
>>
>> We also suspect that, although Hudson has the same # of tests as
>> before, that the test report link for a given test name is
>> last-test-wins. I.e. if TestCase.test-part1 failed, but a separate
>> TestCase.test-part1 in another test suite later succeeded...the test
>> report link for both is a success message.
>>
>> The original behavior is highly desirable with the way our
>> infrastructure is currently setup...should I file an issue, or is this
>> intentional and not likely to be reversed?
>>
>>
>> Joshua
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Test naming in Hudson 1.219

David Vrzalik-2
On Tuesday 10 June 2008 02:54:54 Kohsuke Kawaguchi wrote:
> David Vrzalik wrote:
> > Yes, it has changed and it 's going to hurt us as well :(. See the last
> > comment on the issue 1233 for more informations...
>
> OK, I should have known this better. Can someone file an issue for this?

Issue #1832.
We'll have to solve this problem somehow before next hudson upgrade :-/. I
would be happy if I don't have to maintain another custom patch...


--
David Vrzalik

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Test naming in Hudson 1.219

Kohsuke Kawaguchi
Administrator
David Vrzalik wrote:

> On Tuesday 10 June 2008 02:54:54 Kohsuke Kawaguchi wrote:
>> David Vrzalik wrote:
>> > Yes, it has changed and it 's going to hurt us as well :(. See the last
>> > comment on the issue 1233 for more informations...
>>
>> OK, I should have known this better. Can someone file an issue for this?
>
> Issue #1832.
> We'll have to solve this problem somehow before next hudson upgrade :-/. I
> would be happy if I don't have to maintain another custom patch...
JUnit result parsing is really a mine field. Seemingly every change
causes some regressions somewhere.

You can help me by adding more test result XML files and expected results.

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Test naming in Hudson 1.219

David Vrzalik-2
On Tuesday 10 June 2008 18:44:46 Kohsuke Kawaguchi wrote:

> David Vrzalik wrote:
> > On Tuesday 10 June 2008 02:54:54 Kohsuke Kawaguchi wrote:
> >> David Vrzalik wrote:
> >> > Yes, it has changed and it 's going to hurt us as well :(. See the
> >> > last comment on the issue 1233 for more informations...
> >>
> >> OK, I should have known this better. Can someone file an issue for this?
> >
> > Issue #1832.
> > We'll have to solve this problem somehow before next hudson upgrade :-/.
> > I would be happy if I don't have to maintain another custom patch...
>
> JUnit result parsing is really a mine field. Seemingly every change
> causes some regressions somewhere.
>
> You can help me by adding more test result XML files and expected results.

Thanks. I attached shortened versions of two original junit xml fles and
expected junitResult.xml - as it looked before the change.  
(The second junitResult is valid, I didn't figure out how to delete the first
one from the issue)

--
David Vrzalik

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Test naming in Hudson 1.219

David Vrzalik-2
Kohsuke, do you think you'll be able to look at this issue? OK, I know you are
able :), I just want to ask if this is in your pipeline for next few weeks. We
can't upgrade before this is solved :(. You know, 1.202 is nice, but we don't
want to stick with it forever ;).

On Wednesday 11 June 2008 11:44:41 David Vrzalik wrote:

> On Tuesday 10 June 2008 18:44:46 Kohsuke Kawaguchi wrote:
> > David Vrzalik wrote:
> > > On Tuesday 10 June 2008 02:54:54 Kohsuke Kawaguchi wrote:
> > >> David Vrzalik wrote:
> > >> > Yes, it has changed and it 's going to hurt us as well :(. See the
> > >> > last comment on the issue 1233 for more informations...
> > >>
> > >> OK, I should have known this better. Can someone file an issue for
> > >> this?
> > >
> > > Issue #1832.
> > > We'll have to solve this problem somehow before next hudson upgrade
> > > :-/. I would be happy if I don't have to maintain another custom
> > > patch...
> >
> > JUnit result parsing is really a mine field. Seemingly every change
> > causes some regressions somewhere.
> >
> > You can help me by adding more test result XML files and expected
> > results.
>
> Thanks. I attached shortened versions of two original junit xml fles and
> expected junitResult.xml - as it looked before the change.
> (The second junitResult is valid, I didn't figure out how to delete the
> first one from the issue)

--
David Vrzalik

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]