Why am I not seeing test results using "aggregate test results"?

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Why am I not seeing test results using "aggregate test results"?

epaga4711
Hi,
I have a build job which has a test job downstream of it. I set the build job to "aggregate test results", run it, the tests run (and I can see the test results in the test job), but I don't see any test results in the build job.

Note that I've tried:
 * manually specifying the test job by name
 * running the tests more than once so I get data to show
 * clicking on "publish test results" in the build job (just don't know what to enter in the pattern if anything)

would very much appreciate help!

Thanks,
epaga4711
Reply | Threaded
Open this post in threaded view
|

Re: Why am I not seeing test results using "aggregate test results"?

stephenconnolly
because aggregate is only for the current job?

On Fri, May 16, 2008 at 1:27 PM, epaga4711 <[hidden email]> wrote:

Hi,
I have a build job which has a test job downstream of it. I set the build
job to "aggregate test results", run it, the tests run (and I can see the
test results in the test job), but I don't see any test results in the build
job.

Note that I've tried:
 * manually specifying the test job by name
 * running the tests more than once so I get data to show
 * clicking on "publish test results" in the build job (just don't know what
to enter in the pattern if anything)

would very much appreciate help!

Thanks,
epaga4711
--
View this message in context: http://www.nabble.com/Why-am-I-not-seeing-test-results-using-%22aggregate-test-results%22--tp17273519p17273519.html
Sent from the Hudson users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Why am I not seeing test results using "aggregate test results"?

epaga
Really? The help text for "Aggregate downstream test results" is:

Because tests often dominates the execution time, a Hudson best practice involves splitting test executions into different jobs, possibly in multiple different jobs.

When you do that, setting test aggregation is a convenient way of collecting all the test results from such downstream test jobs and display it along with the build that they are testing. In this way, people can see the overall test status of the given build quickly.

----
So the way I understand this is that the build job which is aggregating downstream test results should show the test results of its downstream test jobs.

Am I misunderstanding something here? Thanks...
epaga4711

Stephen Connolly-2 wrote
because aggregate is only for the current job?

On Fri, May 16, 2008 at 1:27 PM, epaga4711 <johnstephi@googlemail.com>
wrote:

>
> Hi,
> I have a build job which has a test job downstream of it. I set the build
> job to "aggregate test results", run it, the tests run (and I can see the
> test results in the test job), but I don't see any test results in the
> build
> job.
>
> Note that I've tried:
>  * manually specifying the test job by name
>  * running the tests more than once so I get data to show
>  * clicking on "publish test results" in the build job (just don't know
> what
> to enter in the pattern if anything)
>
> would very much appreciate help!
>
> Thanks,
> epaga4711
> --
> View this message in context:
> http://www.nabble.com/Why-am-I-not-seeing-test-results-using-%22aggregate-test-results%22--tp17273519p17273519.html
> Sent from the Hudson users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@hudson.dev.java.net
> For additional commands, e-mail: users-help@hudson.dev.java.net
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Why am I not seeing test results using "aggregate test results"?

stephenconnolly
Hmmm I had not seen that one!

Perhaps you are correct then!

On Fri, May 16, 2008 at 2:31 PM, epaga <[hidden email]> wrote:

Really? The help text for "Aggregate downstream test results" is:

Because tests often dominates the execution time, a Hudson best practice
involves splitting test executions into different jobs, possibly in multiple
different jobs.

When you do that, setting test aggregation is a convenient way of collecting
all the test results from such downstream test jobs and display it along
with the build that they are testing. In this way, people can see the
overall test status of the given build quickly.

----
So the way I understand this is that the build job which is aggregating
downstream test results should show the test results of its downstream test
jobs.

Am I misunderstanding something here? Thanks...
epaga4711


Stephen Connolly-2 wrote:
>
> because aggregate is only for the current job?
>
> On Fri, May 16, 2008 at 1:27 PM, epaga4711 <[hidden email]>
> wrote:
>
>>
>> Hi,
>> I have a build job which has a test job downstream of it. I set the build
>> job to "aggregate test results", run it, the tests run (and I can see the
>> test results in the test job), but I don't see any test results in the
>> build
>> job.
>>
>> Note that I've tried:
>>  * manually specifying the test job by name
>>  * running the tests more than once so I get data to show
>>  * clicking on "publish test results" in the build job (just don't know
>> what
>> to enter in the pattern if anything)
>>
>> would very much appreciate help!
>>
>> Thanks,
>> epaga4711
>> --
>> View this message in context:
>> http://www.nabble.com/Why-am-I-not-seeing-test-results-using-%22aggregate-test-results%22--tp17273519p17273519.html
>> Sent from the Hudson users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>
>

--
View this message in context: http://www.nabble.com/Why-am-I-not-seeing-test-results-using-%22aggregate-test-results%22--tp17273519p17274759.html
Sent from the Hudson users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Why am I not seeing test results using "aggregate test results"?

Kohsuke Kawaguchi
Administrator
In reply to this post by epaga4711
epaga4711 wrote:

> Hi,
> I have a build job which has a test job downstream of it. I set the build
> job to "aggregate test results", run it, the tests run (and I can see the
> test results in the test job), but I don't see any test results in the build
> job.
>
> Note that I've tried:
>  * manually specifying the test job by name
>  * running the tests more than once so I get data to show
>  * clicking on "publish test results" in the build job (just don't know what
> to enter in the pattern if anything)
Is the fingerprint set up between two jobs, so that Hudson can track
which test run is for which build run?

If that's the case, I guess the issue is that Hudson needs to detect
this user error.

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Why am I not seeing test results using "aggregate test results"?

epaga
OK thanks that helped! I used to copy the built jar file into a central directory but now I'm using the artefact feature plus ant's <get> task and it's working better. Yeah, Hudson noticing this user error (and also documenting it in the help for test result aggregation) would be really nice.

BUT... (sorry) ... I still have a question. What do I enter in the "Publish JUnit test result report"'s "Test report XMLs" field in the build job? I left it empty and it seemed to work for a couple of builds, but then I started seeing the build error "Is this really a JUnit report file? Your configuration must be matching too many files".

Any thoughts?

Thanks,
epaga

kohsuke wrote
epaga4711 wrote:
> Hi,
> I have a build job which has a test job downstream of it. I set the build
> job to "aggregate test results", run it, the tests run (and I can see the
> test results in the test job), but I don't see any test results in the build
> job.
>
> Note that I've tried:
>  * manually specifying the test job by name
>  * running the tests more than once so I get data to show
>  * clicking on "publish test results" in the build job (just don't know what
> to enter in the pattern if anything)

Is the fingerprint set up between two jobs, so that Hudson can track
which test run is for which build run?

If that's the case, I guess the issue is that Hudson needs to detect
this user error.

--
Kohsuke Kawaguchi
Sun Microsystems                   kohsuke.kawaguchi@sun.com

 
Reply | Threaded
Open this post in threaded view
|

Re: Why am I not seeing test results using "aggregate test results"?

Erik Ramfelt
On Mon, May 19, 2008 at 10:42 AM, epaga <[hidden email]> wrote:

>
> OK thanks that helped! I used to copy the built jar file into a central
> directory but now I'm using the artefact feature plus ant's <get> task and
> it's working better. Yeah, Hudson noticing this user error (and also
> documenting it in the help for test result aggregation) would be really
> nice.
>
> BUT... (sorry) ... I still have a question. What do I enter in the "Publish
> JUnit test result report"'s "Test report XMLs" field in the build job? I
> left it empty and it seemed to work for a couple of builds, but then I
> started seeing the build error "Is this really a JUnit report file? Your
> configuration must be matching too many files".

The field should contain a reference to your JUnit XML files, if you
dont have any then you should not enable the JUnit reports.

For example if your junit XML files are located in
"build/test-reports/A.xml", then you could enter
"build/test-reports/**/*.xml" That way Hudson will look into the
specified folder and add the results from the XML files.

>
> Any thoughts?
>
> Thanks,
> epaga
>
>
> kohsuke wrote:
>>
>> epaga4711 wrote:
>>> Hi,
>>> I have a build job which has a test job downstream of it. I set the build
>>> job to "aggregate test results", run it, the tests run (and I can see the
>>> test results in the test job), but I don't see any test results in the
>>> build
>>> job.
>>>
>>> Note that I've tried:
>>>  * manually specifying the test job by name
>>>  * running the tests more than once so I get data to show
>>>  * clicking on "publish test results" in the build job (just don't know
>>> what
>>> to enter in the pattern if anything)
>>
>> Is the fingerprint set up between two jobs, so that Hudson can track
>> which test run is for which build run?
>>
>> If that's the case, I guess the issue is that Hudson needs to detect
>> this user error.
>>
>> --
>> Kohsuke Kawaguchi
>> Sun Microsystems                   [hidden email]
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Why-am-I-not-seeing-test-results-using-%22aggregate-test-results%22--tp17273519p17313665.html
> Sent from the Hudson users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Why am I not seeing test results using "aggregate test results"?

epaga
Thanks for the response, Erik, unfortunately that's not my problem.

I know the idea of providing a pattern for finding the XML files, the problem is that i'm in the build job, not the test job. so if i enter a pattern, it fails because it doesn't find anything because the test results are in the test job (and haven't been created yet since the test job has yet to be started downstream). if i do enter a path, it says "Is this really a JUnit report file? Your configuration must be matching too many files."

Again: I'm trying to aggregate the test results from downstream test jobs and display them in the build job.

Thanks,
epaga

Erik Ramfelt wrote
On Mon, May 19, 2008 at 10:42 AM, epaga <jgoering@inetsoftware.de> wrote:
>
The field should contain a reference to your JUnit XML files, if you
dont have any then you should not enable the JUnit reports.

For example if your junit XML files are located in
"build/test-reports/A.xml", then you could enter
"build/test-reports/**/*.xml" That way Hudson will look into the
specified folder and add the results from the XML files.
Reply | Threaded
Open this post in threaded view
|

Re: Why am I not seeing test results using "aggregate test results"?

epaga
I meant if I DON'T enter a path i get that error message. sorry.
epaga

epaga wrote
...if i do enter a path, it says "Is this really a JUnit report file? Your configuration must be matching too many files."
Reply | Threaded
Open this post in threaded view
|

Re: Why am I not seeing test results using "aggregate test results"?

epaga
OK I think I found a workaround for the problem: I simply make sure my build job runs a single pseudo-test so it at least has the test results from that test, I then specify a pattern to that single test.

This seems to me to be a bug to have to do this workaround (unless I'm overlooking some setting somewhere):

if your build job does not run any tests itself but you want it to aggregate and display test results from downstream test jobs, then specifying no pattern for test files causes EVERY file to be tried to be read, and specifying a pattern causes NO test file to be found. Turning off "export JUnit test results" causes the results to not be displayed.

Is my assessment correct, should I go ahead with my workaround or am I overlooking something?

Thanks,
epaga
Reply | Threaded
Open this post in threaded view
|

Re: Why am I not seeing test results using "aggregate test results"?

Erik Ramfelt
On Tue, May 20, 2008 at 9:26 AM, epaga <[hidden email]> wrote:
>
> OK I think I found a workaround for the problem: I simply make sure my build
> job runs a single pseudo-test so it at least has the test results from that
> test, I then specify a pattern to that single test.
>
> This seems to me to be a bug to have to do this workaround (unless I'm
> overlooking some setting somewhere):

Yes, in my pov it would be an issue (but I havent used that feature
before). Could you log the issue on the issue tracker
(https://hudson.dev.java.net/issues) with a step by step guide on how
to reproduce the issue?

//Erik

> if your build job does not run any tests itself but you want it to aggregate
> and display test results from downstream test jobs, then specifying no
> pattern for test files causes EVERY file to be tried to be read, and
> specifying a pattern causes NO test file to be found. Turning off "export
> JUnit test results" causes the results to not be displayed.
>
> Is my assessment correct, should I go ahead with my workaround or am I
> overlooking something?
>
> Thanks,
> epaga
> --
> View this message in context: http://www.nabble.com/Why-am-I-not-seeing-test-results-using-%22aggregate-test-results%22--tp17273519p17334784.html
> Sent from the Hudson users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Why am I not seeing test results using "aggregate test results"?

epaga
Yeah, unfortunately my workaround didn't work either (if i specify test results of the pseudo test, it only shows the result from that one test, and doesn't seem to aggregate the test results.

OK I'll file an issue...

To reproduce it should be pretty simple...
1. Create a build job which creates an artifact and fingerprints it.
2. Create a test job downstream from the build job which uses the fingerprinted artifact from the build job.
3. Set the build job to aggregate test results.
4. Run build job and test job multiple times.
5. Now try to set the build job to display its aggregated test results without breaking the build job - this doesn't seem to be possible.
   a. Specifying no pattern for the test results causes an IO error because it tries reading ALL files in the build job as test results.
   b. Specifying a pattern for test results which is not correct causes an error because it can't find anything.
   c. creating a pseudo test which is additionally run by the build job causes ONLY this pseudo test result to be shown in the test results of the build job.
Reply | Threaded
Open this post in threaded view
|

Re: Why am I not seeing test results using "aggregate test results"?

Joshua McKinnon
Wouldn't it make more sense to specify no test pattern in the main job
(i.e. uncheck publish junit report entirely) since it has no test
reports itself, and instead specify publish junit report with the
appropriate pattern in each downstream test job (or is that already
being done)? The fact that 'aggregate downstream test reports' is
checked should make the main build smart enough to pick up whatever
test reports the downstream build found - it shouldn't be the main
job's responsibility to specify where its downstream test reports are,
since there could be multiple downstream jobs with test results and
they could all have different patterns, and it should be legal for a
job w/ aggregate test results to not publish its own junit reports and
just include that of its downstream jobs.

This is merely my own speculation, I haven't actually been able to try
this due to Issue1574, but that's how I thought it would work. Then
again I am using freestyle jobs and not sharing artifacts and maybe
this is primarily for maven2 jobs. I'm curious if that's been tried,
though.



On Wed, May 21, 2008 at 5:33 AM, epaga <[hidden email]> wrote:

>
> Yeah, unfortunately my workaround didn't work either (if i specify test
> results of the pseudo test, it only shows the result from that one test, and
> doesn't seem to aggregate the test results.
>
> OK I'll file an issue...
>
> To reproduce it should be pretty simple...
> 1. Create a build job which creates an artifact and fingerprints it.
> 2. Create a test job downstream from the build job which uses the
> fingerprinted artifact from the build job.
> 3. Set the build job to aggregate test results.
> 4. Run build job and test job multiple times.
> 5. Now try to set the build job to display its aggregated test results
> without breaking the build job - this doesn't seem to be possible.
>   a. Specifying no pattern for the test results causes an IO error because
> it tries reading ALL files in the build job as test results.
>   b. Specifying a pattern for test results which is not correct causes an
> error because it can't find anything.
>   c. creating a pseudo test which is additionally run by the build job
> causes ONLY this pseudo test result to be shown in the test results of the
> build job.
> --
> View this message in context: http://www.nabble.com/Why-am-I-not-seeing-test-results-using-%22aggregate-test-results%22--tp17273519p17359471.html
> Sent from the Hudson users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]