Static Analysis...How about some collaboration?

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Static Analysis...How about some collaboration?

iamsteveholmes
Hey folks!
Has there been any news on the static analysis plugins?  I may have some time to work on a PMD plugin but I would definitely like to see Checkstyle, PMD CPD, FindBug, Cobertura, etc.  I have also been looking at a tool called qalab at:

http://qalab.sourceforge.net/

Anyone gotten these woking with Hudson?  Anyone else working on plugins?  I'm very open to collaboration.
-Steve
Reply | Threaded
Open this post in threaded view
|

Re: Static Analysis...How about some collaboration?

Kohsuke Kawaguchi
Administrator
Steve Holmes wrote:

> Hey folks!
> Has there been any news on the static analysis plugins?  I may have some
> time to work on a PMD plugin but I would definitely like to see Checkstyle,
> PMD CPD, FindBug, Cobertura, etc.  I have also been looking at a tool called
> qalab at:
>
> http://qalab.sourceforge.net/
>
> Anyone gotten these woking with Hudson?  Anyone else working on plugins?
> I'm very open to collaboration.
I believe [1] is relevant. I don't know if Steve Holmes is still
monitoring the list ... oh wait! that's you!

As I wrote in the previous e-mail in [1], I created the empty PMD
directory. So why don't we get started with PMD, and let's see if there
are other folks interested in joining hands.


I think Stephen said one of his friends has been or is doing some
plugins in your list. Hope he chimes in.


[1]
http://www.nabble.com/Current-state-of-unreleased-analysis-plugins-%28checkstyle%2C-pmd%2C-findbugs%2C-cobertura%2C-etc.%29-tf4057339.html#a11525772
--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Static Analysis...How about some collaboration?

Stephen Connolly-2
Kohsuke Kawaguchi wrote:
> I think Stephen said one of his friends has been or is doing some
> plugins in your list. Hope he chimes in.
Peter told me is working on getting checkstyle ready for release.  I
think he just wants to refactor one or two things to tidy up and put in
health thresholds to configure the health reports per project (he has
health reporting but it's with hard limits, something like any more than
20 checkstyle errors is thunderclouds)

Not sure if he's got a java.net id yet...

-Stephen.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Static Analysis...How about some collaboration?

Ulli Hafner-2
In reply to this post by iamsteveholmes
Hi,

I'm currently working on a findbugs plugin.
I'll upload it to the CVS soon. However, it's not yet finished.
Up to now it just scans the findbugs.xml files and reports the
number of bugs (and sets the build to UNSTABLE if a threshold
is reached). This was the item with highest priority in our
Project (where warnings are treated as errors).

I'll try to add some more things next week:
- Healthy State
- Warnings Trend graph
- Display of results

Maybe we should consolidate on these topics? This code needs
to be done only once. Findbugs, checkstyle and pmd all create
a number of warnings which are part of maven modules, packages,
and classes.

Actually, I'm not sure how to display these results. The easiest
way is to show the maven reports, however not all checkers support
multi-module maven builds up to now (and not everybody uses maven).
Any ideas?

Moreover, when showing a project trend graph, I would rather prefer
one graph with differnt warning counters for pmd, checkstyle and
findbugs than showing three graphs. Maybe it's better to have a graph
showing the healthy percentage? What do you thik?

Best regards,
Ulli

> -----Original Message-----
> From: Steve Holmes [mailto:[hidden email]]
> Sent: Friday, July 20, 2007 10:06 PM
> To: [hidden email]
> Subject: Static Analysis...How about some collaboration?
>
>
> Hey folks!
> Has there been any news on the static analysis plugins?  I
> may have some time to work on a PMD plugin but I would
> definitely like to see Checkstyle, PMD CPD, FindBug,
> Cobertura, etc.  I have also been looking at a tool called qalab at:
>
> http://qalab.sourceforge.net/
>
> Anyone gotten these woking with Hudson?  Anyone else working
> on plugins?
> I'm very open to collaboration.
> -Steve
> --
> View this message in context:
> http://www.nabble.com/Static-Analysis...How-about-some-collabo
> ration--tf4119327.html#a11715149
> Sent from the Hudson users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Static Analysis...How about some collaboration?

Arnaud LACOUR
Will your code be bound to projects using Maven ?
Our project is all wrapped in ant but we do use PMD and findBugs right
now. Being able to see all those metrics consolidated into one graph
certainly has value and I was about to start wrapping our findbugs and
PMD targets in order to convert the reports to something that would be
parseable by Nigel's plot plugin as this seems to be a workable and
quick route ...
of course a dedicated plugin wouldn't hurt it it made matters easier ;)

my 2 pennies
-=arnaud=-

On 7/22/07, Hafner Ullrich <[hidden email]> wrote:

> Hi,
>
> I'm currently working on a findbugs plugin.
> I'll upload it to the CVS soon. However, it's not yet finished.
> Up to now it just scans the findbugs.xml files and reports the
> number of bugs (and sets the build to UNSTABLE if a threshold
> is reached). This was the item with highest priority in our
> Project (where warnings are treated as errors).
>
> I'll try to add some more things next week:
> - Healthy State
> - Warnings Trend graph
> - Display of results
>
> Maybe we should consolidate on these topics? This code needs
> to be done only once. Findbugs, checkstyle and pmd all create
> a number of warnings which are part of maven modules, packages,
> and classes.
>
> Actually, I'm not sure how to display these results. The easiest
> way is to show the maven reports, however not all checkers support
> multi-module maven builds up to now (and not everybody uses maven).
> Any ideas?
>
> Moreover, when showing a project trend graph, I would rather prefer
> one graph with differnt warning counters for pmd, checkstyle and
> findbugs than showing three graphs. Maybe it's better to have a graph
> showing the healthy percentage? What do you thik?
>
> Best regards,
> Ulli
>
> > -----Original Message-----
> > From: Steve Holmes [mailto:[hidden email]]
> > Sent: Friday, July 20, 2007 10:06 PM
> > To: [hidden email]
> > Subject: Static Analysis...How about some collaboration?
> >
> >
> > Hey folks!
> > Has there been any news on the static analysis plugins?  I
> > may have some time to work on a PMD plugin but I would
> > definitely like to see Checkstyle, PMD CPD, FindBug,
> > Cobertura, etc.  I have also been looking at a tool called qalab at:
> >
> > http://qalab.sourceforge.net/
> >
> > Anyone gotten these woking with Hudson?  Anyone else working
> > on plugins?
> > I'm very open to collaboration.
> > -Steve
> > --
> > View this message in context:
> > http://www.nabble.com/Static-Analysis...How-about-some-collabo
> > ration--tf4119327.html#a11715149
> > Sent from the Hudson users mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Static Analysis...How about some collaboration?

Kohsuke Kawaguchi
Administrator
In reply to this post by Ulli Hafner-2
Hafner Ullrich wrote:
> Hi,
>
> I'm currently working on a findbugs plugin.
> I'll upload it to the CVS soon. However, it's not yet finished.

Cool! You don't need to wait for the plugin to be finished before
commiting them into CVS ;-)  there's a bunch of half-baked plugins in
the repository.


> Moreover, when showing a project trend graph, I would rather prefer
> one graph with differnt warning counters for pmd, checkstyle and
> findbugs than showing three graphs. Maybe it's better to have a graph
> showing the healthy percentage? What do you thik?

This is an interesting idea. But how you you scale Y-axis?


--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: Static Analysis...How about some collaboration?

Ulli Hafner-2
In reply to this post by Arnaud LACOUR
> Will your code be bound to projects using Maven ?

Well, actually the code is bound to the findbugs.xml files, which should
be the same for ant or maven projects.

Ulli

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Static Analysis...How about some collaboration?

Ulli Hafner-2
In reply to this post by Kohsuke Kawaguchi
> Hafner Ullrich wrote:
> > Hi,
> >
> > I'm currently working on a findbugs plugin.
> > I'll upload it to the CVS soon. However, it's not yet finished.
>
> Cool! You don't need to wait for the plugin to be finished
> before commiting them into CVS ;-)  there's a bunch of
> half-baked plugins in the repository.

I'll commit the sources tonight (CVS is blocked here from work).

>
>
> > Moreover, when showing a project trend graph, I would rather prefer
> > one graph with differnt warning counters for pmd, checkstyle and
> > findbugs than showing three graphs. Maybe it's better to
> have a graph
> > showing the healthy percentage? What do you thik?
>
> This is an interesting idea. But how you you scale Y-axis?

Well, for zero-warning projects the number of warnings should be quite
small (<50;-)

However, as suggested by Stephen Connolly I'll integrate the hudson
healthy concept for the analysis tool. Maybe we could use this health
percentage in the graph. E.g., a user definable threshold (number of
warnings) for each analysis tool that defines the 0% and 100% limit and
linear interpolation between.

Best regards, Ulli

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Static Analysis...How about some collaboration?

Stephen Connolly-2
In reply to this post by Ulli Hafner-2
Hafner Ullrich wrote
Moreover, when showing a project trend graph, I would rather prefer
one graph with differnt warning counters for pmd, checkstyle and
findbugs than showing three graphs. Maybe it's better to have a graph
showing the healthy percentage? What do you thik?
I'm liking the idea of having a health graph.

I have some concerns which I will have to play around with:

1. With the thresholds, I am torn between using the most recent threshold, or using the threshold at the time the build took place.  If we use the current threshold, then the % graph has meaning, even if it goes off the scale.  If we use the threshold at the time the build took place, then we know how the project was fairing in relation to it's targets.

My concern is that when you start using static analysis, or measuring code coverage, or in fact using any project health metric, the metric is likely to be poor.  If it is a big project, there might be a lot of effort in order to get out of the lightening clouds.  When developers don't see feedback, they loose motivation, and then the weather reports are not as useful.

That is the reason why I favour project customisable metrics.  The next issue is when management come and look at the historical graph and they see that (relative to the current targets) everything was crap until now and we'd been telling them how things were on target... so if the graph is based on the current targets, we have to explain that the targets changed as we met them... on the other hand, if it's based on the historical targets, we have to explain why it keeps on going to pot (when we reset the targets to push another drive)

Maybe this is something that could be customised.

2. At present, we only ask for the health from the current build.  If we are doing a graph, that could be asking a lot of questions of every build.  May need to store the health reports in each build's serialized xml so that we don't kill hudson's performance generating these graphs all the time (this is more of an issue as health reports are calculated by more than one plugin.  With, e.g. the JUnit graph, or clover graph.  Only that plugin is responsible for the graph, and where it is causing problems you can either disable the plugin, or we can optimize that plugin once we know where the bottlekneck is.  With health reports, we are at the mercy of each plugin's calculations... if they are slow, the graph would kill hudson's performance)

3. If we store the cached health reports... then we are kind of stuck with using the historical thresholds... unless we store the thresholds in the health report... so we have a new HelathReport constructor that takes (0% threshold, 100% threshold, currentValue, iconUrl, tooltipText) and it does the interpolation for you.

4. I'm currently toying with the idea of taking junit tests as a special case for health reports.  0 failures -> 100% health. Any failures -> at most 80%.  And scaling between 80% and 0% based on a configurable threshold or two... which might be absolute number of failing tests, or % of failing tests.  The reason being that it can be somewhat incongruent to see the yellow ball (for a failing test) and clear skies (for 99% health with only one failing test).  If I go with this JUnit health calculation, it might look strange on a graph...

The more I think about this, the more I think it might be better to have a separate way to contribute to a static analysis metric graph.  Co-opting the health report may not be the best option.  (Although, any contributor to a static analysis metric graph should be able to provide a health report.  We could add a method to HealthReportingAction, e.g. Collection<StaticAnalysisMetric> getStaticAnalysisMetrics() that by default returns an empty set and build the graph from the aggregated metrics for all actions in the build. OK, so this is almost co-opting health reports, but not quite!)

-Stephen.

Reply | Threaded
Open this post in threaded view
|

Re: Static Analysis...How about some collaboration?

iamsteveholmes
In reply to this post by iamsteveholmes
What about using something like qalab or Glean to help with reports?  I like the idea of metrics and thunderclouds and all that but a consolidated report would be very nice.  

http://jbrugge.com/glean/index.html

http://qalab.sourceforge.net/

-Steve

Steve Holmes wrote
Hey folks!
Has there been any news on the static analysis plugins?  I may have some time to work on a PMD plugin but I would definitely like to see Checkstyle, PMD CPD, FindBug, Cobertura, etc.  I have also been looking at a tool called qalab at:

http://qalab.sourceforge.net/

Anyone gotten these woking with Hudson?  Anyone else working on plugins?  I'm very open to collaboration.
-Steve
Reply | Threaded
Open this post in threaded view
|

Re: Static Analysis...How about some collaboration?

iamsteveholmes
A little off topic but how do I get at the CVS repository?  I'm not having any luck using intellij.  Maybe my address is wrong:

:pserver:guest@cvs.dev.java.net:/cvs

-Steve
Steve Holmes wrote
What about using something like qalab or Glean to help with reports?  I like the idea of metrics and thunderclouds and all that but a consolidated report would be very nice.  

http://jbrugge.com/glean/index.html

http://qalab.sourceforge.net/

-Steve

Steve Holmes wrote
Hey folks!
Has there been any news on the static analysis plugins?  I may have some time to work on a PMD plugin but I would definitely like to see Checkstyle, PMD CPD, FindBug, Cobertura, etc.  I have also been looking at a tool called qalab at:

http://qalab.sourceforge.net/

Anyone gotten these woking with Hudson?  Anyone else working on plugins?  I'm very open to collaboration.
-Steve
Reply | Threaded
Open this post in threaded view
|

Re: Static Analysis...How about some collaboration?

Kohsuke Kawaguchi
Administrator
Steve Holmes wrote:
> A little off topic but how do I get at the CVS repository?  I'm not having
> any luck using intellij.  Maybe my address is wrong:
>
> :pserver:[hidden email]:/cvs

That is correct.

   cvs -d:pserver:[hidden email]:/cvs co hudson/hudson

would check out the whole thing.


--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Static Analysis...How about some collaboration?

Kohsuke Kawaguchi
Administrator
In reply to this post by Stephen Connolly-2
Stephen Connolly wrote:

>
> Hafner Ullrich wrote:
>>
>> Moreover, when showing a project trend graph, I would rather prefer
>> one graph with differnt warning counters for pmd, checkstyle and
>> findbugs than showing three graphs. Maybe it's better to have a graph
>> showing the healthy percentage? What do you thik?
>>
> I'm liking the idea of having a health graph.
>
> I have some concerns which I will have to play around with:
>
> 1. With the thresholds, I am torn between using the most recent threshold,
> or using the threshold at the time the build took place.  If we use the
> current threshold, then the % graph has meaning, even if it goes off the
> scale.  If we use the threshold at the time the build took place, then we
> know how the project was fairing in relation to it's targets.
>
> My concern is that when you start using static analysis, or measuring code
> coverage, or in fact using any project health metric, the metric is likely
> to be poor.  If it is a big project, there might be a lot of effort in order
> to get out of the lightening clouds.  When developers don't see feedback,
> they loose motivation, and then the weather reports are not as useful.
>
> That is the reason why I favour project customisable metrics.  The next
> issue is when management come and look at the historical graph and they see
> that (relative to the current targets) everything was crap until now and
> we'd been telling them how things were on target... so if the graph is based
> on the current targets, we have to explain that the targets changed as we
> met them... on the other hand, if it's based on the historical targets, we
> have to explain why it keeps on going to pot (when we reset the targets to
> push another drive)
>
> Maybe this is something that could be customised.
You probably already know, but I'm really keen on reducing the number of
customizations. Being easy to configure is a very important value
proposition for Hudson.

> 2. At present, we only ask for the health from the current build.  If we are
> doing a graph, that could be asking a lot of questions of every build.  May
> need to store the health reports in each build's serialized xml so that we
> don't kill hudson's performance generating these graphs all the time (this
> is more of an issue as health reports are calculated by more than one
> plugin.  With, e.g. the JUnit graph, or clover graph.  Only that plugin is
> responsible for the graph, and where it is causing problems you can either
> disable the plugin, or we can optimize that plugin once we know where the
> bottlekneck is.  With health reports, we are at the mercy of each plugin's
> calculations... if they are slow, the graph would kill hudson's performance)
What I do in other parts of Hudson is to cache data set used to generate
a graph. Note that all build records get loaded into memory at the
start-up, so in the past, that has been sufficient, and I think it's the
same with health report trend.

> 3. If we store the cached health reports... then we are kind of stuck with
> using the historical thresholds... unless we store the thresholds in the
> health report... so we have a new HelathReport constructor that takes (0%
> threshold, 100% threshold, currentValue, iconUrl, tooltipText) and it does
> the interpolation for you.

The other problem of storing health trend data elsewhere is that you are
now duplicating data in two places, and thus it gets out of sync.

> 4. I'm currently toying with the idea of taking junit tests as a special
> case for health reports.  0 failures -> 100% health. Any failures -> at most
> 80%.  And scaling between 80% and 0% based on a configurable threshold or
> two... which might be absolute number of failing tests, or % of failing
> tests.  The reason being that it can be somewhat incongruent to see the
> yellow ball (for a failing test) and clear skies (for 99% health with only
> one failing test).  If I go with this JUnit health calculation, it might
> look strange on a graph...
>
> The more I think about this, the more I think it might be better to have a
> separate way to contribute to a static analysis metric graph.  Co-opting the
> health report may not be the best option.  (Although, any contributor to a
> static analysis metric graph should be able to provide a health report.  We
> could add a method to HealthReportingAction, e.g.
> Collection<StaticAnalysisMetric> getStaticAnalysisMetrics() that by default
> returns an empty set and build the graph from the aggregated metrics for all
> actions in the build. OK, so this is almost co-opting health reports, but
> not quite!)
>
> -Stephen.
>
>

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: Static Analysis...How about some collaboration?

Ulli Hafner-2
> > 2. At present, we only ask for the health from the current
> build.  If
> > we are doing a graph, that could be asking a lot of
> questions of every
> > build.  May need to store the health reports in each build's
> > serialized xml so that we don't kill hudson's performance
> generating
> > these graphs all the time (this is more of an issue as
> health reports
> > are calculated by more than one plugin.  With, e.g. the
> JUnit graph,
> > or clover graph.  Only that plugin is responsible for the
> graph, and
> > where it is causing problems you can either disable the
> plugin, or we
> > can optimize that plugin once we know where the bottlekneck
> is.  With
> > health reports, we are at the mercy of each plugin's
> calculations...
> > if they are slow, the graph would kill hudson's performance)
>
> What I do in other parts of Hudson is to cache data set used
> to generate a graph. Note that all build records get loaded
> into memory at the start-up, so in the past, that has been
> sufficient, and I think it's the same with health report trend.
> > 3. If we store the cached health reports... then we are
> kind of stuck
> > with using the historical thresholds... unless we store the
> thresholds
> > in the health report... so we have a new HelathReport
> constructor that
> > takes (0% threshold, 100% threshold, currentValue, iconUrl,
> > tooltipText) and it does the interpolation for you.
>
> The other problem of storing health trend data elsewhere is
> that you are now duplicating data in two places, and thus it
> gets out of sync.

Hmm, I'm not sure whether I got the point. Currently I'm storing for
each
build the number of warnings in the build.xml file. When using a healthy
percentage,
I would store this percentage value based on the current thresholds.
This value
should not change in the future if you decide to adapt the thresholds.
I.e., it should be quite easy to display a graph (even based on the plot
plugin).

Ulli

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Static Analysis...How about some collaboration?

Kohsuke Kawaguchi
Administrator
2007/7/24, Hafner Ullrich <[hidden email]>:
> Hmm, I'm not sure whether I got the point. Currently I'm storing for
> each
> build the number of warnings in the build.xml file.

That is the right thing to do. build.xml for each build gets loaded
into memory, so the values you need to access quickly, like the # of
warnings, should be stored as a part of build.xml (which really just
means define them as fields in your Action class.)

> When using a healthy
> percentage,
> I would store this percentage value based on the current thresholds.
> This value
> should not change in the future if you decide to adapt the thresholds.
> I.e., it should be quite easy to display a graph (even based on the plot
> plugin).

That is also a right thing to do (aside from the question of whether
you want values to change as the threshold changes or not.)

What I meant we don't want to do is to create a single data file for
storing all the data points necessary to plot a health report grpah.

--
Kohsuke Kawaguchi

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Static Analysis...How about some collaboration?

Stephen Connolly-2
In reply to this post by iamsteveholmes
Steve Holmes wrote
Hey folks!
Has there been any news on the static analysis plugins?  I may have some time to work on a PMD plugin but I would definitely like to see Checkstyle, PMD CPD, FindBug, Cobertura, etc.  I have also been looking at a tool called qalab at:

http://qalab.sourceforge.net/

Anyone gotten these woking with Hudson?  Anyone else working on plugins?  I'm very open to collaboration.
-Steve
FYI,

Hope Peter doesn't mind me saying that:

Peter's plugin is looking very good, having just seen a demo!

It currently does Checkstyle, Findbugs and PMD.

-Stephen