Benchmarking Jenkins using Java Microbenchmark Harness (JMH)

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Benchmarking Jenkins using Java Microbenchmark Harness (JMH)

Abhyudaya Sharma
Hi everyone,

As a part of this year's Google Summer of Code, I have been working on a
framework for allowing Java Microbenchmark Harness (JMH) benchmarks to be
run with Jenkins. The basic framework was implemented in the Role Strategy
Plugin through this pull request:
https://github.com/jenkinsci/role-strategy-plugin/pull/63 . For each
benchmark, a temporary Jenkins instance is created, similar to how
JenkinsRule from Jenkins Test Harness does it. The benchmarks are run
through a unit test which allows them to be integrated with the normal pull
request builds (if required). Apart from configuring the Jenkins instance
for test using Java code, I've also added support for easily configuring the
instance by YAML files using Jenkins Configuration as Code. The benchmark
reports are generated as JSON files which are compatible with the jmh-report
plugin. I have implemented a couple of benchmarks for the Role Strategy
Plugin which can be seen in the Role Strategy Plugin GitHub repository. I
have attached screenshots of these results as visualized by the jmh-report
plugin. To make the framework easier to use, I'm also working on running the
benchmarks through a Maven profile.

To make it available to the rest of the community, I would like to propose
to make it either a part of Jenkins Test Harness or a part of new library so
it can be used by other plugins and perhaps by Jenkins core itself. I would
to love to hear your feedback, comments and suggestions. It would be even
better if you could join the project's weekly meetings (Tuesdays and Fridays
at 7:00 AM UTC) and help us improve this framework.

Thanks
Abhyudaya
GitHub: AbhyudayaSharma


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/006301d5151d%246159d610%24240d8230%24%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Annotation 2019-05-28 114729.png (341K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Benchmarking Jenkins using Java Microbenchmark Harness (JMH)

Oleg Nenashev
Hi all,

Just to facilitate this thread, I am +1 regarding having micro-benchmarking support in Jenkins Test Harness. There are many stories related to performance degradation in Jenkins, and having a framework could help us to have some checks for the critical functionality like permission checks. The end goal of the project is to improve Role Strategy performance (which is far from perfect now), but it would be great if we could reuse the framework in other components.

Tomorrow at 7am UTC we will be doing a recorded demo of the current framework state, and everybody is welcome to join.

Best regards,
Oleg

On Tuesday, May 28, 2019 at 8:20:14 AM UTC+2, [hidden email] wrote:
Hi everyone,

As a part of this year's Google Summer of Code, I have been working on a
framework for allowing Java Microbenchmark Harness (JMH) benchmarks to be
run with Jenkins. The basic framework was implemented in the Role Strategy
Plugin through this pull request:
<a href="https://github.com/jenkinsci/role-strategy-plugin/pull/63" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Frole-strategy-plugin%2Fpull%2F63\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGQppXbuVpAsX-p5wEABV_5ac-npA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Frole-strategy-plugin%2Fpull%2F63\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGQppXbuVpAsX-p5wEABV_5ac-npA&#39;;return true;">https://github.com/jenkinsci/role-strategy-plugin/pull/63 . For each
benchmark, a temporary Jenkins instance is created, similar to how
JenkinsRule from Jenkins Test Harness does it. The benchmarks are run
through a unit test which allows them to be integrated with the normal pull
request builds (if required). Apart from configuring the Jenkins instance
for test using Java code, I've also added support for easily configuring the
instance by YAML files using Jenkins Configuration as Code. The benchmark
reports are generated as JSON files which are compatible with the jmh-report
plugin. I have implemented a couple of benchmarks for the Role Strategy
Plugin which can be seen in the Role Strategy Plugin GitHub repository. I
have attached screenshots of these results as visualized by the jmh-report
plugin. To make the framework easier to use, I'm also working on running the
benchmarks through a Maven profile.

To make it available to the rest of the community, I would like to propose
to make it either a part of Jenkins Test Harness or a part of new library so
it can be used by other plugins and perhaps by Jenkins core itself. I would
to love to hear your feedback, comments and suggestions. It would be even
better if you could join the project's weekly meetings (Tuesdays and Fridays
at 7:00 AM UTC) and help us improve this framework.

Thanks
Abhyudaya
GitHub: AbhyudayaSharma


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/09147b86-6565-4b83-a4f8-3fbf48baa6cf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Benchmarking Jenkins using Java Microbenchmark Harness (JMH)

Matt Sicker
I'm +1 in using JMH and exploring benchmarks of various functionality.
Seems like a good idea, especially with all the reinvented database
functionality implicit in Jenkins and plugins.

On Thu, May 30, 2019 at 12:57 AM Oleg Nenashev <[hidden email]> wrote:

>
> Hi all,
>
> Just to facilitate this thread, I am +1 regarding having micro-benchmarking support in Jenkins Test Harness. There are many stories related to performance degradation in Jenkins, and having a framework could help us to have some checks for the critical functionality like permission checks. The end goal of the project is to improve Role Strategy performance (which is far from perfect now), but it would be great if we could reuse the framework in other components.
>
> Tomorrow at 7am UTC we will be doing a recorded demo of the current framework state, and everybody is welcome to join.
>
> Best regards,
> Oleg
>
> On Tuesday, May 28, 2019 at 8:20:14 AM UTC+2, [hidden email] wrote:
>>
>> Hi everyone,
>>
>> As a part of this year's Google Summer of Code, I have been working on a
>> framework for allowing Java Microbenchmark Harness (JMH) benchmarks to be
>> run with Jenkins. The basic framework was implemented in the Role Strategy
>> Plugin through this pull request:
>> https://github.com/jenkinsci/role-strategy-plugin/pull/63 . For each
>> benchmark, a temporary Jenkins instance is created, similar to how
>> JenkinsRule from Jenkins Test Harness does it. The benchmarks are run
>> through a unit test which allows them to be integrated with the normal pull
>> request builds (if required). Apart from configuring the Jenkins instance
>> for test using Java code, I've also added support for easily configuring the
>> instance by YAML files using Jenkins Configuration as Code. The benchmark
>> reports are generated as JSON files which are compatible with the jmh-report
>> plugin. I have implemented a couple of benchmarks for the Role Strategy
>> Plugin which can be seen in the Role Strategy Plugin GitHub repository. I
>> have attached screenshots of these results as visualized by the jmh-report
>> plugin. To make the framework easier to use, I'm also working on running the
>> benchmarks through a Maven profile.
>>
>> To make it available to the rest of the community, I would like to propose
>> to make it either a part of Jenkins Test Harness or a part of new library so
>> it can be used by other plugins and perhaps by Jenkins core itself. I would
>> to love to hear your feedback, comments and suggestions. It would be even
>> better if you could join the project's weekly meetings (Tuesdays and Fridays
>> at 7:00 AM UTC) and help us improve this framework.
>>
>> Thanks
>> Abhyudaya
>> GitHub: AbhyudayaSharma
>>
>>
> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/09147b86-6565-4b83-a4f8-3fbf48baa6cf%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Matt Sicker
Senior Software Engineer, CloudBees

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAEot4oyK3AVFN92Mnir8dwsVXCWhWPfqeXgKWsDHGk6g4jBdOg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Benchmarking Jenkins using Java Microbenchmark Harness (JMH)

Oleg Nenashev
Hi all,

You can find the video recording of the demo here: https://www.youtube.com/watch?v=sr28UADG1AE
There is also an ongoing pull request to make it a part of Jenkins Test Harness: https://github.com/jenkinsci/jenkins-test-harness/pull/135

Best regards,
Oleg

On Thursday, May 30, 2019 at 6:44:38 PM UTC+2, Matt Sicker wrote:
I'm +1 in using JMH and exploring benchmarks of various functionality.
Seems like a good idea, especially with all the reinvented database
functionality implicit in Jenkins and plugins.

On Thu, May 30, 2019 at 12:57 AM Oleg Nenashev <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="IZpUk3APCAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">o.v.n...@...> wrote:

>
> Hi all,
>
> Just to facilitate this thread, I am +1 regarding having micro-benchmarking support in Jenkins Test Harness. There are many stories related to performance degradation in Jenkins, and having a framework could help us to have some checks for the critical functionality like permission checks. The end goal of the project is to improve Role Strategy performance (which is far from perfect now), but it would be great if we could reuse the framework in other components.
>
> Tomorrow at 7am UTC we will be doing a recorded demo of the current framework state, and everybody is welcome to join.
>
> Best regards,
> Oleg
>
> On Tuesday, May 28, 2019 at 8:20:14 AM UTC+2, [hidden email] wrote:
>>
>> Hi everyone,
>>
>> As a part of this year's Google Summer of Code, I have been working on a
>> framework for allowing Java Microbenchmark Harness (JMH) benchmarks to be
>> run with Jenkins. The basic framework was implemented in the Role Strategy
>> Plugin through this pull request:
>> <a href="https://github.com/jenkinsci/role-strategy-plugin/pull/63" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Frole-strategy-plugin%2Fpull%2F63\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGQppXbuVpAsX-p5wEABV_5ac-npA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Frole-strategy-plugin%2Fpull%2F63\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGQppXbuVpAsX-p5wEABV_5ac-npA&#39;;return true;">https://github.com/jenkinsci/role-strategy-plugin/pull/63 . For each
>> benchmark, a temporary Jenkins instance is created, similar to how
>> JenkinsRule from Jenkins Test Harness does it. The benchmarks are run
>> through a unit test which allows them to be integrated with the normal pull
>> request builds (if required). Apart from configuring the Jenkins instance
>> for test using Java code, I've also added support for easily configuring the
>> instance by YAML files using Jenkins Configuration as Code. The benchmark
>> reports are generated as JSON files which are compatible with the jmh-report
>> plugin. I have implemented a couple of benchmarks for the Role Strategy
>> Plugin which can be seen in the Role Strategy Plugin GitHub repository. I
>> have attached screenshots of these results as visualized by the jmh-report
>> plugin. To make the framework easier to use, I'm also working on running the
>> benchmarks through a Maven profile.
>>
>> To make it available to the rest of the community, I would like to propose
>> to make it either a part of Jenkins Test Harness or a part of new library so
>> it can be used by other plugins and perhaps by Jenkins core itself. I would
>> to love to hear your feedback, comments and suggestions. It would be even
>> better if you could join the project's weekly meetings (Tuesdays and Fridays
>> at 7:00 AM UTC) and help us improve this framework.
>>
>> Thanks
>> Abhyudaya
>> GitHub: AbhyudayaSharma
>>
>>
> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="IZpUk3APCAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jenkin...@googlegroups.com.
> To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-dev/09147b86-6565-4b83-a4f8-3fbf48baa6cf%40googlegroups.com" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-dev/09147b86-6565-4b83-a4f8-3fbf48baa6cf%40googlegroups.com&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-dev/09147b86-6565-4b83-a4f8-3fbf48baa6cf%40googlegroups.com&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-dev/09147b86-6565-4b83-a4f8-3fbf48baa6cf%40googlegroups.com.
> For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.



--
Matt Sicker
Senior Software Engineer, CloudBees

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/676323d7-7d32-43be-9003-d0d8eab9e02e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

RE: Benchmarking Jenkins using Java Microbenchmark Harness (JMH)

Abhyudaya Sharma

Hi everyone,

 

Just to update you all, the framework for running JMH benchmarks is almost ready to be merged into Jenkins Test Harness. Please see this pull request. There is also a pull request to the plugin-pom for running benchmarks through a Maven profile. For ease of use in plugins hosted on ci.jenkins.io, there is also a work-in-progress pull request to add support for running them as a pipeline step to Jenkins’ pipeline library.

 

I would love to hear your feedback and comments.

 

Thanks

Abhyudaya Sharma

GitHub: AbhyudayaSharma

 

From: [hidden email] <[hidden email]> On Behalf Of Oleg Nenashev
Sent: Tuesday, June 4, 2019 1:20 PM
To: Jenkins Developers <[hidden email]>
Subject: Re: Benchmarking Jenkins using Java Microbenchmark Harness (JMH)

 

Hi all,

 

You can find the video recording of the demo here: https://www.youtube.com/watch?v=sr28UADG1AE

There is also an ongoing pull request to make it a part of Jenkins Test Harness: https://github.com/jenkinsci/jenkins-test-harness/pull/135

 

Best regards,

Oleg

On Thursday, May 30, 2019 at 6:44:38 PM UTC+2, Matt Sicker wrote:

I'm +1 in using JMH and exploring benchmarks of various functionality.
Seems like a good idea, especially with all the reinvented database
functionality implicit in Jenkins and plugins.

On Thu, May 30, 2019 at 12:57 AM Oleg Nenashev <<a href="javascript:" target="_blank">o.v.n...@...> wrote:


>
> Hi all,
>
> Just to facilitate this thread, I am +1 regarding having micro-benchmarking support in Jenkins Test Harness. There are many stories related to performance degradation in Jenkins, and having a framework could help us to have some checks for the critical functionality like permission checks. The end goal of the project is to improve Role Strategy performance (which is far from perfect now), but it would be great if we could reuse the framework in other components.
>
> Tomorrow at 7am UTC we will be doing a recorded demo of the current framework state, and everybody is welcome to join.
>
> Best regards,
> Oleg
>
> On Tuesday, May 28, 2019 at 8:20:14 AM UTC+2, [hidden email] wrote:
>>
>> Hi everyone,
>>
>> As a part of this year's Google Summer of Code, I have been working on a
>> framework for allowing Java Microbenchmark Harness (JMH) benchmarks to be
>> run with Jenkins. The basic framework was implemented in the Role Strategy
>> Plugin through this pull request:
>> https://github.com/jenkinsci/role-strategy-plugin/pull/63 . For each
>> benchmark, a temporary Jenkins instance is created, similar to how
>> JenkinsRule from Jenkins Test Harness does it. The benchmarks are run
>> through a unit test which allows them to be integrated with the normal pull
>> request builds (if required). Apart from configuring the Jenkins instance
>> for test using Java code, I've also added support for easily configuring the
>> instance by YAML files using Jenkins Configuration as Code. The benchmark
>> reports are generated as JSON files which are compatible with the jmh-report
>> plugin. I have implemented a couple of benchmarks for the Role Strategy
>> Plugin which can be seen in the Role Strategy Plugin GitHub repository. I
>> have attached screenshots of these results as visualized by the jmh-report
>> plugin. To make the framework easier to use, I'm also working on running the
>> benchmarks through a Maven profile.
>>
>> To make it available to the rest of the community, I would like to propose
>> to make it either a part of Jenkins Test Harness or a part of new library so
>> it can be used by other plugins and perhaps by Jenkins core itself. I would
>> to love to hear your feedback, comments and suggestions. It would be even
>> better if you could join the project's weekly meetings (Tuesdays and Fridays
>> at 7:00 AM UTC) and help us improve this framework.
>>
>> Thanks
>> Abhyudaya
>> GitHub: AbhyudayaSharma
>>
>>
> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank">jenkin...@....
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/09147b86-6565-4b83-a4f8-3fbf48baa6cf%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Matt Sicker
Senior Software Engineer, CloudBees

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/W1jrwTEFnz0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/676323d7-7d32-43be-9003-d0d8eab9e02e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/00d901d51f80%24359a0960%24a0ce1c20%24%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Benchmarking Jenkins using Java Microbenchmark Harness (JMH)

Abhyudaya Sharma
In reply to this post by Abhyudaya Sharma
Hi everyone,

The Micro-benchmarking framework is now ready and is available through Plugin POM 3.46 or through Jenkins Test Harness 2.51. There is now a blog post summarizing it on jenkins.io. You can find examples for the benchmarks in the Role Strategy Plugin here. Looking forward to your feedback and comments.

Thanks
Abhyudaya Sharma

On Tuesday, 28 May 2019 11:50:14 UTC+5:30, Abhyudaya Sharma wrote:
Hi everyone,

As a part of this year's Google Summer of Code, I have been working on a
framework for allowing Java Microbenchmark Harness (JMH) benchmarks to be
run with Jenkins. The basic framework was implemented in the Role Strategy
Plugin through this pull request:
<a href="https://github.com/jenkinsci/role-strategy-plugin/pull/63" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Frole-strategy-plugin%2Fpull%2F63\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGQppXbuVpAsX-p5wEABV_5ac-npA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Frole-strategy-plugin%2Fpull%2F63\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGQppXbuVpAsX-p5wEABV_5ac-npA&#39;;return true;">https://github.com/jenkinsci/role-strategy-plugin/pull/63 . For each
benchmark, a temporary Jenkins instance is created, similar to how
JenkinsRule from Jenkins Test Harness does it. The benchmarks are run
through a unit test which allows them to be integrated with the normal pull
request builds (if required). Apart from configuring the Jenkins instance
for test using Java code, I've also added support for easily configuring the
instance by YAML files using Jenkins Configuration as Code. The benchmark
reports are generated as JSON files which are compatible with the jmh-report
plugin. I have implemented a couple of benchmarks for the Role Strategy
Plugin which can be seen in the Role Strategy Plugin GitHub repository. I
have attached screenshots of these results as visualized by the jmh-report
plugin. To make the framework easier to use, I'm also working on running the
benchmarks through a Maven profile.

To make it available to the rest of the community, I would like to propose
to make it either a part of Jenkins Test Harness or a part of new library so
it can be used by other plugins and perhaps by Jenkins core itself. I would
to love to hear your feedback, comments and suggestions. It would be even
better if you could join the project's weekly meetings (Tuesdays and Fridays
at 7:00 AM UTC) and help us improve this framework.

Thanks
Abhyudaya
GitHub: AbhyudayaSharma


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/7d49f6f8-79ed-49ab-a9ab-f94b376caf4a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Benchmarking Jenkins using Java Microbenchmark Harness (JMH)

Ulli Hafner
Is there a way to use the JMH within the individual steps of a plugin? The examples I have seen so far start with a given Jenkins instance and then call methods on Jenkins object model.

So what is supported is something like 

jenkins.doSomething();

to measure the performance of one of Jenkins API methods.

or 

job.scheduleBuild2()

to measure the performance of a whole build pipeline. 

What I’m interested in my warnings plugin is the performance of individual steps in the pipeline. 
E.g., if I have the following steps:

    checkout scm
    withMaven(maven: 'mvn-default', mavenLocalRepo: '/var/data/m2repository', mavenOpts: '-Xmx768m -Xms512m') {
        sh 'mvn -ntp -V -e clean verify -f plugin/pom.xml -Dmaven.test.failure.ignore -DskipITs'
    }
    recordIssues tools: [java(), javaDoc()], aggregatingResults: 'true', id: 'java', name: 'Java'

I am interested only in the results of the last step. 

It would be even better if I can obtain the performance results of the individual activities in my plugin. I.e. the recordWarnings step actually is composed of several steps:
- parse warnings
- compute fingerprints
- collect blames from Git
etc. 

Is this supported somehow yet?


Am 22.06.2019 um 07:32 schrieb Abhyudaya Sharma <[hidden email]>:

Hi everyone,

The Micro-benchmarking framework is now ready and is available through Plugin POM 3.46 or through Jenkins Test Harness 2.51. There is now a blog post summarizing it on jenkins.io. You can find examples for the benchmarks in the Role Strategy Plugin here. Looking forward to your feedback and comments.

Thanks
Abhyudaya Sharma

On Tuesday, 28 May 2019 11:50:14 UTC+5:30, Abhyudaya Sharma wrote:
Hi everyone, 

As a part of this year's Google Summer of Code, I have been working on a 
framework for allowing Java Microbenchmark Harness (JMH) benchmarks to be 
run with Jenkins. The basic framework was implemented in the Role Strategy 
Plugin through this pull request: 
https://github.com/jenkinsci/role-strategy-plugin/pull/63 . For each 
benchmark, a temporary Jenkins instance is created, similar to how 
JenkinsRule from Jenkins Test Harness does it. The benchmarks are run 
through a unit test which allows them to be integrated with the normal pull 
request builds (if required). Apart from configuring the Jenkins instance 
for test using Java code, I've also added support for easily configuring the 
instance by YAML files using Jenkins Configuration as Code. The benchmark 
reports are generated as JSON files which are compatible with the jmh-report 
plugin. I have implemented a couple of benchmarks for the Role Strategy 
Plugin which can be seen in the Role Strategy Plugin GitHub repository. I 
have attached screenshots of these results as visualized by the jmh-report 
plugin. To make the framework easier to use, I'm also working on running the 
benchmarks through a Maven profile. 

To make it available to the rest of the community, I would like to propose 
to make it either a part of Jenkins Test Harness or a part of new library so 
it can be used by other plugins and perhaps by Jenkins core itself. I would 
to love to hear your feedback, comments and suggestions. It would be even 
better if you could join the project's weekly meetings (Tuesdays and Fridays 
at 7:00 AM UTC) and help us improve this framework. 

Thanks 
Abhyudaya 
GitHub: AbhyudayaSharma 



-- 
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/7d49f6f8-79ed-49ab-a9ab-f94b376caf4a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/F0FEB18C-9B6F-495A-A4F8-31BAE16271C0%40gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Benchmarking Jenkins using Java Microbenchmark Harness (JMH)

Mark Waite-2
Rishabh has created two JMH benchmarks for specific git operations inside the Jenkins git plugin.  He's running those micro benchmarks without a JenkinsRule because the git plugin does not require a JenkinsRule to operate at that level.

I assume you need a JenkinsRule because you're running a pipeline.  However, it seems like you could make the pre-steps be part of the initial creation of state, then run the micro benchmark solely on the subset of the pipeline that interests you.

On Sun, Jun 7, 2020 at 7:52 AM Ullrich Hafner <[hidden email]> wrote:
Is there a way to use the JMH within the individual steps of a plugin? The examples I have seen so far start with a given Jenkins instance and then call methods on Jenkins object model.

So what is supported is something like 

jenkins.doSomething();

to measure the performance of one of Jenkins API methods.

or 

job.scheduleBuild2()

to measure the performance of a whole build pipeline. 

What I’m interested in my warnings plugin is the performance of individual steps in the pipeline. 
E.g., if I have the following steps:

    checkout scm
    withMaven(maven: 'mvn-default', mavenLocalRepo: '/var/data/m2repository', mavenOpts: '-Xmx768m -Xms512m') {
        sh 'mvn -ntp -V -e clean verify -f plugin/pom.xml -Dmaven.test.failure.ignore -DskipITs'
    }
    recordIssues tools: [java(), javaDoc()], aggregatingResults: 'true', id: 'java', name: 'Java'

I am interested only in the results of the last step. 

It would be even better if I can obtain the performance results of the individual activities in my plugin. I.e. the recordWarnings step actually is composed of several steps:
- parse warnings
- compute fingerprints
- collect blames from Git
etc. 

Is this supported somehow yet?


Am 22.06.2019 um 07:32 schrieb Abhyudaya Sharma <[hidden email]>:

Hi everyone,

The Micro-benchmarking framework is now ready and is available through Plugin POM 3.46 or through Jenkins Test Harness 2.51. There is now a blog post summarizing it on jenkins.io. You can find examples for the benchmarks in the Role Strategy Plugin here. Looking forward to your feedback and comments.

Thanks
Abhyudaya Sharma

On Tuesday, 28 May 2019 11:50:14 UTC+5:30, Abhyudaya Sharma wrote:
Hi everyone, 

As a part of this year's Google Summer of Code, I have been working on a 
framework for allowing Java Microbenchmark Harness (JMH) benchmarks to be 
run with Jenkins. The basic framework was implemented in the Role Strategy 
Plugin through this pull request: 
https://github.com/jenkinsci/role-strategy-plugin/pull/63 . For each 
benchmark, a temporary Jenkins instance is created, similar to how 
JenkinsRule from Jenkins Test Harness does it. The benchmarks are run 
through a unit test which allows them to be integrated with the normal pull 
request builds (if required). Apart from configuring the Jenkins instance 
for test using Java code, I've also added support for easily configuring the 
instance by YAML files using Jenkins Configuration as Code. The benchmark 
reports are generated as JSON files which are compatible with the jmh-report 
plugin. I have implemented a couple of benchmarks for the Role Strategy 
Plugin which can be seen in the Role Strategy Plugin GitHub repository. I 
have attached screenshots of these results as visualized by the jmh-report 
plugin. To make the framework easier to use, I'm also working on running the 
benchmarks through a Maven profile. 

To make it available to the rest of the community, I would like to propose 
to make it either a part of Jenkins Test Harness or a part of new library so 
it can be used by other plugins and perhaps by Jenkins core itself. I would 
to love to hear your feedback, comments and suggestions. It would be even 
better if you could join the project's weekly meetings (Tuesdays and Fridays 
at 7:00 AM UTC) and help us improve this framework. 

Thanks 
Abhyudaya 
GitHub: AbhyudayaSharma 



-- 
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/7d49f6f8-79ed-49ab-a9ab-f94b376caf4a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/F0FEB18C-9B6F-495A-A4F8-31BAE16271C0%40gmail.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGneTpq4OpuTjKaFu1%3D-7QCbO2ZqFQmxpaERWgp1AS1ew%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Benchmarking Jenkins using Java Microbenchmark Harness (JMH)

Rishabh Budhouliya
From Role Strategy Plugin or Configuration as a Code Plugin, the JMH examples show us that the intention is to provide a JMH environment which is fairly similar to a JUnit test environment. For example, as Mark mentioned, a JenkinsState was created to provide a JenkinsRule in the JMH environment to benchmark associated functionalities.

JMH provides a static state class where you can initialize almost everything you need to benchmark a step or a particular operation. I used that approach to benchmark git operations without using a JenkinsRule by just creating a custom class which would provide me fresh local repos for the lifetime of the benchmark. (the idea was simply copied from JUnit @Rule TemporaryFolder class)

So it is fair to assume that anything we can test in a JUnit test, we can do that in a JMH benchmark too.

On Sunday, June 7, 2020 at 10:26:31 PM UTC+5:30, Mark Waite wrote:
Rishabh has created two JMH benchmarks for specific git operations inside the Jenkins git plugin.  He's running those micro benchmarks without a JenkinsRule because the git plugin does not require a JenkinsRule to operate at that level.

I assume you need a JenkinsRule because you're running a pipeline.  However, it seems like you could make the pre-steps be part of the initial creation of state, then run the micro benchmark solely on the subset of the pipeline that interests you.

On Sun, Jun 7, 2020 at 7:52 AM Ullrich Hafner <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="UkLpkW0BAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">ullric...@...> wrote:
Is there a way to use the JMH within the individual steps of a plugin? The examples I have seen so far start with a given Jenkins instance and then call methods on Jenkins object model.

So what is supported is something like 

jenkins.doSomething();

to measure the performance of one of Jenkins API methods.

or 

job.scheduleBuild2()

to measure the performance of a whole build pipeline. 

What I’m interested in my warnings plugin is the performance of individual steps in the pipeline. 
E.g., if I have the following steps:

    checkout scm
    withMaven(maven: 'mvn-default', mavenLocalRepo: '/var/data/m2repository', mavenOpts: '-Xmx768m -Xms512m') {
        sh 'mvn -ntp -V -e clean verify -f plugin/pom.xml -Dmaven.test.failure.ignore -DskipITs'
    }
    recordIssues tools: [java(), javaDoc()], aggregatingResults: 'true', id: 'java', name: 'Java'

I am interested only in the results of the last step. 

It would be even better if I can obtain the performance results of the individual activities in my plugin. I.e. the recordWarnings step actually is composed of several steps:
- parse warnings
- compute fingerprints
- collect blames from Git
etc. 

Is this supported somehow yet?


Am 22.06.2019 um 07:32 schrieb Abhyudaya Sharma <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="UkLpkW0BAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">sharmaa...@...>:

Hi everyone,

The Micro-benchmarking framework is now ready and is available through Plugin POM 3.46 or through Jenkins Test Harness 2.51. There is now a <a href="https://jenkins.io/blog/2019/06/21/performance-testing-jenkins/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fjenkins.io%2Fblog%2F2019%2F06%2F21%2Fperformance-testing-jenkins%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFzxd64I0gJMxp0U0WLQX80nzzOwQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fjenkins.io%2Fblog%2F2019%2F06%2F21%2Fperformance-testing-jenkins%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFzxd64I0gJMxp0U0WLQX80nzzOwQ&#39;;return true;">blog post summarizing it on <a href="http://jenkins.io/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fjenkins.io%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEo1lmJhX4WqqsewrasDhmLvTPo3A&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fjenkins.io%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEo1lmJhX4WqqsewrasDhmLvTPo3A&#39;;return true;">jenkins.io. You can find examples for the benchmarks in the Role Strategy Plugin <a href="https://github.com/jenkinsci/role-strategy-plugin/tree/master/src/test/java/jmh/benchmarks" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Frole-strategy-plugin%2Ftree%2Fmaster%2Fsrc%2Ftest%2Fjava%2Fjmh%2Fbenchmarks\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFRvNjSUjVR6yuCuGUcuvD46yuc1A&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Frole-strategy-plugin%2Ftree%2Fmaster%2Fsrc%2Ftest%2Fjava%2Fjmh%2Fbenchmarks\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFRvNjSUjVR6yuCuGUcuvD46yuc1A&#39;;return true;">here. Looking forward to your feedback and comments.

Thanks
Abhyudaya Sharma

On Tuesday, 28 May 2019 11:50:14 UTC+5:30, Abhyudaya Sharma wrote:
Hi everyone, 

As a part of this year's Google Summer of Code, I have been working on a 
framework for allowing Java Microbenchmark Harness (JMH) benchmarks to be 
run with Jenkins. The basic framework was implemented in the Role Strategy 
Plugin through this pull request: 
<a href="https://github.com/jenkinsci/role-strategy-plugin/pull/63" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Frole-strategy-plugin%2Fpull%2F63\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGQppXbuVpAsX-p5wEABV_5ac-npA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Frole-strategy-plugin%2Fpull%2F63\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGQppXbuVpAsX-p5wEABV_5ac-npA&#39;;return true;">https://github.com/jenkinsci/role-strategy-plugin/pull/63 . For each 
benchmark, a temporary Jenkins instance is created, similar to how 
JenkinsRule from Jenkins Test Harness does it. The benchmarks are run 
through a unit test which allows them to be integrated with the normal pull 
request builds (if required). Apart from configuring the Jenkins instance 
for test using Java code, I've also added support for easily configuring the 
instance by YAML files using Jenkins Configuration as Code. The benchmark 
reports are generated as JSON files which are compatible with the jmh-report 
plugin. I have implemented a couple of benchmarks for the Role Strategy 
Plugin which can be seen in the Role Strategy Plugin GitHub repository. I 
have attached screenshots of these results as visualized by the jmh-report 
plugin. To make the framework easier to use, I'm also working on running the 
benchmarks through a Maven profile. 

To make it available to the rest of the community, I would like to propose 
to make it either a part of Jenkins Test Harness or a part of new library so 
it can be used by other plugins and perhaps by Jenkins core itself. I would 
to love to hear your feedback, comments and suggestions. It would be even 
better if you could join the project's weekly meetings (Tuesdays and Fridays 
at 7:00 AM UTC) and help us improve this framework. 

Thanks 
Abhyudaya 
GitHub: AbhyudayaSharma 



-- 
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" style="font-family:Helvetica;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank" gdf-obfuscated-mailto="UkLpkW0BAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jenkin...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-dev/7d49f6f8-79ed-49ab-a9ab-f94b376caf4a%40googlegroups.com?utm_medium=email&amp;utm_source=footer" style="font-family:Helvetica;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-dev/7d49f6f8-79ed-49ab-a9ab-f94b376caf4a%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-dev/7d49f6f8-79ed-49ab-a9ab-f94b376caf4a%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-dev/7d49f6f8-79ed-49ab-a9ab-f94b376caf4a%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" style="font-family:Helvetica;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="UkLpkW0BAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jenkin...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-dev/F0FEB18C-9B6F-495A-A4F8-31BAE16271C0%40gmail.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-dev/F0FEB18C-9B6F-495A-A4F8-31BAE16271C0%40gmail.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-dev/F0FEB18C-9B6F-495A-A4F8-31BAE16271C0%40gmail.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-dev/F0FEB18C-9B6F-495A-A4F8-31BAE16271C0%40gmail.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/7826c83d-81f1-4b2f-ace9-ea3dc254f952o%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: Benchmarking Jenkins using Java Microbenchmark Harness (JMH)

Ulli Hafner
In reply to this post by Mark Waite-2
I see, that makes sense. Seems that I was just following the wrong path…  
Do you have a link to those Git performance tests? Are they in the git or in the git-client plugin?

Am 07.06.2020 um 18:56 schrieb Mark Waite <[hidden email]>:

Rishabh has created two JMH benchmarks for specific git operations inside the Jenkins git plugin.  He's running those micro benchmarks without a JenkinsRule because the git plugin does not require a JenkinsRule to operate at that level.

I assume you need a JenkinsRule because you're running a pipeline.  However, it seems like you could make the pre-steps be part of the initial creation of state, then run the micro benchmark solely on the subset of the pipeline that interests you.

On Sun, Jun 7, 2020 at 7:52 AM Ullrich Hafner <[hidden email]> wrote:
Is there a way to use the JMH within the individual steps of a plugin? The examples I have seen so far start with a given Jenkins instance and then call methods on Jenkins object model.

So what is supported is something like 

jenkins.doSomething();

to measure the performance of one of Jenkins API methods.

or 

job.scheduleBuild2()

to measure the performance of a whole build pipeline. 

What I’m interested in my warnings plugin is the performance of individual steps in the pipeline. 
E.g., if I have the following steps:

    checkout scm
    withMaven(maven: 'mvn-default', mavenLocalRepo: '/var/data/m2repository', mavenOpts: '-Xmx768m -Xms512m') {
        sh 'mvn -ntp -V -e clean verify -f plugin/pom.xml -Dmaven.test.failure.ignore -DskipITs'
    }
    recordIssues tools: [java(), javaDoc()], aggregatingResults: 'true', id: 'java', name: 'Java'

I am interested only in the results of the last step. 

It would be even better if I can obtain the performance results of the individual activities in my plugin. I.e. the recordWarnings step actually is composed of several steps:
- parse warnings
- compute fingerprints
- collect blames from Git
etc. 

Is this supported somehow yet?


Am 22.06.2019 um 07:32 schrieb Abhyudaya Sharma <[hidden email]>:

Hi everyone,

The Micro-benchmarking framework is now ready and is available through Plugin POM 3.46 or through Jenkins Test Harness 2.51. There is now a blog post summarizing it on jenkins.io. You can find examples for the benchmarks in the Role Strategy Plugin here. Looking forward to your feedback and comments.

Thanks
Abhyudaya Sharma

On Tuesday, 28 May 2019 11:50:14 UTC+5:30, Abhyudaya Sharma wrote:
Hi everyone, 

As a part of this year's Google Summer of Code, I have been working on a 
framework for allowing Java Microbenchmark Harness (JMH) benchmarks to be 
run with Jenkins. The basic framework was implemented in the Role Strategy 
Plugin through this pull request: 
https://github.com/jenkinsci/role-strategy-plugin/pull/63 . For each 
benchmark, a temporary Jenkins instance is created, similar to how 
JenkinsRule from Jenkins Test Harness does it. The benchmarks are run 
through a unit test which allows them to be integrated with the normal pull 
request builds (if required). Apart from configuring the Jenkins instance 
for test using Java code, I've also added support for easily configuring the 
instance by YAML files using Jenkins Configuration as Code. The benchmark 
reports are generated as JSON files which are compatible with the jmh-report 
plugin. I have implemented a couple of benchmarks for the Role Strategy 
Plugin which can be seen in the Role Strategy Plugin GitHub repository. I 
have attached screenshots of these results as visualized by the jmh-report 
plugin. To make the framework easier to use, I'm also working on running the 
benchmarks through a Maven profile. 

To make it available to the rest of the community, I would like to propose 
to make it either a part of Jenkins Test Harness or a part of new library so 
it can be used by other plugins and perhaps by Jenkins core itself. I would 
to love to hear your feedback, comments and suggestions. It would be even 
better if you could join the project's weekly meetings (Tuesdays and Fridays 
at 7:00 AM UTC) and help us improve this framework. 

Thanks 
Abhyudaya 
GitHub: AbhyudayaSharma 



-- 
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/7d49f6f8-79ed-49ab-a9ab-f94b376caf4a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/F0FEB18C-9B6F-495A-A4F8-31BAE16271C0%40gmail.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGneTpq4OpuTjKaFu1%3D-7QCbO2ZqFQmxpaERWgp1AS1ew%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/2E90C090-8789-4A5D-B783-FC4883568AA0%40gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: Benchmarking Jenkins using Java Microbenchmark Harness (JMH)

Rishabh Budhouliya
The benchmark module in the git-client plugin: https://github.com/jenkinsci/git-client-plugin/tree/master/src/test/java/jmh/benchmark
A JMH benchmark on git fetch: https://github.com/jenkinsci/git-client-plugin/blob/master/src/test/java/jmh/benchmark/GitClientFetchBenchmark.java

On Monday, June 8, 2020 at 4:57:51 PM UTC+5:30, Ullrich Hafner wrote:
I see, that makes sense. Seems that I was just following the wrong path…  
Do you have a link to those Git performance tests? Are they in the git or in the git-client plugin?

Am 07.06.2020 um 18:56 schrieb Mark Waite <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="yoKBpBI-AwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">mark.e...@...>:

Rishabh has created two JMH benchmarks for specific git operations inside the Jenkins git plugin.  He's running those micro benchmarks without a JenkinsRule because the git plugin does not require a JenkinsRule to operate at that level.

I assume you need a JenkinsRule because you're running a pipeline.  However, it seems like you could make the pre-steps be part of the initial creation of state, then run the micro benchmark solely on the subset of the pipeline that interests you.

On Sun, Jun 7, 2020 at 7:52 AM Ullrich Hafner <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="yoKBpBI-AwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">ullric...@...> wrote:
Is there a way to use the JMH within the individual steps of a plugin? The examples I have seen so far start with a given Jenkins instance and then call methods on Jenkins object model.

So what is supported is something like 

jenkins.doSomething();

to measure the performance of one of Jenkins API methods.

or 

job.scheduleBuild2()

to measure the performance of a whole build pipeline. 

What I’m interested in my warnings plugin is the performance of individual steps in the pipeline. 
E.g., if I have the following steps:

    checkout scm
    withMaven(maven: 'mvn-default', mavenLocalRepo: '/var/data/m2repository', mavenOpts: '-Xmx768m -Xms512m') {
        sh 'mvn -ntp -V -e clean verify -f plugin/pom.xml -Dmaven.test.failure.ignore -DskipITs'
    }
    recordIssues tools: [java(), javaDoc()], aggregatingResults: 'true', id: 'java', name: 'Java'

I am interested only in the results of the last step. 

It would be even better if I can obtain the performance results of the individual activities in my plugin. I.e. the recordWarnings step actually is composed of several steps:
- parse warnings
- compute fingerprints
- collect blames from Git
etc. 

Is this supported somehow yet?


Am 22.06.2019 um 07:32 schrieb Abhyudaya Sharma <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="yoKBpBI-AwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">sharmaa...@...>:

Hi everyone,

The Micro-benchmarking framework is now ready and is available through Plugin POM 3.46 or through Jenkins Test Harness 2.51. There is now a <a href="https://jenkins.io/blog/2019/06/21/performance-testing-jenkins/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fjenkins.io%2Fblog%2F2019%2F06%2F21%2Fperformance-testing-jenkins%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFzxd64I0gJMxp0U0WLQX80nzzOwQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fjenkins.io%2Fblog%2F2019%2F06%2F21%2Fperformance-testing-jenkins%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFzxd64I0gJMxp0U0WLQX80nzzOwQ&#39;;return true;">blog post summarizing it on <a href="http://jenkins.io/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fjenkins.io%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEo1lmJhX4WqqsewrasDhmLvTPo3A&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fjenkins.io%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEo1lmJhX4WqqsewrasDhmLvTPo3A&#39;;return true;">jenkins.io. You can find examples for the benchmarks in the Role Strategy Plugin <a href="https://github.com/jenkinsci/role-strategy-plugin/tree/master/src/test/java/jmh/benchmarks" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Frole-strategy-plugin%2Ftree%2Fmaster%2Fsrc%2Ftest%2Fjava%2Fjmh%2Fbenchmarks\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFRvNjSUjVR6yuCuGUcuvD46yuc1A&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Frole-strategy-plugin%2Ftree%2Fmaster%2Fsrc%2Ftest%2Fjava%2Fjmh%2Fbenchmarks\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFRvNjSUjVR6yuCuGUcuvD46yuc1A&#39;;return true;">here. Looking forward to your feedback and comments.

Thanks
Abhyudaya Sharma

On Tuesday, 28 May 2019 11:50:14 UTC+5:30, Abhyudaya Sharma wrote:
Hi everyone, 

As a part of this year's Google Summer of Code, I have been working on a 
framework for allowing Java Microbenchmark Harness (JMH) benchmarks to be 
run with Jenkins. The basic framework was implemented in the Role Strategy 
Plugin through this pull request: 
<a href="https://github.com/jenkinsci/role-strategy-plugin/pull/63" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Frole-strategy-plugin%2Fpull%2F63\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGQppXbuVpAsX-p5wEABV_5ac-npA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Frole-strategy-plugin%2Fpull%2F63\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGQppXbuVpAsX-p5wEABV_5ac-npA&#39;;return true;">https://github.com/jenkinsci/role-strategy-plugin/pull/63 . For each 
benchmark, a temporary Jenkins instance is created, similar to how 
JenkinsRule from Jenkins Test Harness does it. The benchmarks are run 
through a unit test which allows them to be integrated with the normal pull 
request builds (if required). Apart from configuring the Jenkins instance 
for test using Java code, I've also added support for easily configuring the 
instance by YAML files using Jenkins Configuration as Code. The benchmark 
reports are generated as JSON files which are compatible with the jmh-report 
plugin. I have implemented a couple of benchmarks for the Role Strategy 
Plugin which can be seen in the Role Strategy Plugin GitHub repository. I 
have attached screenshots of these results as visualized by the jmh-report 
plugin. To make the framework easier to use, I'm also working on running the 
benchmarks through a Maven profile. 

To make it available to the rest of the community, I would like to propose 
to make it either a part of Jenkins Test Harness or a part of new library so 
it can be used by other plugins and perhaps by Jenkins core itself. I would 
to love to hear your feedback, comments and suggestions. It would be even 
better if you could join the project's weekly meetings (Tuesdays and Fridays 
at 7:00 AM UTC) and help us improve this framework. 

Thanks 
Abhyudaya 
GitHub: AbhyudayaSharma 



-- 
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" style="font-family:Helvetica;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank" gdf-obfuscated-mailto="yoKBpBI-AwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jenkin...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-dev/7d49f6f8-79ed-49ab-a9ab-f94b376caf4a%40googlegroups.com?utm_medium=email&amp;utm_source=footer" style="font-family:Helvetica;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-dev/7d49f6f8-79ed-49ab-a9ab-f94b376caf4a%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-dev/7d49f6f8-79ed-49ab-a9ab-f94b376caf4a%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-dev/7d49f6f8-79ed-49ab-a9ab-f94b376caf4a%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" style="font-family:Helvetica;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="yoKBpBI-AwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jenkin...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-dev/F0FEB18C-9B6F-495A-A4F8-31BAE16271C0%40gmail.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-dev/F0FEB18C-9B6F-495A-A4F8-31BAE16271C0%40gmail.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-dev/F0FEB18C-9B6F-495A-A4F8-31BAE16271C0%40gmail.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-dev/F0FEB18C-9B6F-495A-A4F8-31BAE16271C0%40gmail.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="yoKBpBI-AwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jenkin...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGneTpq4OpuTjKaFu1%3D-7QCbO2ZqFQmxpaERWgp1AS1ew%40mail.gmail.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGneTpq4OpuTjKaFu1%3D-7QCbO2ZqFQmxpaERWgp1AS1ew%40mail.gmail.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGneTpq4OpuTjKaFu1%3D-7QCbO2ZqFQmxpaERWgp1AS1ew%40mail.gmail.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGneTpq4OpuTjKaFu1%3D-7QCbO2ZqFQmxpaERWgp1AS1ew%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/80b754e1-faf0-49d7-9b32-a65a452603f8o%40googlegroups.com.