About Jenkins slowness for version 2.138 in multi branch pipeline project

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

About Jenkins slowness for version 2.138 in multi branch pipeline project

Mahesh Wabale
Hi Team , 

We are using observing Jenkins slowness issues for jenkins version 2.138 in multi branch pipeline project . 
We have also used NFS 4.1 . 

After Jenkins restart it works for few days but at some stage it stuck and becomes unresponsive . As per observation only restart will solve this issue for next few days . We require frequent restart due to this issue . Have anyone observed similar issues , or  this is open bug in Jenkins ?    


--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/319f90ff-5410-4495-bb46-51d61745b3e8%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: About Jenkins slowness for version 2.138 in multi branch pipeline project

Mark Waite-2
I'm not aware of an open bug in Jenkins related to being slow over NFS, though Jenkins is quite disc intensive and runs best with local disc drives.

You might refer to the following articles for more information:
  • Jenkins pipeline durability settings documentation ("Will Higher-Performance Durability Settings Help Me? Yes, if your Jenkins instance uses NFS, magnetic storage, runs many Pipelines at once, or shows high iowait.")
  • CloudBees NFS guide (multiple pages of tuning recommendations)


On Fri, Jan 17, 2020 at 5:42 AM Mahesh Wabale <[hidden email]> wrote:
Hi Team , 

We are using observing Jenkins slowness issues for jenkins version 2.138 in multi branch pipeline project . 
We have also used NFS 4.1 . 

After Jenkins restart it works for few days but at some stage it stuck and becomes unresponsive . As per observation only restart will solve this issue for next few days . We require frequent restart due to this issue . Have anyone observed similar issues , or  this is open bug in Jenkins ?    


--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/319f90ff-5410-4495-bb46-51d61745b3e8%40googlegroups.com.


--
Thanks!
Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtH0igk%3DzrjNvmNt%2B9KWgen4oweRfhhHQC_sg%3DrZTMKtYg%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: About Jenkins slowness for version 2.138 in multi branch pipeline project

Mahesh Wabale
Thanks Marc , 

We will check if it helps , Currently we have "maximum durability" settings configured now , we are planning to make it to "performance-optimized"  .  We have observed couple of time disc IO is very high sometimes . 

On Monday, January 20, 2020 at 5:54:38 AM UTC+5:30, Mark Waite wrote:
I'm not aware of an open bug in Jenkins related to being slow over NFS, though Jenkins is quite disc intensive and runs best with local disc drives.

You might refer to the following articles for more information:
  • Jenkins <a href="https://www.google.com/url?q=https%3A%2F%2Fjenkins.io%2Fdoc%2Fbook%2Fpipeline%2Fscaling-pipeline%2F%23will-higher-performance-durability-settings-help-me&amp;sa=D&amp;sntz=1&amp;usg=AFQjCNEtvgPugqm3XG5_99K4YYJi6vCf6Q" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fjenkins.io%2Fdoc%2Fbook%2Fpipeline%2Fscaling-pipeline%2F%23will-higher-performance-durability-settings-help-me\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEtvgPugqm3XG5_99K4YYJi6vCf6Q&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fjenkins.io%2Fdoc%2Fbook%2Fpipeline%2Fscaling-pipeline%2F%23will-higher-performance-durability-settings-help-me\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEtvgPugqm3XG5_99K4YYJi6vCf6Q&#39;;return true;">pipeline durability settings documentation ("Will Higher-Performance Durability Settings Help Me? Yes, if your Jenkins instance uses NFS, magnetic storage, runs many Pipelines at once, or shows high iowait.")
  • CloudBees <a href="https://support.cloudbees.com/hc/en-us/articles/217479948-NFS-Guide" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F217479948-NFS-Guide\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHa4LWaUx485443uzldQwSMF_2r3Q&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F217479948-NFS-Guide\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHa4LWaUx485443uzldQwSMF_2r3Q&#39;;return true;">NFS guide (multiple pages of tuning recommendations)


On Fri, Jan 17, 2020 at 5:42 AM Mahesh Wabale <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="2FZe9pNCDAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">mahes...@...> wrote:
Hi Team , 

We are using observing Jenkins slowness issues for jenkins version 2.138 in multi branch pipeline project . 
We have also used NFS 4.1 . 

After Jenkins restart it works for few days but at some stage it stuck and becomes unresponsive . As per observation only restart will solve this issue for next few days . We require frequent restart due to this issue . Have anyone observed similar issues , or  this is open bug in Jenkins ?    


--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="2FZe9pNCDAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jenkins...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/319f90ff-5410-4495-bb46-51d61745b3e8%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/319f90ff-5410-4495-bb46-51d61745b3e8%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/319f90ff-5410-4495-bb46-51d61745b3e8%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/319f90ff-5410-4495-bb46-51d61745b3e8%40googlegroups.com.


--
Thanks!
Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/8a711ba9-4433-4e62-9170-50e7f5c82bdd%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: About Jenkins slowness for version 2.138 in multi branch pipeline project

Mahesh Wabale
We are using following JAVA_OPTS CONFIG in Jenkins deployment .

"name": "JAVA_OPTS",
                "value": "-XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+ParallelRefProcEnabled -XX:+UseStringDeduplication -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=20 -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:G1SummarizeRSetStatsPeriod=1 -Xms2048m -Xmx8192m"   , and 48 GB ram allocated for master Jenkins pod as per initial observation GC is working correctly its not going beyond 8 gb . But observing RAM USAGE of master pod is continusly increasing day by day . Some time it reaches to memory 48 GB seems like it taking memory for cache .How can we control ram memory size , or is it possible to free cache memory from Jenkins Docker image which can solve my problem .

On Mon, Jan 20, 2020, 3:45 PM Mahesh Wabale <[hidden email]> wrote:
Thanks Marc , 

We will check if it helps , Currently we have "maximum durability" settings configured now , we are planning to make it to "performance-optimized"  .  We have observed couple of time disc IO is very high sometimes . 

On Monday, January 20, 2020 at 5:54:38 AM UTC+5:30, Mark Waite wrote:
I'm not aware of an open bug in Jenkins related to being slow over NFS, though Jenkins is quite disc intensive and runs best with local disc drives.

You might refer to the following articles for more information:
  • Jenkins pipeline durability settings documentation ("Will Higher-Performance Durability Settings Help Me? Yes, if your Jenkins instance uses NFS, magnetic storage, runs many Pipelines at once, or shows high iowait.")
  • CloudBees NFS guide (multiple pages of tuning recommendations)


On Fri, Jan 17, 2020 at 5:42 AM Mahesh Wabale <[hidden email]> wrote:
Hi Team , 

We are using observing Jenkins slowness issues for jenkins version 2.138 in multi branch pipeline project . 
We have also used NFS 4.1 . 

After Jenkins restart it works for few days but at some stage it stuck and becomes unresponsive . As per observation only restart will solve this issue for next few days . We require frequent restart due to this issue . Have anyone observed similar issues , or  this is open bug in Jenkins ?    


--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/319f90ff-5410-4495-bb46-51d61745b3e8%40googlegroups.com.


--
Thanks!
Mark Waite

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/8MnTICGxQ-o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/8a711ba9-4433-4e62-9170-50e7f5c82bdd%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CABOY2ebxtXCGqDXKcfk7kzh0CgjvajTmyiOSni4EPF516gy9iA%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: About Jenkins slowness for version 2.138 in multi branch pipeline project

Mark Waite-2
You might install the Jenkins Health Advisor by CloudBees and see if it has suggestions for your instance.  It will check for conditions that have been found to cause problems in other installations and report them to you in a daily e-mail message.


On Mon, Jan 20, 2020 at 6:49 AM Mahesh Wabale <[hidden email]> wrote:
We are using following JAVA_OPTS CONFIG in Jenkins deployment .

"name": "JAVA_OPTS",
                "value": "-XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+ParallelRefProcEnabled -XX:+UseStringDeduplication -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=20 -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:G1SummarizeRSetStatsPeriod=1 -Xms2048m -Xmx8192m"   , and 48 GB ram allocated for master Jenkins pod as per initial observation GC is working correctly its not going beyond 8 gb . But observing RAM USAGE of master pod is continusly increasing day by day . Some time it reaches to memory 48 GB seems like it taking memory for cache .How can we control ram memory size , or is it possible to free cache memory from Jenkins Docker image which can solve my problem .

On Mon, Jan 20, 2020, 3:45 PM Mahesh Wabale <[hidden email]> wrote:
Thanks Marc , 

We will check if it helps , Currently we have "maximum durability" settings configured now , we are planning to make it to "performance-optimized"  .  We have observed couple of time disc IO is very high sometimes . 

On Monday, January 20, 2020 at 5:54:38 AM UTC+5:30, Mark Waite wrote:
I'm not aware of an open bug in Jenkins related to being slow over NFS, though Jenkins is quite disc intensive and runs best with local disc drives.

You might refer to the following articles for more information:
  • Jenkins pipeline durability settings documentation ("Will Higher-Performance Durability Settings Help Me? Yes, if your Jenkins instance uses NFS, magnetic storage, runs many Pipelines at once, or shows high iowait.")
  • CloudBees NFS guide (multiple pages of tuning recommendations)


On Fri, Jan 17, 2020 at 5:42 AM Mahesh Wabale <[hidden email]> wrote:
Hi Team , 

We are using observing Jenkins slowness issues for jenkins version 2.138 in multi branch pipeline project . 
We have also used NFS 4.1 . 

After Jenkins restart it works for few days but at some stage it stuck and becomes unresponsive . As per observation only restart will solve this issue for next few days . We require frequent restart due to this issue . Have anyone observed similar issues , or  this is open bug in Jenkins ?    


--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/319f90ff-5410-4495-bb46-51d61745b3e8%40googlegroups.com.


--
Thanks!
Mark Waite

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/8MnTICGxQ-o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/8a711ba9-4433-4e62-9170-50e7f5c82bdd%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CABOY2ebxtXCGqDXKcfk7kzh0CgjvajTmyiOSni4EPF516gy9iA%40mail.gmail.com.


--
Thanks!
Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtFRJ-UPhCzrjbTU%2BZrMgoT-wrMSJyzUUxAE2YWEoyn0mg%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: About Jenkins slowness for version 2.138 in multi branch pipeline project

Mahesh Wabale
Thanks Marc , we will definitely update this settings and observe performance . 

For now below fix worked for me , not observing any issues recently with my application  . 

1.upgrade jdk8 version to recommended version (1.8.0_222-b10 and above) . 
2.Install monitoring plugin in Jenkins. 
3.Udate NFS settings with recommended settings (
RPCNFSDCOUNT=16 (default is 8
sunrpc.tcp_slot_table_entries = 128) . 
4.Auto clean up for sessions with scheduler (https://wiki.jenkins.io/display/JENKINS/Invalidate+Jenkins+HTTP+sessions
5.Add monitoring script to monitor active thread , JVM usage , thread deadlocks (https://wiki.jenkins.io/display/JENKINS/Monitoring+Scripts )
6.Free cache memory whenever it reaches to MAX memory limit , below steps help me to clean cache memory for jenkins master pod. 

I am able to free cache memory for Jenkins master pod docker container . As per docker image behaviour its taking resources from k8s node where it is deployed . 

You can verify memory usage by below commands from docker container . 

bash-4.4$ cat /sys/fs/cgroup/memory/memory.limit_in_bytes

bash-4.4$ cat /sys/fs/cgroup/memory/memory.max_usage_in_bytes

bash-4.4$ cat /sys/fs/cgroup/memory/memory.stat | grep cache

bash-4.4$ 


Solution , delete cache memory for k8s node where your Jenkins master pod docker container is running , there will be no downtime required for Jenkins service . 


Thanks ,
Mahesh

On Monday, January 20, 2020 at 8:07:33 PM UTC+5:30, Mark Waite wrote:
You might install the Jenkins Health Advisor by CloudBees and see if it has suggestions for your instance.  It will check for conditions that have been found to cause problems in other installations and report them to you in a daily e-mail message.

<a href="https://jenkins.io/blog/2019/11/22/jenkins-health-advisor-by-cloudbees/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fjenkins.io%2Fblog%2F2019%2F11%2F22%2Fjenkins-health-advisor-by-cloudbees%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGxgASY-EgMkIqjlhS7KRWrOnTK3w&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fjenkins.io%2Fblog%2F2019%2F11%2F22%2Fjenkins-health-advisor-by-cloudbees%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGxgASY-EgMkIqjlhS7KRWrOnTK3w&#39;;return true;">https://jenkins.io/blog/2019/11/22/jenkins-health-advisor-by-cloudbees/  

On Mon, Jan 20, 2020 at 6:49 AM Mahesh Wabale <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="-n-kKB9xDAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">mahes...@...> wrote:
We are using following JAVA_OPTS CONFIG in Jenkins deployment .

"name": "JAVA_OPTS",
                "value": "-XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+ParallelRefProcEnabled -XX:+UseStringDeduplication -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=20 -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:G1SummarizeRSetStatsPeriod=1 -Xms2048m -Xmx8192m"   , and 48 GB ram allocated for master Jenkins pod as per initial observation GC is working correctly its not going beyond 8 gb . But observing RAM USAGE of master pod is continusly increasing day by day . Some time it reaches to memory 48 GB seems like it taking memory for cache .How can we control ram memory size , or is it possible to free cache memory from Jenkins Docker image which can solve my problem .

On Mon, Jan 20, 2020, 3:45 PM Mahesh Wabale <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="-n-kKB9xDAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">mahes...@...> wrote:
Thanks Marc , 

We will check if it helps , Currently we have "maximum durability" settings configured now , we are planning to make it to "performance-optimized"  .  We have observed couple of time disc IO is very high sometimes . 

On Monday, January 20, 2020 at 5:54:38 AM UTC+5:30, Mark Waite wrote:
I'm not aware of an open bug in Jenkins related to being slow over NFS, though Jenkins is quite disc intensive and runs best with local disc drives.

You might refer to the following articles for more information:
  • Jenkins <a href="https://www.google.com/url?q=https%3A%2F%2Fjenkins.io%2Fdoc%2Fbook%2Fpipeline%2Fscaling-pipeline%2F%23will-higher-performance-durability-settings-help-me&amp;sa=D&amp;sntz=1&amp;usg=AFQjCNEtvgPugqm3XG5_99K4YYJi6vCf6Q" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fjenkins.io%2Fdoc%2Fbook%2Fpipeline%2Fscaling-pipeline%2F%23will-higher-performance-durability-settings-help-me\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEtvgPugqm3XG5_99K4YYJi6vCf6Q&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fjenkins.io%2Fdoc%2Fbook%2Fpipeline%2Fscaling-pipeline%2F%23will-higher-performance-durability-settings-help-me\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEtvgPugqm3XG5_99K4YYJi6vCf6Q&#39;;return true;">pipeline durability settings documentation ("Will Higher-Performance Durability Settings Help Me? Yes, if your Jenkins instance uses NFS, magnetic storage, runs many Pipelines at once, or shows high iowait.")
  • CloudBees <a href="https://support.cloudbees.com/hc/en-us/articles/217479948-NFS-Guide" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F217479948-NFS-Guide\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHa4LWaUx485443uzldQwSMF_2r3Q&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fsupport.cloudbees.com%2Fhc%2Fen-us%2Farticles%2F217479948-NFS-Guide\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHa4LWaUx485443uzldQwSMF_2r3Q&#39;;return true;">NFS guide (multiple pages of tuning recommendations)


On Fri, Jan 17, 2020 at 5:42 AM Mahesh Wabale <[hidden email]> wrote:
Hi Team , 

We are using observing Jenkins slowness issues for jenkins version 2.138 in multi branch pipeline project . 
We have also used NFS 4.1 . 

After Jenkins restart it works for few days but at some stage it stuck and becomes unresponsive . As per observation only restart will solve this issue for next few days . We require frequent restart due to this issue . Have anyone observed similar issues , or  this is open bug in Jenkins ?    


--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/319f90ff-5410-4495-bb46-51d61745b3e8%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/319f90ff-5410-4495-bb46-51d61745b3e8%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/319f90ff-5410-4495-bb46-51d61745b3e8%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/319f90ff-5410-4495-bb46-51d61745b3e8%40googlegroups.com.


--
Thanks!
Mark Waite

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit <a href="https://groups.google.com/d/topic/jenkinsci-users/8MnTICGxQ-o/unsubscribe" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/topic/jenkinsci-users/8MnTICGxQ-o/unsubscribe&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/topic/jenkinsci-users/8MnTICGxQ-o/unsubscribe&#39;;return true;">https://groups.google.com/d/topic/jenkinsci-users/8MnTICGxQ-o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to <a href="javascript:" rel="nofollow" target="_blank" gdf-obfuscated-mailto="-n-kKB9xDAAJ" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jenkins...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/8a711ba9-4433-4e62-9170-50e7f5c82bdd%40googlegroups.com?utm_medium=email&amp;utm_source=footer" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/8a711ba9-4433-4e62-9170-50e7f5c82bdd%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/8a711ba9-4433-4e62-9170-50e7f5c82bdd%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/8a711ba9-4433-4e62-9170-50e7f5c82bdd%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="-n-kKB9xDAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jenkins...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/CABOY2ebxtXCGqDXKcfk7kzh0CgjvajTmyiOSni4EPF516gy9iA%40mail.gmail.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/CABOY2ebxtXCGqDXKcfk7kzh0CgjvajTmyiOSni4EPF516gy9iA%40mail.gmail.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/CABOY2ebxtXCGqDXKcfk7kzh0CgjvajTmyiOSni4EPF516gy9iA%40mail.gmail.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/CABOY2ebxtXCGqDXKcfk7kzh0CgjvajTmyiOSni4EPF516gy9iA%40mail.gmail.com.


--
Thanks!
Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/6f369000-dc41-4e16-b2e9-416625c49ebd%40googlegroups.com.