Quantcast

Intermittent 504s when browser is accessing ajaxBuildQueue and ajaxExecutors endpoints

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Intermittent 504s when browser is accessing ajaxBuildQueue and ajaxExecutors endpoints

Prasad Velidi

Hello,


Recently we have been seeing 504 errors on the ELB status logs (AWS) for Jenkins. When investigated, we came to know that, to keep the page on browser in sync with latest information, the browser keeps on polling data using ajaxBuildQueue and ajaxExecutors endpoints. When the logs for these requests and their responses are observed, we saw that intermittently 504 errors are popping up. This tripped our thresholds for 5xx errors. We want to know how to eradicate the 504s as it might also impacts performance if we start to see more of these errors.


When observed, we found 26 of the 504s on the logs over a period of 15 mins. While many of the requests land on ELB one after the other, only one out of 10 subsequent requests to the mentioned endpoints produced the 504s.

 

Not sure if this is related to performance tuning where in we tune the thresholds in configuration somewhere or is it something to do with jenkins itself. Any suggestions are highly appreciated.


Regards,
Prasad Velidi

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/2593f9a0-89ab-420e-9016-2c83dede6da1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Intermittent 504s when browser is accessing ajaxBuildQueue and ajaxExecutors endpoints

teilo
This could be a number of things without any data to correlate

1. Garbage collection performing stop the world collection (Google 'Sam van oort jenkins garbage collection 
cloudbees')
2. Not enough http handlers in jenkins and someone is making http requests that are not quick so you exhaust the thread pool (look at' java - jar jenkins.war - - help' also look in a support bundle for slow requests)
3. Jenkins being starved of underlying resources (cpu,  memory paging disk io..)

There's probably plenty of others as well but this are possibly the biggest 3 off the top of my head. 

On 18 May 2017 8:23 am, Prasad Velidi <[hidden email]> wrote:

Hello,


Recently we have been seeing 504 errors on the ELB status logs (AWS) for Jenkins. When investigated, we came to know that, to keep the page on browser in sync with latest information, the browser keeps on polling data using ajaxBuildQueue and ajaxExecutors endpoints. When the logs for these requests and their responses are observed, we saw that intermittently 504 errors are popping up. This tripped our thresholds for 5xx errors. We want to know how to eradicate the 504s as it might also impacts performance if we start to see more of these errors.


When observed, we found 26 of the 504s on the logs over a period of 15 mins. While many of the requests land on ELB one after the other, only one out of 10 subsequent requests to the mentioned endpoints produced the 504s.

 

Not sure if this is related to performance tuning where in we tune the thresholds in configuration somewhere or is it something to do with jenkins itself. Any suggestions are highly appreciated.


Regards,
Prasad Velidi

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/2593f9a0-89ab-420e-9016-2c83dede6da1%40googlegroups.com?utm_medium&#61;email&amp;utm_source&#61;footer">https://groups.google.com/d/msgid/jenkinsci-users/2593f9a0-89ab-420e-9016-2c83dede6da1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/a79acd6e-db63-4411-a88c-e76bfbdaeaec%40email.android.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Intermittent 504s when browser is accessing ajaxBuildQueue and ajaxExecutors endpoints

Prasad Velidi
Thank you for your valuable inputs. You have covered more things than what i had in mind. Appreciate your help.

On Friday, 19 May 2017 00:23:12 UTC+5:30, James Nord wrote:
This could be a number of things without any data to correlate

1. Garbage collection performing stop the world collection (Google 'Sam van oort jenkins garbage collection 
cloudbees')
2. Not enough http handlers in jenkins and someone is making http requests that are not quick so you exhaust the thread pool (look at' java - jar jenkins.war - - help' also look in a support bundle for slow requests)
3. Jenkins being starved of underlying resources (cpu,  memory paging disk io..)

There's probably plenty of others as well but this are possibly the biggest 3 off the top of my head. 

On 18 May 2017 8:23 am, Prasad Velidi <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="7vONbTfpAQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">prasad...@...> wrote:

Hello,


Recently we have been seeing 504 errors on the ELB status logs (AWS) for Jenkins. When investigated, we came to know that, to keep the page on browser in sync with latest information, the browser keeps on polling data using ajaxBuildQueue and ajaxExecutors endpoints. When the logs for these requests and their responses are observed, we saw that intermittently 504 errors are popping up. This tripped our thresholds for 5xx errors. We want to know how to eradicate the 504s as it might also impacts performance if we start to see more of these errors.


When observed, we found 26 of the 504s on the logs over a period of 15 mins. While many of the requests land on ELB one after the other, only one out of 10 subsequent requests to the mentioned endpoints produced the 504s.

 

Not sure if this is related to performance tuning where in we tune the thresholds in configuration somewhere or is it something to do with jenkins itself. Any suggestions are highly appreciated.


Regards,
Prasad Velidi

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="7vONbTfpAQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/jenkinsci-users/2593f9a0-89ab-420e-9016-2c83dede6da1%40googlegroups.com?utm_medium=email&amp;utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/2593f9a0-89ab-420e-9016-2c83dede6da1%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/msgid/jenkinsci-users/2593f9a0-89ab-420e-9016-2c83dede6da1%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter&#39;;return true;">https://groups.google.com/d/msgid/jenkinsci-users/2593f9a0-89ab-420e-9016-2c83dede6da1%40googlegroups.com.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/5f294058-d0ef-43af-88f9-3219cf3aa3db%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Loading...