What are reasons for high load on Jenkins master?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

What are reasons for high load on Jenkins master?

dokaspar
Hi,

We are running Jenkins with about 20 build agents on separate VMs (with up to 4 executors each). The master has 4 CPUs, so I would expect a load of about 4 when there's "a lot of work" to be done.

However, no job is ever scheduled on the master itself, and still we can observe very high load levels on the master when many jobs are running concurrently on the agents. For example, one of our test suites launches about 30 jobs (lasting 5-20min each) in a 1h time slot and at that moment the master reaches a load of up to 40, with all CPUs fully occupied! However, looking at the UI, all jobs clearly run on the agents, and the master simply shows as 'Idle'.

When I look at Jenkins' own monitoring, I see a huge amount of Computer.threadPoolForRemoting and similar threads with a summary stating Threads on master: Number = 415, Maximum = 511, Total started = 7,731,717.

Can anyone give some insight on what could be going on? What can be the reasons for high load on the master when all the jobs are running on agents?

Regards,
Dominik

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/104aaf8c-47b3-43ea-8bee-758e5c804c5en%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: What are reasons for high load on Jenkins master?

Kamil Dzierżanowski
Hi Dominik,

High load on the master might be caused by incorrectly implemented Jenkinsfiles which contain complex logic, thus inducing load on the master instead of the nodes. Furthermore, this load will not be visible in the node overview. I think this is the best point to start investigating this issue from, see this piece of documentation for explanation: https://www.jenkins.io/doc/book/pipeline/pipeline-best-practices/#making-sure-to-use-groovy-code-in-pipelines-as-glue

Best regards,
Kamil

On Friday, September 25, 2020 at 2:26:47 PM UTC+2 Dominik wrote:
Hi,

We are running Jenkins with about 20 build agents on separate VMs (with up to 4 executors each). The master has 4 CPUs, so I would expect a load of about 4 when there's "a lot of work" to be done.

However, no job is ever scheduled on the master itself, and still we can observe very high load levels on the master when many jobs are running concurrently on the agents. For example, one of our test suites launches about 30 jobs (lasting 5-20min each) in a 1h time slot and at that moment the master reaches a load of up to 40, with all CPUs fully occupied! However, looking at the UI, all jobs clearly run on the agents, and the master simply shows as 'Idle'.

When I look at Jenkins' own monitoring, I see a huge amount of Computer.threadPoolForRemoting and similar threads with a summary stating Threads on master: Number = 415, Maximum = 511, Total started = 7,731,717.

Can anyone give some insight on what could be going on? What can be the reasons for high load on the master when all the jobs are running on agents?

Regards,
Dominik

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1ff872c7-8b83-43bc-a339-7f07bfdce6ffn%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: What are reasons for high load on Jenkins master?

dokaspar
Hi Kamil,

Thanks for the link. Although we do use pipelines for many projects, the set of test jobs I mentioned are plain old 'freestyle' jobs.
They include one or multiple Maven build steps which run some more or less heavy unit and integration tests (which all happens on agents anyway).
When looking at top/htop on the server during those tests, it's really just the Jenkins main process rotating at 380-400% CPU usage (of 4 CPUs).

Regards,
Dominik

On Friday, September 25, 2020 at 3:28:23 PM UTC+2 [hidden email] wrote:
Hi Dominik,

High load on the master might be caused by incorrectly implemented Jenkinsfiles which contain complex logic, thus inducing load on the master instead of the nodes. Furthermore, this load will not be visible in the node overview. I think this is the best point to start investigating this issue from, see this piece of documentation for explanation: https://www.jenkins.io/doc/book/pipeline/pipeline-best-practices/#making-sure-to-use-groovy-code-in-pipelines-as-glue

Best regards,
Kamil

On Friday, September 25, 2020 at 2:26:47 PM UTC+2 Dominik wrote:
Hi,

We are running Jenkins with about 20 build agents on separate VMs (with up to 4 executors each). The master has 4 CPUs, so I would expect a load of about 4 when there's "a lot of work" to be done.

However, no job is ever scheduled on the master itself, and still we can observe very high load levels on the master when many jobs are running concurrently on the agents. For example, one of our test suites launches about 30 jobs (lasting 5-20min each) in a 1h time slot and at that moment the master reaches a load of up to 40, with all CPUs fully occupied! However, looking at the UI, all jobs clearly run on the agents, and the master simply shows as 'Idle'.

When I look at Jenkins' own monitoring, I see a huge amount of Computer.threadPoolForRemoting and similar threads with a summary stating Threads on master: Number = 415, Maximum = 511, Total started = 7,731,717.

Can anyone give some insight on what could be going on? What can be the reasons for high load on the master when all the jobs are running on agents?

Regards,
Dominik

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/dd63521c-1aa6-4d1f-94eb-d6a573908286n%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: What are reasons for high load on Jenkins master?

dokaspar
We upgraded our Jenkins instance yesterday (from version 2.190.1 to 2.249.1) and the heavy load on the master "magically disappeared".
The exact same test suite now runs (and 'spams' all the agents), but the load on master stays below a value of 3 (with 4 CPUs).

On Friday, September 25, 2020 at 5:05:11 PM UTC+2 Dominik wrote:
Hi Kamil,

Thanks for the link. Although we do use pipelines for many projects, the set of test jobs I mentioned are plain old 'freestyle' jobs.
They include one or multiple Maven build steps which run some more or less heavy unit and integration tests (which all happens on agents anyway).
When looking at top/htop on the server during those tests, it's really just the Jenkins main process rotating at 380-400% CPU usage (of 4 CPUs).

Regards,
Dominik

On Friday, September 25, 2020 at 3:28:23 PM UTC+2 [hidden email] wrote:
Hi Dominik,

High load on the master might be caused by incorrectly implemented Jenkinsfiles which contain complex logic, thus inducing load on the master instead of the nodes. Furthermore, this load will not be visible in the node overview. I think this is the best point to start investigating this issue from, see this piece of documentation for explanation: https://www.jenkins.io/doc/book/pipeline/pipeline-best-practices/#making-sure-to-use-groovy-code-in-pipelines-as-glue

Best regards,
Kamil

On Friday, September 25, 2020 at 2:26:47 PM UTC+2 Dominik wrote:
Hi,

We are running Jenkins with about 20 build agents on separate VMs (with up to 4 executors each). The master has 4 CPUs, so I would expect a load of about 4 when there's "a lot of work" to be done.

However, no job is ever scheduled on the master itself, and still we can observe very high load levels on the master when many jobs are running concurrently on the agents. For example, one of our test suites launches about 30 jobs (lasting 5-20min each) in a 1h time slot and at that moment the master reaches a load of up to 40, with all CPUs fully occupied! However, looking at the UI, all jobs clearly run on the agents, and the master simply shows as 'Idle'.

When I look at Jenkins' own monitoring, I see a huge amount of Computer.threadPoolForRemoting and similar threads with a summary stating Threads on master: Number = 415, Maximum = 511, Total started = 7,731,717.

Can anyone give some insight on what could be going on? What can be the reasons for high load on the master when all the jobs are running on agents?

Regards,
Dominik

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/f170a0e9-2e47-451a-9483-d2fd363aced9n%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: What are reasons for high load on Jenkins master?

Ivan Fernandez Calvo
remoting has tons of improvements between those two versions 3.33 vs 4.5, if you agents ar JNLP the new default protocols is JNLP 4 that is it's better than JNLP 3 IIRC it has an important performance issue 

El jueves, 1 de octubre de 2020 a las 12:33:53 UTC+2, Dominik escribió:
We upgraded our Jenkins instance yesterday (from version 2.190.1 to 2.249.1) and the heavy load on the master "magically disappeared".
The exact same test suite now runs (and 'spams' all the agents), but the load on master stays below a value of 3 (with 4 CPUs).

On Friday, September 25, 2020 at 5:05:11 PM UTC+2 Dominik wrote:
Hi Kamil,

Thanks for the link. Although we do use pipelines for many projects, the set of test jobs I mentioned are plain old 'freestyle' jobs.
They include one or multiple Maven build steps which run some more or less heavy unit and integration tests (which all happens on agents anyway).
When looking at top/htop on the server during those tests, it's really just the Jenkins main process rotating at 380-400% CPU usage (of 4 CPUs).

Regards,
Dominik

On Friday, September 25, 2020 at 3:28:23 PM UTC+2 [hidden email] wrote:
Hi Dominik,

High load on the master might be caused by incorrectly implemented Jenkinsfiles which contain complex logic, thus inducing load on the master instead of the nodes. Furthermore, this load will not be visible in the node overview. I think this is the best point to start investigating this issue from, see this piece of documentation for explanation: https://www.jenkins.io/doc/book/pipeline/pipeline-best-practices/#making-sure-to-use-groovy-code-in-pipelines-as-glue

Best regards,
Kamil

On Friday, September 25, 2020 at 2:26:47 PM UTC+2 Dominik wrote:
Hi,

We are running Jenkins with about 20 build agents on separate VMs (with up to 4 executors each). The master has 4 CPUs, so I would expect a load of about 4 when there's "a lot of work" to be done.

However, no job is ever scheduled on the master itself, and still we can observe very high load levels on the master when many jobs are running concurrently on the agents. For example, one of our test suites launches about 30 jobs (lasting 5-20min each) in a 1h time slot and at that moment the master reaches a load of up to 40, with all CPUs fully occupied! However, looking at the UI, all jobs clearly run on the agents, and the master simply shows as 'Idle'.

When I look at Jenkins' own monitoring, I see a huge amount of Computer.threadPoolForRemoting and similar threads with a summary stating Threads on master: Number = 415, Maximum = 511, Total started = 7,731,717.

Can anyone give some insight on what could be going on? What can be the reasons for high load on the master when all the jobs are running on agents?

Regards,
Dominik

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/8b729806-0cf6-4fdd-a598-236d623220f1n%40googlegroups.com.