Too many job in ci.jenkins.io

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Too many job in ci.jenkins.io

Rick
hi,

Does anyone notice that there're too many jobs in ci.jenkins.io. That cause many later PRs can't be built.

Regards,

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAMM7nTFVb0sCnpa9eAZiYDCwT3c6Ntt49VtG00fCkhXbjt7vTA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Too many job in ci.jenkins.io

Baptiste MATHUS
Each merge generally rebuilds all ongoing PRs. Given the core takes ~2.5h, necessitates 5 or 6 agents IIRC, and there are almost 100 open PRs, this causes quite a spike each time.

We are working on improving the situation to reduce the open PRs, basically review and merge or close.

HTH

Le ven. 8 févr. 2019 à 04:11, Rick <[hidden email]> a écrit :
hi,

Does anyone notice that there're too many jobs in ci.jenkins.io. That cause many later PRs can't be built.

Regards,

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAMM7nTFVb0sCnpa9eAZiYDCwT3c6Ntt49VtG00fCkhXbjt7vTA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANWgJS7hMRfx93ntJjONwq1T1amziyYjDV%2BkNA%3DgMUzge8aBcQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Too many job in ci.jenkins.io

Oliver Gondža-2
Just the other day, my path was crossed by a github bot[1] closing idle
PRs. Not that I would be delighted by the experience, but it might be
something to help us with the housekeeping.

[1] https://github.com/probot/stale

On 08/02/2019 08.40, Baptiste Mathus wrote:

> Each merge generally rebuilds all ongoing PRs. Given the core takes
> ~2.5h, necessitates 5 or 6 agents IIRC, and there are almost 100 open
> PRs, this causes quite a spike each time.
>
> We are working on improving the situation to reduce the open PRs,
> basically review and merge or close.
>
> HTH
>
> Le ven. 8 févr. 2019 à 04:11, Rick <[hidden email]
> <mailto:[hidden email]>> a écrit :
>
>     hi,
>
>     Does anyone notice that there're too many jobs in ci.jenkins.io
>     <http://ci.jenkins.io>. That cause many later PRs can't be built.
>
>     Regards,
>     Rick
>
>     --
>     https://github.com/LinuxSuRen
>
>     --
>     You received this message because you are subscribed to the Google
>     Groups "Jenkins Developers" group.
>     To unsubscribe from this group and stop receiving emails from it,
>     send an email to [hidden email]
>     <mailto:[hidden email]>.
>     To view this discussion on the web visit
>     https://groups.google.com/d/msgid/jenkinsci-dev/CAMM7nTFVb0sCnpa9eAZiYDCwT3c6Ntt49VtG00fCkhXbjt7vTA%40mail.gmail.com
>     <https://groups.google.com/d/msgid/jenkinsci-dev/CAMM7nTFVb0sCnpa9eAZiYDCwT3c6Ntt49VtG00fCkhXbjt7vTA%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>     For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to [hidden email]
> <mailto:[hidden email]>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-dev/CANWgJS7hMRfx93ntJjONwq1T1amziyYjDV%2BkNA%3DgMUzge8aBcQ%40mail.gmail.com 
> <https://groups.google.com/d/msgid/jenkinsci-dev/CANWgJS7hMRfx93ntJjONwq1T1amziyYjDV%2BkNA%3DgMUzge8aBcQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.


--
oliver

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/2a393fe5-c032-e885-e1e1-2adddf30780d%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Too many job in ci.jenkins.io

Jesse Glick-4
In reply to this post by Baptiste MATHUS
On Fri, Feb 8, 2019 at 2:40 AM Baptiste Mathus <[hidden email]> wrote:
> Each merge generally rebuilds all ongoing PRs.

It does not have to be this way.

https://issues.jenkins-ci.org/browse/INFRA-1633

Closing potentially valid, though slow-moving, PRs just to prevent
Jenkins from endlessly rebuilding them seems perverse when the actual
problem can be corrected directly.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr355LqhOH3rk3n0TnvXk%3D8ykb0vrsXjj_sWdHR3%3Dog7vg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Too many job in ci.jenkins.io

Kohsuke Kawaguchi
Administrator
This excessive rebuilding behaviour also causes lots of ops overhead by eating up disk space.

I think we need to change the current behaviour of ci.jenkins.io.

2019年2月8日(金) 6:25 Jesse Glick <[hidden email]>:
On Fri, Feb 8, 2019 at 2:40 AM Baptiste Mathus <[hidden email]> wrote:
> Each merge generally rebuilds all ongoing PRs.

It does not have to be this way.

https://issues.jenkins-ci.org/browse/INFRA-1633

Closing potentially valid, though slow-moving, PRs just to prevent
Jenkins from endlessly rebuilding them seems perverse when the actual
problem can be corrected directly.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr355LqhOH3rk3n0TnvXk%3D8ykb0vrsXjj_sWdHR3%3Dog7vg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


--
Kohsuke Kawaguchi

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAN4CQ4xJej8oHrzFnDeMTCz7siLHKdyJYRGiGhp%3Dv8NYy4exSw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Too many job in ci.jenkins.io

R. Tyler Croy
(replies inline)

On Fri, 08 Feb 2019, Kohsuke Kawaguchi wrote:

> This excessive rebuilding behaviour also causes lots of ops overhead by
> eating up disk space.


The disk space issues we experience are due to unaddressed scalability tickets
in Jenkins Pipeline, we would be seeing them regardless.


>
> I think we need to change the current behaviour of ci.jenkins.io.


I have been opposed to the change, but I don't consider my voice the final say
here.

My objection is that I fundamentally do not believe the green check mark on a
pull request is accurate unless the pull request is rebuilt and tested against
the latest HEAD of the target branch, even in cases of fast-forwards.


There are 88 pull requests open on Jenkins core right now, and I think it's
absolutely false to characterize anything that hasn't been updated in more than
three to six months as "valid."

I believe Daniel and Oleg have done a great job trying to keep things labeled
and organized, so we can tell that of that 88 we have 27 marked as needing more
review and 21 marked as works in progress, both going back for 2-3 years.


If core alone is causing a denial of service on ci.jenkins.io (it's not), and
we are unwilling to address the root cause, one easy solution is to add a
specific label for core only, which would be a fixed set of agents, allowing
other workloads to bypass the bottleneck.


If the maintainers of Jenkins core wish to change the settings for
core in ci.jenkins.io, I know at least one of them has the privileges to do so.


Personally, I like the probot suggestion from Oliver a lot more. Stale pull
requests are a boat anchor which drags down an open source community, and that
is IMHO the underlying problem worth fixing  here.



Cheers
--
GitHub:  https://github.com/rtyler

GPG Key ID: 0F2298A980EE31ACCA0A7825E5C92681BEF6CEA2

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/20190209011052.GR23906%40grape.brokenco.de.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Too many job in ci.jenkins.io

Gavin Mogan
I also like the idea of stalebot, I don't know much about the state of Jenkins core, but stale PRs are frustrating and it's not hard to comment when the warning goes out.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/57c20354-9beb-43c4-8000-2e122c0dd548%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Too many job in ci.jenkins.io

teilo
In reply to this post by R. Tyler Croy
I agree with Tyler, however...

Jenkins-x addresses the issue of merging something broken by a later commit by not allowing anyone to merge and merging by machine.  it uses Tide (part of prow) to do this and groups any prs together in a merge commit and retests them in a single batch before merging (in the happy case or breaking the merge into smaller units to identify the bad pr).

the branchsource plugin has all the info required and it could (if anyone had the time and inclination) be extended to merge like this.
then you don't need to rebuild everything all the time, just before a potential merge.


On Sat, 9 Feb 2019, 01:11 R. Tyler Croy <[hidden email] wrote:
(replies inline)

On Fri, 08 Feb 2019, Kohsuke Kawaguchi wrote:

> This excessive rebuilding behaviour also causes lots of ops overhead by
> eating up disk space.


The disk space issues we experience are due to unaddressed scalability tickets
in Jenkins Pipeline, we would be seeing them regardless.


>
> I think we need to change the current behaviour of ci.jenkins.io.


I have been opposed to the change, but I don't consider my voice the final say
here.

My objection is that I fundamentally do not believe the green check mark on a
pull request is accurate unless the pull request is rebuilt and tested against
the latest HEAD of the target branch, even in cases of fast-forwards.


There are 88 pull requests open on Jenkins core right now, and I think it's
absolutely false to characterize anything that hasn't been updated in more than
three to six months as "valid."

I believe Daniel and Oleg have done a great job trying to keep things labeled
and organized, so we can tell that of that 88 we have 27 marked as needing more
review and 21 marked as works in progress, both going back for 2-3 years.


If core alone is causing a denial of service on ci.jenkins.io (it's not), and
we are unwilling to address the root cause, one easy solution is to add a
specific label for core only, which would be a fixed set of agents, allowing
other workloads to bypass the bottleneck.


If the maintainers of Jenkins core wish to change the settings for
core in ci.jenkins.io, I know at least one of them has the privileges to do so.


Personally, I like the probot suggestion from Oliver a lot more. Stale pull
requests are a boat anchor which drags down an open source community, and that
is IMHO the underlying problem worth fixing  here.



Cheers
--
GitHub:  https://github.com/rtyler

GPG Key ID: 0F2298A980EE31ACCA0A7825E5C92681BEF6CEA2

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/20190209011052.GR23906%40grape.brokenco.de.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAPzq3pf0o8Aw8FYj%3DHMBe-PLa8WHKkFesVLayb0Hksmda3JqmA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Too many job in ci.jenkins.io

Jesse Glick-4
In reply to this post by R. Tyler Croy
On Fri, Feb 8, 2019 at 8:11 PM R. Tyler Croy <[hidden email]> wrote:
> If core alone is causing a denial of service on ci.jenkins.io (it's not)

But it is. I have personally had my work obstructed on multiple
occasions by core PR builds (though `git-plugin` has been a lesser
contributor).

> one easy solution is to add a specific label for core only

Though this would make for poor resource utilitization. Baptiste
already suggested using a Jenkins plugin to prioritize plugin builds
over core builds, which would be better than nothing, though it would
not address the wasted cost issue.

James Nord wrote:
> Jenkins-x addresses the issue…

Yes, Tide’s behavior is already mentioned in INFRA-1633. This is
certainly one option, though it would involve nontrivial new feature
development as well as a workflow change.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1q31HSHxdAKZxP7wSTwYHHXkRys8DzpN%2BEq14Y62cbMg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.