Help with blocking job execution per node

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Help with blocking job execution per node

Martin Weber-2
Hi list,

my plugins adds a build wrapper that starts and stop a daemon (similar to xvfb
[1] plugin).
Only one daemon instance can be run at a time on a single node, so I want to
block job execution automatically, without forcing my users to install and
configure Throttle Concurrent Builds plugin [2].

I tried to implement a QueueTaskDispatcher that blocks if a job is running
that also has my plugin wrapper enabled. It works mostly, but if I have 3 jobs
with my wrapper enabled in the queue and 2 executors on the node, *both* of
the 2 blocked jobs start execution after the first job has finished.

Can anyone point me to the right direction to solve this (allow only one
running job with my wrapper enabled per node)?

TIA,
        Martin

[1] https://plugins.jenkins.io/xvfb
[2] https://plugins.jenkins.io/throttle-concurrents

--
Cd wrttn wtht vwls s mch trsr.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/6641805.XLH7yz7z48%40linux.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Help with blocking job execution per node

stephenconnolly

On Thu 20 Jul 2017 at 19:13, Martin Weber <[hidden email]> wrote:
Hi list,

my plugins adds a build wrapper that starts and stop a daemon (similar to xvfb
[1] plugin).
Only one daemon instance can be run at a time on a single node, so I want to
block job execution automatically, without forcing my users to install and
configure Throttle Concurrent Builds plugin [2].

I tried to implement a QueueTaskDispatcher that blocks if a job is running
that also has my plugin wrapper enabled. It works mostly, but if I have 3 jobs
with my wrapper enabled in the queue and 2 executors on the node, *both* of
the 2 blocked jobs start execution after the first job has finished.

Yep because both jobs transition at the same time.

What you need to do is keep track of if you let a job through and wait say 100ms before letting another go through (but round-robin which one you pick)

A cheap way might be to be use a 5% probability of letting a job through when none are running... 

That gives you a 1:400 chance of two jobs in the same "round" and most jobs should start in under 1 second if they can while avoiding having to deal with concurrency issues so much.

If 1:400 is too high, you'll need to bite the bullet and sort out the concurrency to determine when a round starts and then only allow one per round to be unblocked (pick a different one each round)


Can anyone point me to the right direction to solve this (allow only one
running job with my wrapper enabled per node)?

TIA,
        Martin

[1] https://plugins.jenkins.io/xvfb
[2] https://plugins.jenkins.io/throttle-concurrents

--
Cd wrttn wtht vwls s mch trsr.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/6641805.XLH7yz7z48%40linux.
For more options, visit https://groups.google.com/d/optout.
--
Sent from my phone

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BnPnMy7tUgQxvyHPXx2nYZQ9DZ6YN3Yd4DDHZYbAiHz%2BACT%3DA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Help with blocking job execution per node

Markus Winter
Another thing to consider:
Someone might have configured 2 agents on the same machine.
You could also make the wrapper block and wait until the other job is finished if the daemon is already running. Maybe combine with the proposal from Stephen.

Another question, is this daemon specific to the job that is executed or could the deamon serve both jobs?


On 20.07.2017 23:01, Stephen Connolly wrote:

On Thu 20 Jul 2017 at 19:13, Martin Weber <[hidden email]> wrote:
Hi list,

my plugins adds a build wrapper that starts and stop a daemon (similar to xvfb
[1] plugin).
Only one daemon instance can be run at a time on a single node, so I want to
block job execution automatically, without forcing my users to install and
configure Throttle Concurrent Builds plugin [2].

I tried to implement a QueueTaskDispatcher that blocks if a job is running
that also has my plugin wrapper enabled. It works mostly, but if I have 3 jobs
with my wrapper enabled in the queue and 2 executors on the node, *both* of
the 2 blocked jobs start execution after the first job has finished.

Yep because both jobs transition at the same time.

What you need to do is keep track of if you let a job through and wait say 100ms before letting another go through (but round-robin which one you pick)

A cheap way might be to be use a 5% probability of letting a job through when none are running... 

That gives you a 1:400 chance of two jobs in the same "round" and most jobs should start in under 1 second if they can while avoiding having to deal with concurrency issues so much.

If 1:400 is too high, you'll need to bite the bullet and sort out the concurrency to determine when a round starts and then only allow one per round to be unblocked (pick a different one each round)


Can anyone point me to the right direction to solve this (allow only one
running job with my wrapper enabled per node)?

TIA,
        Martin

[1] https://plugins.jenkins.io/xvfb
[2] https://plugins.jenkins.io/throttle-concurrents

--
Cd wrttn wtht vwls s mch trsr.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/6641805.XLH7yz7z48%40linux.
For more options, visit https://groups.google.com/d/optout.
--
Sent from my phone
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BnPnMy7tUgQxvyHPXx2nYZQ9DZ6YN3Yd4DDHZYbAiHz%2BACT%3DA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/46c83c19-2e5e-8c2b-4eff-18efe2186d68%40gmx.de.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Help with blocking job execution per node

Martin Weber-2
In reply to this post by stephenconnolly
Am Donnerstag, 20. Juli 2017, 21:01:13 CEST schrieb Stephen Connolly:
> On Thu 20 Jul 2017 at 19:13, Martin Weber <[hidden email]> wrote:


> If 1:400 is too high, you'll need to bite the bullet and sort out the
> concurrency to determine when a round starts and then only allow one per
> round to be unblocked (pick a different one each round)

How can I be informed when a round starts? Implement a QueueListener?

--
Cd wrttn wtht vwls s mch trsr.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/6949603.eISL3IcJ9R%40linux.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Help with blocking job execution per node

stephenconnolly
May need to extend core to support advertising it.

If it's more than 100ms it is likely a new round

On Fri 21 Jul 2017 at 19:56, Martin Weber <[hidden email]> wrote:
Am Donnerstag, 20. Juli 2017, 21:01:13 CEST schrieb Stephen Connolly:
> On Thu 20 Jul 2017 at 19:13, Martin Weber <[hidden email]> wrote:


> If 1:400 is too high, you'll need to bite the bullet and sort out the
> concurrency to determine when a round starts and then only allow one per
> round to be unblocked (pick a different one each round)

How can I be informed when a round starts? Implement a QueueListener?

--
Cd wrttn wtht vwls s mch trsr.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/6949603.eISL3IcJ9R%40linux.
For more options, visit https://groups.google.com/d/optout.
--
Sent from my phone

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BnPnMxpnmDe_xGwo8UAS1%2BvsCKaSMR3QQvu%2BMUifvj3t-QqJQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: Help with blocking job execution per node

Martin Weber-2
Am Freitag, 21. Juli 2017, 20:01:41 CEST schrieb Stephen Connolly:
> May need to extend core to support advertising it.
>
> If it's more than 100ms it is likely a new round

OK, thanks You!
Have a nice weekend. (I'm at UTC +2)

Martin

--
Cd wrttn wtht vwls s mch trsr.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/1816682.ttlQY7hRkx%40linux.
For more options, visit https://groups.google.com/d/optout.