Groovy pipeline to lock a resource on a node that is online

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Groovy pipeline to lock a resource on a node that is online

David Riemens
Hi, 
I have several jobs that require exlusive access to some resource of which a given agent node can have multiple, but limited to some maximum number which is less than nr of executors of the agent.

We solve this by using the Lockable Resources plugin. We define a resource per what is available on a node like: <res>_<nr>_<node>
and give each of those resources a generic label 'VirtualRes'.
In the job code we can then use:

     lock(label: 'VirtualRes', quantity: 1, variable: 'res_locked')
     {
     }

and within the code block, derive the nodename from the resource by string extraction, and assign the work to that node.

This normally works fine, until there is some issue with one of the nodes going offline.

As the LockableResources are defined on the master itself without any link to the node, the resources of any offline node
are still available. Consequently a lock will be granted, but the worker job will be queued if the node is not online.

I am looking for a way to lock a resource with additional check that the node is online.


Due to [JENKINS-50176] my code for the lock already has some surrounding 'retry()' loop with exception handling for
the res_locked being null. I could extend that (rather ugly) code to also check for the node being offline, but I am hoping
that there is a cleaner solution.
Anyone ??

thx, M.

PS: Obviously, it would be so much nicer if lockable resources could be allocated in the node definition; then they would be unavailable if the node goes offline. And also having multiple of the same resource would be the latest request.







--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/775169529.312360.1590567499764%40mail.yahoo.com.
Reply | Threaded
Open this post in threaded view
|

Re: Groovy pipeline to lock a resource on a node that is online

David Riemens
today I gave it a try to:
* add a check within the lock { ... } block to see if the derived node is online. If so, I throw an exception,
   and thus end up in the retry(). Unfortunately, upon the next try, I just get again the resource for the 
   node that was offline. After 3 retries, my script bails out with fatal error
* on top of the isNodeOnline() check I added also code to, if the node is offline, set the still locked resource
   to 'reserved'. Also this code works, and I see the resource getting reserved immediately.
   Unfortunately, the 'reserved' seems to not work for the jobs itself: in the next try the same resourtce is getting locked
   again (I think this should be considered a bug). In a next run of the same jobs, the resources is 'skipped'.

All in all, I am slowly losing hope of this getting to work wothout real effort by the plugin maintainers.

On Wednesday, May 27, 2020, 10:18:19 AM GMT+2, monger_39 <[hidden email]> wrote:


Hi, 
I have several jobs that require exlusive access to some resource of which a given agent node can have multiple, but limited to some maximum number which is less than nr of executors of the agent.

We solve this by using the Lockable Resources plugin. We define a resource per what is available on a node like: <res>_<nr>_<node>
and give each of those resources a generic label 'VirtualRes'.
In the job code we can then use:

     lock(label: 'VirtualRes', quantity: 1, variable: 'res_locked')
     {
     }

and within the code block, derive the nodename from the resource by string extraction, and assign the work to that node.

This normally works fine, until there is some issue with one of the nodes going offline.

As the LockableResources are defined on the master itself without any link to the node, the resources of any offline node
are still available. Consequently a lock will be granted, but the worker job will be queued if the node is not online.

I am looking for a way to lock a resource with additional check that the node is online.


Due to [JENKINS-50176] my code for the lock already has some surrounding 'retry()' loop with exception handling for
the res_locked being null. I could extend that (rather ugly) code to also check for the node being offline, but I am hoping
that there is a cleaner solution.
Anyone ??

thx, M.

PS: Obviously, it would be so much nicer if lockable resources could be allocated in the node definition; then they would be unavailable if the node goes offline. And also having multiple of the same resource would be the latest request.







--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1416975228.1802191.1601995319478%40mail.yahoo.com.