Support graceful deletion

[blackliner]

Hi, let me know if you want to add this. I played around with removing tags and then polling the runners until they are idle, to avoid killing jobs that are still running: https://github.com/vbem/multi-runners/compare/main...blackliner:multi-runners:patch-1?expand=1

[vbem]

Hi,

Thank you for the creative idea. This will be useful in scenarios where periodic reconstruction of runners is required.

I went through your code and have the following suggestions:

1. The runner ID can be fetch from local JSON file .runner under the runner home directory rather than remote API calls.
2. It's advised to add a --wait option which specifies the wait seconds (defaults to zero) rather than --graceful flag, and when --wait is not zero, the graceful shutdown process will be enabled.
3. The timeout loop can be optimized with (current_time - start_time) checking to eliminate the time consuming of curl and other codes.

However, this brings a life cycle management issue into current design:

1. Previously, the runner has only 2 status in the state machine - existing and non-existing. Thus the runner can be thought as stateless and the life-cycle-management can be easily be done via add and del.
2. But the "graceful shutdown" brings a new status into the state machine - decommissioning. Remove all labels make a runner from "existing" status into "decommissioning" status, which may introduce orphan runners, for example, for some reason ("busy" API call failed, the long-running script was terminated, etc.). These orphan runners need more effort to clean up.

So in order to ensure the effectiveness and simplicity of the state machine, currently we can't introduce this "graceful shutdown" feature.

I'll look for other more stateless approaches to help reset runners without killing running jobs.

[blackliner]

Thanks, good suggestions, didn't know about the .runner file, this is super helfpul!

I have this running in our infra, will give feedback how it goes and how often we run into stale runners. But honestly, as this is part of the terraform destroy procedure (for onprem runners) and part of the lifecylce hooks in AWS, eventually we will just delete/terminate the instances, and the runner will be cleaned up by github.com eventually.