Feature Request: Graceful node draining before termination

[JoelSpeed]

It doesn't look to me as if the code uses any graceful draining of the Kubernetes nodes at present?

It would be good to add this into the node termination procedure so that pods are moved off of the instance before termination on AWS.

I'm thinking about cases when terminating a node would disrupt a PDB, if there's no budget then ideally you wouldn't terminate the node until there is budget

[Joseph-Irving]

We could add this as a flag --drain-nodes or something. You could get into a weird state where it's draining the node it's on so you would have to work around that.

It was intentional to have it aggressively terminate nodes as it will expose single points of failure and kind of simulate a node breaking.

[Joseph-Irving]

So we could get around the draining problem by just filtering out the surtr pod by label when draining the node so that should work actually