Indirect node tainting using Conditions

[sftim]

Rather than directly setting a taint on a Node, manage conditions for the status of that Node and allow the control plane to then have another part of the controller rely on that condition information to taint the node.

To make this work, admission control could mutate incoming Nodes to have that condition imposed immediately, leaving that in place until there is evidence that the required DaemonSets are healthy and ready.

[sftim]

There's more information about this in https://github.com/kubernetes/design-proposals-archive/blob/main/scheduling/taint-node-by-condition.md and https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/382-taint-node-by-condition#readme

An example of software that sets conditions on a node: https://github.com/kubernetes/node-problem-detector

[sftim]

Setting conditions on Nodes also allows tools like draino to deschedule pods if the important DaemonSet becomes unhealthy.