Floating IP support for NodeDNSRecordSet

[do-joe]

This PR adds to support to NodeDNSRecordSet allowing it to use the assigned Reserved IP Addresses in the DNS record it managed. This was done with the main changes:

1. The FloatingIPPool resource uses an annotation on the node to denote a Reserved IP Address assignment.
2. The nodematch controller has been updated so the any node updates that indicate a change in match status or a change in the annotation value are sent along to ip and nodends controllers so they are aware of changes in the annotation value.
3. Update the NodeDNSRecordSet to use the assigned Reserved IP address assignment as indicated in the node annotation.

There was also a small optimization to the ip controller's handling of node updates. Now when the FloatingIPPool is configured with a dnsRecordSet a DNS update is not made if the assignment of the Reserved IP Address does not change. Prior to this change any change in match status to a node triggered a DNS update regardless if the node was assigned a Reserved IP Address. So for example if I had 3 matching nodes, but only 2 IP addresses, and the node that did not have an Reserved IP Address was cordoned a DNS update would have been even though the IP addresses in the DNS record would not have changed.

The following scenarios were tested in validated in a live environment:

FloatingIPPool

• Annotation added when node is a match
• Annotation removed when node is a no longer a match and then added when match again
• Removes annotation when desiredIPs is reduced to less than matching nodes and adds back when increased to number of matching nodes
• Handles no longer matched and moving reserved IP to new node
• No DNS calls made when matching node without IP address changes match status
• Handles droplet being terminated
• Reserved IP and Annotations left untouched when FloatingIPPool resource removed (mirrors current behavior)

NodeDNSRecordSet

• Using External IP still the default
• Use value of annotation when annotation set as addressType
• DNS record removed then no longer a match and then added when match again
• DNS record is when annotation added
• DNS records updated when annotation is changed
• DNS record removed when annotation removed
• Handles cluster upgrade

[jcodybaker]

This looks great! I'd swear you've been slinging Go for a long while.

[jcodybaker]

This is a small nit--only a matter of efficiency and pattern. Feel free to leave this as is if changing it turns into a mess.

It'd be ideal to pull this from the nodeinformer in the node match controller. That will pull the latest node resource from our local memory, rather than fetching from the Kubernetes API. I made a similar call in my (closed) first attempt at this which might be relevant. In that case I was getting all nodes, but there is also a Get() option I believe

[do-joe]

Thanks for the review and feedback. I agree that we should get the data from cache where we can. It wasn't very straight forward as the node match controllers are properties of FloatingIPPools and the floating ip controller needs some way to find the node in the various node match controller instances.

I took the approach to just iterate through all the node match controllers. I assume that most deployments wont have many FloatingIPPools so this iteration isn't a big deal. I felt this was the more straight forward approach.

An alternative is to keep a NodeNameToPool mapping in the floating ip controller. This would involve passing in a NodeNameToPoolUpdater function when creating the FloatingIPPool and node match controller. I didn't feel like that was as straight forward, and ended up with having to worry about ensuring the mapping was always up to date. Not that its particularly hard, but it was more state to manage, where the other approach keeps all the state in the node match controller.

Happy to switch to use the mapping approach if you think it's best.

[jcodybaker]

Hey Joe, LGTM. I think you should be good to merge provide you've tried it and it works as expected. If you're not able to merge, tag me on Slack and I'll merge it.