Allowing to take a public IP address out of the pool of existing primary IP's

[jerve]

When the tool creates a new cloud server in Hetzner (for example when adding a node), it gets a random public IP assigned from Hetzner. In our case, this is often problematic because a lot of those IP's are blacklisted somewhere. The steps we perform then manually are:

1. Power off the server
2. unassign the ip
3. under "primary ips" tab in Hetzner, create some new IP's until we get a non-blacklisted one
4. assign the non-blacklisted IP to the server
5. power on the server

It would be a lot easier for us if we could just specify that the tool should pick a random unassigned IP from the pool of primary IPs. In that case, we can predefine a fixed set of non-blacklisted primary IPs and not worry about it anymore when creating a new node. When no free IP's are left in the pool, the tool should ideally fail with an error.

What are you guys thinking about this? are there more elegant solutions to this?
I contacted Hetzner, there is nothing they can do about the IP blacklist. Also, using egress gateway is a level of complexity I would not like to add to my cluster.

[vitobotta]

Hi there,

I'd be happy to look into this at some point, but it won't be a high priority compared to other tasks I have in mind. This is the first time someone has requested this feature in three years, so I prioritize features and improvements that benefit the company I work for and most other users.

Technically, it wouldn't be complex to implement if you just wanted to specify a list of IP addresses to choose from. However, there are a few considerations to keep in mind. First, creating a cluster would involve more API calls. Second, there's the issue of exhausting the IP pool and needing to create additional servers. Third, this feature wouldn't work with autoscaling because it's not possible to specify the ID of the IP to use with the cluster autoscaler.

I can look into this further, but I can't guarantee when it will be implemented or when I'll be able to provide an update.

On a related note, I've only encountered this issue a few times when nodes need to pull images from registries. However, k3s can set up an embedded registry mirror in k3s, if enabled in the configuration file. This means that as long as the image can be downloaded by at least one node, other nodes or peers can take the image from that node instead of pulling it from the registry again. Have you tried this?

If the issue isn't related to pulling images, another alternative could be using a node with a floating IP for egress, as documented in https://vitobotta.github.io/hetzner-k3s/Floating_IP_egress/.

[jerve]

I read about the ingress. What is unclear to me is:
can I just change the setting of an existing cluster from flannel to cilium -> will that work? could not find that in the docs

[jerve]

Also, I think the issue with the blacklisted IPs might be underestimated. We experience issues with:

• sending logs/traces to datadog
• sending e-mails -> no error on sending but e-mails do not get delivered or end up in spam folder
• pulling images
• ...

[vitobotta]

I believe I've heard of someone who successfully swapped a CNI for another in an existing cluster. However, I've never attempted this myself, and even if it's feasible, it would undoubtedly cause a complete outage until all the nodes in the cluster are restarted and so on. I think it would be too risky, but again, I've never had to do something like this.