Worker nodes created with the root password access

[aidas-emersoft]

I'm new to using this tool, but the first impression is great!

The cluster I created has one master and two worker nodes. Master VM can be SSH into using the key and port provided in cluster_config.yaml file when creating a cluster. However, once worker nodes are created, Hetzner sends an email with the root password. Isn't it insecure, and why would't the same SSH key used for master be injected into the worker nodes during cluster creation?

Thank you!

[vitobotta]

Hi, glad you like it :)

Did you perhaps already have the same key added to your Hetzner project? Usually, hetzner-k3s uploads the SSH key you specify in the config file to your Hetzner project and names it after the cluster. However, if a key with the same fingerprint already exists in your Hetzner project, hetzner-k3s can't upload the key because two keys with identical fingerprints but different names cannot coexist in the same project. So when new nodes are created, they reference a key by the cluster name that doesn't exist in this scenario. As a result, Hetzner sends you emails with the root passwords. This doesn't occur when there are no keys in your Hetzner project that match the fingerprint of the key specified in the config file. In such cases, Hetzner can inject the key directly into the nodes, eliminating the need for a root password and an email.

To sum up, if you had already added the same key to your project, that's why you're receiving those emails. You can rename the existing key using the same name as your cluster, so new nodes created by hetzner-k3s will have the key correctly injected.

The only weird thing you mentioned is not getting an email for the master. I'm wondering if this could be a delivery issue with that specific email or something similar, because the key configuration is the same across all nodes in your cluster. By the way, can you also SSH manually into the worker nodes?

To confirm whether my explanation fits your situation: if there's indeed a key in your Hetzner project with the same fingerprint as specified in the config file, you can rename that key to match your cluster's name or simply delete it. This way, Hetzner-k3s will recreate the key with the expected name for you.

Next, try increasing your node pool by one and then run the create command again to add a new node. See if you receive an email from Hetzner for the newly created node.

[aidas-emersoft]

Hi @vitobotta, thank you for such a detailed explanation! I had a thought this might be the case.

I had indeed previously added the same key to the same project.

I can SSH to both master and worker nodes, but master node seems to have been provisioned with the SSH key in discussion (strange), whereas the worker nodes require the root password.

I can not simple delete the existing key, as I reference to it by it's name in Terraform, and might need to spin more VMs in the same project later. And yes, I know it's recommended to have a separate project per cluster, but my intention is to add an existing database VM to the newly created private network, so that my k8 pods can connect to the existing DB.

It seems the best way to solve this particular issue is to have a set of dedicated SSH keys for this and future clusters.

[vitobotta]

Sure, the simplest solution for now is to use a different key. I'll add it to my list of tasks to make it possible to specify an existing key, just like we do for the private network. For now, using a different key should work fine.

[aidas-emersoft]

Definitely works as expected with a new set of keys. Thanks!

[TobiasBrandtBruker]

Is this somewhere stated in the docs? This issue has driven me crazy over the last few days and since I am not the owner of the cluster I did not receive any mails. Glad, i finally found this thread! But I would recommend to add this to the troubleshooting section.

[vitobotta]

I have added this section to https://vitobotta.github.io/hetzner-k3s/Creating_a_cluster/:

If you already have SSH keys in your Hetzner project, it's best to use a different key for your cluster—unless there's already a key with the same name and fingerprint as the one in your config file. Hetzner doesn't allow two keys with the same fingerprint in one project. So if you've already added the key from your config but under a different name, Hetzner won't let you add it again. In that case, hetzner-k3s will skip creating the key and won't inject any SSH key into the cluster nodes. Without an SSH key, Hetzner sets up the nodes with password login instead. That means you'll get an email for each node with its root password. Managing several nodes this way can get tricky. To avoid this, just use a new keypair for your cluster—unless the project already has a key that matches both the name and fingerprint in your config.