Autoscale or initial setup is taking too long

[ArgirisGkogkidis]

So, I do have a bastion machine which I also use as NAT for the nodes.

The cluster-config.yml has this:

additional_pre_k3s_commands:
- echo "auto enp7s0" >> /etc/network/interfaces
- echo "iface enp7s0 inet dhcp" >> /etc/network/interfaces
- echo "    post-up ip route add default via 10.0.0.1"  >> /etc/network/interfaces
- echo "dns-nameservers 8.8.8.8 1.1.1.1" >> /etc/network/interfaces
- echo "[Resolve]" > /etc/systemd/resolved.conf
- echo "DNS=185.12.64.2 185.12.64.1" >> /etc/systemd/resolved.conf
- echo "FallbackDNS=8.8.8.8" >> /etc/systemd/resolved.conf
- ifdown enp7s0 && ifup enp7s0
- systemctl start systemd-resolved
- systemctl enable systemd-resolved

additional_post_k3s_commands:
  - 'export DEBIAN_FRONTEND=noninteractive; apt-get update -y && apt-get install -y fail2ban ca-certificates curl gnupg lsb-release unattended-upgrades systemd-resolved systemd-timesyncd'
  - 'dpkg-reconfigure --priority=low unattended-upgrades'
  - 'sed -i "s#^//\\?Unattended-Upgrade::Automatic-Reboot .*#Unattended-Upgrade::Automatic-Reboot \\"true\\";#" /etc/apt/apt.conf.d/50unattended-upgrades'
  - 'grep -q "Unattended-Upgrade::Automatic-Reboot-Time" /etc/apt/apt.conf.d/50unattended-upgrades || echo "Unattended-Upgrade::Automatic-Reboot-Time \\"04:00\\";" >> /etc/apt/apt.conf.d/50unattended-upgrades'
  - 'systemctl enable --now unattended-upgrades fail2ban systemd-resolved systemd-timesyncd'
  - apt update
  - apt upgrade -y

If I do SSH to the node, I can normally ping Google, quay.io, or anything. I do not really understand the following,

ubectl -n kube-system get pods -o wide --field-selector spec.nodeName=k3s-small-scalled-40a701cf91362e86
NAME                 READY   STATUS              RESTARTS   AGE   IP         NODE                                        NOMINATED NODE   READINESS GATES
cilium-envoy-fkgrj   0/1     ContainerCreating   0          10m   10.0.0.7  k3s-small-scalled-40a701cf91362e86   <none>           <none>
cilium-nwlbm         0/1     Init:0/6            0          10m   10.0.0.7  k3s-small-scalled-40a701cf91362e86   <none>           <none>

and then:

  Normal   Scheduled               11m   default-scheduler  Successfully assigned kube-system/cilium-nwlbm to k3s-small-scalled-40a701cf91362e86
  Warning  FailedCreatePodSandBox  11m   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox "658e2a8b95976061027e1e1d00bf8bcfeb16a62c07a9c3ab50a48086737db19f": failed to get sandbox image "rancher/mirrored-pause:3.6": failed to pull image "rancher/mirrored-pause:3.6": failed to pull and unpack image "docker.io/rancher/mirrored-pause:3.6": failed to copy: httpReadSeeker: failed open: failed to do request: Get "https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/sha256:74c4244427b7312c5b901fe0f67cbc53683d06f4f24c6faee65d4182bf0fa893": dial tcp: lookup registry-1.docker.io: Try again
  Warning  FailedCreatePodSandBox  11m   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox "8437599a7ca80ab8c2b8095bd78556055bfe48630115d3fa7d2d6a43efecb097": failed to get sandbox image "rancher/mirrored-pause:3.6": failed to pull image "rancher/mirrored-pause:3.6": failed to pull and unpack image "docker.io/rancher/mirrored-pause:3.6": failed to resolve reference "docker.io/rancher/mirrored-pause:3.6": failed to do request: Head "https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.6": dial tcp: lookup registry-1.docker.io: Try again
  Normal   Pulling                 10m   kubelet            Pulling image "quay.io/cilium/cilium:v1.17.2@sha256:3c4c9932b5d8368619cb922a497ff2ebc8def5f41c18e410bcc84025fcd385b1"

Is this network configuration issue or with the versions? I was expecting that the cluster init or scale should be fast not 15minutes ±

[ArgirisGkogkidis]

So it seems that something was broken with the NAT gateway. I deleted everything and redone cluster + NAT. now the deployment is faster and autoscale takes like 3-5 minutes to spin up new nodes.

[vitobotta]

Glad you sorted it out. would you mind describing your configuration in detail? It may be worth adding to the docs.

[ArgirisGkogkidis]

Of course, I plan to share my full experience later, but here is what happened so far.

Step 1

Create a NAT server.
As is already in the docs NAT Server, just take the cloud-init for the server side.
In my case, the same machine is also used as a Bastion/VPN, because I want to hide the cluster nodes in my setup, thus some extra changes happened there, but do not affect the NAT.

Super important Do not forget to create the route 0.0.0.0/0 to point to your NAT server (example 10.0.0.2)

Step 2

Warning: In case you already have a cluster up and you want to rebuild from scratch, DELETE any backups from etcd (if you are using that. I was using S3 backup and I had to also delete those, otherwise something with caching is going crazy.

On the first attempt, I had the networking like this:

     ssh:
       - office_static_ip/32
       - 10.0.0.0/16
       - 10.8.0.0/24                # WireGuard VPN cidr if used
    api:                            # k8s API (6443) reachable only privately
       - office_static_ip/32
       - 10.0.0.0/16
       - 10.8.0.0/24

But eventually, I changed that to 0.0.0.0/0 as shown below.
Another change was on additional_pre_k3s_commands. On my first attempt, I tried to recreate it as it was in the NAT documentation for the client.
So initially, I had this setting:

additional_pre_k3s_commands:
 - echo "auto enp7s0" >> /etc/network/interfaces
 - echo "iface enp7s0 inet dhcp" >> /etc/network/interfaces
 - echo "    post-up ip route add default via 10.0.0.1"  >> /etc/network/interfaces
 - echo "dns-nameservers 8.8.8.8 1.1.1.1" >> /etc/network/interfaces
 - echo "[Resolve]" > /etc/systemd/resolved.conf
 - echo "DNS=185.12.64.2 185.12.64.1" >> /etc/systemd/resolved.conf
 - echo "FallbackDNS=8.8.8.8" >> /etc/systemd/resolved.conf
 - ifdown enp7s0 && ifup enp7s0
 - systemctl start systemd-resolved
 - systemctl enable systemd-resolved

Then I tried the same as is in the documentation Private_clusters_with_public_network_interface_disabled, that didn't work as expected, so, again mix and match. I updated the DNS to Hetzner's. Probably something could be done and avoid this change, but I was feeling tired to retry it.

So final cluster_config.yaml is the following:

hetzner_token: your-api-token
cluster_name: k3s-cluster
kubeconfig_path: "./kubeconfig"
k3s_version: v1.34.1+k3s1

networking:
  ssh:
    port: 22
    use_agent: false # set to true if your key has a passphrase
    public_key_path: "~/.ssh/id_ed25519.pub"
    private_key_path: "~/.ssh/id_ed25519"
  allowed_networks:
    ssh:
      - 0.0.0.0/0
    api:
      - 0.0.0.0/0
    custom_firewall_rules:
      - description: "Cilium VXLAN overlay"
        direction: in
        protocol: udp
        port: 8472
        source_ips: [ "10.0.0.0/16" ]

  public_network:
    ipv4: false
    ipv6: false
  private_network:
    enabled: true
    subnet: 10.0.0.0/16
    existing_network_name: "cluster-network"
  cni:
    enabled: true
    encryption: false
    mode: cilium
    # cilium:
      # Optional: specify a path to a custom values file for Cilium Helm chart
      # When specified, this file will be used instead of the default values
      # helm_values_path: "./cilium-values.yaml"
      # chart_version: "v1.17.2"

  # cluster_cidr: 10.244.0.0/16 # optional: a custom IPv4/IPv6 network CIDR to use for pod IPs
  # service_cidr: 10.43.0.0/16 # optional: a custom IPv4/IPv6 network CIDR to use for service IPs. Warning, if you change this, you should also change cluster_dns!
  # cluster_dns: 10.43.0.10 # optional: IPv4 Cluster IP for coredns service. Needs to be an address from the service_cidr range


# manifests:
#   cloud_controller_manager_manifest_url: "https://github.com/hetznercloud/hcloud-cloud-controller-manager/releases/download/v1.23.0/ccm-networks.yaml"
#   csi_driver_manifest_url: "https://raw.githubusercontent.com/hetznercloud/csi-driver/v2.12.0/deploy/kubernetes/hcloud-csi.yml"
#   system_upgrade_controller_deployment_manifest_url: "https://github.com/rancher/system-upgrade-controller/releases/download/v0.14.2/system-upgrade-controller.yaml"
#   system_upgrade_controller_crd_manifest_url: "https://github.com/rancher/system-upgrade-controller/releases/download/v0.14.2/crd.yaml"
#   cluster_autoscaler_manifest_url: "https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/hetzner/examples/cluster-autoscaler-run-on-master.yaml"
#   cluster_autoscaler_container_image_tag: "v1.32.0"

datastore:
  mode: etcd # etcd (default) or external
  # external_datastore_endpoint: postgres://....
  etcd:
      snapshot_retention: 24
      snapshot_schedule_cron: "0 * * * *"
      # S3 snapshot configuration (optional)
      s3_enabled: true
      s3_endpoint: "s3service-endpoint" # Can also be set with ETCD_S3_ENDPOINT environment variable
      s3_region: "eu-central-1" # Can also be set with ETCD_S3_REGION environment variable
      s3_bucket: "whatever-bucketr" # Can also be set with ETCD_S3_BUCKET environment variable
      s3_access_key: "123412341234" # Can also be set with ETCD_S3_ACCESS_KEY environment variable
      s3_secret_key: "1234123412345" # Can also be set with ETCD_S3_SECRET_KEY environment variable
      s3_folder: "k3s-cluster"
      s3_force_path_style: false

schedule_workloads_on_masters: false

image: debian-12 # optional: default is ubuntu-24.04
autoscaling_image: debian-12 # optional, defaults to the `image` setting

masters_pool:
  instance_type: cx23
  instance_count: 3 # for HA; you can also create a single master cluster for dev and testing (not recommended for production)
  locations: # You can choose a single location for single master clusters or if you prefer to have all masters in the same location. For regional clusters (which are only available in the eu-central network zone), each master needs to be placed in a separate location.
    - fsn1
    - hel1
    - nbg1

worker_node_pools:
- name: small-scalled
  instance_type: cx33
  location: fsn1
  autoscaling:
    enabled: true
    min_instances: 1
    max_instances: 4

- name: medium-autoscaled
  instance_type: cx43
  location: fsn1
  autoscaling:
    enabled: true
    min_instances: 0
    max_instances: 3

cluster_autoscaler:
  scan_interval: "10s"                        # How often cluster is reevaluated for scale up or down
  scale_down_delay_after_add: "10m"           # How long after scale up that scale down evaluation resumes
  scale_down_delay_after_delete: "10s"        # How long after node deletion that scale down evaluation resumes
  scale_down_delay_after_failure: "3m"        # How long after scale down failure that scale down evaluation resumes
  max_node_provision_time: "15m"              # Maximum time CA waits for node to be provisioned

embedded_registry_mirror:
  enabled: false # Enables fast p2p distribution of container images between nodes for faster pod startup. Check if your k3s version is compatible before enabling this option. You can find more information at https://docs.k3s.io/installation/registry-mirror

addons:
  csi_driver:
    enabled: false   # Hetzner CSI driver (default true). Set to false to skip installation.
  traefik:
    enabled: true  # built-in Traefik ingress controller. Disabled by default.
  servicelb:
    enabled: false  # built-in ServiceLB. Disabled by default.
  metrics_server:
    enabled: true  # Kubernetes metrics-server addon. Disabled by default.
  cloud_controller_manager:
    enabled: true   # Hetzner Cloud Controller Manager (default true). Disabling stops automatic LB provisioning for Service objects.
  cluster_autoscaler:
    enabled: true   # Cluster Autoscaler addon (default true). Set to false to omit autoscaling.

protect_against_deletion: true
create_load_balancer_for_the_kubernetes_api: false # Just a heads up: right now, we can’t limit access to the load balancer by IP through the firewall. This feature hasn’t been added by Hetzner yet.
k3s_upgrade_concurrency: 1 # how many nodes to upgrade at the same time

additional_pre_k3s_commands:
- apt update
- apt upgrade -y
- apt install ifupdown resolvconf -y
- apt autoremove -y hc-utils
- apt purge -y hc-utils
- echo "auto enp7s0" > /etc/network/interfaces.d/60-private
- echo "iface enp7s0 inet dhcp" >> /etc/network/interfaces.d/60-private
- echo "    post-up ip route add default via 10.0.0.1"  >> /etc/network/interfaces.d/60-private
- echo "[Resolve]" > /etc/systemd/resolved.conf
- echo "DNS=185.12.64.2 185.12.64.1" >> /etc/systemd/resolved.conf
- ifdown enp7s0
- ifup enp7s0
- systemctl start resolvconf
- systemctl enable resolvconf
- echo "nameserver 185.12.64.2" >> /etc/resolvconf/resolv.conf.d/head
- echo "nameserver 185.12.64.1" >> /etc/resolvconf/resolv.conf.d/head
- resolvconf --enable-updates
- resolvconf -u

additional_post_k3s_commands:
  - 'export DEBIAN_FRONTEND=noninteractive; apt-get update -y && apt-get install -y fail2ban ca-certificates curl gnupg lsb-release unattended-upgrades systemd-resolved systemd-timesyncd'
  - 'dpkg-reconfigure --priority=low unattended-upgrades'
  - 'sed -i "s#^//\\?Unattended-Upgrade::Automatic-Reboot .*#Unattended-Upgrade::Automatic-Reboot \\"true\\";#" /etc/apt/apt.conf.d/50unattended-upgrades'
  - 'grep -q "Unattended-Upgrade::Automatic-Reboot-Time" /etc/apt/apt.conf.d/50unattended-upgrades || echo "Unattended-Upgrade::Automatic-Reboot-Time \\"04:00\\";" >> /etc/apt/apt.conf.d/50unattended-upgrades'
  - 'systemctl enable --now unattended-upgrades fail2ban systemd-resolved systemd-timesyncd'

Step 3

As I have decided to use Traefik for ingress, my initial assumption, which was wrong, was that the setup would create a Hetzner Load Balancer to expose the cluster. So, in order to achieve this, and avoid messing that early with shared storage, I decided to use cert-manager.

So, first step, we need to change the following to traefik.

kubectl -n kube-system patch svc traefik \
  -p '{
    "metadata": {
      "annotations": {
        "load-balancer.hetzner.cloud/location": "fsn1",
        "load-balancer.hetzner.cloud/use-private-ip": "true",
        "load-balancer.hetzner.cloud/name": "traefik-lb",
        "load-balancer.hetzner.cloud/http-redirect-https": "false"
      }
    }
  }'

Install cert-manager to avoid having traefik, to save acme.json and manage shared storage for HA traefik.

kubectl create namespace cert-manager

helm repo add jetstack https://charts.jetstack.io
helm repo update

helm upgrade --install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --set installCRDs=true

verify:

kubectl -n cert-manager get pods

Then use cert-manager-cloudflare.yaml

apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token
  namespace: cert-manager
type: Opaque
stringData:
  api-token: "cf-api-token"
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-cloudflare
spec:
  acme:
    email: info@foobar.com               # your email for LE notifications
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-cloudflare-key
    solvers:
      - dns01:
          cloudflare:
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: api-token

And apply

kubectl apply -f cert-manager-cloudflare.yaml

Finally, update traefik with the following traefik-config.yaml

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik          # must match HelmChart name
  namespace: kube-system 
spec:
  valuesContent: |-
    # Extra static arguments passed to Traefik
    additionalArguments:
      # Redirect HTTP -> HTTPS
      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.web.http.redirections.entrypoint.permanent=true"

    deployment:
      replicas: 2
    
    # Since we use cert-manager, we do not need persistence storage, everything is a secret to the cluster
    persistence:
      enabled: false

    # Force websecure has TLS enabled (most k3s defaults already do this, but this is a safe, explicit override aligned with the Traefik chart)
    ports:
      websecure:
        tls:
          enabled: true

At this stage, you should be able to start deploying your applications.

Currently, I did test to deploy a NextJS application with the following yaml.

apiVersion: v1
kind: Namespace
metadata:
  name: project-main-site
---
apiVersion: v1
kind: Secret
metadata:
  name: project-env
  namespace: project-main-site
type: Opaque
stringData:
  EMAIL_USER: "info@foobar.com"
  EMAIL_PASS: "123412341234"
  DISCORD_WEBHOOK_URL: "https://discord.com/api/webhooks/........-"
  NEXT_PUBLIC_MEASUREMENT_ID: "G-12341234"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: project-main-site
  namespace: project-main-site
spec:
  replicas: 2
  selector:
    matchLabels:
      app: project-main-site
  template:
    metadata:
      labels:
        app: project-main-site
    spec:
      # you need to create secret, to access private registry
      imagePullSecrets:
        - name: regcred
      containers:
        - name: app
          image: registry.foobar.com/websites/nextjs-site
          env:
            - name: NODE_ENV
              value: "production"
            - name: EMAIL_USER
              valueFrom:
                secretKeyRef:
                  name: project-env
                  key: EMAIL_USER
            - name: EMAIL_PASS
              valueFrom:
                secretKeyRef:
                  name: project-env
                  key: EMAIL_PASS
            - name: DISCORD_WEBHOOK_URL
              valueFrom:
                secretKeyRef:
                  name: project-env
                  key: DISCORD_WEBHOOK_URL
            - name: NEXT_PUBLIC_MEASUREMENT_ID
              valueFrom:
                secretKeyRef:
                  name: project-env
                  key: NEXT_PUBLIC_MEASUREMENT_ID
          ports:
            - containerPort: 80
          resources:
            requests:
              cpu: "100m"
              memory: "256Mi"
            limits:
              cpu: "500m"
              memory: "512Mi"
          volumeMounts:
            - name: logs
              mountPath: /usr/src/app/dist/logs
      volumes:
        - name: logs
          emptyDir: {} 
---
apiVersion: v1
kind: Service
metadata:
  name: project-main-site
  namespace: project-main-site
spec:
  selector:
    app: project-main-site
  ports:
    - port: 80
      targetPort: 3000
      protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: project-main-site
  namespace: project-main-site
  annotations:
    kubernetes.io/ingress.class: traefik
    cert-manager.io/cluster-issuer: letsencrypt-cloudflare
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
spec:
  tls:
    - hosts:
        - foobar.com
      secretName: foobar-com-tls    # cert-manager will create/maintain this
  rules:
    - host: foobar.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: project-main-site
                port:
                  number: 80

What comes next

Next step for me is to test the CSI driver. One of the use cases I need to manage, while dummy and probably not ideal, is WordPress hosting. At first, I was thinking Longhorn to have RWX, but the longer I thought about it, the drawback of shared storage between nodes is that if I need 1TB I need to spend a lot $$ in nodes with high storage. So, the hcloud volumes, at least for the initial setup, seem to be a nice solution, although I will not have RWX. I saw some posts about a solution around this topic, but if there is any suggestion, feel free to share.

[vitobotta]

Thanks a lot! I will need to investigate adding proper support for dedicated servers at some point and this info can be helpful with that too.

[ArgirisGkogkidis]

Thank you for this great tool. I will get back with more feedback and some suggestions/feature requests, let's say.
For now, I have identified 2 pain points:

1. It would be nice to have Longhorn set up, optional of course, but similar as you enable csi_driver. Perhaps there is something better, still, I need to investigate.
2. Usually, on VM setup, I want to block port 22, switch to another, and remove root user, create another user and sudo -i to do sudo things. Could this be done in this script as an extra layer of security? or would it complicate things more than it needs to?

[vitobotta]

1. I see Longhorn as something out of scope for the tool. hetzner-k3s' goal is to provide you easily and quickly with a fully functional, production grade cluster in Hetzner Cloud. the Hetzner CSI driver, since it's from Hetzner, is part of the package. Any other software defined storage solution is something that should be handled separately. Otherwise we'd have to support Longhorn and other stuff. So I don't think I would add built in support for Longhorn.
2. Yeah it would complicate things with no much benefit. The only thing running on these nodes is k3s really, and that has to ruin as root anyway. Password authentication via SSH is disabled, as only pubkey authentication is allowed. Plus, the SSH port can be customized if you want, and it's always protected by the firewall as long as you don't add 0.0.0.0/0 (or the ipv6 equivalent) to the allowed networks list.