Multi zone masters on HA setup?

[GiovanniBarile]

Hello, is there a way to create a multi zone control plane? For example
1x master in Helsinki, 1x in Falkenstein and 1x in Nuremberg ?
So it can really be HA, this way if something happens in a specific datacenter, all the masters go down at the same time

Thank you

[vitobotta]

Hi, it’s on the plan, but it’s not available right now. By default, hetzner-k3s uses etcd for the control plane data storage, and etcd is really sensitive to latency issues. So, having masters in different regions wouldn’t work well without a lot of adjustments and tests. Based on my experience, etcd can be quite challenging in setups like this.

hetzner-k3s v2 lets you use a relational database, such as Postgres or MySQL, instead of etcd, which solves the latency problem. This part is already in place, but I still need to add support for configuration options that let you choose where to place the masters. I can’t give you an exact timeframe for this because I have a lot of other tasks and limited time.

[vitobotta]

Hi, it's locations plural :)

[jobetinfosec]

Hello Vito,
I changed that to locations, but now an error shows up and the script stops:
[Configuration] - All must be in valid locations and in the same same network zone when using a private network

[vitobotta]

Can you share your full config?

[jobetinfosec]

Here you are:

hetzner_token: token
cluster_name: test-k3s
kubeconfig_path: "./kubeconfig"
k3s_version: v1.32.0+k3s1

networking:
  ssh:
    port: 22
    use_agent: false
    public_key_path: "~/.ssh/id_rsa.pub"
    private_key_path: "~/.ssh/id_rsa"
  allowed_networks:
    ssh:
      - xxx/32
    api:
      - 0.0.0.0/0
  public_network:
    ipv4: true
    ipv6: true
  private_network:
    enabled : true
    subnet: 10.3.0.0/16
    existing_network_name: ""
  cni:
    enabled: true
    encryption: false
    mode: flannel

masters_pool:
  instance_type: cpx21
  instance_count: 3
  locations:
  - fns1
  - hel1
  - nbg1

worker_node_pools:
- name: node
  instance_type: cpx21
  instance_count: 3
  location: fsn1

protect_against_deletion: false

[jobetinfosec]

@vitobotta

Any advice?

[jobetinfosec]

Found culprit...
Indentation issue...
This is the right indentation:

locations:
    - fsn1
    - hel1
    - nbg1

For God's sake!! :)

[vitobotta]

Glad you sorted it :)

[BartKrol]

Hey @vitobotta, sorry for old thread necromancy, but I think I have similar problem. What's the best way to have multiple clusters around the world? Is it creating config for all of them? If I want to have cluster in Ashburn and Hillsboro that means the same right, two files?

[vitobotta]

Hi, yes, each cluster needs its own config file. Putting multiple cluster definitions in the same config file would be a mess.

[BartKrol]

Weirdly I had a problem setting hil cluster - I still have an issue with it :/. It's like hil is not really correct for the config?

[vitobotta]

There is no difference in hetzner-k3s between the different locations. Maybe Hetzner is having problems in that location right now. What is the issue?

[BartKrol]

hetzner-k3s create --config hetzner-k3s_cluster_config-hil.yaml
 _          _                            _    _____     
| |__   ___| |_ _____ __   ___ _ __     | | _|___ / ___ 
| '_ \ / _ \ __|_  / '_ \ / _ \ '__|____| |/ / |_ \/ __|
| | | |  __/ |_ / /| | | |  __/ | |_____|   < ___) \__ \
|_| |_|\___|\__/___|_| |_|\___|_|       |_|\_\____/|___/

Version: 2.3.2

[Configuration] Validating configuration...
[Configuration] Some information in the configuration file requires your attention, aborting.
[Configuration]  - All workers must be in valid locations and in the same same network zone as the masters when using a private network. If the masters are located in Ashburn, then all the worker must be located in Ashburn too. Same thing for Hillsboro and Singapore. If the masters are located in Germany and/or Finland, then also the workers must all be located in either Germany or Finland since these locations belong to the same network zone.

My config

hetzner_token: [TOKEN]
cluster_name: app-hil
kubeconfig_path: "./kubeconfig" 
k3s_version: v1.32.0+k3s1
protect_against_deletion: false

networking:
  ssh:
    port: 22
    use_agent: false
    public_key_path: "~/.ssh/id_ed25519.pub"
    private_key_path: "~/.ssh/id_ed25519"
  allowed_networks:
    ssh:
      - 0.0.0.0/0
    api:
      - 0.0.0.0/0

masters_pool:
  instance_type: cpx21
  instance_count: 1
  location: hil

worker_node_pools:
  - name: small
    instance_type: cpx21
    instance_count: 3
    location: hil
    assign_public_ip: false

[vitobotta]

Please use the latest version v2.3.8 - it has many fixes since v2.3.2, including one for detecting network zones, which might be the issue you are having here.