Not clear how vmnet-helper should be used to connect multiple VMs to the same network.

[nohajc]

Each time vmnet-helper is started, it creates a new bridge interface with an associated subnet. Do I just reuse the file descriptor or socket file for all VMs which I want to have in the same network or do I need to do some extra work here?

On the other hand, the output configuration always has one MAC address for one particular guest. So I'm not really sure how to obtain config for subsequent guests on the same network.

Sorry if this sounds like a newbie question. :)

EDIT: I am mainly interested in the shared operation mode.

EDIT2: Do I need vmnet-broker for this?

[nirs]

vmnet-helper is per vm - each vm need a helper program to forward packets between the vm file descriptor and the vmnet-interface.

The architecture is:

vm1 (libkrun) <--> vment-helper (vmnet interface) <--> vmnet network
vm2 (libkrun) <--> vment-helper (vmnet interface) <--> vment network
vm3 (libkrun) <--> vment-helper (vmnet interface) <--> vment network

Usually you will see a new bridge (e.g. bridge100) with 3 members (vmnet0, vmnet1, vmnet2).

VMs communicate via the vmnet network (192.168.105.0/24). vmnet-helper is not a switch, it is a cable connecting the VM process to the vmnet network.

All modes work int same way. The difference are:

• host mode does not have NAT, you can access only the host and other vms
• shared mode has NAT, you can access the host, other vms, and other networks
• bridged mode is connecting to a physical interface - like your host, you get the IP from the DHCP server in your LAN and other hosts can access the vms.

[nohajc]

Does this mean each vm / helper needs to use a distinct subnet? According to the documentation 192.168.105.0/24 is the default and I can see my bridge interface gets 192.168.105.1 assigned.

So I assume I always need to use a non-conflicting IP range. Does vmnet-helper validate this somehow or is it purely my responsibility?

[nirs]

no, if you use a different subnet the vm will not be able communicate. If you don't specify --start-address, --end-address and --subnet-mask you get 192.168.105.1, 192.168.105.254, and 255.255.255.0.

You want to specify a unique UUID for every vm (e.g. --interface-id 9F5D834C-8C51-433C-B039-AACAB35F1154) and vmnet will allocate the same mac address for every vm. Then the system DHCP server will assign the next available IP for each vm, starting at 192.168.105.2. 192.168.105.1 is the host address on this network.

If you don't specify --interface-id you will get random mac address each time you start the vm, and new IP address. This makes it harder to use the vms since the IP change on every run.

You can also assign the mac address yourself, for example compute a hash of the vm name and take the first 6 bytes for the mac address. See how we do this here:

vmnet-helper/vmnet/mac.py

Line 7 in b9e0218

def random_mac_address():

[nohajc]

Frankly, I'm still confused. An example would help.

I'm trying to understand how the bridge setup / routing table looks like from the host perspective when I have multiple VMs, each with its helper process.

You also said vmnet-helper isn't a switch but a network bridge is a software switch. So, is it possible to associate multiple VMs with the same bridge or do I need vmnet-broker to do that?

In my mind, each host bridge corresponds to one vm network because how else would you know where to route the traffic?

So, does the vmnet framework only create the bridge once for every network and remove it when the last process using it had been terminated?

Maybe it would help to have a diagram which shows how vmnet-helper interacts with the OS framework. Or just a note regarding the lifetimes of process/vm interface/vm bridge.

I understand it's probably not surprising for people experienced with virtual networking but I'm still kind of learning the basics.

Thank you!

[nirs]

Frankly, I'm still confused. An example would help.

I'm trying to understand how the bridge setup / routing table looks like from the host perspective when I have multiple VMs, each with its helper process.

You also said vmnet-helper isn't a switch but a network bridge is a software switch. So, is it possible to associate multiple VMs with the same bridge or do I need vmnet-broker to do that?

Yes, all vms started with same --start-address, --end-address, and --subnet-mask are sharing the network (default 192.168.105.0/24) using bridge (typically bridge100).

In my mind, each host bridge corresponds to one vm network because how else would you know where to route the traffic?

Yes, every network has a bridge.

So, does the vmnet framework only create the bridge once for every network and remove it when the last process using it had been terminated?

Yes, when the last vmnet-helper process shuts down, the bridge disappers.

Maybe it would help to have a diagram which shows how vmnet-helper interacts with the OS framework. Or just a note regarding the lifetimes of process/vm interface/vm bridge.

vmnet-helper is simple - it copies packets from a socket to vment interface. The vmnet framework is creating and destroying the bridges as needed.

Architecture diagram

flowchart TB
  subgraph vmnet-client1
    subgraph vm1["vm1 192.168.105.2"]
      socket1
    end
    subgraph helper1["vmnet-helper1"]
      socket2
    end

    socket1 <--> socket2
  end

  subgraph vmnet-client2
    subgraph vm2["vm2 192.168.105.3"]
      socket3
    end
    subgraph helper2["vmnet-helper2"]
      socket4
    end

    socket3 <--> socket4
  end

  subgraph vmnet-client3
    subgraph vm3["vm3 192.168.105.4"]
      socket5
    end
    subgraph helper3["vmnet-helper3"]
      socket6
    end

    socket5 <--> socket6
  end

  subgraph vmnet
    vmnet1
    vmnet2
    vmnet3
    bridge["bridge100 192.168.105.1"]

    vmnet1 <--> bridge
    vmnet2 <--> bridge
    vmnet3 <--> bridge
  end

  helper1 <--> vmnet1
  helper2 <--> vmnet2
  helper3 <--> vmnet3

Loading

[nohajc]

This is great, thank you. It didn't occur to me the bridges are managed automatically, especially when I read the vmnet-broker documentation and motivation:

The catch: vmnet networks are bound to the process that creates them. When the process exits, the network is destroyed, even if VMs are still using it.

So, this isn't exactly true, is it? If I had a VM helper process A which creates VM network 1, then started a helper process B and also attached it to VM network 1, I could exit VM helper A but the network would still exist until helper B terminated as well.

At first, I was under the impression I need the broker to manage vm network lifetime but that seems to be already taken care of be the OS framework.

[nirs]

What I show here is the legacy way. In macOS 26, if you use the virtualization framework, there is native vment support, so you create the vmnet interface in the vm process, and you don't need to copy packets - the virtualization framework handles the data path for you, and you get 3-6 times better performance with less cpu usage.

But in the new model if you want to use the same network (e.g. 192.168.64.0/24) with multiple vms, you need to create the network in another process and pass the network serialization to the vm processes. You need to keep the process that created the network running while the vms are using the network. This is what vmnet-broker does for you.

vmnet-helper v0.10.0 will support this model - instead of starting it with --{start,end}-address, you strat it with a network name (e.g. "shared"). It gets the "shared" network from vmnet-broker and start the vmnet interface with the network. This allows vms using native vment support (vfkit, lima, utm, tart) to be in the same network as vms using libkrun or qemu.

vmnet-helper and vmnet-broker architecture

flowchart TB
  subgraph vm1["vfkit vm 192.168.64.2"]
  end

  subgraph vmnet-client1
    subgraph vm2["libkrun vm 192.168.64.3"]
      socket1
    end
    subgraph helper1["vmnet-helper1"]
      socket2
    end

    socket1 <--> socket2
  end

  subgraph vmnet-broker
    network1["shared network 192.168.64.0/24"]
  end

  subgraph vmnet
    vmnet1
    vmnet2
    bridge["bridge101 192.168.64.1"]

    vmnet1 <--> bridge
    vmnet2 <--> bridge
  end

  vm1 <--> vmnet1
  vm1 --> vmnet-broker

  helper1 <--> vmnet2
  helper1 --> vmnet-broker

  vmnet-broker --> vmnet

Loading

[nohajc]

Ok, now it makes perfect sense. Thank you for further clarification.

That means if I work with a libkrun VM, I wouldn't get any performance advantage using the broker except it would allow me to communicate with Virtualization framework VMs.

[nirs]

@nohajc feel free to review #167. I hope it makes things more clear.