Windows: spotty virtio nic functionality

[iximeow]

Some boots of a Windows Server 2022 guest seem to be unable to use the virtio network device, while others can only use it for a few seconds before something happens and the guest is unable to use the network.

here's the interesting viona bits as a Windows VM gets weird:

MMIO NOTIFY write c000a000+2 = 7       # <-- I'm pinging this VM from elsewhere on my network
MMIO NOTIFY write c000a000+2 = 7
MMIO NOTIFY write c000a000+2 = 1
MMIO NOTIFY write c000a000+2 = 1
MMIO NOTIFY write c000a000+2 = 1
MMIO NOTIFY write c000a000+2 = 1
MMIO NOTIFY write c000a000+2 = 1
MMIO NOTIFY write c000a000+2 = 1
MMIO NOTIFY write c000a000+2 = 3
MMIO NOTIFY write c000a000+2 = 3
MMIO NOTIFY write c000a000+2 = 3       # <-- and this is right as `ping` stops seeing the VM
RING_RESET 0x0                         # <-- it's reset the device, but why?
RING_RESET 0x1
RING_RESET 0x2
RING_RESET 0x3
RING_RESET 0x4
RING_RESET 0x5
RING_RESET 0x6
RING_RESET 0x7
RING_RESET 0x8
RING_RESET 0x9
RING_RESET 0xa
RING_RESET 0xb
RING_RESET 0xc
RING_RESET 0xd
RING_RESET 0xe
RING_RESET 0xf
RING_RESET 0x10
RING_RESET 0x11
RING_RESET 0x12
RING_RESET 0x13
RING_RESET 0x14
RING_RESET 0x15
RING_SET_MSI 0x1 addr=0 msg=0
RING_SET_MSI 0x0 addr=0 msg=0
RING_SET_MSI 0x3 addr=0 msg=0
RING_SET_MSI 0x2 addr=0 msg=0
RING_SET_MSI 0x5 addr=0 msg=0
RING_SET_MSI 0x4 addr=0 msg=0
RING_SET_MSI 0x7 addr=0 msg=0
RING_SET_MSI 0x6 addr=0 msg=0
SET_FEATURES 0x1104388ab
SET_PAIRS 0xb
SET_USEPAIRS 0xb
RING_INIT_MODERN 0x1 233496000/23349e000/23349f040 800
RING_SET_MSI 0x1 addr=0 msg=0
RING_INIT_MODERN 0x0 237ff1000/237ff2000/237ff2240 100
RING_SET_MSI 0x0 addr=0 msg=0
RING_INIT_MODERN 0x3 235002000/23500a000/23500b040 800
RING_SET_MSI 0x3 addr=0 msg=0
RING_INIT_MODERN 0x2 238158000/238159000/238159240 100
RING_SET_MSI 0x2 addr=0 msg=0
RING_INIT_MODERN 0x5 230505000/23050d000/23050e040 800
RING_SET_MSI 0x5 addr=0 msg=0
RING_INIT_MODERN 0x4 231468000/231469000/231469240 100
RING_SET_MSI 0x4 addr=0 msg=0
RING_INIT_MODERN 0x7 232057000/23205f000/232060040 800
RING_SET_MSI 0x7 addr=0 msg=0
RING_INIT_MODERN 0x6 232f99000/232f9a000/232f9a240 100
RING_SET_MSI 0x6 addr=0 msg=0
RING_SET_MSI 0x1 addr=fee0100c msg=49a1
RING_SET_MSI 0x0 addr=fee0100c msg=4991
RING_SET_MSI 0x3 addr=fee0200c msg=49b1
RING_SET_MSI 0x2 addr=fee0200c msg=49a1
RING_SET_MSI 0x5 addr=fee0400c msg=49b1
RING_SET_MSI 0x4 addr=fee0400c msg=49a1
RING_SET_MSI 0x7 addr=fee0800c msg=49b1
RING_SET_MSI 0x6 addr=fee0800c msg=49a1
SET_FEATURES 0x1104388ab
SET_PAIRS 0xb
ioctl(SET_PAIRS) returning 0x10
MMIO NOTIFY write c000a000+2 = 16
SET_USEPAIRS 0x4
MMIO NOTIFY write c000a000+2 = 0
RING_KICK 0x0
MMIO NOTIFY write c000a000+2 = 2
RING_KICK 0x2
MMIO NOTIFY write c000a000+2 = 4
RING_KICK 0x4
MMIO NOTIFY write c000a000+2 = 6
RING_KICK 0x6
                                       # <-- it just keeps resetting the NIC at this point

this seems to have been the case since the modern VirtIO/multiqueue stuff landed in 6990099 . further, flipping off VIRTIO_NET_F_MQ from the virtio features seems to work fine or at least as long as I've been writing up this issue (a minute or three)

[iximeow]

bonus datapoint: having VIRTIO_NET_F_MQ but PROPOLIS_MAX_MQ_PAIRS == 1 also seems to avoid .. whatever is happening.

[iximeow]

I have a sneaking suspicion that

SET_PAIRS 0xb
ioctl(SET_PAIRS) returning 0x10

is less innocent than we'd thought. #1051 adds some probes around the vinoa/vnic PCI state and I see we're getting the device to NEEDS_RESET at this point. I think the fact that Linux tolerates this is probably a bug.

if Windows has some kind of background task that would go kick the NEEDS_RESET devices and locks them out until they come up without NEEDS_RESET, I think we'd end up seeing basically this.