Instance network interface is gone after guest reboot

[askfongjojo]

The issue is found on dogfood and still reproducible on a racklet running commit c765b3539203e34f65cd402f139cf604035d5993 (i.e. #9746).

The instance in question has an ephemeral IP and a ssh key added at create time, and password auth configured. I was unable to SSH into it after sudo reboot now but could still access it via password auth on serial console. The instance was up and running and still listening to port 22. All the opte and swadm nat entries on the sled and both switches were intact.

The problem turned out to be the guest's losing its private IP after guest reboot:

before:

ubuntu@vm1-1:~$ ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether a8:40:25:f0:00:00 brd ff:ff:ff:ff:ff:ff
    inet 172.30.0.5/32 metric 100 scope global dynamic enp0s8
       valid_lft 86387sec preferred_lft 86387sec
    inet6 fe80::aa40:25ff:fef0:0/64 scope link 
       valid_lft forever preferred_lft forever
ubuntu@vm1-1:~$ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         172.30.0.1      0.0.0.0         UG        0 0          0 enp0s8
1.1.1.1         172.30.0.1      255.255.255.255 UGH       0 0          0 enp0s8
9.9.9.9         172.30.0.1      255.255.255.255 UGH       0 0          0 enp0s8
172.30.0.1      0.0.0.0         255.255.255.255 UH        0 0          0 enp0s8

after:

ubuntu@vm1-1:~$ ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
ubuntu@vm1-1:~$ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface

[askfongjojo]

The instance nic is fixed after a stop and start. It's running noble, with 4vcpu and 16 Gb memory.

I have one instance that doesn't have the lost nic behavior. The only difference is in size (32 vcpu, 96 Gb memory). It has consistently come back with the nic configured after guest reboot.

I've placed the guest logs in /staff/core/propolis-1030 for these two instances in case there is anything useful from there.

[iximeow]

I took a very very quick look here, this is easily reproducible with propolis-standalone ( ... once you actually are on appropriately recent illumos bits ... )

comparing viona.d output from a fresh boot versus an in-place reboot they're pretty similar up until..

SET_FEATURES 0x1104388ab
SET_PAIRS 0xb

where on a fresh boot the next line is

SET_USEPAIRS 0xb

but on a reboot it's

ioctl(SET_PAIRS) returning 0x10

I think that's viona_ioc_link_setpairs() -> viona_link_qalloc() -> link->l_vrings[i].vr_state != VRS_RESET and the attendant EBUSY. then we bail before set_usepairs so the device doesn't ever communicate back to the host, it looks like?

the rings would be set back to VRS_RESET here at ring_reset: so I expect we're not fully signalling the reset to viona on a reboot ... somehow ...

[iximeow]

so, in Propolis when we self.hdl.set_pairs(PROPOLIS_MAX_MQ_PAIRS) we're getting viona to go set up 0xb rings. but if the guest doesn't negotiate multiqueue, we only manage two rings (plus the admin queue). then when we reset the device we get to ring_reset for two actual I/O queues but leave the rest untouched.

in my case, I've got a Debian 13 guest that seems to end up here, and looking at 0xfffffebefed24000::print struct viona_vring vr_state (and knowing each viona_vring is 0x208 bytes long), I see the first and second rings are VRS_RESET, but the third and on VRS_INIT.

... we probably should drop number of rings tracked by VirtQueues back to the default of 3 across the reset too, instead of leaving that unchanged?

[dancrossnyc]

FWIW, there's clearly a bug here; my bad.

I think that dropping the number of rings down makes sense. Comparing against c-bhyve (which Andy pointed me to), the behavior is to reset all queues/ring pairs and reset to the initial state.

I'd probably add a method to VirtQueues to do this, as it can walk over the entire list, regardless of how many are actually used.