jammy still cannot get back its vnic after guest reboot

[askfongjojo]

ubuntu 22.04 still suffers from the vnic not configured issue fixed in #9828. This guest OS version has always been laggy in network setup but now it seems to be failing consistently. Users may see this kind of syslog on the serial console:

[    6.314734] cloud-init[736]: Cloud-init v. 25.1.4-0ubuntu0~22.04.1 running 'init-local' at Sun, 08 Feb 2026 03:54:17 +0000. Up 6.28 seconds.
[  OK  ] Finished Cloud-init: Local Stage (pre-network).
[  OK  ] Reached target Preparation for Network.
         Starting Network Configuration...
[  OK  ] Started Network Configuration.
         Starting Wait for Network to be Configured...
         Starting Network Name Resolution...
[  OK  ] Started Network Name Resolution.
[  OK  ] Reached target Network.
[  OK  ] Reached target Host and Network Name Lookups.
[FAILED] Failed to start Wait for Network to be Configured.
See 'systemctl status systemd-networkd-wait-online.service' for details.
         Starting Cloud-init: Network Stage...
[  126.982059] cloud-init[753]: Cloud-init v. 25.1.4-0ubuntu0~22.04.1 running 'init' at Sun, 08 Feb 2026 03:56:18 +0000. Up 126.95 seconds.
[  126.996223] cloud-init[753]: ci-info: ++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++
[  127.004558] cloud-init[753]: ci-info: +--------+------+------------------------------+-----------+-------+-------------------+
[  127.012597] cloud-init[753]: ci-info: | Device |  Up  |           Address            |    Mask   | Scope |     Hw-Address    |
[  127.020902] cloud-init[753]: ci-info: +--------+------+------------------------------+-----------+-------+-------------------+
[  127.029196] cloud-init[753]: ci-info: | enp0s8 | True | fe80::aa40:25ff:fef0:b958/64 |     .     |  link | a8:40:25:f0:b9:58 |
[  127.037427] cloud-init[753]: ci-info: |   lo   | True |          127.0.0.1           | 255.0.0.0 |  host |         .         |
[  127.045586] cloud-init[753]: ci-info: |   lo   | True |           ::1/128            |     .     |  host |         .         |
[  127.053696] cloud-init[753]: ci-info: +--------+------+------------------------------+-----------+-------+-------------------+
[  127.061754] cloud-init[753]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
[  127.067884] cloud-init[753]: ci-info: +-------+-------------+---------+-----------+-------+
[  127.074017] cloud-init[753]: ci-info: | Route | Destination | Gateway | Interface | Flags |
[  127.080205] cloud-init[753]: ci-info: +-------+-------------+---------+-----------+-------+
[  127.086207] cloud-init[753]: ci-info: |   1   |  fe80::/64  |    ::   |   enp0s8  |   U   |
[  127.092220] cloud-init[753]: ci-info: |   3   |    local    |    ::   |   enp0s8  |   U   |
[  127.098290] cloud-init[753]: ci-info: |   4   |  multicast  |    ::   |   enp0s8  |   U   |
[  127.104422] cloud-init[753]: ci-info: +-------+-------------+---------+-----------+-------+
[  OK  ] Finished Cloud-init: Network Stage.
[  OK  ] Reached target Cloud-config availability.
[  OK  ] Reached target Network is Online.

so they may be able to tell what is wrong. The vnic problem can be resolved with instance stop-start operations.

Since it's an older linux version, it may not be as user-impacting as the situation prior to the #9828 fix.

[leftwo]

I'm running a focal instance on London, with the propolis from (now merged) PR 1047.
The focal instance initially came up okay, but after a reboot it also reported the same problem:

[  OK  ] Reached target Network.
[  OK  ] Reached target Host and Network Name Lookups.
[FAILED] Failed to start Wait for Network to be Configured.
See 'systemctl status systemd-networkd-wait-online.service' for details.
         Starting Initial cloud-ini… (metadata service crawler)...

[leftwo]

I've run through a few different situations.

So far for me, it's only the 2nd boot of a brand new jammy instance that does not get a working network.
Follow on reboots do get working network. This initial reboot is performed from inside the guest with the "reboot" command.

If I stop then start the instance after the initial creation, the network does work (does not show a problem).

If I snapshot the root disk after a 2nd reboot of my jammy image, and create an instance with the snapshot, that new instance has working network after a reboot, so once the network is working, it looks like it continues working.

[leftwo]

On dogfood, my 24.04 instance showed this issue.
1st boot, good.
2nd boot (reboot from the guest) network not working.
3rd boot (reboot from the guest) good
4th boot (reboot from the guest) good

ubuntu@aaatest1:~$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

[iximeow]

I've got a patch up at #1051 that we've given a quick check on London (I built a Propolis with it and swapped out propolis-server.tar.gz with a patched one). Alan's VM seems OK there, and I'll let him say if he's tested other images since. (my desktop has a wedged bhyve for other reasons that we really should understand, so I'm a bit hesitant to restart it at the moment..)

I'm not entirely sure how I didn't see this when validating 1df49a4. one may ask, reasonably, what is the point of tracking the peak value of SET_USEPAIR for later resets? at least from looking at Alan's Ubuntu VM we can see that enabling multiqueue gets SET_USEPAIR to 0xb, but then later the guest specifically goes down to four pairs:

$ rg SET_USEPAIR
watchme_first_boot
51:SET_USEPAIRS 0xb
276:SET_USEPAIRS 0x4

watchme_first_reboot
46:SET_USEPAIRS 0x1
179:SET_USEPAIRS 0xb
204:SET_USEPAIRS 0x4

watchme_second_reboot
28:SET_USEPAIRS 0x1
70:SET_USEPAIRS 0xb
295:SET_USEPAIRS 0x4

watchme_third_reboot
43:SET_USEPAIRS 0x1
85:SET_USEPAIRS 0xb
310:SET_USEPAIRS 0x4

so we do need to track that we should reset all pairs before a future SET_PAIRS (or that'll EBUSY and get the device into NEEDS_RESET for that different reason...)

the biggest confusion, to me, aside from why I didn't see this in reboots without a boot device (so that OVMF would have been doing the ill-fated reset of 0xb pairs with us only having ioctl'd SET_PAIRS to 1), is why Alan's VM is only doing that for only the first reboot:

iximeow@castle /staff/core/propolis-1049 $ rg 'SET_FEATURE'
watchme_first_boot
49:SET_FEATURES 0x1104388ab
272:SET_FEATURES 0x1104388ab

watchme_first_reboot
104:SET_FEATURES 0x10020           # very weird!
177:SET_FEATURES 0x1104388ab
200:SET_FEATURES 0x1104388ab

watchme_second_reboot
68:SET_FEATURES 0x1104388ab
291:SET_FEATURES 0x1104388ab

watchme_third_reboot
83:SET_FEATURES 0x1104388ab
306:SET_FEATURES 0x1104388ab

I noticed back in the boot order stuff that Ubuntu messes with boot order settings, so and I've never tried to understand what the exact mechanism is there, but I suspect that would play a role. efibootmgr should be able to tell us what the boot settings are to validate that. but given this seems to be a further improvement to the state of things and the extant behavior is clearly wrong, I'm OK leaving that as a follow up question..

[iximeow]

with #1051 merged I think we can close this, but I'm going to wait until we've actually seen Angela's jammy image consistently have networking...