[inventory] Add unhealthy zpools from each sled

[karencfv]

This is the second PR for #9412

This commit adds a list of unhealthy zpools to the sled agent inventory's health monitor. In a follow-up PR this information will be added to the DB

Successful unhealthy zpool retrieval:

$ zpool list -Hpo health,name
FAULTED fakepool1
FAULTED fakepool2
ONLINE  rpool

$ curl -H "api-version: 15.0.0"  http://[::1]:54963/inventory | jq
<...>
  "health_monitor": {
    "smf_services_in_maintenance": {
      "ok": {
        "services": [
          {
            "fmri": "svc:/site/fake-service2:default",
            "zone": "global"
          }
        ],
        "errors": [],
        "time_of_status": "2026-01-12T07:15:06.913644164Z"
      }
    },
    "unhealthy_zpools": {
      "ok": {
        "zpools": [
          "fakepool1",
          "fakepool2"
        ],
        "errors": [],
        "time_of_status": "2026-01-12T07:15:06.888312628Z"
      }
    }
  }

Response contains errors:

$ curl -H "api-version: 15.0.0"  http://[::1]:54963/inventory | jq
<...>
  "health_monitor": {
    "smf_services_in_maintenance": {
      "ok": {
        "services": [],
        "errors": [],
        "time_of_status": "2026-01-12T07:10:00.642206104Z"
      }
    },
    "unhealthy_zpools": {
      "ok": {
        "zpools": [],
        "errors": [
          "Failed to parse output: Unrecognized zpool 'health': fakepool1",
          "Failed to parse output: Unrecognized zpool 'health': fakepool2",
          "Failed to parse output: Unrecognized zpool 'health': rpool"
        ],
        "time_of_status": "2026-01-12T07:10:00.591985515Z"
      }
    }
  }

[jgallagher]

Does this need to be optional? If we have a result, it must have been collected at some time, right?

[karencfv]

This is meant for when this runs on non-illumos environements

https://github.com/oxidecomputer/omicron/pull/9615/files#diff-07cfb91c0643fac6671e8b81317c92cbe806ab07e2af6fdce16db75315245e32R446-R453

The idea is that there is no time of status because no command was ever run.

[karencfv]

Also there is a point in time where health monitor may not have run yet -> #9589 (comment)

[karencfv]

I went back and forth between just having a list of unhealthy zpools, or associating each zpool with it's state. In the end I went with listing the zpools only, but I'm not convinced. We'll be including the information of the health checks in the support bundle, and it'd be useful for them to be able to see what state each zpool is in. Thoughts? @davepacheco @jgallagher

[jgallagher]

Associating each zpool with its state sounds good to me; having an explicit entry for "this zpool was healthy" seems safer than inferring "any zpool that isn't explicitly listed as unhealthy must have been healthy".

[karencfv]

Hmmm, I was thinking of only including the "unhealthy" zpools with their associated statuses in this list. Similarly with the svcs_in_maintenance I only added the services in maintenance with their associated zones. If I were to include the "healthy" zpools then it wouldn't really be consistent with the services in maintenance no?

My take on the health checks is to only report on things that are in an unhealthy state. Thoughts?