TQ: Support adding sleds via trust quorum #9650

andrewjstone · 2026-01-14T21:47:26Z

This PR introduces two new external APIs to allow adding multiple sleds to a rack at once and to query status about the ongoing operation. It also adds an omdb command for more detailed status. Much more omdb to come in the near future.

This PR also introduces a background task for driving the trust quorum reconfiguration to completion. Reconfiguration is driven by two steps. Synchronously updating the DB in the new external endpoint handler and then asynchronously trying to commit the operation via the background task.

I tested this on a4x2 and it works as expected. See the trace from the original external API test below:

➜  oxide.rs git:(main) ✗ echo '{"rack_id": "0dbef452-a6dd-4831-bbdc-769ea3353f28", "sled_ids": [{"part": "PPP-PPPPPPP","serial": "00000000002"}]}' | target/debug/oxide --profile recovery api /v1/trust-quorum/new-members --method POST --input -
➜  oxide.rs git:(main) ✗ target/debug/oxide --profile recovery  api /v1/trust-quorum/config/latest/0dbef452-a6dd-4831-bbdc-769ea3353f28
{
  "abort_reason": null,
  "commit_crash_tolerance": 1,
  "coordinator": {
    "part_number": "PPP-PPPPPPP",
    "serial_number": "00000000003"
  },
  "encrypted_rack_secrets": null,
  "epoch": 2,
  "last_committed_epoch": 1,
  "members": {
    "PPP-PPPPPPP:00000000000": {
      "share_digest": null,
      "state": "unacked",
      "time_committed": null,
      "time_prepared": null
    },
    "PPP-PPPPPPP:00000000001": {
      "share_digest": null,
      "state": "unacked",
      "time_committed": null,
      "time_prepared": null
    },
    "PPP-PPPPPPP:00000000002": {
      "share_digest": null,
      "state": "unacked",
      "time_committed": null,
      "time_prepared": null
    },
    "PPP-PPPPPPP:00000000003": {
      "share_digest": null,
      "state": "unacked",
      "time_committed": null,
      "time_prepared": null
    }
  },
  "rack_id": "0dbef452-a6dd-4831-bbdc-769ea3353f28",
  "state": "preparing",
  "threshold": 3,
  "time_aborted": null,
  "time_committed": null,
  "time_committing": null,
  "time_created": "2026-01-14T21:32:18.780136Z"
}
➜  oxide.rs git:(main) ✗ target/debug/oxide --profile recovery  api /v1/trust-quorum/config/latest/0dbef452-a6dd-4831-bbdc-769ea3353f28
{
  "abort_reason": null,
  "commit_crash_tolerance": 1,
  "coordinator": {
    "part_number": "PPP-PPPPPPP",
    "serial_number": "00000000003"
  },
  "encrypted_rack_secrets": null,
  "epoch": 2,
  "last_committed_epoch": 1,
  "members": {
    "PPP-PPPPPPP:00000000000": {
      "share_digest": "fcfb09128c84d82cc81b200c6c682510f63160a4417856f4041b1886445e8b14",
      "state": "prepared",
      "time_committed": null,
      "time_prepared": "2026-01-14T21:32:55.826622Z"
    },
    "PPP-PPPPPPP:00000000001": {
      "share_digest": "d8cad02bd3bccd08109a79e3bf6d8dab0d460a0ba879bf42887dc0fc8d855786",
      "state": "prepared",
      "time_committed": null,
      "time_prepared": "2026-01-14T21:32:55.848235Z"
    },
    "PPP-PPPPPPP:00000000002": {
      "share_digest": "dd57ad8e271734fabfe97d6180d6da3e5c3805e17dacf58e0f2a6d5ed7f1242b",
      "state": "prepared",
      "time_committed": null,
      "time_prepared": "2026-01-14T21:32:55.806644Z"
    },
    "PPP-PPPPPPP:00000000003": {
      "share_digest": "6b27327ca49976ccca83972e6578ef195c99489e62811e8d0a0cb061fca9c0c4",
      "state": "prepared",
      "time_committed": null,
      "time_prepared": "2026-01-14T21:32:55.837154Z"
    }
  },
  "rack_id": "0dbef452-a6dd-4831-bbdc-769ea3353f28",
  "state": "preparing",
  "threshold": 3,
  "time_aborted": null,
  "time_committed": null,
  "time_committing": null,
  "time_created": "2026-01-14T21:32:18.780136Z"
}
➜  oxide.rs git:(main) ✗ target/debug/oxide --profile recovery  api /v1/trust-quorum/config/latest/0dbef452-a6dd-4831-bbdc-769ea3353f28
{
  "abort_reason": null,
  "commit_crash_tolerance": 1,
  "coordinator": {
    "part_number": "PPP-PPPPPPP",
    "serial_number": "00000000003"
  },
  "encrypted_rack_secrets": {
    "data": "53de7731deec3f298a7f5067e256a63bb2869a91c9710d9b23dbf3d261d1b730039d9cb11b543c14906ff77cd409d32953959e9ff8933858",
    "salt": "ec609ed5ff7aee94e2e88ad94af56e0cbb8a66a683294005c7888f60a627956a"
  },
  "epoch": 2,
  "last_committed_epoch": 1,
  "members": {
    "PPP-PPPPPPP:00000000000": {
      "share_digest": "fcfb09128c84d82cc81b200c6c682510f63160a4417856f4041b1886445e8b14",
      "state": "committed",
      "time_committed": "2026-01-14T21:33:03.864617Z",
      "time_prepared": "2026-01-14T21:32:55.826622Z"
    },
    "PPP-PPPPPPP:00000000001": {
      "share_digest": "d8cad02bd3bccd08109a79e3bf6d8dab0d460a0ba879bf42887dc0fc8d855786",
      "state": "committed",
      "time_committed": "2026-01-14T21:33:03.864617Z",
      "time_prepared": "2026-01-14T21:32:55.848235Z"
    },
    "PPP-PPPPPPP:00000000002": {
      "share_digest": "dd57ad8e271734fabfe97d6180d6da3e5c3805e17dacf58e0f2a6d5ed7f1242b",
      "state": "committed",
      "time_committed": "2026-01-14T21:33:03.864617Z",
      "time_prepared": "2026-01-14T21:32:55.806644Z"
    },
    "PPP-PPPPPPP:00000000003": {
      "share_digest": "6b27327ca49976ccca83972e6578ef195c99489e62811e8d0a0cb061fca9c0c4",
      "state": "committed",
      "time_committed": "2026-01-14T21:33:03.864617Z",
      "time_prepared": "2026-01-14T21:32:55.837154Z"
    }
  },
  "rack_id": "0dbef452-a6dd-4831-bbdc-769ea3353f28",
  "state": "committed",
  "threshold": 3,
  "time_aborted": null,
  "time_committed": "2026-01-14T21:33:04.652543Z",
  "time_committing": "2026-01-14T21:32:55.861158Z",
  "time_created": "2026-01-14T21:32:18.780136Z"
}
➜  oxide.rs git:(main) ✗

This PR introduces two new external APIs to allow adding multiple sleds to a rack at once and to query status about the ongoing operation. Both are currently experimental and live under `/v1/trust-quorum`. They need to be moved under `/system/hardware` like the original `sled-add` command. They also need to be reworked to not report trust quorum specific details if it can be avoided. Most of that should be in omdb for debugging. I may add some of that support to this PR. This PR also introduces a background task for driving the trust quorum reconfiguration to completion. Reconfiguration is driven by two steps. Synchronously updating the DB in the new external endpoint handler and then asynchronously trying to commit the operation via the background task. I tested this on a4x2 and it works as expected. See the trace from the original external API test below: ``` ➜ oxide.rs git:(main) ✗ echo '{"rack_id": "0dbef452-a6dd-4831-bbdc-769ea3353f28", "sled_ids": [{"part": "PPP-PPPPPPP","serial": "00000000002"}]}' | target/debug/oxide --profile recovery api /v1/trust-quorum/new-members --method POST --input - ➜ oxide.rs git:(main) ✗ target/debug/oxide --profile recovery api /v1/trust-quorum/config/latest/0dbef452-a6dd-4831-bbdc-769ea3353f28 { "abort_reason": null, "commit_crash_tolerance": 1, "coordinator": { "part_number": "PPP-PPPPPPP", "serial_number": "00000000003" }, "encrypted_rack_secrets": null, "epoch": 2, "last_committed_epoch": 1, "members": { "PPP-PPPPPPP:00000000000": { "share_digest": null, "state": "unacked", "time_committed": null, "time_prepared": null }, "PPP-PPPPPPP:00000000001": { "share_digest": null, "state": "unacked", "time_committed": null, "time_prepared": null }, "PPP-PPPPPPP:00000000002": { "share_digest": null, "state": "unacked", "time_committed": null, "time_prepared": null }, "PPP-PPPPPPP:00000000003": { "share_digest": null, "state": "unacked", "time_committed": null, "time_prepared": null } }, "rack_id": "0dbef452-a6dd-4831-bbdc-769ea3353f28", "state": "preparing", "threshold": 3, "time_aborted": null, "time_committed": null, "time_committing": null, "time_created": "2026-01-14T21:32:18.780136Z" } ➜ oxide.rs git:(main) ✗ target/debug/oxide --profile recovery api /v1/trust-quorum/config/latest/0dbef452-a6dd-4831-bbdc-769ea3353f28 { "abort_reason": null, "commit_crash_tolerance": 1, "coordinator": { "part_number": "PPP-PPPPPPP", "serial_number": "00000000003" }, "encrypted_rack_secrets": null, "epoch": 2, "last_committed_epoch": 1, "members": { "PPP-PPPPPPP:00000000000": { "share_digest": "fcfb09128c84d82cc81b200c6c682510f63160a4417856f4041b1886445e8b14", "state": "prepared", "time_committed": null, "time_prepared": "2026-01-14T21:32:55.826622Z" }, "PPP-PPPPPPP:00000000001": { "share_digest": "d8cad02bd3bccd08109a79e3bf6d8dab0d460a0ba879bf42887dc0fc8d855786", "state": "prepared", "time_committed": null, "time_prepared": "2026-01-14T21:32:55.848235Z" }, "PPP-PPPPPPP:00000000002": { "share_digest": "dd57ad8e271734fabfe97d6180d6da3e5c3805e17dacf58e0f2a6d5ed7f1242b", "state": "prepared", "time_committed": null, "time_prepared": "2026-01-14T21:32:55.806644Z" }, "PPP-PPPPPPP:00000000003": { "share_digest": "6b27327ca49976ccca83972e6578ef195c99489e62811e8d0a0cb061fca9c0c4", "state": "prepared", "time_committed": null, "time_prepared": "2026-01-14T21:32:55.837154Z" } }, "rack_id": "0dbef452-a6dd-4831-bbdc-769ea3353f28", "state": "preparing", "threshold": 3, "time_aborted": null, "time_committed": null, "time_committing": null, "time_created": "2026-01-14T21:32:18.780136Z" } ➜ oxide.rs git:(main) ✗ target/debug/oxide --profile recovery api /v1/trust-quorum/config/latest/0dbef452-a6dd-4831-bbdc-769ea3353f28 { "abort_reason": null, "commit_crash_tolerance": 1, "coordinator": { "part_number": "PPP-PPPPPPP", "serial_number": "00000000003" }, "encrypted_rack_secrets": { "data": "53de7731deec3f298a7f5067e256a63bb2869a91c9710d9b23dbf3d261d1b730039d9cb11b543c14906ff77cd409d32953959e9ff8933858", "salt": "ec609ed5ff7aee94e2e88ad94af56e0cbb8a66a683294005c7888f60a627956a" }, "epoch": 2, "last_committed_epoch": 1, "members": { "PPP-PPPPPPP:00000000000": { "share_digest": "fcfb09128c84d82cc81b200c6c682510f63160a4417856f4041b1886445e8b14", "state": "committed", "time_committed": "2026-01-14T21:33:03.864617Z", "time_prepared": "2026-01-14T21:32:55.826622Z" }, "PPP-PPPPPPP:00000000001": { "share_digest": "d8cad02bd3bccd08109a79e3bf6d8dab0d460a0ba879bf42887dc0fc8d855786", "state": "committed", "time_committed": "2026-01-14T21:33:03.864617Z", "time_prepared": "2026-01-14T21:32:55.848235Z" }, "PPP-PPPPPPP:00000000002": { "share_digest": "dd57ad8e271734fabfe97d6180d6da3e5c3805e17dacf58e0f2a6d5ed7f1242b", "state": "committed", "time_committed": "2026-01-14T21:33:03.864617Z", "time_prepared": "2026-01-14T21:32:55.806644Z" }, "PPP-PPPPPPP:00000000003": { "share_digest": "6b27327ca49976ccca83972e6578ef195c99489e62811e8d0a0cb061fca9c0c4", "state": "committed", "time_committed": "2026-01-14T21:33:03.864617Z", "time_prepared": "2026-01-14T21:32:55.837154Z" } }, "rack_id": "0dbef452-a6dd-4831-bbdc-769ea3353f28", "state": "committed", "threshold": 3, "time_aborted": null, "time_committed": "2026-01-14T21:33:04.652543Z", "time_committing": "2026-01-14T21:32:55.861158Z", "time_created": "2026-01-14T21:32:18.780136Z" } ➜ oxide.rs git:(main) ✗ ```

andrewjstone · 2026-01-15T03:12:03Z

sled-agent/src/bootstrap/trust_quorum_setup.rs

 /// testing.
-pub const TRUST_QUORUM_INTEGRATION_ENABLED: bool = false;
+//pub const TRUST_QUORUM_INTEGRATION_ENABLED: bool = false;
+pub const TRUST_QUORUM_INTEGRATION_ENABLED: bool = true;


Reset this before merge.

jgallagher

Looks good! Just a bunch of nits and small suggestions

dev-tools/omdb/src/bin/omdb/nexus.rs

nexus/db-queries/src/db/datastore/sled.rs

jgallagher · 2026-01-15T18:31:07Z

nexus/external-api/src/lib.rs

+    async fn trust_quorum_get_latest_config(
+        rqctx: RequestContext<Self::Context>,
+        path_params: Path<params::RackPath>,
+    ) -> Result<HttpResponseOk<Option<RackMembershipChange>>, HttpError>;


Very happy to defer to others with more opinions on the external API, but a few questions based mostly on the update sync this week:

Will the trust_quorum_... bits of these names leak out into the OpenAPI spec? (I think so, because we default the operation ID to match the method name?)

I'm a little surprised this is a "get latest" and not "get the result of an operation I started", but maybe I misunderstood? I thought we wanted something like "add sled is async, and returns an identifier that can be used to check the progress of the operation".

Are all the fields of RackMembershipChange meaningful to an operator? (I'm mostly squinting at epoch, but maybe that's closely related to the previous bullet.)

Ah, so I left the method names the same, but changed the endpoints. I did not realize that the method names leaked.

I really like the return a token / identifier mechanism in general, and it could possibly work here. However there are two wrinkles:

The initial request could timeout and the trust quorum reconfiguration could be in progress or even committed by the time the user polled. What would they poll with if they didn't get back the token? I suppose they could ask for a token for the latest configuration, but that brings up the next point.

There's an inherent TOCTTOU here, where the user can see that their configuration committed but another user could have started a later one. Maybe that's not a problem and the user is only concerned about their own and can always ask for a token back.

While writing this, I think you sold me on the token idea. The user will submit a request and get back the epoch as the token for the configuration. Then they will poll that epoch. We will also provide another api to get the latest epoch.

I think this all takes me to your last question. In this case the epoch is the identifier to know which configuration a user is dealing with. I could change this to version or generation in the API and map that to an Epoch, but I think that will generally make things more confusing for support. I personally just want to call it epoch, and not try to map the same concept to a different word :)

Fixed up in 9832d33

nexus/types/src/external_api/views.rs

nexus/src/app/background/tasks/trust_quorum.rs

andrewjstone · 2026-01-15T23:19:04Z

Thanks for the comprehensive review @jgallagher. I think I fixed up everything. Let me know if you see anything else!

jgallagher

Changes LGTM, just a couple minor style suggestions I didn't notice the first time around.

Happy to approve with a couple caveats:

Your comment about resetting TRUST_QUORUM_INTEGRATION_ENABLED still needs to be applied
Would strongly prefer an external-API-focused set of eyes to review those bits (maybe @ahl?)

jgallagher · 2026-01-16T15:06:21Z

nexus/external-api/src/lib.rs

+        rqctx: RequestContext<Self::Context>,
+        path_params: Path<params::RackPath>,
+        req: TypedBody<params::AddSledsRequest>,
+    ) -> Result<HttpResponseOk<Epoch>, HttpError>;


Do the type names leak out into the API doc? If so, maybe RackMembershipEpoch would be more clear? If not, ignore this.

No leaking as far as I can tell.

"properties": { "epoch": { "description": "The generation / version of the configuration", "type": "integer", "format": "uint64", "minimum": 0 },

nexus/types/src/external_api/views.rs

nexus/src/app/background/tasks/trust_quorum.rs

andrewjstone · 2026-01-17T03:49:09Z

Tested in a4x2 with all the latest changes.

➜  oxide.rs git:(main) ✗ echo '{"sled_ids": [{"part": "PPP-PPPPPPP","serial": "00000000002"}]}' |  target/debug/oxide --profile recovery api /v1/system/hardware/racks/3df0b7cd-50ed-498e-afa4-41f178d393f9/sleds --method POST --input -
2
➜  oxide.rs git:(main) ✗ target/debug/oxide --profile recovery api /v1/system/hardware/racks/3df0b7cd-50ed-498e-afa4-41f178d393f9/sleds/2
{
  "epoch": 2,
  "members": [
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000000"
    },
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000001"
    },
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000002"
    },
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000003"
    }
  ],
  "rack_id": "3df0b7cd-50ed-498e-afa4-41f178d393f9",
  "state": "committed",
  "time_aborted": null,
  "time_committed": "2026-01-17T03:45:33.946797Z",
  "time_created": "2026-01-17T03:45:02.262102Z",
  "unacknowledged_members": []
}

I even was able to catch the bg task doing something in OMDB:

root@oxz_switch:~# omdb nexus --nexus-internal-url='http://[fd00:17:1:d01::6]:12232' background-tasks show trust_quorum_manager
note: using Nexus URL http://[fd00:17:1:d01::6]:12232
task: "trust_quorum_manager"
  configured period: every 1m
  currently executing: no
  last completed activation: iter 19, triggered by a periodic timer firing
    started at 2026-01-17T03:45:26.931Z (34s ago) and ran for 7643ms
Trust Quorum status for rack_id: 3df0b7cd-50ed-498e-afa4-41f178d393f9, epoch: 2:
    Commit Status from client PPP-PPPPPPP:00000000003:
      acked: PPP-PPPPPPP:00000000000
      pending:
      errors:
    Commit Status from client PPP-PPPPPPP:00000000001:
      acked: PPP-PPPPPPP:00000000001
      pending:
      errors:
    Commit Status from client PPP-PPPPPPP:00000000000:
      acked: PPP-PPPPPPP:00000000002, PPP-PPPPPPP:00000000003
      pending:
      errors:

All that remains is for someone to take a look over the external API.

andrewjstone force-pushed the tq-nexus-api-2 branch from 04173b6 to 2d8fc67 Compare January 14, 2026 23:41

andrewjstone added 4 commits January 15, 2026 00:55

Migrate get api to lockstep and rename some paths

0e6e070

Add omdb status

f335e3f

Make membership status less trust quorum specific

e059b6b

remove unrelated comment

33d05d2

andrewjstone commented Jan 15, 2026

View reviewed changes

andrewjstone changed the title ~~WIP: TQ: Support adding sleds via trust quorum~~ TQ: Support adding sleds via trust quorum Jan 15, 2026

andrewjstone marked this pull request as ready for review January 15, 2026 16:43

andrewjstone requested review from davepacheco and jgallagher January 15, 2026 16:43

andrewjstone added 2 commits January 15, 2026 16:44

remove stale comment

8c5d5bb

clippy

f722d9f

jgallagher reviewed Jan 15, 2026

View reviewed changes

andrewjstone commented Jan 15, 2026

View reviewed changes

nexus/src/app/background/tasks/trust_quorum.rs Outdated Show resolved Hide resolved

andrewjstone added 5 commits January 15, 2026 21:02

Actually commit bg task config changes

4cc270e

Better external API

9832d33

Fix return value and errors for bgtask/omdb

20787fc

more review fixes

f0d0ed7

fill in missing nexus config

a77313c

jgallagher approved these changes Jan 16, 2026

View reviewed changes

review updates and fix tests

3c883c1

TQ: Support adding sleds via trust quorum #9650

Are you sure you want to change the base?

TQ: Support adding sleds via trust quorum #9650

Uh oh!

Conversation

andrewjstone commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andrewjstone Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

jgallagher left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jgallagher Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

andrewjstone Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewjstone Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andrewjstone commented Jan 15, 2026

Uh oh!

jgallagher left a comment

Choose a reason for hiding this comment

Uh oh!

jgallagher Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

andrewjstone Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andrewjstone commented Jan 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

andrewjstone commented Jan 14, 2026 •

edited

Loading

andrewjstone Jan 15, 2026 •

edited

Loading

andrewjstone Jan 16, 2026 •

edited

Loading