Support Bundle: Automatic Deletion

[smklein]

Welp, we're talking in the FMA sync about doing auto-collection of support bundles (yay) but this means we're going to be automatically triggering creation of a resource that still requires manual deletion (boo).

This will inevitably result in an out-of-space error. Ergo: This issue tracks "auto-deletion" of older bundles, within some reasonable bounds.

[smklein]

Here's my proposal:

• Add configs for a "minimum number of bundles to keep", to avoid an overly aggressive auto-deletion systems (and to avoid problems on small deployments)
• Add configs for a "target free bundle buffer" - on a system with D debug datasets, where bundles may be stored, we could ask for a "free bundle buffer" of count N. If the number of active bundles is larger than "D - N", we can start deleting the oldest active bundles automatically
• Perform this pruning as the first step of the support bundle collection task, marking active bundles as Destroying. This matches what would happen if these bundles were manually deleted via the HTTP API.

The "free bundle buffer" size means we're effectively storing N fewer bundles on the rack, but it also means "we'll try to make space for N new bundles between each execution of the support bundle collection background task".

Practically, in a rack with with 16 sleds + 10 U.2 disks on each sled, that's roughly ~160 debug datasets where we can store bundles. Trying to keep ~16 bundles free (10% for a half-populated rack, 5% for a fully-populated rack) seems like a reasonable starting point for most of our deployments? (I'm thinking I'd store this as a build-time variable within the Nexus configuration, but we could store it in the database and make it more dynamic as needed).

[hawkw]

Add configs for a "minimum number of bundles to keep", to avoid an overly aggressive auto-deletion systems (and to avoid problems on small deployments)

Would these configs be set in the Nexus config file (i.e. by Oxide), or configured by the operator? 'Cause I can kinda imagine an argument for the operator to be able to turn this knob someday.

[smklein]

I have implemented a version of this with the former (... honestly, because it seemed like less work) but I agree, it seems fine to allow some dynamicism to these values.

Idk if this is too in the weeds, but: it kinda seems like these values should be scaling a bit with the size of the deployment? So I 100% buy the argument that they could be a little flexible (e.g., in a massive, multi-rack deployment, "16 free" might not be enough) but I'm not sure that the operator should need to be aware of this detail to make that correct?

To summary: Would be okay making it an operator knob, I'm just not sure how I'd reasonably advise someone to tweak these values, when the answer feels a bit like "well, we know what these should be, so we're going to tweak them automatically for you".