Support bundle: automatic deletion to maintain free dataset buffer #9662
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
smklein
force-pushed
the
support-bundle-auto-delete
branch
from
5bc38cc to
c78018e
Compare
smklein
force-pushed
the
support-bundle-auto-delete
branch
from
c78018e to
1c94bc0
Compare
- Use COUNT queries instead of loading all dataset/bundle rows - Use GROUP BY to get used_datasets and active_bundles in a single query - Add LIMIT to the deletion candidates query - Exclude Failed bundles from used_datasets (their dataset was expunged) These changes allow the auto-deletion query to scale to systems with thousands of datasets without loading all rows into memory.
Replace the two-step find-then-update approach with a single atomic CTE query that calculates how many deletions are needed AND performs them in one database operation. The previous approach had a time-of-check to time-of-use (TOCTTOU) issue where multiple Nexuses running concurrently could over-delete bundles: 1. Nexus A queries: free=2, needs 1 deletion -> gets B1 2. Nexus A transitions B1: Active -> Destroying 3. Nexus B queries: free=2 (unchanged, Destroying still occupies), needs 1 -> gets B2 4. Nexus B transitions B2: Active -> Destroying 5. Result: 2 bundles deleted when only 1 was needed The new atomic query: - Calculates free datasets and needed deletions - Respects min_bundles_to_keep constraint - Finds the N oldest Active bundles - Transitions them to Destroying state atomically - Returns the IDs of deleted bundles Also adds tests verifying: - Bundles are actually transitioned to Destroying state - Failed bundles don't count as occupying datasets - Explain test for SQL validity - Expectorate test for SQL output
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Implements automatic deletion of support bundles to maintain a buffer of free debug datasets for new allocations.
target_free_datasetsandmin_bundles_to_keep(Option<NonZeroU32>)support_bundle_auto_delete()CTE + unit testsauto_delete_bundles()phase before existing phasesHow it works
The
SupportBundleCollectorbackground task now runs three phases:Deletion respects
min_bundles_to_keepto protect small systems from aggressive cleanup.Tests
test_auto_deletion_no_bundles- No deletion when there are enough free datasets and no bundles existtest_auto_deletion_enough_free_datasets- No deletion when free datasets already meet the targettest_auto_deletion_deletes_oldest_first- Verifies oldest bundles (by time_created) are selected for deletiontest_auto_deletion_respects_min_bundles_to_keep- Deletion is limited when it would leave fewer thanmin_bundles_to_keepbundlestest_auto_deletion_min_bundles_prevents_all_deletion- No deletion whenmin_bundles_to_keepexceeds active bundle counttest_auto_deletion_only_selects_active_bundles- Only Active bundles are deleted; Collecting/Destroying bundles are skipped but still count as occupying datasetstest_auto_deletion_verifies_state_transition- Verifies bundles are actually transitioned to Destroying state in the databasetest_auto_deletion_failed_bundles_dont_occupy_datasets- Failed bundles don't count toward used dataset count (their dataset was expunged)test_auto_delete_query_explains- Validates the CTE is valid SQL via EXPLAINexpectorate_auto_delete_query- Captures SQL output for inspection/change detectiontest_support_bundle_auto_deletion()Fixes #9660