Enhancement: Automatic Rolling Reservoir & Prompt Retirement

[sharmaanchita]

Problem
The system lacks an automated mechanism to manage prompt lifecycle (active → retired → public). Currently, prompts persist indefinitely unless manually revealed, undermining contamination resistance.

Basis of issue

1. Automatic prompt retirement based on:
   • Age
   • Low sampling weight
   • Exposure thresholds
2. Fixed-capacity rolling reservoir (k) for live prompts
3. Automated state transitions: secret → retired → public
4. Longitudinal contamination archive for retired prompts

Importance of issue

1. Core requirement for contamination-resistant benchmarks (per paper)
2. Prevents benchmark staleness and training leakage
3. Maintains discriminative power over time

Current Implementation Gap

1. isRevealed flag exists in DB
2. No automated triggers or lifecycle orchestration

Implementation checklist

1. Background job enforces reservoir capacity
2. Prompts auto-retire based on defined policies
3. State transitions are logged and auditable
4. Retired prompts are excluded from live evaluation

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🔗 Similar Issues

Related Issues

• Implement Prompt Versioning with Immutable Benchmark References and Result Diffing #53
• Enhancement: Add prompt metadata and failure reason annotation for better benchmarking analysis #51
• On prompt creation automatically mark the author as someone leaving positive feedback on the prompt #44

👤 Suggested Assignees

• SanthoshPradeep
• atharvayeole1912-sys
• mistrz-g

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord or schedule a call!