Enhancement: Synchronized Cohort-Based Evaluation

[sharmaanchita]

Problem
All evaluations are currently asynchronous, making scores incomparable across models and time.

Basis of issue

1. Time-boxed evaluation rounds (e.g., quarterly cohorts)
2. Shared prompt batches per cohort
3. Cohort registration and locking mechanism
4. Split evaluation budget (e.g., 70% cohort / 30% immediate)

Importance

1. Enables fair, head-to-head comparisons
2. Reduces contamination surface
3. Aligns with paper’s benchmarking methodology

Current Implementation gap

1. Immediate scoring only
2. No cohort coordination

Implementation Checklist

1. Cohort scheduler with fixed evaluation windows
2. Models in a cohort evaluated on identical prompt sets
3. Cohort scores stored and compared independently
4. Immediate scoring remains available but limited

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🔗 Similar Issues

Related Issues

• New idea: Sealed Execution Environment for Model Evaluation #59
• Enhancement: Automatic Rolling Reservoir & Prompt Retirement #58
• Implement Prompt Versioning with Immutable Benchmark References and Result Diffing #53
• Benchmark leaderboards become hard to interpret when most models score near-perfect #48
• Expand Benchmark Create/Edit page #42

👤 Suggested Assignees

• sharmaanchita
• SanthoshPradeep
• Meghhanaa
• mistrz-g

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord or schedule a call!