New idea: Sealed Execution Environment for Model Evaluation

[sharmaanchita]

Problem
Models are evaluated via direct API calls, exposing prompts to providers and allowing potential memorization or training leakage.

Basis of issue

1. Isolated execution environment (sandboxed SDK or enclave)
2. Network egress restrictions during inference
3. Prevention of prompt visibility prior to scoring
4. Secure prompt delivery and response capture

Importance

1. Central security guarantee of the paper
2. Prevents prompt harvesting by model providers
3. Without this, contamination resistance is fundamentally broken

Current Implementation Gap

1. Models access prompts via OpenRouter / direct APIs
2. No isolation or prompt secrecy guarantees

Implementation checklist

1. Prompts executed inside a sealed environment
2. No outbound network access during inference
3. Model providers cannot log or store prompts
4. Scoring occurs post-execution, not inline

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🔗 Similar Issues

Related Issues

• Enhancement: Automatic Rolling Reservoir & Prompt Retirement #58
• Implement Prompt Versioning with Immutable Benchmark References and Result Diffing #53
• Benchmark leaderboards become hard to interpret when most models score near-perfect #48

👤 Suggested Assignees

• sharmaanchita
• SanthoshPradeep
• Meghhanaa

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord or schedule a call!

[reisepass]

You mean kinda like how dynabench does it https://github.com/mlcommons/dynabench/blob/main/docs/overview.md or directly jumping into TEEs like https://docs.near.ai/cloud/private-inference/ ?

We have not done this yet because most of the top models are not available to run on any other environment other than the owner.