Benchmark leaderboards become hard to interpret when most models score near-perfect

[Meghhanaa]

While exploring PeerBench benchmarks (for example, Logical Puzzles – Polish), I noticed that most modern models achieve almost perfect scores on nearly all prompts.

Since many of these prompts are taken from well-known, published puzzle books, it is very likely that models have already seen similar data during training. Because of this, the leaderboard does not clearly reflect real reasoning differences between models.

Why this matters

• PeerBench aims to provide fair and cheat-proof AI benchmarking. However, when benchmarks become saturated:
• Leaderboards lose their ability to differentiate models
• Small score differences may appear meaningful when they are not
• Users may misinterpret results as true performance gains
• Without any indicator, it is difficult for users to judge the reliability of benchmark results.

Suggested Improvement

It would be helpful to add simple benchmark quality signals, such as:

• A warning when most models score near-perfect
• An indicator that prompts come from well-known or published sources
• A short explanation that results may reflect memorization rather than reasoning
• This additional context would help users better understand leaderboard results.

Expected Impact

• Better interpretation of benchmark results
• Increased trust in PeerBench evaluations
• Stronger alignment with the project’s goal of meaningful benchmarking

[reisepass]

One of the core ideas of peerbench is that people can keep parts of the data hidden so there is no way for LLM's to train on it. Not all benchmarks have done this yet, maybe an idea would be to enforce it that no one can be fully open.

So something to display would be what % of the data is hidden and if all is open then its filtered out by defualt or has a flag.

---

Other considerations we have had included automated difficulty rating. Humanities last exam for instnace has a system that when a question is introduced you must show that no current model can solve it. We started in this direction also putting in a minimum requirement of 2 models getting it wrong in the creat prompt but then removed this just to make things more flexible.

Happy to hear more thoughts on this topic

[Meghhanaa]

Thanks for the explanation, this really helps clarify the thinking behind PeerBench.

I agree that hidden data is a strong idea and probably the cleanest way to avoid training data leakage in the long run. Making this more visible to users would already add a lot of clarity.

Showing something like the percentage of hidden vs open prompts, or a simple flag when a benchmark is fully open, sounds very useful. That way users can quickly understand how much prior exposure might be affecting the leaderboard, without completely blocking open benchmarks.

On the difficulty enforcement point, I also see why strict rules (like requiring multiple models to fail) can reduce flexibility, especially while the platform is still evolving. A softer signal, such as a saturation or difficulty indicator, feels like a good middle ground between guidance and freedom.

Happy to think more on this if helpful, but even small transparency signals like these would already make leaderboard results much easier to interpret.