About the evaluation performance

[c-box]

Thanks for the great benchmark!

I am currently trying to reproduce some model results on C_simQA using OpenCompass, using Qwen-2.5-72B-instruct as the judge model. However, the performance I've evaluated is significantly higher than the performance reported in the paper.

I'm wondering if the judge model could lead to such a large difference? After reviewing some cases, I found that the overall judge results are in line with expectations.

Do you have any suggestions?

[OpenStellarTeam]

Sorry for the late response—we only just saw your question due to some account management issues.

As shown in the appendix of our official release, we provide experiments using different judge models. Our findings also show that different judge models can lead to some variation in scores, mainly due to differences in interpreting the "not_attempted" category, as well as the difference in judging ability. However, we found that regardless of which judge model is used, the relative ranking among different models remains consistent.

We recommend using GPT-4o as the judge model if possible. If that's not available, using the same judge model for all evaluations will still reflect the relative improvements between models.

Thank you very much for using our benchmark! Please feel free to reach out if you have any other questions or suggestions.