[doc] [zh_cn]Add docs for judge evaluate and gedit extend benchmark#168
[doc] [zh_cn]Add docs for judge evaluate and gedit extend benchmark#168GaoHuaZhang merged 7 commits intoAISBench:masterfrom
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! 此拉取请求通过引入新的中文文档来扩展AISBench的评估能力。它详细介绍了如何使用裁判模型进行评估，这对于没有标准答案或需要更深入评估过程的场景至关重要。此外，还新增了关于GEdit-Bench基准测试的文档，为真实世界指令图像编辑提供了评估指南。这些文档的添加旨在提升用户对AISBench工具的理解和使用体验，使其能够更灵活地应对复杂的模型评估需求。 Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
| D --> E[从裁判模型判断结果中提取相关指标得分] | ||
| E --> F((精度数据)) | ||
| F --> G[呈现结果] | ||
|
|
||
| classDef green fill:#90EE90,stroke:#228B22,stroke-width:2px; | ||
| ``` |
There was a problem hiding this comment.
关于--mode infer_judge的mermaid图示与文字描述存在不一致。文字描述指出该模式“不进行指标提取”，但图示却包含了“从裁判模型判断结果中提取相关指标得分”和“呈现结果”的步骤。请修正图示以准确反映该模式的行为，即图示应在“裁判模型的判断结果”处结束。
| D --> E[从裁判模型判断结果中提取相关指标得分] | |
| E --> F((精度数据)) | |
| F --> G[呈现结果] | |
| classDef green fill:#90EE90,stroke:#228B22,stroke-width:2px; | |
| ``` | |
| graph LR; | |
| A[基于给定数据集执行推理] --> B((被测模型的推理结果)) | |
| B --> C[裁判模型对被测模型推理结果进行判断]:::green | |
| C --> D((裁判模型的判断结果)):::green | |
| classDef green fill:#90EE90,stroke:#228B22,stroke-width:2px; |
| D --> E[基于精度数据生成汇总报告] | ||
| E --> F((呈现结果)) | ||
| ``` | ||
| 当时有些测评场景中没有标准答案，或者不仅需要看标准答案，还要看得到标准答案的过程是否正确。这个时候常规的答案提取方式就无法满足这样的评测场景。因此需要引入裁判模型对被测模型推理结果进行判断。有裁判模型介入的整体测评流程如下： |
There was a problem hiding this comment.
在描述一般场景时，使用“当时”可能不够准确，因为它暗示了一个特定的时间点。建议使用“然而，在某些”或“在某些”来更普遍地引入这种场景。
| 当时有些测评场景中没有标准答案，或者不仅需要看标准答案，还要看得到标准答案的过程是否正确。这个时候常规的答案提取方式就无法满足这样的评测场景。因此需要引入裁判模型对被测模型推理结果进行判断。有裁判模型介入的整体测评流程如下： | |
| 然而，在某些测评场景中没有标准答案，或者不仅需要看标准答案，还要看得到标准答案的过程是否正确。 |
|
|
||
| ### 运行命令前置准备 | ||
| - `--models`: 使用`vllm_api_general_chat`模型任务，需要准备支持`v1/chat/completions`子服务的推理服务，可以参考🔗 [VLLM启动OpenAI 兼容服务器](https://docs.vllm.com.cn/en/latest/getting_started/quickstart.html#openai-compatible-server)启动推理服务（被测模型是一个推理服务，裁判模型是另一个推理服务，快速入门如果图方便也可以共用一个服务） | ||
| - `--datasets`: 使用`aime2025_gen_0_shot_llmjudge`数据集任务，需要准备aime2025数据集，可以从🔗 [opencompass |
There was a problem hiding this comment.
“opencompass 提供的”可以简化为“opencompass 的”，使表达更简洁。
| - `--datasets`: 使用`aime2025_gen_0_shot_llmjudge`数据集任务，需要准备aime2025数据集，可以从🔗 [opencompass | |
| 提供的aime2025数据集压缩包 |
| ### 运行命令前置准备 | ||
| - `--models`: 使用`vllm_api_general_chat`模型任务，需要准备支持`v1/chat/completions`子服务的推理服务，可以参考🔗 [VLLM启动OpenAI 兼容服务器](https://docs.vllm.com.cn/en/latest/getting_started/quickstart.html#openai-compatible-server)启动推理服务（被测模型是一个推理服务，裁判模型是另一个推理服务，快速入门如果图方便也可以共用一个服务） | ||
| - `--datasets`: 使用`aime2025_gen_0_shot_llmjudge`数据集任务，需要准备aime2025数据集，可以从🔗 [opencompass | ||
| 提供的aime2025数据集压缩包](http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/aime2025.zip)下载。将解压后的`aime2025/`文件夹部署到AISBench评测工具根路径下的`ais_bench/datasets`文件夹下。 |
There was a problem hiding this comment.
对于文件夹的操作，使用“部署到”可能过于正式，通常用于软件或服务的部署。建议使用“放置到”或“复制到”来更准确地描述将文件夹放到指定位置的操作。
| 提供的aime2025数据集压缩包](http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/aime2025.zip)下载。将解压后的`aime2025/`文件夹部署到AISBench评测工具根路径下的`ais_bench/datasets`文件夹下。 | |
| 将解压后的`aime2025/`文件夹放置到AISBench评测工具根路径下的`ais_bench/datasets`文件夹下。 |
|
|
||
| ### 测评指标（MLLM 自动评分，满分10分） | ||
| - **G_SC, Q_SC（语义一致性）**：编辑结果与指令的匹配度 | ||
| - **G_PQ, Q_PQ（图像质量）**：清晰度、细节保留、无伪影 |
There was a problem hiding this comment.
Q_0可能是一个笔误，为了与G_O保持一致性，建议修改为Q_O。
| - **G_PQ, Q_PQ（图像质量）**：清晰度、细节保留、无伪影 | |
| G_O, Q_O（综合得分） |
| | gedit-2-PQ_judge | 16dd59 | PQ | gen | - | - | 7.22 | - | | ||
| | gedit-3-SC_judge | 16dd59 | SC | gen | - | - | - | 7.31 | | ||
| | gedit-3-PQ_judge | 16dd59 | PQ | gen | - | - | - | 7.24 | | ||
|
|
There was a problem hiding this comment.
日志路径中lmm_exmaple存在拼写错误，应为lmm_example。
| [2026-03-04 15:40:45,583] [ais_bench] [INFO] write markdown summary to /workplace/benchmark/ais_bench/configs/lmm_example/outputs/default/20260213_150110/summary/summary_20260304_152835.md |
2 participants
Thanks for your contribution; we appreciate it a lot. The following instructions will make your pull request healthier and help you get feedback more easily. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
感谢您的贡献，我们非常重视。以下说明将使您的拉取请求更健康，更易于获得反馈。如果您不理解某些项目，请不要担心，只需提交拉取请求并从维护人员那里寻求帮助即可。
PR Type / PR类型
Related Issue | 关联 Issue
Fixes #(issue ID / issue 编号) / Relates to #(issue ID / issue 编号)
🔍 Motivation / 变更动机
Please describe the motivation of this PR and the goal you want to achieve through this PR.
请描述您的拉取请求的动机和您希望通过此拉取请求实现的目标。
Add docs for judge evaluate and gedit extend benchmark
📝 Modification / 修改内容
Please briefly describe what modification is made in this PR.
请简要描述此拉取请求中进行的修改。
Add docs for judge evaluate and gedit extend benchmark
📐 Associated Test Results / 关联测试结果
Please provide links to the related test results, such as CI pipelines, test reports, etc.
请提供相关测试结果的链接，例如 CI 管道、测试报告等。
Does the modification introduce changes that break the backward compatibility of the downstream repositories? If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.
是否引入了会破坏下游存储库向后兼容性的更改？如果是，请描述它如何破坏兼容性，以及下游项目应该如何修改其代码以保持与此 PR 的兼容性。
If the modification introduces performance degradation, please describe the impact of the performance degradation and the expected performance improvement.
如果引入了性能下降，请描述性能下降的影响和预期的性能改进。
🌟 Use cases (Optional) / 使用案例（可选）
If this PR introduces a new feature, it is better to list some use cases here and update the documentation.
如果此拉取请求引入了新功能，最好在此处列出一些用例并更新文档。
✅ Checklist / 检查列表
Before PR:
After PR:
👥 Collaboration Info / 协作信息
🌟 Useful CI Command / 实用的CI命令
/gemini review/gemini summary/gemini help/readthedocs build