[doc] [en]Add docs for judge evaluate and gedit extend benchmark#170
[doc] [en]Add docs for judge evaluate and gedit extend benchmark#170GaoHuaZhang merged 2 commits intoAISBench:masterfrom
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the English documentation for the AISBench tool by introducing detailed guides for advanced evaluation methodologies. It provides a new tutorial on utilizing 'judge models' for scenarios where traditional ground truth comparisons are insufficient, and integrates documentation for the GEdit-Bench, a benchmark for real-world image editing. These additions improve the clarity and scope of the evaluation framework, making it more accessible for users to understand and implement complex benchmarking tasks. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
SJTUyh
changed the title
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
The pull request introduces comprehensive documentation for the Judge Model evaluation and GEdit-Bench, enhancing the clarity and usability of the AISBench tool. The new documentation covers the motivation, quick start, configuration, execution details, and various evaluation scenarios for the Judge Model, along with a detailed guide for GEdit-Bench evaluation. The changes also correctly integrate these new documentation pages into the main English documentation structure. Some minor improvements in clarity and consistency within the new documentation have been identified.
| | --- | --- | --- | --- | --- | --- | | ||
| | aime2025_gen | Generative task for the AIME2025 dataset | Accuracy | 0-shot | Chat format | aime2025_gen_0_shot_chat_prompt.py | No newline at end of file | ||
| | aime2025_gen | Generative task for the AIME2025 dataset | Accuracy | 0-shot | Chat format | aime2025_gen_0_shot_chat_prompt.py | | ||
| | aime2025_gen_0_shot_llmjudge | AIME2025 | Generative task for the AIME2025 dataset | Accuracy evaluated by judge model | 0-shot | Chat format | aime2025_gen_0_shot_llmjudge.py | |
There was a problem hiding this comment.
The 'Introduction' for aime2025_gen_0_shot_llmjudge is currently AIME2025, which appears to be the dataset name rather than an introduction to the task itself. For consistency with the previous entry, consider providing a brief description of the task.
| | aime2025_gen_0_shot_llmjudge | AIME2025 | Generative task for the AIME2025 dataset | Accuracy evaluated by judge model | 0-shot | Chat format | aime2025_gen_0_shot_llmjudge.py | | |
| | aime2025_gen_0_shot_llmjudge | Generative task for the AIME2025 dataset with judge model evaluation | Accuracy evaluated by judge model | 0-shot | Chat format | aime2025_gen_0_shot_llmjudge.py | |
| ais_bench --models vllm_api_general_chat --datasets aime2025_gen_0_shot_llmjudge | ||
| ``` | ||
|
|
||
| > Note: Judge model dataset tasks differ from regular dataset tasks in configuration, but both types of dataset tasks can be mixed in a single dataset task. |
There was a problem hiding this comment.
This note could be more specific. Clarify what aspects of the configuration differ and what 'mixed in a single dataset task' implies (e.g., running both types in one command or within one config file).
|
|
||
| ### Pre-run Preparation | ||
|
|
||
| - `--models`: Using `vllm_api_general_chat` model task requires preparing an inference service that supports `v1/chat/completions` sub-service. You can refer to 🔗 [VLLM Launch OpenAI Compatible Server](https://docs.vllm.com.cn/en/latest/getting_started/quickstart.html#openai-compatible-server) to start the inference service (the tested model is one inference service, and the judge model is another inference service; for quick start, you can also share one service if convenient). |
There was a problem hiding this comment.
The sentence is a bit long and could be rephrased for better readability. Consider splitting it or simplifying the phrasing.
| - `--models`: Using `vllm_api_general_chat` model task requires preparing an inference service that supports `v1/chat/completions` sub-service. You can refer to 🔗 [VLLM Launch OpenAI Compatible Server](https://docs.vllm.com.cn/en/latest/getting_started/quickstart.html#openai-compatible-server) to start the inference service (the tested model is one inference service, and the judge model is another inference service; for quick start, you can also share one service if convenient). | |
| to start the inference service (typically, the tested model and the judge model each require a separate inference service; however, for quick start, you can share one service if convenient). |
| ``` | ||
|
|
||
| - The configuration method for `vllm_api_general_chat` corresponding to the tested model task configuration file is the same as in Quick Start, not repeated here. | ||
| - In the `aime2025_gen_0_shot_llmjudge` corresponding judge model dataset task configuration file, you need to modify the judge model configuration: |
There was a problem hiding this comment.
The phrasing "In the ... corresponding judge model dataset task configuration file" is a bit awkward. Consider rephrasing for better flow.
| - In the `aime2025_gen_0_shot_llmjudge` corresponding judge model dataset task configuration file, you need to modify the judge model configuration: | |
| For the `aime2025_gen_0_shot_llmjudge` judge model dataset task configuration file, you need to modify the judge model configuration: |
| pred_postprocessor=dict(type=extract_non_reasoning_content), | ||
| ), | ||
| ``` | ||
| The meaning of judge model task configuration is exactly the same as the tested model task configuration. |
There was a problem hiding this comment.
This statement is a bit redundant after providing the detailed configuration block. If kept, it could be rephrased to emphasize that the parameters within the judge_model dictionary have similar meanings to those in the tested model's configuration.
| # ====== User configuration parameters ========= | ||
| qwen_image_edit_models[0]["path"] = "/path/to/Qwen-Image-Edit-2509/" # Modify to actual model weight path | ||
| qwen_image_edit_models[0]["infer_kwargs"]["num_inference_steps"] = 50 # Modify to the required inference steps | ||
| device_list = [0] # [0, 1, 2, 3] Modify to the actual available NPU device ID list, not necessarily in order, each device will separately load a weight |
There was a problem hiding this comment.
The comment [0, 1, 2, 3] is an example, but the current value is [0]. It might be clearer to show a more representative example or explicitly state that [0] is just a placeholder for a single device.
| device_list = [0] # [0, 1, 2, 3] Modify to the actual available NPU device ID list, not necessarily in order, each device will separately load a weight | |
| device_list = [0] # Example: [0, 1, 2, 3]. Modify to the actual available NPU device ID list; each device will separately load a weight. |
| judge_model=dict( | ||
| attr="service", | ||
| type=VLLMCustomAPIChat, | ||
| abbr=f"{metric}_judge", # Be added after dataset abbr |
There was a problem hiding this comment.
The comment "Be added after dataset abbr" is a bit vague. Clarify what abbr represents in this context (e.g., "This abbreviation is appended to the dataset abbreviation to form a unique identifier for the judge model.").
| | gedit-3-SC_judge | 16dd59 | SC | gen | - | - | - | 7.31 | | ||
| | gedit-3-PQ_judge | 16dd59 | PQ | gen | - | - | - | 7.24 | | ||
|
|
||
| [2026-03-04 15:40:45,583] [ais_bench] [INFO] write markdown summary to /workplace/benchmark/ais_bench/configs/lmm_exmaple/outputs/default/20260213_150110/summary/summary_20260304_152835.md |
There was a problem hiding this comment.
There is a typo in the path: lmm_exmaple should be lmm_example.
| [2026-03-04 15:40:45,583] [ais_bench] [INFO] write markdown summary to /workplace/benchmark/ais_bench/configs/lmm_exmaple/outputs/default/20260213_150110/summary/summary_20260304_152835.md | |
| [2026-03-04 15:40:45,583] [ais_bench] [INFO] write markdown summary to /workplace/benchmark/ais_bench/configs/lmm_example/outputs/default/20260213_150110/summary/summary_20260304_152835.md |
| ```bash | ||
| # | ||
| # python3 -m ais_bench.tools.dataset_processors.gedit.display_results --config_path {CONFIG_PATH} --timestamp_path {TIMESTAMP_PATH} | ||
| python3 -m ais_bench.tools.dataset_processors.gedit.display_results --config_path ./multi_device_run_qwen_image_edit.py --timestamp_path outputs/default/20260213_150110/ |
There was a problem hiding this comment.
The timestamp 20260213_150110 is hardcoded in the example. It would be beneficial to add a note indicating that this is an example timestamp and the actual value will vary based on execution.
| python3 -m ais_bench.tools.dataset_processors.gedit.display_results --config_path ./multi_device_run_qwen_image_edit.py --timestamp_path outputs/default/20260213_150110/ | |
| python3 -m ais_bench.tools.dataset_processors.gedit.display_results --config_path ./multi_device_run_qwen_image_edit.py --timestamp_path outputs/default/{ACTUAL_TIMESTAMP}/ |
|
|
||
| ```bash | ||
| # python3 -m ais_bench.tools.dataset_processors.gedit.display_results --config_path {CONFIG_PATH} --timestamp_path {TIMESTAMP_PATH} | ||
| python3 -m ais_bench.tools.dataset_processors.gedit.convert_results --config_path ./multi_device_run_qwen_image_edit.py --timestamp_path outputs/default/20260213_150110/ |
There was a problem hiding this comment.
Similar to the previous comment, the timestamp 20260213_150110 is hardcoded. A note about it being an example would improve clarity.
| python3 -m ais_bench.tools.dataset_processors.gedit.convert_results --config_path ./multi_device_run_qwen_image_edit.py --timestamp_path outputs/default/20260213_150110/ | |
| python3 -m ais_bench.tools.dataset_processors.gedit.convert_results --config_path ./multi_device_run_qwen_image_edit.py --timestamp_path outputs/default/{ACTUAL_TIMESTAMP}/ |
2 participants
Thanks for your contribution; we appreciate it a lot. The following instructions will make your pull request healthier and help you get feedback more easily. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
感谢您的贡献，我们非常重视。以下说明将使您的拉取请求更健康，更易于获得反馈。如果您不理解某些项目，请不要担心，只需提交拉取请求并从维护人员那里寻求帮助即可。
PR Type / PR类型
Related Issue | 关联 Issue
Fixes #(issue ID / issue 编号) / Relates to #(issue ID / issue 编号)
🔍 Motivation / 变更动机
Please describe the motivation of this PR and the goal you want to achieve through this PR.
请描述您的拉取请求的动机和您希望通过此拉取请求实现的目标。
add en docs for Judge Model
📝 Modification / 修改内容
Please briefly describe what modification is made in this PR.
请简要描述此拉取请求中进行的修改。
add en docs for Judge Model
📐 Associated Test Results / 关联测试结果
Please provide links to the related test results, such as CI pipelines, test reports, etc.
请提供相关测试结果的链接，例如 CI 管道、测试报告等。
Does the modification introduce changes that break the backward compatibility of the downstream repositories? If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.
是否引入了会破坏下游存储库向后兼容性的更改？如果是，请描述它如何破坏兼容性，以及下游项目应该如何修改其代码以保持与此 PR 的兼容性。
If the modification introduces performance degradation, please describe the impact of the performance degradation and the expected performance improvement.
如果引入了性能下降，请描述性能下降的影响和预期的性能改进。
🌟 Use cases (Optional) / 使用案例（可选）
If this PR introduces a new feature, it is better to list some use cases here and update the documentation.
如果此拉取请求引入了新功能，最好在此处列出一些用例并更新文档。
✅ Checklist / 检查列表
Before PR:
After PR:
👥 Collaboration Info / 协作信息
🌟 Useful CI Command / 实用的CI命令
/gemini review/gemini summary/gemini help/readthedocs build