[bugfix]bugfix for judge infer

[SJTUyh]

Thanks for your contribution; we appreciate it a lot. The following instructions will make your pull request healthier and help you get feedback more easily. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
感谢您的贡献，我们非常重视。以下说明将使您的拉取请求更健康，更易于获得反馈。如果您不理解某些项目，请不要担心，只需提交拉取请求并从维护人员那里寻求帮助即可。

PR Type / PR类型

• Feature（功能新增）
• Bugfix（Bug 修复）
• Docs（文档更新）
• CI/CD（持续集成/持续部署）
• Refactor（代码重构）
• Perf（性能优化）
• Dependency（依赖项更新）
• Test-Cases（测试用例更新）
• Other（其他）

Related Issue | 关联 Issue
Fixes #(issue ID / issue 编号) / Relates to #(issue ID / issue 编号)

🔍 Motivation / 变更动机

Please describe the motivation of this PR and the goal you want to achieve through this PR.
请描述您的拉取请求的动机和您希望通过此拉取请求实现的目标。
裁判模型相关代码合入后，只跑普通数据集基本功能不可用

📝 Modification / 修改内容

Please briefly describe what modification is made in this PR.
请简要描述此拉取请求中进行的修改。
做了兼容性修改

📐 Associated Test Results / 关联测试结果

Please provide links to the related test results, such as CI pipelines, test reports, etc.
请提供相关测试结果的链接，例如 CI 管道、测试报告等。

只跑普通数据集：

同时跑普通数据集和裁判模型数据集：

⚠️ BC-breaking (Optional) / 向后不兼容变更（可选）

Does the modification introduce changes that break the backward compatibility of the downstream repositories? If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.
是否引入了会破坏下游存储库向后兼容性的更改？如果是，请描述它如何破坏兼容性，以及下游项目应该如何修改其代码以保持与此 PR 的兼容性。

⚠️ Performance degradation (Optional) / 性能下降（可选）

If the modification introduces performance degradation, please describe the impact of the performance degradation and the expected performance improvement.
如果引入了性能下降，请描述性能下降的影响和预期的性能改进。

🌟 Use cases (Optional) / 使用案例（可选）

If this PR introduces a new feature, it is better to list some use cases here and update the documentation.
如果此拉取请求引入了新功能，最好在此处列出一些用例并更新文档。

✅ Checklist / 检查列表

Before PR:

• Pre-commit or other linting tools are used to fix the potential lint issues. / 使用预提交或其他 linting 工具来修复潜在的 lint 问题。
• Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests. / 修复的 Bug 已完全由单元测试覆盖，导致 Bug 的情况应在单元测试中添加。
• The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness. / 此拉取请求中的修改已完全由单元测试覆盖。如果不是，请添加更多单元测试以确保正确性。
• All relevant documentation (API docs, docstrings, example tutorials) has been updated to reflect these changes. / 所有相关文档（API 文档、文档字符串、示例教程）已更新以反映这些更改。

After PR:

• If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects. / 如果此拉取请求对下游或其他相关项目有潜在影响，应在那些项目中测试此 PR。
• CLA has been signed and all committers have signed the CLA in this PR. / CLA 已签署，且本 PR 中的所有提交者均已签署 CLA。

👥 Collaboration Info / 协作信息

• Suggested Reviewers / 建议审核人: @xxx
• Relevant Module Owners / 相关模块负责人: @xxx
• Other Collaboration Notes / 其他协作说明：

🌟 Useful CI Command / 实用的CI命令

Command / 命令	Introduction / 介绍
/gemini review	Performs a code review for the current pull request in its current state by Gemini. / 对当前拉取请求在当前状态下由 Gemini 执行代码审核。
/gemini summary	Provides a summary of the current pull request in its current state by Gemini. / 对当前拉取请求在当前状态下由 Gemini 提供摘要。
/gemini help	Displays a list of available commands of Gemini. / 显示 Gemini 可用命令的列表。
/readthedocs build	Triggers a build of the documentation for the current pull request in its current state by Read the Docs. / 触发当前拉取请求在当前状态下由 Read the Docs 构建文档。

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a compatibility issue within the judge inference module. Previously, the system would encounter problems when processing datasets that lacked specific judge inference configurations. The changes introduce robust checks to detect the absence of judge inference settings, allowing the system to either adapt its behavior or skip the judge inference process entirely, thereby ensuring broader compatibility and preventing failures with diverse dataset types.

Highlights

• Judge Infer Compatibility: Implemented checks in the JudgeInfer worker to gracefully handle configurations where judge inference is not specified for any dataset, preventing errors and allowing the worker to skip judge inference when not applicable.
• Configuration Handling: Modified the update_cfg method to iterate through all datasets to determine the judge_model_type, ensuring that the worker can adapt to mixed configurations where some datasets might not require judge inference.
• Execution Flow Control: Added an early exit condition in the do_work method of JudgeInfer to prevent unnecessary processing if no judge model type is detected, improving robustness for non-judge inference scenarios.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• ais_bench/benchmark/cli/workers.py
  • Introduced self.judge_model_type to store the detected judge model type.
  • Modified update_cfg to iterate through all datasets to find judge_infer_cfg and set self.judge_model_type.
  • Added a conditional return in update_cfg to skip further configuration if no judge model type is found.
  • Updated get_task_type to use self.judge_model_type for determining the task type.
  • Implemented an early return in do_work if self.judge_model_type is None, effectively skipping judge inference.

Activity

• No human activity has been recorded on this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

The pull request introduces robust error handling for the JudgeInfer worker by ensuring that judge inference tasks are only attempted when a judge_infer_cfg is explicitly present in the configuration. This prevents runtime errors when the configuration does not include judge inference, making the system more stable. The changes correctly initialize and check self.judge_model_type in both update_cfg and do_work methods, leading to a graceful skip if no judge inference is configured.

[gemini-code-assist]

The loop iterates through cfg["datasets"] and updates self.judge_model_type if judge_infer_cfg is found. If multiple datasets have judge_infer_cfg with potentially different judge_model.attr values, self.judge_model_type will be set to the value from the last dataset in the list that contains judge_infer_cfg. This self.judge_model_type is then used globally for get_task_type(). If the system expects all judge_infer_cfgs to have a consistent judge_model.attr or if different judge_model.attr values across datasets should lead to different behaviors, this current implementation might lead to unexpected results. Consider adding a check to ensure consistency across all judge_infer_cfgs if this is a requirement, or explicitly document that only the last encountered judge_model.attr is used.