Skip to content

Conversation

@m7md7sien
Copy link
Contributor

@m7md7sien m7md7sien commented Feb 12, 2026

Description

  • Add check_for_unsupported_tools validation for agentic evaluators
  • Support new content types: function_call, mcp_approval_request, openapi_call, and their outputs
  • Fix minor typos in docstrings

All SDK Contribution checklist:

  • The pull request does not introduce [breaking changes]
  • CHANGELOG is updated for new features, bug fixes or other significant changes.
  • I have read the contribution guidelines.

General Guidelines and Best Practices

  • Title of the pull request is clear and informative.
  • There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

  • Pull request includes test coverage for the included changes.

@m7md7sien m7md7sien requested a review from a team as a code owner February 12, 2026 20:14
Copilot AI review requested due to automatic review settings February 12, 2026 20:14
@github-actions github-actions bot added the Evaluation Issues related to the client library for Azure AI Evaluation label Feb 12, 2026
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request enhances the Azure AI Evaluation SDK's validator infrastructure to support new content types and adds validation for unsupported tools in agentic evaluators. The changes enable the SDK to handle emerging tool call patterns (function calls, MCP approval workflows, OpenAPI calls) while preventing usage of known unsupported tools.

Changes:

  • Added six new content types to support function calls, MCP approval requests/responses, and OpenAPI calls with their outputs
  • Implemented unsupported tools validation via check_for_unsupported_tools flag in conversation validators
  • Fixed typo in task navigation efficiency evaluator docstring
  • Relaxed validation for tool result fields from requiring dictionary type to only requiring field existence

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
_validation_constants.py Added six new ContentType enum values for function_call, function_call_output, mcp_approval_request, mcp_approval_response, openapi_call, and openapi_call_output
_conversation_validator.py Added UNSUPPORTED_TOOLS list, check_for_unsupported_tools parameter, _validate_field_exists method, and validation logic for new content types
_tool_definitions_validator.py Added check_for_unsupported_tools parameter and propagated it to parent class
_tool_calls_validator.py Added check_for_unsupported_tools parameter and propagated it to parent class
_tool_output_utilization.py Enabled unsupported tools validation by setting check_for_unsupported_tools=True
_tool_input_accuracy.py Enabled unsupported tools validation by setting check_for_unsupported_tools=True
_tool_call_success.py Enabled unsupported tools validation by setting check_for_unsupported_tools=True
_tool_call_accuracy.py Enabled unsupported tools validation by setting check_for_unsupported_tools=True
_groundedness.py Enabled unsupported tools validation by setting check_for_unsupported_tools=True
_task_navigation_efficiency.py Fixed typo in docstring (removed extra quote before "call_tool_A")

@m7md7sien m7md7sien enabled auto-merge (squash) February 12, 2026 20:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Evaluation Issues related to the client library for Azure AI Evaluation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant