Skip to content

Conversation

@AAgnihotry
Copy link
Contributor

@AAgnihotry AAgnihotry commented Jan 9, 2026

Summary

  • Add is_tool_simulated(tool_name: str) -> bool function that checks whether a tool will be simulated based on the current evaluation context
  • Add _normalize_tool_name helper to handle underscore/space normalization in tool names (e.g., my_tool matches config with my tool)
  • Export is_tool_simulated from uipath.eval.mocks public API
  • Add "Evaluation output" span as a child of each evaluator span in evaluation tracing containing:
    • value: the evaluation score
    • justification: extracted from evaluation details
    • evaluatorId: the ID of the evaluator
  • Bump version to 2.4.8

Test plan

  • Run existing tests to ensure no regressions
  • Add unit tests for _normalize_tool_name function
  • Add unit tests for is_tool_simulated with various scenarios:
    • No evaluation context
    • Null mocking strategy
    • LLM strategy with simulated/non-simulated tools
    • Mockito strategy with simulated/non-simulated tools
    • Underscore/space normalization handling
  • Add tests for new "Evaluation output" span:
    • Span name and type
    • Required attributes (value, evaluatorId, justification)
    • Span hierarchy (child of evaluator)
    • Justification extraction from different detail formats

🤖 Generated with Claude Code

Development Package

  • Use uipath pack --nolock to get the latest dev build from this PR (requires version range).
  • Add this package as a dependency in your pyproject.toml:
[project]
dependencies = [
  # Exact version:
  "uipath==2.4.8.dev1010783672",

  # Any version from PR
  "uipath>=2.4.8.dev1010780000,<2.4.8.dev1010790000"
]

[[tool.uv.index]]
name = "testpypi"
url = "https://test.pypi.org/simple/"
publish-url = "https://test.pypi.org/legacy/"
explicit = true

[tool.uv.sources]
uipath = { index = "testpypi" }

[tool.uv]
override-dependencies = [
    "uipath>=2.4.8.dev1010780000,<2.4.8.dev1010790000",
]

related PR

https://github.com/UiPath/uipath-agents-python/pull/79

image

@github-actions github-actions bot added test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-llamaindex Triggers tests in the uipath-llamaindex-python repository labels Jan 9, 2026
@AAgnihotry AAgnihotry added the build:dev Create a dev build from the pr label Jan 9, 2026
@AAgnihotry AAgnihotry changed the title feat(mocks): add is_tool_simulated function to check if tool is mocked feat: fix the tool mock name and add the Evaluation Output Span Jan 9, 2026
@AAgnihotry AAgnihotry merged commit 2da6547 into main Jan 9, 2026
86 checks passed
@AAgnihotry AAgnihotry deleted the feat/toolMocking branch January 9, 2026 01:46
@AAgnihotry AAgnihotry restored the feat/toolMocking branch January 9, 2026 01:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build:dev Create a dev build from the pr test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-llamaindex Triggers tests in the uipath-llamaindex-python repository

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants