Skip to content

fix: redact sensitive data from evaluation SDK log messages #45176

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
slister1001 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix: redact sensitive data from evaluation SDK log messages #45176

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 8 additions & 10 deletions sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_common/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -706,7 +706,7 @@ def reformat_conversation_history(query, logger=None, include_system_messages=Fa
# Lower percentage of mode in Likert scale (73.4% vs 75.4%)
# Lower pairwise agreement between LLMs (85% vs 90% at the pass/fail level with threshold of 3)
if logger:
logger.warning(f"Conversation history could not be parsed, falling back to original query: {query}")
logger.warning("Conversation history could not be parsed, falling back to original query")
return query


Expand Down Expand Up @@ -761,15 +761,15 @@ def reformat_agent_response(response, logger=None, include_tool_messages=False):
# If no message could be extracted, likely the format changed, fallback to the original response in that case
if logger:
logger.debug(
f"Empty agent response extracted, likely due to input schema change. Falling back to using the original response: {response}"
"Empty agent response extracted, likely due to input schema change. Falling back to original response"
)
return response
return "\n".join(agent_response)
except:
except Exception:
# If the agent response cannot be parsed for whatever reason (e.g. the converter format changed), the original response is returned
Copy link

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bare except: clause should specify Exception to avoid catching system-exiting exceptions like KeyboardInterrupt and SystemExit. This is inconsistent with the exception handling patterns used elsewhere in the file (see lines 700 and 786 which use except Exception as e:). Consider changing this to except Exception: for consistency and better practice.

Copilot uses AI. Check for mistakes.
# This is a fallback to ensure that the evaluation can still proceed. See comments on reformat_conversation_history for more details.
if logger:
logger.debug(f"Agent response could not be parsed, falling back to original response: {response}")
logger.debug("Agent response could not be parsed, falling back to original response")
return response


Expand All @@ -787,9 +787,7 @@ def reformat_tool_definitions(tool_definitions, logger=None):
# If the tool definitions cannot be parsed for whatever reason, the original tool definitions are returned
# This is a fallback to ensure that the evaluation can still proceed. See comments on reformat_conversation_history for more details.
if logger:
logger.warning(
f"Tool definitions could not be parsed, falling back to original definitions: {tool_definitions}"
)
logger.debug("Tool definitions could not be parsed, falling back to original definitions")
return tool_definitions


Expand Down Expand Up @@ -915,9 +913,9 @@ def upload(path: str, container_client: ContainerClient, logger=None):

except Exception as e:
raise EvaluationException(
message=f"Error uploading file: {e}",
internal_message=f"Error uploading file: {e}",
message=f"Error uploading file: {type(e).__name__}",
internal_message=f"Error uploading file: {type(e).__name__}",
target=ErrorTarget.RAI_CLIENT,
category=ErrorCategory.UPLOAD_ERROR,
blame=ErrorBlame.SYSTEM_ERROR,
)
) from e
Loading