Structured Output and Multiple Outputs in LLM-as-a-Judge Evaluators

[mmabrouk]

Problem Statement

Currently, LLM-as-a-judge evaluators in Agenta return a score and basic feedback. They cannot use structured output formats or generate multiple output fields. This limits the depth of insights teams can extract from automated evaluations.

Proposed Solution

We are enhancing LLM-as-a-judge evaluators with two capabilities:

Structured Output Support
Define JSON schemas for evaluator outputs to ensure consistent, parseable results.

Multiple Output Fields
Generate not just a score, but also additional fields such as:

• Explanation: detailed reasoning for the score
• Confidence level: how confident the judge is in its assessment
• Suggestions: specific improvement recommendations
• Issue categories: classification of problems found (accuracy, tone, safety, etc.)

These fields will be configurable when setting up the evaluator. Teams will be able to analyze and filter evaluation results based on any of these fields.

[mmabrouk]

Shipped: https://docs.agenta.ai/changelog/customize-llm-as-a-judge-output-schemas