Skip to content

[3138] fix(api,frontend): guard live evaluations and clarify missing query filter message #3327

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
ashrafchowdury wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

[3138] fix(api,frontend): guard live evaluations and clarify missing query filter message #3327

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions api/oss/src/core/evaluations/service.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1338,6 +1338,12 @@ async def create(
log.info("[EVAL] [failure] missing simple evaluation data")
return None

if evaluation.flags.is_live and not evaluation.data.query_steps:
Copy link
Contributor

@junaway junaway Dec 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not correct though, since evaluation without queries do exist.

log.info(
"[EVAL] [failure] missing query for live evaluation (query_steps required)"
)
return None

# ----------------------------------------------------------------------
log.info("[EVAL] [create]")
log.info("[EVAL] [scope] ", project_id=project_id, user_id=user_id)
Expand Down
10 changes: 8 additions & 2 deletions web/oss/src/components/pages/evaluations/onlineEvaluation/OnlineEvaluationDrawer.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -289,15 +289,21 @@ const OnlineEvaluationDrawer = ({open, onClose, onCreate}: OnlineEvaluationDrawe
const queryResponse = await createSimpleQuery({query: queryPayload})
const queryId = queryResponse.query?.id
if (!queryId) {
throw new Error("Unable to create query for online evaluation.")
message.error(
"Unable to create the query required for live evaluation. Please add at least one query filter and try again.",
)
return
}

const revisionResponse = await retrieveQueryRevision({
query_ref: {id: queryId},
})
const queryRevisionId = revisionResponse.query_revision?.id
if (!queryRevisionId) {
throw new Error("Unable to resolve query revision for online evaluation.")
message.error(
"Unable to resolve the query revision required for live evaluation. Please try again.",
)
return
}

// Prefer preview evaluator artifact id; fall back to selected config id if preview not available
Expand Down