Trt 2389 test summaries #4868
Trt 2389 test summaries #4868
Conversation
|
Pipeline controller notification For optional jobs, comment This repository is configured in: automatic mode |
WalkthroughAdds TestSummaryByPeriodRow (including FirstSampleDate) and a BigQuery-backed ListTestSummaryByPeriod API with retry and mock support; introduces a "tests" data-type path in the historical analyzer that fetches, formats, and writes test summaries; adds CLI validation and helper utilities for test-summary files. Changes
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes ✨ Finishing touches
Comment |
|
/hold |
openshift-ci
bot
added
the
do-not-merge/hold
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: neisw The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/analyzer.go (1)
25-52: Newdata-type=testsflow is cleanly separated; consider future configurabilityThe early branch in
RunandrunTestsDataTypecleanly bypass the comparison logic fordata-type=tests, fetching summaries once and writing them directly viaformatTestOutput, without disturbing the alerts/disruptions path. The hardcodedsuiteName,daysBack, andminTestCountare reasonable defaults, but if this grows beyond a single use case, promoting them to flags or config would improve long‑term flexibility.Also applies to: 100-132
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
Cache: Disabled due to data retention organization setting
Knowledge base: Disabled due to Reviews -> Disable Knowledge Base setting
📒 Files selected for processing (7)
pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go(1 hunks)pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go(2 hunks)pkg/jobrunaggregator/jobrunaggregatorlib/retrying_ci_data_client.go(1 hunks)pkg/jobrunaggregator/jobrunaggregatorlib/util.go(1 hunks)pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/analyzer.go(2 hunks)pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/cmd.go(2 hunks)pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/util.go(1 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**
⚙️ CodeRabbit configuration file
-Focus on major issues impacting performance, readability, maintainability and security. Avoid nitpicks and avoid verbosity.
Files:
pkg/jobrunaggregator/jobrunaggregatorlib/util.gopkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.gopkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.gopkg/jobrunaggregator/jobrunaggregatorlib/retrying_ci_data_client.gopkg/jobrunaggregator/jobrunhistoricaldataanalyzer/analyzer.gopkg/jobrunaggregator/jobrunhistoricaldataanalyzer/cmd.gopkg/jobrunaggregator/jobrunhistoricaldataanalyzer/util.go
🧬 Code graph analysis (3)
pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go (1)
pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go (1)
TestSummaryByPeriodRow(9-20)
pkg/jobrunaggregator/jobrunaggregatorlib/retrying_ci_data_client.go (1)
pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go (1)
TestSummaryByPeriodRow(9-20)
pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/util.go (2)
pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go (1)
TestSummaryByPeriodRow(9-20)pkg/api/types.go (1)
Release(452-466)
🔇 Additional comments (6)
pkg/jobrunaggregator/jobrunaggregatorlib/util.go (1)
39-58: BigQuery label addition is consistent and low‑riskThe new
bigQueryLabelValueTestSummaryByPeriodconstant matches existing naming/usage patterns for query labels; no issues from a maintainability or behavior standpoint.pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/cmd.go (1)
29-49:data-type=testswiring and validation look coherentAdding
"tests"tosupportedDataTypesand relaxing only the--current/--previous-releaserequirements (while still enforcing BigQuery auth/coords when--newis empty) matches the new tests path, which fetches directly from BigQuery without doing comparisons. No obvious correctness or UX regressions here.Also applies to: 51-77
pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/util.go (1)
170-185: Test-summary formatting helper is straightforward and consistentThe in-place sort by release, failure count (desc), then test name mirrors the query’s ordering and existing
formatOutputpatterns, andjson.MarshalIndentkeeps output stable for diffs. Behavior for empty input is defensive and safe.pkg/jobrunaggregator/jobrunaggregatorlib/retrying_ci_data_client.go (1)
188-196: Retry wrapper mirrors existing client behavior
ListTestSummaryByPeriodon the retrying client follows the sameretry.OnError(slowBackoff, isReadQuotaError, …)pattern as the other methods, so retry semantics stay consistent and localized to quota-style failures.If you haven’t already, run a quick integration test hitting the real BigQuery-backed implementation to confirm retry behavior and error wrapping are as expected.
pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go (1)
1-20: Struct shape correctly matches the BigQuery query schema
TestSummaryByPeriodRow’s field names, types, andbigquerytags line up with theListTestSummaryByPeriodSELECT clause (SUMs asint64, ratios/durations asfloat64, dates ascivil.Date), so decoding should be straightforward.Please confirm via a small end‑to‑end query (or unit test with a fake
RowIterator) that BigQuery scanning into this struct works without field/tag mismatches.pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go (1)
70-76: Test-summary query is well‑structured and consistent with existing CIDataClient patternsThe new
ListTestSummaryByPeriodmethod is documented clearly on the interface and the implementation’s SQL matches theTestSummaryByPeriodRowschema (field aliases ↔ struct tags). The aggregation logic (SUM/SAFE_DIVIDE/AVG, window viaCURRENT_DATE()anddaysBack,HAVING SUM(test_count) > @min_test_count) and labeling (bigQueryLabelValueTestSummaryByPeriod) align with how other queries in this client are written, so it should be straightforward to operate and debug.If possible, run this against a representative dataset to confirm:
- Parameter binding for
days_back/min_test_countis accepted by BigQuery.- The
DATE_SUB(CURRENT_DATE(), INTERVAL @days_back DAY)filter yields the expected window.- The decoded
TestSummaryByPeriodRowvalues (includingFailureRateandPeriodStart/End) match manual query results.Also applies to: 1107-1173
neisw
force-pushed
the
TRT-2389-test-summaries
branch
from
c5366f4 to
cee2982
Compare
|
/test images |
1 similar comment
|
/test images |
|
/hold cancel |
openshift-ci
bot
removed
the
do-not-merge/hold
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go (1)
1107-1107: Consider validating input parameters.Adding validation for
daysBackandminTestCountwould prevent unexpected behavior from negative or extreme values.🛡️ Suggested validation
func (c *ciDataClient) ListTestSummaryByPeriod(ctx context.Context, suiteName, releaseName string, daysBack, minTestCount int) ([]jobrunaggregatorapi.TestSummaryByPeriodRow, error) { + if daysBack < 1 { + return nil, fmt.Errorf("daysBack must be at least 1, got %d", daysBack) + } + if minTestCount < 0 { + return nil, fmt.Errorf("minTestCount must be non-negative, got %d", minTestCount) + } + // Query to summarize test results for a specific suite and release over a time period
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Cache: Disabled due to data retention organization setting
Knowledge base: Disabled due to Reviews -> Disable Knowledge Base setting
📒 Files selected for processing (2)
pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.gopkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go
🧰 Additional context used
📓 Path-based instructions (1)
**
⚙️ CodeRabbit configuration file
-Focus on major issues impacting performance, readability, maintainability and security. Avoid nitpicks and avoid verbosity.
Files:
pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.gopkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go
🧬 Code graph analysis (1)
pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go (1)
pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go (1)
TestSummaryByPeriodRow(9-21)
🔇 Additional comments (2)
pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go (1)
70-76: LGTM! Well-documented interface addition.The new method signature is clear and follows existing patterns. The parameter documentation helps callers understand the expected inputs.
pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go (1)
1-21: LGTM! Clean data structure definition.The struct is well-designed with:
- Appropriate field types (
civil.Datefor date-only values,int64for counts,float64for rates/averages)- Correct BigQuery tags matching the query fields
- Clear documentation including the purpose of
FirstSampleDate
| WITH earliest_test_dates AS ( | ||
| -- Pre-compute the earliest date each test was seen across all releases for this suite | ||
| SELECT | ||
| test_name, | ||
| MIN(date) AS first_sample_date | ||
| FROM | ||
| DATA_SET_LOCATION.TestsSummaryByDate | ||
| WHERE | ||
| suite = @suite_name | ||
| AND date >= DATE_SUB(CURRENT_DATE(), INTERVAL 100 DAY) | ||
| AND date <= CURRENT_DATE() | ||
| GROUP BY | ||
| test_name | ||
| ) |
There was a problem hiding this comment.
Hardcoded 100-day window in CTE may cause incomplete first_sample_date.
The CTE uses a fixed 100-day lookback (line 1123), while the main query uses the parametrized @days_back. If daysBack > 100, the first_sample_date will be incomplete because the CTE won't scan beyond 100 days. Consider either:
- Documenting this as a design constraint (e.g., "first_sample_date is limited to 100 days of history")
- Making the CTE window configurable (e.g.,
MAX(100, @days_back)or a separate parameter)
🔧 Possible fix to align CTE window with daysBack
WHERE
suite = @suite_name
- AND date >= DATE_SUB(CURRENT_DATE(), INTERVAL 100 DAY)
+ AND date >= DATE_SUB(CURRENT_DATE(), INTERVAL @days_back DAY)
AND date <= CURRENT_DATE()📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| WITH earliest_test_dates AS ( | |
| -- Pre-compute the earliest date each test was seen across all releases for this suite | |
| SELECT | |
| test_name, | |
| MIN(date) AS first_sample_date | |
| FROM | |
| DATA_SET_LOCATION.TestsSummaryByDate | |
| WHERE | |
| suite = @suite_name | |
| AND date >= DATE_SUB(CURRENT_DATE(), INTERVAL 100 DAY) | |
| AND date <= CURRENT_DATE() | |
| GROUP BY | |
| test_name | |
| ) | |
| WITH earliest_test_dates AS ( | |
| -- Pre-compute the earliest date each test was seen across all releases for this suite | |
| SELECT | |
| test_name, | |
| MIN(date) AS first_sample_date | |
| FROM | |
| DATA_SET_LOCATION.TestsSummaryByDate | |
| WHERE | |
| suite = @suite_name | |
| AND date >= DATE_SUB(CURRENT_DATE(), INTERVAL @days_back DAY) | |
| AND date <= CURRENT_DATE() | |
| GROUP BY | |
| test_name | |
| ) |
|
@neisw: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
2 participants
Adds
testsdata typeProduces test_summaries.json
Need to review the importance of this if we are pivoting to openshift/origin#30604