Skip to content

Conversation

@portalhacker
Copy link
Contributor

Reduce compute by limiting the test unique_stg_ga4__events_event_key to the last 7 days (event_timestamp is already in the hash so no need to test all partitions)

Description & motivation

By default the unique_stg_ga4__events_event_key test scans all partitions which can be very expensive, at 2TB in this example:

SCR-20250508-lpzr

Scanning all partitions is not required as the event_key already includes the event timestamp.
The same test on the same data is 100 times less expensive at 22GB when adding a partition filter on the last 7 days.

SCR-20250508-lqwc

Checklist

  • I have verified that these changes work locally
  • I have updated the README.md (if applicable)
  • I have added tests & descriptions to my models (and macros if applicable)
  • I have run dbt test to validate existing tests
  • I have run python -m pytest . to validate existing tests

For a reason I can't explain most integration test are failing with FileNotFoundError despite being in the unit_tests directory, existing files in other directories, and being able to view the file find the file on direct executions like python -c "from dbt.tests.util import read_file; print(read_file('../macros/default_channel_grouping.sql'))"

ERROR test_macro_default_channel_grouping.py::TestDefaultChannelGrouping::test_mock_run_and_check - FileNotFoundError: [Errno 2] No such file or directory: '../macros/default_channel_grouping.sql'
[... 8 others]
========================================================== 4 passed, 141 warnings, 9 errors in 70.85s (0:01:10) ==========================================================

portalhacker and others added 2 commits May 8, 2025 12:49
…t_key` to the last 7 days (`event_timestamp` is already in the hash so no need to test all partitions)
@mtcarlone mtcarlone self-assigned this Nov 14, 2025
@mtcarlone
Copy link
Collaborator

Hi - getting up to speed with the project. Please give me a little bit of time to go through this and get acquainted / review. Thanks in advance!

@mtcarlone
Copy link
Collaborator

@portalhacker can you please merge Velir:main into this branch to re-trigger CI? Thank you!

Copy link
Collaborator

@mtcarlone mtcarlone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. Pending CI

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants