Filter partitions on test unique_stg_ga4__events_event_key
#365
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reduce compute by limiting the test
unique_stg_ga4__events_event_keyto the last 7 days (event_timestampis already in the hash so no need to test all partitions)Description & motivation
By default the unique_stg_ga4__events_event_key test scans all partitions which can be very expensive, at 2TB in this example:
Scanning all partitions is not required as the event_key already includes the event timestamp.
The same test on the same data is 100 times less expensive at 22GB when adding a partition filter on the last 7 days.
Checklist
dbt testto validate existing testspython -m pytest .to validate existing testsFor a reason I can't explain most integration test are failing with FileNotFoundError despite being in the unit_tests directory, existing files in other directories, and being able to view the file find the file on direct executions like
python -c "from dbt.tests.util import read_file; print(read_file('../macros/default_channel_grouping.sql'))"ERROR test_macro_default_channel_grouping.py::TestDefaultChannelGrouping::test_mock_run_and_check - FileNotFoundError: [Errno 2] No such file or directory: '../macros/default_channel_grouping.sql'
[... 8 others]
========================================================== 4 passed, 141 warnings, 9 errors in 70.85s (0:01:10) ==========================================================