Skip to content

Conversation

@shangyian
Copy link
Collaborator

@shangyian shangyian commented Jan 5, 2026

Summary

This PR adds support for automatically detecting and using pre-aggregated tables during SQL generation in v3 SQL building. When a pre-aggregation exists that can satisfy a query's requirements, we will read from the pre-aggregated table instead of computing from source data, which helps to improve query perf.

We find pre-aggregations that satisfy query requirements based on:

  • The parent node (node_revision_id)
  • Grain coverage (preagg grain is superset of requested grain)
  • Measure coverage (preagg measures cover required measures by expression hash)
  • Availability state (materialized and available)

When a matching pre-agg is found, we generate SQL that reads from the pre-aggregation's materialized table and re-aggregates it to the requested grain using merge functions. This also preserves all metadata (column types, semantic entities).

This PR also adds include_temporal_filters and lookback_window as parameters for incremental materialization.

Test Plan

  • PR has an associated issue: #
  • make check passes
  • make test shows 100% unit test coverage

Deployment Plan

@netlify
Copy link

netlify bot commented Jan 5, 2026

Deploy Preview for thriving-cassata-78ae72 canceled.

Name Link
🔨 Latest commit 1e13b8a
🔍 Latest deploy log https://app.netlify.com/projects/thriving-cassata-78ae72/deploys/695bf54578d8bf000858b726

@shangyian shangyian marked this pull request as ready for review January 5, 2026 18:54
@shangyian shangyian merged commit 05ccb45 into DataJunction:main Jan 5, 2026
28 of 29 checks passed
@shangyian shangyian deleted the preagg-substitution branch January 5, 2026 19:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant