Support partial/delta refreshes of materialized views

[niteshpurohit]

Feature Request

Description:

Implement a strategy for partial or incremental refreshes, where only the changed or new data is updated in the materialized view, rather than performing a full refresh.

Use Case:

For very large tables, a full refresh can be prohibitively slow. An incremental strategy can reduce refresh times from hours to minutes, enabling near real-time data.

Proposed Solution:

• This is highly dependent on the database and application logic.
• One approach: require a last_updated_at column on source tables.
• The refresh logic would then delete rows from the mat view corresponding to recent source table updates, and then re-insert the new/updated data.
• This would likely be a new, advanced refresh strategy selectable by the user.

RSpec Tests:

• Implement a testable incremental refresh strategy.
• Seed data, then update a subset of it.
• Run the incremental refresh and verify that the view is updated correctly without performing a full refresh.
• Verify that a full refresh is triggered if the incremental strategy cannot be used.

Checklist:

• I have searched for a similar feature request and didn't find any duplicates.
• I have included all the required information.
• I have tagged the issue with the label "enhancement".

---