[Epic] New dbt Engine: Native SQL Parsing with SQLGlot and Full dbt Compatibility

[ismailsimsek]

This is an epic to fundamentally upgrade the core dbt parsing and compilation process by integrating a robust SQL parser, like SQLGlot. This foundational change will enable a native understanding of SQL, which will unlock powerful new features, and allow for a more streamlined user experience.

---

🚀 Motivation / Problem

The current dbt compilation primarily relies on Jinja rendering, which limits its ability to understand the underlying SQL structure natively. This results in key limitations:

1. Manual Dependency Management: Requires explicit ref() and source() calls even when dependencies are clearly visible in the SQL code.
2. Limited Advanced Features: Hindrance in implementing modern features from other tools (like sqlmesh), such as pure Python-based macros and more advanced dependency handling.
3. Optimization Constraints: Difficulty in performing SQL-aware optimizations and advanced query transformations.

✨ Proposed Solution / Scope

The goal is to implement a new dbt parser/compiler that uses SQLGlot to natively understand and process SQL models, while maintaining full backwards compatibility with existing dbt projects.

Core Features (Phase 1)

• Native Dependency Resolution: Automatically parse SQL code and create model dependencies without requiring the use of ref() or source(). Users should be able to write pure, standard SQL models.
• Full dbt-Core Compatibility: Ensure the new engine can correctly parse and compile all existing dbt constructs, including:
  • ref() and source() function calls.
  • Other dbt and custom Jinja macros used within SQL. config() and other
• Modern Macro Support: Integrate support for advanced features like Python-based macros (similar to sqlmesh).

Optional / Future Enhancements (Phase 2)

• Simplify Complexities: Explore opportunities to simplify or abstract away existing dbt complexities where native SQL parsing makes them redundant.
• SQL-Aware Optimizations: Leverage SQLGlot's parsing capabilities to enable advanced compilation features (e.g., query rewriting, dialect conversion).

🎯 End Result

A next-generation dbt engine that offers the simplicity and native SQL capabilities of tools like sqlmesh, combined with the wide adoption and ecosystem of dbt. This will significantly improve the developer experience by allowing users to focus on writing standard SQL and leveraging powerful Python-based macros.

🤝 Community Support:

This is a foundational innovation that requires deep software engineering and architectural expertise. We are actively seeking collaboration and contributions from experienced individuals in the community. Please join the discussion, feel free to submit your design doc or mvp solution.

We are looking forward to community support and discussion to help drive this foundational innovation. The help wanted label has been applied.