Data Engineering Pipeline

This project is a hands-on data engineering pipeline built using the modern data stack. It covers the full workflow from raw data ingestion to transformation, modeling, and analytics.

Tech Stack

• Dagster
• PostgreSQL
• dbt
• Python
• Docker

Dagster DAG overview

Repository Structure

├── big_star_warehouse
│    ├── README.md
│    ├── dbt_project.yml
│    ├── packages.yml
│    ├── models/
│    │   ├── staging/
│    │   ├── marts/
│    ├── target/
├── dagster_pipelien
│    ├── dagster_pipeline/
│    │   ├── __init__.py
│    │   ├── assets.py
│    │   ├── definitions.py
│    │   ├── schedules.py
├── README.md
├── docker-compose.yml
├── setup.py
├── .gitignore
├── .env
└── requirements.txt

Local setup

1. Create virtual environment
python -m venv env && source env/bin/activate

2. Install dependencies
python setup.py

👋 Built by Saurabh Joshi
If you found this helpful, feel free to connect or ⭐ the repo!