E-Commerce Event Analytics Pipeline

Complete end-to-end analytics pipeline for e-commerce event data transformation, attribution modeling, and monitoring.

Author: Kushal Project: Puffy E-Commerce Analytics Date: November 2025

📋 Table of Contents

Overview
Project Structure
Prerequisites
Quick Start
Part-by-Part Instructions
Full Pipeline Execution
Output Files
Troubleshooting
Technical Documentation

Overview

This project implements a production-ready data pipeline for e-commerce event analytics, covering:

Data Quality Analysis - Identify and fix data issues before transformation
Data Transformation - Convert raw events into analytics-ready tables (Medallion architecture)
Business Insights - Generate executive-ready reports with actionable recommendations
Monitoring & Validation - Ensure pipeline accuracy with statistical anomaly detection

Key Features:

📊 290 conversions, $294,461 revenue analyzed (14-day period)
🎯 First-click & last-click attribution modeling
📈 Statistical monitoring with Z-score anomaly detection
✅ 0.000000% revenue error - perfect data integrity
📄 Professional PDF reports for stakeholders

Project Structure

puffy/
├── README.md                          # This file
├── requirements.txt                   # Python dependencies
├── run_all_pipelines.py              # End-to-end orchestrator script
│
├── part1-data-quality/               # Part 1: Data Quality Analysis
│   ├── eda.py                        # Exploratory data analysis script
│   ├── session_timeout_analysis.png  # Session timeout visualization
│   └── data_quality_report.md        # Data quality assessment
│
├── part2-transformation/             # Part 2: Data Transformation Pipeline
│   ├── transformation.py             # Main transformation script (Bronze→Silver→Gold)
│   ├── create_star_schema.py         # Star schema diagram generator
│   ├── documentation.md              # Comprehensive transformation docs (96 KB)
│   ├── validation.md                 # Validation proof (18 checks)
│   ├── attribution_report.md         # Attribution analysis report
│   │
│   ├── sessions.csv                  # Silver: 38,266 sessions
│   ├── user_journeys.csv             # Silver: User-level aggregations
│   ├── conversion_events.csv         # Silver: 290 conversions
│   ├── fct_events.csv                # Silver: 48,184 events
│   ├── fct_sessions.csv              # Silver: Session facts
│   ├── fct_orders.csv                # Silver: Order facts
│   ├── fct_touchpoints.csv           # Silver: Marketing touchpoints
│   ├── dim_user.csv                  # Silver: User dimension
│   │
│   ├── fct_attribution.csv           # Gold: Attribution facts (first/last click)
│   ├── attribution_first_click.csv   # Gold: First-click aggregations
│   ├── attribution_last_click.csv    # Gold: Last-click aggregations
│   ├── agg_channel_performance.csv   # Gold: Channel metrics
│   ├── agg_user_engagement.csv       # Gold: User engagement metrics
│   ├── agg_session_analytics.csv     # Gold: Session analytics
│   └── agg_funnel_analysis.csv       # Gold: Conversion funnel metrics
│
├── part3-analysis/                   # Part 3: Executive Summary
│   ├── executive_summary.md          # Executive-ready markdown report (8.5 KB)
│   ├── executive_summary.pdf         # Professional PDF report (352 KB)
│   ├── convert_to_pdf.py             # PDF conversion script
│   └── venv/                         # Virtual environment for PDF generation
│
└── part4-monitoring/                 # Part 4: Monitoring & Validation
    ├── documentation.md              # Monitoring system design (21 KB, 573 lines)
    ├── monitor.py                    # Monitoring implementation (30 KB, 786 lines)
    ├── README.md                     # Monitoring usage guide (13 KB)
    ├── monitoring_report.md          # Detailed monitoring analysis (14 KB)
    ├── monitoring_report.pdf         # Professional PDF report (307 KB)
    ├── monitoring_report_YYYYMMDD.json  # JSON monitoring results
    └── convert_monitoring_to_pdf.py  # PDF conversion script

Total Deliverables:

4 Python pipelines (1,824 lines of code)
6 PDF reports (659 KB professional documentation)
17 CSV outputs (92 MB transformed data)
Comprehensive documentation (150+ pages)

Prerequisites

System Requirements

Python: 3.8 or higher
Operating System: macOS, Linux, or Windows
RAM: Minimum 4 GB (8 GB recommended)
Disk Space: ~500 MB (for data + outputs)

Verify Python Installation

python3 --version
# Should output: Python 3.8.x or higher

If Python is not installed:

macOS: brew install python3
Linux: sudo apt-get install python3 python3-pip
Windows: Download from python.org

Quick Start

Option 1: Run Full Pipeline (Recommended)

# 1. Navigate to project directory
cd /Users/apple/Documents/FlutterDev/puffy

# 2. Install dependencies
pip3 install -r requirements.txt

# 3. Run entire pipeline (all 4 parts)
python3 run_all_pipelines.py

Expected Runtime: ~5-10 minutes (with PDF generation)

What This Does:

✅ Analyzes data quality (Part 1)
✅ Transforms raw events → analytics tables (Part 2)
✅ Generates executive summary PDF (Part 3)
✅ Runs monitoring checks + report PDF (Part 4)

Option 2: Fast Execution (Skip PDFs)

# Faster execution - skip PDF generation
python3 run_all_pipelines.py --skip-pdf

Expected Runtime: ~2-3 minutes

Option 3: Skip Data Quality Analysis

# Use existing Part 1 results, run Parts 2-4 only
python3 run_all_pipelines.py --skip-part1

Part-by-Part Instructions

Part 1: Data Quality Analysis

Purpose: Identify data quality issues, validate session timeout, explore distributions

Input: ../part1-data-quality/cleaned_events.csv (from previous data cleaning step)

Steps:

# Navigate to Part 1 directory
cd part1-data-quality

# Run exploratory data analysis
python3 eda.py

Outputs:

session_timeout_analysis.png - Visualization proving 30-minute timeout is optimal
data_quality_report.md - Detailed data quality assessment
Console output with statistics and findings

What It Does:

Analyzes session timeout (validates 30-minute cutoff)
Calculates null rates (74.4% null referrers identified)
Detects outliers (1 revenue outlier: $15,888)
Validates data distributions
Generates recommendations for Part 2 transformation

Expected Runtime: ~30 seconds

Part 2: Data Transformation

Purpose: Transform raw events into analytics-ready tables using Medallion architecture (Bronze → Silver → Gold)

Input: ../part1-data-quality/cleaned_events.csv

Steps:

# Navigate to Part 2 directory
cd part2-transformation

# Run transformation pipeline
python3 transformation.py

What It Does:

Bronze Layer (Raw):

Loads cleaned events (49,963 rows)

Silver Layer (Sessionized & Cleaned):

Creates sessions using 30-minute timeout (38,266 sessions)
Sessionizes events (48,184 events with session_id)
Identifies conversions (290 orders, $294,461 revenue)
Builds user journeys
Creates fact tables: fct_events, fct_sessions, fct_orders, fct_touchpoints
Creates dimension: dim_user

Gold Layer (Business-Ready):

Generates attribution models (first-click, last-click)
Aggregates channel performance
Creates user engagement metrics
Builds session analytics
Generates funnel analysis

Additional Outputs:

attribution_report.md - Marketing attribution analysis
validation.md - Data validation proof (18 checks, all PASS)
documentation.md - Complete transformation documentation (96 KB)

Expected Runtime: ~1-2 minutes

Optional: Generate star schema diagram

# Requires graphviz: brew install graphviz
python3 create_star_schema.py
# Output: star_schema_diagram.png

Part 3: Executive Summary

Purpose: Generate executive-ready business performance report with insights and recommendations

Input: Transformation outputs from Part 2

Steps:

# Navigate to Part 3 directory
cd part3-analysis

# Executive summary already created, just verify
ls -lh executive_summary.md

# Generate PDF (optional but recommended)
# First, set up virtual environment
python3 -m venv venv
source venv/bin/activate
pip install markdown weasyprint

# Convert to PDF
python convert_to_pdf.py

Outputs:

executive_summary.md - 2-page markdown report
executive_summary.pdf - Professional PDF (352 KB)

Report Contents:

Business Performance Overview - $294,461 revenue, $1,015 AOV, $7.7M run rate
What's Working - Referral dominance (50%), fast conversions (87.6% single-touch)
What's Concerning - Revenue concentration risk, mobile conversion gap (33% lower)
Key Behavioral Patterns - 12-minute median journey, mobile-first audience
Strategic Recommendations - Prioritized actions (Week 1, Month 1, Q1)
Growth Potential - $2.6-3.4M incremental annual revenue opportunity

Expected Runtime: ~1 minute

Part 4: Monitoring & Validation

Purpose: Validate pipeline accuracy, detect anomalies, ensure data integrity

Input: Transformation outputs from Part 2

Steps:

# Navigate to project root
cd /Users/apple/Documents/FlutterDev/puffy

# Run monitoring checks
python3 part4-monitoring/monitor.py \
  --data-dir part2-transformation \
  --output-dir part4-monitoring

What It Monitors:

Priority 0: Critical (Pipeline Blockers)

✅ Data Freshness - Is data arriving daily? (< 30 hours old)
✅ Revenue Integrity - Does revenue reconcile? (0.000000% error - PERFECT)
✅ Pipeline Health - All stages completed? (All ratios within bounds)

Priority 1: Data Quality 4. ⚠️ Volume Anomaly - Statistical Z-score detection (expected CRITICAL for batched data) 5. ✅ Data Quality - Null rates, duplicates, invalid values (All PASS)

Priority 2: Business Metrics 6. ✅ Business Metrics - Conversion rate, AOV, revenue/session (All normal)

Outputs:

monitoring_report_YYYYMMDD.json - Machine-readable results
monitoring_report.md - Detailed human-readable analysis (14 KB)
monitoring_report.pdf - Professional PDF report (307 KB)

Expected Results:

Overall Status: 🔴 CRITICAL (volume anomaly - false positive for batched data)

✅ Revenue Integrity: PASS (0.000000% error)
✅ Pipeline Health: PASS (all stages operational)
🔴 Volume Anomaly: CRITICAL (expected - 14 days batched vs daily incremental)
✅ Data Quality: PASS (73.4% null referrers, 0% invalid revenue)
✅ Business Metrics: PASS (CVR, AOV, RPS within normal ranges)

Note: Volume anomaly CRITICAL status is expected behavior because we're loading 14 days of historical data as a single batch. In production with daily incremental loads of ~3,500 events/day, this would PASS.

Expected Runtime: ~30 seconds

Optional: Generate monitoring PDF

cd part4-monitoring

# Use Part 3 venv for PDF generation
../part3-analysis/venv/bin/python convert_monitoring_to_pdf.py

Full Pipeline Execution

Run All Parts Automatically

# Navigate to project directory
cd /Users/apple/Documents/FlutterDev/puffy

# Install dependencies (first time only)
pip3 install -r requirements.txt

# Run full pipeline
python3 run_all_pipelines.py

Expected Output:

████████████████████████████████████████████████████████████████████████████████
█                                                                              █
█        E-COMMERCE EVENT ANALYTICS - END-TO-END PIPELINE ORCHESTRATOR         █
█                                                                              █
████████████████████████████████████████████████████████████████████████████████

🕐 Started: 2025-11-30 15:30:00
📁 Working Directory: /Users/apple/Documents/FlutterDev/puffy

================================================================================
  PART 1: DATA QUALITY ANALYSIS
================================================================================

────────────────────────────────────────────────────────────────────────────────
📍 1.1: Running Exploratory Data Analysis (EDA)
────────────────────────────────────────────────────────────────────────────────

▶ Generate session timeout analysis and visualizations
  Command: python3 eda.py

[... EDA output ...]

✅ Success: Generate session timeout analysis and visualizations

📊 Checking outputs...
   ✅ Found: session_timeout_analysis.png (621,525 bytes)
   ✅ Found: data_quality_report.md (12,345 bytes)

================================================================================
  PART 2: DATA TRANSFORMATION (Bronze → Silver → Gold)
================================================================================

────────────────────────────────────────────────────────────────────────────────
📍 2.1: Running Data Transformation Pipeline
────────────────────────────────────────────────────────────────────────────────

[... Transformation output ...]

✅ Success: Transform raw events → sessions → conversions → attribution

📊 Checking transformation outputs...
   ✅ Found: sessions.csv (16,794,197 bytes, 38,266 rows)
   ✅ Found: conversion_events.csv (75,613 bytes, 290 rows)
   ✅ Found: fct_attribution.csv (63,417 bytes, 580 rows)
   [... more files ...]

================================================================================
  PART 3: EXECUTIVE SUMMARY REPORT GENERATION
================================================================================

────────────────────────────────────────────────────────────────────────────────
📍 3.1: Verifying Executive Summary
────────────────────────────────────────────────────────────────────────────────

   ✅ Found: executive_summary.md (8,755 bytes)

────────────────────────────────────────────────────────────────────────────────
📍 3.2: Converting Executive Summary to PDF
────────────────────────────────────────────────────────────────────────────────

Converting executive_summary.md to PDF...
✓ PDF created: executive_summary.pdf

   ✅ PDF generated: 352,000 bytes

================================================================================
  PART 4: MONITORING & VALIDATION
================================================================================

────────────────────────────────────────────────────────────────────────────────
📍 4.1: Running Pipeline Monitoring Checks
────────────────────────────────────────────────────────────────────────────────

ℹ️  Note: Volume anomaly detection may show CRITICAL status
   This is expected for batched historical data (14 days loaded at once)
   In production with daily incremental loads, this would PASS

[... Monitoring output ...]

✅ Monitoring report generated: monitoring_report_20251130.json
   Size: 4,000 bytes

================================================================================
  PIPELINE EXECUTION SUMMARY
================================================================================

⏱️  Total Execution Time: 180.5 seconds (3.0 minutes)

📊 Results by Part:

   ✅ SUCCESS            │ Part 1: Data Quality Analysis
   ✅ SUCCESS            │ Part 2: Data Transformation
   ✅ SUCCESS            │ Part 3: Executive Summary
   ✅ SUCCESS            │ Part 4: Monitoring & Validation

✅ PIPELINE COMPLETED SUCCESSFULLY

📂 Output Locations:
   • Part 1: /Users/apple/Documents/FlutterDev/puffy/part1-data-quality/
   • Part 2: /Users/apple/Documents/FlutterDev/puffy/part2-transformation/
   • Part 3: /Users/apple/Documents/FlutterDev/puffy/part3-analysis/
   • Part 4: /Users/apple/Documents/FlutterDev/puffy/part4-monitoring/

Command-Line Options

# Full pipeline (default)
python3 run_all_pipelines.py

# Skip Part 1 (use existing data quality results)
python3 run_all_pipelines.py --skip-part1

# Skip PDF generation (faster, markdown reports only)
python3 run_all_pipelines.py --skip-pdf

# Skip both (fastest)
python3 run_all_pipelines.py --skip-part1 --skip-pdf

# Help
python3 run_all_pipelines.py --help

Output Files

Part 1: Data Quality Analysis

File	Size	Description
`session_timeout_analysis.png`	621 KB	Visualization proving 30-min timeout is optimal
`data_quality_report.md`	~12 KB	Data quality assessment and recommendations

Part 2: Data Transformation

Silver Layer (Fact Tables):

File	Rows	Size	Description
`sessions.csv`	38,266	16.8 MB	Sessionized events with 30-min timeout
`user_journeys.csv`	16,026	4.7 MB	User-level aggregations
`conversion_events.csv`	290	76 KB	Purchase events with attribution metadata
`fct_events.csv`	48,184	18.2 MB	Event-level facts
`fct_sessions.csv`	38,266	16.8 MB	Session-level facts
`fct_orders.csv`	290	45 KB	Order-level facts
`fct_touchpoints.csv`	38,266	2.5 MB	Marketing touchpoint facts
`dim_user.csv`	16,026	4.7 MB	User dimension

Gold Layer (Aggregations):

File	Description
`fct_attribution.csv`	Attribution facts (first-click, last-click)
`attribution_first_click.csv`	First-click channel aggregations
`attribution_last_click.csv`	Last-click channel aggregations
`agg_channel_performance.csv`	Channel-level metrics
`agg_user_engagement.csv`	User engagement metrics
`agg_session_analytics.csv`	Session analytics
`agg_funnel_analysis.csv`	Conversion funnel metrics

Documentation:

File	Size	Description
`documentation.md`	96 KB	Complete transformation documentation
`validation.md`	46 KB	Validation proof (18 checks, all PASS)
`attribution_report.md`	5.4 KB	Marketing attribution analysis

Part 3: Executive Summary

File	Size	Description
`executive_summary.md`	8.5 KB	Markdown executive summary
`executive_summary.pdf`	352 KB	Professional PDF report

Part 4: Monitoring & Validation

File	Size	Description
`documentation.md`	21 KB	Monitoring system design (573 lines)
`monitor.py`	30 KB	Monitoring implementation (786 lines)
`README.md`	13 KB	Monitoring usage guide
`monitoring_report_YYYYMMDD.json`	4 KB	Machine-readable monitoring results
`monitoring_report.md`	14 KB	Detailed monitoring analysis
`monitoring_report.pdf`	307 KB	Professional PDF report

Troubleshooting

Common Issues

1. `ModuleNotFoundError: No module named 'pandas'`

Problem: Python dependencies not installed

Solution:

pip3 install -r requirements.txt

2. `FileNotFoundError: cleaned_events.csv not found`

Problem: Input data file missing

Solution: Ensure cleaned_events.csv exists in part1-data-quality/ directory.

ls -lh part1-data-quality/cleaned_events.csv

If missing, you need to run the data cleaning step (Part 0) first.

3. PDF Generation Fails

Problem: weasyprint dependencies missing (macOS)

Solution:

# Install system dependencies
brew install pango cairo

# Reinstall weasyprint
pip3 install --force-reinstall weasyprint

4. `Permission denied` when running scripts

Problem: Scripts not executable

Solution:

chmod +x run_all_pipelines.py
chmod +x part4-monitoring/monitor.py

5. Virtual Environment Issues (Part 3 PDF)

Problem: PDF generation fails in Part 3

Solution:

# Recreate virtual environment
cd part3-analysis
rm -rf venv
python3 -m venv venv
source venv/bin/activate
pip install markdown weasyprint
python convert_to_pdf.py

6. Monitoring Shows All CRITICAL

Problem: Volume anomaly detection triggers false positives

Explanation: This is expected behavior when running on batched historical data (14 days loaded at once). The monitoring system expects daily incremental loads.

Solution: No action needed. In production with daily loads of ~3,500 events, this would PASS.

Technical Documentation

Architecture Overview

Medallion Architecture (Bronze → Silver → Gold):

┌─────────────────────────────────────────────────────────────┐
│  BRONZE LAYER (Raw Data)                                    │
│  • cleaned_events.csv (49,963 events)                       │
│  • Direct copy from data cleaning step                      │
└─────────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────────┐
│  SILVER LAYER (Cleaned & Sessionized)                       │
│  • Sessionization (30-min timeout → 38,266 sessions)        │
│  • Event-level facts (48,184 events)                        │
│  • User journeys (16,026 users)                             │
│  • Conversions identified (290 orders, $294K revenue)       │
│  • Marketing touchpoints extracted                          │
└─────────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────────┐
│  GOLD LAYER (Business-Ready Analytics)                      │
│  • Attribution models (first-click, last-click)             │
│  • Channel performance aggregations                         │
│  • User engagement metrics                                  │
│  • Session analytics                                        │
│  • Funnel analysis (Session → Cart → Checkout → Purchase)  │
└─────────────────────────────────────────────────────────────┘

Key Technical Decisions

1. Session Timeout: 30 Minutes

Empirically validated via EDA (Part 1)
95th percentile of inter-event times: 28.7 minutes
Balances accuracy vs granularity

2. Attribution Window: 7 Days

Industry standard for e-commerce
Captures majority of customer journeys
87.6% convert in single/zero-touch (median: 12 minutes)

3. User Identification: Cookie-Based

client_id = user_id (cookie-based tracking)
No cross-device tracking in current implementation
Trade-off documented in Part 2 documentation.md

4. Marketing Channel Classification:

Direct (43.3% revenue)
Referral (50.7% revenue)
Organic Search (6.0% revenue)
Paid Search, Social, Email (currently $0)

5. Monitoring Approach: Statistical Z-Score

Z > 3.0 = WARNING (99.7% confidence, 1 in 370 days)
Z > 4.0 = CRITICAL (99.99% confidence)
Adapts to business growth and seasonality

Performance Metrics

Pipeline Performance:

Input: 49,963 events
Output: 17 CSV files, 92 MB total
Execution Time: ~3-5 minutes (full pipeline)
Memory Usage: < 2 GB RAM

Data Quality:

Revenue Conservation: 0.000000% error
Event Conservation: 100% (48,184 + 1,779 excluded = 49,963)
Null Referrer Rate: 73.4% (tracked, stable)

Business Metrics:

Conversions: 290 orders
Revenue: $294,461.40
Average Order Value: $1,015.38
Conversion Rate: 0.76%
Revenue Per Session: $7.70

Additional Resources

Documentation by Part

Part 1:

part1-data-quality/data_quality_report.md - Data quality assessment

Part 2:

part2-transformation/documentation.md - Complete transformation guide (96 KB)
part2-transformation/validation.md - Validation proof (18 checks)
part2-transformation/attribution_report.md - Attribution analysis

Part 3:

part3-analysis/executive_summary.md - Executive insights
part3-analysis/executive_summary.pdf - Professional PDF

Part 4:

part4-monitoring/documentation.md - Monitoring system design (21 KB)
part4-monitoring/README.md - Monitoring usage guide (13 KB)
part4-monitoring/monitoring_report.md - Detailed analysis
part4-monitoring/monitoring_report.pdf - Professional PDF

Contact & Support

Questions? Review the detailed documentation in each part's directory.

Issues? Check the Troubleshooting section above.

Summary

This project delivers a complete, production-ready analytics pipeline with:

✅ Data Quality Validation - 30-min timeout validated, 74.4% null referrers identified ✅ Transformation Pipeline - Bronze → Silver → Gold with 0% revenue error ✅ Business Insights - $2.6-3.4M growth opportunity identified ✅ Monitoring System - Statistical anomaly detection with 99.7% confidence ✅ Professional Reports - 6 PDF reports totaling 659 KB

Total Deliverables:

4 Python pipelines (1,824 lines)
17 CSV outputs (92 MB)
6 PDF reports (659 KB)
150+ pages documentation

Ready for production deployment! 🚀

Version: 1.0 Last Updated: 2025-11-30 Author: Kushal

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
part1-data-quality		part1-data-quality
part2-transformation		part2-transformation
part3-analysis		part3-analysis
part4-monitoring		part4-monitoring
README.md		README.md
requirements.txt		requirements.txt
run_all_pipelines.py		run_all_pipelines.py
value_over_ai.md		value_over_ai.md
value_over_ai.pdf		value_over_ai.pdf

Kloud23/puffy-task

Folders and files

Latest commit

History

Repository files navigation

E-Commerce Event Analytics Pipeline

📋 Table of Contents

Overview

Project Structure

Prerequisites

System Requirements

Verify Python Installation

Quick Start

Option 1: Run Full Pipeline (Recommended)

Option 2: Fast Execution (Skip PDFs)

Option 3: Skip Data Quality Analysis

Part-by-Part Instructions

Part 1: Data Quality Analysis

Part 2: Data Transformation

Part 3: Executive Summary

Part 4: Monitoring & Validation

Full Pipeline Execution

Run All Parts Automatically

Command-Line Options

Output Files

Part 1: Data Quality Analysis

Part 2: Data Transformation

Part 3: Executive Summary

Part 4: Monitoring & Validation

Troubleshooting

Common Issues

1. ModuleNotFoundError: No module named 'pandas'

2. FileNotFoundError: cleaned_events.csv not found

3. PDF Generation Fails

4. Permission denied when running scripts

5. Virtual Environment Issues (Part 3 PDF)

6. Monitoring Shows All CRITICAL

Technical Documentation

Architecture Overview

Key Technical Decisions

Performance Metrics

Additional Resources

Documentation by Part

Contact & Support

Summary

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Languages

1. `ModuleNotFoundError: No module named 'pandas'`

2. `FileNotFoundError: cleaned_events.csv not found`

4. `Permission denied` when running scripts

Packages