CI/CD	License	Python	Streamlit	Docker	GCP	Terraform

📰 Briefly

🔗 Live Demo

Briefly is a lightweight, AI-powered ETL pipeline that pulls trending news headlines, summarizes them using Google's Gemini API, and displays them in a clean web app interface. It's built with Python, Streamlit, and GCP — ideal for showcasing real-time NLP + data engineering skills.

🚀 Features

• Extract top news stories from Hacker News
• Summarize headlines using Gemini 1.5 Pro
• Display summaries in a dynamic Streamlit app
• Top navigation bar with Feed and Trending views
• Light/Dark theme toggle in the header
• Live date range and source filtering in the sidebar
• Preview logos for each article (with fallback)
• Optional support for BigQuery or CSV export
• Free-tier compatible (Google Gemini 1.5)

🧱 Tech Stack

• Python (ETL scripts)
• BigQuery (cloud data warehouse)
• Gemini API (summarization)
• Streamlit (web UI)
• Terraform (infra-as-code)
• Docker (optional for app deployment)

📂 Project Structure

briefly/
├── docker-compose.yaml
├── Dockerfile
├── etl
│   ├── __pycache__
│   ├── extract.py
│   ├── insert_sample_data.py
│   ├── list_models.py
│   ├── load.py
│   ├── run_pipeline.py
│   ├── setup_bigquery.py
│   ├── summarize.py
│   ├── test_bigquery.py
│   └── transform.py
├── LICENSE
├── notebooks
├── README.md
├── requirements.txt
├── terraform
│   ├── main.tf
│   ├── outputs.tf
│   ├── provider.tf
│   ├── terraform.tfstate
│   ├── terraform.tfstate.backup
│   ├── terraform.tfvars
│   └── variables.tf
├── venv
│   ├── bin
│   ├── etc
│   ├── include
│   ├── lib
│   ├── pyvenv.cfg
│   └── share
└── webapp
    └── app.py

🛠 System Requirements

To get started with this project, you'll need the following tools installed:

• Python 3.11+
• Terraform 1.3+
• Google Cloud SDK – required for authenticating with GCP and managing infrastructure
• Streamlit – installed via pip install -r requirements.txt

🔑 Environment Setup

1. Clone the repo
2. Create a .env file:

   GEMINI_API_KEY=your-api-key-here
   

3. Ensure your Google Cloud credentials are available:

   export GOOGLE_APPLICATION_CREDENTIALS=/path/to/your/service_account.json
   export GCP_PROJECT=your-gcp-project-id
   

   (Required for BigQuery integration)

4. Install dependencies:

   pip install -r requirements.txt
   

🧪 Run Locally

# Create and activate your virtual environment (if needed)
python3 -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Run full ETL pipeline (extract, summarize, and load into BigQuery)
python etl/run_pipeline.py

# Launch the frontend dashboard
streamlit run webapp/app.py

📡 BigQuery Integration

If you want to store and analyze summaries in BigQuery:

1. Set your GCP credentials and project ID as environment variables:

   export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service_account.json
   export GCP_PROJECT=your-gcp-project-id

2. Run the setup script to create the dataset and table:

   python etl/setup_bigquery.py

3. Use etl/run_pipeline.py to automatically push new summaries to BigQuery.

Summaries are stored in the briefly_data.summaries table with fields like url, title, summary, source, published_at, and summarized_at.

🏗️ Terraform Infrastructure

You can provision the required GCP infrastructure using Terraform:

1. Navigate to the Terraform directory:

   cd terraform/

2. Set your environment credentials (if not already):

   export GOOGLE_APPLICATION_CREDENTIALS=./.secrets/terraform-admin-key.json

3. Initialize the Terraform project:

   terraform init

4. Review the plan:

   terraform plan

5. Apply the infrastructure:

   terraform apply

Terraform will create:

• A BigQuery dataset and summaries table
• A service account with bigquery.user permissions
• GitHub Actions CI/CD validation pipeline

🧹 Terraform Cleanup and Remote Backend (Optional)

Destroy Infrastructure

To tear down all Terraform-managed resources:

terraform destroy

This will prompt you to confirm deletion of all provisioned infrastructure.

---

Use a Remote Backend (Optional but Recommended)

For team collaboration and state consistency, configure a remote backend using Google Cloud Storage (GCS):

1. Create a GCS bucket (e.g. briefly-terraform-state)

2. Enable versioning on the bucket:

   gsutil versioning set on gs://briefly-terraform-state

3. Add a backend config to your provider.tf or main.tf:

terraform {
  backend "gcs" {
    bucket  = "briefly-terraform-state"
    prefix  = "terraform/state"
  }
}

4. Reinitialize Terraform to migrate local state:

terraform init -migrate-state

This ensures your Terraform state is versioned, backed up, and team-ready.

📜 License

MIT — free to use, extend, and showcase.

✅ Project Status

This project is complete and production-ready. Further improvements (e.g. CI deployment, testing automation, or remote backends) can be added as future enhancements.