Skip to content

Add containerized services, observability, Wikimedia API guards, and ops docs#3

Open
artvandelay wants to merge 1 commit intomainfrom
codex/add-docker-support-for-services
Open

Add containerized services, observability, Wikimedia API guards, and ops docs#3
artvandelay wants to merge 1 commit intomainfrom
codex/add-docker-support-for-services

Conversation

@artvandelay
Copy link
Owner

Motivation

  • Provide production-ready service scaffolding for ingest/enrich/detect/report pipelines with containerization and observability.
  • Protect the Wikimedia ingest path with rate-limiting and circuit-breaking to avoid outages and throttling.
  • Add storage migrations and reliable backup/restore tooling to make data operations repeatable.
  • Ship runbooks and configuration documentation so operators can restart services and perform backfills.

Description

  • Add FastAPI service entrypoints: src/nlbt/services/ingest.py, enrich.py, detect.py, and report.py with structured logging and metrics integrated via src/nlbt/services/observability.py.
  • Implement src/nlbt/services/wikimedia.py with a RateLimiter, CircuitBreaker, and WikimediaClient used by the ingest service to guard Wikimedia API calls.
  • Add container Dockerfiles under services/*, database migrations and schema snapshots under storage/, and operational scripts scripts/db_migrate.sh, scripts/db_backup.sh, and scripts/db_restore.sh.
  • Document configuration and runbooks in docs/configuration.md and docs/runbooks.md, update env.example, pyproject.toml (adds runtime deps), and reference the new materials from README.md.

Testing

  • No automated tests were run against these changes.
  • Files were validated by listing and viewing the added modules and scripts as part of the rollout (no unit/integration test execution).

Codex Task

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant