Skip to content

[PIPELINE] Add --force-cache mode and local dev Makefile #10

@wi287g

Description

@wi287g

Prevent hammering ICE/ACLU servers during local development.

Tasks

  • Verify python scripts/fetch_287g.py --force-cache works end-to-end (html already saved by first run)
  • Add a Makefile (or scripts/run_pipeline.sh) with:
    • make fetch — live fetch, save cache
    • make local--force-cache only (fast, no network)
    • make merge — run merge step
    • make all — full pipeline
  • Document in README under "Running the pipeline locally"

Metadata

Metadata

Assignees

No one assigned

    Labels

    M3-automationMilestone 3: AutomationdocumentationREADME, methodology, data dictionarypipelinePython scripts that fetch/parse/merge data

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions