-
Notifications
You must be signed in to change notification settings - Fork 1
Open
Labels
M3-automationMilestone 3: AutomationMilestone 3: AutomationdocumentationREADME, methodology, data dictionaryREADME, methodology, data dictionarypipelinePython scripts that fetch/parse/merge dataPython scripts that fetch/parse/merge data
Milestone
Description
Prevent hammering ICE/ACLU servers during local development.
Tasks
- Verify
python scripts/fetch_287g.py --force-cacheworks end-to-end (html already saved by first run) - Add a
Makefile(orscripts/run_pipeline.sh) with:make fetch— live fetch, save cachemake local—--force-cacheonly (fast, no network)make merge— run merge stepmake all— full pipeline
- Document in README under "Running the pipeline locally"
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
M3-automationMilestone 3: AutomationMilestone 3: AutomationdocumentationREADME, methodology, data dictionaryREADME, methodology, data dictionarypipelinePython scripts that fetch/parse/merge dataPython scripts that fetch/parse/merge data