-
Notifications
You must be signed in to change notification settings - Fork 2
Open
Description
Issue #152 suggested this feature idea
When working with data from different sources, different days, different URLs etc, the pipeline should be able to be launched with variable names
E.g: data team periodically downloads "employees.csv" from multiple companies and it's also important what day it's downloaded
> python3 -m phaser run employee_pipeline output "sources/employees.csv" --var company_name="Example" --var date="2024-06-06"
I guess this would be done like this:
- in the CLI, add an argument name="--var", with the 'append' action (using python's argparse library)
- split each value of 'var' on the equal sign
- after the pipeline is created, get its context and use add_variable for each value
Metadata
Metadata
Assignees
Labels
No labels