Prefect 3 Upgrade

[davramov]

Preparing for upgrading our Prefect Server from v2.x to v3.4.2

Other notes

• Adding "file_path" basename to flow run names for easier identification in the UI (addressing Flow names should contain dataset name #82)
• including a new files to streamline deployment with
  • orchestration/flows/bl832/prefect.yaml : Defines which flows from which files to deploy, and assign them to a work pool and queue
  • init_832_work_pools.sh : Creates work pools if they don't already exist, and deploys all of the flows defined above in prefect.yaml
• updating Prefect version in requirements.txt
• updating pytests and flows scripts to address updates within Prefect

[davramov]

To do:
Update Prefect JSON Blocks (deprecated) to Prefect Variables. Existing relevant JSON Blocks to replicate:

• [alcf-allocation-root-path] -- {"alcf-allocation-root-path":"/eagle/IRIProd/ALS"}
• [bl832-settings] -- {"delete_spot832_files_after_days":1,"delete_data832_files_after_days":35}
• [globus-settings] -- {"@doc_max_wait_seconds_desc":"Maximum time to wait around for a globus job. The job may still complete, but if globus tasks take longer, an error will be thrown","max_wait_seconds":600}
• [pruning-config] -- {"max_wait_seconds":120, "delete_alcf832_files_after_days":1, "delete_data832_files_after_days":14, "delete_nersc832_files_after_days":7}
• [decision-settings] -- {"nersc_recon_flow/nersc_recon_flow":true, "new_832_file_flow/new_file_832":true, "alcf_recon_flow/alcf_recon_flow":false}

[dylanmcreynolds]

Do you need to preserve the scripts in "legacy"? In general I like to keep the trail free of dead branches :)

[davramov]

Do you need to preserve the scripts in "legacy"? In general I like to keep the trail free of dead branches :)

No, I can delete these 👍