Issue 57: 7.3.3 Flows #60
Conversation
kkamdin
left a comment
kkamdin
left a comment
There was a problem hiding this comment.
I did a fairly thorough review but then got to some comments about how this PR depends on another PR down in move.py and the other flows/bl733 files so I didn't review those super thoroughly... I'll definitely take another pass but wanted to submit the comments so far.
Let me know if you prefer to pair to go over my questions/comments, I know typing in replies and going back and forth can be a big time sink.
|
Looks good. How do I indicate that my review is no longer "pending" in the UI? Or can I just hit the "merge" button? :D |
Sweeeet. I need to push one more change so it stages data on Lamarr, then it should be ready to merge (: |
…uses the transfer_controller class. Created a Config733() class and updated config.yml with a few endpoints, anticipating a globus endpoint at the beamline. Updated the test_globus_flow.py pytest for the new move.py script.
…icating next steps, included a create_deployments_733.sh script.
…nce that PR is merged, let's use the prune_controller
… a class for now.
…y to avoid real calls to Prefect in the pytest
…s with Prefect3.x 🤔
…efect flows following the 3.4.2 best practices. Created a new config 'orchestration/flows/bl733/prefect.yaml' that defines the flows for bl733. Updating bl733 documentation to reflect these changes.
…re that the flow run names match the file names that are being processed.
…ted with Prefect 3. Hard coding the max_wait_seconds=600 variable for now.
…datetime. Also adding a check to ensure days_from_now is not negative
…l, for improved maintainability.
…IENT_SECRET in Prefect Secret blocks if set as environment variables
…733 -> NERSC with a test.txt file), and scheduling it to run every 12 hours
…l733) as a prefix as per Garrett's suggestion
…ove task, and updating prefect.yaml to reflect that
… prefect visibility. Also, adding RuntimeError to the move_733_flight_check to ensure the flow fails if the transfer fails
… for 733, and a transfer call in move.py. Updating pytest to reflect the additional transfer. Still waiting for credentials on Lamarr to verify the movement there.
3 participants
Creating this PR to start work on extending our Prefect workflows to beamline 7.3.3 once the Globus endpoint is set up as described here #57. I anticipate transferring data from 733 to NERSC as an initial step.