7.0.1.1 Flows

config.yml

@@ -1,5 +1,42 @@
 globus:
   globus_endpoints:
+
+    # 7.0.1.1 ENDPOINTS
+
+    bl7011-als-cosmic-scattering:
+      root_path: /data
+      uri: bl7011-als-cosmic-scattering.lbl.gov
+      uuid: e19da22e-4af5-4901-acae-d7e702def054
+      name: bl7011-als-cosmic-scattering
+
+    bl7011-nersc_alsdev:
+      root_path: /global/cfs/cdirs/als/data_mover/7.0.1.1
+      uri: nersc.gov
+      uuid: d40248e6-d874-4f7b-badd-2c06c16f1a58
+      name: bl7011-nersc_alsdev
+
+    bl7011-compute-dtn:
+      root_path: /
+      uri: compute-dtn.als.lbl.gov
+      uuid: 5ae9485d-87a4-4349-82ac-e214963ed823
+      name: bl7011-compute-dtn
+
+    # 7.0.1.2 ENDPOINTS
+
+    nersc7012:
+      root_path: /global/cfs/cdirs/als/gsharing/data_mover/7012
+      uri: nersc.gov
+      uuid: d40248e6-d874-4f7b-badd-2c06c16f1a58
+      name: nersc7012
+
+    data7012:
+      root_path: /
+      uri: hpc.lbl.gov
+      uuid: 741b96e1-1b98-42a8-918d-daacc24c145f
+      name: data7012
+
+    # 8.3.2 ENDPOINTS
+
     spot832:
       root_path: /
       uri: spot832.lbl.gov
@@ -90,18 +127,6 @@ globus:
       uuid: df82346e-9a15-11ea-b3c4-0ae144191ee3
       name: nersc832
 
-    nersc7012:
-      root_path: /global/cfs/cdirs/als/gsharing/data_mover/7012
-      uri: nersc.gov
-      uuid: d40248e6-d874-4f7b-badd-2c06c16f1a58
-      name: nersc7012
-
-    data7012:
-      root_path: /
-      uri: hpc.lbl.gov
-      uuid: 741b96e1-1b98-42a8-918d-daacc24c145f
-      name: data7012
-
   globus_apps:
     als_transfer:
       client_id: ${GLOBUS_CLIENT_ID}

docs/mkdocs/docs/bl7011.md

@@ -0,0 +1,127 @@
+# Beamline 7.0.1.1 Flows
+
+This page documents the workflows supported by Splash Flows at [ALS Beamline 7.0.1.1 (COSMIC Scattering)](https://als.lbl.gov/beamlines/7-0-1-1/).
+
+## Data at 7.0.1.1
+
+At Beamline 7.0.1.1, users generate data in an HDF5 format containing a background subtracted stack of 2D images with associated Labview metadata. Depending on the experiment, the file sizes can be greater than 100GB. A ROI is exported for each dataset.
+
+## File Watcher
+
+There is a file watcher on the system `QNAP` that listens for new scans that have finished writing to disk. From there, a Prefect Flow we call `dispatcher` kicks off the downstream steps:
+- Copy scans in real time from a Globus collection on the `compute-dtn` server to `NERSC CFS` using Globus Transfer.
+- Copy project data to `NERSC HPSS` for long-term storage (TBD).
+- Analysis on HPC systems (TBD).
+- Ingest into SciCat (TBD).
+- Schedule data pruning from `QNAP` and `NERSC CFS`.
+
+## Prefect Configuration
+
+### Registered Flows
+
+#### `dispatcher.py`
+
+The Dispatcher Prefect Flow manages the logic for handling the order and execution of data tasks. Once a new file is written, the `dispatcher()` Flow is called. In this case, the dispatcher handles the synchronous call to `move.py`, with a potential to add additional steps (e.g. scheduling remote HPC analysis code).
+
+#### `move.py`
+
+Flow to process a new file at BL 7.0.1.1
+1. Copy the file from `compute-dtn` to `NERSC CFS` and ingest the file path and metadata into SciCat.
+2. Schedule pruning from `QNAP`.
+3. Copy the file from `NERSC CFS` to `NERSC HPSS`. Ingest the archived file path in SciCat.
+4. Schedule pruning from `NERSC CFS`.
+
+## VM Details
+
+The computing backend runs on a VM in the B15 server room that is managed by ALS IT staff.
+
+**Name**: `flow-xpcs`
+**OS**: `Ubuntu 20.02 LTS` ... **at some point should be updated to `Ubuntu 24.04 LTS`**
+
+## Flow Diagram
+
+## Flow Diagram
+```mermaid
+sequenceDiagram
+    participant DET as Detector/<br/>File Watcher
+    participant DISP as Prefect<br/>Dispatcher
+    participant D7011 as bl7011 compute-dtn<br/>Storage
+    participant GLOB as Globus<br/>Transfer
+    participant CFS as NERSC<br/>CFS
+    participant CAT as SciCat<br/>Metadata
+    participant SFAPI as SFAPI
+    participant HPC as HPC<br/>Compute
+    participant HPSS as HPSS<br/>Tape
+
+    %% Initial Trigger
+    DET->>DET: Monitor filesystem
+    DET->>DISP: Trigger on new file
+    DISP->>DISP: Coordinate flows
+
+    %% Flow 1: new_file_7011
+    rect rgb(220, 230, 255)
+        note over DISP,CAT: FLOW 1: new_file_7011
+        DISP->>GLOB: Init transfer
+        activate GLOB
+        GLOB->>D7011: Initiate copy
+        activate D7011
+        D7011-->>GLOB: Copy initiated
+        deactivate D7011
+        %% note right of GLOB: Transfer in progress
+        GLOB-->>DISP: Transfer complete
+        deactivate GLOB
+
+        DISP->>CAT: Register metadata
+    end
+
+    %% Flow 2: HPSS Transfer
+    rect rgb(220, 255, 230)
+        note over DISP,CAT: FLOW 2: Scheduled HPSS Transfer
+        DISP->>SFAPI: Submit tape job
+        activate SFAPI
+        SFAPI->>HPSS: Initiate archive
+        activate HPSS
+        HPSS-->>SFAPI: Archive complete
+        deactivate HPSS
+        SFAPI-->>DISP: Job complete
+        deactivate SFAPI
+
+        DISP->>CAT: Update metadata
+    end
+
+    %% Flow 3: HPC Analysis
+    rect rgb(255, 230, 230)
+        note over DISP,HPC: FLOW 3: HPC Downstream Analysis
+        DISP->>SFAPI: Submit compute job
+        activate SFAPI
+        SFAPI->>HPC: Execute job
+        activate HPC
+        HPC->>HPC: Process data
+        HPC-->>SFAPI: Compute complete
+        deactivate HPC
+        SFAPI-->>DISP: Job complete
+        deactivate SFAPI
+
+        DISP->>CAT: Update metadata
+    end
+
+    %% Flow 4: Scheduled Pruning
+    rect rgb(255, 255, 220)
+        note over DISP,CAT: FLOW 4: Scheduled Pruning
+        DISP->>DISP: Scheduled pruning trigger
+
+        DISP->>D7011: Prune old files
+        activate D7011
+        D7011->>D7011: Delete expired data
+        D7011-->>DISP: Pruning complete
+        deactivate D7011
+
+        DISP->>CFS: Prune old files
+        activate CFS
+        CFS->>CFS: Delete expired data
+        CFS-->>DISP: Pruning complete
+        deactivate CFS
+
+        DISP->>CAT: Update metadata
+    end
+  ```

docs/mkdocs/mkdocs.yml

@@ -13,8 +13,11 @@ nav:
 - Home: index.md
 - Installation and Requirements: install.md
 - Getting Started: getting_started.md
-- Compute at ALCF: alcf832.md
-- Compute at NERSC: nersc832.md
+- Beamline Implementations:
+  - Beamline 7.0.1.1 - COSMIC Scattering: bl7011.md
+  - Beamline 8.3.2 - Microtomography:
+    - Compute at ALCF: alcf832.md
+    - Compute at NERSC: nersc832.md
 - Orchestration: orchestration.md
 - Configuration: configuration.md
 # - Troubleshooting: troubleshooting.md