introduce execution graph

[luxass]

The pipeline builds an execution graph during processing that captures the relationships between sources, files, routes, artifacts, and outputs. This graph can be used for visualization, debugging, or understanding data flow.

Accessing the Graph

The execution graph is returned as part of the pipeline result.

import { definePipeline } from "@ucdjs/pipelines";

const pipeline = definePipeline({
  versions: ["16.0.0"],
  source: mySource,
  routes: [myRoute],
});

const result = await pipeline.run();

console.log(result.graph.nodes); // All nodes in the graph
console.log(result.graph.edges); // All edges connecting nodes

Graph Structure

The graph consists of nodes and edges. Each node represents an entity in the pipeline, and edges represent relationships between them.

interface PipelineGraph {
  nodes: PipelineGraphNode[];
  edges: PipelineGraphEdge[];
}

interface PipelineGraphNode {
  id: string;
  type: "source" | "file" | "route" | "artifact" | "output";
  // Additional properties based on type
}

interface PipelineGraphEdge {
  from: string;
  to: string;
  type: "provides" | "matched" | "resolved" | "depends";
}

Example Graph

For a pipeline processing two files with one route, the graph might look like:

// Nodes
{ id: "source:16.0.0", type: "source", version: "16.0.0" }
{ id: "file:16.0.0:LineBreak.txt", type: "file", file: { name: "LineBreak.txt", ... } }
{ id: "file:16.0.0:Scripts.txt", type: "file", file: { name: "Scripts.txt", ... } }
{ id: "route:16.0.0:line-break", type: "route", routeId: "line-break" }
{ id: "output:16.0.0:0", type: "output", outputIndex: 0, property: "Line_Break" }

// Edges
{ from: "source:16.0.0", to: "file:16.0.0:LineBreak.txt", type: "provides" }
{ from: "file:16.0.0:LineBreak.txt", to: "route:16.0.0:line-break", type: "matched" }
{ from: "route:16.0.0:line-break", to: "output:16.0.0:0", type: "resolved" }

Use Cases

The graph enables several workflows like generating visual diagrams of pipeline execution, identifying which routes processed which files, tracking artifact dependencies between routes, and debugging why certain files were or weren't processed.