This project is part of a complete end-to-end trading system:
- Main Repository: fpga-trading-systems
- Project Number: 28 of 30
- Category: C++ Application
- Dependencies: Projects 24 (Order Gateway), 25 (Market Maker), 26 (Order Execution)
Platform: Linux Technology: C++20, Process Management, POSIX Shared Memory, Prometheus Status: Completed
Project 28 is the system orchestrator that integrates Projects 24 (Order Gateway with PCIe + XGBoost), 25 (Market Maker FSM), and 26 (Order Execution Engine) into a unified, production-ready FPGA trading system. This is the portfolio centerpiece demonstrating end-to-end low-latency trading from PCIe data ingestion through GPU inference to order execution and position management.
- Unified System Management: Single command to start/stop entire trading system
- Process Orchestration: Manages lifecycle of all components in correct dependency order
- Health Monitoring: Real-time health checks via process and Prometheus endpoints
- Graceful Shutdown: Proper cleanup of processes, shared memory, and resources
- Automatic Dependency Resolution: Ensures components start only when dependencies are ready
- Shared Memory Management: Creates and cleans up Disruptor ring buffers
┌────────────────────────────────────────────────────────────────────┐
│ PROJECT 28: SYSTEM ORCHESTRATOR │
│ • Process spawning and monitoring │
│ • Health checks (process, Prometheus) │
│ • Shared memory lifecycle management │
│ • Graceful shutdown coordination │
└───────────────────────────┬────────────────────────────────────────┘
│
┌───────────────────┼───────────────────┐
│ │ │
▼ ▼ ▼
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ PROJECT 24 │ │ PROJECT 25 │ │ PROJECT 26 │
│ Order Gateway │ │ Market Maker │ │ Order Execution │
│ │ │ FSM │ │ Engine │
│ PCIe Listener │ │ Disruptor │ │ Disruptor │
│ XGBoost GPU │ │ Consumer │ │ Consumer │
│ Disruptor │ │ Strategy │ │ Simulated │
│ Producer │ │ Position │ │ Fills │
│ │ │ Tracker │ │ │
└──────────────────┘ └──────────────────┘ └──────────────────┘
│ │ │
└───────────────────┼───────────────────┘
│
Shared Memory (IPC):
• /dev/shm/bbo_ring_gateway (P24 → P25)
• /dev/shm/order_ring_mm (P25 → P26)
• /dev/shm/fill_ring_oe (P26 → P25)
1. FPGA (Project 23) → PCIe DMA → Project 24 (Order Gateway)
• PCIeListener reads 56-byte BBO packets (with magic header) from /dev/xdma0_c2h_0
• XGBoostPredictor runs GPU inference (84% accuracy, ~10-100 μs)
2. Project 24 → Disruptor (bbo_ring_gateway) → Project 25 (Market Maker)
• Lock-free ring buffer for BBO + prediction data
• ~0.5 μs IPC latency
3. Project 25 → Quote Generation → Position Management
• MarketMakerFSM uses predictions to adjust edge
• Position tracking and risk management
4. Project 25 → Disruptor (order_ring_mm) → Project 26 (Order Execution)
• Lock-free ring buffer for order submission
• ~0.5 μs IPC latency
5. Project 26 → Simulated Fill → Fill Notification
• Configurable latency (default 50 μs)
• Full/partial fill support
6. Project 26 → Disruptor (fill_ring_oe) → Project 25
• Lock-free ring buffer for fill notification
• ~0.5 μs IPC latency
7. Project 25 → Position Update → Next Trading Decision
• Real-time position tracking and PnL calculation
Target End-to-End Latency: ~75-165 μs (FPGA → XGBoost → Order → Fill → Position Update)
- OS: Linux (Ubuntu 22.04+ recommended)
- CPU: Multi-core processor (8+ cores recommended)
- GPU: NVIDIA RTX 5090 (for XGBoost GPU inference)
- Memory: 8 GB RAM minimum
- Disk: 1 GB free space
- PCIe: FPGA with XDMA driver installed
# Ubuntu/Debian
sudo apt update
sudo apt install -y build-essential cmake nlohmann-json3-dev
# CUDA Toolkit (for XGBoost GPU)
# Follow NVIDIA CUDA installation guide
# XDMA Driver
# Build and install from Xilinx dma_ip_driversAll three component projects must be built before starting the orchestrator:
- Project 24 (Order Gateway - PCIe + XGBoost)
- Project 25 (Market Maker FSM)
- Project 26 (Order Execution Engine)
# From fpga-trading-systems root directory
# Build Project 24
cd 24-order-gateway
mkdir -p build && cd build
cmake .. && make -j$(nproc)
cd ../..
# Build Project 25
cd 25-market-maker
mkdir -p build && cd build
cmake .. && make -j$(nproc)
cd ../..
# Build Project 26
cd 26-order-execution
mkdir -p build && cd build
cmake .. && make -j$(nproc)
cd ../..
# Build Project 28 (Orchestrator)
cd 28-complete-system
mkdir -p build && cd build
cmake .. && make -j$(nproc)
cd ../..cd 28-complete-system/build
./trading_system_orchestrator ../config/system_config.jsonThe orchestrator will print component status on startup:
========================================
Trading System Status:
========================================
Order Gateway (PCIe + XGBoost): RUNNING (PID: 12345)
Market Maker FSM: RUNNING (PID: 12346)
Order Execution Engine: RUNNING (PID: 12347)
========================================
Trading system is running.
Press Ctrl+C to stop.
Press Ctrl+C in the orchestrator terminal:
- Orchestrator will send SIGTERM to all components
- Wait for graceful shutdown (10s timeout)
- Cleanup shared memory
{
"system": {
"name": "FPGA Trading System (PCIe + XGBoost)",
"version": "2.0.0",
"log_level": "INFO",
"enable_prometheus": true,
"enable_auto_restart": false,
"healthcheck_interval_ms": 500,
"startup_timeout_seconds": 30,
"shutdown_timeout_seconds": 10
},
"project_24": {
"name": "Order Gateway (PCIe + XGBoost)",
"executable": "../../24-order-gateway/build/order_gateway",
"config_file": "../../24-order-gateway/config.json",
"working_directory": "../../24-order-gateway/build",
"startup_delay_ms": 0,
"healthcheck": {
"type": "process",
"timeout_ms": 1000
}
},
"project_25": {
"name": "Market Maker (Disruptor Consumer)",
"executable": "../../25-market-maker/build/market_maker",
"config_file": "../../25-market-maker/config.json",
"working_directory": "../../25-market-maker/build",
"startup_delay_ms": 2000,
"healthcheck": {
"type": "process",
"timeout_ms": 1000
},
"depends_on": ["project_24"]
},
"project_26": {
"name": "Order Execution (Simulated Fills)",
"executable": "../../26-order-execution/build/order_execution_engine",
"config_file": "../../26-order-execution/config.json",
"working_directory": "../../26-order-execution/build",
"startup_delay_ms": 3000,
"healthcheck": {
"type": "process",
"timeout_ms": 1000
},
"depends_on": ["project_25"]
},
"shared_memory": {
"bbo_ring": {
"path": "/dev/shm/bbo_ring_gateway",
"size_bytes": 262144,
"cleanup_on_start": true,
"cleanup_on_stop": true
},
"order_ring": {
"path": "/dev/shm/order_ring_mm",
"size_bytes": 262144,
"cleanup_on_start": true,
"cleanup_on_stop": true
},
"fill_ring": {
"path": "/dev/shm/fill_ring_oe",
"size_bytes": 262144,
"cleanup_on_start": true,
"cleanup_on_stop": true
}
},
"performance": {
"enable_cpu_pinning": false,
"cpu_affinity": {
"project_24": [2, 3],
"project_25": [4, 5],
"project_26": [6, 7],
"orchestrator": [0]
},
"enable_realtime_scheduling": false
}
}-
Load Configuration
- Parse system_config.json
- Validate component paths and dependencies
-
Cleanup Stale Resources
- Remove old shared memory segments
- Clean up zombie processes
-
Start Components in Order
- Project 24 (Order Gateway)
- Spawn process
- Wait for process to start
- Mark as RUNNING
- Project 25 (Market Maker FSM)
- Wait for startup_delay (2s)
- Verify Project 24 is running
- Spawn process
- Mark as RUNNING
- Project 26 (Order Execution Engine)
- Wait for startup_delay (3s)
- Verify Project 25 is running
- Spawn process
- Mark as RUNNING
- Project 24 (Order Gateway)
-
Start Monitoring Loop
- Health checks every 500ms
- Detect crashed components
- Optional auto-restart
- Receive Shutdown Signal (SIGINT or SIGTERM)
- Stop Monitoring Thread
- Stop Components in Reverse Order
- Project 26 (Order Execution)
- Project 25 (Market Maker)
- Project 24 (Order Gateway)
- For Each Component:
- Send SIGTERM
- Wait up to 10s for graceful exit
- If timeout, send SIGKILL
- Wait for process to exit (waitpid)
- Cleanup Shared Memory
shm_unlink(/dev/shm/bbo_ring_gateway)shm_unlink(/dev/shm/order_ring_mm)shm_unlink(/dev/shm/fill_ring_oe)
- Reap Zombie Processes
- Exit
| Stage | Latency | Notes |
|---|---|---|
| PCIe DMA Read (P24) | ~1-2 μs | XDMA C2H transfer |
| BBO Parse (P24) | ~0.1-0.5 μs | Binary to struct |
| Feature Extract (P24) | ~1-5 μs | BBO to DMatrix |
| XGBoost GPU (P24) | ~10-100 μs | CUDA inference |
| Disruptor P24→P25 | ~0.5 μs | Lock-free IPC |
| Market Maker FSM (P25) | ~1-2 μs | Strategy logic |
| Disruptor P25→P26 | ~0.5 μs | Lock-free IPC |
| Simulated Fill (P26) | ~50 μs | Configurable |
| Disruptor P26→P25 | ~0.5 μs | Lock-free IPC |
| Position Update (P25) | ~0.1 μs | PnL calculation |
| Total End-to-End | ~75-165 μs | FPGA to position update |
28-complete-system/
├── CMakeLists.txt # Build configuration
├── config/
│ └── system_config.json # Unified system configuration
├── include/
│ ├── system_orchestrator.h # Orchestrator class header
│ ├── metrics_aggregator.h # Metrics collection
│ └── prometheus_server.h # Prometheus HTTP server
├── src/
│ ├── main.cpp # Entry point
│ └── system_orchestrator.cpp # Orchestrator implementation
└── README.md # This file
- 23-order-book/ - FPGA order book (PCIe source)
- 24-order-gateway/ - Order Gateway (PCIe + XGBoost)
- 25-market-maker/ - Market Maker FSM
- 26-order-execution/ - Order Execution Engine
FPGA (Project 23)
↓ PCIe Gen2 x4 (/dev/xdma0_c2h_0)
Project 24: Order Gateway
├─ PCIeListener (56-byte BBO packets with magic header)
├─ BBOValidator (filter invalid data)
└─ Disruptor Producer (raw BBO)
↓ Shared Memory (/dev/shm/bbo_ring_gateway)
Project 25: Market Maker
├─ Disruptor Consumer (raw BBO)
├─ XGBoostPredictor (84% accuracy, ~10-100μs)
├─ MarketMakerFSM (strategy logic)
├─ PositionTracker (PnL management)
└─ OrderProducer (orders to P26)
↓ Shared Memory (/dev/shm/order_ring_mm)
Project 26: Order Execution
├─ Order Consumer
├─ Simulated Fill (~50 μs latency)
└─ Fill Producer (fills to P25)
↓ Shared Memory (/dev/shm/fill_ring_oe)
Project 25: (receives fills)
└─ Updates position, PnL
Project 28: System Orchestrator ← YOU ARE HERE
└─ Manages lifecycle of P24, P25, P26
Symptoms: Orchestrator reports "Failed to start" for a component
Diagnosis:
# Check if executable exists
ls -lh 24-order-gateway/build/order_gateway
ls -lh 25-market-maker/build/market_maker
ls -lh 26-order-execution/build/order_execution_engine
# Check if component crashes immediately
./24-order-gateway/build/order_gatewaySolution: Build missing components, fix crashes
Symptoms: Stale /dev/shm/* exists after crash
Solution:
# Manual cleanup
rm -f /dev/shm/bbo_ring_gateway /dev/shm/order_ring_mm /dev/shm/fill_ring_oeSymptoms: Project 24 fails to open /dev/xdma0_c2h_0
Solution:
# Check if XDMA driver loaded
lsmod | grep xdma
# Load driver
sudo modprobe xdma
# Check device
ls -la /dev/xdma0_*This project demonstrates:
- System Integration Skills: Orchestrating multiple C++ processes with complex dependencies
- Process Management: Spawning, monitoring, health checking, graceful shutdown
- IPC Expertise: Shared memory, lock-free ring buffers (Disruptor pattern)
- GPU Integration: XGBoost CUDA inference in production trading pipeline
- Production Readiness: Configuration management, error handling, resource cleanup
- Complete Trading System: Working demonstration of FPGA → GPU → Strategy → Execution loop
Interview Talking Points:
- "Built orchestrator managing 3 C++ microservices with <165μs end-to-end latency"
- "Integrated XGBoost GPU inference (84% accuracy) into real-time trading pipeline"
- "Designed dependency resolution ensuring components start in correct order"
- "Managed shared memory lifecycle for lock-free IPC (Disruptor pattern)"
- "Created production-grade startup/shutdown with proper resource cleanup"
Build Time: ~15 seconds Hardware Status: Tested with full P23→P24→P25→P26 pipeline