Skip to content

Latest commit

 

History

History
802 lines (648 loc) · 22.5 KB

architecture.md

File metadata and controls

802 lines (648 loc) · 22.5 KB

🏗️ Reframe v3.1.1: Technical Architecture & Design

Table of Contents

Overview

Reframe v3.1.1 is an enterprise-grade bidirectional SWIFT MT ↔ ISO 20022 transformation engine built in Rust. This release introduces a unified package-based architecture with configuration-driven design, providing financial institutions with a transparent, high-performance, and fully auditable message transformation solution.

What's New in v3.1.1

  • Package-Based Architecture: External workflow packages with reframe-package.json configuration
  • Configuration-Driven: Runtime configuration via reframe.config.json with environment variable overrides
  • Unified Transformation Engine: Single bidirectional engine handling both MT→MX and MX→MT transformations
  • GraphQL API: Query transformation history with custom fields support
  • Custom Fields: Package-specific computed fields using JSONLogic expressions
  • Database Integration: MongoDB persistence with audit trail
  • Metadata Support: User-provided metadata for workflow customization
  • Hot-Reload Capability: Update workflows without service restart
  • Simplified API: Unified endpoints with automatic message detection
  • SR2025 Compliance: Full implementation of SWIFT's November 2025 standards release

Core Design Principles

🔍 Complete Transparency

  • All transformation logic externalized in human-readable JSON
  • Package-based workflow organization
  • Full audit trail for compliance and debugging
  • No proprietary black boxes or hidden logic

⚡ High Performance

  • Sub-millisecond transformation latency
  • Rust-powered for memory safety and speed
  • Async runtime with Tokio for concurrent processing
  • Zero-copy parsing where possible

🔧 Configuration-Driven Architecture

  • External package-based workflow system
  • Runtime configuration via reframe.config.json
  • Environment variable overrides for deployment flexibility
  • Hot-reloadable workflow configurations

📊 Workflow-Driven

  • Declarative transformation pipelines
  • JSONLogic-based business rules
  • Version-controlled configuration
  • Visual workflow representation

🔄 True Bidirectional Design

  • Unified engine for both directions
  • Automatic message format detection
  • Symmetric transformation capabilities
  • Consistent handling of edge cases

🏢 Enterprise-Ready

  • Production-grade error handling
  • Comprehensive monitoring and metrics
  • Container-native deployment
  • Horizontal scaling support

System Architecture

High-Level Architecture Diagram

graph TB
    subgraph "External Systems"
        CLIENT[REST API Clients]
        WEB[Web Applications]
        CLI[CLI Tools]
    end

    subgraph "API Gateway Layer"
        HTTP[Axum HTTP Server<br/>REST API v3.1.1]
        AUTH[Authentication]
    end

    subgraph "Processing Core"
        ROUTER[Intelligent Message Router<br/>Auto Format Detection]

        subgraph "Unified Engines"
            TRANSFORM_ENGINE[Transform Engine<br/>Bidirectional MT ↔ MX]
            GENERATION_ENGINE[Generation Engine<br/>Sample Messages]
            VALIDATION_ENGINE[Validation Engine<br/>Message Compliance]
        end
    end

    subgraph "Message Processing"
        subgraph "Parsers"
            PARSE_MT[MT Parser<br/>SWIFT MT Format]
            PARSE_MX[MX Parser<br/>ISO 20022 XML]
        end

        subgraph "Publishers"
            PUB_MT[MT Publisher<br/>Generate MT]
            PUB_MX[MX Publisher<br/>Generate XML]
        end

        subgraph "Generators"
            SAMPLE_GEN[Sample Generator<br/>Test Data Creation]
            BAH_GEN[BAH Generator<br/>Business Headers]
        end
    end

    subgraph "Workflow Engine"
        DATAFLOW[Dataflow-rs<br/>Pipeline Orchestration]
        DATALOGIC[Datalogic-rs<br/>Business Logic]
        VALIDATOR[SR2025 Validator<br/>Compliance Rules]
    end

    subgraph "Package Management"
        PKG_MGR[Package Manager<br/>Discovery & Loading]

        subgraph "External Package"
            PKG_CONFIG[reframe-package.json<br/>Package Configuration]
            WF_TRANSFORM[Transform Workflows<br/>Bidirectional Rules]
            WF_GENERATE[Generate Workflows<br/>Sample Creation]
            WF_VALIDATE[Validate Workflows<br/>Compliance Checks]
            SCENARIOS[Test Scenarios<br/>Sample Data]
        end
    end

    subgraph "Configuration"
        CONFIG[reframe.config.json<br/>Runtime Configuration]
        ENV_VARS[Environment Variables<br/>Deployment Overrides]
    end

    subgraph "Observability"
        LOGS[Structured Logging<br/>tracing/env_logger]
        METRICS[Prometheus Metrics]
        HEALTH[Health Checks]
    end

    %% Connections
    CLIENT --> HTTP
    WEB --> HTTP
    CLI --> HTTP

    HTTP --> AUTH
    AUTH --> ROUTER

    ROUTER -->|Auto-detect format| TRANSFORM_ENGINE
    ROUTER -->|Generate request| GENERATION_ENGINE
    ROUTER -->|Validate request| VALIDATION_ENGINE

    TRANSFORM_ENGINE --> PARSE_MT
    TRANSFORM_ENGINE --> PARSE_MX
    TRANSFORM_ENGINE --> DATAFLOW
    TRANSFORM_ENGINE --> PUB_MT
    TRANSFORM_ENGINE --> PUB_MX

    GENERATION_ENGINE --> SAMPLE_GEN
    GENERATION_ENGINE --> DATAFLOW

    VALIDATION_ENGINE --> PARSE_MT
    VALIDATION_ENGINE --> PARSE_MX
    VALIDATION_ENGINE --> DATAFLOW

    DATAFLOW --> DATALOGIC
    DATAFLOW --> VALIDATOR
    DATAFLOW --> WF_TRANSFORM
    DATAFLOW --> WF_GENERATE
    DATAFLOW --> WF_VALIDATE

    SAMPLE_GEN --> SCENARIOS

    TRANSFORM_ENGINE --> PKG_MGR
    GENERATION_ENGINE --> PKG_MGR
    VALIDATION_ENGINE --> PKG_MGR

    PKG_MGR --> PKG_CONFIG
    PKG_MGR --> CONFIG
    CONFIG --> ENV_VARS

    ROUTER --> LOGS
    DATAFLOW --> LOGS
    HTTP --> METRICS
    HTTP --> HEALTH
Loading

Core Components

1. Main Server (src/main.rs)

  • Initializes Axum HTTP server
  • Sets up unified route handlers
  • Manages engine lifecycle
  • Loads configuration from reframe.config.json

2. API Handlers (src/handlers.rs)

  • Unified Transform Handler: Auto-detects message format and direction
  • Generate Handler: Creates sample messages for testing
  • Validate Handler: Validates MT and MX messages
  • Admin Handlers: Workflow reload and package management
  • Request validation with metadata support
  • Response formatting and error handling

3. Package Manager (src/package_manager.rs)

  • Package discovery and loading from reframe.config.json
  • Package validation and version compatibility
  • Configuration management with environment variable overrides
  • Runtime package information and metadata

4. Transformation Engines (src/engine.rs)

  • Transform Engine: Unified bidirectional MT ↔ ISO 20022 transformations
  • Generation Engine: Sample message creation for both MT and MX
  • Validation Engine: Message compliance checking for both formats
  • Workflow orchestration using dataflow-rs
  • Hot-reload capability without service restart
  • Package-based workflow loading

5. Message Parsers

  • MT Parser (src/parse_mt.rs): Custom SWIFT MT parser with field validation
  • MX Parser (src/parse_mx.rs): ISO 20022 XML parsing with schema validation
  • Validation Helpers (src/validation_helpers.rs): Common validation utilities

6. Message Generators

  • MX Generator (src/mx_generator.rs): JSON to ISO 20022 XML conversion
  • MT Generator (src/mt_generator.rs): Structured data to SWIFT MT format
  • Sample Generator (src/sample_generator.rs): Test data creation using datafake-rs

7. Message Publishers

  • MT Publisher (src/publish_mt.rs): Format and publish MT messages
  • MX Publisher (src/publish_mx.rs): Format and publish MX messages with proper namespaces

8. Custom Fields Module (src/custom_fields.rs)

  • Package-specific computed fields using JSONLogic expressions
  • Three storage strategies: precompute, runtime, and hybrid
  • Field evaluation at storage and query time
  • Error handling with graceful degradation

9. Database Integration (src/database/)

  • MongoDB Client (mongodb.rs): Audit trail persistence with custom field support
  • Conversion Utilities (conversion.rs): Message to/from database document conversion
  • Custom field filtering and querying
  • Transaction history and analytics

10. GraphQL API (src/graphql/)

  • Schema (schema.rs): GraphQL query and mutation definitions
  • Types (types.rs): GraphQL types with custom fields support
  • Message querying with filters and pagination
  • Package-specific custom field resolvers

11. Scenario Management (src/scenario_loader.rs)

  • Load test scenarios from package
  • Generate realistic test data
  • Support for multiple message variants

12. OpenAPI Documentation (src/openapi.rs)

  • Swagger/OpenAPI spec generation with runtime configuration
  • Interactive API documentation
  • Request/response examples
  • Configurable server URLs and metadata

10. Type Definitions (src/types.rs)

  • Common data structures and types
  • Request/response models with metadata support
  • AppState with three unified engines
  • Error types for structured error handling

Package Management

Package Structure

External packages follow this standardized structure:

reframe-package-swift-cbpr/
├── reframe-package.json    # Package configuration
├── transform/
│   ├── index.json          # Unified workflow index
│   ├── outgoing/           # MT → MX transformations
│   │   ├── parse-mt.json
│   │   ├── MT103/
│   │   │   ├── bah-mapping.json
│   │   │   ├── document-mapping.json
│   │   │   ├── precondition.json
│   │   │   └── postcondition.json
│   │   └── combine-xml.json
│   └── incoming/           # MX → MT transformations
│       ├── parse-mx.json
│       └── pacs008/
│           ├── 01-variant-detection.json
│           ├── 02-preconditions.json
│           └── ...
├── generate/
│   ├── index.json
│   ├── generate-mt.json
│   └── generate-mx.json
├── validate/
│   ├── index.json
│   ├── validate-mt.json
│   └── validate-mx.json
└── scenarios/
    ├── index.json
    ├── outgoing/
    │   └── mt103_to_pacs008_cbpr_standard.json
    └── incoming/
        └── pacs008_to_mt103_cbpr_standard.json

Package Configuration

reframe-package.json:

{
  "id": "swift-cbpr-mt-mx",
  "name": "SWIFT CBPR+ MT <-> MX Transformations - SR2025",
  "version": "2.0.0",
  "description": "Official SWIFT CBPR+ transformations",
  "author": "Plasmatic Team",
  "license": "Apache-2.0",
  "engine_version": "3.1.1",
  "required_plugins": ["swift-mt-message", "mx-message"],
  "workflows": {
    "transform": {
      "path": "transform",
      "description": "Unified bidirectional transformation workflows",
      "entry_point": "index.json"
    },
    "generate": {
      "path": "generate",
      "description": "Sample message generation",
      "entry_point": "index.json"
    },
    "validate": {
      "path": "validate",
      "description": "Message validation workflows",
      "entry_point": "index.json"
    }
  },
  "scenarios": {
    "path": "scenarios",
    "description": "Test scenarios",
    "entry_point": "index.json"
  }
}

Configuration System

Runtime Configuration

reframe.config.json (project root):

{
  "packages": [
    {
      "path": "../reframe-package-swift-cbpr",
      "enabled": true
    }
  ],
  "server": {
    "host": "0.0.0.0",
    "port": 3000,
    "runtime": {
      "tokio_worker_threads": "auto",
      "thread_name_prefix": "reframe-worker"
    }
  },
  "logging": {
    "format": "compact",
    "level": "info",
    "show_target": false,
    "show_thread": false,
    "show_file_location": false,
    "show_span_events": false,
    "file_output": {
      "enabled": false,
      "directory": "./logs",
      "file_prefix": "reframe",
      "rotation": "daily"
    }
  },
  "api_docs": {
    "enabled": true,
    "title": "Reframe API",
    "version": "3.1.1",
    "description": "Enterprise-grade bidirectional SWIFT MT ↔ ISO 20022 transformation service",
    "contact": {
      "name": "Plasmatic Team",
      "email": "enquires@goplasmatic.io"
    },
    "license": {
      "name": "Apache 2.0",
      "identifier": "Apache-2.0",
      "url": "https://opensource.org/license/apache-2-0"
    },
    "external_docs_url": "https://sandbox.goplasmatic.io",
    "server_url": "http://localhost:3000"
  },
  "defaults": {
    "package_id": null,
    "package_path": "../reframe-package-swift-cbpr"
  }
}

Configuration Priority System

Configuration values are resolved in this order (highest to lowest priority):

  1. Environment Variables (highest priority)
  2. reframe.config.json file
  3. Built-in Defaults (lowest priority)

Environment Variables

Variable Description Default
REFRAME_PACKAGE_PATH Path to workflow package ../reframe-package-swift-cbpr
TOKIO_WORKER_THREADS Async runtime worker threads CPU count
API_SERVER_URL OpenAPI server URL override http://localhost:3000
RUST_LOG Logging level info

Core Libraries

Dataflow-rs (Workflow Engine)

  • Purpose: JSON-based workflow orchestration
  • Features:
    • Declarative pipeline definitions
    • Conditional routing
    • Data transformation operators
    • Error handling and recovery

Datalogic-rs (Logic Engine)

  • Purpose: JSONLogic implementation for business rules
  • Features:
    • Complex conditional evaluation
    • Variable substitution
    • Mathematical operations
    • String manipulation

Datafake-rs (Data Generation)

  • Purpose: Realistic test data generation
  • Features:
    • Faker.js compatible API
    • Custom generators
    • Locale support
    • Reproducible data with seeds

Swift-MT-Message (MT Library)

  • Purpose: SWIFT MT message handling
  • Version: 3.0+
  • Features:
    • All MT message types
    • Field validation
    • Block structure parsing
    • SR2025 compliance

MX-Message (ISO 20022 Library)

  • Purpose: ISO 20022 message handling
  • Version: 3.0+
  • Features:
    • Complete message catalog
    • XML serialization/deserialization
    • Schema validation
    • BAH v3 support

Data Flow & Processing

Unified Transformation Flow

sequenceDiagram
    participant Client
    participant Router
    participant Transform Engine
    participant Parser (MT/MX)
    participant Workflow Engine
    participant Publisher (MX/MT)

    Client->>Router: POST /api/transform
    Note over Router: Auto-detect format<br/>from message content
    Router->>Transform Engine: Route message
    Transform Engine->>Parser (MT/MX): Parse message
    Parser (MT/MX)-->>Transform Engine: Structured data
    Note over Transform Engine: Set metadata<br/>including user metadata
    Transform Engine->>Workflow Engine: Execute transformation
    Note over Workflow Engine: Load appropriate workflows<br/>from package (outgoing/incoming)
    Workflow Engine-->>Transform Engine: Transformed data
    Transform Engine->>Publisher (MX/MT): Generate output
    Publisher (MX/MT)-->>Client: Transformed message
Loading

Metadata Flow

User-provided metadata is merged into the workflow execution context:

{
  "message": "<message content>",
  "metadata": {
    "priority": "high",
    "custom_field": "value"
  }
}

The workflow can access these values:

{
  "logic": {
    "if": [
      {"==": [{"var": "metadata.priority"}, "high"]},
      "URGENT",
      "NORMAL"
    ]
  }
}

SR2025 Compliance

Key SR2025 Updates Implemented

  1. Business Application Header v3

    • Enhanced party identification
    • Improved service level codes
    • Extended priority options

  2. Structured Remittance Information

    • Creditor reference information
    • Structured reference types
    • Document adjustment details

  3. Enhanced Data Quality

    • Mandatory UETR (Unique End-to-end Transaction Reference)
    • LEI (Legal Entity Identifier) support
    • Improved address structures

  4. New Message Types

    • camt.105 - Billing report
    • camt.106 - Investigation response
    • camt.107 - Non-deliverable information
    • Additional pain and pacs variants

  5. Validation Enhancements

    • Cross-field validation rules
    • Business rule enforcement
    • Format validation improvements

Technology Stack

Core Technologies

Component Technology Version Purpose
Language Rust 1.75+ Core implementation
Async Runtime Tokio 1.47 Asynchronous I/O
Web Framework Axum 0.8 HTTP server
XML Processing quick-xml 0.38 Fast XML parsing
JSON Processing serde_json 1.0 JSON serialization
API Documentation utoipa 5.4 OpenAPI generation
Logging tracing/env_logger Latest Structured logging
Configuration arc-swap Latest Lock-free config updates

Performance Architecture

Design Optimizations

  1. Async Processing

    • Tokio runtime for non-blocking I/O
    • Concurrent request handling
    • Efficient resource utilization
    • Automatic CPU core scaling

  2. Engine Caching

    • Pre-loaded workflow engines
    • Arc-swap for zero-downtime reloads
    • Shared engine state across threads

  3. Zero-Copy Parsing

    • Direct memory access for string operations
    • Minimal allocations during parsing
    • Reuse of buffers where possible

  4. Configuration System

    • Lock-free configuration reads with ArcSwap
    • Hot-reload without service interruption
    • Package-based workflow loading

Tokio Runtime Configuration

The async runtime automatically scales with available CPU cores:

# Auto-detection (default, uses all CPU cores)
cargo run --release

# Explicit configuration
TOKIO_WORKER_THREADS=4 cargo run --release

# Low-resource environment
TOKIO_WORKER_THREADS=1 cargo run

Deployment Architecture

Container Architecture

# Multi-stage build
FROM rust:1.75 as builder
# Build optimized binary

FROM debian:bookworm-slim
# Minimal runtime with only required libraries
# Non-root user execution
# Health check endpoint

Deployment Options

  1. Docker Standalone

    • Single container deployment
    • Volume mounts for packages and config
    • Environment-based configuration

  2. Docker Compose

    • Multi-container orchestration
    • Service dependencies
    • Network isolation

  3. Kubernetes

    • Horizontal pod autoscaling
    • ConfigMaps for configuration and workflows
    • Ingress controllers
    • Service mesh integration

  4. Cloud Native

    • AWS ECS/Fargate
    • Azure Container Instances
    • Google Cloud Run
    • OpenShift

Scaling Strategy

  • Horizontal Scaling: Stateless design allows linear scaling
  • Load Balancing: Round-robin or least-connections
  • Auto-scaling: Based on CPU/memory or request rate
  • Geographic Distribution: Deploy close to users

Monitoring & Observability

Health Checks

GET /health
{
  "status": "healthy",
  "timestamp": "2025-10-14T10:00:00Z",
  "engines": {
    "transform": "ready",
    "generation": "ready",
    "validation": "ready"
  },
  "capabilities": [
    "bidirectional_transformation",
    "sample_generation",
    "message_validation",
    "workflow_reload"
  ],
  "config": {
    "thread_count": 8
  }
}

Logging Strategy

// Structured logging example
info!(
    message_type = "MT103",
    direction = "outgoing",
    duration_ms = 5,
    "Transformation completed successfully"
);

Metrics Collection

Key metrics to monitor:

  • Request throughput (requests/second)
  • Transformation latency (milliseconds)
  • Error rates by message type
  • Engine workflow execution time
  • Memory usage and GC pressure

API Endpoints

Unified Transformation API

  • POST /api/transform: Unified bidirectional transformation
    • Auto-detects message format (MT or MX)
    • Supports optional message_type_hint for routing
    • Handles both MT → MX and MX → MT
    • Accepts user metadata for workflow customization

Sample Generation

  • POST /api/generate: Generate sample messages
    • Supports both MT and MX message types
    • Uses scenarios from package

Validation

  • POST /api/validate: Validate messages
    • Auto-detects and validates MT or MX
    • Returns validation status and errors

Administration

  • POST /admin/reload-workflows: Hot reload workflows from package
  • GET /api/packages: List loaded packages with details
  • GET /health: Health check with engine status
  • GET /swagger-ui: Interactive API documentation
  • GET /api-docs/openapi.json: OpenAPI specification

Future Roadmap

Near Term

  • Kafka/RabbitMQ integration
  • Enhanced monitoring dashboard
  • Multi-package support (beyond single default package)
  • Real-time WebSocket notifications

Medium Term

  • Plugin system for custom functions
  • Multi-tenant support
  • Cloud-native SaaS offering
  • Performance optimization tools

Long Term

  • Real-time streaming transformations
  • Machine learning for mapping suggestions
  • Predictive error detection
  • Cross-border payment optimization

Conclusion

Reframe v3.1.1 represents a significant advancement with its unified package-based architecture. The configuration-driven design with external packages provides:

  • Scalability: From single instances to global deployments
  • Maintainability: Clear separation of concerns and modular design
  • Extensibility: Easy to add new message types via packages
  • Observable: Comprehensive monitoring and debugging capabilities
  • Secure: Multiple layers of security and compliance features
  • Flexible: Runtime configuration without recompilation

For more information, see our other documentation: