Tesser RFC-005: Unified State Reconciliation Engine

[EwigMidori]

Tesser RFC-005: Unified State Reconciliation Engine

Metadata	Detail
Status	Done in #90 (Targeting v0.9.3)
Type	Core Feature / Reliability
Dependencies	tesser-portfolio (Portfolio State), Exchange Client Interface
Objective	Create a single, robust mechanism to detect and resolve discrepancies between local state and exchange state, both at startup and during runtime.

1. Motivation

1.1 The State Drift Problem

A trading system's local state (positions, balances, open orders) can easily diverge from the exchange's state due to process crashes, network partitions, or exchange-side events (e.g., funding payments, order expirations). Currently, Tesser has a basic spawn_reconciliation_loop for runtime checks and separate, yet-to-be-defined logic for startup recovery.

1.2 The Risk of Dual Systems

Maintaining two separate reconciliation systems—one for startup and one for runtime—is a significant architectural risk. It violates the DRY (Don't Repeat Yourself) principle and inevitably leads to subtle behavioral inconsistencies. For example, the definition of a "zombie" order (an order existing on the exchange but not tracked locally) might differ between the two systems, leading to unpredictable behavior and making the system difficult to maintain and reason about.

This RFC proposes a unified architecture to solve state drift reliably and maintainably.

---

2. Proposed Architecture

We will introduce a Unified Reconciliation Engine built on a layered design. The core idea is to separate the detection of state differences from the resolution of those differences.

2.1 Layer 1: The Core StateDiffer

This will be a pure, stateless component responsible for one task: comparing a snapshot of the local LiveState with a snapshot of the remote exchange state.

• Input: Local state (orders, positions, balances) and Remote state.
• Output: A structured ReconciliationReport. This report does not perform any actions; it simply categorizes all discrepancies found.

The ReconciliationReport will contain categorized lists, such as:

• Order Diffs:
  • Matched Orders: Exist in both states (but may have different filled quantities).
  • Ghost Orders: Exist locally but are missing on the exchange (implying they were filled or canceled during a disconnect).
  • Zombie Orders: Exist on the exchange but are not tracked locally.
• Position & Balance Diffs: Lists any discrepancies in asset quantities.

This component becomes the single source of truth for identifying state drift.

2.2 Layer 2: Contextual Handlers

These components consume the ReconciliationReport and execute state-modifying actions. The key is that we will have different handlers for different operational contexts.

A. The StartupHandler

• Context: Runs once, blocking the engine from starting until complete.
• Behavior: Aggressive and authoritative. It prioritizes establishing a perfectly synchronized state baseline as quickly as possible.
• Actions:
  • For position/balance diffs: Hard Resets the local portfolio using the exchange's state as the absolute truth.
  • For ghost orders: Logs them as resolved and discards them, since the portfolio state has already been corrected.
  • For zombie orders: Executes a pre-configured, decisive policy (e.g., Cancel All).

B. The RuntimeHandler

• Context: Runs periodically in the background (refactoring the existing spawn_reconciliation_loop).
• Behavior: Fine-grained and event-driven. It aims to correct minor drifts without disrupting the running strategy.
• Actions:
  • For position/balance diffs: This is a critical error at runtime. It should trigger an alert and potentially engage a global liquidate_only mode, but it must not perform a hard reset.
  • For ghost orders: This is the most critical runtime task. The handler must query the exchange for the order's historical fills, generate Fill events, and dispatch them to the portfolio to ensure accounting remains accurate.
  • For zombie orders: Dispatches a command to cancel the order.

---

3. Implementation Plan

3.1 Phase 1: Core Differ & Runtime Refactor

• Implement the stateless StateDiffer and the ReconciliationReport data structures.
• Refactor the existing spawn_reconciliation_loop to use the new StateDiffer and evolve its logic into the RuntimeHandler.
• Crucially, implement the logic for the RuntimeHandler to fetch historical fills for Ghost Orders and translate them into internal events.

3.2 Phase 2: Startup Handler & Integration

• Implement the StartupHandler with its "hard reset" logic.
• Enhance Portfolio::from_exchange_state to allow the preservation of key metrics (like peak_equity for drawdown calculations) during a reset.
• Integrate the StartupHandler into the engine's boot sequence, ensuring it runs after loading from disk but before the strategy logic begins.

3.3 Phase 3: Configuration & Policies

• Expose configuration options for reconciliation behavior, such as the policy for handling Zombie Orders (Cancel, Adopt, or Ignore).
• Add robust logging throughout the reconciliation process to ensure every action taken is auditable.

---

4. Future Benefits

1. Reliability: By unifying the logic, we eliminate a major source of potential bugs and ensure consistent behavior, whether recovering from a 5-second network blip or a 5-hour outage.
2. Maintainability: Developers only need to modify the StateDiffer to change how discrepancies are detected, making the system easier to upgrade and debug.
3. Atomicity: The startup handler ensures the system begins from a known-good, consistent state, preventing strategies from acting on stale or incorrect data. This is fundamental to Tesser's "Reliability First" philosophy.