[Feature]: Transaction mechanism for atomic multi-subsystem operations

[qin-ctx]

RFC Discussion: #115

Problem Statement

OpenViking's core write operations (rm, mv, add_resource, session.commit) coordinate across multiple subsystems — VikingFS, VectorDB, and QueueManager. Currently there is no atomicity guarantee: if a failure occurs mid-operation (e.g., VectorDB update fails after FS move succeeds), the system is left in an inconsistent state with no way to recover.

For example:

• rm() deletes files from FS, then fails to clean up VectorDB records → orphan index entries
• mv() moves files in FS, then VectorDB URI update fails → search returns stale URIs
• session.commit() archives messages, then LLM call times out → partial commit with no recovery
• Process crash during add_resource → enqueued semantic generation never happens

Proposed Solution

Implement a write-ahead journal + undo log transaction mechanism that wraps multi-subsystem operations with automatic rollback on failure and crash recovery on restart.

Architecture

TransactionContext (async context manager)
  ├── TransactionManager (lifecycle, journal, lock coordination)
  │     ├── TransactionJournal (AGFS-persisted WAL at /local/_system/transactions/)
  │     ├── PathLock (fencing-token-based distributed locks)
  │     └── Crash Recovery (journal replay on startup)
  ├── UndoLog (ordered list of reversible sub-operations)
  └── PostActions (deferred work after commit, e.g. enqueue_semantic)

Key Design Decisions

1. Undo log, not WAL redo: We record how to reverse each completed sub-operation. On rollback, entries are replayed in reverse order. This is simpler than redo logging since our operations are heterogeneous (FS + VectorDB).

2. Fencing tokens for locks: Lock files contain {tx_id}:{monotonic_ns} instead of just transaction ID. This enables stale lock detection and prevents ABA problems during concurrent acquisition.

3. Two-phase session commit: session.commit() is split into two independent transactions with a checkpoint file between them:

   • Phase 1 (Archive): Lock session → write archive → clear messages → checkpoint(status=archived)
   • LLM call (no transaction — can safely retry)
   • Phase 2 (Memory): Lock session → write memories → checkpoint(status=completed) → post_action(enqueue_semantic)

4. Post-actions: Deferred work (like enqueue_semantic) is registered during the transaction but executed after commit. If the process crashes after commit but before post-actions complete, the journal retains them for replay on restart.

5. Best-effort rollback: Each undo step is independently try-caught. A failing rollback entry doesn't prevent subsequent entries from running. fs_rm is marked as non-reversible (skip on rollback).

Operation-specific strategies

Operation	Lock Mode	Order	Rollback
rm()	rm (bottom-up tree) or normal (parent for files)	VectorDB delete → FS delete	Restore VectorDB from snapshot
mv()	mv (src tree + dst parent)	FS move → VectorDB URI update	Reverse FS move
add_resource (finalize)	normal (parent dir)	FS move + post_action(enqueue_semantic)	Delete final dir
session.commit	normal (session dir) × 2	Two-phase with checkpoint	Phase-specific cleanup

Crash Recovery

On TransactionManager.start(), scan /local/_system/transactions/ for residual journals:

Journal Status	Recovery Action
COMMITTED + post_actions	Replay post_actions → cleanup
COMMITTED / RELEASED	Cleanup locks + journal
EXEC (age > 300s)	Execute rollback → cleanup
INIT / ACQUIRE	Cleanup locks + journal

Alternatives Considered

• Database-level transactions (SQLite): Only covers VectorDB, not AGFS filesystem operations.
• Saga pattern with compensating transactions: More complex orchestration; undo log achieves the same goal with simpler code.
• Event sourcing: Overkill for the current write patterns; would require rewriting the storage layer.

Use Case

Any production deployment where:

• Process crashes or restarts can leave data inconsistent
• Concurrent operations on the same directory tree need coordination
• session.commit() with LLM calls can timeout, leaving partial state
• add_resource must guarantee that semantic generation is eventually enqueued

Example API (Optional)

from openviking.storage.transaction import TransactionContext, get_transaction_manager

tx_manager = get_transaction_manager()

async with TransactionContext(tx_manager, "my_operation", [path], lock_mode="normal") as tx:
    # Record undo entry before each sub-operation
    seq = tx.record_undo("fs_write_new", {"uri": path})
    await do_something(path)
    tx.mark_completed(seq)

    # Register deferred post-commit work
    tx.add_post_action("enqueue_semantic", {"uri": uri, ...})

    # Commit (or auto-rollback on exception / missing commit)
    await tx.commit()

Additional Context

• Design document: test_scripts/transaction-design.md
• Existing skeleton code in openviking/storage/transaction/ has been extended
• The Session.commit() method changes from sync to async (def commit → async def commit)
• 52 new tests cover unit (undo, journal, path_lock, context_manager) and integration (rm/mv rollback, crash recovery, post_actions, concurrent locks)

Contribution

• I am willing to contribute to implementing this feature

[r266-tech]

Great design! A few observations from an operational perspective as an OpenViking deployer:

1. Post-action retry policy
The current design defers enqueue_semantic as a post-action replayed on crash recovery. But what about repeated failures (e.g., LLM provider rate-limiting or outage)? A bounded retry with exponential backoff + dead-letter mechanism would prevent infinite replay loops. Something like:

• Max retries: configurable (default 3)
• On exhaustion: move to a _failed_post_actions/ journal for manual inspection

2. Journal compaction / TTL
For long-running instances, /local/_system/transactions/ could accumulate completed journals. A periodic cleanup (e.g., remove COMMITTED journals older than 24h) would keep the directory lean. This matters for start() scan performance.

3. Lock timeout observability
The fencing-token design is solid. One suggestion: emit a structured log/event when a stale lock is detected and force-acquired. This helps operators understand contention patterns without digging into journal files.

4. Real-world consistency issue we've hit
In our deployment, we've observed orphaned VectorDB entries after interrupted rm() operations — exactly the scenario described. The undo-log approach with VectorDB-first ordering makes sense. One edge case worth testing: what happens if the VectorDB snapshot used for rollback is itself stale (e.g., concurrent write to the same URI)?

Looking forward to this landing — it addresses real pain points. Happy to help test.

[r266-tech]

💡 TransactionJournal Contribution Proposal

Hi, I'd like to contribute the TransactionJournal (WAL) component for this feature.

Why TransactionJournal first?

• Highest independence — no dependency on TransactionContext or ResourceManager changes
• Unlocks all downstream components (TransactionContext, ResourceManager integration)
• Complements the likely TransactionContext focus, enabling parallel development

Proposed Design

• JournalEntry dataclass: txn_id, operation (enum: CREATE/UPDATE/DELETE/MOVE), target_uri, old_data/new_data, timestamp, status (PENDING/COMMITTED/ROLLED_BACK)
• JournalSummary: lightweight view for listing/monitoring
• TransactionJournal class with async interface:
  • begin(txn_id) / record(entry) / commit(txn_id) / rollback(txn_id)
  • recover() — replays incomplete transactions on startup
  • compact(before_timestamp) — garbage collection of old committed entries
• Storage: append-only file per transaction under data/transactions/
• Full test coverage with pytest

Scope

Aiming for a focused PR covering:

1. Data models (JournalEntry, JournalSummary, operation enum)
2. TransactionJournal core implementation
3. Recovery mechanism
4. Compaction
5. Unit tests

Happy to discuss design choices or adjust scope. Already have a detailed implementation plan ready.

---

Related: PR #402 (URI normalization) and PR #403 (PDF bookmark headings) are also from me — building familiarity with the codebase.

[qin-ctx]

Hi @r266-tech — thanks for the detailed feedback and the offer to contribute! Great timing to flag this — we're actively working on this feature right now and want to avoid duplicating effort.

Current Progress

The transaction mechanism is largely implemented. Here's what's already in place:

Core implementation (openviking/storage/transaction/):

• TransactionJournal — AGFS-persisted WAL at /local/_system/transactions/, with begin / record_undo / commit / rollback and crash recovery scan on startup (this is done, so the component you proposed is already covered)
• TransactionRecord — data models for journal entries and transaction state machine (INIT → ACQUIRE → EXEC → COMMITTED → RELEASED)
• PathLock — fencing-token-based distributed locks ({tx_id}:{monotonic_ns})
• UndoLog — ordered undo entries with best-effort rollback (each step independently try-caught)
• TransactionContext — async context manager wrapping the full lifecycle
• TransactionManager — orchestrates lifecycle, lock coordination, and crash recovery

Integration: rm(), mv(), add_resource finalize, and two-phase session.commit() are all being wired in.

Tests (tests/transaction/): 52 tests covering unit (undo, journal, path_lock, context_manager) and integration (rm/mv rollback, crash recovery, post_actions, concurrent locks).

On Your Feedback

All four points are well-taken:

1. Post-action retry with backoff + dead-letter — not yet implemented. Retry count/delay fields are reserved in the journal entry, but the bounded-retry + _failed_post_actions/ fallback is missing. Clear gap.
2. Journal compaction / TTL — also not done. The start() scan will get slower over time without it.
3. Stale lock observability — we emit a debug log today, but no structured event. Worth adding before the first production release.
4. Stale VectorDB snapshot on rollback — good edge case. The VectorDB-first ordering in rm() mitigates most scenarios, but a concurrent write to the same URI after the snapshot but before rollback is a real gap we should test explicitly.

Where You Could Contribute

Given the core is built, the most valuable areas for contribution are:

• Post-action retry policy — bounded retries + exponential backoff + _failed_post_actions/ dead-letter directory (your point 1 is already a solid spec)
• Journal compaction — background task or on-startup GC for COMMITTED journals older than a configurable TTL
• Stale-lock structured events — a small observability hook when force-acquiring a lock
• Edge-case tests — the concurrent-write-during-rollback scenario you raised; chaos/fault-injection style tests for crash recovery paths

We'll have a draft PR up soon. Happy to loop you in once it's open so you can pick up one of the above. Does any of these areas interest you?

[r266-tech]

Thanks @qin-ctx for the thorough status update! Great to see the core is already solid.

I'd love to pick up Post-action retry policy — it's the most self-contained piece and I already have a concrete spec in mind:

• Bounded retries with exponential backoff (configurable max_retries, base_delay_s, max_delay_s)
• Dead-letter directory at _failed_post_actions/ for actions that exhaust retries
• Structured logging for each retry attempt + final dead-letter write
• Tests covering: retry exhaustion → dead-letter, transient failure → eventual success, backoff timing

Happy to also tackle journal compaction as a follow-up once the retry policy is merged.

Please loop me in on the draft PR — I'll base my work on it. 🚀

[qin-ctx]

Post-action retry is yours!

Quick context on the current implementation: _execute_post_actions() in transaction_manager.py is single-shot fire-and-forget today — failure logs a warning and the action is dropped. Your spec covers exactly what's missing.

A few design notes before you start:

On retry necessity: crash recovery already handles the "process died before post-actions ran" case — the startup scan replays any COMMITTED journal with pending post-actions. What retry needs to solve is transient failures while the process is still alive (queue temporarily unavailable, enqueue timeout, etc.). So in-memory retry state is sufficient — no need to persist back to the journal.

On retry strategy: recommend only retrying on transient errors (asyncio.TimeoutError, connection-class exceptions) and routing permanent failures (unknown action type, bad params) straight to dead-letter to avoid pointless retries. Suggested backoff: max_retries=3, base_delay=1s, max_delay=30s, with jitter to avoid thundering herd.

Dead-letter path: suggest per-transaction scoping at /local/_system/transactions/{tx_id}/_failed/ — easier to correlate failures back to their originating transaction.

Suggest splitting post-action retry and journal compaction into two separate PRs to keep each reviewable. I'll tag you once the draft PR is up.