🚧 Proposal: Core Transport Layer Primitives – `jods.persist`, `jods.sync`, `jods.stream`

[clamstew]

This issue consolidates design requirements and implementation direction for three key primitives that expand JODS into a full-stack reactive state platform:

• #5 jods.persist(...) – storage adapter to persist store state to disk, memory, or remote.
• #7 jods.sync(...) – bidirectional state replication across environments (client ↔ server, peer ↔ peer).
• #9 jods.stream(...) – one-way async feed of external data (SSE, HTTP streams, etc.) into a store.

Together with defineStore() and withJods() (Remix), these complete the lifecycle of: initialize → mutate → hydrate → persist → sync → stream.

---

✅ Shared Design Principles

• Unified Sink Model: All incoming state updates—whether local mutations, synced diffs, streamed payloads, or restored snapshots—should flow through a common reconciliation pathway. This enables onUpdate, signals, and devtools to behave consistently regardless of source.

• Schema-Driven Trust Boundary: If a defineStore() uses a Zod schema, it should act as the boundary for validating all incoming external data. Invalid updates should be ignored and optionally logged.

• No Dropped State: Streaming or syncing should never drop updates by default. If necessary for performance (e.g. UI rendering), downstream systems can batch. But the reactive core should maintain every change for time-travel/debugging.

• Pluggable, Imperative API: All primitives should be low-level, composable JS calls (persist(store, opts)) rather than builder chains (persist(...).andSync(...)) to preserve flexibility, composability, and Unix-style separation of concerns.

---

1. 🧠 jods.persist(store, opts?) - #5

Purpose: Store state to localStorage, IndexedDB, sessionStorage, or a remote adapter.

Core Behavior:

• On init, loads persisted state and merges into store.
• Subscribes to store updates and saves them (debounced).
• Should be opt-in per store or partial (with a filter).
• Can use schema versioning to manage breaking changes.

Key Features:

• Supports both sync and async storage engines.
• Optional version field on defineStore() could seed a cache key (e.g., jods:cart:v1.2.0) and rotate on breaking schema changes.
• Default error behavior: catch + log invalid saves; never crash app.
• Hook support: onLoad, onSaveError.

Example:

jods.persist(store, {
  storage: localStorage,
  key: 'cart',
  version: '1.0.2',
  filter: (key) => !key.startsWith('secure_'),
});

---

2. 🔄 jods.sync(socket, store, opts?) - #7

Purpose: Two-way syncing of state across network boundaries (WS, postMessage, devtools, etc.)

Core Behavior:

• Wraps a SocketLike interface (with .send() + onmessage).
• Diffs outbound updates (via onUpdate) and sends as patches.
• Receives remote patches, validates, and applies via unified sink.
• Optionally handshake on connect to align divergent stores.

Must Support:

• Partial syncing (via key allow-list).
• Conflict resolution via timestamps (even in v1).
• Error logging, not crashing (e.g., schema validation fails).
• Events: onPatchRejected, onConnectionLost.

Example:

jods.sync(socket, store, {
  allowKeys: ['items', 'note'],
  mode: 'peer', // future: 'primary', 'readOnly'
});

Conflict Strategy:

• Each patch carries a timestamp.
• Local patches can queue if in-flight.
• Remote patches are ordered and applied only if newer.

---

3. 📡 jods.stream(readableStream | generator, store, opts?) - #9

Purpose: One-way push of streamed external data into a store (HTTP streams, SSE, etc.)

Core Behavior:

• Consumes JSON stream chunks or structured diffs.
• Applies them as patches (or replaces) on arrival.
• Default behavior: validate each chunk against schema.

Design Notes:

• Support { type: 'patch' | 'replace' } per chunk.
• Optionally batch in requestAnimationFrame to avoid UI jank.
• Expose jods.applyPatch() for manual patch usage.

Example:

await jods.stream(eventSource, store, {
  validate: true,
  batch: true,
  onError: (err) => logger.warn('Stream parse failed', err),
});

** Expose jods.applyPatch() alludes to the design principle from earlier about exposing the internals for advanced usage.

---

🧪 Devtools Integration

Existing: [src/react/debugger.tsx](https://github.com/clamstew/jods/blob/main/src/react/debugger.tsx)

• Expand debugger to visualize:

  • Incoming patches (sync or stream)
  • Source (local, persisted, streamed, remote)
  • Rejected patches
  • Versioning or hydration flows

• Add dev command to simulate patches manually

• Possibly: a debug transportLog for showing outbound and inbound event traffic

---

📚 Documentation Plan

New sections on the docs site should include:

• Core Concepts: transport layer, schema validation, sync vs stream vs persist

• How-To Guides:

  • Offline caching with jods.persist
  • Real-time sync between tabs or clients with jods.sync
  • Streaming updates from an async function or HTTP response

• Examples:

  • Chat app (sync)
  • Live dashboard (stream)
  • Offline cart (persist)
  • Remix integration walkthrough

---

🔬 Implementation & Testing Strategy

• Use [jods-remix-starter](https://github.com/clamstew/jods-remix-starter) to test:

  • Persist + Sync together (ensure no feedback loops)
  • Sync + Stream on same store
  • Schema versioning / migration edge cases

• Plan to build:

  • jods.transport.test.ts
  • jods-remix.e2e.ts Playwright flows (multi-tab sync, SSR → stream handoff)

• Cursor can be used to scaffold these workflows iteratively.

---

🚫 Not in Scope Yet

We anticipate building toward these capabilities, but they are not included in the initial implementation of persist, stream, or sync:

• CRDT/OT Conflict Resolution – Future work may implement true peer-to-peer conflict resolution via conflict-free replicated data types (CRDT) or operational transforms (OT).
• Chained API (persist(...).andSync(...)) – While expressive, this approach limits composability and is harder to scale. We'll revisit if dev feedback demands it.
• Server-Pushed Transforms – Applying a patch or command that maps to a defined action handler or reducer (e.g., transform via cart.addItem) is a more opinionated abstraction and could appear in v2+.
• Persistent Retry Queues – For now, we assume sync happens only when connected. An offline queue or sync-on-reconnect flow (e.g., local writes buffered until network resumes) could follow in a dedicated feature.

---

📌 TL;DR

This issue defines the shape of JODS' transport architecture. V1 of persist, stream, and sync aim to be composable primitives that respect schema boundaries, maintain signal-based updates, and push state coherently across runtime boundaries—without relying on last-write-wins or dropping updates.

This enables JODS to truly become the "active model" layer of a Remix app, with Remix as the router/controller and JODS as the data layer—coherent, reactive, and cross-context.

---

Informed by Issues #5, #7, #9, and PRs #20 and #22.

[clamstew]

🕱️ Future Scope: Deeper Dives on Deferred Transport Features

This comment elaborates on three advanced transport layer features that were marked as 🚫 Not in Scope Yet for initial implementation in the main JODS transport proposal:

• CRDT / Operational Transform (OT) Conflict Resolution
• Server-Pushed Transforms
• Persistent Retry Queues

These concepts are now laid out clearly enough to hand off for exploration, prototyping, or independent contribution.

---

🧼 CRDT / Operational Transform Conflict Resolution

What:
Enable peer-to-peer or multi-client editing of a shared JODS store without relying on a central authority or last-write-wins. Conflict-free replicated data types (CRDTs) and operational transforms (OT) are well-established techniques for managing concurrent writes in distributed systems.

Why:
Currently, jods.sync() assumes a timestamp-based merge model. While that works for client-server sync, it doesn’t scale well to concurrent edits from multiple peers (e.g. two users typing in the same field).

How:

• Abstract patches into higher-level "intentful ops" (e.g., insertAt(index, value) vs. items = [...]).
• Maintain metadata per key (e.g., per-character IDs in a text array).
• Implement sync middleware or a sync.mode = 'crdt' option.
• Optionally expose a CRDT primitive (e.g. jods.crdtStore(...)) that replaces store().
• Begin with text + array merging (e.g. for chat or form builders).
• Use Yjs or Automerge as inspiration.

Testing:

• Simulate 3+ peer edits with dropped and reordered messages.
• Verify that all peers converge on the same state.
• Devtools: visualize history and conflict resolution steps.

---

💡 Server-Pushed Transforms

What:
Allow incoming jods.sync or jods.stream events to be mapped to defined actions instead of directly patching state. Example: turning { type: 'ADD_ITEM', payload: { id: 123 } } into cart.actions.addItem(...).

Why:
Decouples data wire format from internal state structure. Aligns with Remix-style loader/action conventions and lets the server retain logic control (e.g. validation, logging, auth).

How:

• Add jods.transform hook to intercept incoming patches/events.
• Allow transform to return an action function, or a new patch.
• Expose action map metadata from defineStore().
• Could use { kind: 'action', name: 'addItem', args: [...] } format.
• Middleware signature: transform({ event, store }) => Patch | Action | null

Example:

jods.sync(socket, cart, {
  transform: ({ event, store }) => {
    if (event.kind === 'action' && event.name === 'addItem') {
      return cart.actions.addItem(event.args);
    }
  }
});

Testing:

• Event wire logs: simulate action events being replayed.
• Mismatch scenarios: server sends unknown action name.
• Integration with Zod: validate payload against handler args.

---

🛠️ Persistent Retry Queues

What:
If a store syncs data to a remote and the network is offline, persist those updates locally and retry once the connection is restored.

Why:
Offline-first UX is increasingly expected in modern apps. Retry queues prevent data loss and allow client-side state to remain fluid even during disconnections.

How:

• Hook into jods.sync() and onUpdate() to enqueue failed patches.
• Use IndexedDB or navigator.storage.persist() to store the queue.
• On reconnect, flush queue and send updates.
• Ensure patch order and deduping.
• Mark queue entries with UUID + timestamp.

API Sketch:

jods.sync(socket, store, {
  retry: true, // or retryQueue: customAdapter
});

Edge Considerations:

• Replay on reconnect must not duplicate updates.
• May need conflict resolution fallback (CRDT).
• Possibly serialize with schema version or hash.

Testing:

• Simulate lost connection, enqueue updates.
• Reload browser with queue intact, verify replay.
• Devtools: show queue depth and last replay.

---

❌ Chained API

Considered and indefinitely deferred. While persist(...).andSync(...).andStream(...) is syntactically appealing, it compromises:

• Composability
• Predictability
• Static analysis

Encouraging clear, one-purpose calls keeps the mental model simpler and fits Unix-style pipeline philosophy better.

---

PRs welcome on any of the above, provided they:

• Respect schema-based validation boundaries
• Reuse onUpdate, patching, or loader hydration pipelines
• Plug cleanly into defineStore() + Remix

Future milestone: these may combine under a unified jods.transport(...) pipeline that abstracts connectors (local, sync, stream, retry, etc).

[clamstew]

💿📦 Clarify SSR Hydration vs Local Persistence Strategy

💡 Insight: When combining SSR hydration via withJods() 💿 and local persistence through jods.persist() 📦, it’s not yet clear which data source should take precedence. A page load might rehydrate from Remix’s server loader and also apply stored data from a previous session. If the two conflict, behavior could become unpredictable. ⚠️

✨ Suggestion: Define and document a precedence strategy:

• 🧩 Should persisted state 📦 override SSR loader data 💿, or vice versa?
• 🔄 Offer an optional merge strategy: e.g., apply persisted state only if it's newer or has unsynced changes.
• 📋 Provide examples or recipes for resolving state reconciliation.

---

🔌👂 Provide Teardown/Unsubscribe Hooks for Transports

💡 Insight: There should be a reliable way to dispose of sync 🔄, persist 📦, or stream 🌊 connections when a component unmounts or a route changes. Long-lived connections like WebSockets or SSEs can otherwise lead to memory leaks or ghost updates. 👻

✨ Suggestion: Ensure transports return a teardown handle:

• 🧰 Example: const stop = jods.sync(socket, store); → stop() removes listeners 👂.
• 🛑 For streams 🌊: accept AbortSignal or return a cancel function.
• 📚 Document cleanup best practices clearly.

---

🔄📡 Address Multi-Source Update Coordination

💡 Insight: When using multiple inputs like jods.sync() 🔄 and jods.stream() 🌊 on the same store 📦, the library should explain how simultaneous updates from different channels are resolved. The internal ordering and merge strategy between update sources isn’t documented. 🤔

✨ Suggestion: Clarify how JODS handles this scenario:

• ✅ Confirm that multiple transports can attach to a single store 📦.
• 🔍 Document patch ordering logic: e.g., does sync win if it has a timestamp 🕰️? Does stream apply immediately?
• 🚧 If not fully solved, flag it as a best practice to isolate keys or namespace updates.

---

🌐🧠 Multi-Tab and P2P Sync Considerations

💡 Insight: Multi-tab state sync (e.g., two tabs editing a form simultaneously) is a real-world use case. Developers may not realize that jods.sync() 🔄 can support this using local peer channels.

✨ Suggestion: Highlight browser-native sync options:

• 📡 Provide examples using BroadcastChannel or localStorage events as SocketLike interfaces.
• 🔌 Consider adding a built-in multi-tab adapter for convenience.
• 👂 Document how to prevent infinite loops or patch reapplication.

---

🧩💿 Define Server-Side State Scope for jods.sync

💡 Insight: Remix loaders and actions 💿 are request-bound. For jods.sync() 🔄 to operate in real-time, a persistent server context must be maintained. Developers may mistakenly expect that Remix can handle continuous connections.

✨ Suggestion: Clarify persistent backend expectations:

• 🧠 Recommend use of WebSocket servers, background workers, or edge runtimes.
• 🔧 Provide minimal setup examples with jods.sync() running in a stateful context.
• 📦 Offer a fallback HTTP-only sync pattern for request/response cases.

---

🕰️🔍 Acknowledge Timestamp Limitations in Conflict Resolution

💡 Insight: Timestamp-based patch resolution 🕰️ is simple but unreliable for concurrent edits across peers. It assumes clock sync and offers no causal tracking 🔍.

✨ Suggestion: Be explicit about limitations:

• ⚠️ State that timestamp merge is only safe in centralized systems.
• ⏳ Recommend server-based time authorities where possible.
• 🧬 Reaffirm plans to support CRDT or OT-based approaches in future for robust peer sync.

---

🏎️🔧 Expand on Performance Backpressure Options

💡 Insight: High-frequency updates, such as from streams 🌊 or telemetry sync 🔄, need throttling or batching to prevent UI jank or memory pressure.

✨ Suggestion: Expose performance controls:

• 🧰 Support batch: true (uses requestAnimationFrame) or batch: ms delay tuning.
• 🔄 Apply these across sync 🔄 and stream 🌊 for a unified developer experience.
• 📋 Document impact on reactivity 🔄 and update timing expectations.

---

🔐📚 Emphasize Validation and Security Hooks

💡 Insight: Beyond schema validation 📐, developers may need more control over what can be persisted 📦, synced 🔄, or streamed 🌊. Especially in untrusted environments, fine-grained filters or modes are essential.

✨ Suggestion: Offer scoped transport permissions:

• 📌 Add allowKeys, filter, or mode: "receive-only" options.
• 🛡️ Let developers constrain exposure of sensitive fields.
• 💿 Remind Remix users to enforce access controls server-side and treat schema validation as a secondary line of defense.
• 🧰 Offer examples showing selective field exposure and restricted sync behavior.