Test coverage gap: README claims vs actual verification

[oldsj]

Problem

The README makes claims that need verification through the docs-as-specs pipeline.

Current State (Jan 2026)

After running /validate-docs, the coverage is better than expected:

Category	Coverage
README features with specs	6/7 (86%)
Spec assertions with tests	34/35 (97%)

Critical Gap: Agent Workflow

The README prominently features this workflow:

┌─────────────┐     ┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Spawn     │────►│    Work     │────►│     PR      │────►│    Close    │
│   (main)    │     │  (k8s ns)   │     │  (GitHub)   │     │  (summary)  │
└─────────────┘     └─────────────┘     └─────────────┘     └─────────────┘

This has NO spec and NO tests. It's the headline feature but completely unverified.

Other Gaps

• chat.md: "Messages persist across page reloads" - no test
• Terminology: Specs say "Sessions", some UI/tests say "INBOX"
• Skipped tests: Several spec behaviors have skipped tests due to flakiness

Infrastructure Claims (Out of Scope for E2E)

These README claims are about infrastructure, not user behavior:

• "Workers continue after browser closes" - DBOS guarantee
• "Task state persists across restarts" - DBOS guarantee
• "Workers in isolated K8s namespaces" - Deployment architecture

These should be verified via integration tests, not E2E.

Action Items

1. Create docs/specs/agent-workflow.md
2. Add message persistence test to chat spec coverage
3. Fix skipped tests or remove spec assertions they cover
4. Standardize terminology (Sessions vs INBOX)

What IS Working Well

The docs-as-specs approach is solid:

• sessions.md → 21/21 assertions tested (100%)
• layout.md → 7/7 assertions tested (100%)
• chat.md → 6/7 assertions tested (86%)

The pipeline works, we just need to extend it to the agent workflow.

[oldsj]

Documentation Validation Report (2026-01-15)

Ran /validate-docs to audit README → Specs → Tests coverage.

README Features → Specs

README Feature	Spec	Status
Main thread conversation	docs/specs/chat.md	✅
Sessions (background work)	docs/specs/sessions.md	✅
Notifications	docs/specs/sessions.md	✅
Mobile layout (bottom tabs)	docs/specs/layout.md	✅
Desktop layout (sidebar)	docs/specs/layout.md	✅
Agent Workflow (Spawn → Work → PR → Close)	NO SPEC	❌

Spec Assertions → Tests

Spec	Coverage	Notes
chat.md	6/7 (86%)	Missing: "Messages persist across page reloads"
sessions.md	21/21 (100%)	Fully covered
layout.md	7/7 (100%)	Fully covered

Critical Gaps Found

1. Agent Workflow has NO spec - README prominently features:

   Spawn → Work (k8s ns) → PR (GitHub) → Close (summary)
   

   This is a major advertised feature with no corresponding spec defining expected behavior.

2. Message persistence untested - chat.md claims "Messages persist across page reloads" but no test reloads and verifies.

3. Terminology mismatch - Specs say "Sessions" but some tests/UI use "INBOX". Should standardize.

Skipped Tests (Spec behaviors not actively verified)

• 04-session-expanded-view.spec.ts: logs tab switching, collapse toggle
• 05-notification-toast.spec.ts: dismiss button, max 3 notifications
• e2e/user-journey.spec.ts: session spawning via conversation

Recommendations

1. Create docs/specs/agent-workflow.md to document agent lifecycle
2. Add message persistence test (send → reload → verify)
3. Align terminology across specs, tests, and UI

Summary

Category	Coverage
README features with specs	6/7 (86%)
Total spec assertions tested	34/35 (97%)

The main gap is the Agent Workflow - it's the headline feature in the README diagram but has no spec or tests. The rest of the docs-as-specs system is working well.