layered-memory

A persistent, token-efficient memory system for OpenClaw agents — inspired by OS memory paging and the Unix "everything is a file" philosophy.

The Problem

OpenClaw agents suffer from three compounding memory problems:

1. Context compression = amnesia — Long sessions get compressed; early context disappears. The next session starts from zero.
2. Skill injection overhead — 20+ skill descriptions injected at startup consume tokens on things that may never be used in that session.
3. All-or-nothing memory — Either the agent remembers nothing, or you stuff everything into context and blow the window. Neither is right.

Existing approaches (always-on context injection, RAG) treat memory as an injection problem. We treat it as a retrieval problem — and the file system already solved the analogous problem in operating systems decades ago.

The Solution

Layered Memory applies two ideas from computer science to agent memory management:

1. Progressive Disclosure (borrowed from OpenClaw Skills)

OpenClaw skills already do this well: only a short description lives in the system prompt; the full SKILL.md is loaded on demand. We apply the same pattern to memory:

System Prompt  →  MEMORY.md only (~500 tokens, always present)
On Demand      →  daily logs, task snapshots, lesson files (loaded when needed)

2. Everything is a File (Unix philosophy)

Memory lives in plain Markdown files — not in model weights, not in opaque vector DBs. Any tool — grep, semantic search, or a human editor — can read, verify, and correct it.

File system (disk)   =  Long-term memory  (scalable for local workloads)
Context window (RAM) =  Working memory    (limited, precious)
Agent reads file     =  Paging memory into context
Session ends         =  RAM clears; disk persists

Architecture

┌──────────────────────────────────────────────────────┐
│  Context Window  (always loaded, target ≤500 tok)    │
│  └── MEMORY.md  — curated facts/prefs/rules          │
└────────────────────┬─────────────────────────────────┘
                     │  load on demand
                     ▼
┌─────────────────────────────────────────────┐
│  File System  (survives sessions & restarts) │
│  ├── memory/YYYY-MM-DD.md  — daily raw log  │
│  ├── current-task.md       — task snapshot  │
│  ├── memory/lessons/       — learned lessons│
│  └── (optional) vector index — semantic search│
└─────────────────────────────────────────────┘

When to load from disk

Trigger	Load what
Starting a new task	Search for past experience on this topic
Hitting an error	Search for similar past errors + fixes
Resuming a session	Read current-task.md for re-entrancy
User asks about the past	Retrieve relevant daily log entries

Decision flow

Session starts
    ├─ current-task.md exists?  → YES → recall task snapshot → resume
    │                            NO  → proceed normally
    └─ New task begins?
           └─ recall past experience → act → remember outcome

Error occurs (2nd time)
    └─ recall error history → apply fix → remember resolution

Weekly
    └─ summarize: distill daily logs → MEMORY.md → prune stale entries

Memory Protocol

Four operations the agent follows consistently:

Operation	When	Target
remember	After significant events, decisions, errors	memory/YYYY-MM-DD.md (raw); MEMORY.md (distilled)
recall	Before starting tasks; on errors; on session resume	grep or semantic search across memory files
summarize	Weekly, or when daily logs exceed ~50 entries	Distill daily logs → MEMORY.md (keep under ~500 tok)
forget	On explicit request or TTL expiry (default: 90 days)	Move to memory/archive/ — no hard deletes

Fallback rule: if recall returns no results, proceed without — do not hallucinate past context.

Failure Modes & Mitigations

Failure	Mitigation
Recall miss (relevant memory not found)	Fall back to keyword search + time-window filter; surface to user for manual confirmation
Memory pollution (wrong info recorded)	Human-editable plain files — easy to correct; optional confidence tags ([low], [uncertain])
File bloat over time	Weekly summarize + 90-day TTL + memory/archive/ rotation
Stale preferences conflict with new ones	Newer entry takes precedence; conflicts flagged with [SUPERSEDES: date] tag

Scope & Non-Goals

This skill is designed for:

• Personal agent deployments with a single user
• Local-first, privacy-preserving setups
• Workloads where memory grows over weeks/months, not gigabytes

This skill does not:

• Guarantee 100% recall — file search has coverage limits
• Replace vector databases for large-scale or multi-user deployments
• Validate the factual accuracy of stored memories
• Handle concurrent multi-agent writes safely

vs. Always-on Context Injection

Dimension	Always-on Injection	Layered Memory
Token cost	Paid upfront every session	Paid only when retrieval is triggered
Always-loaded overhead	Grows with memory size	Bounded: target ≤500 tok, configurable (MEMORY.md only)
Compression resilience	❌ Injected content vanishes on compression	✅ Files persist; re-retrievable any time
Auditability	Hidden in prompts	Plain files — human-readable and editable
Works best for	Small, always-relevant context	Large, growing memory with sparse access patterns

These approaches are complementary. Best practice: always inject MEMORY.md (curated summary, small); retrieve everything else on demand.

Real-World Validation

Deployed on a personal OpenClaw instance over 30+ days across multiple ongoing projects.

Observed qualitative outcomes:

• Session re-entrancy: Multi-step tasks spanning days resume from current-task.md without re-establishing context from scratch.
• Error learning: Recurring errors (e.g. misconfigured API keys, deployment edge cases) are resolved on first recurrence instead of being debugged again from zero.
• Bounded context overhead: Always-loaded memory targets ≤500 tokens (configurable) regardless of how many weeks of logs accumulate — daily logs do not grow the context footprint.
• Human-correctable: Errors in memory have been corrected by directly editing markdown files, with no tooling required.

Note: Controlled quantitative benchmarks (token counts, resolution times) are a planned next step. The above reflects observed workflow impact.

Quick Start (10 minutes)

# 1. Clone
git clone https://github.com/tankyhsu/layered-memory ~/.openclaw/skills/layered-memory

# 2. Bootstrap file structure
cd ~/.openclaw/skills/layered-memory && ./setup.sh

# 3. Edit your MEMORY.md
open ~/openclaw/MEMORY.md   # add your agent's core facts & preferences

# 4. Reference SKILL.md in your OpenClaw agent config
# The agent will now follow the layered memory protocol automatically

After setup, the agent will:

• Load MEMORY.md on every session start (~500 tokens, one-time cost)
• Search memory files before starting tasks or hitting recurring errors
• Write daily logs and task snapshots as work progresses
• Summarize logs into MEMORY.md on a weekly cadence

What's Included

layered-memory/
├── SKILL.md              # OpenClaw skill definition + trigger rules
├── setup.sh              # One-command bootstrap
├── templates/
│   ├── MEMORY.md         # Long-term memory template
│   ├── daily.md          # Daily log template
│   └── task.md           # Task snapshot template (re-entrancy)
├── LICENSE               # MIT
└── README.md             # This file

Estimated Monthly API Cost

Mode	Assumptions	Cost
File-only (grep search)	Fully local, no external calls	$0
With semantic search (light use)	~100 queries/month via SiliconFlow BGE-M3 at ~$0.0001/query	~$0.01 / month
With semantic search (heavy use)	~10,000 queries/month at ~$0.0001/query	~$1 / month

Cost scales with retrieval frequency; file storage and write operations are free.

Pricing is provider-dependent and subject to change; figures above are illustrative.

License

MIT — free to use, modify, and distribute.

---

Built on the shoulders of giants: Unix file system philosophy + OpenClaw's progressive skill disclosure pattern.