Openclaw Memory Toolkit
OpenClaw Memory Toolkit is a memory layer for AI agents. It remembers what matters across sessions: it extracts durable facts from your conversations, resolves contradictions instead of hoarding them, and lets you ask what it knew on any past date.
Memory Pipeline Skill
Complete memory management pipeline for OpenClaw agents: extraction, archiving, scoring, consolidation, health monitoring, and ontology — local by default (Ollama runs locally via HTTP).
⚠️ One opt-in exception: trace_extractor.py can send memory/session excerpts
to Ollama cloud (https://ollama.com) when OLLAMA_API_KEY is configured.
With no key it stays local; TRACE_LLM_LOCAL_ONLY=1 refuses every cloud call. The
destination is printed before each send. See the Security Notes below.
⛔ MUST — releasing this skill. Every change to this skill goes through
scripts/release.sh, without exception. A change is not done untilscripts/release.sh checkpasses green. Then, and only then, sync to the installed skill and tag viascripts/release.sh release vX.Y.Z "msg". Never edit the installed skill directly. Never tag or publish on a red gate — fix the drift or the invariant, never bypass the gate. Pipeline: local repo → installed skill → GitHub → ClawHub (manual).
Pipeline Overview
Nightly Cron (23h)
│
├─ 1. trace_extractor.py # Extract decisions/errors/facts from sessions
├─ 2. auto_archive.py # Archive daily notes >21 days
├─ 3. scoring.py # Score all memories with temporal decay
├─ 4. consolidate_advisor.py # Suggest consolidations (agent reviews)
├─ 5. conflict_resolver.py # Arbitrate contradictory facts (NLI lifecycle)
├─ 6. memory_health.py # Periodic health check (weekly)
└─ 7. hybrid_search.py # Hybrid search: FTS5 + sqlite-vec + RRF
All scripts are standalone and composable. Run individually or as a pipeline.
Fact lifecycle (introduced in v3.0.0) — current release v4.0.6
The search DB no longer just accumulates facts: every fact carries a lifecycle
(active / superseded / disputed) and only active facts are ever
retrieved. conflict_resolver.py runs the four-step consistency pipeline:
atomic extraction → targeted retrieval of concurrent active facts → NLI
classification (CONTRADICTION / REDUNDANT / COMPATIBLE) → traceable state
update. A weak contradiction is escalated to disputed rather than silently
destroying an established fact; pending surfaces the queue and resolve --confirm|--reject lifts the ambiguity. compact.py moves terminal facts into a
cold archive (memories_archive + JSONL audit) so the hot FTS5/vector indexes
stay lean without losing traceability.
Scripts
1. trace_extractor.py — Session extraction
Extracts decisions, errors, facts, and patterns from OpenClaw session transcripts and daily notes. Updates daily notes with extracted items, appends entities to the ontology graph.
# Nightly (pattern-based, fast ~5s)
python3 scripts/trace_extractor.py --days 1
# Deep extraction (LLM-powered, ~60-180s)
python3 scripts/trace_extractor.py --days 3 --llm
# With session transcripts
python3 scripts/trace_extractor.py --days 1 --llm --session-file /path/to/session.jsonl
# Preview only
python3 scripts/trace_extractor.py --days 1 --llm --dry-run
Categories extracted:
- 🟢 DECISIONS — new choices, config changes, migrations
- 🔴 ERRORS — bugs, failures, workarounds
- 🔵 FACTS — new versions, configs, status changes
- ⬆️ PROMOTIONS — items worth promoting to MEMORY.md
Output: Daily notes updated, ontology entities added, .trace-extracted flag.
2. auto_archive.py — Daily note archiving
Moves daily notes older than N days to memory/archive/YYYY-MM/ subdirectories.
python3 scripts/auto_archive.py # Archive notes > 21 days
python3 scripts/auto_archive.py --days 30 # Custom threshold
python3 scripts/auto_archive.py --dry-run # Preview only
python3 scripts/auto_archive.py --verbose # Show each file
Idempotent. Only moves YYYY-MM-DD*.md files. Zero dependencies.
3. scoring.py — Temporal decay scoring
Scores all memory items using exponential recency decay, category weights, frequency boost, entity boost, and completion penalty.
python3 scripts/scoring.py # Score all memories
python3 scripts/scoring.py --verbose # Show top 20
python3 scripts/scoring.py --threshold 0.3 # Filter by min score
python3 scripts/scoring.py --dry-run # Don't write output
Scoring formula:
score = weight_category × recency_decay × frequency_boost × entity_boost × completion_penalty
recency_decay = exp(-ln(2) × days_old / HALF_LIFE_DAYS)
Category weights: DECISIONS ×3, ERRORS ×2, FACTS ×1.5, PATTERNS ×1.2, TRANSIENT ×1
Output: memory/scores.json — full ranking with stats and promotion candidates.
4. consolidate_advisor.py — Consolidation suggestions
Analyzes recent daily notes + scores.json to identify clusters, promotions, stale items, and duplicates. Writes consolidation_report.json by default. Modifies MEMORY.md only with --apply-promotions flag (requires confirmation).
python3 scripts/consolidate_advisor.py # Last 7 days
python3 scripts/consolidate_advisor.py --days 14 # Custom window
python3 scripts/consolidate_advisor.py --verbose # All suggestions
python3 scripts/consolidate_advisor.py --no-llm # Skip LLM (fallback)
python3 scripts/consolidate_advisor.py --apply-promotions # Write to MEMORY.md
Output: memory/consolidation_report.json — clusters, promotions, stale items, duplicates.
LLM optional (Ollama) for cluster summaries. Falls back to text-based with --no-llm.
5. memory_health.py — System health check
Comprehensive diagnostics: trace extraction, LoCoMo benchmark, MEMORY.md size, ontology health, daily notes hygiene, index status, drift detection.
READ-ONLY by default: writes nothing to disk. Use --output-dir <path> to save
JSON reports and SVG trend charts.
python3 scripts/memory_health.py # Full health check (read-only)
python3 scripts/memory_health.py --quick # Skip benchmark & LLM (read-only)
python3 scripts/memory_health.py --benchmark # Benchmark only (read-only)
python3 scripts/memory_health.py --deep # LLM + sessions + benchmark (weekly)
python3 scripts/memory_health.py --output-dir results/ # Save reports to disk
python3 scripts/memory_health.py --fix # Fix mode (DESTRUCTIVE)
Output: results/YYYY-MM-DD.json — only with --output-dir.
Destructive actions (--fix): Moves daily notes >14 days old to archive/,
rewrites ontology file (dedup + clean). Creates timestamped backup in memory/backup/
before modifying. Requires interactive confirmation or --force flag.
6. ontology_compact.py — Ontology graph GC
Compacts memory/ontology/graph.jsonl (an append-only operation log) by replaying
it into a consolidated state: one line per active entity, superseded records dropped.
Safe by design: backs up first (MD5-verified), writes to a temp file, validates
that the entity set and contents are identical, and only then swaps in place.
Idempotent: skips when the gain is below --min-gain (default 5%).
python3 scripts/ontology_compact.py --dry-run # Report only
python3 scripts/ontology_compact.py # Compact (threshold 5%)
python3 scripts/ontology_compact.py --min-gain 10 # Skip unless >=10% smaller
Run weekly. Typical gain on a never-compacted log: ~60-65%.
7. hybrid_search.py — Hybrid search (FTS5 + sqlite-vec + RRF)
Hybrid memory search combining lexical (BM25 via SQLite FTS5) and semantic (vector via sqlite-vec) retrieval using Reciprocal Rank Fusion (RRF, k=60).
# Initialize DB with schema
python3 scripts/hybrid_search.py init
# Index all memory files
python3 scripts/hybrid_search.py index
# Search
python3 scripts/hybrid_search.py query "project_alpha"
python3 scripts/hybrid_search.py query "roadmap EIIDP" --top 10
# Lexical only (BM25)
python3 scripts/hybrid_search.py query "2026-08-17" --lexical-only
# Vector only (semantic)
python3 scripts/hybrid_search.py query "memory decay scoring" --vector-only
# JSON output for programmatic use
python3 scripts/hybrid_search.py query "leadership coaching" --json
# Stats
python3 scripts/hybrid_search.py stats
# Index a single file
python3 scripts/hybrid_search.py add path/to/file.md --category skill
How it works:
query → ┬─ vector_search (nomic-embed-text, top 20) ──┐
└─ lexical_search (FTS5/BM25, top 20) ────────┤
↓
RRF(k=60) fusion
↓
min_score filter (≥0.015)
↓
temporal boost (optional)
↓
source deduplication
↓
top K results
RRF ignores raw scores and uses only ranks: rrf(d) = Σ 1/(k + rank_m(d)).
Source deduplication groups by file, returning the best chunk per source.
Gemini vigilance #1 — min_rrf_score (noise threshold):
Chunks appearing in neither top-20 list have RRF score ~0 = pure noise.
Filtered by default at 0.015. Override with --min-score 0 to disable.
Gemini vigilance #3 — temporal_boost (decay weighting):
RRF score is multiplied by (1 + 0.1 * normalized_score) where normalized_score
comes from the score column (populated by scoring.py temporal decay).
Gives slight priority to recent facts when context conflicts.
Disable with --no-temporal-boost.
Schema: Single SQLite file with three synchronized tables:
memories— content, category, layer, source, score, timestampsmemories_fts— FTS5 virtual table (external content, auto-synced via triggers)memories_vec— vec0 virtual table (float[768], nomic-embed-text)
Layers: episodic (daily notes), semantic (long-term facts, ontology), procedural (skills, config)
Requirements: sqlite-vec (pip install in venv), Ollama with nomic-embed-text
Output: hybrid-search/agent_memory.db — SQLite DB with FTS5 + vec0 indexes.
8. conflict_resolver.py — Fact lifecycle & dispute resolution
Four-step consistency pipeline over the search DB: atomic extraction →
targeted retrieval of concurrent active facts → NLI classification → traceable
state update. Only active facts are ever retrieved. A new fact that
contradicts an active one supersedes it (superseded_by now written); a weak
contradiction is escalated to disputed instead of silently destroying an
established fact.
# Analyse one candidate fact (read-only)
python3 hybrid-search/conflict_resolver.py check "On a migré la BDD sur MySQL 8" --subject serveur_prod
# Batch arbitration from JSONL (dry-run; --apply to persist)
python3 hybrid-search/conflict_resolver.py arbitrate facts.jsonl
python3 hybrid-search/conflict_resolver.py arbitrate facts.jsonl --apply --force
# What is waiting for a human? (cheap, deterministic)
python3 hybrid-search/conflict_resolver.py pending
# Lift the ambiguity explicitly
python3 hybrid-search/conflict_resolver.py resolve 42 --confirm
python3 hybrid-search/conflict_resolver.py resolve 42 --reject --replacement "corrected fact"
--confirm restores the wording to active and supersedes any rival on the
same subject; --reject [--replacement] supersedes it and optionally inserts a
corrected fact. Only disputed rows are eligible. --no-llm gives a
conservative lexical fallback; the LLM endpoint is loopback-only.
9. compact.py — Cold storage & lifecycle compaction
Moves terminal facts (superseded, optionally disputed) out of the hot tables
into a cold archive so FTS5/BM25 rank space and the vector index stop carrying
dead rows while full traceability is kept.
python3 hybrid-search/compact.py --stats # hot vs cold sizes
python3 hybrid-search/compact.py --dry-run # what would be archived
python3 hybrid-search/compact.py --apply --min-age-days 30
python3 hybrid-search/compact.py --restore 42 # rehydrate one fact
Rows are copied verbatim into memories_archive in the same SQLite file and
appended to a dated JSONL audit under memory/audit/, then DELETEd from the hot
table (fires the triggers → removed from FTS5 and memories_vec). Never a hard
delete of data, only a move. Dry-run is the default; mutation requires
--apply; a retention guard refuses rows newer than --min-age-days; the DB is
backed up via SQLite's backup() API and MD5-checked before any write. Requires
sqlite-vec (the vec0 delete trigger fires on DELETE).
Ontology
The ontology graph stores entities and relations as JSONL. A YAML schema defines allowed types and relations.
Entity types: Person, Organization, Project, Task, Document, Event, Skill, Device, Service, Tool, Infrastructure, Concept, Location, Pet, BugFix, SecurityEvent, Integration, Feature, Software, Configuration
Relation types: reports_to, has_owner, includes, depends_on, manages, uses, integrated_with, located_at, fixes, monitors
Files:
memory/ontology/graph.jsonl— entity and relation recordsmemory/ontology/schema.yaml— type and relation definitionsmemory/ontology/graph-index.json— search index
Configuration
Environment variables with defaults:
| Variable | Default | Description |
|---|---|---|
WORKSPACE | ~/.openclaw/workspace | OpenClaw workspace path |
OLLAMA_URL | http://localhost:11434 | Ollama API URL |
OLLAMA_MODEL | glm-5.2 | Model for LLM extraction/summaries |
Scoring constants (top of scoring.py):
| Parameter | Default | Description |
|---|---|---|
HALF_LIFE_DAYS | 14 | Recency decay half-life |
MAX_SCORE | 5.0 | Score cap |
PROMOTE_THRESHOLD | 2.0 | Min score for promotion |
ARCHIVE_THRESHOLD | 0.05 | Score below = archive candidate |
Memory health thresholds (top of memory_health.py):
| Parameter | Default | Description |
|---|---|---|
MEMORY_MAX_SIZE | 5000 | MEMORY.md max size in bytes |
DAILY_NOTES_MAX_AGE | 14 | Days before archiving |
Requirements
- Python 3.10+
- Ollama (optional — LLM extraction and cluster summaries)
- No pip packages required — pure stdlib (sqlite-vec optional for hybrid search)
Nightly Cron Integration
Recommended nightly pipeline (after trace extraction):
# In nightly cron (23h):
python3 scripts/trace_extractor.py --days 1
python3 scripts/auto_archive.py
python3 scripts/scoring.py
python3 scripts/consolidate_advisor.py --no-llm # quiet mode
Weekly health check (Monday, separate cron):
python3 scripts/memory_health.py --quick
Monthly deep check (manual):
python3 scripts/memory_health.py --deep
Design Principles
- Local by default — no cloud account, no paid dependency.
trace_extractor.pyis the only cloud-capable script, opt-in viaOLLAMA_API_KEY, disclosed each run, cancellable viaTRACE_LLM_LOCAL_ONLY=1 - Composable — each script is standalone, can run independently
- Safe by default — dry-run available for all analysis scripts; some nightly cron commands modify files by default (archive, scores, consolidation report). Review cron commands before deploying.
- Human-in-the-loop — consolidation suggestions, not auto-merge
- Pipeline-friendly — scripts chain naturally, outputs feed inputs
- Zero dependencies — pure Python stdlib (except sqlite-vec for hybrid search)
- Strata-aware — episodic, semantic, and procedural memory are separated
License
MIT — free to use, modify, and share.
Security Notes
- ⚠️ Nightly cron modifies files by default:
auto_archive.pymoves files,scoring.pywrites scores.json,consolidate_advisor.pywrites consolidation_report.json. Review cron commands before deploying. - ⚠️
--fixmode is destructive:memory-health.py --fixmoves daily notes to archive/ and rewrites ontology. Requires interactive confirmation or--forceflag. Creates timestamped backups inmemory/backup/before modifying. - ⚠️
--forceflag: The--forceflag exists onconsolidate_advisor.pyandmemory-health.pyfor non-interactive/cron use. It skips confirmation prompts. Only use in trusted automation with backups in place. - ⚠️
--apply-promotionsmodifies MEMORY.md:consolidate_advisor.py --apply-promotionsappends entries to MEMORY.md. Requires interactive confirmation or--forceflag. - ⚠️ Memory and session content IS transmitted to an LLM (
trace_extractor.py): extraction sends an excerpt of daily notes (and, with--session-file, session transcript text) to a language model. Primary transport is Ollama cloud (https://ollama.com,POST /api/chat) whenOLLAMA_API_KEYis configured — content leaves this machine. Local fallback is Ollama at127.0.0.1:11434. SetTRACE_LLM_LOCAL_ONLY=1to refuse every cloud call and force local-only. The destination is printed on each run ([Security] ⚠️ CLOUD TRANSMISSION: …). - ⚠️ PII sanitization is best-effort, not a guarantee:
sanitize_pii()removes API keys, tokens, JWTs, emails, passwords, PEM keys, French phone numbers and long opaque blobs before any LLM submission — but regex scrubbing cannot catch every secret format. The transport decision is the primary control, not the filter. - ⚠️ OLLAMA_URL should stay localhost: local LLM calls (cluster summaries, embeddings) send memory text to Ollama. Keep
OLLAMA_URL=http://localhost:11434to prevent data from leaving the machine. - ⚠️ Subprocess and urlopen are intentional local calls: Scripts use
subprocess.runto call other local Python scripts (trace_extractor, locomo_test) andurllib.request.urlopento call the local Ollama HTTP API. These are intentional local-only calls. KeepOLLAMA_URLon localhost to prevent data from leaving the machine. - ⚠️ Memory files may contain sensitive data: Review all files before indexing with hybrid search. The
scoring.pyscript skips files matching secret patterns (.secrets/,*.env,credentials*,*token*,*password*,.git/). - ⚠️ Hybrid search consent warnings:
hybrid_search.pyindexcommand displays a consent warning before batch embedding. Use--yesto skip in automation.addcommand prints a one-line embedding notice (use--quietto suppress). - ⚠️
memory-health.pyis READ-ONLY by default: No files or charts are written to disk without--output-dir <path>. SVG trend charts and JSON reports require this flag. - ⚠️ Scope confinement: All scripts restrict file scanning to the designated memory directory (
WORKSPACE/memory/), plus an explicit allowlist of three root config files that the indexer legitimately reads (MEMORY.md,TOOLS.md, the skill's ownSKILL.md). No parent traversal (../) and no sibling skill enumeration (skills/*/SKILL.md). Paths are validated withPath.resolve().is_relative_to(WORKSPACE). - ⚠️
--session-fileis a deliberate, explicit exception:trace_extractor.py --session-file <path>accepts one absolute path outside the workspace, because a session transcript does not live undermemory/. It is never scanned automatically — no global session directory walk exists. Only pass paths you own and accept sending to the configured LLM transport. - ⚠️ Subprocess calls use fixed argument lists: All
subprocess.runcalls use hardcoded[sys.executable, ...]argument lists — no environment variable injection possible. Script paths are validated against workspace confinement. - ⚠️ No PII in test fixtures:
run_tests.pyuses anonymized query terms (project_alpha,sample_note_01) — no real project names, personal names, or sensitive references. - ⚠️
--fixmode is destructive:memory-health.py --fixmoves daily notes to archive/ and rewrites ontology. Requires interactive confirmation or--forceflag. Creates timestamped backups inmemory/backup/before modifying.
Security & audit triage: see
docs/SECURITY-AUDIT-NOTES.mdfor which scanner findings are closed in code and which are accepted false positives.
Related skills
- Downloads
- 1.4kAll time
- Security audit
- Review
- Last updated
- 1mo ago
- Current version
- v4.0.6
- License
- MIT-0
