memex/docs/DESIGN-RATIONALE.md

Design Rationale — Signal & Noise

Why each part of this repo exists. This is the "why" document; the other docs are the "what" and "how."

Before implementing anything, the design was worked out interactively with Claude as a structured Signal & Noise analysis of Andrej Karpathy's original persistent-wiki pattern:

Interactive version: eric-turner.com/memex/signal-and-noise.html — tabs for pros/cons, vs RAG, use-case fits, signal breakdown, mitigations

Self-contained archive: artifacts/signal-and-noise.html — same content, works offline

The analysis walks through the pattern's seven genuine strengths, seven places where it needs an engineering layer to be practical, and the concrete extension for each. memex is the implementation of those extensions. If you want to understand why a component exists, the interactive version has the longer-form argument; this document is the condensed written version.

---

Where the pattern is genuinely strong

The analysis found seven strengths that hold up under scrutiny. This repo preserves all of them:

Strength	How this repo keeps it
Knowledge compounds over time	Every ingest adds to the existing wiki rather than restarting; conversation mining and URL harvesting continuously feed new material in
Zero maintenance burden on humans	Cron-driven harvest + hygiene; the only manual step is staging review, and that's fast because the AI already compiled the page
Token-efficient at personal scale	index.md fits in context; qmd kicks in only at 50+ articles; the wake-up briefing is ~200 tokens
Human-readable & auditable	Plain markdown everywhere; every cross-reference is visible; git history shows every change
Future-proof & portable	No vendor lock-in; you can point any agent at the same tree tomorrow
Self-healing via lint passes	wiki-hygiene.py runs quick checks daily and full (LLM) checks weekly
Path to fine-tuning	Wiki pages are high-quality synthetic training data once purified through hygiene

---

Where memex extends the pattern

Karpathy's gist is a concept pitch. He was explicit that he was sharing an "idea file" for others to build on, not publishing a working implementation. The analysis identified eight places where the core idea needs an engineering layer to become practical day-to-day. The first seven emerged from the original Signal & Noise review; the eighth (conversation distillation) surfaced after building the other layers and realizing that the conversations themselves were being mined, summarized, indexed, and scanned for URLs — but the knowledge inside them was never becoming wiki pages.

1. Claim freshness and reversibility

The gap: Unlike RAG — where a hallucination is ephemeral and the next query starts clean — an LLM-maintained wiki is stateful. If a claim is wrong at ingest time, it stays wrong until something corrects it. For the pattern to work long-term, claims need a way to earn trust over time and lose it when unused.

How memex extends it:

• confidence field — every page carries high/medium/low with decay based on last_verified. Wrong claims aren't treated as permanent — they age out visibly.
• Archive + restore — decayed pages get moved to archive/ where they're excluded from default search. If they get referenced again they're auto-restored with confidence: medium (never straight to high — they have to re-earn trust).
• Raw harvested material is immutable — raw/harvested/*.md files are the ground truth. Every compiled wiki page can be traced back to its source via the sources: frontmatter field.
• Full-mode contradiction detection — wiki-hygiene.py --full uses sonnet to find conflicting claims across pages. Report-only (humans decide which side wins).
• Staging review — automated content goes to staging/ first. Nothing enters the live wiki without human approval, so errors have two chances to get caught (AI compile + human review) before they become persistent.

2. Scalable search beyond the context window

The gap: The pattern works beautifully up to ~100 articles, where index.md still fits in context. Karpathy's own wiki was right at the ceiling. Past that point, the agent needs a real search layer — loading the full index stops being practical.

How memex extends it:

• qmd from day one — qmd (BM25 + vector + LLM re-ranking) is set up in the default configuration so the agent never has to load the full index. At 50+ pages, qmd search replaces cat index.md.
• Wing/room structural filtering — conversations are partitioned by project code (wing) and topic (room, via the topics: frontmatter). Retrieval is pre-narrowed to the relevant wing before search runs. This extends the effective ceiling because qmd works on a relevant subset, not the whole corpus.
• Hygiene full mode flags redundancy — duplicate detection auto-merges weaker pages into stronger ones, keeping the corpus lean.
• Archive excludes stale content — the wiki-archive collection has includeByDefault: false, so archived pages don't eat context until explicitly queried.

3. Traceable sources for every claim

The gap: In precision-sensitive domains (API specs, version constraints, legal records, medical protocols), LLM-generated content needs to be verifiable against a source. For the pattern to work in those contexts, every claim needs to trace back to something immutable.

How memex extends it:

• Staging workflow — every automated page goes through human review. For precision-critical content, that review IS the cross-check. The AI does the drafting; you verify.
• compilation_notes field — staging pages include the AI's own explanation of what it did and why. Makes review faster — you can spot-check the reasoning rather than re-reading the whole page.
• Immutable raw sources — every wiki claim traces back to a specific file in raw/harvested/ with a SHA-256 content_hash. Verification means comparing the claim to the source, not "trust the LLM."
• confidence: low for precision domains — the agent's instructions (via CLAUDE.md) tell it to flag low-confidence content when citing. Humans see the warning before acting.

Residual trade-off: For truly mission-critical data (legal, medical, compliance), no amount of automation replaces domain-expert review. If that's your use case, treat this repo as a drafting tool, not a canonical source.

4. Continuous feed without manual discipline

The gap: Community analysis of 120+ comments on Karpathy's gist converged on one clear finding: this is the #1 friction point. Most people who try the pattern get the folder structure right and still end up with a wiki that slowly becomes unreliable because they stop feeding it. Six-week half-life is typical.

How memex extends it (this is the biggest layer):

• Automation replaces human discipline — daily cron runs wiki-maintain.sh (harvest + hygiene + qmd reindex); weekly cron runs --full mode. You don't need to remember anything.
• Conversation mining is the feed — you don't need to curate sources manually. Every Claude Code session becomes potential ingest. The feed is automatic and continuous, as long as you're doing work.
• last_verified refreshes from conversation references — when the summarizer links a conversation to a wiki page via related:, the hygiene script picks that up and bumps last_verified. Pages stay fresh as long as they're still being discussed.
• Decay thresholds force attention — pages without refresh signals for 6/9/12 months get downgraded and eventually archived. The wiki self-trims.
• Hygiene reports — reports/hygiene-YYYY-MM-DD-needs-review.md flags the things that do need human judgment. Everything else is auto-fixed.

This is the single biggest layer memex adds. Nothing about it is exotic — it's a cron-scheduled pipeline that runs the scripts you'd otherwise have to remember to run. That's the whole trick.

5. Keeping the human engaged with their own knowledge

The gap: Hacker News critics pointed out that the bookkeeping Karpathy outsources — filing, cross-referencing, summarizing — is precisely where genuine understanding forms. If the LLM does all of it, you can end up with a comprehensive wiki you haven't internalized. For the pattern to be an actual memory aid and not a false one, the human needs touchpoints that keep them engaged.

How memex extends it:

• Staging review is a forcing function — you see every automated page before it lands. Even skimming forces engagement with the material.
• qmd query "..." for exploration — searching the wiki is an active process, not passive retrieval. You're asking questions, not pulling a file.
• The wake-up briefing — context/wake-up.md is a 200-token digest the agent reads at session start. You read it too (or the agent reads it to you) — ongoing re-exposure to your own knowledge base.

Caveat: memex is designed as augmentation, not replacement. It's most valuable when you engage with it actively — reading your own wake-up briefing, spot-checking promoted pages, noticing decay flags. If you only consult the wiki through the agent and never look at it yourself, you've outsourced the learning. That's a usage pattern choice, not an architecture problem.

6. Hybrid retrieval — structure and semantics

The gap: Explicit wikilinks catch direct topic references but miss semantic neighbors that use different wording. At scale, the pattern benefits from vector similarity to find cross-topic connections the human (or the LLM at ingest time) didn't think to link manually.

How memex extends it:

• qmd is hybrid (BM25 + vector) — not just keyword search. Vector similarity is built into the retrieval pipeline from day one.
• Structural navigation complements semantic search — project codes (wings) and topic frontmatter narrow the search space before the hybrid search runs. Structure + semantics is stronger than either alone.
• Missing cross-reference detection — full-mode hygiene asks the LLM to find pages that should link to each other but don't, then auto-adds them. This is the explicit-linking approach catching up to semantic retrieval over time.

Residual trade-off: At enterprise scale (millions of documents), a proper vector DB with specialized retrieval wins. This repo is for personal / small-team scale where the hybrid approach is sufficient.

7. Cross-machine collaboration

The gap: Karpathy's gist describes a single-user, single-machine setup. In practice, people work from multiple machines (laptop, workstation, server) and sometimes collaborate with small teams. The pattern needs a sync story that handles concurrent writes gracefully.

How memex extends it:

• Git-based sync with merge-union — concurrent writes on different machines auto-resolve because markdown is set to merge=union in .gitattributes. Both sides win.
• State file sync — .harvest-state.json and .hygiene-state.json are committed, so two machines running the same pipeline agree on what's already been processed instead of re-doing the work.
• Network boundary as access gate — the suggested deployment is over Tailscale or a VPN, so the network enforces who can reach the wiki at all. Simple and sufficient for personal/family/small-team use.

Explicit scope: memex is deliberately not enterprise knowledge management. No audit trails, no fine-grained permissions, no compliance story. If you need any of that, you need a different architecture. This is for the personal and small-team case where git + Tailscale is the right amount of rigor.

8. Closing the MemPalace loop — conversation distillation

The gap: The mining pipeline extracts Claude Code sessions into transcripts, classifies them by memory type (fact/discovery/preference/ advice/event/tooling), and tags them with topics. The URL harvester scans them for cited links. Hygiene refreshes last_verified on any wiki page that appears in a conversation's related: field. But none of those steps actually compile the knowledge inside the conversations themselves into wiki pages. A decision made in a session, a root cause found during debugging, a pattern spotted in review — these stay in the conversation summaries (searchable but not synthesized) until a human manually writes them up. That's the last piece of the MemPalace model that wasn't wired through: closet content was never becoming the source for the wiki proper.

How memex extends it:

• wiki-distill.py runs as Phase 1a of wiki-maintain.sh, before URL harvesting. The ordering is deliberate: conversation content should drive the page, and URL harvesting should only supplement what the conversations are already covering.
• Narrow today-filter with historical rollup — daily runs only look at topics appearing in TODAY's summarized conversations, but for each such topic the script pulls in ALL historical conversations sharing that topic. Processing scope stays small; LLM context stays wide. Old topics that resurface in new sessions automatically trigger a re-distillation of the full history on that topic.
• First-run bootstrap — the very first run uses a 7-day lookback to seed the state. After that, daily runs stay narrow.
• High-signal halls only — distill reads hall_facts, hall_discoveries, and hall_advice bullets. Skips hall_events (temporal, not knowledge), hall_preferences (user working style), and hall_tooling (often low-signal). These are the halls the MemPalace taxonomy treats as "canonical knowledge" vs "context."
• claude -p compile step — each topic group (topic + all matching conversations + their high-signal halls) is sent to claude -p with the current wiki index. The model decides whether to create a new page, update an existing one, emit both, or skip (topic not substantive enough or already well-covered).
• Staging output with distill provenance — new/updated pages land in staging/ with staged_by: wiki-distill, distill_topic, and distill_source_conversations frontmatter fields. Every page traces back to the exact conversations it was distilled from.
• State file .distill-state.json tracks processed conversations by content hash and topic set, so re-runs only process what actually changed. A conversation gets re-distilled if its body changes OR if it gains a new topic not seen at previous distill time.

Why this matters: Without distillation, the MemPalace integration was incomplete — the closet summaries existed, the structural metadata existed, qmd could search them, but knowledge discovered during work never escaped the conversation archive. You could find "we had a debugging session about X last month" but couldn't find "here's the canonical page on X that captures what we learned." This extension turns the MemPalace layer from a searchable archive into a proper ingest pipeline for the wiki.

Residual consideration: Summarization quality is now load-bearing. The distill step trusts the summarizer's classification of bullets into halls. If the summarizer puts a debugging dead-end in hall_discoveries, it may enter the wiki compilation pipeline. The MIN_BULLETS_PER_TOPIC filter (default 2) and the LLM's own substantiveness check (it can choose skip with a reason) together catch most noise, and the staging review catches the rest.

---

The biggest layer — active upkeep

The other six extensions are important, but this is the one that makes or breaks the pattern in practice. The community data is unambiguous:

• People who automate the lint schedule → wikis healthy at 6+ months
• People who rely on "I'll remember to lint" → wikis abandoned at 6 weeks

The entire automation layer of this repo exists to remove upkeep as a thing the human has to think about:

Cadence	Job	Purpose
Every 15 min	wiki-sync.sh	Commit/pull/push — cross-machine sync
Every 2 hours	wiki-sync.sh full	Full sync + qmd reindex
Every hour	mine-conversations.sh --extract-only	Capture new Claude Code sessions (no LLM)
Daily 2am	summarize-conversations.py --claude + index	Classify + summarize (LLM)
Daily 3am	wiki-maintain.sh	Distill + harvest + quick hygiene + reindex
Weekly Sun 4am	wiki-maintain.sh --hygiene-only --full	LLM-powered duplicate/contradiction/cross-ref detection

If you disable all of these, you get the same outcome as every abandoned wiki: six-week half-life. The scripts aren't optional convenience — they're the load-bearing automation that lets the pattern actually compound over months and years instead of requiring a disciplined human to keep it alive.

---

What was borrowed from where

This repo is a synthesis of two ideas with an automation layer on top:

From Karpathy

• The core pattern: LLM-maintained persistent wiki, compile at ingest time instead of retrieve at query time
• Separation of raw/ (immutable sources) from wiki/ (compiled pages)
• CLAUDE.md as the schema that disciplines the agent
• Periodic "lint" passes to catch orphans, contradictions, missing refs
• The idea that the wiki becomes fine-tuning material over time

From mempalace

• Wings = per-person or per-project namespaces → this repo uses project codes (mc, wiki, web, etc.) as the same thing in conversations/<project>/
• Rooms = topics within a wing → the topics: frontmatter on conversation files
• Halls = memory-type corridors (fact / event / discovery / preference / advice / tooling) → the halls: frontmatter field classified by the summarizer
• Closets = summary layer → the summary body of each summarized conversation
• Drawers = verbatim archive, never lost → the extracted conversation transcripts under conversations/<project>/*.md
• Tunnels = cross-wing connections → the related: frontmatter linking conversations to wiki pages
• Wing + room structural filtering gives a documented +34% retrieval boost over flat search

The MemPalace taxonomy solved a problem Karpathy's pattern doesn't address: how do you navigate a growing corpus without reading everything? The answer is to give the corpus structural metadata at ingest time, then filter on that metadata before doing semantic search. This repo borrows that wholesale.

What this repo adds

• Automation layer tying the pieces together with cron-friendly orchestration
• Staging pipeline as a human-in-the-loop checkpoint for automated content
• Confidence decay + auto-archive + auto-restore as the "retention curve" that community analysis identified as critical for long-term wiki health
• qmd integration as the scalable search layer (chosen over ChromaDB because it uses the same markdown storage as the wiki — one index to maintain, not two)
• Hygiene reports with fixed vs needs-review separation so automation handles mechanical fixes and humans handle ambiguity
• Cross-machine sync via git with markdown merge-union so the same wiki lives on multiple machines without merge hell

---

What memex deliberately doesn't try to do

Five things memex is explicitly scoped around — not because they're unsolvable, but because solving them well requires a different kind of architecture than a personal/small-team wiki. If any of these are dealbreakers for your use case, memex is probably not the right fit:

1. Enterprise scale — millions of documents, hundreds of users, RBAC, compliance: these need real enterprise knowledge management infrastructure. memex is tuned for personal and small-team use.
2. True semantic retrieval at massive scale — qmd hybrid search works great up to thousands of pages. At millions, a dedicated vector database with specialized retrieval wins.
3. Replacing your own learning — memex is an augmentation layer, not a substitute for reading. Used well, it's a memory aid; used as a bypass, it just lets you forget more.
4. Precision-critical source of truth — for legal, medical, or regulatory data, memex is a drafting tool. Human domain-expert review still owns the final call.
5. Access control — the network boundary (Tailscale) is the fastest path to "only authorized people can reach it." memex itself doesn't enforce permissions inside that boundary.

These are scope decisions, not unfinished work. memex is the best personal/small-team answer to Karpathy's pattern I could build; it's not trying to be every answer.

---

Further reading

• The original Karpathy gist — the concept
• mempalace — the structural memory layer
• Signal & Noise interactive analysis — the design rationale this document summarizes (live interactive version)
• artifacts/signal-and-noise.html — self-contained archive of the same analysis, works offline
• README — the concept pitch
• ARCHITECTURE.md — component deep-dive
• SETUP.md — installation
• CUSTOMIZE.md — adapting for non-Claude-Code setups