memex/docs/ARCHITECTURE.md

# Architecture

Eleven scripts across three conceptual layers. This document walks through
what each one does, how they talk to each other, and where the seams are
for customization.

> **See also**: [`DESIGN-RATIONALE.md`](DESIGN-RATIONALE.md) — the *why*
> behind each component, with links to the interactive design artifact.

## Borrowed concepts

The architecture is a synthesis of two external ideas with an automation
layer on top. The terminology often maps 1:1, so it's worth calling out
which concepts came from where:

### From Karpathy's persistent-wiki gist

| Concept | How this repo implements it |
|---------|-----------------------------|
| Immutable `raw/` sources | `raw/` directory — never modified by the agent |
| LLM-compiled `wiki/` pages | `patterns/` `decisions/` `concepts/` `environments/` |
| Schema file disciplining the agent | `CLAUDE.md` at the wiki root |
| Periodic "lint" passes | `wiki-hygiene.py --quick` (daily) + `--full` (weekly) |
| Wiki as fine-tuning material | Clean markdown body is ready for synthetic training data |

### From [mempalace](https://github.com/milla-jovovich/mempalace)

MemPalace gave us the structural memory taxonomy that turns a flat
corpus into something you can navigate without reading everything. The
concepts map directly:

| MemPalace term | Meaning | How this repo implements it |
|----------------|---------|-----------------------------|
| **Wing** | Per-person or per-project namespace | Project code in `conversations/<code>/` (set by `PROJECT_MAP` in `extract-sessions.py`) |
| **Room** | Topic within a wing | `topics:` frontmatter field on summarized conversation files |
| **Closet** | Summary layer — high-signal compressed knowledge | The summary body written by `summarize-conversations.py --claude` |
| **Drawer** | Verbatim archive, never lost | The extracted transcript under `conversations/<wing>/*.md` (before summarization) |
| **Hall** | Memory-type corridor (fact / event / discovery / preference / advice / tooling) | `halls:` frontmatter field classified by the summarizer |
| **Tunnel** | Cross-wing connection — same topic in multiple projects | `related:` frontmatter linking conversations to wiki pages and to each other |

The key benefit of wing + room filtering is documented in MemPalace's
benchmarks as a **+34% retrieval boost** over flat search — because
`qmd` can search a pre-narrowed subset of the corpus instead of
everything. This is why the wiki scales past the Karpathy-pattern's
~50K token ceiling without needing a full vector DB rebuild.

### What this repo adds

Automation + lifecycle management on top of both:

- **Automation layer** — cron-friendly orchestration via `wiki-maintain.sh`
- **Staging pipeline** — human-in-the-loop checkpoint for automated content
- **Confidence decay + auto-archive + auto-restore** — the retention curve
- **`qmd` integration** — the scalable search layer (chosen over ChromaDB
  because it uses markdown storage like the wiki itself)
- **Hygiene reports** — fixed vs needs-review separation
- **Cross-machine sync** — git with markdown merge-union

---

## Overview

```
     ┌─────────────────────────────────┐
     │       SYNC LAYER                │
     │  wiki-sync.sh                   │  (git commit/pull/push, qmd reindex)
     └─────────────────────────────────┘
                     │
     ┌─────────────────────────────────┐
     │       MINING LAYER              │
     │  extract-sessions.py            │  (Claude Code JSONL → markdown)
     │  summarize-conversations.py     │  (LLM classify + summarize)
     │  update-conversation-index.py   │  (regenerate indexes + wake-up)
     │  mine-conversations.sh          │  (orchestrator)
     └─────────────────────────────────┘
                     │
     ┌─────────────────────────────────┐
     │    AUTOMATION LAYER             │
     │  wiki_lib.py  (shared helpers)  │
     │  wiki-distill.py                │  (conversations → staging) ← closes MemPalace loop
     │  wiki-harvest.py                │  (URL → raw → staging)
     │  wiki-staging.py                │  (human review)
     │  wiki-hygiene.py                │  (decay, archive, repair, checks)
     │  wiki-maintain.sh               │  (orchestrator)
     └─────────────────────────────────┘
```

Each layer is independent — you can run the mining layer without the
automation layer, or vice versa. The layers communicate through files on
disk (conversation markdown, raw harvested pages, staging pages, wiki
pages), never through in-memory state.

---

## Mining layer

### `extract-sessions.py`

Parses Claude Code JSONL session files from `~/.claude/projects/` into
clean markdown transcripts under `conversations/<project-code>/`.
Deterministic, no LLM calls. Incremental — tracks byte offsets in
`.mine-state.json` so it safely re-runs on partially-processed sessions.

Key features:
- Summarizes tool calls intelligently: full output for `Bash` and `Skill`,
  paths-only for `Read`/`Glob`/`Grep`, path + summary for `Edit`/`Write`
- Caps Bash output at 200 lines to prevent transcript bloat
- Handles session resumption — if a session has grown since last extraction,
  it appends new messages without re-processing old ones
- Maps Claude project directory names to short wiki codes via `PROJECT_MAP`

### `summarize-conversations.py`

Sends extracted transcripts to an LLM for classification and summarization.
Supports two backends:

1. **`--claude` mode** (recommended): Uses `claude -p` with
   haiku for short sessions (≤200 messages) and sonnet for longer ones.
   Runs chunked over long transcripts, keeping a rolling context window.

2. **Local LLM mode** (default, omit `--claude`): Uses a local
   `llama-server` instance at `localhost:8080` (or WSL gateway:8081 on
   Windows Subsystem for Linux). Requires llama.cpp installed and a GGUF
   model loaded.

Output: adds frontmatter to each conversation file — `topics`, `halls`
(fact/discovery/preference/advice/event/tooling), and `related` wiki
page links. The `related` links are load-bearing: they're what
`wiki-hygiene.py` uses to refresh `last_verified` on pages that are still
being discussed.

### `update-conversation-index.py`

Regenerates three files from the summarized conversations:

- `conversations/index.md` — catalog of all conversations grouped by project
- `context/wake-up.md` — a ~200-token briefing the agent loads at the start
  of every session ("current focus areas, recent decisions, active
  concerns")
- `context/active-concerns.md` — longer-form current state

The wake-up file is important: it's what gives the agent *continuity*
across sessions without forcing you to re-explain context every time.

### `mine-conversations.sh`

Orchestrator chaining extract → summarize → index. Supports
`--extract-only`, `--summarize-only`, `--index-only`, `--project <code>`,
and `--dry-run`.

---

## Automation layer

### `wiki_lib.py`

The shared library. Everything in the automation layer imports from here.
Provides:

- `WikiPage` dataclass — path + frontmatter + body + raw YAML
- `parse_page(path)` — safe markdown parser with YAML frontmatter
- `parse_yaml_lite(text)` — subset YAML parser (no external deps, handles
  the frontmatter patterns we use)
- `serialize_frontmatter(fm)` — writes YAML back in canonical key order
- `write_page(page, ...)` — full round-trip writer
- `page_content_hash(page)` — body-only SHA-256 for change detection
- `iter_live_pages()` / `iter_staging_pages()` / `iter_archived_pages()`
- Shared constants: `WIKI_DIR`, `STAGING_DIR`, `ARCHIVE_DIR`, etc.

All paths honor the `WIKI_DIR` environment variable, so tests and
alternate installs can override the root.

### `wiki-distill.py`

**Closes the MemPalace loop.** Reads the *content* of summarized
conversations — not the URLs they cite — and compiles wiki pages from
the high-signal hall entries (`hall_facts`, `hall_discoveries`,
`hall_advice`). Runs as Phase 1a in `wiki-maintain.sh`, before URL
harvesting.

**Scope filter (deliberately narrow)**:
1. Find all summarized conversations dated TODAY
2. Extract their `topics:` — this is the "topics-of-today" set
3. For each topic in that set, pull ALL summarized conversations across
   history that share that topic (full historical context via rollup)
4. Extract `hall_facts` + `hall_discoveries` + `hall_advice` bullet
   content from each conversation's body
5. Send the topic group (topic + matching conversations + halls) to
   `claude -p` with the current `index.md`
6. Model emits a JSON `actions` array with `new_page` / `update_page` /
   `skip` verdicts; the script writes each to `staging/<type>/`

**First-run bootstrap**: the very first run uses a 7-day lookback
instead of today-only, so the state file gets seeded with a reasonable
starting set. After that, daily runs stay narrow.

**Self-triggering**: dormant topics that resurface in a new
conversation automatically pull in all historical conversations on
that topic via the rollup. No manual intervention needed to
reprocess old knowledge when it becomes relevant again.

**Model routing**: haiku for short topic groups (< 15K chars prompt,
< 20 bullets), sonnet for longer ones.

**State** lives in `.distill-state.json` — tracks processed
conversations by content hash and topics-at-distill-time. A
conversation is re-processed if its body changes OR if it gains a new
topic not seen at previous distill.

**Staging output** includes distill-specific frontmatter:
- `staged_by: wiki-distill`
- `distill_topic: <topic>`
- `distill_source_conversations: <comma-separated conversation paths>`

Commands:
- `wiki-distill.py` — today-only rollup (default mode after first run)
- `wiki-distill.py --first-run` — 7-day lookback bootstrap
- `wiki-distill.py --topic TOPIC` — explicit single-topic processing
- `wiki-distill.py --project WING` — only today-topics from this wing
- `wiki-distill.py --dry-run` — plan only, no LLM calls, no writes
- `wiki-distill.py --no-compile` — rollup only, skip claude -p step
- `wiki-distill.py --limit N` — stop after N topic groups

### `wiki-harvest.py`

Scans summarized conversations for HTTP(S) URLs, classifies them,
fetches content, and compiles pending wiki pages. Runs as Phase 1b in
`wiki-maintain.sh`, after distill — URL content is treated as a
supplement to conversation-driven knowledge, not the primary source.

URL classification:
- **Harvest** (Type A/B) — docs, articles, blogs → fetch and compile
- **Check** (Type C) — GitHub issues, Stack Overflow — only harvest if
  the topic is already covered in the wiki (to avoid noise)
- **Skip** (Type D) — internal domains, localhost, private IPs, chat tools

Fetch cascade (tries in order, validates at each step):
1. `trafilatura -u <url> --markdown --no-comments --precision`
2. `crwl <url> -o markdown-fit`
3. `crwl <url> -o markdown-fit -b "user_agent_mode=random" -c "magic=true"` (stealth)
4. Conversation-transcript fallback — pull inline content from where the
   URL was mentioned during the session

Validated content goes to `raw/harvested/<domain>-<path>.md` with
frontmatter recording source URL, fetch method, and a content hash.

Compilation step: sends the raw content + `index.md` + conversation
context to `claude -p`, asking for a JSON verdict:
- `new_page` — create a new wiki page
- `update_page` — update an existing page (with `modifies:` field)
- `both` — do both
- `skip` — content isn't substantive enough

Result lands in `staging/<type>/` with `origin: automated`,
`status: pending`, and all the staging-specific frontmatter that gets
stripped on promotion.

### `wiki-staging.py`

Pure file operations — no LLM calls. Human review pipeline for automated
content.

Commands:
- `--list` / `--list --json` — pending items with metadata
- `--stats` — counts by type/source + age stats
- `--review` — interactive a/r/s/q loop with preview
- `--promote <path>` — approve, strip staging fields, move to live, update
  main index, rewrite cross-refs, preserve `origin: automated` as audit trail
- `--reject <path> --reason "..."` — delete, record in
  `.harvest-state.json` rejected_urls so the harvester won't re-create
- `--promote-all` — bulk approve everything
- `--sync` — regenerate `staging/index.md`, detect drift

### `wiki-hygiene.py`

The heavy lifter. Two modes:

**Quick mode** (no LLM, ~1 second on a 100-page wiki, run daily):
- Backfill `last_verified` from `last_compiled`/git/mtime
- Refresh `last_verified` from conversation `related:` links — this is
  the "something's still being discussed" signal
- Auto-restore archived pages that are referenced again
- Repair frontmatter (missing/invalid fields get sensible defaults)
- Apply confidence decay per thresholds (6/9/12 months)
- Archive stale and superseded pages
- Detect index drift (pages on disk not in index, stale index entries)
- Detect orphan pages (no inbound links) and auto-add them to index
- Detect broken cross-references, fuzzy-match to the intended target
  via `difflib.get_close_matches`, fix in place
- Report empty stubs (body < 100 chars)
- Detect state file drift (references to missing files)
- Regenerate `staging/index.md` and `archive/index.md` if out of sync

**Full mode** (LLM-powered, run weekly — extends quick mode with):
- Missing cross-references (haiku, batched 5 pages per call)
- Duplicate coverage (sonnet — weaker merged into stronger, auto-archives
  the loser with `archived_reason: Merged into <winner>`)
- Contradictions (sonnet, **report-only** — the human decides)
- Technology lifecycle (regex + conversation comparison — flags pages
  mentioning `Node 18` when recent conversations are using `Node 20`)

State lives in `.hygiene-state.json` — tracks content hashes per page so
full-mode runs can skip unchanged pages. Reports land in
`reports/hygiene-YYYY-MM-DD-{fixed,needs-review}.md`.

### `wiki-maintain.sh`

Top-level orchestrator:

```
Phase 1a: wiki-distill.py    (unless --no-distill or --harvest-only / --hygiene-only)
Phase 1b: wiki-harvest.py    (unless --distill-only / --hygiene-only)
Phase 2:  wiki-hygiene.py    (--full for the weekly pass, else quick)
Phase 3:  qmd update && qmd embed     (unless --no-reindex or --dry-run)
```

Ordering is deliberate: distill runs before harvest so that
conversation content drives the page shape, and URL harvesting only
supplements what the conversations are already covering. Flags pass
through to child scripts. Error-tolerant: if one phase fails, the
others still run. Logs to `scripts/.maintain.log`.

---

## Sync layer

### `wiki-sync.sh`

Git-based sync for cross-machine use. Commands:

- `--commit` — stage and commit local changes
- `--pull` — `git pull` with markdown merge-union (keeps both sides on conflict)
- `--push` — push to origin
- `full` — commit + pull + push + qmd reindex
- `--status` — read-only sync state report

The `.gitattributes` file sets `*.md merge=union` so markdown conflicts
auto-resolve by keeping both versions. This works because most conflicts
are additive (two machines both adding new entries).

---

## State files

Three JSON files track per-pipeline state:

| File | Owner | Synced? | Purpose |
|------|-------|---------|---------|
| `.mine-state.json` | `extract-sessions.py`, `summarize-conversations.py` | No (gitignored) | Per-session byte offsets — local filesystem state, not portable |
| `.distill-state.json` | `wiki-distill.py` | Yes (committed) | Processed conversations (content hash + topics seen), rejected topics, first-run flag |
| `.harvest-state.json` | `wiki-harvest.py` | Yes (committed) | URL dedup — harvested/skipped/failed/rejected URLs |
| `.hygiene-state.json` | `wiki-hygiene.py` | Yes (committed) | Page content hashes, deferred issues, last-run timestamps |

Harvest and hygiene state need to sync across machines so both
installations agree on what's been processed. Mining state is per-machine
because Claude Code session files live at OS-specific paths.

---

## Module dependency graph

```
wiki_lib.py  ─┬─>  wiki-distill.py
              ├─>  wiki-harvest.py
              ├─>  wiki-staging.py
              └─>  wiki-hygiene.py

wiki-maintain.sh  ─>  wiki-distill.py   (Phase 1a — conversations → staging)
                  ─>  wiki-harvest.py   (Phase 1b — URLs → staging)
                  ─>  wiki-hygiene.py   (Phase 2)
                  ─>  qmd (external)    (Phase 3)

mine-conversations.sh  ─>  extract-sessions.py
                       ─>  summarize-conversations.py
                       ─>  update-conversation-index.py

extract-sessions.py     (standalone — reads Claude JSONL)
summarize-conversations.py  ─>  claude CLI (or llama-server)
update-conversation-index.py  ─>  qmd (external)
```

`wiki_lib.py` is the only shared Python module — everything else is
self-contained within its layer.

---

## Extension seams

The places to modify when customizing:

1. **`scripts/extract-sessions.py`** — `PROJECT_MAP` controls how Claude
   project directories become wiki "wings". Also `KEEP_FULL_OUTPUT_TOOLS`,
   `SUMMARIZE_TOOLS`, `MAX_BASH_OUTPUT_LINES` to tune transcript shape.

2. **`scripts/update-conversation-index.py`** — `PROJECT_NAMES` and
   `PROJECT_ORDER` control how the index groups conversations.

3. **`scripts/wiki-harvest.py`** —
   - `SKIP_DOMAIN_PATTERNS` — your internal domains
   - `C_TYPE_URL_PATTERNS` — URL shapes that need topic-match before harvesting
   - `FETCH_DELAY_SECONDS` — rate limit between fetches
   - `COMPILE_PROMPT_TEMPLATE` — what the AI compile step tells the LLM
   - `SONNET_CONTENT_THRESHOLD` — size cutoff for haiku vs sonnet

4. **`scripts/wiki-hygiene.py`** —
   - `DECAY_HIGH_TO_MEDIUM` / `DECAY_MEDIUM_TO_LOW` / `DECAY_LOW_TO_STALE`
     — decay thresholds in days
   - `EMPTY_STUB_THRESHOLD` — what counts as a stub
   - `VERSION_REGEX` — which tools/runtimes to track for lifecycle checks
   - `REQUIRED_FIELDS` — frontmatter fields the repair step enforces

5. **`scripts/summarize-conversations.py`** —
   - `CLAUDE_LONG_THRESHOLD` — haiku/sonnet routing cutoff
   - `MINE_PROMPT_FILE` — the LLM system prompt for summarization
   - Backend selection (claude vs llama-server)

6. **`CLAUDE.md`** at the wiki root — the instructions the agent reads
   every session. This is where you tell the agent how to maintain the
   wiki, what conventions to follow, when to flag things to you.

See [`docs/CUSTOMIZE.md`](CUSTOMIZE.md) for recipes.