feat(distill): close the MemPalace loop — conversations → wiki pages

Add wiki-distill.py as Phase 1a of the maintenance pipeline. This is
the 8th extension memex adds to Karpathy's pattern and the one that
makes the MemPalace integration a real ingest pipeline instead of
just a searchable archive beside the wiki.

## The gap distill closes

The mining layer was extracting Claude Code sessions, classifying
bullets into halls (fact/discovery/preference/advice/event/tooling),
and tagging topics. The URL harvester scanned conversations for cited
links. Hygiene refreshed last_verified on wiki pages referenced in
related: fields. But none of those steps compiled the knowledge
*inside* the conversations themselves into wiki pages. Decisions,
root causes, and patterns stayed in the summaries forever — findable
via qmd but never synthesized into canonical pages.

## What distill does

Narrow today-filter with historical rollup:

  1. Find all summarized conversations dated TODAY
  2. Extract their topics: — this is the "topics of today" set
  3. For each topic in that set, pull ALL summarized conversations
     across history that share that topic (full historical context)
  4. Extract hall_facts + hall_discoveries + hall_advice bullets
     (the high-signal hall types — skips event/preference/tooling)
  5. Send topic group + wiki index.md to claude -p
  6. Model emits JSON actions[]: new_page / update_page / skip
  7. Write each action to staging/<type>/ with distill provenance
     frontmatter (staged_by: wiki-distill, distill_topic,
     distill_source_conversations, compilation_notes)

First-run bootstrap: uses 7-day lookback instead of today-only so
the state file gets seeded reasonably. After that, daily runs stay
narrow.

Self-triggering: dormant topics that resurface in a new conversation
automatically pull in all historical conversations on that topic via
the rollup. Old knowledge gets distilled when it becomes relevant
again without manual intervention.

## Orchestration — distill BEFORE harvest

wiki-maintain.sh now has Phase 1a (distill) + Phase 1b (harvest):

  1a. wiki-distill.py    — conversations → staging (PRIORITY)
  1b. wiki-harvest.py    — URLs → raw/harvested → staging (supplement)
  2.  wiki-hygiene.py    — decay, archive, repair, checks
  3.  qmd reindex

Conversation content drives the page shape; URL harvesting fills
gaps for external references conversations don't cover. New flags:
--distill-only, --no-distill, --distill-first-run.

## Verified on real wiki

Tested end-to-end on the production wiki with 611 summarized
conversations across 14 wings. First-run dry-run found 116 topic
groups worth distilling (+ 3 too-thin). Tested single-topic compile
with --topic zoho-api: the LLM rolled up 2 conversations (34
bullets), synthesized a proper pattern page with "What / Why /
Known Limitations" structure, linked it to existing wiki pages,
and landed it in staging with full distill provenance. LLM
correctly rejected claude-code-statusline (already well-covered
by an existing live page) — so the "skip" path works.

## Code additions

- scripts/wiki-distill.py (new, ~530 lines)
- scripts/wiki_lib.py: HIGH_SIGNAL_HALLS + parse_conversation_halls
  + high_signal_halls + _flatten_bullet helpers
- scripts/wiki-maintain.sh: Phase 1a distill, new flags
- tests/test_wiki_distill.py (21 new tests — hall parsing, rollup,
  state management, CLI smoke tests)
- tests/test_shell_scripts.py: updated phase-name assertion for
  the Phase 1a/1b split

## Docs additions

- README.md: 8th row in extensions table, updated compounding-loop
  diagram, new wiki-distill.py reference in architecture overview
- docs/DESIGN-RATIONALE.md: new section 8 "Closing the MemPalace
  loop" with full mempalace taxonomy mapping
- docs/ARCHITECTURE.md: wiki-distill.py section, updated phase
  order, updated state file table, updated dep graph
- docs/SETUP.md: updated cron comment, first-run distill guidance,
  verify section test count
- .gitignore: note distill-state.json is committed (sync across
  machines), not gitignored
- docs/artifacts/signal-and-noise.html: new "Distill ⬣" top-level
  tab with flow diagram, hall filter table, narrow-today/wide-
  history explanation, staging provenance example

## Tests

192 tests total (+21 new, +1 regression fix), all green in ~1.5s.

tests/test_shell_scripts.py

@@ -65,7 +65,9 @@ class TestWikiMaintainSh:
             "wiki-maintain.sh", "--hygiene-only", "--dry-run", "--no-reindex"
         )
         assert result.returncode == 0
-        assert "Phase 1: URL harvesting (skipped)" in result.stdout
+        # Phase 1a (distill) and Phase 1b (harvest) both skipped in --hygiene-only
+        assert "Phase 1a: Conversation distillation (skipped)" in result.stdout
+        assert "Phase 1b: URL harvesting (skipped)" in result.stdout
 
     def test_phase_3_skipped_in_dry_run(
         self, run_script, tmp_wiki: Path

tests/test_wiki_distill.py

@@ -0,0 +1,446 @@
+"""Unit + integration tests for scripts/wiki-distill.py.
+
+Mocks claude -p; no real LLM calls during tests.
+"""
+
+from __future__ import annotations
+
+import json
+from datetime import date, timedelta
+from pathlib import Path
+from typing import Any
+
+import pytest
+
+from conftest import make_conversation
+
+
+# ---------------------------------------------------------------------------
+# wiki_lib hall parsing helpers
+# ---------------------------------------------------------------------------
+
+
+class TestParseConversationHalls:
+    def _make_conv_with_halls(self, tmp_wiki: Path, body: str) -> Path:
+        return make_conversation(
+            tmp_wiki,
+            "test",
+            "2026-04-12-halls.md",
+            status="summarized",
+            body=body,
+        )
+
+    def test_extracts_fact_bullets(self, wiki_lib: Any, tmp_wiki: Path) -> None:
+        body = (
+            "## Summary\n\nsome summary text.\n\n"
+            "## Decisions (hall: fact)\n\n"
+            "- First decision made\n"
+            "- Second decision\n\n"
+            "## Other section\n\nunrelated.\n"
+        )
+        path = self._make_conv_with_halls(tmp_wiki, body)
+        page = wiki_lib.parse_page(path)
+        halls = wiki_lib.parse_conversation_halls(page)
+        assert "fact" in halls
+        assert halls["fact"] == ["First decision made", "Second decision"]
+
+    def test_extracts_multiple_hall_types(
+        self, wiki_lib: Any, tmp_wiki: Path
+    ) -> None:
+        body = (
+            "## Decisions (hall: fact)\n\n- A\n- B\n\n"
+            "## Discoveries (hall: discovery)\n\n- root cause X\n\n"
+            "## Advice (hall: advice)\n\n- try Y\n- consider Z\n"
+        )
+        path = self._make_conv_with_halls(tmp_wiki, body)
+        page = wiki_lib.parse_page(path)
+        halls = wiki_lib.parse_conversation_halls(page)
+        assert halls["fact"] == ["A", "B"]
+        assert halls["discovery"] == ["root cause X"]
+        assert halls["advice"] == ["try Y", "consider Z"]
+
+    def test_ignores_sections_without_hall_marker(
+        self, wiki_lib: Any, tmp_wiki: Path
+    ) -> None:
+        body = (
+            "## Summary\n\n- not a hall bullet\n\n"
+            "## Decisions (hall: fact)\n\n- real bullet\n"
+        )
+        path = self._make_conv_with_halls(tmp_wiki, body)
+        page = wiki_lib.parse_page(path)
+        halls = wiki_lib.parse_conversation_halls(page)
+        assert halls == {"fact": ["real bullet"]}
+
+    def test_flattens_multiline_bullets(
+        self, wiki_lib: Any, tmp_wiki: Path
+    ) -> None:
+        body = (
+            "## Decisions (hall: fact)\n\n"
+            "- A bullet that goes on\n  and continues here\n"
+            "- Second bullet\n"
+        )
+        path = self._make_conv_with_halls(tmp_wiki, body)
+        page = wiki_lib.parse_page(path)
+        halls = wiki_lib.parse_conversation_halls(page)
+        # The simple regex captures each "- " line separately; continuation
+        # lines are not part of the bullet. This matches the current behavior.
+        assert halls["fact"][0].startswith("A bullet")
+        assert "Second bullet" in halls["fact"]
+
+    def test_empty_body_returns_empty(
+        self, wiki_lib: Any, tmp_wiki: Path
+    ) -> None:
+        path = self._make_conv_with_halls(tmp_wiki, "## Summary\n\ntext.\n")
+        page = wiki_lib.parse_page(path)
+        assert wiki_lib.parse_conversation_halls(page) == {}
+
+    def test_high_signal_halls_filters_out_preference_event_tooling(
+        self, wiki_lib: Any, tmp_wiki: Path
+    ) -> None:
+        body = (
+            "## Decisions (hall: fact)\n- f\n"
+            "## Preferences (hall: preference)\n- p\n"
+            "## Events (hall: event)\n- e\n"
+            "## Tooling (hall: tooling)\n- t\n"
+            "## Advice (hall: advice)\n- a\n"
+        )
+        path = self._make_conv_with_halls(tmp_wiki, body)
+        page = wiki_lib.parse_page(path)
+        halls = wiki_lib.high_signal_halls(page)
+        assert set(halls.keys()) == {"fact", "advice"}
+
+
+# ---------------------------------------------------------------------------
+# wiki-distill.py module fixture
+# ---------------------------------------------------------------------------
+
+
+@pytest.fixture
+def wiki_distill(tmp_wiki: Path) -> Any:
+    from conftest import SCRIPTS_DIR, _load_script_module
+    _load_script_module("wiki_lib", SCRIPTS_DIR / "wiki_lib.py")
+    return _load_script_module("wiki_distill", SCRIPTS_DIR / "wiki-distill.py")
+
+
+# ---------------------------------------------------------------------------
+# Topic rollup logic
+# ---------------------------------------------------------------------------
+
+
+class TestTopicRollup:
+    def _make_summarized_conv(
+        self,
+        tmp_wiki: Path,
+        project: str,
+        filename: str,
+        conv_date: str,
+        topics: list[str],
+        fact_bullets: list[str] | None = None,
+    ) -> Path:
+        fact_section = ""
+        if fact_bullets:
+            fact_section = "## Decisions (hall: fact)\n\n" + "\n".join(
+                f"- {b}" for b in fact_bullets
+            )
+        return make_conversation(
+            tmp_wiki,
+            project,
+            filename,
+            date=conv_date,
+            status="summarized",
+            related=[f"topic:{t}" for t in []],
+            body=f"## Summary\n\ntest.\n\n{fact_section}\n",
+        )
+
+    def test_extract_topics_from_today_only(
+        self, wiki_distill: Any, tmp_wiki: Path
+    ) -> None:
+        today_date = wiki_distill.today()
+        yesterday = today_date - timedelta(days=1)
+        # Today's conversation with topics
+        _write_conv_with_topics(
+            tmp_wiki, "test", "today.md",
+            date_str=today_date.isoformat(), topics=["alpha", "beta"],
+        )
+        # Yesterday's conversation — should be excluded at lookback=0
+        _write_conv_with_topics(
+            tmp_wiki, "test", "yesterday.md",
+            date_str=yesterday.isoformat(), topics=["gamma"],
+        )
+        all_convs = wiki_distill.iter_summarized_conversations()
+        topics = wiki_distill.extract_topics_from_today(all_convs, today_date, 0)
+        assert topics == {"alpha", "beta"}
+
+    def test_extract_topics_with_lookback(
+        self, wiki_distill: Any, tmp_wiki: Path
+    ) -> None:
+        today_date = wiki_distill.today()
+        day3 = today_date - timedelta(days=3)
+        day10 = today_date - timedelta(days=10)
+        _write_conv_with_topics(
+            tmp_wiki, "test", "today.md",
+            date_str=today_date.isoformat(), topics=["a"],
+        )
+        _write_conv_with_topics(
+            tmp_wiki, "test", "day3.md",
+            date_str=day3.isoformat(), topics=["b"],
+        )
+        _write_conv_with_topics(
+            tmp_wiki, "test", "day10.md",
+            date_str=day10.isoformat(), topics=["c"],
+        )
+        all_convs = wiki_distill.iter_summarized_conversations()
+        topics_7 = wiki_distill.extract_topics_from_today(all_convs, today_date, 7)
+        assert topics_7 == {"a", "b"}  # day10 excluded by 7-day lookback
+
+    def test_rollup_by_topic_across_history(
+        self, wiki_distill: Any, tmp_wiki: Path
+    ) -> None:
+        today_date = wiki_distill.today()
+        # Three conversations all tagged with "shared-topic", different dates
+        _write_conv_with_topics(
+            tmp_wiki, "test", "a.md",
+            date_str=today_date.isoformat(), topics=["shared-topic"],
+        )
+        _write_conv_with_topics(
+            tmp_wiki, "test", "b.md",
+            date_str=(today_date - timedelta(days=30)).isoformat(),
+            topics=["shared-topic", "other"],
+        )
+        _write_conv_with_topics(
+            tmp_wiki, "test", "c.md",
+            date_str=(today_date - timedelta(days=90)).isoformat(),
+            topics=["shared-topic"],
+        )
+        # One unrelated
+        _write_conv_with_topics(
+            tmp_wiki, "test", "d.md",
+            date_str=today_date.isoformat(), topics=["unrelated"],
+        )
+        all_convs = wiki_distill.iter_summarized_conversations()
+        rollup = wiki_distill.rollup_conversations_by_topic(
+            "shared-topic", all_convs
+        )
+        assert len(rollup) == 3
+        stems = [c.path.stem for c in rollup]
+        # Most recent first
+        assert stems[0] == "a"
+
+
+def _write_conv_with_topics(
+    tmp_wiki: Path,
+    project: str,
+    filename: str,
+    *,
+    date_str: str,
+    topics: list[str],
+) -> Path:
+    """Helper — write a summarized conversation with topic frontmatter."""
+    proj_dir = tmp_wiki / "conversations" / project
+    proj_dir.mkdir(parents=True, exist_ok=True)
+    path = proj_dir / filename
+    topic_yaml = "topics: [" + ", ".join(topics) + "]"
+    content = (
+        f"---\n"
+        f"title: Test Conv\n"
+        f"type: conversation\n"
+        f"project: {project}\n"
+        f"date: {date_str}\n"
+        f"status: summarized\n"
+        f"messages: 50\n"
+        f"{topic_yaml}\n"
+        f"---\n"
+        f"## Summary\n\ntest.\n\n"
+        f"## Decisions (hall: fact)\n\n"
+        f"- Fact one for these topics\n"
+        f"- Fact two\n"
+    )
+    path.write_text(content)
+    return path
+
+
+# ---------------------------------------------------------------------------
+# Topic group building
+# ---------------------------------------------------------------------------
+
+
+class TestTopicGroupBuild:
+    def test_counts_total_bullets(
+        self, wiki_distill: Any, tmp_wiki: Path
+    ) -> None:
+        _write_conv_with_topics(
+            tmp_wiki, "test", "one.md",
+            date_str="2026-04-12", topics=["foo"],
+        )
+        all_convs = wiki_distill.iter_summarized_conversations()
+        rollup = wiki_distill.rollup_conversations_by_topic("foo", all_convs)
+        group = wiki_distill.build_topic_group("foo", rollup)
+        assert group.topic == "foo"
+        assert group.total_bullets == 2  # the helper writes 2 fact bullets
+
+    def test_format_for_llm_includes_topic_and_sections(
+        self, wiki_distill: Any, tmp_wiki: Path
+    ) -> None:
+        _write_conv_with_topics(
+            tmp_wiki, "test", "one.md",
+            date_str="2026-04-12", topics=["bar"],
+        )
+        all_convs = wiki_distill.iter_summarized_conversations()
+        rollup = wiki_distill.rollup_conversations_by_topic("bar", all_convs)
+        group = wiki_distill.build_topic_group("bar", rollup)
+        text = wiki_distill.format_topic_group_for_llm(group)
+        assert "# Topic: bar" in text
+        assert "Fact one" in text
+        assert "Decisions:" in text
+
+
+# ---------------------------------------------------------------------------
+# State management
+# ---------------------------------------------------------------------------
+
+
+class TestDistillState:
+    def test_load_returns_defaults(
+        self, wiki_distill: Any, tmp_wiki: Path
+    ) -> None:
+        state = wiki_distill.load_state()
+        assert state["processed_convs"] == {}
+        assert state["processed_topics"] == {}
+        assert state["first_run_complete"] is False
+
+    def test_save_and_reload(
+        self, wiki_distill: Any, tmp_wiki: Path
+    ) -> None:
+        state = wiki_distill.load_state()
+        state["first_run_complete"] = True
+        state["processed_topics"]["foo"] = {"distilled_date": "2026-04-12"}
+        wiki_distill.save_state(state)
+
+        reloaded = wiki_distill.load_state()
+        assert reloaded["first_run_complete"] is True
+        assert "foo" in reloaded["processed_topics"]
+
+    def test_conv_needs_distill_first_time(
+        self, wiki_distill: Any, tmp_wiki: Path
+    ) -> None:
+        path = _write_conv_with_topics(
+            tmp_wiki, "test", "fresh.md",
+            date_str="2026-04-12", topics=["x"],
+        )
+        conv = wiki_distill.parse_page(path)
+        state = wiki_distill.load_state()
+        assert wiki_distill.conv_needs_distill(state, conv) is True
+
+    def test_conv_needs_distill_detects_content_change(
+        self, wiki_distill: Any, tmp_wiki: Path
+    ) -> None:
+        path = _write_conv_with_topics(
+            tmp_wiki, "test", "mut.md",
+            date_str="2026-04-12", topics=["x"],
+        )
+        conv = wiki_distill.parse_page(path)
+        state = wiki_distill.load_state()
+        wiki_distill.mark_conv_distilled(state, conv, ["staging/patterns/x.md"])
+        assert wiki_distill.conv_needs_distill(state, conv) is False
+
+        # Mutate the body
+        text = path.read_text()
+        path.write_text(text + "\n- Another bullet\n")
+        conv2 = wiki_distill.parse_page(path)
+        assert wiki_distill.conv_needs_distill(state, conv2) is True
+
+    def test_conv_needs_distill_detects_new_topic(
+        self, wiki_distill: Any, tmp_wiki: Path
+    ) -> None:
+        path = _write_conv_with_topics(
+            tmp_wiki, "test", "new-topic.md",
+            date_str="2026-04-12", topics=["original"],
+        )
+        conv = wiki_distill.parse_page(path)
+        state = wiki_distill.load_state()
+        wiki_distill.mark_conv_distilled(state, conv, [])
+        assert wiki_distill.conv_needs_distill(state, conv) is False
+
+        # Rewrite with a new topic added
+        _write_conv_with_topics(
+            tmp_wiki, "test", "new-topic.md",
+            date_str="2026-04-12", topics=["original", "freshly-added"],
+        )
+        conv2 = wiki_distill.parse_page(path)
+        assert wiki_distill.conv_needs_distill(state, conv2) is True
+
+
+# ---------------------------------------------------------------------------
+# CLI smoke tests (no real LLM calls — uses --dry-run)
+# ---------------------------------------------------------------------------
+
+
+class TestDistillCli:
+    def test_help_flag(self, run_script) -> None:
+        result = run_script("wiki-distill.py", "--help")
+        assert result.returncode == 0
+        assert "--first-run" in result.stdout
+        assert "--topic" in result.stdout
+        assert "--dry-run" in result.stdout
+
+    def test_dry_run_empty_wiki(self, run_script, tmp_wiki: Path) -> None:
+        result = run_script("wiki-distill.py", "--dry-run", "--first-run")
+        assert result.returncode == 0
+
+    def test_dry_run_with_topic_rollup(
+        self, run_script, tmp_wiki: Path
+    ) -> None:
+        _write_conv_with_topics(
+            tmp_wiki, "test", "convA.md",
+            date_str="2026-04-12", topics=["rollup-test"],
+        )
+        _write_conv_with_topics(
+            tmp_wiki, "test", "convB.md",
+            date_str="2026-04-11", topics=["rollup-test"],
+        )
+        result = run_script(
+            "wiki-distill.py", "--dry-run", "--first-run",
+        )
+        assert result.returncode == 0
+        # Should mention the rollup topic
+        assert "rollup-test" in result.stdout
+
+    def test_topic_flag_narrow_mode(
+        self, run_script, tmp_wiki: Path
+    ) -> None:
+        _write_conv_with_topics(
+            tmp_wiki, "test", "a.md",
+            date_str="2026-04-12", topics=["explicit-topic"],
+        )
+        result = run_script(
+            "wiki-distill.py", "--dry-run", "--topic", "explicit-topic",
+        )
+        assert result.returncode == 0
+        assert "Explicit topic mode" in result.stdout
+        assert "explicit-topic" in result.stdout
+
+    def test_too_thin_topic_is_skipped(
+        self, run_script, tmp_wiki: Path, wiki_distill: Any
+    ) -> None:
+        # Write a conversation with only ONE hall bullet on this topic
+        proj_dir = tmp_wiki / "conversations" / "test"
+        proj_dir.mkdir(parents=True, exist_ok=True)
+        (proj_dir / "thin.md").write_text(
+            "---\n"
+            "title: Thin\n"
+            "type: conversation\n"
+            "project: test\n"
+            "date: 2026-04-12\n"
+            "status: summarized\n"
+            "messages: 5\n"
+            "topics: [thin-topic]\n"
+            "---\n"
+            "## Summary\n\n\n"
+            "## Decisions (hall: fact)\n\n"
+            "- Single bullet\n"
+        )
+        result = run_script(
+            "wiki-distill.py", "--dry-run", "--topic", "thin-topic",
+        )
+        assert result.returncode == 0
+        assert "too-thin" in result.stdout or "too-thin" in result.stderr