feat(web): fetch-and-index with FTS5 retrieval layer #159

Background

Current web filter defuddles a URL once, returns the markdown, discards it. Today's weak-filter audit shows curl leaking 833K tokens at 11% savings — much of that is URL fetches of stable doc content that gets re-fetched every session.

Problem

• Repeated web https://docs.example.com/page on the same URL re-fetches and re-defuddles every time
• No way to retrieve fragments of a previously-fetched page without re-fetching
• Large defuddled outputs still dump into context even when only a section is wanted

Proposed Design (clean-room)

Three additions, each independently useful:

1. Index on fetch

contextcrawler web <url> writes the defuddled markdown into a new web_cache table (separate from history.db, at ~/Library/Application Support/rtk/web_cache.db on macOS):

CREATE TABLE web_cache (
    id INTEGER PRIMARY KEY,
    url TEXT UNIQUE,
    fetched_at TEXT,
    markdown TEXT,
    content_hash TEXT
);
CREATE VIRTUAL TABLE web_search USING fts5(
    url UNINDEXED,
    body,
    tokenize = 'porter'
);

2. TTL cache (24h default)

Repeat web <url> calls within TTL skip the network and return a one-line cache hint:

[cache hit] docs.example.com/api — last fetched 2h ago, 8 sections indexed.
Run `contextcrawler search "<terms>" --in <url>` to retrieve.

--force bypasses cache.

3. New contextcrawler search

contextcrawler search "authentication tokens"
contextcrawler search "rate limit" --in https://docs.example.com/api

FTS5 BM25 across the cache. Returns top-N matching snippets with surrounding context. No re-fetch.

Why now

• The curl filter audit (PR feat(gain): release-boundary slicing for --weak-filters #150 weak-filters output) shows 833K tokens leaked on URL fetches at 11% savings — much of that is repeat-fetch of stable doc content
• Substrate for feat(session): cut-down session continuity — snapshot + resume #160 (session continuity cut-down) — same FTS5 infra can host session events later
• SQLite FTS5 is built-in, no new deps

Effort

M (4-6 days). Schema, FTS5 setup, chunker (split markdown by headings, code blocks intact), cache logic with 24h TTL + 14-day cleanup, search subcommand, snapshot tests.

Out of scope (v1)

• Trigram tokeniser (defer until Porter shows insufficient on real fixtures)
• Levenshtein typo correction (defer)
• Multi-URL parallel fetch / RRF reranking (defer)
• Pre-existing read and grep filter integration (separate issue)