From 5ec2e143e549c113cc02dee96bcd5f07b15628b5 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 15 Aug 2026 06:54:29 -0400 Subject: [PATCH] docs(knowledge): make map-corpus self-contained instead of citing a pruned Brief MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The shipped skill cited its authoring-time planning document by label (`Q19`, "Brief criterion 2/6", "Brief acceptance criterion 4"). That document lived at docs/topics/docsite-digest/PLAN.md, which exists in no git ref — it was never committed, so no pointer the topic-docs lifecycle accepts can be built for it, and a consumer never had it at all. Each site now states what it needs: the deferred rung-3 decision carries its own fork (a presence-gated /firecrawl:firecrawl map seam versus a recorded reimplementation), its user-reserved arbiter, and its trigger; the deferred repo-tree enumeration rung carries its trigger; the `Q19` label is dropped from SKILL.md, discovery/link-map-format.md, discovery/check_linkmap.py and the eval set; and the whole-snapshot hash in extraction/node-manifest-format.md points at reference/citation-shape.md, which owns that fact. No behavior, schema, gate, exit code, or argument change. Co-authored-by: Claude Opus 5 --- plugins/knowledge/.claude-plugin/plugin.json | 2 +- plugins/knowledge/CHANGELOG.md | 15 +++++++++++++++ plugins/knowledge/skills/map-corpus/SKILL.md | 15 +++++++++------ .../skills/map-corpus/discovery/check_linkmap.py | 2 +- .../map-corpus/discovery/link-map-format.md | 16 +++++++++------- .../knowledge/skills/map-corpus/evals/evals.json | 4 ++-- .../extraction/node-manifest-format.md | 4 ++-- .../map-corpus/verification/inventory-format.md | 6 +++--- .../verification/test_check_inventory.py | 5 ++--- 9 files changed, 44 insertions(+), 25 deletions(-) diff --git a/plugins/knowledge/.claude-plugin/plugin.json b/plugins/knowledge/.claude-plugin/plugin.json index e68b84eb97..7ee410d6f4 100644 --- a/plugins/knowledge/.claude-plugin/plugin.json +++ b/plugins/knowledge/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", "name": "knowledge", - "version": "0.12.1", + "version": "0.12.2", "description": "Ingest external knowledge into durable, synthesized artifacts. Ships a book-distillation pipeline (PDF/EPUB into concept-organized, author-attributed skill reference files), a YouTube pipeline (watch, transcript, link harvest, and repo-applicability synthesis), a course-digest pipeline (extract and synthesize online video courses \u2014 Dometrain, Teachable \u2014 into repo-applicable recommendations), a docpage-digest pipeline (single online documentation page into a verified knowledge slice with dual verification \u2014 one cross-vendor verifier \u2014 and an interview handoff), and a map-corpus pipeline (multi-resource corpus into a classified link map, deterministic node manifests, gate-verified relevance inventory, and an approved queue of docpage-digest runs), plus a re-runnable setup action; a configurable library directory governs where synthesized artifacts land in the consuming repo.", "author": { "name": "Melodic Software", diff --git a/plugins/knowledge/CHANGELOG.md b/plugins/knowledge/CHANGELOG.md index e254c28d9f..f5b5ef14e2 100644 --- a/plugins/knowledge/CHANGELOG.md +++ b/plugins/knowledge/CHANGELOG.md @@ -4,6 +4,21 @@ All notable changes to the `knowledge` plugin are recorded here. The `version` i `.claude-plugin/plugin.json` is the delivery vehicle — a consumer receives a change only after that version increases. +## [0.12.2] + +### Changed + +- **`map-corpus` states its own deferred decisions instead of pointing outside itself.** The skill + cited an authoring-time planning document by label, which no consumer ever receives — an + unresolvable reference on a shipped surface. Each site is now self-contained: the deferred rung-3 + decision states its own fork (a presence-gated `/firecrawl:firecrawl map` seam versus a recorded + reimplementation), its user-reserved arbiter, and its trigger; the deferred repo-tree enumeration + rung states its trigger; the opaque `Q19` label is dropped from `SKILL.md`, + `discovery/link-map-format.md`, `discovery/check_linkmap.py`, and the eval set; and the + whole-snapshot hash in `extraction/node-manifest-format.md` now points at + `reference/citation-shape.md`, its actual owner. No behavior, schema, gate, exit code, or + argument changes. + ## [0.12.1] ### Added diff --git a/plugins/knowledge/skills/map-corpus/SKILL.md b/plugins/knowledge/skills/map-corpus/SKILL.md index 60802f0379..18b06dc0b3 100644 --- a/plugins/knowledge/skills/map-corpus/SKILL.md +++ b/plugins/knowledge/skills/map-corpus/SKILL.md @@ -89,13 +89,16 @@ Seeds come in two kinds, told apart by their normalized path: snapshot into `discovery/`: **rung 1** `/llms.txt`; **rung 2** the sitemap (`sitemap.xml`, a markdown variant such as `sitemap.md` when the site serves one, or the location robots.txt declares). An origin seed where **neither rung resolves → stop loudly**: - rung 3 (in-page link extraction) is deliberately absent, and its design — a presence-gated - `/firecrawl:firecrawl map` seam versus a recorded reimplementation — is a user-reserved - decision (Brief Q19) triggered by exactly this stop. + rung 3 (in-page link extraction) is deliberately absent. Its design is a **user-reserved** + decision — a presence-gated `/firecrawl:firecrawl map` seam with a documented in-skill fallback, + versus a recorded reason this skill reimplements in-page extraction (a bare unguarded + cross-plugin reference is barred, and `dependencies` are reserved for hard requires). This stop + IS that decision's trigger: report it and ask, never crawl unasked. - **Resource seed** (non-root path, e.g. a raw repo file URL) — itself a corpus resource: a link-map row with rung `seed`, no discovery at its origin. Human-enumerated resource seeds are - the V1 ingress for repository files (a repo-tree enumeration rung is deferred, recorded in the - Brief). **V1 requires at least one origin seed** — the link-map gate needs a discovery basis; + the V1 ingress for repository files; a repo-tree enumeration rung (`git ls-tree` / tree API) is + deferred, and its trigger is the first corpus whose repository half is too large to enumerate by + hand. **V1 requires at least one origin seed** — the link-map gate needs a discovery basis; a resource-seeds-ONLY corpus is outside V1 mapper scope (recorded deferral in `discovery/link-map-format.md`): route those URLs to direct `/knowledge:docpage-digest` runs. **GitHub blob URLs:** seed the `raw.githubusercontent.com` form — a `blob` URL snapshots the @@ -178,7 +181,7 @@ Emit a continuation prompt when pausing mid-pipeline (slug, first unticked phase ## What this skill does NOT do - **Does not digest.** Verdicts and evidence tokens, yes; digests are `docpage-digest`'s job. -- **Does not crawl in-page links.** Discovery is rungs 1–2; rung 3 is user-reserved (Q19). +- **Does not crawl in-page links.** Discovery is rungs 1–2; rung 3 is user-reserved (see Phase 1). - **Does not commit or graduate.** The slice is untracked and self-ignoring. - **Does not route non-web ingest types.** A YouTube/course/book URL discovered in the corpus is classified `companion` for the interview, never dispatched to sibling pipelines. diff --git a/plugins/knowledge/skills/map-corpus/discovery/check_linkmap.py b/plugins/knowledge/skills/map-corpus/discovery/check_linkmap.py index 5823bafee4..54756d9c96 100755 --- a/plugins/knowledge/skills/map-corpus/discovery/check_linkmap.py +++ b/plugins/knowledge/skills/map-corpus/discovery/check_linkmap.py @@ -118,7 +118,7 @@ def main(argv=None) -> int: fail(2, "at least one --discovery output is required: a link map with " "no discovery basis cannot demonstrate coverage. Two cases " "land here: (1) an origin seed resolved neither llms.txt nor " - "a sitemap -- stop loudly (rung 3 is deferred by Q19); " + "a sitemap -- stop loudly (rung 3 is deferred, user-reserved); " "(2) a corpus of resource seeds only -- outside V1 gate scope " "by recorded deferral (see link-map-format.md); route those " "URLs to direct docpage-digest runs instead.") diff --git a/plugins/knowledge/skills/map-corpus/discovery/link-map-format.md b/plugins/knowledge/skills/map-corpus/discovery/link-map-format.md index 9eb4f25c0e..858fb75425 100644 --- a/plugins/knowledge/skills/map-corpus/discovery/link-map-format.md +++ b/plugins/knowledge/skills/map-corpus/discovery/link-map-format.md @@ -14,9 +14,9 @@ of what it saw. An agent classifies; it does not get to choose what needs classi user says so), snapshot it into the slice under `discovery/`. 2. **Rung 2 — sitemap**: fetch the site's sitemap (`sitemap.xml`, `sitemap.md`, or a robots.txt-declared location), snapshot likewise. -3. **Rung 3 — in-page link extraction: NOT IN V1.** Deferred question Q19 (Brief): reaching it +3. **Rung 3 — in-page link extraction: NOT IN V1.** Deferred and USER-RESERVED: reaching it requires either a presence-gated `/firecrawl:firecrawl map` seam or a recorded reason to - reimplement — USER-RESERVED. The trigger is the first corpus whose seeds resolve neither an + reimplement. The trigger is the first corpus whose seeds resolve neither an `llms.txt` nor a sitemap. Until then a corpus with neither artifact stops loudly at discovery. Fetching is the skill's job (WebFetch/curl per the skill's own text, channel recorded); parsing @@ -91,7 +91,8 @@ nobody parsed. - `rows` — exactly one row per distinct URL across seeds + every discovery output. Each row: - `url` — normalized URL. - `rungs` — non-empty subset of `seed` | `llms-txt` | `sitemap-xml` | `sitemap-md`, the - provenance of every appearance. (`in-page` joins this enum only when Q19 is decided.) + provenance of every appearance. (`in-page` joins this enum only when the deferred rung-3 + decision above is made.) - `classification` — exactly one of: - `in-corpus` — fetched, snapshotted, node-extracted, inventoried, queued for digestion. - `companion` — same corpus context but a different ingest type (e.g. a repo, a video); @@ -111,12 +112,13 @@ same recorded deferral as the repo-tree enumeration rung). Checks: 1. All inputs parse (duplicate JSON keys rejected at any depth); unknown or missing keys rejected in the link map, its rows, AND each discovery output; schema literals exact; seeds and row URLs must be in normalized form. Failures name the file/row; exit 2 for unusable input. -2. **Classification coverage** (Brief criterion 2): every URL in every discovery output and every - seed has exactly one row; every row's URL traces back to at least one discovery output or the - seed list (no phantom rows); every row carries a valid classification and non-empty reason; +2. **Classification coverage** — this gate's reason to exist: every URL in every discovery output + and every seed has exactly one row; every row's URL traces back to at least one discovery + output or the seed list (no phantom rows); every row carries a valid classification and + non-empty reason; every row's `rungs` match where the URL actually appeared, exactly. 3. **Bounds**: `in-corpus` row count ≤ `bounds.max_resources`, else a named failure — the - bound-breach stop that forces the run back to the user (Brief criterion 6). + bound-breach stop that forces the run back to the user. 4. A clean run prints what it exercised (files, row/URL counts, per-classification tally). Exit codes: 0 pass; 1 named check failures; 2 unusable input; 3 internal gate bug. diff --git a/plugins/knowledge/skills/map-corpus/evals/evals.json b/plugins/knowledge/skills/map-corpus/evals/evals.json index 8468245825..cf2edb0208 100644 --- a/plugins/knowledge/skills/map-corpus/evals/evals.json +++ b/plugins/knowledge/skills/map-corpus/evals/evals.json @@ -28,10 +28,10 @@ "id": 3, "name": "no-discovery-artifacts-stops-loudly", "prompt": "Run /knowledge:map-corpus \"internal wiki\" https://wiki.internal.example where the site serves neither llms.txt nor any sitemap.", - "expected_output": "Stops loudly at discovery: reports that neither rung 1 (llms.txt) nor rung 2 (sitemap) resolved, names the deferred rung-3 decision (Q19, user-reserved firecrawl seam vs reimplementation), and asks the user rather than crawling in-page links.", + "expected_output": "Stops loudly at discovery: reports that neither rung 1 (llms.txt) nor rung 2 (sitemap) resolved, names the deferred rung-3 decision (user-reserved: a presence-gated firecrawl map seam versus a recorded reimplementation), and asks the user rather than crawling in-page links.", "expectations": [ "Does not extract in-page links as a fallback", - "Names the Q19 deferral and its user-reserved arbiter", + "Names the deferred rung-3 decision and that its arbiter is the user", "Does not silently classify the corpus as empty" ] }, diff --git a/plugins/knowledge/skills/map-corpus/extraction/node-manifest-format.md b/plugins/knowledge/skills/map-corpus/extraction/node-manifest-format.md index a065996f28..deed694f00 100644 --- a/plugins/knowledge/skills/map-corpus/extraction/node-manifest-format.md +++ b/plugins/knowledge/skills/map-corpus/extraction/node-manifest-format.md @@ -14,8 +14,8 @@ agent chose to enumerate. machine paths, locale-dependent text, or unordered collections enter the output. The snapshot is recorded by basename only. 3. **Self-verifying spans.** `content_sha256` is SHA-256 over the raw snapshot bytes - `[start_byte, end_byte)`. `snapshot.sha256` is SHA-256 over the whole snapshot as fetched (the - Brief's captured content-hash assumption). + `[start_byte, end_byte)`. `snapshot.sha256` is SHA-256 over the whole snapshot as fetched — the + hash the tracked citation shape carries (`${CLAUDE_PLUGIN_ROOT}/reference/citation-shape.md`). 4. **Document order.** `nodes` is ordered by `start_byte`; `index` is the 0-based position. 5. **Fail loudly.** Empty snapshot, unreadable file, unknown extension, non-UTF-8 BOM, CR-only line endings, or a partition self-check failure exits non-zero with a message. The extractor diff --git a/plugins/knowledge/skills/map-corpus/verification/inventory-format.md b/plugins/knowledge/skills/map-corpus/verification/inventory-format.md index f8b956025f..8a943331f5 100644 --- a/plugins/knowledge/skills/map-corpus/verification/inventory-format.md +++ b/plugins/knowledge/skills/map-corpus/verification/inventory-format.md @@ -54,7 +54,7 @@ Emitters must never emit a duplicate key. - `quote` — non-empty string, verbatim from the snapshot. - `start_byte` / `end_byte` — the quote's exact span in RAW SNAPSHOT bytes (end exclusive). -## Evidence-token byte mapping (the rule the Brief owed step 2) +## Evidence-token byte mapping The manifest is byte-addressed; quotes are text. The mapping rule: @@ -94,5 +94,5 @@ Exit codes: 0 all checks passed; 1 one or more named check failures; 2 unusable (parse/IO/schema-literal errors); 3 internal invariant violation. A clean run prints what it exercised (file names, row count, node count, field list) — silence is never a pass, and a pass covers only what was printed. The gate was written and tested to fail loudly on unparsable and -malformed input BEFORE being made a required artifact (Brief acceptance criterion 4; two prior -gates in this codebase shipped fail-open and were caught by verifiers). +malformed input BEFORE being made a required artifact (two prior gates in this codebase shipped +fail-open and were caught by verifiers). diff --git a/plugins/knowledge/skills/map-corpus/verification/test_check_inventory.py b/plugins/knowledge/skills/map-corpus/verification/test_check_inventory.py index db72fca2b5..c2ff600df6 100755 --- a/plugins/knowledge/skills/map-corpus/verification/test_check_inventory.py +++ b/plugins/knowledge/skills/map-corpus/verification/test_check_inventory.py @@ -1,9 +1,8 @@ #!/usr/bin/env python3 """Tests for check_inventory.py. Stdlib unittest; run: python test_check_inventory.py -The fail-loud cases here are the Brief's acceptance criterion 4: this gate is -proven to fail loudly on unparsable and malformed input BEFORE it is made a -required artifact. +The fail-loud cases here are why this file exists: the gate is proven to fail +loudly on unparsable and malformed input BEFORE it is made a required artifact. """ import copy