feat(rag): tag document-summary rows with a document_context provenance origin

docs/branch-review-records/1048a2a62bd4a4306cb90f6c144e94427f6cf21d5a9f449f983b170ddb35a4b5.record.md

@@ -0,0 +1 @@
+| 2026-08-17 | claude/g1-rag-document-context-qn9ubx | f59b358d5ddba2480c585196b262afe1f4181632 | packet G1 document_context provenance tag + ledger issueRowFingerprint display-id fix + two queued G1 closures (supersedes the f5f146a record: two further commits landed on this branch) | approved — G1 is a provenance tag only, deriveConfidence and synthetic_similarity_count unchanged and pinned; ledger fix restores issues:done for ULID-backed rows with null-safety preserved; no retrieval behaviour change, no canary per owner decision 2026-08-17 Option B | verify:pr-local all 18 stages green at this head; full unit 642 files / 6879 passed 4 skipped; eval:rag:offline 36 golden 25 suites 603/603; 4 G1 pins + 3 ledger pins mutation-checked |

docs/branch-review-records/3f5dad0c7109ddc3f59bd3ce42fa9df1587d3cbcd938a01b8950f1f1e746472a.record.md

@@ -0,0 +1 @@
+| 2026-08-17 | claude/g1-rag-document-context-qn9ubx | f5f146a497a4ed04fc1db99b7b8f90a00a1cea30 | packet G1 — document_context similarity_origin provenance tag (types.ts, answer-stream-contract.ts, rag-row-contracts.ts, rag.ts telemetry export) + 4 discriminating pins + H5a/HANDOVER docs | approved — provenance tag only; deriveConfidence and synthetic_similarity_count unchanged and pinned; no retrieval behaviour change, no canary per owner decision 2026-08-17 Option B | vitest 5 touched suites 58/58; full unit 642 files / 6879 passed 4 skipped; eval:rag:offline 36 golden 25 suites 603/603; check:rag:fixtures; verify:pr-local all 18 stages green (lint, typecheck, test, build, eval:rag:offline); 4 pins mutation-checked |

docs/clinical-hazard-analysis.md

@@ -133,6 +133,31 @@ keep telemetry's `synthetic_similarity_count` from counting it. Option A (tag as
 downgrade without a measured safety gain. Implementation packet: `docs/rag-improvement/HANDOVER.md`
 §G1 (no retrieval behaviour change; no canary).
 
+**H5a — Option B implemented (packet G1, 2026-08-17).** `"document_context"` is now a declared
+value of the `similarity_origin` union ([`types.ts`](../src/lib/types.ts)), accepted by the
+streamed-preview client-source validator
+([`answer-stream-contract.ts`](../src/lib/answer-stream-contract.ts)), and stamped on every row
+`buildDocumentSummaryResults` builds
+([`rag-row-contracts.ts`](../src/lib/rag/rag-row-contracts.ts)). The fabricated `similarity: 1`
+is therefore no longer indistinguishable from a perfect cosine at any surface that reads a row.
+Per the decision, `deriveConfidence` ([`rag-answer-support.ts`](../src/lib/rag/rag-answer-support.ts))
+is **unchanged** — it still excludes only `"synthetic_text"` from `strongestNonSynthetic`, so a
+document summary still reaches `"high"` — and `synthetic_similarity_count`
+([`rag.ts`](../src/lib/rag/rag.ts)) still counts only `"synthetic_text"`, keeping the RC9
+imputed-score signal a single population. Four discriminating pins hold the split: the summary
+rows carry the tag ([`rag-retrieval-row-contract.test.ts`](../tests/rag-retrieval-row-contract.test.ts));
+two `"document_context"` citations at ≥ 0.82 still yield `"high"` while the identical scores
+tagged `"synthetic_text"` still yield `"medium"`; and the telemetry counter ignores the new value
+([`rag-score.test.ts`](../tests/rag-score.test.ts)). Each was mutation-checked — folding
+`"document_context"` into the `deriveConfidence` exclusion (rejected Option A), widening the
+telemetry filter, and removing the stamp each turn the matching pin red.
+
+**Residual after G1.** This closes the _legibility_ half of the live residual, not the underlying
+question of whether a constant 1.0 should ever contribute to a confidence label. The tag now makes
+that reachable: any future gate can discriminate the document-summary route without re-deriving
+provenance. The fast-path half of H5a (`"synthetic_text"`, already excluded from `"high"`) and the
+`scoreValue` gap in the row above are unchanged by this packet.
+
 ### H6 — Adversarial / corrupted document content changes the answer
 
 _Pathway: uploaded document text → model context → answer. Detailed in the companion threat model; summarized here as a hazard._

docs/outstanding-issues-inbox/34c0f9bf-22fe-495b-a828-f73bbd4cfddf.json

@@ -0,0 +1,11 @@
+{
+  "version": 2,
+  "id": "34c0f9bf-22fe-495b-a828-f73bbd4cfddf",
+  "createdOn": "2026-08-17",
+  "action": "done",
+  "payload": {
+    "id": "#0MSNT8",
+    "outcome": "Implemented as packet G1 on branch claude/g1-rag-document-context-qn9ubx. Every element of the queued Option B scope landed: document_context added to the similarity_origin union in src/lib/types.ts and to src/lib/answer-stream-contract.ts (as an allow-set, so the union and the stream validator cannot drift), stamped in buildDocumentSummaryResults (src/lib/rag/rag-row-contracts.ts), deriveConfidence left unchanged, rag.ts synthetic_similarity_count left counting only synthetic_text, and docs/clinical-hazard-analysis.md H5a updated to mark the decision implemented. Four discriminating pins, each mutation-checked against the change it exists to catch: summary rows carry the tag (tests/rag-retrieval-row-contract.test.ts); two document_context citations at >= 0.82 still yield high while the identical scores tagged synthetic_text still yield medium, and the telemetry counter ignores the new value (tests/rag-score.test.ts); the stream validator accepts every declared union member with a compile-time exhaustiveness guard (tests/answer-incremental-delivery.test.ts). Gates: vitest 58/58 across the five touched suites, full unit 642 files / 6879 passed, eval:rag:offline 36 golden cases / 603 tests, check:rag:fixtures, verify:pr-local all 18 stages green. No canary per the decision. HANDOVER status row updated. Closes the paired governance question row #J912J9.",
+    "baseRowFingerprint": "5d3f833d9e969448a55703d05c2423ccd92eb9176410abf285825e88bcc0daec"
+  }
+}

docs/outstanding-issues-inbox/fb7d42c0-ee67-426e-907e-f2c306afc38b.json

@@ -0,0 +1,11 @@
+{
+  "version": 2,
+  "id": "fb7d42c0-ee67-426e-907e-f2c306afc38b",
+  "createdOn": "2026-08-17",
+  "action": "done",
+  "payload": {
+    "id": "#J912J9",
+    "outcome": "Answered and implemented (packet G1, PR for branch claude/g1-rag-document-context-qn9ubx). Owner decided Option B on 2026-08-17: document-summary rows keep the high confidence label, and the fabricated similarity gets its own provenance value rather than being folded into synthetic_text. Landed: document_context added to the similarity_origin union (src/lib/types.ts) and to the streamed-preview client-source validator (src/lib/answer-stream-contract.ts), and stamped in buildDocumentSummaryResults (src/lib/rag/rag-row-contracts.ts). Per the decision deriveConfidence (src/lib/rag/rag-answer-support.ts) is unchanged and still excludes only synthetic_text, and rag.ts synthetic_similarity_count still counts only synthetic_text; both are pinned by discriminating tests that go red on the rejected Option A fold. Option A (tag as synthetic_text so summaries cap at medium) recorded as rejected. docs/clinical-hazard-analysis.md H5a marks the decision implemented and names the residual: the tag closes the legibility gap, not the deeper question of whether a constant 1.0 should contribute to a confidence label, but any future gate can now discriminate the route without re-deriving provenance. No retrieval behaviour change; no canary.",
+    "baseRowFingerprint": "e072b25f5773f1945471f8081b307f5f74766ef400aaad8c01d27f186066eea6"
+  }
+}

docs/rag-improvement/HANDOVER.md

@@ -76,7 +76,7 @@ generation-quality verdict on fallback`), merged 2026-08-13 — structured
 | S1b        | A1 rung 3 (R1): pre-deadline strong routing for dosing class                                                                                          | `claude/s1b-rag-dosing-routing-6u1mik`           | #2035                 | Merged 2026-08-17 (PR #2035, merge `92f7618`)                             | canary pair pending: baseline run 32025082010 (`2bd146eed`) -> post-merge dispatch (owner-approved); offline 586/586 + verify:pr-local heavy scope green                                                                                                                                                                                           |
 | S1c        | A1 residuals R2 + R3: claim-support strictness                                                                                                        | `claude/s1c-residuals-r2-r3-4pb1at`              | #2052                 | Merged 2026-08-17 (merge `b8e774bcd`; follow-ups #2063, #2065)            | canary pair: baseline run 32049952885 -> post run 32052479537 (`084f63799`): recall 1.0/1.0, zero per-case rr regressions, answer gate 44/44                                                                                                                                                                                                       |
 | S1d        | A1 final-gate gap recovery: hedged cited low-confidence fast answers must recover extractively, not collapse to a citation-free `provider_source_gap` | `claude/s1d-final-gate-gap-recovery-dxgrn2`      | #2054                 | Merged 2026-08-17 (merge `0bbd64fbc`); canary pending (owner approval)    | needs post-merge canary pair (baseline run 32039841070, `e6ad0d5db`); offline: verify:pr-local heavy green, eval:rag:offline 604/604, 6 new discriminating fixtures                                                                                                                                                                                |
-| G1         | Governance: provenance tag for document-summary rows (Option B)                                                                                       | `claude/rag-g1-document-context-origin-<suffix>` | —                     | Ready — disjoint; owner decided Option B 2026-08-17                       | no canary (no behaviour change)                                                                                                                                                                                                                                                                                                                    |
+| G1         | Governance: provenance tag for document-summary rows (Option B)                                                                                       | `claude/g1-rag-document-context-qn9ubx`          | #2053                 | Implemented 2026-08-17 - awaiting review/merge                            | no canary by decision (provenance tag only). Offline: 58/58 across the five touched suites; `eval:rag:offline`, `check:rag:fixtures`, `verify:pr-local` - see PR body                                                                                                                                                                              |
 | S2         | A2 (+A3): composition menu + moderate length                                                                                                          | `claude/rag-a2-composition-<suffix>`             | —                     | Blocked on the S1d canary pair (S1c pair green); dispatch when green      | canary pair + `eval:answer-quality` + Gate E                                                                                                                                                                                                                                                                                                       |
 | S2b        | A3: moderate length (if separate review needed)                                                                                                       | `claude/rag-a3-length-<suffix>`                  | —                     | Blocked on S2                                                             | —                                                                                                                                                                                                                                                                                                                                                  |
 | S3         | A4: follow-up suggestion refinement                                                                                                                   | `claude/rag-a4-follow-ups-<suffix>`              | —                     | Blocked on S2 + S2b                                                       | —                                                                                                                                                                                                                                                                                                                                                  |

scripts/check-outstanding-issues.mjs

@@ -33,7 +33,7 @@ import { execFileSync } from "node:child_process";
 import { readFileSync } from "node:fs";
 import { createHash } from "node:crypto";
 
-import { canonicalLegacyIssueId, issueIdCitations, parseIssueIdCell } from "./issue-id.mjs";
+import { canonicalLegacyIssueId, isIssueDisplayId, issueIdCitations, parseIssueIdCell } from "./issue-id.mjs";
 
 export const ISSUES_PATH = "docs/outstanding-issues.md";
 
@@ -250,22 +250,34 @@ export function parseIssues(markdown) {
   };
 }
 
+// Two id generations coexist in the ledger: legacy zero-padded sequential ids
+// (`#001`) and the Crockford display locators minted from a row's ULID
+// (`#J912J9`). Rows of both kinds are addressed by display id everywhere else,
+// but this lookup resolved only the numeric form — so for every row created
+// after the ULID migration it returned null, and `ledger-inbox.mjs` reads a null
+// fingerprint as "no such row" and refuses the request. The visible symptom was
+// `npm run issues:done '#J912J9'` failing with "is not in Open items" against a
+// row plainly present in Open items, which made the optimistic-concurrency check
+// unreachable for exactly the rows that have it available (they carry a ULID).
 export function issueRowFingerprint(markdown, issueId) {
-  const trimmed = String(issueId).trim();
-  const numericMatch = trimmed.match(/^#(\d+)$/);
-  const number = numericMatch ? Number(numericMatch[1]) : null;
-  if (numericMatch && !Number.isFinite(number)) return null;
-
-  // Legacy numeric ids match by number (tolerating zero-padding differences);
-  // ULID-suffix display ids minted by reconcile match by the exact display id.
-  // Without the second arm, issues:done could never close a reconciled row.
-  const row = parseIssues(markdown).rows.find(
-    (entry) =>
-      entry.table === "open" &&
-      entry.valid &&
-      entry.raw &&
-      (numericMatch ? entry.number === number : entry.id === trimmed),
-  );
+  const id = String(issueId).trim();
+  const legacy = id.match(/^#(\d+)$/);
+  const number = legacy ? Number(legacy[1]) : null;
+  if (legacy) {
+    if (!Number.isFinite(number)) return null;
+  } else if (!isIssueDisplayId(id)) {
+    return null;
+  }
+
+  const open = parseIssues(markdown).rows.filter((entry) => entry.table === "open" && entry.valid && entry.raw);
+  // Exact display id first, and only then the legacy numeric interpretation.
+  // Crockford's alphabet includes 0-9, so a ULID-derived locator can be entirely
+  // digits (`#041061`) and is indistinguishable from a legacy id by pattern
+  // alone — branching on shape would silently miss exactly those rows. The
+  // fallback preserves the old behaviour of resolving a non-canonical legacy id
+  // (`#5`) to its zero-padded row, and the exact-id arm preserves
+  // `issues:done` closing ULID-suffix display ids minted by reconcile.
+  const row = open.find((entry) => entry.id === id) ?? (legacy ? open.find((entry) => entry.number === number) : null);
   if (!row) return null;
   const normalized = `| ${cells(row.raw).join(" | ")} |`;
   return createHash("sha256").update(normalized).digest("hex");

src/lib/answer-stream-contract.ts

@@ -47,6 +47,10 @@ const answerSectionKinds = new Set([
   "verification",
 ]);
 const answerSectionSupportLevels = new Set(["direct", "partial", "nearby", "unsupported"]);
+// Mirrors the `similarity_origin` union in `types.ts`. Kept as an allow-set rather than a
+// chain of `!==` comparisons so adding a provenance value there cannot silently leave the
+// streamed preview rejecting a payload the `final` response accepts.
+const similarityOriginValues = new Set(["cosine", "synthetic_text", "document_context"]);
 const citationProvenanceValues = new Set([
   "model_selected",
   "section_selected",
@@ -165,8 +169,7 @@ function isClientSource(value: unknown): value is SearchResult {
   if (
     "similarity_origin" in value &&
     value.similarity_origin !== undefined &&
-    value.similarity_origin !== "cosine" &&
-    value.similarity_origin !== "synthetic_text"
+    !similarityOriginValues.has(value.similarity_origin as string)
   ) {
     return false;
   }

src/lib/rag/rag-row-contracts.ts

@@ -183,7 +183,23 @@ export function assertIndexUnitRows(rows: unknown, rpc: string): asserts rows is
   assertRowsAgainst(z.array(indexUnitRowSchema), rows, rpc);
 }
 
-/** Build and validate the locally retrieved rows used as document-summary context. */
+/**
+ * Build and validate the locally retrieved rows used as document-summary context.
+ *
+ * `similarity: 1` here is a constant, not a measured cosine: `summarizeDocument` loads every
+ * committed chunk of one document, so there is no query to score against — the document IS the
+ * query. The `similarity_origin: "document_context"` tag makes that provenance explicit rather
+ * than leaving a fabricated 1.0 indistinguishable from a perfect vector match (H5a live
+ * residual, `docs/clinical-hazard-analysis.md`; owner decision 2026-08-17, Option B).
+ *
+ * The tag is deliberately NOT `"synthetic_text"`. That value marks scores imputed from lexical
+ * or structural match strength on the general answer path, where a title hit can masquerade as
+ * semantic evidence; `deriveConfidence` (`rag-answer-support.ts`) therefore excludes it from the
+ * "high" bar. This route has no match strength to inflate and its citations are verified by the
+ * same grounding pipeline, so the confidence derivation is unchanged and the "high" label stays.
+ * Reusing `"synthetic_text"` here would silently cap every document summary at "medium" —
+ * recorded as rejected Option A. `tests/rag-score.test.ts` pins both halves of that split.
+ */
 export function buildDocumentSummaryResults(
   chunks: unknown[],
   document: { title: string; file_name: string; metadata?: unknown },
@@ -194,6 +210,7 @@ export function buildDocumentSummaryResults(
     file_name: document.file_name,
     source_metadata: normalizeOptionalSourceMetadata(document.metadata),
     similarity: 1,
+    similarity_origin: "document_context" as const,
     images: [],
   }));
   assertRetrievalRows(results, "document_summary_context");

src/lib/rag/rag.ts

@@ -537,8 +537,8 @@ function provenanceLayerKeys(result: SearchResult) {
   return layers;
 }
 
-/** Record search score telemetry. */
-function recordSearchScoreTelemetry(telemetry: SearchTelemetry, results: SearchResult[]) {
+/** Record search score telemetry. Exported for `tests/rag-score.test.ts`; not a route surface. */
+export function recordSearchScoreTelemetry(telemetry: SearchTelemetry, results: SearchResult[]) {
   if (!results.length) {
     telemetry.top_score = 0;
     telemetry.second_top_score = 0;
@@ -570,6 +570,7 @@ function recordSearchScoreTelemetry(telemetry: SearchTelemetry, results: SearchR
   telemetry.score_spread = Number(Math.max(0, telemetry.top_score - telemetry.second_top_score).toFixed(4));
   telemetry.score_distinct_documents = new Set(results.map((result) => result.document_id)).size;
   telemetry.retrieval_candidate_count = results.length;
+  // Strict equality, deliberately: "document_context" rows carry the constant 1 with no match strength to inflate; counting them here would mix two populations and make the RC9 signal unreadable. Pinned by tests/rag-score.test.ts.
   telemetry.synthetic_similarity_count = results.filter(
     (result) => result.similarity_origin === "synthetic_text",
   ).length;