diff --git a/docs/branch-review-records/b2ecb1cee8bf7fcc71dfc33aafe348fd300805816d36dc8164eed9ed3fc356cf.record.md b/docs/branch-review-records/b2ecb1cee8bf7fcc71dfc33aafe348fd300805816d36dc8164eed9ed3fc356cf.record.md new file mode 100644 index 0000000000..beb8f133d7 --- /dev/null +++ b/docs/branch-review-records/b2ecb1cee8bf7fcc71dfc33aafe348fd300805816d36dc8164eed9ed3fc356cf.record.md @@ -0,0 +1 @@ +| 2026-08-17 | claude/rag-d3-s5-followups | 60e61bb9b89e2a62cefa7e89e8f6497c09e8a900 | docs/rag-improvement S5 follow-ups: HANDOVER S5/S2 rows, COORDINATION §7, 4 inbox requests (docs-only) | docs-only; S5 landed; follow-ups queued | verify:pr-local docs scope | diff --git a/docs/outstanding-issues-inbox/44e79b9e-5530-4b14-a431-b7a87683cf30.json b/docs/outstanding-issues-inbox/44e79b9e-5530-4b14-a431-b7a87683cf30.json new file mode 100644 index 0000000000..c16db7dd74 --- /dev/null +++ b/docs/outstanding-issues-inbox/44e79b9e-5530-4b14-a431-b7a87683cf30.json @@ -0,0 +1,14 @@ +{ + "version": 2, + "id": "44e79b9e-5530-4b14-a431-b7a87683cf30", + "createdOn": "2026-08-17", + "action": "add", + "payload": { + "pri": "P3", + "type": "task", + "summary": "Owner decision: enable RAG_TELEMETRY_EXTENDED (verification_latency_ms projection) in production once a dashboard consumer exists", + "detail": "Packet S5 (PR #2056, merge 093f9340c) landed the B1 telemetry gap assessment: the one proven gap is verification_latency_ms, now persisted behind RAG_TELEMETRY_EXTENDED (typed, default false) via the allow-listed projection module with canary-absence tests. Enabling it in production is an owner decision gated on a dashboard consumer existing (no consumer today), and is a Railway env change (provider-backed, explicit approval; rollback = set false). Next: when a dashboard question needs verification latency, set RAG_TELEMETRY_EXTENDED=true on the Database service after confirming the canary-absence tests are still green on main. Stop: do not enable speculatively; do not add unproven fields.", + "source": "RAG programme coordinator, packet S5 (PR #2056) follow-ups, 2026-08-17", + "issueUlid": "01M08C1AH72AB2NJRQFCXG3446" + } +} diff --git a/docs/outstanding-issues-inbox/48745dbc-b7a1-40d5-aec7-818274814291.json b/docs/outstanding-issues-inbox/48745dbc-b7a1-40d5-aec7-818274814291.json new file mode 100644 index 0000000000..4531eaaf96 --- /dev/null +++ b/docs/outstanding-issues-inbox/48745dbc-b7a1-40d5-aec7-818274814291.json @@ -0,0 +1,14 @@ +{ + "version": 2, + "id": "48745dbc-b7a1-40d5-aec7-818274814291", + "createdOn": "2026-08-17", + "action": "add", + "payload": { + "pri": "P2", + "type": "issue", + "summary": "Adversarial divergence (S5 harness pin): scope-other-owner-document — abstains in substance but the review fallback still cites in-scope evidence", + "detail": "Pinned in tests/rag-adversarial-harness.test.ts KNOWN_DIVERGENCES (self-expiring). Observed shape: grounded false, confidence unsupported, but cited chunk ids [syn-scope-owner-a] — the answer correctly abstains from the other-owner document, yet the review fallback attaches an in-scope citation to an unsupported answer. Fixture: scripts/fixtures/rag-adversarial-cases.v1.json case scope-other-owner-document (category scope_or_tenant). Tenancy/no-read invariant held (the other-owner content is never read). Next: decide whether an unsupported abstention may carry any citation; if not, strip citations on the abstention path (RAG-surface change; own PR; harness pin flips; canary pair). Stop: do not delete the pin without the behaviour change.", + "source": "RAG programme coordinator, packet S5 (PR #2056) follow-ups, 2026-08-17", + "issueUlid": "01M08C19VRC2D9JFG95HAJRFR7" + } +} diff --git a/docs/outstanding-issues-inbox/63e7cebb-9a42-4c98-9520-9a1b7f5874a8.json b/docs/outstanding-issues-inbox/63e7cebb-9a42-4c98-9520-9a1b7f5874a8.json new file mode 100644 index 0000000000..50c1358638 --- /dev/null +++ b/docs/outstanding-issues-inbox/63e7cebb-9a42-4c98-9520-9a1b7f5874a8.json @@ -0,0 +1,14 @@ +{ + "version": 2, + "id": "63e7cebb-9a42-4c98-9520-9a1b7f5874a8", + "createdOn": "2026-08-17", + "action": "add", + "payload": { + "pri": "P2", + "type": "issue", + "summary": "Adversarial divergence (S5 harness pin): scope-guessed-chunk-id — review fallback returns a grounded source pointer echoing the query instead of refusing", + "detail": "Pinned in tests/rag-adversarial-harness.test.ts KNOWN_DIVERGENCES (self-expiring). Observed shape: grounded true, cited [syn-scope-guess-a], and the guessed (never-retrieved) chunk id syn-not-retrieved-zzz is never resolved into content — the no-read invariant holds — but the review fallback returns a grounded source pointer that echoes the query text rather than refusing the guessed-id request. Fixture: scripts/fixtures/rag-adversarial-cases.v1.json case scope-guessed-chunk-id (category scope_or_tenant). Next: decide whether a query naming an unretrieved chunk id should refuse rather than fall back to a source pointer (RAG-surface change; own PR; harness pin flips; canary pair). Stop: do not delete the pin without the behaviour change.", + "source": "RAG programme coordinator, packet S5 (PR #2056) follow-ups, 2026-08-17", + "issueUlid": "01M08C1A7XNTAV3DNZ50RZ2C05" + } +} diff --git a/docs/outstanding-issues-inbox/71399b63-cce0-46d4-9fc3-c5fdc617289d.json b/docs/outstanding-issues-inbox/71399b63-cce0-46d4-9fc3-c5fdc617289d.json new file mode 100644 index 0000000000..34f61b8625 --- /dev/null +++ b/docs/outstanding-issues-inbox/71399b63-cce0-46d4-9fc3-c5fdc617289d.json @@ -0,0 +1,14 @@ +{ + "version": 2, + "id": "71399b63-cce0-46d4-9fc3-c5fdc617289d", + "createdOn": "2026-08-17", + "action": "add", + "payload": { + "pri": "P2", + "type": "issue", + "summary": "Adversarial divergence (S5 harness pin): cite-mismatched-attribution — offline document-match listing cites every retrieved document, not only the claim-bearing one", + "detail": "Pinned in tests/rag-adversarial-harness.test.ts KNOWN_DIVERGENCES (self-expiring: the harness asserts the normative B0 fixture contract still FAILS; when behaviour reaches the fixture expectation the pin goes red and must be deleted). Observed shape: cited chunk ids [syn-cite-attrib-a, syn-cite-attrib-b], grounded true, answerQualityTier source_only — the document-match listing attributes both retrieved documents although only one carries the claim. Fixture: scripts/fixtures/rag-adversarial-cases.v1.json case cite-mismatched-attribution (category citation_fabrication). Safety invariants (network, budget, canary absence, forbidden substrings, tenancy) hold; only citation attribution precision diverges. Next: decide whether the document-match listing should cite only claim-bearing documents (RAG-surface change; own PR; RAG impact line; offline harness proves the pin flips; canary pair). Stop: do not delete the pin without the behaviour change; do not weaken the fixture.", + "source": "RAG programme coordinator, packet S5 (PR #2056) follow-ups, 2026-08-17", + "issueUlid": "01M08C19FYVXB8XATJE3TYZJMS" + } +} diff --git a/docs/rag-improvement/COORDINATION.md b/docs/rag-improvement/COORDINATION.md index 0a22f53173..cd5edba948 100644 --- a/docs/rag-improvement/COORDINATION.md +++ b/docs/rag-improvement/COORDINATION.md @@ -176,17 +176,19 @@ Always a fresh owner ask, every single time: - **Done (2026-08-17):** #1895, #1899, #1908 (2026-08-13); **#2022 S1** (squash `2bd146eed`); **#2024 D1** (`78fe906b8`); **#2035 S1b** (merge `92f7618c0`); **#2036 S4** (`f5b093291`); - **#2037 #212 T4** (`1726537b7`); **#2045 R0 reconcile** (28 requests; #212 closed). Canary + **#2037 #212 T4** (`1726537b7`); **#2045 R0 reconcile** (28 requests; #212 closed); **#2048 D2** (S1d packet); **#2056 S5** (merge `093f9340c`: B1 `verification_latency_ms` behind `RAG_TELEMETRY_EXTENDED`, B2 `eval:rag:adversarial:offline` with 3 self-expiring divergence pins). Canary pairs: S1 baseline 32025082010 → S1b post 32039841070, recall 1.0/1.0, zero per-case rr - regressions, answer gate 45/45. The first S1b post run (32038751592) was red on one non-golden + regressions, answer gate 44/44. The first S1b post run (32038751592) was red on one non-golden case; root-caused to the finalizer gap-recovery hole → packet S1d, not S1b. - **Owner decisions:** R1 before S2; governance Option B; S1d lands before S2. -- **Wave 1 (dispatch now, parallel, disjoint files):** S1c (`rag-claim-support.ts`), S1d - (`rag-extractive-answer.ts` finalizer / `rag.ts` fast-failure block), G1 (`types.ts`, - `answer-stream-contract.ts`, `rag-row-contracts.ts`), S5 (harness/telemetry), S6 (`eval/docling/`). - Prompts: HANDOVER §7. Merge order for the RAG-behaviour ones: S1c and S1d each get their own - canary pair; merge one, run its canary, then the next. -- **Wave 2:** S2 (+S2b) after S1c and S1d canaries are green. **Wave 3:** S3; S7+ owner decisions. +- **Wave 1 status:** S1c (#2052 + follow-ups #2063/#2065) merged, canary pair green + (32049952885 → 32052479537); S1d (#2054, merge `0bbd64fbc`) merged, **canary pending owner + approval**; S6 (#2057) merged. **Still to dispatch: G1** (`types.ts`, + `answer-stream-contract.ts`, `rag-row-contracts.ts`; no canary). S5 follow-ups queued as + inbox requests: three adversarial divergence pins (`cite-mismatched-attribution`, + `scope-other-owner-document`, `scope-guessed-chunk-id`) and the owner decision on enabling + `RAG_TELEMETRY_EXTENDED` in production. +- **Wave 2:** S2 (+S2b) once the S1d canary pair is green (S1c pair already green). **Wave 3:** S3; S7+ owner decisions. - **Waiting on owner:** merges as PRs open; canary approvals for S1c, S1d, S2. - **Live board (artifact, owner-private):** RAG Master Plan v2 — `https://claude.ai/code/artifact/d5dba709-0df3-40e3-8a45-15997231533d`. diff --git a/docs/rag-improvement/HANDOVER.md b/docs/rag-improvement/HANDOVER.md index d9d9f32967..6e79fff7b0 100644 --- a/docs/rag-improvement/HANDOVER.md +++ b/docs/rag-improvement/HANDOVER.md @@ -67,25 +67,25 @@ generation-quality verdict on fallback`), merged 2026-08-13 — structured ## 2. Status table — update in every programme PR -| Packet | Scope | Branch | PR | State | Canary / evidence refs | -| ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | --------------------- | --------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Guide | Programme guide | `claude/rag-plan-review-guide-vhrls9` | #1895 | Merged 2026-08-13 | docs-only | -| Handover | Multi-session handover + coordination | `claude/rag-plan-review-guide-vhrls9` | #1908 / #2024 | Merged 2026-08-13; coordination layer PR #2024 | docs-only | -| S0 | A1 phase 1: structured fallback diagnostics | `claude/lithium-generation-quality-debug-ji1vce` | #1899 | Merged 2026-08-13 | offline 93/93 focused | -| S1 | A1 phase 2: rung-1 verification-faithfulness fixes | `claude/s1-rag-mitigation-231-86c182` | #2022 | Merged 2026-08-17 (squash `2bd146eed`, landed by content) | 8 pre-fix + 5 post-fix live probes 2026-08-17; offline 583/583; canary pair run 31964560921 (baseline `8f8d111ab`) -> run 32025082010 (`2bd146eed`): recall 1.0/1.0, zero per-case rr regressions, answer gate 44/44 (denominator reconciled by S5; see baseline-record §3); rung-2 measurement in `docs/audit/live-drift-forensics-2026-08.md` §5 | -| S1b | A1 rung 3 (R1): pre-deadline strong routing for dosing class | `claude/s1b-rag-dosing-routing-6u1mik` | #2035 | Merged 2026-08-17 (PR #2035, merge `92f7618`) | canary pair pending: baseline run 32025082010 (`2bd146eed`) -> post-merge dispatch (owner-approved); offline 586/586 + verify:pr-local heavy scope green | -| S1c | A1 residuals R2 + R3: claim-support strictness | `claude/s1c-residuals-r2-r3-4pb1at` | #2052 | PR open 2026-08-17 | offline 613/613 + verify:pr-local heavy scope green; R3 measured before loosening: 87->78 sole-overlap rejections (46->42 unique), zero protective flips; canary pair planned post-merge: baseline run 32039841070 (`92f7618`) -> post-merge dispatch (owner-approved) | -| S1d | A1 final-gate gap recovery: hedged cited low-confidence fast answers must recover extractively, not collapse to a citation-free `provider_source_gap` | `claude/s1d-final-gate-gap-recovery-dxgrn2` | #2054 | In review — PR open 2026-08-17 (`recoverFinalGateGapExtractively` in the finalizer; terminal gaps pinned) | needs post-merge canary pair (baseline run 32039841070, `e6ad0d5db`); offline: verify:pr-local heavy green, eval:rag:offline 604/604, 6 new discriminating fixtures | -| G1 | Governance: provenance tag for document-summary rows (Option B) | `claude/rag-g1-document-context-origin-` | — | Ready — disjoint; owner decided Option B 2026-08-17 | no canary (no behaviour change) | -| S2 | A2 (+A3): composition menu + moderate length | `claude/rag-a2-composition-` | — | Blocked on S1b + S1c | canary pair + `eval:answer-quality` + Gate E | -| S2b | A3: moderate length (if separate review needed) | `claude/rag-a3-length-` | — | Blocked on S2 | — | -| S3 | A4: follow-up suggestion refinement | `claude/rag-a4-follow-ups-` | — | Blocked on S2 + S2b | — | -| S4 | B0: adversarial fixtures + baseline + register | `claude/packet-s4-adversarial-fixtures-5ho5tp` | #2036 | Merged 2026-08-17 (squash `f5b093291`) | Offline only: `check:rag:adversarial-fixtures` 24 cases / 8 categories / 6 canaries; `eval:rag:offline` 24 suites, 597 tests. Baseline `scripts/fixtures/rag-adversarial-baseline.v1.json` marks the three provider-backed gates `pending_owner_run` | -| S5 | B1+B2: telemetry assessment + offline harness | `claude/s5-rag-telemetry-harness-2wvis7` | #2056 | Merged 2026-08-17 (merge `093f934`) | Offline only: `eval:rag:adversarial:offline` 25/25 (24 cases + canary-free report; 3 divergences pinned in `KNOWN_DIVERGENCES`); B1 gap = `verification_latency_ms` behind `RAG_TELEMETRY_EXTENDED` (default false); canary-absence tests green; 44/44 denominator reconciled | -| S6 | B3: Docling lab benchmark | `claude/packet-s6-docling-lab-d6foa6` | #2057 | Open — awaiting review (2026-08-17) | Offline only: `check:docling-lab` 36 fixtures / 10 hostile / 6 canaries + Gate B template valid; `verify:pr-local` heavy plan failed:(none); contract test 20/20; legacy smoke 46 docs, 10/10 hostile contained, canary-clean report. Verdict is a separate owner dispatch of `docling-lab.yml` | -| S7+ | B4 shadow / B5 Ragas / B6 reranker / B7 DSPy | — | — | Gated — owner decision | — | -| #212 T1–T3 | Runtime row contracts (rag.ts, rag-candidate-sources.ts, src/app/api) — sibling stream sharing `src/lib/rag/**` | — | #1946 / #1981 / #2023 | Merged (T3 squash `440a34f71` 2026-08-17) | see the #212 ledger row; RAG surface complete for the cast class | -| #212 T4 | Runtime row contracts: `worker/main.ts` (11 casts) — sibling stream | `claude/ledger-212-tranche-4-worker-q3y6i4` | #2037 | Merged 2026-08-17 (squash `1726537b7`); #212 closed by reconcile PR #2045 | Governance Preflight complete; audit: 1 inbound cast (claim rows, per-row fail-soft) + 2 read-back param casts contracted, 9 outbound/interop left; closes #212 (inbox `done` queued in the PR) | +| Packet | Scope | Branch | PR | State | Canary / evidence refs | +| ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | --------------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Guide | Programme guide | `claude/rag-plan-review-guide-vhrls9` | #1895 | Merged 2026-08-13 | docs-only | +| Handover | Multi-session handover + coordination | `claude/rag-plan-review-guide-vhrls9` | #1908 / #2024 | Merged 2026-08-13; coordination layer PR #2024 | docs-only | +| S0 | A1 phase 1: structured fallback diagnostics | `claude/lithium-generation-quality-debug-ji1vce` | #1899 | Merged 2026-08-13 | offline 93/93 focused | +| S1 | A1 phase 2: rung-1 verification-faithfulness fixes | `claude/s1-rag-mitigation-231-86c182` | #2022 | Merged 2026-08-17 (squash `2bd146eed`, landed by content) | 8 pre-fix + 5 post-fix live probes 2026-08-17; offline 583/583; canary pair run 31964560921 (baseline `8f8d111ab`) -> run 32025082010 (`2bd146eed`): recall 1.0/1.0, zero per-case rr regressions, answer gate 44/44 (denominator reconciled by S5; see baseline-record §3); rung-2 measurement in `docs/audit/live-drift-forensics-2026-08.md` §5 | +| S1b | A1 rung 3 (R1): pre-deadline strong routing for dosing class | `claude/s1b-rag-dosing-routing-6u1mik` | #2035 | Merged 2026-08-17 (PR #2035, merge `92f7618`) | canary pair pending: baseline run 32025082010 (`2bd146eed`) -> post-merge dispatch (owner-approved); offline 586/586 + verify:pr-local heavy scope green | +| S1c | A1 residuals R2 + R3: claim-support strictness | `claude/s1c-residuals-r2-r3-4pb1at` | #2052 | Merged 2026-08-17 (merge `b8e774bcd`; follow-ups #2063, #2065) | canary pair: baseline run 32049952885 -> post run 32052479537 (`084f63799`): recall 1.0/1.0, zero per-case rr regressions, answer gate 44/44 | +| S1d | A1 final-gate gap recovery: hedged cited low-confidence fast answers must recover extractively, not collapse to a citation-free `provider_source_gap` | `claude/s1d-final-gate-gap-recovery-dxgrn2` | #2054 | Merged 2026-08-17 (merge `0bbd64fbc`); canary pending (owner approval) | needs post-merge canary pair (baseline run 32039841070, `e6ad0d5db`); offline: verify:pr-local heavy green, eval:rag:offline 604/604, 6 new discriminating fixtures | +| G1 | Governance: provenance tag for document-summary rows (Option B) | `claude/rag-g1-document-context-origin-` | — | Ready — disjoint; owner decided Option B 2026-08-17 | no canary (no behaviour change) | +| S2 | A2 (+A3): composition menu + moderate length | `claude/rag-a2-composition-` | — | Blocked on the S1d canary pair (S1c pair green); dispatch when green | canary pair + `eval:answer-quality` + Gate E | +| S2b | A3: moderate length (if separate review needed) | `claude/rag-a3-length-` | — | Blocked on S2 | — | +| S3 | A4: follow-up suggestion refinement | `claude/rag-a4-follow-ups-` | — | Blocked on S2 + S2b | — | +| S4 | B0: adversarial fixtures + baseline + register | `claude/packet-s4-adversarial-fixtures-5ho5tp` | #2036 | Merged 2026-08-17 (squash `f5b093291`) | Offline only: `check:rag:adversarial-fixtures` 24 cases / 8 categories / 6 canaries; `eval:rag:offline` 24 suites, 597 tests. Baseline `scripts/fixtures/rag-adversarial-baseline.v1.json` marks the three provider-backed gates `pending_owner_run` | +| S5 | B1+B2: telemetry assessment + offline harness | `claude/s5-rag-telemetry-harness-2wvis7` | #2056 | Merged 2026-08-17 (merge `093f9340c`); post-merge canary run 32049952885 | Offline only: `eval:rag:adversarial:offline` 25/25 (24 cases + canary-free report; 3 divergences pinned in `KNOWN_DIVERGENCES`); B1 gap = `verification_latency_ms` behind `RAG_TELEMETRY_EXTENDED` (default false); canary-absence tests green; 44/44 denominator reconciled | +| S6 | B3: Docling lab benchmark | `claude/packet-s6-docling-lab-d6foa6` | #2057 | Merged 2026-08-17 (merge `5a6418636`) | Offline only: `check:docling-lab` 36 fixtures / 10 hostile / 6 canaries + Gate B template valid; `verify:pr-local` heavy plan failed:(none); contract test 20/20; legacy smoke 46 docs, 10/10 hostile contained, canary-clean report. Verdict is a separate owner dispatch of `docling-lab.yml` | +| S7+ | B4 shadow / B5 Ragas / B6 reranker / B7 DSPy | — | — | Gated — owner decision | — | +| #212 T1–T3 | Runtime row contracts (rag.ts, rag-candidate-sources.ts, src/app/api) — sibling stream sharing `src/lib/rag/**` | — | #1946 / #1981 / #2023 | Merged (T3 squash `440a34f71` 2026-08-17) | see the #212 ledger row; RAG surface complete for the cast class | +| #212 T4 | Runtime row contracts: `worker/main.ts` (11 casts) — sibling stream | `claude/ledger-212-tranche-4-worker-q3y6i4` | #2037 | Merged 2026-08-17 (squash `1726537b7`); #212 closed by reconcile PR #2045 | Governance Preflight complete; audit: 1 inbound cast (claim rows, per-row fail-soft) + 2 read-back param casts contracted, 9 outbound/interop left; closes #212 (inbox `done` queued in the PR) | Update rule: the session that opens a packet's PR edits its row (branch, PR number, state) in the same PR. A later session updating another packet may also correct stale rows