From 60e61bb9b89e2a62cefa7e89e8f6497c09e8a900 Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Tue, 18 Aug 2026 01:24:56 +0800 Subject: [PATCH 1/2] docs(rag): mark S5 merged, re-key S2's blocker, queue the S5 follow-ups MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - HANDOVER: S5 row Merged (PR #2056, merge 093f9340c, post-merge canary run 32049952885); S2 row now blocked on S1c + S1d (each with its own canary). - COORDINATION §7: S5 and D2 in the done list; wave 1 without S5; follow-ups. - Four immutable inbox requests: the three self-expiring KNOWN_DIVERGENCES pins from tests/rag-adversarial-harness.test.ts (P2 issues) and the owner decision on enabling RAG_TELEMETRY_EXTENDED in production (P3 task). Co-Authored-By: Claude Fable 5 --- .../44e79b9e-5530-4b14-a431-b7a87683cf30.json | 14 ++++++++++++++ .../48745dbc-b7a1-40d5-aec7-818274814291.json | 14 ++++++++++++++ .../63e7cebb-9a42-4c98-9520-9a1b7f5874a8.json | 14 ++++++++++++++ .../71399b63-cce0-46d4-9fc3-c5fdc617289d.json | 14 ++++++++++++++ docs/rag-improvement/COORDINATION.md | 7 +++++-- docs/rag-improvement/HANDOVER.md | 4 ++-- 6 files changed, 63 insertions(+), 4 deletions(-) create mode 100644 docs/outstanding-issues-inbox/44e79b9e-5530-4b14-a431-b7a87683cf30.json create mode 100644 docs/outstanding-issues-inbox/48745dbc-b7a1-40d5-aec7-818274814291.json create mode 100644 docs/outstanding-issues-inbox/63e7cebb-9a42-4c98-9520-9a1b7f5874a8.json create mode 100644 docs/outstanding-issues-inbox/71399b63-cce0-46d4-9fc3-c5fdc617289d.json diff --git a/docs/outstanding-issues-inbox/44e79b9e-5530-4b14-a431-b7a87683cf30.json b/docs/outstanding-issues-inbox/44e79b9e-5530-4b14-a431-b7a87683cf30.json new file mode 100644 index 0000000000..c16db7dd74 --- /dev/null +++ b/docs/outstanding-issues-inbox/44e79b9e-5530-4b14-a431-b7a87683cf30.json @@ -0,0 +1,14 @@ +{ + "version": 2, + "id": "44e79b9e-5530-4b14-a431-b7a87683cf30", + "createdOn": "2026-08-17", + "action": "add", + "payload": { + "pri": "P3", + "type": "task", + "summary": "Owner decision: enable RAG_TELEMETRY_EXTENDED (verification_latency_ms projection) in production once a dashboard consumer exists", + "detail": "Packet S5 (PR #2056, merge 093f9340c) landed the B1 telemetry gap assessment: the one proven gap is verification_latency_ms, now persisted behind RAG_TELEMETRY_EXTENDED (typed, default false) via the allow-listed projection module with canary-absence tests. Enabling it in production is an owner decision gated on a dashboard consumer existing (no consumer today), and is a Railway env change (provider-backed, explicit approval; rollback = set false). Next: when a dashboard question needs verification latency, set RAG_TELEMETRY_EXTENDED=true on the Database service after confirming the canary-absence tests are still green on main. Stop: do not enable speculatively; do not add unproven fields.", + "source": "RAG programme coordinator, packet S5 (PR #2056) follow-ups, 2026-08-17", + "issueUlid": "01M08C1AH72AB2NJRQFCXG3446" + } +} diff --git a/docs/outstanding-issues-inbox/48745dbc-b7a1-40d5-aec7-818274814291.json b/docs/outstanding-issues-inbox/48745dbc-b7a1-40d5-aec7-818274814291.json new file mode 100644 index 0000000000..4531eaaf96 --- /dev/null +++ b/docs/outstanding-issues-inbox/48745dbc-b7a1-40d5-aec7-818274814291.json @@ -0,0 +1,14 @@ +{ + "version": 2, + "id": "48745dbc-b7a1-40d5-aec7-818274814291", + "createdOn": "2026-08-17", + "action": "add", + "payload": { + "pri": "P2", + "type": "issue", + "summary": "Adversarial divergence (S5 harness pin): scope-other-owner-document — abstains in substance but the review fallback still cites in-scope evidence", + "detail": "Pinned in tests/rag-adversarial-harness.test.ts KNOWN_DIVERGENCES (self-expiring). Observed shape: grounded false, confidence unsupported, but cited chunk ids [syn-scope-owner-a] — the answer correctly abstains from the other-owner document, yet the review fallback attaches an in-scope citation to an unsupported answer. Fixture: scripts/fixtures/rag-adversarial-cases.v1.json case scope-other-owner-document (category scope_or_tenant). Tenancy/no-read invariant held (the other-owner content is never read). Next: decide whether an unsupported abstention may carry any citation; if not, strip citations on the abstention path (RAG-surface change; own PR; harness pin flips; canary pair). Stop: do not delete the pin without the behaviour change.", + "source": "RAG programme coordinator, packet S5 (PR #2056) follow-ups, 2026-08-17", + "issueUlid": "01M08C19VRC2D9JFG95HAJRFR7" + } +} diff --git a/docs/outstanding-issues-inbox/63e7cebb-9a42-4c98-9520-9a1b7f5874a8.json b/docs/outstanding-issues-inbox/63e7cebb-9a42-4c98-9520-9a1b7f5874a8.json new file mode 100644 index 0000000000..50c1358638 --- /dev/null +++ b/docs/outstanding-issues-inbox/63e7cebb-9a42-4c98-9520-9a1b7f5874a8.json @@ -0,0 +1,14 @@ +{ + "version": 2, + "id": "63e7cebb-9a42-4c98-9520-9a1b7f5874a8", + "createdOn": "2026-08-17", + "action": "add", + "payload": { + "pri": "P2", + "type": "issue", + "summary": "Adversarial divergence (S5 harness pin): scope-guessed-chunk-id — review fallback returns a grounded source pointer echoing the query instead of refusing", + "detail": "Pinned in tests/rag-adversarial-harness.test.ts KNOWN_DIVERGENCES (self-expiring). Observed shape: grounded true, cited [syn-scope-guess-a], and the guessed (never-retrieved) chunk id syn-not-retrieved-zzz is never resolved into content — the no-read invariant holds — but the review fallback returns a grounded source pointer that echoes the query text rather than refusing the guessed-id request. Fixture: scripts/fixtures/rag-adversarial-cases.v1.json case scope-guessed-chunk-id (category scope_or_tenant). Next: decide whether a query naming an unretrieved chunk id should refuse rather than fall back to a source pointer (RAG-surface change; own PR; harness pin flips; canary pair). Stop: do not delete the pin without the behaviour change.", + "source": "RAG programme coordinator, packet S5 (PR #2056) follow-ups, 2026-08-17", + "issueUlid": "01M08C1A7XNTAV3DNZ50RZ2C05" + } +} diff --git a/docs/outstanding-issues-inbox/71399b63-cce0-46d4-9fc3-c5fdc617289d.json b/docs/outstanding-issues-inbox/71399b63-cce0-46d4-9fc3-c5fdc617289d.json new file mode 100644 index 0000000000..34f61b8625 --- /dev/null +++ b/docs/outstanding-issues-inbox/71399b63-cce0-46d4-9fc3-c5fdc617289d.json @@ -0,0 +1,14 @@ +{ + "version": 2, + "id": "71399b63-cce0-46d4-9fc3-c5fdc617289d", + "createdOn": "2026-08-17", + "action": "add", + "payload": { + "pri": "P2", + "type": "issue", + "summary": "Adversarial divergence (S5 harness pin): cite-mismatched-attribution — offline document-match listing cites every retrieved document, not only the claim-bearing one", + "detail": "Pinned in tests/rag-adversarial-harness.test.ts KNOWN_DIVERGENCES (self-expiring: the harness asserts the normative B0 fixture contract still FAILS; when behaviour reaches the fixture expectation the pin goes red and must be deleted). Observed shape: cited chunk ids [syn-cite-attrib-a, syn-cite-attrib-b], grounded true, answerQualityTier source_only — the document-match listing attributes both retrieved documents although only one carries the claim. Fixture: scripts/fixtures/rag-adversarial-cases.v1.json case cite-mismatched-attribution (category citation_fabrication). Safety invariants (network, budget, canary absence, forbidden substrings, tenancy) hold; only citation attribution precision diverges. Next: decide whether the document-match listing should cite only claim-bearing documents (RAG-surface change; own PR; RAG impact line; offline harness proves the pin flips; canary pair). Stop: do not delete the pin without the behaviour change; do not weaken the fixture.", + "source": "RAG programme coordinator, packet S5 (PR #2056) follow-ups, 2026-08-17", + "issueUlid": "01M08C19FYVXB8XATJE3TYZJMS" + } +} diff --git a/docs/rag-improvement/COORDINATION.md b/docs/rag-improvement/COORDINATION.md index 0a22f53173..37a286a794 100644 --- a/docs/rag-improvement/COORDINATION.md +++ b/docs/rag-improvement/COORDINATION.md @@ -176,14 +176,17 @@ Always a fresh owner ask, every single time: - **Done (2026-08-17):** #1895, #1899, #1908 (2026-08-13); **#2022 S1** (squash `2bd146eed`); **#2024 D1** (`78fe906b8`); **#2035 S1b** (merge `92f7618c0`); **#2036 S4** (`f5b093291`); - **#2037 #212 T4** (`1726537b7`); **#2045 R0 reconcile** (28 requests; #212 closed). Canary + **#2037 #212 T4** (`1726537b7`); **#2045 R0 reconcile** (28 requests; #212 closed); **#2048 D2** (S1d packet); **#2056 S5** (merge `093f9340c`: B1 `verification_latency_ms` behind `RAG_TELEMETRY_EXTENDED`, B2 `eval:rag:adversarial:offline` with 3 self-expiring divergence pins). Canary pairs: S1 baseline 32025082010 → S1b post 32039841070, recall 1.0/1.0, zero per-case rr regressions, answer gate 45/45. The first S1b post run (32038751592) was red on one non-golden case; root-caused to the finalizer gap-recovery hole → packet S1d, not S1b. - **Owner decisions:** R1 before S2; governance Option B; S1d lands before S2. - **Wave 1 (dispatch now, parallel, disjoint files):** S1c (`rag-claim-support.ts`), S1d (`rag-extractive-answer.ts` finalizer / `rag.ts` fast-failure block), G1 (`types.ts`, - `answer-stream-contract.ts`, `rag-row-contracts.ts`), S5 (harness/telemetry), S6 (`eval/docling/`). + `answer-stream-contract.ts`, `rag-row-contracts.ts`), S6 (`eval/docling/`). S5 is merged. + S5 follow-ups queued as inbox requests: three adversarial divergence pins + (`cite-mismatched-attribution`, `scope-other-owner-document`, `scope-guessed-chunk-id`) and + the owner decision on enabling `RAG_TELEMETRY_EXTENDED` in production. Prompts: HANDOVER §7. Merge order for the RAG-behaviour ones: S1c and S1d each get their own canary pair; merge one, run its canary, then the next. - **Wave 2:** S2 (+S2b) after S1c and S1d canaries are green. **Wave 3:** S3; S7+ owner decisions. diff --git a/docs/rag-improvement/HANDOVER.md b/docs/rag-improvement/HANDOVER.md index 75ac30fd79..006d27df64 100644 --- a/docs/rag-improvement/HANDOVER.md +++ b/docs/rag-improvement/HANDOVER.md @@ -77,11 +77,11 @@ generation-quality verdict on fallback`), merged 2026-08-13 — structured | S1c | A1 residuals R2 + R3: claim-support strictness | `claude/rag-a1-r2-r3-claim-support-` | — | Ready — dispatch now (S1b merged, canary pair green 2026-08-17) | needs canary pair | | S1d | A1 final-gate gap recovery: hedged cited low-confidence fast answers must recover extractively, not collapse to a citation-free `provider_source_gap` | `claude/rag-a1-final-gate-gap-recovery-` | — | Ready — dispatch now; parallel-safe with S1c (different file); lands before S2 | needs canary pair; evidence: canary runs 32038751592 (red) vs 32039841070 (green), 3/3 live probes on the extractive branch | | G1 | Governance: provenance tag for document-summary rows (Option B) | `claude/rag-g1-document-context-origin-` | — | Ready — disjoint; owner decided Option B 2026-08-17 | no canary (no behaviour change) | -| S2 | A2 (+A3): composition menu + moderate length | `claude/rag-a2-composition-` | — | Blocked on S1b + S1c | canary pair + `eval:answer-quality` + Gate E | +| S2 | A2 (+A3): composition menu + moderate length | `claude/rag-a2-composition-` | — | Blocked on S1c + S1d (each merged with its own green canary pair) | canary pair + `eval:answer-quality` + Gate E | | S2b | A3: moderate length (if separate review needed) | `claude/rag-a3-length-` | — | Blocked on S2 | — | | S3 | A4: follow-up suggestion refinement | `claude/rag-a4-follow-ups-` | — | Blocked on S2 + S2b | — | | S4 | B0: adversarial fixtures + baseline + register | `claude/packet-s4-adversarial-fixtures-5ho5tp` | #2036 | Merged 2026-08-17 (squash `f5b093291`) | Offline only: `check:rag:adversarial-fixtures` 24 cases / 8 categories / 6 canaries; `eval:rag:offline` 24 suites, 597 tests. Baseline `scripts/fixtures/rag-adversarial-baseline.v1.json` marks the three provider-backed gates `pending_owner_run` | -| S5 | B1+B2: telemetry assessment + offline harness | `claude/s5-rag-telemetry-harness-2wvis7` | #2056 | PR open 2026-08-17 | Offline only: `eval:rag:adversarial:offline` 25/25 (24 cases + canary-free report; 3 divergences pinned in `KNOWN_DIVERGENCES`); B1 gap = `verification_latency_ms` behind `RAG_TELEMETRY_EXTENDED` (default false); canary-absence tests green; 44/44 denominator reconciled | +| S5 | B1+B2: telemetry assessment + offline harness | `claude/s5-rag-telemetry-harness-2wvis7` | #2056 | Merged 2026-08-17 (merge `093f9340c`); post-merge canary run 32049952885 | Offline only: `eval:rag:adversarial:offline` 25/25 (24 cases + canary-free report; 3 divergences pinned in `KNOWN_DIVERGENCES`); B1 gap = `verification_latency_ms` behind `RAG_TELEMETRY_EXTENDED` (default false); canary-absence tests green; 44/44 denominator reconciled | | S6 | B3: Docling lab benchmark | `claude/rag-b3-docling-lab-` | — | Ready — dispatch now (S4 merged) | — | | S7+ | B4 shadow / B5 Ragas / B6 reranker / B7 DSPy | — | — | Gated — owner decision | — | | #212 T1–T3 | Runtime row contracts (rag.ts, rag-candidate-sources.ts, src/app/api) — sibling stream sharing `src/lib/rag/**` | — | #1946 / #1981 / #2023 | Merged (T3 squash `440a34f71` 2026-08-17) | see the #212 ledger row; RAG surface complete for the cast class | From cb73bba549db576129578d12a102bb755728d70d Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Tue, 18 Aug 2026 01:24:58 +0800 Subject: [PATCH 2/2] docs(ledger): record the D3 S5 follow-up review Co-Authored-By: Claude Fable 5 --- ...fcc71dfc33aafe348fd300805816d36dc8164eed9ed3fc356cf.record.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 docs/branch-review-records/b2ecb1cee8bf7fcc71dfc33aafe348fd300805816d36dc8164eed9ed3fc356cf.record.md diff --git a/docs/branch-review-records/b2ecb1cee8bf7fcc71dfc33aafe348fd300805816d36dc8164eed9ed3fc356cf.record.md b/docs/branch-review-records/b2ecb1cee8bf7fcc71dfc33aafe348fd300805816d36dc8164eed9ed3fc356cf.record.md new file mode 100644 index 0000000000..beb8f133d7 --- /dev/null +++ b/docs/branch-review-records/b2ecb1cee8bf7fcc71dfc33aafe348fd300805816d36dc8164eed9ed3fc356cf.record.md @@ -0,0 +1 @@ +| 2026-08-17 | claude/rag-d3-s5-followups | 60e61bb9b89e2a62cefa7e89e8f6497c09e8a900 | docs/rag-improvement S5 follow-ups: HANDOVER S5/S2 rows, COORDINATION §7, 4 inbox requests (docs-only) | docs-only; S5 landed; follow-ups queued | verify:pr-local docs scope |