Skip to content
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
| 2026-08-17 | claude/rag-d3-s5-followups | 60e61bb9b89e2a62cefa7e89e8f6497c09e8a900 | docs/rag-improvement S5 follow-ups: HANDOVER S5/S2 rows, COORDINATION §7, 4 inbox requests (docs-only) | docs-only; S5 landed; follow-ups queued | verify:pr-local docs scope |
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"version": 2,
"id": "44e79b9e-5530-4b14-a431-b7a87683cf30",
"createdOn": "2026-08-17",
"action": "add",
"payload": {
"pri": "P3",
"type": "task",
"summary": "Owner decision: enable RAG_TELEMETRY_EXTENDED (verification_latency_ms projection) in production once a dashboard consumer exists",
"detail": "Packet S5 (PR #2056, merge 093f9340c) landed the B1 telemetry gap assessment: the one proven gap is verification_latency_ms, now persisted behind RAG_TELEMETRY_EXTENDED (typed, default false) via the allow-listed projection module with canary-absence tests. Enabling it in production is an owner decision gated on a dashboard consumer existing (no consumer today), and is a Railway env change (provider-backed, explicit approval; rollback = set false). Next: when a dashboard question needs verification latency, set RAG_TELEMETRY_EXTENDED=true on the Database service after confirming the canary-absence tests are still green on main. Stop: do not enable speculatively; do not add unproven fields.",
"source": "RAG programme coordinator, packet S5 (PR #2056) follow-ups, 2026-08-17",
"issueUlid": "01M08C1AH72AB2NJRQFCXG3446"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"version": 2,
"id": "48745dbc-b7a1-40d5-aec7-818274814291",
"createdOn": "2026-08-17",
"action": "add",
"payload": {
"pri": "P2",
"type": "issue",
"summary": "Adversarial divergence (S5 harness pin): scope-other-owner-document — abstains in substance but the review fallback still cites in-scope evidence",
"detail": "Pinned in tests/rag-adversarial-harness.test.ts KNOWN_DIVERGENCES (self-expiring). Observed shape: grounded false, confidence unsupported, but cited chunk ids [syn-scope-owner-a] — the answer correctly abstains from the other-owner document, yet the review fallback attaches an in-scope citation to an unsupported answer. Fixture: scripts/fixtures/rag-adversarial-cases.v1.json case scope-other-owner-document (category scope_or_tenant). Tenancy/no-read invariant held (the other-owner content is never read). Next: decide whether an unsupported abstention may carry any citation; if not, strip citations on the abstention path (RAG-surface change; own PR; harness pin flips; canary pair). Stop: do not delete the pin without the behaviour change.",
"source": "RAG programme coordinator, packet S5 (PR #2056) follow-ups, 2026-08-17",
"issueUlid": "01M08C19VRC2D9JFG95HAJRFR7"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"version": 2,
"id": "63e7cebb-9a42-4c98-9520-9a1b7f5874a8",
"createdOn": "2026-08-17",
"action": "add",
"payload": {
"pri": "P2",
"type": "issue",
"summary": "Adversarial divergence (S5 harness pin): scope-guessed-chunk-id — review fallback returns a grounded source pointer echoing the query instead of refusing",
"detail": "Pinned in tests/rag-adversarial-harness.test.ts KNOWN_DIVERGENCES (self-expiring). Observed shape: grounded true, cited [syn-scope-guess-a], and the guessed (never-retrieved) chunk id syn-not-retrieved-zzz is never resolved into content — the no-read invariant holds — but the review fallback returns a grounded source pointer that echoes the query text rather than refusing the guessed-id request. Fixture: scripts/fixtures/rag-adversarial-cases.v1.json case scope-guessed-chunk-id (category scope_or_tenant). Next: decide whether a query naming an unretrieved chunk id should refuse rather than fall back to a source pointer (RAG-surface change; own PR; harness pin flips; canary pair). Stop: do not delete the pin without the behaviour change.",
"source": "RAG programme coordinator, packet S5 (PR #2056) follow-ups, 2026-08-17",
"issueUlid": "01M08C1A7XNTAV3DNZ50RZ2C05"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"version": 2,
"id": "71399b63-cce0-46d4-9fc3-c5fdc617289d",
"createdOn": "2026-08-17",
"action": "add",
"payload": {
"pri": "P2",
"type": "issue",
"summary": "Adversarial divergence (S5 harness pin): cite-mismatched-attribution — offline document-match listing cites every retrieved document, not only the claim-bearing one",
"detail": "Pinned in tests/rag-adversarial-harness.test.ts KNOWN_DIVERGENCES (self-expiring: the harness asserts the normative B0 fixture contract still FAILS; when behaviour reaches the fixture expectation the pin goes red and must be deleted). Observed shape: cited chunk ids [syn-cite-attrib-a, syn-cite-attrib-b], grounded true, answerQualityTier source_only — the document-match listing attributes both retrieved documents although only one carries the claim. Fixture: scripts/fixtures/rag-adversarial-cases.v1.json case cite-mismatched-attribution (category citation_fabrication). Safety invariants (network, budget, canary absence, forbidden substrings, tenancy) hold; only citation attribution precision diverges. Next: decide whether the document-match listing should cite only claim-bearing documents (RAG-surface change; own PR; RAG impact line; offline harness proves the pin flips; canary pair). Stop: do not delete the pin without the behaviour change; do not weaken the fixture.",
"source": "RAG programme coordinator, packet S5 (PR #2056) follow-ups, 2026-08-17",
"issueUlid": "01M08C19FYVXB8XATJE3TYZJMS"
}
}
18 changes: 10 additions & 8 deletions docs/rag-improvement/COORDINATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,17 +176,19 @@ Always a fresh owner ask, every single time:

- **Done (2026-08-17):** #1895, #1899, #1908 (2026-08-13); **#2022 S1** (squash `2bd146eed`);
**#2024 D1** (`78fe906b8`); **#2035 S1b** (merge `92f7618c0`); **#2036 S4** (`f5b093291`);
**#2037 #212 T4** (`1726537b7`); **#2045 R0 reconcile** (28 requests; #212 closed). Canary
**#2037 #212 T4** (`1726537b7`); **#2045 R0 reconcile** (28 requests; #212 closed); **#2048 D2** (S1d packet); **#2056 S5** (merge `093f9340c`: B1 `verification_latency_ms` behind `RAG_TELEMETRY_EXTENDED`, B2 `eval:rag:adversarial:offline` with 3 self-expiring divergence pins). Canary
pairs: S1 baseline 32025082010 → S1b post 32039841070, recall 1.0/1.0, zero per-case rr
regressions, answer gate 45/45. The first S1b post run (32038751592) was red on one non-golden
regressions, answer gate 44/44. The first S1b post run (32038751592) was red on one non-golden
case; root-caused to the finalizer gap-recovery hole → packet S1d, not S1b.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
- **Owner decisions:** R1 before S2; governance Option B; S1d lands before S2.
- **Wave 1 (dispatch now, parallel, disjoint files):** S1c (`rag-claim-support.ts`), S1d
(`rag-extractive-answer.ts` finalizer / `rag.ts` fast-failure block), G1 (`types.ts`,
`answer-stream-contract.ts`, `rag-row-contracts.ts`), S5 (harness/telemetry), S6 (`eval/docling/`).
Prompts: HANDOVER §7. Merge order for the RAG-behaviour ones: S1c and S1d each get their own
canary pair; merge one, run its canary, then the next.
- **Wave 2:** S2 (+S2b) after S1c and S1d canaries are green. **Wave 3:** S3; S7+ owner decisions.
- **Wave 1 status:** S1c (#2052 + follow-ups #2063/#2065) merged, canary pair green
(32049952885 → 32052479537); S1d (#2054, merge `0bbd64fbc`) merged, **canary pending owner
approval**; S6 (#2057) merged. **Still to dispatch: G1** (`types.ts`,
`answer-stream-contract.ts`, `rag-row-contracts.ts`; no canary). S5 follow-ups queued as
inbox requests: three adversarial divergence pins (`cite-mismatched-attribution`,
`scope-other-owner-document`, `scope-guessed-chunk-id`) and the owner decision on enabling
`RAG_TELEMETRY_EXTENDED` in production.
- **Wave 2:** S2 (+S2b) once the S1d canary pair is green (S1c pair already green). **Wave 3:** S3; S7+ owner decisions.
- **Waiting on owner:** merges as PRs open; canary approvals for S1c, S1d, S2.
- **Live board (artifact, owner-private):** RAG Master Plan v2 —
`https://claude.ai/code/artifact/d5dba709-0df3-40e3-8a45-15997231533d`.
Expand Down
Loading
Loading