Skip to content

feat(eval): Gate E blinded before/after capture and pairing tooling (#E0N0QC) #2208

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
BigSimmo merged 7 commits into from
Aug 21, 2026
Merged

feat(eval): Gate E blinded before/after capture and pairing tooling (#E0N0QC) #2208

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
41bca3c
feat(eval): Gate E blinded before/after capture and pairing tooling (…
BigSimmo Aug 20, 2026
af8afeb
chore(gate-e): prettier formatting and generated docs counts
BigSimmo Aug 20, 2026
8c0f451
docs(ledger): review record for PR #2208
BigSimmo Aug 21, 2026
0fd6349
Merge branch 'main' into claude/gate-e-blinded-eval-b6076d
BigSimmo Aug 21, 2026
40d0f86
fix(gate-e): address PR #2208 review — cited-source rendering, pair-l…
BigSimmo Aug 21, 2026
61a95b9
Merge remote branch sync (main merged into PR #2208 branch)
BigSimmo Aug 21, 2026
985b800
Merge branch 'main' into claude/gate-e-blinded-eval-b6076d
BigSimmo Aug 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ...ords/2d2b3c78d86d6c1599ede00358ad6ffc66a5adba0563fd5dee8656f0ef55ea87.record.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
| 2026-08-21 | claude/gate-e-blinded-eval-b6076d | af8afeba09916f44462ba53609560f0118ca0940 | Gate E blinded-eval capture tooling: eval-answer-quality --extra-cases + gate-outcome dump fields, new scripts/blind-answer-pairs.ts build/unblind, tests, docs (PR #2208) | PR #2208 opened; offline-only, no retrieval behaviour change (#E0N0QC); paid v18-vs-v19 capture pending owner approval | focused vitest 36/36; offline contract 26 suites/627; adversarial fixtures 24 recorded + harness 25/25; check:rag:fixtures 36/26; typecheck 0; lint 0; docs checks green; full suite: load-flake timeouts only, disjoint sets, unrelated files |
38 changes: 38 additions & 0 deletions docs/rag-improvement/HANDOVER.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,11 @@ generation-quality verdict on fallback`), merged 2026-08-13 — structured
`answerSections.maxItems` 6, adversarial baseline re-captured for v19 (`baseline-record.md` §4). Its canary
pair 32100681177 -> 32111839806 is green, and `eval:answer-quality` was neutral within
nondeterminism (owner blinded read pending). **Track A is complete with S3 (A4).**
- **Gate E tooling landed (2026-08-21, offline-only; `#E0N0QC`):** `eval-answer-quality`
gained `--extra-cases` (owner capture-only questions) and gate-outcome dump fields, and
`scripts/blind-answer-pairs.ts` builds the blinded reading pack / verdict sheet /
assignment key and unblinds verdicts. The paid v18-vs-v19 capture and the owner's blinded
read are still pending — procedure in §2a below; every provider step needs owner approval.
- **Owner decisions 2026-08-17:** (1) **R1 before S2** — A2/A3 add answer length, and length
under the still-unbudgeted strong retry pushes more dosing queries into `provider_timeout`,
not fewer; (2) **governance Option B** for the document-summary `similarity: 1` question
Expand Down Expand Up @@ -94,6 +99,7 @@ generation-quality verdict on fallback`), merged 2026-08-13 — structured
| S6 | B3: Docling lab benchmark | `claude/packet-s6-docling-lab-d6foa6` | #2057 | Merged 2026-08-17 (merge `5a6418636`) | Offline only: `check:docling-lab` 36 fixtures / 10 hostile / 6 canaries + Gate B template valid; `verify:pr-local` heavy plan failed:(none); contract test 20/20; legacy smoke 46 docs, 10/10 hostile contained, canary-clean report. Verdict is a separate owner dispatch of `docling-lab.yml` |
| S6b | Gate B run: docling-lab dispatch, verdict, decision record | `claude/docling-gate-b-eval-5czgln` | (this PR) | Gate B **PASS** 2026-08-18 (evidence run 32176604314 at `8a92378`); four latent harness defects found+fixed en route (setuptools pin, libGL, torch.compile/no-toolchain, HTML-entity scoring) | All five gates pass at pre-agreed 0 pp margins: parse 36/36 both engines, exactness 162/162 both, table F1 parity at ceiling (agreed 0 pp target; fixtures.v2 hardness follow-up queued), hostile 10/10 contained / 0 crash / 0 canary echo, resources max 12.6 s P95 / 1.40 GiB vs 120 s / 6 GiB caps; record: `docs/rag-improvement/gate-b-decision-record-2026-08-18.{md,json}`, validated `--final` |
| S7 | B4: Docling worker shadow mode | `claude/docling-worker-shadow-mode-b6fa17` | #2170 | Merged 2026-08-19 (squash `5437c309f`), landed by content; default legacy — shadow is an operator flag (Railway); ledger #9DGA6R closed | Offline only: `WORKER_DOCUMENT_EXTRACTOR_MODE=legacy\ | shadow`(default legacy) +`WORKER_SHADOW_EXTRACTION_COHORT_PERCENT`(1–5, default 2) +`WORKER_DOCLING_PYTHON_BIN`; shadow runs after `commitDocumentIndexGeneration`, aggregate record in `documents.metadata.shadow_extraction` via the existing final metadata merge, no chunk/embedding/index/table-fact/`document_index_quality`writes; bounded 120 s / 40 pages / 1 process; vitest`tests/worker-shadow-extraction.test.ts`19/19 + Python unittest 7/7; caveats: table-heavy leg passed at parity-on-ceiling (fixtures.v2 first), eager-mode latency budgeted by the three bounds; rollback`WORKER_DOCUMENT_EXTRACTOR_MODE=legacy` |
| Gate E | Blinded before/after read tooling (capture `--extra-cases` + gate-outcome dump fields; `blind-answer-pairs.ts` build/unblind) | `claude/gate-e-blinded-eval-b6076d` | (this PR) | Tooling merged, offline-only; paid v18 (4ea310e48) vs v19 capture and the owner's blinded read still pending (`#E0N0QC`; procedure in §2a) | Offline only: blind-answer-pairs + eval-answer-quality focused suites 36/36 (blinding swap-proof, key round-trip, byte-stable builds); eval:rag:offline + eval:rag:adversarial:offline unchanged; no canary — no runtime behaviour change |
| S8+ | B5 Ragas / B6 reranker / B7 DSPy | — | — | Still gated — owner decision | — |
| #212 T1–T3 | Runtime row contracts (rag.ts, rag-candidate-sources.ts, src/app/api) — sibling stream sharing `src/lib/rag/**` | — | #1946 / #1981 / #2023 | Merged (T3 squash `440a34f71` 2026-08-17) | see the #212 ledger row; RAG surface complete for the cast class |
| #212 T4 | Runtime row contracts: `worker/main.ts` (11 casts) — sibling stream | `claude/ledger-212-tranche-4-worker-q3y6i4` | #2037 | Merged 2026-08-17 (squash `1726537b7`); #212 closed by reconcile PR #2045 | Governance Preflight complete; audit: 1 inbound cast (claim rows, per-row fail-soft) + 2 read-back param casts contracted, 9 outbound/interop left; closes #212 (inbox `done` queued in the PR) |
Expand All @@ -102,6 +108,38 @@ Update rule: the session that opens a packet's PR edits its row (branch, PR numb
state) in the same PR. A later session updating another packet may also correct stale rows
it can verify from GitHub/git state. Keep rows one line.

## 2a. Gate E blinded read — owner procedure

Compares prompt v18 (commit `4ea310e48`, the S2 baseline canary half) against v19 (current
`main`) on the 30 `answerQualityEvalCases` plus up to ~10 owner-chosen live questions.
`scripts/eval-answer-quality.ts` and `scripts/eval-utils.ts` are byte-identical between
`4ea310e48` and current `main` and every new dump field is defensive, so the updated script
runs unmodified in a v18 worktree. Steps 3 and 4 are **provider-backed and paid** (OpenAI +
Supabase, ~80 cache-bypassed answers, est ~$4–8); everything else is offline.

1. _(offline)_ Author `.local/gate-e/extra-cases.json` in the main checkout:
`{"questions":[{"id":"live-01","question":"..."}, ...]}` — ids `live-*`; the tool rejects
collisions with the fixed `quality-*` ids.
2. _(offline; handles credentials)_ `git worktree add ..\gate-e-v18 4ea310e48`, then in it
`npm ci --include=dev`; copy in from main: `scripts\eval-answer-quality.ts`,
`.local\gate-e\extra-cases.json`, and `.env.local` (never commit; delete before removing
the worktree).
3. **PROVIDER (paid)** — in the v18 worktree:
`npm run eval:answer-quality -- --json --extra-cases .local/gate-e/extra-cases.json --dump-answers output/gate-e/dump-v18.json > output/gate-e/summary-v18.json`
4. **PROVIDER (paid)** — in the main checkout at current HEAD: same command with
`dump-v19` in both paths.
5. _(offline)_ Copy the v18 dump into the main checkout's `output/gate-e/`, then:
`node scripts/run-tsx.mjs scripts/blind-answer-pairs.ts build --before output/gate-e/dump-v18.json --after output/gate-e/dump-v19.json --before-label v18-4ea310e48 --after-label v19-<HEAD-sha> --out-dir output/gate-e/blind`
— check the printed pair count (expect 30 + the extra questions) and any unpaired ids.
6. _(offline, human)_ Open **only** `output/gate-e/blind/reading-pack.md`; record every
verdict in `verdict-sheet.md` (`verdict=A|B|tie|neither`, optional `notes=`). Do not open
`assignment-key.json` until every verdict is recorded.
7. _(offline)_
`node scripts/run-tsx.mjs scripts/blind-answer-pairs.ts unblind --key output/gate-e/blind/assignment-key.json --verdicts output/gate-e/blind/verdict-sheet.md --out output/gate-e/blind/unblinded-report.md`
8. _(offline)_ Record the Gate E verdict (tallies + dump digests from the key) in the S2 and
Gate E rows above via a docs PR + ledger append; delete `.env.local` from the v18
worktree, then `git worktree remove ..\gate-e-v18`.

## 3. Session packets

Every packet inherits the standing rules in §6. "Done" for a packet always ends at an open
Expand Down
9 changes: 7 additions & 2 deletions docs/scripts-index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Scripts index

Curated map of `scripts/` (249 files) and the `package.json` script surface (252 entries),
Curated map of `scripts/` (250 files) and the `package.json` script surface (252 entries),
grouped by purpose. This is orientation, not an exhaustive per-file listing — the authoritative
command list is `package.json`, and `npm run docs:check-scripts` verifies every `npm run <x>`
referenced in docs resolves to a real script. `npm run docs:update` refreshes the exact counts above.
Expand Down Expand Up @@ -97,7 +97,12 @@ validation of the synthetic adversarial fixture dataset and its baseline record;
`eval-rag-adversarial-offline.mjs` (packet B2: fixture validation then the offline Vitest
adversarial harness `tests/rag-adversarial-harness.test.ts`; `npm run eval:rag:adversarial:offline`,
routed by `ci-change-scope.mjs` to RAG-surface PRs only; fails closed on missing fixture,
network attempt, or round-trip budget breach).
network attempt, or round-trip budget breach),
`blind-answer-pairs.ts` (Gate E offline blinded A/B pairing over two
`eval-answer-quality --dump-answers` artefacts — `build` emits reading-pack/verdict-sheet/
assignment-key under `output/` or `.local/` only, `unblind` resolves recorded verdicts back to
version labels; pure file transformation, no provider access, node-builtin imports only;
`/issues` `#E0N0QC`; run via `node scripts/run-tsx.mjs scripts/blind-answer-pairs.ts`).
Golden fixtures:
`scripts/fixtures/rag-retrieval-golden.json`, `scripts/fixtures/assertion-golden.json`.
Adversarial fixtures: `scripts/fixtures/rag-adversarial-cases.v1.json` (+ its schema) and
Expand Down
Loading
Loading