Skip to content

feat(eval): isolated Docling lab benchmark harness + Gate B decision-record template (packet S6 / B3) #2057

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
BigSimmo merged 7 commits into from
Aug 17, 2026
Merged

feat(eval): isolated Docling lab benchmark harness + Gate B decision-record template (packet S6 / B3) #2057

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
798725d
feat(eval): add the isolated Docling lab benchmark harness (packet S6…
claude Aug 17, 2026
356948a
docs(rag): record packet S6 handoff — HANDOVER S6 row + review-ledger…
claude Aug 17, 2026
be0a679
Merge origin/main into claude/packet-s6-docling-lab-d6foa6 (resolve S…
claude Aug 17, 2026
0a899d2
Merge branch 'main' into claude/packet-s6-docling-lab-d6foa6
BigSimmo Aug 17, 2026
e06967e
Merge branch 'main' into claude/packet-s6-docling-lab-d6foa6
BigSimmo Aug 17, 2026
0531220
Merge branch 'main' into claude/packet-s6-docling-lab-d6foa6
BigSimmo Aug 17, 2026
8bc9950
Merge branch 'main' into claude/packet-s6-docling-lab-d6foa6
BigSimmo Aug 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 58 additions & 0 deletions .github/workflows/docling-lab.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# Docling lab benchmark — manual dispatch only (docs/rag-improvement/README.md §B3,
# HANDOVER packet S6). Deliberately NOT part of pr-required or any pull_request
# trigger: the benchmark verdict is a separate owner-reviewed run, and this workflow
# exists so that run has a reproducible, sandboxed home.
#
# The sandbox contract lives in eval/docling/run-lab.sh: the docker build stage has
# network for hash-verified installs and the docling model prefetch; the benchmark
# itself runs with --network=none, non-root, and CPU/memory/pids/wall-clock/output
# limits. Only the aggregate report (numeric allowlist + canary-leak scan) is
# uploaded — raw per-document extraction output never leaves the runner.
name: Docling lab benchmark

on:
workflow_dispatch:

concurrency:
group: docling-lab
cancel-in-progress: false

permissions:
contents: read

jobs:
benchmark:
name: Sandboxed extraction benchmark
runs-on: ubuntu-24.04
timeout-minutes: 90
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- name: Setup Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version-file: ".nvmrc"
cache: npm
cache-dependency-path: package-lock.json

- name: Install dependencies
run: npm ci --include=dev

- name: Validate lab contract before spending a build
run: node eval/docling/report/build-report.mjs --validate-only

- name: Build sandbox image and run benchmark
run: bash eval/docling/run-lab.sh

# Success-only by design: on failure there is no report, and raw per-document
# output must never be uploaded from the runner.
- name: Upload aggregate report
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: docling-lab-report-${{ github.run_id }}
path: eval/docling/out/report/
if-no-files-found: error
retention-days: 14
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,9 @@ test-output.txt
/sample-documents/
/tmp/

# docling lab run output (generated fixtures, raw measurements, reports)
/eval/docling/out/

# python
__pycache__/
.pytest_cache/
Expand Down
1 change: 1 addition & 0 deletions ...ords/6ea4a18560d833a6c86edeeebf1828e2a11801d2e55027be54889272b93e7a96.record.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
| 2026-08-17 | claude/packet-s6-docling-lab-d6foa6 | 798725d04c85070f90c7496c5c7808713dd2d2a5 | eval/docling isolated Docling lab benchmark harness + Gate B decision-record template (packet S6/B3, PR #2057) | PR #2057 open — harness only, no benchmark verdict; hard boundaries respected (worker/extractors/database untouched) | verify:pr-local heavy plan failed:(none); check:docling-lab passed (36 fixtures/10 hostile/6 canaries); docling-lab-contract test 20/20; check:github-actions passed; legacy engine smoke 46 docs 10/10 hostile contained canary-clean |
27 changes: 14 additions & 13 deletions docs/codebase-index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,19 +34,20 @@ Structured map for AI agents and onboarding. For live routes, see `docs/site-map

Smaller top-level directories that are easy to miss:

| Path | Purpose |
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `data/` | Committed clinical **snapshot exports** loaded at runtime by `src/lib/` (differentials, forms, medications, services, specifiers). Regenerate via the matching `scripts/import-*-export.ts` / `build-*-index.mjs`; do not hand-edit. Distinct from `src/data/`, which holds hand-authored static content. |
| `eslint-rules/` | Repo-specific lint rules enforced by `npm run lint` (button wiring, hardcoded hex, type/icon scale, z-index ladder) |
| `mockups/` | Notes for the design-scratch routes under `src/app/mockups/` (the routes themselves 404 in production) |
| `plugins/` | `plugins/clinical-kb/` Codex plugin manifest and workflow skill |
| `.agents/` | Canonical single-word skill catalogue (`npm run skills`); `npm run check:skills` also validates Claude, Cursor, and plugin skill policies |
| `.claude/` | Claude Code agents, skills, hooks, settings — plus the `.claude/worktrees/` working copies |
| `.codex/` | Trusted Desktop/CLI config; tracked `config.toml` has disabled, secret-free Figma, Supabase, Railway, and Sentry MCP templates. Hosted ChatGPT/Codex apps are installed and authenticated separately; OAuth stays in the host credential store. |
| `.cursor/` | Cursor project rules and local-agent configuration |
| `.design-sync/` | Generated design-system package metadata, validation notes, and project-sync artifacts |
| `.githooks/` | Installed by `npm install`; `pre-push` runs `scripts/guard-push.mjs` (user-owned auto-merge preservation, format, drift staleness, static lint+typecheck, ledger write discipline) |
| `.vscode/` | Shared VS Code workspace recommendations and settings |
| Path | Purpose |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `data/` | Committed clinical **snapshot exports** loaded at runtime by `src/lib/` (differentials, forms, medications, services, specifiers). Regenerate via the matching `scripts/import-*-export.ts` / `build-*-index.mjs`; do not hand-edit. Distinct from `src/data/`, which holds hand-authored static content. |
| `eval/` | Isolated evaluation labs, outside the product/runtime dependency graph. `eval/docling/` is the sandboxed, dispatch-only Docling extraction benchmark (own hashed Python lock + venvs, egress-blocked Docker run, synthetic fixtures + hostile corpus, aggregate-only reports; `docs/rag-improvement/README.md` §B3) |
| `eslint-rules/` | Repo-specific lint rules enforced by `npm run lint` (button wiring, hardcoded hex, type/icon scale, z-index ladder) |
| `mockups/` | Notes for the design-scratch routes under `src/app/mockups/` (the routes themselves 404 in production) |
| `plugins/` | `plugins/clinical-kb/` Codex plugin manifest and workflow skill |
| `.agents/` | Canonical single-word skill catalogue (`npm run skills`); `npm run check:skills` also validates Claude, Cursor, and plugin skill policies |
| `.claude/` | Claude Code agents, skills, hooks, settings — plus the `.claude/worktrees/` working copies |
| `.codex/` | Trusted Desktop/CLI config; tracked `config.toml` has disabled, secret-free Figma, Supabase, Railway, and Sentry MCP templates. Hosted ChatGPT/Codex apps are installed and authenticated separately; OAuth stays in the host credential store. |
| `.cursor/` | Cursor project rules and local-agent configuration |
| `.design-sync/` | Generated design-system package metadata, validation notes, and project-sync artifacts |
| `.githooks/` | Installed by `npm install`; `pre-push` runs `scripts/guard-push.mjs` (user-owned auto-merge preservation, format, drift staleness, static lint+typecheck, ledger write discipline) |
| `.vscode/` | Shared VS Code workspace recommendations and settings |

**Do not commit:** `.next/`, `node_modules/`, `coverage/`, `.env*`, `sample-documents/`, logs.

Expand Down
4 changes: 2 additions & 2 deletions docs/rag-improvement/HANDOVER.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,8 +81,8 @@ generation-quality verdict on fallback`), merged 2026-08-13 — structured
| S2b | A3: moderate length (if separate review needed) | `claude/rag-a3-length-<suffix>` | — | Blocked on S2 | — |
| S3 | A4: follow-up suggestion refinement | `claude/rag-a4-follow-ups-<suffix>` | — | Blocked on S2 + S2b | — |
| S4 | B0: adversarial fixtures + baseline + register | `claude/packet-s4-adversarial-fixtures-5ho5tp` | #2036 | Merged 2026-08-17 (squash `f5b093291`) | Offline only: `check:rag:adversarial-fixtures` 24 cases / 8 categories / 6 canaries; `eval:rag:offline` 24 suites, 597 tests. Baseline `scripts/fixtures/rag-adversarial-baseline.v1.json` marks the three provider-backed gates `pending_owner_run` |
| S5 | B1+B2: telemetry assessment + offline harness | `claude/s5-rag-telemetry-harness-2wvis7` | #2056 | PR open 2026-08-17 | Offline only: `eval:rag:adversarial:offline` 25/25 (24 cases + canary-free report; 3 divergences pinned in `KNOWN_DIVERGENCES`); B1 gap = `verification_latency_ms` behind `RAG_TELEMETRY_EXTENDED` (default false); canary-absence tests green; 44/44 denominator reconciled |
| S6 | B3: Docling lab benchmark | `claude/rag-b3-docling-lab-<suffix>` | — | Ready — dispatch now (S4 merged) | — |
| S5 | B1+B2: telemetry assessment + offline harness | `claude/s5-rag-telemetry-harness-2wvis7` | #2056 | Merged 2026-08-17 (merge `093f934`) | Offline only: `eval:rag:adversarial:offline` 25/25 (24 cases + canary-free report; 3 divergences pinned in `KNOWN_DIVERGENCES`); B1 gap = `verification_latency_ms` behind `RAG_TELEMETRY_EXTENDED` (default false); canary-absence tests green; 44/44 denominator reconciled |
| S6 | B3: Docling lab benchmark | `claude/packet-s6-docling-lab-d6foa6` | #2057 | Open — awaiting review (2026-08-17) | Offline only: `check:docling-lab` 36 fixtures / 10 hostile / 6 canaries + Gate B template valid; `verify:pr-local` heavy plan failed:(none); contract test 20/20; legacy smoke 46 docs, 10/10 hostile contained, canary-clean report. Verdict is a separate owner dispatch of `docling-lab.yml` |
| S7+ | B4 shadow / B5 Ragas / B6 reranker / B7 DSPy | — | — | Gated — owner decision | — |
| #212 T1–T3 | Runtime row contracts (rag.ts, rag-candidate-sources.ts, src/app/api) — sibling stream sharing `src/lib/rag/**` | — | #1946 / #1981 / #2023 | Merged (T3 squash `440a34f71` 2026-08-17) | see the #212 ledger row; RAG surface complete for the cast class |
| #212 T4 | Runtime row contracts: `worker/main.ts` (11 casts) — sibling stream | `claude/ledger-212-tranche-4-worker-q3y6i4` | #2037 | Merged 2026-08-17 (squash `1726537b7`); #212 closed by reconcile PR #2045 | Governance Preflight complete; audit: 1 inbound cast (claim rows, per-row fail-soft) + 2 read-back param casts contracted, 9 outbound/interop left; closes #212 (inbox `done` queued in the PR) |
Expand Down
Loading
Loading