feat(rag): adversarial fixture contract, validator, baseline record and data-flow register (packet S4 / B0)

[BigSimmo]

Summary

Implements programme packet S4 (B0) from docs/rag-improvement/HANDOVER.md, per docs/rag-improvement/README.md §B0. Everything here is offline: no retrieval, ranking, selection, generation, or answer-contract code is touched.

• Adversarial fixture dataset. scripts/fixtures/rag-adversarial-cases.v1.json holds 24 synthetic cases, three in each of the eight categories the guide names: prompt injection, citation fabrication, unsupported claim, empty/conflicting evidence, scope/tenant, provider failure, adversarial metadata, and cost/timeout abuse. Every excerpt, document title, and drug name is invented — the fixtures use made-up agents (zolinapine, meraflox, cortavine) so no real clinical text can be mistaken for a source. scripts/fixtures/rag-adversarial-cases.schema.json publishes the shape.
• PHI-like canaries. Six registered sentinel strings (CANARY-PHI-…) name a patient name, MRN, date of birth, phone number, address, and email address, planted in case text across four categories. They carry their PHI category in words and no digits at all — see the secret-scanner note below — so each is unique, exactly greppable, and conspicuously not a real identifier.
• Validator. New command npm run check:rag:adversarial-fixtures, backed by scripts/check-rag-adversarial-fixtures.mjs (CLI) and scripts/rag-adversarial-contract.mjs (pure functions). It is deterministic and network-free: it parses three repository files and prints counts. It has no dependencies and adds none. The existing check:rag:fixtures is untouched.
• Canary containment is the load-bearing check. The report the command prints is scanned for every registered canary literal before it is written to stdout, and a hit exits 1 without naming the token. The report is built from ids, counts, and gate statuses only — never excerpt text. Two further rules close the obvious leaks: a canary must be declared exactly when it is planted (both directions), and a canary may never be written into forbiddenOutputSubstrings, because a runner echoing that list would itself print the canary.
• Synthetic-only enforcement. Every excerpt title must begin with SYNTHETIC , and a denylist of real clinical source names (EMHS, RANZCP, NICE, Maudsley, Therapeutic Guidelines, and similar) is a hard validation failure rather than a warning.
• Baseline record. scripts/fixtures/rag-adversarial-baseline.v1.json carries the six-field report key — commit_sha, dataset_version, eval_config_version, model_version, embedding_version, index_version — whose field set and order are pinned by test, plus the prompt version (cross-checked against src/lib/rag/rag-versioning.ts at validation time) and semanticRerankEnabled: false. docs/rag-improvement/baseline-record.md explains it.
• No fabricated numbers. The validator requires each gate to be either recorded with a result and the run or file it came from, or pending_owner_run with a stated reason and no result. The three provider-backed gates (36-case retrieval, 44-case answer gate, 30-case answer quality) are recorded as pending, because dispatching them needs owner approval and they were not run for this commit. The two offline gates carry their real measured results.
• Data-flow register. docs/rag-improvement/data-flow-register.md is the Gate A artefact: inputs, processes, sinks with retention and de-identification status, provider egress, and four explicitly recorded gaps (worker temp-path cleanup is asserted rather than proven; provider retention terms are unexecuted under ledger #053; upload does not screen for identifiers; telemetry canary-absence tests are packet B1's deliverable).
• Planned-path allowlists removed. scripts/check-rag-adversarial-fixtures.mjs, scripts/fixtures/rag-adversarial-cases.v1.json, and docs/rag-improvement/data-flow-register.md are gone from the ALLOWLIST in scripts/check-docs-links.mjs, and check:rag:adversarial-fixtures is gone from the allowlist in scripts/check-docs-script-refs.mjs. Those paths and that command now exist, so both gates validate them for real.
• Enforcement without new CI routing. tests/rag-adversarial-fixtures.test.ts runs the same validation as the CLI and is registered in scripts/fixtures/rag-offline-contract-tests.json, so eval:rag:offline and the full unit suite hold the contract on every PR. CI routing for the adversarial runner is packet B2's work and is deliberately not done here — no workflow file and no scripts/ci-change-scope.mjs entry is changed.
• Docs. docs/rag-improvement/HANDOVER.md §2's S4 row is updated in this PR as the handover rule requires; docs/README.md and docs/scripts-index.md register the new documents and scripts.

RAG impact: no retrieval behaviour change — offline fixtures and validation only

Secret scanners, and why the canaries changed shape

The first push of this branch made both secret scanners red, and the fix is worth stating plainly because it is a real finding about how to build a canary.

The original tokens embedded identifier-shaped digit runs — an all-zero MRN body, an 1899 date of birth, an all-zero mobile number — to look like the PHI they simulate. Gitleaks flagged one as generic-api-key at entropy 3.83 and GitGuardian raised two incidents. Neither is a credential; both are true statements about the shape of the strings.

Rather than silence the scanners, the tokens changed: every canary now carries its PHI category in words and contains no digit at all (CANARY-PHI-MRN-NEVER-A-REAL-RECORD). A canary needs to be unique and exactly greppable, not entropy-dense, so nothing of value was lost — and the property is now enforced rather than merely observed. The validator regex is letters-only, rag-adversarial-cases.schema.json documents why, and a unit test asserts both that every committed token complies and that a digit-bearing one is rejected. A future canary cannot reintroduce the failure.

One .gitleaksignore fingerprint is added, because Gitleaks scans the whole PR commit range and the original commit 57fd5ae stays in it even though the tip is clean. It sits alongside the repository's existing synthetic-identifier false positives and follows their documented pattern. GitGuardian's two incidents are SaaS-side against that same historical commit and cannot be resolved from the repository; they need dashboard action or nothing, since the tip carries no match.

Verification

• npm run verify:pr-local

Script changes fail the selector closed to the heavy offline scope, so this ran the full set — lint, typecheck, the full unit suite, build, and eval:rag:offline. Summary line from the final run, on the tree this branch now carries:

PR-local verification summary:
- completed: check:runtime, check:installed-lock-parity, format:changed, check:npm-ci-dry-run, sitemap:check, docs:check-index, docs:check-inventory, docs:check-scripts, docs:check-links, check:branch-review-ledger, check:outstanding-issues, check:ledger-write-discipline, lint, typecheck, test, build, eval:rag:offline, check:medication-interactions, check:medication-lexicon-report
- failed: (none)
- not reached: (none)
[exited with code 0]

Decisive lines from the individual gates:

> npm run test
 Test Files  641 passed (641)
      Tests  6843 passed | 4 skipped (6847)

> npm run eval:rag:offline
Offline RAG fixture and manifest validation passed (36 golden cases, 25 suites).
 Test Files  25 passed (25)
      Tests  603 passed (603)
Offline RAG fixture and production-contract checks passed.

> npm run check:rag:adversarial-fixtures
Adversarial fixture contract passed (24 synthetic cases, 8 categories, 6 canaries).
- prompt_injection: 3 case(s)
- citation_fabrication: 3 case(s)
- unsupported_claim: 3 case(s)
- empty_or_conflicting_evidence: 3 case(s)
- scope_or_tenant: 3 case(s)
- provider_failure: 3 case(s)
- adversarial_metadata: 3 case(s)
- cost_or_timeout_abuse: 3 case(s)
Baseline rag-adversarial-baseline.v1 captured 2026-08-17 at 92f7618c0ccac336ef6c245b89f37b046f4eac32.
- gate retrieval_golden (36 cases): pending_owner_run
- gate answer_gate (44 cases): pending_owner_run
- gate answer_quality (30 cases): pending_owner_run
- gate offline_contract (25 cases): recorded
- gate adversarial_fixtures (24 cases): recorded

npm run format was run over the whole tree and the result is committed.

The two unit suites flagged as pre-existing Windows-host failures — tests/session-start-hook.test.ts and tests/check-playwright-browser-revision.test.ts — are both inside those 641 passing files in this Linux container, so the failure does not reproduce here, no merge-base isolation was needed, and no failure is being carried into this PR. (The FAIL strings visible in the unit-suite log are expected stdout from the check:function-grants negative fixtures, which assert that the guard rejects bad SQL.)

origin/main advanced three times while this branch was open. It was merged in each time, git merge-tree --write-tree is clean against the current base, and two genuine conflicts were resolved by keeping both sides: scripts/fixtures/rag-offline-contract-tests.json (this packet and main each appended a suite) and docs/rag-improvement/HANDOVER.md §2 (this packet's S4 row and packet S1b's row). docs/scripts-index.md conflicted only on its generated count line and was regenerated with npm run docs:update. The S1b row was also corrected from "PR open" to merged, verified from git — PR #2035 landed on main as merge commit 92f7618.

The offline contract list moved from 23 suites / 583 tests to 25 / 603 across those merges: tests/rag-adversarial-fixtures.test.ts from this packet, tests/search-route-round-trip-budget.test.ts from main, and S1b's test changes. The baseline record states all of it so the movement is not mistaken for drift. ragEvalCases is still 44 and answerQualityEvalCases still 30, so neither provider-backed denominator changed.

• npm run verify:ui when UI, routing, styling, browser behavior, reduced-motion, or forced-colors behavior changed

UI verification not run: no component, route, style, or browser-behaviour file is touched.

• npm run verify:release before release or handoff confidence claims

Verification not run: this is a packet handoff, not a release, and verify:release is provider-backed.

• npm run eval:retrieval:quality (must stay 36/36) when retrieval, ranking, selection, chunking, or scoring behavior changed

Verification not run: no retrieval, ranking, selection, chunking, or scoring behaviour is changed. The command is provider-backed and fires only via the owner-approved eval-canary dispatch. verify:pr-local did run eval:rag:offline, which validates the golden fixture and the offline contract suites, and it is green above.

• npm run eval:rag -- --limit 15 + npm run eval:quality -- --rag-only when answer generation, the synthesis prompt, or answer post-processing changed

Verification not run: answer generation, the synthesis prompt, and answer post-processing are untouched. Both commands are provider-backed.

• npm run check:production-readiness when clinical workflow, privacy, environment, Supabase, source governance, or deployment behavior changed

Verification not run: no clinical workflow, privacy, environment, Supabase, source-governance, or deployment behaviour changes. The data-flow register documents existing behaviour; it does not alter any.

• npm run check:deployment-readiness when deployment startup, hosting, or rollout behavior changed

Verification not run: no deployment startup, hosting, or rollout behaviour changes.

Risk and rollout

• Risk: Low. The only executable code added is an offline validator and its unit test; no product code path reads the new fixtures. The two allowlist removals are the one place where an existing gate's behaviour changes — docs:check-links and docs:check-scripts now validate four previously exempted references instead of skipping them. Both gates are green above, and both would fail loudly rather than silently if a referenced path or command were later removed. The .gitleaksignore addition pins one historical-commit fingerprint and does not weaken the rule at tip, which the letters-only canary contract now keeps clean.
• Rollback: revert the commits. Nothing else depends on the new files yet — packets B1, B2, and B3 consume them only after they land — and no schema, migration, flag, cache key, or prompt version is involved, so the revert needs no coordination.
• Provider or production effects: None. No command added or changed here reaches OpenAI, Supabase, Railway, or hosted CI. No provider-backed gate was run for this PR.

Clinical Governance Preflight

• Source-backed claims still require linked source verification before clinical use — unchanged. Nothing in this PR touches citation, verification, or render policy; the fixtures encode the expectation that a claim without a retrieved excerpt behind it must abstain rather than answer.
• No patient-identifiable document workflow was introduced or expanded without explicit governance approval — none is introduced. The opposite is the intent: the canary strings exist so that a patient identifier reaching a report, log, or telemetry sink is detectable, and the data-flow register records in §5 that upload performs no identifier screening today.
• Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy) — unchanged, and named as such in the register's provider-egress table. No Supabase configuration, migration, or connection is touched.
• Service-role keys and private document access remain server-only — unchanged. No environment variable, key handling, or access path is modified, and the validator reads only repository files.
• Demo/synthetic content remains clearly separated from real clinical sources — this PR strengthens that separation. Every adversarial excerpt title must begin with SYNTHETIC , the drugs are invented, and naming a real clinical source in a fixture is a hard validation failure.
• Source metadata, review status, and outdated/unknown-source behavior remain conservative — unchanged in code. Three fixtures pin the conservative expectation: a self-asserted governance status must not upgrade trust, an unattributed similarity: 1 must not by itself raise the confidence label, and an excerpt that supersedes another must have the relationship surfaced rather than silently resolved.
• Deployment classification/TGA SaMD impact was checked when clinical decision-support behavior changed — no clinical decision-support behaviour changes, so classification is unaffected.

Notes

• Open discrepancy surfaced, not resolved. HANDOVER.md §1 records "answer gate 45/45" for canary run 32025082010 at 2bd146eed, but src/lib/rag/rag-eval-cases.ts defines 44 ragEvalCases both at that commit and at this one, and scripts/eval-rag.ts reports its rates over results.length. The baseline records the gate as 44 cases and states the discrepancy rather than guessing which figure is right. It should be reconciled against run 32025082010's own report — packet S5 is the natural place, since its harness consumes this record.
• Deliberately out of scope. No CI workflow, no scripts/ci-change-scope.mjs routing entry, and no scripts/verify-pr-local.mjs selector entry is changed; packet B2 owns routing the adversarial runner. No dependency is added, in particular no JSON Schema runtime — the schema file is the published contract and the .mjs module is the enforcer, with a test asserting that every key the schema marks required is actually rejected when missing.
• Baseline completion is an owner action. With approval, dispatch the eval-canary pair at the merge commit, flip the three pending gates to recorded with their run ids, and update commit_sha in the baseline record.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[coderabbitai]

Caution

Review failed

An error occurred during the review process. Please try again later.

---

_{Comment @coderabbitai help to get the list of available commands.}

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

[gitguardian]

⚠️ GitGuardian has uncovered 4 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
-	-	Generic High Entropy Secret	670afaf	tests/rag-adversarial-fixtures.test.ts	View secret
36220292	Triggered	Generic High Entropy Secret	57fd5ae	scripts/fixtures/rag-adversarial-cases.v1.json	View secret
36220293	Triggered	Generic High Entropy Secret	57fd5ae	scripts/fixtures/rag-adversarial-cases.v1.json	View secret
36220293	Triggered	Generic High Entropy Secret	f08e160	tests/rag-adversarial-fixtures.test.ts	View secret

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secrets safely. Learn here the best practices.
3. Revoke and rotate these secrets.
4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

• following these best practices for managing and storing secrets including API keys and other credentials
• install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

---

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}