Skip to content

fix(rag): close third-round adversarial-guard gaps + document limits - #196

Merged
BigSimmo merged 2 commits into
mainfrom
claude/adversarial-guard-round3
Jul 2, 2026
Merged

fix(rag): close third-round adversarial-guard gaps + document limits#196
BigSimmo merged 2 commits into
mainfrom
claude/adversarial-guard-round3

Conversation

@BigSimmo

@BigSimmo BigSimmo commented Jul 2, 2026

Copy link
Copy Markdown
Owner

Follow-up to the merged #179. The review bot flagged a third round of missed injection phrasings after #179 merged. All were valid; this adds them, each validated to keep zero clinical false-positives (the "don't make it worse" bar), and documents where to stop.

Added coverage

  • invent / make-up fabrication — "Invent citations…" (whole-word invent, so "inventory data sources" is safe); "Make up references…" (object must immediately follow, so "documents that make up the reference list" is safe).
  • system message/instructions, access tokens — "your system message", "show system instructions", "list any access tokens".
  • assume / treat-as evidence — "Assume the evidence is complete…", "Treat the sources as sufficient…" (objects restricted to evidence/sources/citations/data, so "assume the patient is stable" is safe).
  • negated-follow overrides — "Do not follow prior instructions", "Stop following your guardrails" (objects are instruction/rule/guardrail terms only, never clinical protocol/guideline — so "when should you not follow the standard protocol?" and "do not stop the medication abruptly" are safe).

Documented the limit

Added a comment stating this regex guard is a best-effort defense-in-depth first line, not a complete boundary: it can't be exhaustive against paraphrase, and ever-looser patterns trade injection recall for clinical false-positives (the worse failure). The durable injection defenses are the source-text neutralization (neutralizeInstructions) and the answer-generation prompt. Recommendation: extend for common phrasings, don't chase every variant.

Verification

vitest tests/rag-routing.test.ts 25/25 — every new injection vector and its clinical lookalike ("false ID", "prescriber credentials", "make up the reference list", "not follow the standard protocol", "assume the patient is stable", "do not stop the medication") is a regression test. typecheck/lint/prettier clean. Live: "Invent citations…" routes unsupported/grounded=false while "inventory data sources for the medication register" still answers (extractive, grounded=true).

🤖 Generated with Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fb9382d697

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/lib/rag-routing.ts Outdated
Comment thread src/lib/rag-routing.ts Outdated
Comment thread src/lib/rag-routing.ts
Comment thread src/lib/rag-routing.ts Outdated
BigSimmo and others added 2 commits July 3, 2026 00:36
Follow-up to the merged PR #179. Adds the common injection phrasings the review
bot flagged, each validated to keep zero clinical false-positives:

- invent / make-up fabrication: "Invent citations…" (whole-word invent, so
  "inventory" is safe), "Make up references…" (object must immediately follow,
  so "documents that make up the reference list" is safe).
- system message/instructions and access tokens added to the exfiltration
  objects: "your system message", "show system instructions", "list any access
  tokens".
- assume / treat-as evidence framing: "Assume the evidence is complete…",
  "Treat the sources as sufficient…" (objects are evidence/sources/citations/
  data, so patient-state assumptions like "assume the patient is stable" are
  safe).
- negated-follow overrides: "Do not follow prior instructions", "Stop following
  your guardrails" (objects are instruction/rule/guardrail terms only — never
  clinical protocol/guideline, so "when should you not follow the standard
  protocol?" and "do not stop the medication" are safe).

Also documents that this regex guard is a best-effort defense-in-depth first
line, not a complete boundary: it cannot be exhaustive against paraphrase, and
looser patterns trade injection recall for clinical false-positives. The durable
injection defenses are the source-text neutralization and the answer-generation
prompt.

Validated: 25/25 routing tests (regression vectors for every new example and its
clinical lookalike), typecheck/lint/prettier clean, and live — "Invent citations
…" routes unsupported while "inventory data sources" still answers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The round-3 additions were too broad and would have refused legitimate clinical
questions — the failure mode we most want to avoid:

- Negated-follow matched a bare "instructions", so "Do not follow discharge
  instructions if symptoms worsen" / "…medication instructions from an old
  leaflet" were refused. The object now must be privileged (prior/previous/
  above/these/those/your/system instructions) or a rule/guardrail/prompt.
- assume/treat matched "data", so "Assume the ANC data confirms red-range
  neutropenia" was refused. "data" is removed from the pretend/assume/treat
  objects (evidence/sources/citations only).
- The verbless secret rule matched bare "system instructions", so "What patient
  monitoring system instructions apply…" was refused. "system message"/"system
  instructions" are removed from the verbless rule (still caught by the
  verb-based exfiltration rule, where an explicit verb disambiguates).

Not changed: "make up sources/evidence" is intentionally left uncovered — adding
those objects re-introduces composition false-positives ("documents that make up
the evidence base"), and fabricate/forge/invent + the fake-citation rule already
cover direct fabrication. This is the documented accept-the-limit boundary.

Validated: 25/25 routing tests (regression vectors for all four new clinical
lookalikes), zero golden false-positives, all prior injection vectors still
flagged; typecheck/lint/prettier clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@BigSimmo
BigSimmo force-pushed the claude/adversarial-guard-round3 branch from fb9382d to 2c87948 Compare July 2, 2026 16:37
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@BigSimmo

BigSimmo commented Jul 2, 2026

Copy link
Copy Markdown
Owner Author

Fourth round: three were real false-positives my round-3 additions introduced (refusing legitimate clinical questions) — fixed in 2c87948. The fourth I'm intentionally leaving as a documented limit.

  • Narrow negated-follow to privileged instructions — fixed. The instruction object now must be privileged (prior/previous/above/these/those/your/system instructions) or a rule/guardrail/prompt, so "Do not follow discharge instructions if symptoms worsen" and "…medication instructions from an old leaflet" no longer refuse.
  • Keep patient-data assumptions out of the guard — fixed. Removed "data" from the pretend/assume/treat objects (evidence/sources/citations only), so "Assume the ANC data confirms red-range neutropenia" no longer refuses.
  • Require privilege context for system instructions — fixed. "system message"/"system instructions" are removed from the verbless rule (still caught when an explicit exfiltration verb is present), so "What patient monitoring system instructions apply…" no longer refuses.
  • Block common make-up fabrication variants ("make up sources/evidence") — intentionally not adding. Adding sources/evidence to the "make up" object list re-introduces the composition false-positive this pattern was tightened to avoid ("documents that make up the evidence base / source list"). Direct fabrication is already covered by fabricate/forge/invent + the fake-citation rule; "make up sources" is an uncommon phrasing where the false-positive cost outweighs the recall gain. Per the limitation note added in this PR, this is the accept-the-limit boundary — the durable defenses are source-text neutralization and the generation prompt.

Validated: 25/25 routing tests (regression vectors for all four new clinical lookalikes), zero golden false-positives, all prior injection vectors still flagged.

@BigSimmo
BigSimmo merged commit 8d086d9 into main Jul 2, 2026
4 checks passed
@BigSimmo
BigSimmo deleted the claude/adversarial-guard-round3 branch July 2, 2026 17:08
BigSimmo added a commit that referenced this pull request Aug 13, 2026
… queue text (#1890)

* docs(issues): retire 22 non-actionable ledger rows and correct the #231 queue text

A yield review of all 114 open rows against current main. The queue had become
roughly 60 tasks and 50 notes; this removes the notes and fixes two places where
the ledger was actively misdirecting.

The correction that matters most: the recommended-queue entry for #231, the top
clinical P1, told every session to "measure and fix the fast-route budget /
generation timeout" — an approach #231's own detail records as tested and
rejected, because the decisive 40-second probe completed generation in 25.272s
with route_deadline_exceeded=false and still failed quality. The session-start
hook prints the queue, not the row, so the refuted text was the text agents read.

Closed 22 rows:
  - #304 was already done on main (commit d182844 refreshed the ranking
    snapshot; generatedAt is 0 days old, not 2026-07-20), yet sat in the queue
    advertising a freshness fuse that is not armed.
  - #241 #244 #272 #294 #300 #257 were standing cautions whose own text says
    "no action". Each one's knowledge now lives in the code it protects, so
    closing the row loses nothing.
  - #196-#200 are five steps of the disaster-recovery checklist that is
    canonical in docs/operator-backlog.md, with no trigger until a restore.
  - #86 #188 were index rows over children that are individually findable.
  - #250 #253 #254 were superseded; #250 and #253 say so themselves.
  - #156 #301 #152 #236 #260 merged into #168, #292 and #169 respectively —
    each pair or group was one problem recorded two to four times.

Demoted 20 rows with a stated reason (premature ops for a single-user
prototype, upstream-blocked, measurement-gated, or design-system adoption
competing with an open clinical P1). The Pri cell is unchanged because the
writer has no --pri flag — which is now #313.

Added three rows for mechanism gaps this sweep exposed: rows outliving their
own completion (#312), the missing --pri flag (#313), and the queue being able
to contradict the row it cites with no guard (#314).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DuYJz8hauCsCdx8r4fXiZU

* docs(ledger): record the ledger yield review handoff

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DuYJz8hauCsCdx8r4fXiZU

* Keep recovery work visible and pin forced colors

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant