Skip to content

fix(review): INJECTION_SOURCE regex false-positives on benign config/docs prose #3657

Description

@JSONbored

Problem

src/review/prompt-injection.ts's INJECTION_SOURCE array (used by neutralizePromptInjection to redact suspected prompt-injection attempts before untrusted PR/file content reaches the AI reviewer) was empirically confirmed to false-positive on ordinary technical documentation phrasing, most concretely: "Per-repo override of the synthesis merge rule (#2567):" in .gittensory.yml.example (fixed by rewording, in #3641) blocked PR #3635's gate.

The underlying regex itself still has the gap. A follow-up empirical audit (standalone Node script testing all 7 patterns against realistic benign phrases) found similar over-broad matching in every pattern in the array -- notably ones this specific repo is likely to trip given its own subject matter (an AI-review / PR-merge / prompt-engineering product):

  • Pattern 1 (ignore/disregard/forget/override/bypass + previous/the/... + instructions/rules/...): matches "override the default retry policy", "bypass the strict validation guideline".
  • Pattern 2 (you are now): matches "You are now ready to deploy your first worker."
  • Pattern 3 (system/developer prompt): matches "the system prompt sent to the AI reviewer" (this repo builds AI review system prompts).
  • Pattern 4 (approve/merge/... this/the submission/pr/...): matches "the gate will automatically merge the pull request" (this repo's core feature).
  • Pattern 5 (as an ai/assistant ... you must/ignore/approve): matches "As an AI reviewer, you must flag any security issue you find."
  • Pattern 6 (print/reveal/output/... system prompt/rubric/instructions): matches "This script will output the setup instructions to a file."
  • Pattern 7 (pretend/roleplay ... you are/to be): matches "This test mock will pretend to be a GitHub webhook sender."

Fix

Tighten each pattern to a narrower, empirically-verified shape that keeps catching every pinned true-positive fixture in test/unit/prompt-injection.test.ts (including the #2998 adversarial regression suite) while no longer matching the benign phrases above. Add regression fixtures for both the false-positive phrases (must NOT match) and a few narrowing-induced "documented limitation" cases (a small, explicit recall reduction, pinned the same way the existing Unicode-homoglyph/synonym-wording limitations are).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions