Skip to content

feat(audit): record tools/call gate rejections in the scoped audit (see Consiliency/pmcp#296) - #306

Merged
ViperJuice merged 4 commits into
mainfrom
feat/296-gate-audit
Sep 27, 2026
Merged

ViperJuice merged 4 commits into
mainfrom
feat/296-gate-audit

Conversation

@ViperJuice

@ViperJuice ViperJuice commented Sep 27, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Implements #296, the prerequisite for piece B (extra="forbid") of #236. See #296.

  • A tools/call the input-schema gate rejects is now written to the scoped-advisor audit as its own audit.rejection event (terminal_status: "invalid_arguments"). It carries the tool, the failing JSON path (caller-chosen keys redacted to null) and the schema keyword. It never carries the validation message, a value, correlations or any digest of the arguments.
  • Policy is judged before the gate, and read once: a call to a blocked tool is denied whatever its arguments.
  • Every audit.invocation reads only the registry's tool name and the declared top-level keys of arguments that passed the gate. A denied or unregistered call records nothing the caller chose.
  • If describing a rejection raises anything, the gate fails closed.
  • src/ is unchanged from what the plan verified.

Plan: .consiliency/plans/detailed-296-gate-audit-20260926-2141.md (merged in #304 as 8de49e4, rev 5, approved by the panel 4/4).

How this branch was built

  1. b81a164: the plan's five embedded patches, extracted from the merged plan with the plan's own extractor and applied to 8de49e4.
    • git apply --check: ok.
    • cmp against the verified branch wip/296-code @ 3a474ba: all five files identical (src/pmcp/server.py, src/pmcp/scoped_advisor_audit.py, tests/test_scoped_advisor_audit.py, CHANGELOG.md, README.md).
    • The CHANGELOG patch applied cleanly on 8de49e4, so it needed no rebasing.
  2. a6630f7: tests only, for the two items the panel required (below).
  3. a7f2feb: tests only, tightening the log oracle after this PR's board (below).
  4. 53ebe99: tests only, keeping excluded lines' tracebacks in the log oracle after the board's second round (below).

The panel's two required items

  1. The log oracle must check hashes and compare the two calls' logs.
    • _assert_nothing_generated_leaked now checks the same forbidden set against the log text as against the audit. That set is every generated value, plus the sha256 of its raw form, its JSON form and its normalised URL.
    • The log text is every record rendered by pmcp's own text and JSON formatters, tracebacks included.
    • Every sweep call now goes through _LoggedCalls.pair. It keeps the log each call produced, with timestamps and memory addresses removed and the one Validation errors echo argument values (pydantic input_value, jsonschema type/pattern) into responses and logs #297 line shape left out, and asserts the two calls of a pair logged the same thing.
    • The two calls' tags already differ in length and key count, so this rules out any value, hash, length or key count reaching the log.
    • Mixed-shape payloads now take a key's shape from its name, so both calls of a pair agree.
    • The round-5 seat's V1–V4 (a hash or length of a caller value written to a log line) are all killed.
  2. One allowed-path comparison against a real handler, or state the stub scope. Both are done.
    • New test test_generated_caller_values_on_the_real_scoped_handlers sweeps the four tools the scoped policy allows (health, catalog_search, describe, invoke). It uses the real GatewayTools under the real scoped policy, with no stubs and no downstream servers. Each pair of calls is compared on records and on logs, and each record against the one for the declared arguments alone.
    • The real handlers answer success, success, denied and denied respectively.
    • The stub sweep's docstring now states that it runs against _StubGatewayTools and names this test for the real handlers.

PR board follow-up (a7f2feb, tests only)

Codex's blocking finding and the claude seat's S1, S3, S5 and S7:

  • Normalisation strips no 0x… span any more. Before the two calls' logs are compared, each record is rendered by pmcp's formatters at a fixed record time. That time is the only volatile field those formatters print. Only a default object repr's address (<X object at 0x…>) is replaced; the sweep's logs contained none (measured over all of them). A caller-derived count or checksum logged in hex now reaches both the pair comparison and the forbidden-set check.

  • The Validation errors echo argument values (pydantic input_value, jsonschema type/pattern) into responses and logs #297 exclusion matches whole messages. It covers two lines only:

    A line that starts the same way but carries anything more stays in the oracle.

  • Regression tests for both of codex's false negatives:

    • test_the_log_normalisation_keeps_a_hex_count_and_drops_only_the_time
    • test_the_foreign_log_exclusion_matches_only_the_exact_lines
  • Mutants. S1, S3, S5, S7 and codex's two (C1: argument_count=%#x; C2: Unknown tool: payload=<arguments>) are added to the mutant set. Against a6630f7's tests, S1, S3, S5 and C1 survived, while S7 and C2 were killed. Now all six die.

PR board round 2 (53ebe99, tests only)

Codex found that the exact-message exclusion still hid a whole LogRecord. A traceback, exception text or stack attached to an excluded line therefore escaped the oracle. Codex reproduced this with exc_info=ValueError("caller_marker_…") on the excluded InvokeInput message.

  • The rule now. An excluded record is dropped only when it carries none of exc_info, exc_text or stack_info. Otherwise only its message is replaced by a placeholder, and the rendered diagnostics stay in the oracle. The rule is the same in _log_text, in _normalized_log, and therefore in _LoggedCalls.pair.
  • Regression tests. test_an_excluded_line_keeps_its_traceback_and_stack_in_the_oracle covers both excluded shapes (the InvokeInput meta message, and Unknown tool: <exact name>), each with exc_info, exc_text and stack_info carrying a marker. It checks the marker reaches the normalised log, the pair differential and the forbidden-set check. It also checks that a bare excluded line (no diagnostics) is still dropped.
  • Mutants. Four test-file mutants restore the message-only exclusion or narrow the diagnostics rule, and all four are killed:
    • R1: drop excluded records whole (codex's case).
    • R2: ignore stack_info.
    • R3: ignore exc_text.
    • R4: _log_text drops excluded records whole.

Follow-ups (non-blocking, from the board)

  • Unbounded recursion in _declared_property_names (gemini). It recurses over the tool's input schema with no depth limit, so an adversarially deep schema could hit the recursion limit. The schemas are pmcp's own registry definitions, not caller input, and every advertised schema passes check_schema. Noted for a bounded walk if schemas ever come from elsewhere.
  • "Scoped advisor audit channel failed" returns is_error=False (gemini). This is pre-existing. On main (8de49e4), call_tool's except ScopedAdvisorAuditError returns the same {"error": true, "message": "Scoped advisor audit channel failed"} payload inside CallToolResult(content=content), whose is_error defaults to False. The gate's new branch mirrors it on purpose (plan §4). Whether the channel-failed response should set is_error is a separate change for all paths.

Gates (as read from the logs, on this branch)

tests/test_scoped_advisor_audit.py:    162 passed in 21.65s
red on main 8de49e4 (same module, PYTHONPATH=<git archive 8de49e4 src>):    138 failed, 24 passed in 27.26s
ruff check src/ tests/:                 All checks passed!
ruff format --check src/ tests/:        165 files already formatted
mypy src/:                              Success: no issues found in 51 source files
full suite (npm_config_cache, npm_config_store_dir, pnpm_config_store_dir unset;
  -m 'not live and not slow'):          4503 passed, 3 skipped, 25 deselected in 466.67s (0:07:46)   exit=0
mutants (plan's 51 + V1-V4 + S1, S3, S5, S7, C1, C2 + oracle R1-R4):   65/65 killed

The full-suite baseline on main is 4353 passed, 3 skipped, 25 deselected, measured on 876fd33. Main's code and tests are unchanged since then: 8de49e4 adds only the plan document. The difference, +150, is exactly the 150 new test items. The 24 tests that pass on main are the 12 existing ones, the Y3 retention test, the rejection sweep's three no-property tools, and the eight oracle-helper regression tests, which do not depend on src/. Test source is ASCII: no added line has a non-ASCII character, and the look-alike and zero-width spellings are \u escapes.

Not in this PR

All four are documented in the plan's Non-goals.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LT8D8tRmExyQve69DCWRik

ViperJuice and others added 4 commits September 27, 2026 04:19
#296)

Applies the five patches embedded in the merged plan
(.consiliency/plans/detailed-296-gate-audit-20260926-2141.md, rev 5),
byte-identical to the verified code: gate rejections become an
audit.rejection event without argument values; policy is judged before
the gate and read once; every audit.invocation reads only the registry
name and the declared top-level keys of gate-passed arguments; an
exception while describing a rejection fails closed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LT8D8tRmExyQve69DCWRik
…'s log; a real-handler sweep (see #296)

The two items the review panel required on the plan (#304):
- the generated sweep asserts the same forbidden set (every generated
  value and its hashes) against the formatted log text as against the
  audit, and compares the log each call of a pair produced (timestamps and
  addresses removed), so no value, hash, length or key count is logged;
  mixed-shape payloads take a key's shape from its name, so both calls of
  a pair agree;
- the four tools the scoped policy allows are swept against the real
  GatewayTools too, and the stub sweep's docstring states its scope.
Kills the round-5 seat's V1-V4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LT8D8tRmExyQve69DCWRik
PR 306 board (codex blocking; claude seat S1/S3/S5/S7):
- normalisation no longer strips any 0x span: each record is rendered by
  pmcp's formatters at a fixed record time (the only volatile field they
  print), and only a default object repr's address is replaced; a hex
  count or checksum now reaches both oracles;
- the foreign-line exclusion matches whole messages: pydantic's InvokeInput
  error for a non-dict meta (the #297 channel), and
  "Unknown tool: <name>" only for the exact name of the call that logged
  it; a line with anything more stays in the oracle;
- regression tests for both false negatives; S1, S3, S5, S7 and codex's
  hex count and Unknown-tool payload join the mutant set (61/61 killed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LT8D8tRmExyQve69DCWRik
PR 306 board round 2 (codex blocking): the exact-message exclusion hid a
whole LogRecord, so a traceback, exception text or stack attached to an
excluded line escaped the oracle. An excluded record is now dropped only
when it carries none of exc_info, exc_text or stack_info; otherwise only its
message is replaced and the rendered diagnostics stay in the oracle, in
_log_text, _normalized_log and so _LoggedCalls.pair alike. Regression tests
cover both excluded shapes with each kind of diagnostic; oracle mutants
R1-R4, which restore the message-only exclusion, are killed (65/65).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LT8D8tRmExyQve69DCWRik
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant