Repository navigation
feat(audit): record tools/call gate rejections in the scoped audit (see Consiliency/pmcp#296) - #306
Merged
Merged
Conversation
#296) Applies the five patches embedded in the merged plan (.consiliency/plans/detailed-296-gate-audit-20260926-2141.md, rev 5), byte-identical to the verified code: gate rejections become an audit.rejection event without argument values; policy is judged before the gate and read once; every audit.invocation reads only the registry name and the declared top-level keys of gate-passed arguments; an exception while describing a rejection fails closed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LT8D8tRmExyQve69DCWRik
…'s log; a real-handler sweep (see #296) The two items the review panel required on the plan (#304): - the generated sweep asserts the same forbidden set (every generated value and its hashes) against the formatted log text as against the audit, and compares the log each call of a pair produced (timestamps and addresses removed), so no value, hash, length or key count is logged; mixed-shape payloads take a key's shape from its name, so both calls of a pair agree; - the four tools the scoped policy allows are swept against the real GatewayTools too, and the stub sweep's docstring states its scope. Kills the round-5 seat's V1-V4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LT8D8tRmExyQve69DCWRik
PR 306 board (codex blocking; claude seat S1/S3/S5/S7): - normalisation no longer strips any 0x span: each record is rendered by pmcp's formatters at a fixed record time (the only volatile field they print), and only a default object repr's address is replaced; a hex count or checksum now reaches both oracles; - the foreign-line exclusion matches whole messages: pydantic's InvokeInput error for a non-dict meta (the #297 channel), and "Unknown tool: <name>" only for the exact name of the call that logged it; a line with anything more stays in the oracle; - regression tests for both false negatives; S1, S3, S5, S7 and codex's hex count and Unknown-tool payload join the mutant set (61/61 killed). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LT8D8tRmExyQve69DCWRik
PR 306 board round 2 (codex blocking): the exact-message exclusion hid a whole LogRecord, so a traceback, exception text or stack attached to an excluded line escaped the oracle. An excluded record is now dropped only when it carries none of exc_info, exc_text or stack_info; otherwise only its message is replaced and the rendered diagnostics stay in the oracle, in _log_text, _normalized_log and so _LoggedCalls.pair alike. Regression tests cover both excluded shapes with each kind of diagnostic; oracle mutants R1-R4, which restore the message-only exclusion, are killed (65/65). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LT8D8tRmExyQve69DCWRik
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements #296, the prerequisite for piece B (
extra="forbid") of #236. See #296.tools/callthe input-schema gate rejects is now written to the scoped-advisor audit as its ownaudit.rejectionevent (terminal_status: "invalid_arguments"). It carries the tool, the failing JSON path (caller-chosen keys redacted tonull) and the schema keyword. It never carries the validation message, a value, correlations or any digest of the arguments.deniedwhatever its arguments.audit.invocationreads only the registry's tool name and the declared top-level keys of arguments that passed the gate. A denied or unregistered call records nothing the caller chose.src/is unchanged from what the plan verified.Plan:
.consiliency/plans/detailed-296-gate-audit-20260926-2141.md(merged in #304 as8de49e4, rev 5, approved by the panel 4/4).How this branch was built
b81a164: the plan's five embedded patches, extracted from the merged plan with the plan's own extractor and applied to8de49e4.git apply --check: ok.cmpagainst the verified branchwip/296-code@3a474ba: all five files identical (src/pmcp/server.py,src/pmcp/scoped_advisor_audit.py,tests/test_scoped_advisor_audit.py,CHANGELOG.md,README.md).8de49e4, so it needed no rebasing.a6630f7: tests only, for the two items the panel required (below).a7f2feb: tests only, tightening the log oracle after this PR's board (below).53ebe99: tests only, keeping excluded lines' tracebacks in the log oracle after the board's second round (below).The panel's two required items
_assert_nothing_generated_leakednow checks the same forbidden set against the log text as against the audit. That set is every generated value, plus the sha256 of its raw form, its JSON form and its normalised URL._LoggedCalls.pair. It keeps the log each call produced, with timestamps and memory addresses removed and the one Validation errors echo argument values (pydantic input_value, jsonschema type/pattern) into responses and logs #297 line shape left out, and asserts the two calls of a pair logged the same thing.test_generated_caller_values_on_the_real_scoped_handlerssweeps the four tools the scoped policy allows (health,catalog_search,describe,invoke). It uses the realGatewayToolsunder the real scoped policy, with no stubs and no downstream servers. Each pair of calls is compared on records and on logs, and each record against the one for the declared arguments alone.success,success,deniedanddeniedrespectively._StubGatewayToolsand names this test for the real handlers.PR board follow-up (
a7f2feb, tests only)Codex's blocking finding and the claude seat's S1, S3, S5 and S7:
Normalisation strips no
0x…span any more. Before the two calls' logs are compared, each record is rendered by pmcp's formatters at a fixed record time. That time is the only volatile field those formatters print. Only a default object repr's address (<X object at 0x…>) is replaced; the sweep's logs contained none (measured over all of them). A caller-derived count or checksum logged in hex now reaches both the pair comparison and the forbidden-set check.The Validation errors echo argument values (pydantic input_value, jsonschema type/pattern) into responses and logs #297 exclusion matches whole messages. It covers two lines only:
InvokeInputerror for a non-dictmeta, the Validation errors echo argument values (pydantic input_value, jsonschema type/pattern) into responses and logs #297 channel. It is the onlyInvokeInputerror the sweep produces (216 lines, all this shape).Tool execution error: Unknown tool: <name>, only for the exact name of the call that logged it.A line that starts the same way but carries anything more stays in the oracle.
Regression tests for both of codex's false negatives:
test_the_log_normalisation_keeps_a_hex_count_and_drops_only_the_timetest_the_foreign_log_exclusion_matches_only_the_exact_linesMutants. S1, S3, S5, S7 and codex's two (C1:
argument_count=%#x; C2:Unknown tool: payload=<arguments>) are added to the mutant set. Againsta6630f7's tests, S1, S3, S5 and C1 survived, while S7 and C2 were killed. Now all six die.PR board round 2 (
53ebe99, tests only)Codex found that the exact-message exclusion still hid a whole
LogRecord. A traceback, exception text or stack attached to an excluded line therefore escaped the oracle. Codex reproduced this withexc_info=ValueError("caller_marker_…")on the excludedInvokeInputmessage.exc_info,exc_textorstack_info. Otherwise only its message is replaced by a placeholder, and the rendered diagnostics stay in the oracle. The rule is the same in_log_text, in_normalized_log, and therefore in_LoggedCalls.pair.test_an_excluded_line_keeps_its_traceback_and_stack_in_the_oraclecovers both excluded shapes (theInvokeInputmetamessage, andUnknown tool: <exact name>), each withexc_info,exc_textandstack_infocarrying a marker. It checks the marker reaches the normalised log, the pair differential and the forbidden-set check. It also checks that a bare excluded line (no diagnostics) is still dropped.stack_info.exc_text._log_textdrops excluded records whole.Follow-ups (non-blocking, from the board)
_declared_property_names(gemini). It recurses over the tool's input schema with no depth limit, so an adversarially deep schema could hit the recursion limit. The schemas are pmcp's own registry definitions, not caller input, and every advertised schema passescheck_schema. Noted for a bounded walk if schemas ever come from elsewhere.is_error=False(gemini). This is pre-existing. On main (8de49e4),call_tool'sexcept ScopedAdvisorAuditErrorreturns the same{"error": true, "message": "Scoped advisor audit channel failed"}payload insideCallToolResult(content=content), whoseis_errordefaults toFalse. The gate's new branch mirrors it on purpose (plan §4). Whether the channel-failed response should setis_erroris a separate change for all paths.Gates (as read from the logs, on this branch)
The full-suite baseline on main is
4353 passed, 3 skipped, 25 deselected, measured on876fd33. Main's code and tests are unchanged since then:8de49e4adds only the plan document. The difference, +150, is exactly the 150 new test items. The 24 tests that pass on main are the 12 existing ones, the Y3 retention test, the rejection sweep's three no-property tools, and the eight oracle-helper regression tests, which do not depend onsrc/. Test source is ASCII: no added line has a non-ASCII character, and the look-alike and zero-width spellings are\uescapes.Not in this PR
InvokeInputerrors (Validation errors echo argument values (pydantic input_value, jsonschema type/pattern) into responses and logs #297).source_reference_hash, as on main, until piece B lands.record_invocation's correlation and URL reads togateway.invoke.All four are documented in the plan's Non-goals.
🤖 Generated with Claude Code
https://claude.ai/code/session_01LT8D8tRmExyQve69DCWRik