security: redact structured tool results field by field (split out of #234) #290

Split out of #234 by maintainer decision. #234 now covers the core text redactor only; this issue covers redacting structured gateway tool results.

The gap (exists on main today)

gateway.invoke (handlers.py ~2068) and tasks_result (~6523, redact=True by default) pass the downstream result dict to PolicyManager.process_output, which json.dumps it before redacting. A tool whose text content is itself JSON therefore reaches the redactor with escaped quotes, \"password\": \"hunter2\", and keyed rules don't match. This is the dominant MCP result shape (a text content item whose text is JSON), so it's the most common way a secret passes the redactor.

Reproduction on main:

process_output({"content": [{"type": "text", "text": json.dumps(
    {"password": "hunter2", "api_key": "abc123def456"})}], "isError": True}, redact=True)
# -> both values visible

What was tried in #234, and why it moved here

Revision 7 of the #234 plan added PolicyManager._redact_leaves: it redacts each string leaf as written, then as a one-pair {key: leaf} document for key context, before json.dumps. It fixed the case above, and a benign structured result round-tripped byte-identical. But a four-seat review (round 6) found defects that all sit in this new layer, while the core text redactor held. Each is reproduced; use them as this issue's starting acceptance criteria.

1. No cost bound. Each string leaf is scanned whole, twice. For one benign text leaf, the string path takes 0.07 s whatever the size; the dict path took 2.16 s for 1 MB and 6.35 s for 3 MB, linear in the attacker-controlled leaf, on the request path. Slice each leaf to the remaining budget before scanning, extended to the end of any span that starts inside it.
2. JSON nested two levels deep leaks. An HTTP-fetch-shaped result, {"status": 401, "body": json.dumps({"password": "hunter2", "access_token": "abc123def456"})} as a text leaf, leaks both values. Recurse structurally into leaves that parse as JSON, with a depth cap (the reviewer suggested 3), and state that re-dumping normalises whitespace.
3. Leaves past the budget are emitted as "". 25 PEM leaves followed by 5 benign items at the default cap produce a complete-looking document in which every benign text reads "" (8 empty-string leaves, 0 of 5 benign texts visible). An agent acts on false data where the string path gives a visibly cut document. Use an explicit placeholder, or omit the pair, and have the marker say how many leaves were dropped.
4. Key context is lost below the first level. {"password": ["hunter2"]}, {"api_key": ["abc123def456"]} and {"password": {"nested": "hunter2"}} leak: list items get no parent key, and nested objects pass the child key. Tuples are returned unchanged. Thread the parent key through lists, tuples and nested objects.
5. Non-string leaves are never scanned. {"password": 12345678} passes unchanged, and an operator-configured pattern such as \b123456\b no longer protects {"pin": 123456}. Scalars must keep redaction coverage; state any type-change trade.
6. Redaction expansion silently corrupts a capped result. process_output({"password": "a", "note": "x" * 260}, redact=True, max_bytes=300): the redacted serialisation grows to 304 bytes, is sliced to 300, and truncated=False is reported over incomplete JSON. Base truncation on the representation actually emitted.

Acceptance bar

The #234 core redactor (span collection, redact-before-cut over a bounded window, a differential against main as the never-worse test) is the foundation. Plan against it once #234 has merged. Pin both directions with properties: secrets in every structured shape above never appear in the output, and a benign structured result round-trips byte-identical, including large results and results past the budget.

See #234.