Skip to content

docs(mcp-analytics): let servers classify tool failures as expected - #184

Merged
lucasheriques merged 6 commits into
mainfrom
lucas/mcp-expected-errors
Nov 13, 2026
Merged

lucasheriques merged 6 commits into
mainfrom
lucas/mcp-expected-errors

Conversation

@lucasheriques

Copy link
Copy Markdown
Contributor

Why

MCP server owners need to mark tool failures that are expected by design, such as rate limits, records that do not exist, or arguments the agent must fix. Those failures should be separable from the ones the owner has to fix. Today automatic instrumentation can't take an explicit error type, so every isError result is typed "Error". One server owner reported that 81% of its failures were expected (PostHog/posthog#113659).

What changed

  • New error classifier option on automatic instrumentation. It is a server callback that gets the tool name, the type and message the call would carry, and the tool's own failure when the SDK holds it. It may return an explicit type and whether the failure is expected.
  • New property $mcp_error_expected, a JSON boolean on failed calls, set only when the server says. The manual API takes the same flag next to its error type.
  • not_found joins the shared error-type vocabulary. An empty or blank explicit type counts as no type, and explicit types are sanitized as free text.

Scope

  • An expected failure is still a failed call. $mcp_is_error stays true.
  • $exception is unchanged. Whether an expected failure should skip $exception is open, and design.md records it.
  • Only tool calls that end in a thrown error or an isError result. Failed or cancelled tasks and framework-ended input_required sequences keep their fixed types. The manual API never calls the classifier.

Tradeoffs

  • Marking failures in beforeSend needs no spec change, but the hook sees only the sanitized message and runs once per payload.
  • Marking failures in the tool result (_meta) would leak analytics data to the client.

Verification

  • openspec validate --specs --strict passes (69 items), and the archive's delta matches the canonical spec.
  • The feature file parses (42 scenarios). The new scenarios need harness bindings, tracked in tasks.md.
  • Two pre-PR gate rounds, each with one Opus and one Sonnet adversary probing all four SDKs. Seven verified findings are fixed. One of them is an existing Python bug (the low-level adapter puts the handle in $mcp_error_message), which belongs in posthog-python and is left out here.

@dustinbyrne dustinbyrne left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The opt-in classification contract preserves failure status and exception identity, and the canonical requirements match the archive. One optional scenario-setup clarification is inline. Current-main conflicts still require resolution and integration validation.

AI-assisted review.

Comment thread openspec/specs/mcp-analytics/spec.md Resolved
Add an error classifier option and the $mcp_error_expected property.
For a failed tool call, the classifier callback receives the tool name
and the failure (the thrown value, else the isError result). It may
return an explicit error type and whether the failure is expected.
Expected failures keep $mcp_is_error true. $exception is unchanged.
The manual API takes the same flag. not_found joins the shared type
vocabulary.

Why: automatic instrumentation had no way to receive an explicit type,
so every isError result was typed "Error". One server reported 81% of
its failures as expected (PostHog/posthog#113659).

Tested: openspec validate --specs --strict (69 passed); openspec
archive synced the delta; feature file parses with @cucumber/gherkin
(42 scenarios).
The pre-PR gate found the classifier had no defined input for failures
the SDK types itself (failed or cancelled tasks, an input_required
sequence the framework ends), let it relabel those fixed types, left
"the thrown value" ambiguous where frameworks wrap tool errors (Python
FastMCP re-raises as ToolError), and did not say whether the manual
lower-level method calls it.

Now the classifier always gets the tool name and the type and message
the call would carry, plus the tool's own failure (past wrappers) when
the SDK holds it. It runs only for thrown errors and isError results,
at most once, on automatic instrumentation and the manual step path,
never on the lower-level method. The acceptance scenario also checks
that the client gets the same result.

Tested: openspec validate --specs --strict (69 passed); delta and
canonical Failure capture blocks match; feature parses (42 scenarios).
Second gate round fixes. An empty or blank explicit type now counts as
no type (TS dropped $mcp_error_type, Python and Ruby kept "   ", Go fell
back), and explicit types are sanitized as free text (all four SDKs sent
a phc_ token in a caller-passed type unredacted). The manual API never
calls the classifier, since no SDK's result step sends events and the
caller already holds the failure. "At most once" becomes "exactly once".

Tested: openspec validate --specs --strict (69 passed); delta matches
the canonical Failure capture and Manual capture blocks.
…nt-conversation-id wording

Keeps the archived manual-API requirement identical to the synced spec after rebasing onto main (client conversation ids, rule 4 for stateless requests). Tested: openspec validate --specs --strict --no-interactive passes.
… scenario

Review fix (dustinbyrne F1): the scenario compares the client's result with and without instrumentation, so anchoring's appended handle block would differ for an unrelated reason. Matches the public acceptance scenario; both canonical copies updated. Validate totals unchanged from main.
…rors

# Conflicts:
#	openspec/specs/mcp-analytics/spec.md
lucasheriques added a commit that referenced this pull request Nov 12, 2026
What: resolve the three conflicts in openspec/specs/mcp-analytics/spec.md and
resync the archived delta with the merged requirement.
- Configuration table: keep the error message capture row, drop the payload
  personal data row (main removed that option).
- Failure capture: keep the option-off paragraph, reworded because Payload
  privacy no longer applies the personal data pass to error messages; take
  main's $mcp_error_status derivation rules.
- Scenarios: keep both the option-off scenario and main's HTTP status one.
- Archived change: MODIFIED Failure capture delta now mirrors the merged
  requirement; proposal and design no longer claim a personal data pass on
  error messages.

Why: #192 conflicted with main after #212 and the payload personal data removal.
#184's content is not part of this PR and is not depended on.

Tested: openspec validate --specs --strict --no-interactive with
@fission-ai/openspec@1.4.1 (CI's version): 69 passed, 0 failed. Unarchived
changes check: none outside openspec/changes/archive.
@lucasheriques
lucasheriques merged commit b8f3d5c into main Nov 13, 2026
11 checks passed
@lucasheriques
lucasheriques deleted the lucas/mcp-expected-errors branch November 13, 2026 18:20
lucasheriques added a commit that referenced this pull request Nov 13, 2026
Keep both scenario sets in Failure capture (the error classifier ones
from #184 and the error message capture one), add $mcp_error_expected to
what the option leaves unchanged, and re-sync the archived delta with the
merged requirement.

Tested: openspec@1.4.1 validate --specs --strict: 69 passed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants