Repository navigation
docs(mcp-analytics): let servers classify tool failures as expected - #184
Merged
Merged
Conversation
dustinbyrne
approved these changes
Nov 5, 2026
dustinbyrne
left a comment
Contributor
There was a problem hiding this comment.
The opt-in classification contract preserves failure status and exception identity, and the canonical requirements match the archive. One optional scenario-setup clarification is inline. Current-main conflicts still require resolution and integration validation.
AI-assisted review.
Add an error classifier option and the $mcp_error_expected property. For a failed tool call, the classifier callback receives the tool name and the failure (the thrown value, else the isError result). It may return an explicit error type and whether the failure is expected. Expected failures keep $mcp_is_error true. $exception is unchanged. The manual API takes the same flag. not_found joins the shared type vocabulary. Why: automatic instrumentation had no way to receive an explicit type, so every isError result was typed "Error". One server reported 81% of its failures as expected (PostHog/posthog#113659). Tested: openspec validate --specs --strict (69 passed); openspec archive synced the delta; feature file parses with @cucumber/gherkin (42 scenarios).
The pre-PR gate found the classifier had no defined input for failures the SDK types itself (failed or cancelled tasks, an input_required sequence the framework ends), let it relabel those fixed types, left "the thrown value" ambiguous where frameworks wrap tool errors (Python FastMCP re-raises as ToolError), and did not say whether the manual lower-level method calls it. Now the classifier always gets the tool name and the type and message the call would carry, plus the tool's own failure (past wrappers) when the SDK holds it. It runs only for thrown errors and isError results, at most once, on automatic instrumentation and the manual step path, never on the lower-level method. The acceptance scenario also checks that the client gets the same result. Tested: openspec validate --specs --strict (69 passed); delta and canonical Failure capture blocks match; feature parses (42 scenarios).
Second gate round fixes. An empty or blank explicit type now counts as no type (TS dropped $mcp_error_type, Python and Ruby kept " ", Go fell back), and explicit types are sanitized as free text (all four SDKs sent a phc_ token in a caller-passed type unredacted). The manual API never calls the classifier, since no SDK's result step sends events and the caller already holds the failure. "At most once" becomes "exactly once". Tested: openspec validate --specs --strict (69 passed); delta matches the canonical Failure capture and Manual capture blocks.
…nt-conversation-id wording Keeps the archived manual-API requirement identical to the synced spec after rebasing onto main (client conversation ids, rule 4 for stateless requests). Tested: openspec validate --specs --strict --no-interactive passes.
lucasheriques
force-pushed
the
lucas/mcp-expected-errors
branch
from
November 9, 2026 18:15
f88dbe4 to
de7daf8
Compare
… scenario Review fix (dustinbyrne F1): the scenario compares the client's result with and without instrumentation, so anchoring's appended handle block would differ for an unrelated reason. Matches the public acceptance scenario; both canonical copies updated. Validate totals unchanged from main.
…rors # Conflicts: # openspec/specs/mcp-analytics/spec.md
lucasheriques
added a commit
that referenced
this pull request
Nov 12, 2026
What: resolve the three conflicts in openspec/specs/mcp-analytics/spec.md and resync the archived delta with the merged requirement. - Configuration table: keep the error message capture row, drop the payload personal data row (main removed that option). - Failure capture: keep the option-off paragraph, reworded because Payload privacy no longer applies the personal data pass to error messages; take main's $mcp_error_status derivation rules. - Scenarios: keep both the option-off scenario and main's HTTP status one. - Archived change: MODIFIED Failure capture delta now mirrors the merged requirement; proposal and design no longer claim a personal data pass on error messages. Why: #192 conflicted with main after #212 and the payload personal data removal. #184's content is not part of this PR and is not depended on. Tested: openspec validate --specs --strict --no-interactive with @fission-ai/openspec@1.4.1 (CI's version): 69 passed, 0 failed. Unarchived changes check: none outside openspec/changes/archive.
lucasheriques
added a commit
that referenced
this pull request
Nov 13, 2026
Keep both scenario sets in Failure capture (the error classifier ones from #184 and the error message capture one), add $mcp_error_expected to what the option leaves unchanged, and re-sync the archived delta with the merged requirement. Tested: openspec@1.4.1 validate --specs --strict: 69 passed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
MCP server owners need to mark tool failures that are expected by design, such as rate limits, records that do not exist, or arguments the agent must fix. Those failures should be separable from the ones the owner has to fix. Today automatic instrumentation can't take an explicit error type, so every
isErrorresult is typed"Error". One server owner reported that 81% of its failures were expected (PostHog/posthog#113659).What changed
$mcp_error_expected, a JSON boolean on failed calls, set only when the server says. The manual API takes the same flag next to its error type.not_foundjoins the shared error-type vocabulary. An empty or blank explicit type counts as no type, and explicit types are sanitized as free text.Scope
$mcp_is_errorstaystrue.$exceptionis unchanged. Whether an expected failure should skip$exceptionis open, and design.md records it.isErrorresult. Failed or cancelled tasks and framework-endedinput_requiredsequences keep their fixed types. The manual API never calls the classifier.Tradeoffs
beforeSendneeds no spec change, but the hook sees only the sanitized message and runs once per payload._meta) would leak analytics data to the client.Verification
openspec validate --specs --strictpasses (69 items), and the archive's delta matches the canonical spec.$mcp_error_message), which belongs in posthog-python and is left out here.