From ef71c50946c3caf20ff57d34d9fb3d1bba090910 Mon Sep 17 00:00:00 2001 From: Lucas Faria Date: Thu, 5 Nov 2026 09:38:32 -0300 Subject: [PATCH 1/5] docs(mcp-analytics): let servers classify failures as expected Add an error classifier option and the $mcp_error_expected property. For a failed tool call, the classifier callback receives the tool name and the failure (the thrown value, else the isError result). It may return an explicit error type and whether the failure is expected. Expected failures keep $mcp_is_error true. $exception is unchanged. The manual API takes the same flag. not_found joins the shared type vocabulary. Why: automatic instrumentation had no way to receive an explicit type, so every isError result was typed "Error". One server reported 81% of its failures as expected (PostHog/posthog#113659). Tested: openspec validate --specs --strict (69 passed); openspec archive synced the delta; feature file parses with @cucumber/gherkin (42 scenarios). --- acceptance/public/mcp-analytics.feature | 32 ++++++ .../.openspec.yaml | 2 + .../2026-11-05-mcp-error-classifier/design.md | 50 ++++++++ .../proposal.md | 38 ++++++ .../specs/mcp-analytics/spec.md | 108 ++++++++++++++++++ .../2026-11-05-mcp-error-classifier/tasks.md | 23 ++++ openspec/specs/mcp-analytics/spec.md | 87 ++++++++++---- 7 files changed, 317 insertions(+), 23 deletions(-) create mode 100644 openspec/changes/archive/2026-11-05-mcp-error-classifier/.openspec.yaml create mode 100644 openspec/changes/archive/2026-11-05-mcp-error-classifier/design.md create mode 100644 openspec/changes/archive/2026-11-05-mcp-error-classifier/proposal.md create mode 100644 openspec/changes/archive/2026-11-05-mcp-error-classifier/specs/mcp-analytics/spec.md create mode 100644 openspec/changes/archive/2026-11-05-mcp-error-classifier/tasks.md diff --git a/acceptance/public/mcp-analytics.feature b/acceptance/public/mcp-analytics.feature index baef485d..4ac46611 100644 --- a/acceptance/public/mcp-analytics.feature +++ b/acceptance/public/mcp-analytics.feature @@ -146,6 +146,38 @@ Feature: MCP analytics Then the "$mcp_tool_call" event property "$mcp_error_type" should equal "validation" And the "$exception" event's first exception type should name the type "UpstreamTimeout" + @server + Scenario: The error classifier labels an isError result as an expected failure + Given an instrumented MCP server with an error classifier that returns error type "not_found" and expected true and a tool "get_record" that returns a result with isError true and the text "record not found" + When the MCP client calls tool "get_record" with JSON arguments: + """application/json + {"id":"rec_1"} + """ + And pending captures are flushed + Then the "$mcp_tool_call" event property "$mcp_is_error" should equal JSON true + And the "$mcp_tool_call" event property "$mcp_error_type" should equal "not_found" + And the "$mcp_tool_call" event property "$mcp_error_expected" should equal JSON true + And the "$mcp_tool_call" event property "$mcp_error_message" should equal "record not found" + + @server + Scenario: A failed call without a classification has no expected flag + Given an instrumented MCP server with a tool "get_record" that returns a result with isError true and the text "record not found" + When the MCP client calls tool "get_record" with JSON arguments: + """application/json + {"id":"rec_1"} + """ + And pending captures are flushed + Then the "$mcp_tool_call" event property "$mcp_is_error" should equal JSON true + And the "$mcp_tool_call" event should not have property "$mcp_error_expected" + + @server + Scenario: A manually captured failure can be marked expected + Given the SDK's manual MCP capture API records a failed call to tool "query" with error type "rate_limited" and expected true + And pending captures are flushed + Then the "$mcp_tool_call" event property "$mcp_is_error" should equal JSON true + And the "$mcp_tool_call" event property "$mcp_error_type" should equal "rate_limited" + And the "$mcp_tool_call" event property "$mcp_error_expected" should equal JSON true + @server Scenario: A call to a tool that does not exist is not counted as a call Given an instrumented MCP server with a tool "search_docs" that returns the text "ok" diff --git a/openspec/changes/archive/2026-11-05-mcp-error-classifier/.openspec.yaml b/openspec/changes/archive/2026-11-05-mcp-error-classifier/.openspec.yaml new file mode 100644 index 00000000..1491b921 --- /dev/null +++ b/openspec/changes/archive/2026-11-05-mcp-error-classifier/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-11-05 diff --git a/openspec/changes/archive/2026-11-05-mcp-error-classifier/design.md b/openspec/changes/archive/2026-11-05-mcp-error-classifier/design.md new file mode 100644 index 00000000..3285fcef --- /dev/null +++ b/openspec/changes/archive/2026-11-05-mcp-error-classifier/design.md @@ -0,0 +1,50 @@ +## Overview + +The server owner is the only one who knows which failures are expected. The SDK asks them once per +failed tool call, through a callback, and records the answer as data. It does not decide what is +expected, and it does not change what the client receives. + +## Key Decisions + +- **A flag, not a new outcome.** An expected failure keeps `$mcp_is_error: true`. Hiding the failure + (the `beforeSend` workaround in the issue, which sets `$mcp_is_error` to `false`) loses the fact that + the client received an error. A separate property keeps both facts, so anyone reading the data can + count failures with or without expected ones. +- **The callback sees the failure, not only the result.** It gets the tool name and the failure: the + thrown value when the SDK has one, else the `isError` result. A thrown error's own type is the most + reliable thing to classify on (`instanceof RateLimitError`). On frameworks that turn a throw into an + `isError` result before the SDK sees it, the result is all there is. The result is the one the tool + produced, before the SDK appends a conversation handle, the same rule `$mcp_error_message` follows. +- **One object argument.** Platforms pass the failure as one value (an object, a struct, a dataclass) + so it can gain fields later without breaking callbacks. +- **Explicit values win.** A type or flag the caller passes to the manual API wins over the classifier, + and the classifier's type wins over the thrown type, the same precedence explicit types already have. + The classifier's type never renames the `$exception` type, so error tracking still groups by real + types. +- **`$exception` is unchanged.** Whether an expected failure should still emit `$exception` is open + (see below). This change keeps today's behavior, which the host can already change with the + exception autocapture option or the before-send hook. +- **`not_found` joins the vocabulary.** It was the third-largest expected bucket in the reported data + and had no shared label. An ambiguous argument the agent must fix is `validation`. +- **Tool calls only.** Resource reads and listings keep today's rules. Nothing has asked for more. +- **`$mcp_error_expected` is sent only when known.** It is set when the classifier or the caller says + whether the failure is expected, and absent otherwise. Absent reads as not expected. + +## Alternatives Considered + +- **Readers classify by message text.** Possible today, and what the reporting team does. Every reader + has to repeat it for every server, and message text is sanitized and truncated before capture. +- **The host marks expected failures in the before-send hook.** Works without a spec change, but the + hook sees only the sanitized message, never the thrown error or the result, and it runs once per + payload, so the host classifies the `$exception` a second time. +- **Mark the failure inside the tool result (a `_meta` key) or on the thrown error.** The result is + what the client receives, so analytics metadata would leak to it, and on frameworks that flatten a + throw into a result the error's fields are lost. +- **Three-way fault attribution (agent, server, upstream).** More informative, but no reader has asked + for it, and a boolean covers the reported need. + +## Open Questions + +- Should an expected failure skip the `$exception` event? Expected failures are not defects, and each + one creates an error-tracking event the host pays for. Skipping it is a behavior change for every + host that sets a classifier, so it is left out of this change. diff --git a/openspec/changes/archive/2026-11-05-mcp-error-classifier/proposal.md b/openspec/changes/archive/2026-11-05-mcp-error-classifier/proposal.md new file mode 100644 index 00000000..7fe6b47e --- /dev/null +++ b/openspec/changes/archive/2026-11-05-mcp-error-classifier/proposal.md @@ -0,0 +1,38 @@ +## Why + +Some tools fail on purpose. A rate limit, a record that does not exist, or arguments the agent must fix +are outcomes the server returns by design, and the agent is meant to act on them. Today the data cannot +tell them apart from failures the server owner has to fix. An `isError` result from automatic +instrumentation always has `$mcp_error_type: "Error"`, because the spec gives automatic instrumentation +no way to receive an explicit type. One production server reported 148,967 calls in 30 days, with +2,872 failures, all typed `"Error"`. By the owner's own classification, 81% of those failures were +expected (PostHog/posthog#113659). + +## What Changes + +- Add an error classifier option to the Configuration table. It is a server callback that, for a failed + tool call, may return an explicit error type and whether the failure is expected. +- Add `$mcp_error_expected`, a JSON boolean on failed calls, to the Property reference. +- Modify **Failure capture**. Automatic instrumentation takes an explicit type from the classifier. + An expected failure is still a failed call: `$mcp_is_error` stays `true`, and `$exception` is + unchanged. Add `not_found` to the shared type vocabulary. +- Modify **Manual capture**. The result step and the lower-level method take whether the failure is + expected, next to the explicit error type. +- Add scenarios for a classified `isError` result and a manually captured expected failure. + +## Capabilities + +### New Capabilities + +_None._ + +### Modified Capabilities + +- `mcp-analytics`: the **Failure capture** and **Manual capture** requirements, the Configuration + table, and the Property reference. + +## Impact + +Nothing changes for a host that sets no classifier and passes no expected flag. A host that sets one +gets explicit types on failures automatic instrumentation captures, and `$mcp_error_expected` on the +failures it marks. Every SDK that implements MCP analytics adds the option and the manual field. diff --git a/openspec/changes/archive/2026-11-05-mcp-error-classifier/specs/mcp-analytics/spec.md b/openspec/changes/archive/2026-11-05-mcp-error-classifier/specs/mcp-analytics/spec.md new file mode 100644 index 00000000..50125650 --- /dev/null +++ b/openspec/changes/archive/2026-11-05-mcp-error-classifier/specs/mcp-analytics/spec.md @@ -0,0 +1,108 @@ +## MODIFIED Requirements + +### Requirement: Failure capture + +A failed call (an `isError` result or a thrown error) SHALL set `$mcp_is_error: true` and: + +- `$mcp_error_type`: a low-cardinality label, never containing the message. An explicit type wins: one + the caller passes to the manual API, one the error classifier returns, or one the SDK assigns (such + as `validation` for a framework validation error it recognizes), in that order; explicit types + SHOULD come from the shared vocabulary `validation`, `permission`, `missing_context`, `not_found`, + `timeout`, `rate_limited`, `api_4xx`, `api_5xx`, `cancelled`, `input_required`, and `internal`, so + failures group the same way across servers. Otherwise it is the thrown error's own type name (in + JavaScript, its `name` unless that is the generic `"Error"`, then its constructor's name), looking + past wrappers the SDK, standard library, or MCP framework add; otherwise `"Error"`, which is also the + type of an `isError` result without an explicit type. +- `$mcp_error_message`: the thrown message or the text of the tool's own `isError` content (never + the SDK's appended handle), sanitized per "Payload privacy". + +The SDK MAY set `$mcp_error_status` to an upstream HTTP status code. + +A failure can be expected: an outcome the server returns by design for the agent to act on, such as a +rate limit, a record that does not exist, or arguments the agent must fix. Only the server knows which +failures are expected, so the SDK SHALL offer the error classifier option (see "Configuration") on +automatic instrumentation: a server callback that receives the tool name and the failure, and may +return an explicit error type, whether the failure is expected, both, or nothing. The failure is the +thrown value when the SDK has one, else the `isError` result as the tool produced it, before the SDK +appends a conversation handle. The SDK SHALL call it once for each failed tool call that reaches a +final outcome, and for nothing else. When the classifier, or the caller of the manual API, says whether +the failure is expected, the SDK SHALL set `$mcp_error_expected` to that JSON boolean; otherwise it +SHALL NOT set it. A value the caller passes to the manual API wins over the classifier's. An expected +failure is still a failed call: it keeps `$mcp_is_error: true` and every other rule of this +requirement. A classifier that fails or returns nothing leaves the call as it would be without one, and +SHALL NOT change the response. + +When the SDK emits `$exception`, the event SHALL follow the `exception-event-metadata` capability, with +`$exception_source: "mcp.tool_call"`, `mechanism.handled: true` (the framework turns the failure into a +response to the client), and `mechanism.synthetic: true` for an `isError` result that carried no thrown +error; the exception's type SHALL be the thrown error's own type (`"Error"` for an `isError` result), +never the explicit label, so error tracking groups by real types; and the event SHALL carry +`$exception_level: "error"` and the call's `$session_id` and `$mcp_tool_name`. Failure analytics SHALL +NOT depend on `$exception`, which can be disabled. + +#### Scenario: A thrown error is a failed call and still reaches the client +- **GIVEN** an MCP server instrumented by the SDK with a tool "query" whose handler throws an error of a custom type named "UpstreamTimeout" with message "upstream timed out" +- **WHEN** an MCP client calls tool "query" +- **AND** the SDK is flushed +- **THEN** the "$mcp_tool_call" event's property "$mcp_is_error" should be the boolean true +- **AND** its property "$mcp_error_type" should name the "UpstreamTimeout" type (platform spelling, such as "UpstreamTimeout" or "errors.UpstreamTimeout") +- **AND** the client should receive the same failure it would receive without instrumentation + +#### Scenario: The error classifier labels an isError result as an expected failure +- **GIVEN** an MCP server instrumented by the SDK with an error classifier that returns error type "not_found" and expected true, and a tool "get_record" that returns a result with isError true and the text "record not found" +- **WHEN** an MCP client calls tool "get_record" +- **AND** the SDK is flushed +- **THEN** the "$mcp_tool_call" event's property "$mcp_is_error" should be the boolean true +- **AND** its property "$mcp_error_type" should be "not_found" +- **AND** its property "$mcp_error_expected" should be the boolean true +- **AND** its property "$mcp_error_message" should be "record not found" +- **AND** the client should receive the same result it would receive without instrumentation + +#### Scenario: A failed call without a classification has no expected flag +- **GIVEN** an MCP server instrumented by the SDK with no error classifier and a tool "get_record" that returns a result with isError true and the text "record not found" +- **WHEN** an MCP client calls tool "get_record" +- **AND** the SDK is flushed +- **THEN** the "$mcp_tool_call" event should not have property "$mcp_error_expected" + +### Requirement: Manual capture + +An SDK SHALL offer a manual API for servers it cannot instrument automatically, such as a custom +dispatcher, a gateway, or a framework without an adapter. Given the same traffic, its step-based path +SHALL produce the same `$mcp_tool_call` and `$exception` events as automatic instrumentation; for the +events of "Optional events" it MAY offer capture methods. It SHALL work in three steps. A list step +takes the server's tools and returns them with the injected arguments advertised. A call step takes the +tool's registered definition (input schema, description, `_meta`) when the tool exists, the arguments +as the agent sent them, the request's `_meta`, the HTTP headers when there are any, the transport +session when there is one, whether the tool exists (known to exist, known not to, or unknown for a +server with no registry), and the time the request started; it returns the arguments for the handler +and a call context, and for a tool known not to exist it emits `$mcp_unknown_tool` where the SDK does. +The transport session is either a raw id, derived per rule 2 of "Session identity", or a key the caller +keeps for the lifetime of a one-client connection, which the SDK tracks and rotates. A result step +takes the call context and the outcome (the result or the error, an explicit error type and whether the +failure is expected if the caller knows them, and when it ended); it sends the events and returns the +result to deliver, with a minted handle appended. From these inputs the SDK applies every other rule of +this spec itself: the `$mcp_parameters` shape, intent and model, `$session_id`, person and client +identity, failure capture (including the error classifier, for what the caller leaves unset), privacy, +and size bounds. + +A lower-level method MAY also accept a finished call's fields directly, for a server that already holds +them; it is exempt from producing the same events as automatic instrumentation. It sends +`$mcp_parameters` and `$mcp_response` as the caller passes them, takes whether a failure is expected +next to its explicit error type, and follows the rest of this spec for everything else. A `$session_id` +the caller passes is sent unchanged as the transport session unless a conversation handle applies; +without either, each call gets a fresh `ses_`, as rule 3 of "Session identity" gives a +stateless request, so a server on a one-client connection SHOULD pass its connection's id. + +#### Scenario: A manually captured call matches the automatic one +- **GIVEN** an MCP server that dispatches tool "search_docs" itself through the SDK's manual API +- **WHEN** an MCP client calls tool "search_docs" with arguments `{"query": "flags", "context": "find the flags docs"}` +- **AND** the SDK is flushed +- **THEN** the "$mcp_tool_call" event's properties "$mcp_tool_name", "$mcp_intent", "$mcp_parameters", and "$mcp_is_error" should equal those automatic instrumentation sends for the same call +- **AND** its property "$session_id" should be a non-empty string + +#### Scenario: A manually captured failure can be marked expected +- **GIVEN** an MCP server that records a failed call to tool "query" through the SDK's manual API with error type "rate_limited" and expected true +- **WHEN** the SDK is flushed +- **THEN** the "$mcp_tool_call" event's property "$mcp_is_error" should be the boolean true +- **AND** its property "$mcp_error_type" should be "rate_limited" +- **AND** its property "$mcp_error_expected" should be the boolean true diff --git a/openspec/changes/archive/2026-11-05-mcp-error-classifier/tasks.md b/openspec/changes/archive/2026-11-05-mcp-error-classifier/tasks.md new file mode 100644 index 00000000..9cf19a69 --- /dev/null +++ b/openspec/changes/archive/2026-11-05-mcp-error-classifier/tasks.md @@ -0,0 +1,23 @@ +## 1. Spec delta + +- [x] 1.1 Modify the **Failure capture** requirement as a full copied-and-edited block +- [x] 1.2 Add the classifier, `$mcp_error_expected`, and `not_found` to it +- [x] 1.3 Modify the **Manual capture** requirement so the result step and the lower-level method take + the expected flag +- [x] 1.4 Add scenarios for a classified `isError` result and a manual expected failure + +## 2. Prose alignment (applied at archive, outside the requirement-delta mechanism) + +- [x] 2.1 Add the error classifier row to the Configuration table +- [x] 2.2 Add `$mcp_error_expected` to the Property reference +- [x] 2.3 Add both scenarios to `acceptance/public/mcp-analytics.feature` + +## 3. Validation + +- [x] 3.1 Run `openspec validate --specs --strict` +- [x] 3.2 Run `openspec archive` to sync the delta and archive the change + +## 4. Downstream follow-ups (separate changes, not this one) + +- [ ] 4.1 posthog-sdk-test-harness: step bindings for the new scenarios +- [ ] 4.2 Implement the option and the manual field in every SDK that implements MCP analytics diff --git a/openspec/specs/mcp-analytics/spec.md b/openspec/specs/mcp-analytics/spec.md index 004030be..5c31887c 100644 --- a/openspec/specs/mcp-analytics/spec.md +++ b/openspec/specs/mcp-analytics/spec.md @@ -72,6 +72,7 @@ table is the one place defaults are stated. | conversation anchoring | on | Injects `conversation_id`, mints handles, and derives `$session_id` from them; a handle-free client's listing gets no `conversation_id` and its calls get no minted handle (see "Session identity") | | identify | none | Resolves `distinct_id`, `$set`, and `$groups` per request | | exception autocapture | on | Emits `$exception` next to failed calls | +| error classifier | none | Server callback that, for a failed tool call, returns an explicit error type, whether the failure is expected, both, or nothing | | initialize event | on | Sends `$mcp_initialize` for each legacy `initialize`; off sends none and changes nothing else | | tools list event | on | Sends `$mcp_tools_list` for each `tools/list`; off sends none and changes nothing else | | parameter capture | on | Sends `$mcp_parameters` | @@ -109,6 +110,7 @@ means set whenever the SDK can determine the value. Events: "call" is `$mcp_tool | `$mcp_error_type` | failed call | string, low cardinality, at most 256 characters | failed calls | | `$mcp_error_message` | failed call | string, at most 2,048 characters | failed calls | | `$mcp_error_status` | failed call | JSON number, an HTTP status code | may | +| `$mcp_error_expected` | failed call | JSON boolean | when the error classifier or the manual API's caller says | | `$mcp_server_build` | all | string, 1 to 256 characters, an immutable build id (a commit SHA or image digest); rejected at configuration when out of range, never truncated or shrunk | may | | `$mcp_input_keys` | call | array of at most 20 entries: recorded top-level argument names, declared ones (the schema's and accepted aliases) sorted, then others sorted, then one `"[redacted]"` entry if room remains and a name was not recorded or exceeded 64 characters; injected names the tool does not declare are excluded | may | | `$mcp_input_aliases_used` | call | array of at most 20 sorted `":"` strings, each name at most 64 characters, one for each argument the agent sent only under an alias, naming the first alias present in the server's order | may | @@ -231,9 +233,9 @@ conversations. A failed call (an `isError` result or a thrown error) SHALL set `$mcp_is_error: true` and: - `$mcp_error_type`: a low-cardinality label, never containing the message. An explicit type wins: one - the caller passes to the manual API, or one the SDK assigns (such as `validation` for a framework - validation error it recognizes); automatic instrumentation has no other way to receive one; explicit - types SHOULD come from the shared vocabulary `validation`, `permission`, `missing_context`, + the caller passes to the manual API, one the error classifier returns, or one the SDK assigns (such + as `validation` for a framework validation error it recognizes), in that order; explicit types + SHOULD come from the shared vocabulary `validation`, `permission`, `missing_context`, `not_found`, `timeout`, `rate_limited`, `api_4xx`, `api_5xx`, `cancelled`, `input_required`, and `internal`, so failures group the same way across servers. Otherwise it is the thrown error's own type name (in JavaScript, its `name` unless that is the generic `"Error"`, then its constructor's name), looking @@ -242,14 +244,29 @@ A failed call (an `isError` result or a thrown error) SHALL set `$mcp_is_error: - `$mcp_error_message`: the thrown message or the text of the tool's own `isError` content (never the SDK's appended handle), sanitized per "Payload privacy". -The SDK MAY set `$mcp_error_status` to an upstream HTTP status code. When it emits `$exception`, the -event SHALL follow the `exception-event-metadata` capability, with `$exception_source: -"mcp.tool_call"`, `mechanism.handled: true` (the framework turns the failure into a response to the -client), and `mechanism.synthetic: true` for an `isError` result that carried no thrown error; the -exception's type SHALL be the thrown error's own type (`"Error"` for an `isError` result), never the -explicit label, so error tracking groups by real types; and the event SHALL carry `$exception_level: -"error"` and the call's `$session_id` and `$mcp_tool_name`. Failure analytics SHALL NOT depend on -`$exception`, which can be disabled. +The SDK MAY set `$mcp_error_status` to an upstream HTTP status code. + +A failure can be expected: an outcome the server returns by design for the agent to act on, such as a +rate limit, a record that does not exist, or arguments the agent must fix. Only the server knows which +failures are expected, so the SDK SHALL offer the error classifier option (see "Configuration") on +automatic instrumentation: a server callback that receives the tool name and the failure, and may +return an explicit error type, whether the failure is expected, both, or nothing. The failure is the +thrown value when the SDK has one, else the `isError` result as the tool produced it, before the SDK +appends a conversation handle. The SDK SHALL call it once for each failed tool call that reaches a +final outcome, and for nothing else. When the classifier, or the caller of the manual API, says whether +the failure is expected, the SDK SHALL set `$mcp_error_expected` to that JSON boolean; otherwise it +SHALL NOT set it. A value the caller passes to the manual API wins over the classifier's. An expected +failure is still a failed call: it keeps `$mcp_is_error: true` and every other rule of this +requirement. A classifier that fails or returns nothing leaves the call as it would be without one, and +SHALL NOT change the response. + +When the SDK emits `$exception`, the event SHALL follow the `exception-event-metadata` capability, with +`$exception_source: "mcp.tool_call"`, `mechanism.handled: true` (the framework turns the failure into a +response to the client), and `mechanism.synthetic: true` for an `isError` result that carried no thrown +error; the exception's type SHALL be the thrown error's own type (`"Error"` for an `isError` result), +never the explicit label, so error tracking groups by real types; and the event SHALL carry +`$exception_level: "error"` and the call's `$session_id` and `$mcp_tool_name`. Failure analytics SHALL +NOT depend on `$exception`, which can be disabled. #### Scenario: A thrown error is a failed call and still reaches the client - **GIVEN** an MCP server instrumented by the SDK with a tool "query" whose handler throws an error of a custom type named "UpstreamTimeout" with message "upstream timed out" @@ -259,6 +276,22 @@ explicit label, so error tracking groups by real types; and the event SHALL carr - **AND** its property "$mcp_error_type" should name the "UpstreamTimeout" type (platform spelling, such as "UpstreamTimeout" or "errors.UpstreamTimeout") - **AND** the client should receive the same failure it would receive without instrumentation +#### Scenario: The error classifier labels an isError result as an expected failure +- **GIVEN** an MCP server instrumented by the SDK with an error classifier that returns error type "not_found" and expected true, and a tool "get_record" that returns a result with isError true and the text "record not found" +- **WHEN** an MCP client calls tool "get_record" +- **AND** the SDK is flushed +- **THEN** the "$mcp_tool_call" event's property "$mcp_is_error" should be the boolean true +- **AND** its property "$mcp_error_type" should be "not_found" +- **AND** its property "$mcp_error_expected" should be the boolean true +- **AND** its property "$mcp_error_message" should be "record not found" +- **AND** the client should receive the same result it would receive without instrumentation + +#### Scenario: A failed call without a classification has no expected flag +- **GIVEN** an MCP server instrumented by the SDK with no error classifier and a tool "get_record" that returns a result with isError true and the text "record not found" +- **WHEN** an MCP client calls tool "get_record" +- **AND** the SDK is flushed +- **THEN** the "$mcp_tool_call" event should not have property "$mcp_error_expected" + ### Requirement: Session identity Every MCP event SHALL carry a non-empty `$session_id` that groups one agent conversation as far as the @@ -575,21 +608,22 @@ server with no registry), and the time the request started; it returns the argum and a call context, and for a tool known not to exist it emits `$mcp_unknown_tool` where the SDK does. The transport session is either a raw id, derived per rule 3 of "Session identity", or a key the caller keeps for the lifetime of a one-client connection, which the SDK tracks and rotates. A result step -takes the call context and the outcome (the result or the error, an explicit error type if any, and -when it ended); it sends the events and returns the result to deliver, with a minted handle appended. -From these inputs the SDK applies every other rule of this spec itself: the `$mcp_parameters` shape, -intent and model, `$session_id`, person and client identity, failure capture, privacy, and size bounds. +takes the call context and the outcome (the result or the error, an explicit error type and whether the +failure is expected if the caller knows them, and when it ended); it sends the events and returns the +result to deliver, with a minted handle appended. From these inputs the SDK applies every other rule of +this spec itself: the `$mcp_parameters` shape, intent and model, `$session_id`, person and client +identity, failure capture (including the error classifier, for what the caller leaves unset), privacy, +and size bounds. A lower-level method MAY also accept a finished call's fields directly, for a server that already holds them; it is exempt from producing the same events as automatic instrumentation. It sends -`$mcp_parameters` and `$mcp_response` as the caller passes them and follows the rest of this spec for -everything else. A client conversation id applies only when the caller passes the request's `_meta` -together with the `User-Agent` or client name that recognizes its client. A `$session_id` the caller -passes is sent unchanged as the transport session unless a client conversation id or a conversation -handle applies; without any of these, each call gets a fresh -`ses_`, as rule 4 of -"Session identity" gives a stateless request, so a server on a one-client connection SHOULD pass its -connection's id. +`$mcp_parameters` and `$mcp_response` as the caller passes them, takes whether a failure is expected +next to its explicit error type, and follows the rest of this spec for everything else. A client +conversation id applies only when the caller passes the request's `_meta` together with the `User-Agent` +or client name that recognizes its client. A `$session_id` the caller passes is sent unchanged as the +transport session unless a client conversation id or a conversation handle applies; without any of +these, each call gets a fresh `ses_`, as rule 4 of "Session identity" gives a stateless +request, so a server on a one-client connection SHOULD pass its connection's id. #### Scenario: A manually captured call matches the automatic one - **GIVEN** an MCP server that dispatches tool "search_docs" itself through the SDK's manual API @@ -598,6 +632,13 @@ connection's id. - **THEN** the "$mcp_tool_call" event's properties "$mcp_tool_name", "$mcp_intent", "$mcp_parameters", and "$mcp_is_error" should equal those automatic instrumentation sends for the same call - **AND** its property "$session_id" should be a non-empty string +#### Scenario: A manually captured failure can be marked expected +- **GIVEN** an MCP server that records a failed call to tool "query" through the SDK's manual API with error type "rate_limited" and expected true +- **WHEN** the SDK is flushed +- **THEN** the "$mcp_tool_call" event's property "$mcp_is_error" should be the boolean true +- **AND** its property "$mcp_error_type" should be "rate_limited" +- **AND** its property "$mcp_error_expected" should be the boolean true + ### Requirement: Payload privacy Captured strings SHALL NOT carry the credentials and personal data the rules below recognize. Each From afa6e37ff5055c7eda335b6d4fdbf1213494509f Mon Sep 17 00:00:00 2001 From: Lucas Faria Date: Thu, 5 Nov 2026 09:43:59 -0300 Subject: [PATCH 2/5] docs(mcp-analytics): define the classifier input for every failed call The pre-PR gate found the classifier had no defined input for failures the SDK types itself (failed or cancelled tasks, an input_required sequence the framework ends), let it relabel those fixed types, left "the thrown value" ambiguous where frameworks wrap tool errors (Python FastMCP re-raises as ToolError), and did not say whether the manual lower-level method calls it. Now the classifier always gets the tool name and the type and message the call would carry, plus the tool's own failure (past wrappers) when the SDK holds it. It runs only for thrown errors and isError results, at most once, on automatic instrumentation and the manual step path, never on the lower-level method. The acceptance scenario also checks that the client gets the same result. Tested: openspec validate --specs --strict (69 passed); delta and canonical Failure capture blocks match; feature parses (42 scenarios). --- acceptance/public/mcp-analytics.feature | 3 ++- .../2026-11-05-mcp-error-classifier/design.md | 22 ++++++++++++----- .../specs/mcp-analytics/spec.md | 24 +++++++++++-------- openspec/specs/mcp-analytics/spec.md | 24 +++++++++++-------- 4 files changed, 46 insertions(+), 27 deletions(-) diff --git a/acceptance/public/mcp-analytics.feature b/acceptance/public/mcp-analytics.feature index 4ac46611..f2cc023d 100644 --- a/acceptance/public/mcp-analytics.feature +++ b/acceptance/public/mcp-analytics.feature @@ -148,7 +148,7 @@ Feature: MCP analytics @server Scenario: The error classifier labels an isError result as an expected failure - Given an instrumented MCP server with an error classifier that returns error type "not_found" and expected true and a tool "get_record" that returns a result with isError true and the text "record not found" + Given an instrumented MCP server with conversation anchoring disabled, an error classifier that returns error type "not_found" and expected true, and a tool "get_record" that returns a result with isError true and the text "record not found" When the MCP client calls tool "get_record" with JSON arguments: """application/json {"id":"rec_1"} @@ -158,6 +158,7 @@ Feature: MCP analytics And the "$mcp_tool_call" event property "$mcp_error_type" should equal "not_found" And the "$mcp_tool_call" event property "$mcp_error_expected" should equal JSON true And the "$mcp_tool_call" event property "$mcp_error_message" should equal "record not found" + And the MCP client should receive a result with exactly one text block "record not found" and isError true @server Scenario: A failed call without a classification has no expected flag diff --git a/openspec/changes/archive/2026-11-05-mcp-error-classifier/design.md b/openspec/changes/archive/2026-11-05-mcp-error-classifier/design.md index 3285fcef..6f0073f4 100644 --- a/openspec/changes/archive/2026-11-05-mcp-error-classifier/design.md +++ b/openspec/changes/archive/2026-11-05-mcp-error-classifier/design.md @@ -10,13 +10,23 @@ expected, and it does not change what the client receives. (the `beforeSend` workaround in the issue, which sets `$mcp_is_error` to `false`) loses the fact that the client received an error. A separate property keeps both facts, so anyone reading the data can count failures with or without expected ones. -- **The callback sees the failure, not only the result.** It gets the tool name and the failure: the - thrown value when the SDK has one, else the `isError` result. A thrown error's own type is the most - reliable thing to classify on (`instanceof RateLimitError`). On frameworks that turn a throw into an - `isError` result before the SDK sees it, the result is all there is. The result is the one the tool +- **The callback always gets the type and message, and the failure when the SDK holds it.** The type + and message `$mcp_error_type` and `$mcp_error_message` would carry exist for every failed call in + every SDK, so a classifier keyed on them works the same everywhere. The failure itself is richer: a + thrown error's own type is the most reliable thing to classify on (`instanceof RateLimitError`). It + is the tool's own error, past the same wrappers `$mcp_error_type` looks past, because Python's + FastMCP re-raises every tool error as a `ToolError` and a classifier would never match the tool's + type. On frameworks that turn a throw into an `isError` result before the SDK sees it, the result is + all there is, and some manual APIs keep only a type name and message. The result is the one the tool produced, before the SDK appends a conversation handle, the same rule `$mcp_error_message` follows. -- **One object argument.** Platforms pass the failure as one value (an object, a struct, a dataclass) - so it can gain fields later without breaking callbacks. +- **One object argument.** Platforms pass these as one value (an object, a struct, a dataclass) so it + can gain fields later without breaking callbacks. +- **Only failures the tool produced.** A failed or cancelled task and an `input_required` sequence the + framework ends are failures the SDK observes and types itself, with no thrown error or `isError` + result to hand over. The classifier is not called for them, so their fixed types stand in every SDK. +- **Where it runs.** Automatic instrumentation and the manual step path call it. The step path calls it + only for what the caller leaves unset. The lower-level method takes finished fields and has no + failure to hand over, so it never calls it. - **Explicit values win.** A type or flag the caller passes to the manual API wins over the classifier, and the classifier's type wins over the thrown type, the same precedence explicit types already have. The classifier's type never renames the `$exception` type, so error tracking still groups by real diff --git a/openspec/changes/archive/2026-11-05-mcp-error-classifier/specs/mcp-analytics/spec.md b/openspec/changes/archive/2026-11-05-mcp-error-classifier/specs/mcp-analytics/spec.md index 50125650..16d62146 100644 --- a/openspec/changes/archive/2026-11-05-mcp-error-classifier/specs/mcp-analytics/spec.md +++ b/openspec/changes/archive/2026-11-05-mcp-error-classifier/specs/mcp-analytics/spec.md @@ -20,14 +20,19 @@ The SDK MAY set `$mcp_error_status` to an upstream HTTP status code. A failure can be expected: an outcome the server returns by design for the agent to act on, such as a rate limit, a record that does not exist, or arguments the agent must fix. Only the server knows which -failures are expected, so the SDK SHALL offer the error classifier option (see "Configuration") on -automatic instrumentation: a server callback that receives the tool name and the failure, and may -return an explicit error type, whether the failure is expected, both, or nothing. The failure is the -thrown value when the SDK has one, else the `isError` result as the tool produced it, before the SDK -appends a conversation handle. The SDK SHALL call it once for each failed tool call that reaches a -final outcome, and for nothing else. When the classifier, or the caller of the manual API, says whether -the failure is expected, the SDK SHALL set `$mcp_error_expected` to that JSON boolean; otherwise it -SHALL NOT set it. A value the caller passes to the manual API wins over the classifier's. An expected +failures are expected, so the SDK SHALL offer the error classifier option (see "Configuration"): a +server callback that receives one value holding the tool name, the `$mcp_error_type` and +`$mcp_error_message` the call would carry without a classifier, and the failure itself when the SDK +holds it. The failure is the tool's own thrown error, looking past the same wrappers as +`$mcp_error_type`, else the `isError` result as the tool produced it, before the SDK appends a +conversation handle. The classifier may return an explicit error type, whether the failure is expected, +both, or nothing. The SDK SHALL call it at most once per tool call, for a failed call that ends in a +thrown error or an `isError` result, and for nothing else: never for a failed or cancelled task or an +`input_required` sequence the framework ends, whose types "Tool call event" and "Optional events" fix. +On the manual step path the SDK calls it only when the caller leaves the type or the expected flag +unset, and the caller's values win; the lower-level method never calls it. When the classifier or the +manual API's caller says whether the failure is expected, the SDK SHALL set `$mcp_error_expected` to +that JSON boolean; otherwise it SHALL NOT set it. An expected failure is still a failed call: it keeps `$mcp_is_error: true` and every other rule of this requirement. A classifier that fails or returns nothing leaves the call as it would be without one, and SHALL NOT change the response. @@ -82,8 +87,7 @@ takes the call context and the outcome (the result or the error, an explicit err failure is expected if the caller knows them, and when it ended); it sends the events and returns the result to deliver, with a minted handle appended. From these inputs the SDK applies every other rule of this spec itself: the `$mcp_parameters` shape, intent and model, `$session_id`, person and client -identity, failure capture (including the error classifier, for what the caller leaves unset), privacy, -and size bounds. +identity, failure capture (including the error classifier), privacy, and size bounds. A lower-level method MAY also accept a finished call's fields directly, for a server that already holds them; it is exempt from producing the same events as automatic instrumentation. It sends diff --git a/openspec/specs/mcp-analytics/spec.md b/openspec/specs/mcp-analytics/spec.md index 5c31887c..72e0657c 100644 --- a/openspec/specs/mcp-analytics/spec.md +++ b/openspec/specs/mcp-analytics/spec.md @@ -248,14 +248,19 @@ The SDK MAY set `$mcp_error_status` to an upstream HTTP status code. A failure can be expected: an outcome the server returns by design for the agent to act on, such as a rate limit, a record that does not exist, or arguments the agent must fix. Only the server knows which -failures are expected, so the SDK SHALL offer the error classifier option (see "Configuration") on -automatic instrumentation: a server callback that receives the tool name and the failure, and may -return an explicit error type, whether the failure is expected, both, or nothing. The failure is the -thrown value when the SDK has one, else the `isError` result as the tool produced it, before the SDK -appends a conversation handle. The SDK SHALL call it once for each failed tool call that reaches a -final outcome, and for nothing else. When the classifier, or the caller of the manual API, says whether -the failure is expected, the SDK SHALL set `$mcp_error_expected` to that JSON boolean; otherwise it -SHALL NOT set it. A value the caller passes to the manual API wins over the classifier's. An expected +failures are expected, so the SDK SHALL offer the error classifier option (see "Configuration"): a +server callback that receives one value holding the tool name, the `$mcp_error_type` and +`$mcp_error_message` the call would carry without a classifier, and the failure itself when the SDK +holds it. The failure is the tool's own thrown error, looking past the same wrappers as +`$mcp_error_type`, else the `isError` result as the tool produced it, before the SDK appends a +conversation handle. The classifier may return an explicit error type, whether the failure is expected, +both, or nothing. The SDK SHALL call it at most once per tool call, for a failed call that ends in a +thrown error or an `isError` result, and for nothing else: never for a failed or cancelled task or an +`input_required` sequence the framework ends, whose types "Tool call event" and "Optional events" fix. +On the manual step path the SDK calls it only when the caller leaves the type or the expected flag +unset, and the caller's values win; the lower-level method never calls it. When the classifier or the +manual API's caller says whether the failure is expected, the SDK SHALL set `$mcp_error_expected` to +that JSON boolean; otherwise it SHALL NOT set it. An expected failure is still a failed call: it keeps `$mcp_is_error: true` and every other rule of this requirement. A classifier that fails or returns nothing leaves the call as it would be without one, and SHALL NOT change the response. @@ -612,8 +617,7 @@ takes the call context and the outcome (the result or the error, an explicit err failure is expected if the caller knows them, and when it ended); it sends the events and returns the result to deliver, with a minted handle appended. From these inputs the SDK applies every other rule of this spec itself: the `$mcp_parameters` shape, intent and model, `$session_id`, person and client -identity, failure capture (including the error classifier, for what the caller leaves unset), privacy, -and size bounds. +identity, failure capture (including the error classifier), privacy, and size bounds. A lower-level method MAY also accept a finished call's fields directly, for a server that already holds them; it is exempt from producing the same events as automatic instrumentation. It sends From b0485bebc3c3335e9382b19a39be7df828f19a11 Mon Sep 17 00:00:00 2001 From: Lucas Faria Date: Thu, 5 Nov 2026 09:51:00 -0300 Subject: [PATCH 3/5] docs(mcp-analytics): keep the classifier on automatic instrumentation Second gate round fixes. An empty or blank explicit type now counts as no type (TS dropped $mcp_error_type, Python and Ruby kept " ", Go fell back), and explicit types are sanitized as free text (all four SDKs sent a phc_ token in a caller-passed type unredacted). The manual API never calls the classifier, since no SDK's result step sends events and the caller already holds the failure. "At most once" becomes "exactly once". Tested: openspec validate --specs --strict (69 passed); delta matches the canonical Failure capture and Manual capture blocks. --- .../2026-11-05-mcp-error-classifier/design.md | 13 +++--- .../proposal.md | 3 +- .../specs/mcp-analytics/spec.md | 45 ++++++++++--------- openspec/specs/mcp-analytics/spec.md | 45 ++++++++++--------- 4 files changed, 56 insertions(+), 50 deletions(-) diff --git a/openspec/changes/archive/2026-11-05-mcp-error-classifier/design.md b/openspec/changes/archive/2026-11-05-mcp-error-classifier/design.md index 6f0073f4..54604555 100644 --- a/openspec/changes/archive/2026-11-05-mcp-error-classifier/design.md +++ b/openspec/changes/archive/2026-11-05-mcp-error-classifier/design.md @@ -24,11 +24,14 @@ expected, and it does not change what the client receives. - **Only failures the tool produced.** A failed or cancelled task and an `input_required` sequence the framework ends are failures the SDK observes and types itself, with no thrown error or `isError` result to hand over. The classifier is not called for them, so their fixed types stand in every SDK. -- **Where it runs.** Automatic instrumentation and the manual step path call it. The step path calls it - only for what the caller leaves unset. The lower-level method takes finished fields and has no - failure to hand over, so it never calls it. -- **Explicit values win.** A type or flag the caller passes to the manual API wins over the classifier, - and the classifier's type wins over the thrown type, the same precedence explicit types already have. +- **Automatic instrumentation only.** A manual API caller already holds the failure and passes the + type and the expected flag itself, so a callback adds nothing there. It would also run on different + paths in different SDKs, because not every SDK's manual result step sends events. +- **Explicit types are data like any other.** A classifier can derive its label from the raw failure, + so an explicit type is sanitized as free text, and an empty or blank one counts as no type. Without + that rule, today's SDKs send a blank type in some cases and drop the property in others. +- **Explicit values win.** The classifier's type wins over the type the SDK assigns and the thrown type, + the same precedence explicit types already have. The classifier's type never renames the `$exception` type, so error tracking still groups by real types. - **`$exception` is unchanged.** Whether an expected failure should still emit `$exception` is open diff --git a/openspec/changes/archive/2026-11-05-mcp-error-classifier/proposal.md b/openspec/changes/archive/2026-11-05-mcp-error-classifier/proposal.md index 7fe6b47e..abf68def 100644 --- a/openspec/changes/archive/2026-11-05-mcp-error-classifier/proposal.md +++ b/openspec/changes/archive/2026-11-05-mcp-error-classifier/proposal.md @@ -17,7 +17,8 @@ expected (PostHog/posthog#113659). An expected failure is still a failed call: `$mcp_is_error` stays `true`, and `$exception` is unchanged. Add `not_found` to the shared type vocabulary. - Modify **Manual capture**. The result step and the lower-level method take whether the failure is - expected, next to the explicit error type. + expected, next to the explicit error type. The manual API never calls the classifier. +- An empty or blank explicit type is no type, and every explicit type is sanitized as free text. - Add scenarios for a classified `isError` result and a manually captured expected failure. ## Capabilities diff --git a/openspec/changes/archive/2026-11-05-mcp-error-classifier/specs/mcp-analytics/spec.md b/openspec/changes/archive/2026-11-05-mcp-error-classifier/specs/mcp-analytics/spec.md index 16d62146..375b5946 100644 --- a/openspec/changes/archive/2026-11-05-mcp-error-classifier/specs/mcp-analytics/spec.md +++ b/openspec/changes/archive/2026-11-05-mcp-error-classifier/specs/mcp-analytics/spec.md @@ -5,14 +5,16 @@ A failed call (an `isError` result or a thrown error) SHALL set `$mcp_is_error: true` and: - `$mcp_error_type`: a low-cardinality label, never containing the message. An explicit type wins: one - the caller passes to the manual API, one the error classifier returns, or one the SDK assigns (such - as `validation` for a framework validation error it recognizes), in that order; explicit types - SHOULD come from the shared vocabulary `validation`, `permission`, `missing_context`, `not_found`, - `timeout`, `rate_limited`, `api_4xx`, `api_5xx`, `cancelled`, `input_required`, and `internal`, so - failures group the same way across servers. Otherwise it is the thrown error's own type name (in - JavaScript, its `name` unless that is the generic `"Error"`, then its constructor's name), looking - past wrappers the SDK, standard library, or MCP framework add; otherwise `"Error"`, which is also the - type of an `isError` result without an explicit type. + the caller passes to the manual API, one the error classifier returns, or one the SDK assigns (such as + `validation` for a framework validation error it recognizes), in that order; explicit types SHOULD + come from the shared vocabulary `validation`, `permission`, `missing_context`, `not_found`, `timeout`, + `rate_limited`, `api_4xx`, `api_5xx`, `cancelled`, `input_required`, and `internal`, so failures group + the same way across servers. An empty or blank type from any source is no type, so the next source + applies, and an explicit type is sanitized as free text per "Payload privacy" like any other captured + string. Otherwise it is the thrown error's own type name (in JavaScript, its `name` unless that is the + generic `"Error"`, then its constructor's name), looking past wrappers the SDK, standard library, or + MCP framework add; otherwise `"Error"`, which is also the type of an `isError` result without an + explicit type. - `$mcp_error_message`: the thrown message or the text of the tool's own `isError` content (never the SDK's appended handle), sanitized per "Payload privacy". @@ -20,22 +22,21 @@ The SDK MAY set `$mcp_error_status` to an upstream HTTP status code. A failure can be expected: an outcome the server returns by design for the agent to act on, such as a rate limit, a record that does not exist, or arguments the agent must fix. Only the server knows which -failures are expected, so the SDK SHALL offer the error classifier option (see "Configuration"): a -server callback that receives one value holding the tool name, the `$mcp_error_type` and -`$mcp_error_message` the call would carry without a classifier, and the failure itself when the SDK +failures are expected, so automatic instrumentation SHALL offer the error classifier option (see +"Configuration"): a server callback that receives one value holding the tool name, the `$mcp_error_type` +and `$mcp_error_message` the call would carry without a classifier, and the failure itself when the SDK holds it. The failure is the tool's own thrown error, looking past the same wrappers as `$mcp_error_type`, else the `isError` result as the tool produced it, before the SDK appends a conversation handle. The classifier may return an explicit error type, whether the failure is expected, -both, or nothing. The SDK SHALL call it at most once per tool call, for a failed call that ends in a -thrown error or an `isError` result, and for nothing else: never for a failed or cancelled task or an -`input_required` sequence the framework ends, whose types "Tool call event" and "Optional events" fix. -On the manual step path the SDK calls it only when the caller leaves the type or the expected flag -unset, and the caller's values win; the lower-level method never calls it. When the classifier or the -manual API's caller says whether the failure is expected, the SDK SHALL set `$mcp_error_expected` to -that JSON boolean; otherwise it SHALL NOT set it. An expected -failure is still a failed call: it keeps `$mcp_is_error: true` and every other rule of this -requirement. A classifier that fails or returns nothing leaves the call as it would be without one, and -SHALL NOT change the response. +both, or nothing. The SDK SHALL call it exactly once for each failed call that ends in a thrown error or +an `isError` result, and for nothing else: never for a failed or cancelled task or an `input_required` +sequence the framework ends, whose types "Tool call event" and "Optional events" fix. The manual API +never calls it: its caller holds the failure and passes the type and the expected flag itself. When the +classifier or the manual API's caller says whether the failure is expected, the SDK SHALL set +`$mcp_error_expected` to that JSON boolean; otherwise it SHALL NOT set it. An expected failure is still +a failed call: it keeps `$mcp_is_error: true` and every other rule of this requirement. A classifier +that fails or returns nothing leaves the call as it would be without one, and SHALL NOT change the +response. When the SDK emits `$exception`, the event SHALL follow the `exception-event-metadata` capability, with `$exception_source: "mcp.tool_call"`, `mechanism.handled: true` (the framework turns the failure into a @@ -87,7 +88,7 @@ takes the call context and the outcome (the result or the error, an explicit err failure is expected if the caller knows them, and when it ended); it sends the events and returns the result to deliver, with a minted handle appended. From these inputs the SDK applies every other rule of this spec itself: the `$mcp_parameters` shape, intent and model, `$session_id`, person and client -identity, failure capture (including the error classifier), privacy, and size bounds. +identity, failure capture (apart from the error classifier), privacy, and size bounds. A lower-level method MAY also accept a finished call's fields directly, for a server that already holds them; it is exempt from producing the same events as automatic instrumentation. It sends diff --git a/openspec/specs/mcp-analytics/spec.md b/openspec/specs/mcp-analytics/spec.md index 72e0657c..3f36d63c 100644 --- a/openspec/specs/mcp-analytics/spec.md +++ b/openspec/specs/mcp-analytics/spec.md @@ -233,14 +233,16 @@ conversations. A failed call (an `isError` result or a thrown error) SHALL set `$mcp_is_error: true` and: - `$mcp_error_type`: a low-cardinality label, never containing the message. An explicit type wins: one - the caller passes to the manual API, one the error classifier returns, or one the SDK assigns (such - as `validation` for a framework validation error it recognizes), in that order; explicit types - SHOULD come from the shared vocabulary `validation`, `permission`, `missing_context`, `not_found`, - `timeout`, `rate_limited`, `api_4xx`, `api_5xx`, `cancelled`, `input_required`, and `internal`, so - failures group the same way across servers. Otherwise it is the thrown error's own type name (in - JavaScript, its `name` unless that is the generic `"Error"`, then its constructor's name), looking - past wrappers the SDK, standard library, or MCP framework add; otherwise `"Error"`, which is also the - type of an `isError` result without an explicit type. + the caller passes to the manual API, one the error classifier returns, or one the SDK assigns (such as + `validation` for a framework validation error it recognizes), in that order; explicit types SHOULD + come from the shared vocabulary `validation`, `permission`, `missing_context`, `not_found`, `timeout`, + `rate_limited`, `api_4xx`, `api_5xx`, `cancelled`, `input_required`, and `internal`, so failures group + the same way across servers. An empty or blank type from any source is no type, so the next source + applies, and an explicit type is sanitized as free text per "Payload privacy" like any other captured + string. Otherwise it is the thrown error's own type name (in JavaScript, its `name` unless that is the + generic `"Error"`, then its constructor's name), looking past wrappers the SDK, standard library, or + MCP framework add; otherwise `"Error"`, which is also the type of an `isError` result without an + explicit type. - `$mcp_error_message`: the thrown message or the text of the tool's own `isError` content (never the SDK's appended handle), sanitized per "Payload privacy". @@ -248,22 +250,21 @@ The SDK MAY set `$mcp_error_status` to an upstream HTTP status code. A failure can be expected: an outcome the server returns by design for the agent to act on, such as a rate limit, a record that does not exist, or arguments the agent must fix. Only the server knows which -failures are expected, so the SDK SHALL offer the error classifier option (see "Configuration"): a -server callback that receives one value holding the tool name, the `$mcp_error_type` and -`$mcp_error_message` the call would carry without a classifier, and the failure itself when the SDK +failures are expected, so automatic instrumentation SHALL offer the error classifier option (see +"Configuration"): a server callback that receives one value holding the tool name, the `$mcp_error_type` +and `$mcp_error_message` the call would carry without a classifier, and the failure itself when the SDK holds it. The failure is the tool's own thrown error, looking past the same wrappers as `$mcp_error_type`, else the `isError` result as the tool produced it, before the SDK appends a conversation handle. The classifier may return an explicit error type, whether the failure is expected, -both, or nothing. The SDK SHALL call it at most once per tool call, for a failed call that ends in a -thrown error or an `isError` result, and for nothing else: never for a failed or cancelled task or an -`input_required` sequence the framework ends, whose types "Tool call event" and "Optional events" fix. -On the manual step path the SDK calls it only when the caller leaves the type or the expected flag -unset, and the caller's values win; the lower-level method never calls it. When the classifier or the -manual API's caller says whether the failure is expected, the SDK SHALL set `$mcp_error_expected` to -that JSON boolean; otherwise it SHALL NOT set it. An expected -failure is still a failed call: it keeps `$mcp_is_error: true` and every other rule of this -requirement. A classifier that fails or returns nothing leaves the call as it would be without one, and -SHALL NOT change the response. +both, or nothing. The SDK SHALL call it exactly once for each failed call that ends in a thrown error or +an `isError` result, and for nothing else: never for a failed or cancelled task or an `input_required` +sequence the framework ends, whose types "Tool call event" and "Optional events" fix. The manual API +never calls it: its caller holds the failure and passes the type and the expected flag itself. When the +classifier or the manual API's caller says whether the failure is expected, the SDK SHALL set +`$mcp_error_expected` to that JSON boolean; otherwise it SHALL NOT set it. An expected failure is still +a failed call: it keeps `$mcp_is_error: true` and every other rule of this requirement. A classifier +that fails or returns nothing leaves the call as it would be without one, and SHALL NOT change the +response. When the SDK emits `$exception`, the event SHALL follow the `exception-event-metadata` capability, with `$exception_source: "mcp.tool_call"`, `mechanism.handled: true` (the framework turns the failure into a @@ -617,7 +618,7 @@ takes the call context and the outcome (the result or the error, an explicit err failure is expected if the caller knows them, and when it ended); it sends the events and returns the result to deliver, with a minted handle appended. From these inputs the SDK applies every other rule of this spec itself: the `$mcp_parameters` shape, intent and model, `$session_id`, person and client -identity, failure capture (including the error classifier), privacy, and size bounds. +identity, failure capture (apart from the error classifier), privacy, and size bounds. A lower-level method MAY also accept a finished call's fields directly, for a server that already holds them; it is exempt from producing the same events as automatic instrumentation. It sends From de7daf8ebe3522eef051c398ac73461ec12171c3 Mon Sep 17 00:00:00 2001 From: Lucas Faria Date: Mon, 9 Nov 2026 15:15:32 -0300 Subject: [PATCH 4/5] docs(mcp-analytics): sync the archived classifier delta with the client-conversation-id wording Keeps the archived manual-API requirement identical to the synced spec after rebasing onto main (client conversation ids, rule 4 for stateless requests). Tested: openspec validate --specs --strict --no-interactive passes. --- .../specs/mcp-analytics/spec.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/openspec/changes/archive/2026-11-05-mcp-error-classifier/specs/mcp-analytics/spec.md b/openspec/changes/archive/2026-11-05-mcp-error-classifier/specs/mcp-analytics/spec.md index 375b5946..d1485da8 100644 --- a/openspec/changes/archive/2026-11-05-mcp-error-classifier/specs/mcp-analytics/spec.md +++ b/openspec/changes/archive/2026-11-05-mcp-error-classifier/specs/mcp-analytics/spec.md @@ -93,10 +93,12 @@ identity, failure capture (apart from the error classifier), privacy, and size b A lower-level method MAY also accept a finished call's fields directly, for a server that already holds them; it is exempt from producing the same events as automatic instrumentation. It sends `$mcp_parameters` and `$mcp_response` as the caller passes them, takes whether a failure is expected -next to its explicit error type, and follows the rest of this spec for everything else. A `$session_id` -the caller passes is sent unchanged as the transport session unless a conversation handle applies; -without either, each call gets a fresh `ses_`, as rule 3 of "Session identity" gives a -stateless request, so a server on a one-client connection SHOULD pass its connection's id. +next to its explicit error type, and follows the rest of this spec for everything else. A client +conversation id applies only when the caller passes the request's `_meta` together with the `User-Agent` +or client name that recognizes its client. A `$session_id` the caller passes is sent unchanged as the +transport session unless a client conversation id or a conversation handle applies; without any of +these, each call gets a fresh `ses_`, as rule 4 of "Session identity" gives a stateless +request, so a server on a one-client connection SHOULD pass its connection's id. #### Scenario: A manually captured call matches the automatic one - **GIVEN** an MCP server that dispatches tool "search_docs" itself through the SDK's manual API From adb27b74437d189b595686e58fa91c48297c7139 Mon Sep 17 00:00:00 2001 From: Lucas Faria Date: Mon, 9 Nov 2026 15:16:27 -0300 Subject: [PATCH 5/5] docs(mcp-analytics): disable conversation anchoring in the classifier scenario Review fix (dustinbyrne F1): the scenario compares the client's result with and without instrumentation, so anchoring's appended handle block would differ for an unrelated reason. Matches the public acceptance scenario; both canonical copies updated. Validate totals unchanged from main. --- .../2026-11-05-mcp-error-classifier/specs/mcp-analytics/spec.md | 2 +- openspec/specs/mcp-analytics/spec.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/openspec/changes/archive/2026-11-05-mcp-error-classifier/specs/mcp-analytics/spec.md b/openspec/changes/archive/2026-11-05-mcp-error-classifier/specs/mcp-analytics/spec.md index d1485da8..85c47f82 100644 --- a/openspec/changes/archive/2026-11-05-mcp-error-classifier/specs/mcp-analytics/spec.md +++ b/openspec/changes/archive/2026-11-05-mcp-error-classifier/specs/mcp-analytics/spec.md @@ -55,7 +55,7 @@ NOT depend on `$exception`, which can be disabled. - **AND** the client should receive the same failure it would receive without instrumentation #### Scenario: The error classifier labels an isError result as an expected failure -- **GIVEN** an MCP server instrumented by the SDK with an error classifier that returns error type "not_found" and expected true, and a tool "get_record" that returns a result with isError true and the text "record not found" +- **GIVEN** an MCP server instrumented by the SDK with conversation anchoring disabled, an error classifier that returns error type "not_found" and expected true, and a tool "get_record" that returns a result with isError true and the text "record not found" - **WHEN** an MCP client calls tool "get_record" - **AND** the SDK is flushed - **THEN** the "$mcp_tool_call" event's property "$mcp_is_error" should be the boolean true diff --git a/openspec/specs/mcp-analytics/spec.md b/openspec/specs/mcp-analytics/spec.md index 3f36d63c..c2db2fb5 100644 --- a/openspec/specs/mcp-analytics/spec.md +++ b/openspec/specs/mcp-analytics/spec.md @@ -283,7 +283,7 @@ NOT depend on `$exception`, which can be disabled. - **AND** the client should receive the same failure it would receive without instrumentation #### Scenario: The error classifier labels an isError result as an expected failure -- **GIVEN** an MCP server instrumented by the SDK with an error classifier that returns error type "not_found" and expected true, and a tool "get_record" that returns a result with isError true and the text "record not found" +- **GIVEN** an MCP server instrumented by the SDK with conversation anchoring disabled, an error classifier that returns error type "not_found" and expected true, and a tool "get_record" that returns a result with isError true and the text "record not found" - **WHEN** an MCP client calls tool "get_record" - **AND** the SDK is flushed - **THEN** the "$mcp_tool_call" event's property "$mcp_is_error" should be the boolean true