Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions acceptance/public/mcp-analytics.feature
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,39 @@ Feature: MCP analytics
Then the "$mcp_tool_call" event property "$mcp_error_type" should equal "validation"
And the "$exception" event's first exception type should name the type "UpstreamTimeout"

@server
Scenario: The error classifier labels an isError result as an expected failure
Given an instrumented MCP server with conversation anchoring disabled, an error classifier that returns error type "not_found" and expected true, and a tool "get_record" that returns a result with isError true and the text "record not found"
When the MCP client calls tool "get_record" with JSON arguments:
"""application/json
{"id":"rec_1"}
"""
And pending captures are flushed
Then the "$mcp_tool_call" event property "$mcp_is_error" should equal JSON true
And the "$mcp_tool_call" event property "$mcp_error_type" should equal "not_found"
And the "$mcp_tool_call" event property "$mcp_error_expected" should equal JSON true
And the "$mcp_tool_call" event property "$mcp_error_message" should equal "record not found"
And the MCP client should receive a result with exactly one text block "record not found" and isError true

@server
Scenario: A failed call without a classification has no expected flag
Given an instrumented MCP server with a tool "get_record" that returns a result with isError true and the text "record not found"
When the MCP client calls tool "get_record" with JSON arguments:
"""application/json
{"id":"rec_1"}
"""
And pending captures are flushed
Then the "$mcp_tool_call" event property "$mcp_is_error" should equal JSON true
And the "$mcp_tool_call" event should not have property "$mcp_error_expected"

@server
Scenario: A manually captured failure can be marked expected
Given the SDK's manual MCP capture API records a failed call to tool "query" with error type "rate_limited" and expected true
And pending captures are flushed
Then the "$mcp_tool_call" event property "$mcp_is_error" should equal JSON true
And the "$mcp_tool_call" event property "$mcp_error_type" should equal "rate_limited"
And the "$mcp_tool_call" event property "$mcp_error_expected" should equal JSON true

@server
Scenario: A call to a tool that does not exist is not counted as a call
Given an instrumented MCP server with a tool "search_docs" that returns the text "ok"
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-11-05
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
## Overview

The server owner is the only one who knows which failures are expected. The SDK asks them once per
failed tool call, through a callback, and records the answer as data. It does not decide what is
expected, and it does not change what the client receives.

## Key Decisions

- **A flag, not a new outcome.** An expected failure keeps `$mcp_is_error: true`. Hiding the failure
(the `beforeSend` workaround in the issue, which sets `$mcp_is_error` to `false`) loses the fact that
the client received an error. A separate property keeps both facts, so anyone reading the data can
count failures with or without expected ones.
- **The callback always gets the type and message, and the failure when the SDK holds it.** The type
and message `$mcp_error_type` and `$mcp_error_message` would carry exist for every failed call in
every SDK, so a classifier keyed on them works the same everywhere. The failure itself is richer: a
thrown error's own type is the most reliable thing to classify on (`instanceof RateLimitError`). It
is the tool's own error, past the same wrappers `$mcp_error_type` looks past, because Python's
FastMCP re-raises every tool error as a `ToolError` and a classifier would never match the tool's
type. On frameworks that turn a throw into an `isError` result before the SDK sees it, the result is
all there is, and some manual APIs keep only a type name and message. The result is the one the tool
produced, before the SDK appends a conversation handle, the same rule `$mcp_error_message` follows.
- **One object argument.** Platforms pass these as one value (an object, a struct, a dataclass) so it
can gain fields later without breaking callbacks.
- **Only failures the tool produced.** A failed or cancelled task and an `input_required` sequence the
framework ends are failures the SDK observes and types itself, with no thrown error or `isError`
result to hand over. The classifier is not called for them, so their fixed types stand in every SDK.
- **Automatic instrumentation only.** A manual API caller already holds the failure and passes the
type and the expected flag itself, so a callback adds nothing there. It would also run on different
paths in different SDKs, because not every SDK's manual result step sends events.
- **Explicit types are data like any other.** A classifier can derive its label from the raw failure,
so an explicit type is sanitized as free text, and an empty or blank one counts as no type. Without
that rule, today's SDKs send a blank type in some cases and drop the property in others.
- **Explicit values win.** The classifier's type wins over the type the SDK assigns and the thrown type,
the same precedence explicit types already have.
The classifier's type never renames the `$exception` type, so error tracking still groups by real
types.
- **`$exception` is unchanged.** Whether an expected failure should still emit `$exception` is open
(see below). This change keeps today's behavior, which the host can already change with the
exception autocapture option or the before-send hook.
- **`not_found` joins the vocabulary.** It was the third-largest expected bucket in the reported data
and had no shared label. An ambiguous argument the agent must fix is `validation`.
- **Tool calls only.** Resource reads and listings keep today's rules. Nothing has asked for more.
- **`$mcp_error_expected` is sent only when known.** It is set when the classifier or the caller says
whether the failure is expected, and absent otherwise. Absent reads as not expected.

## Alternatives Considered

- **Readers classify by message text.** Possible today, and what the reporting team does. Every reader
has to repeat it for every server, and message text is sanitized and truncated before capture.
- **The host marks expected failures in the before-send hook.** Works without a spec change, but the
hook sees only the sanitized message, never the thrown error or the result, and it runs once per
payload, so the host classifies the `$exception` a second time.
- **Mark the failure inside the tool result (a `_meta` key) or on the thrown error.** The result is
what the client receives, so analytics metadata would leak to it, and on frameworks that flatten a
throw into a result the error's fields are lost.
- **Three-way fault attribution (agent, server, upstream).** More informative, but no reader has asked
for it, and a boolean covers the reported need.

## Open Questions

- Should an expected failure skip the `$exception` event? Expected failures are not defects, and each
one creates an error-tracking event the host pays for. Skipping it is a behavior change for every
host that sets a classifier, so it is left out of this change.
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
## Why

Some tools fail on purpose. A rate limit, a record that does not exist, or arguments the agent must fix
are outcomes the server returns by design, and the agent is meant to act on them. Today the data cannot
tell them apart from failures the server owner has to fix. An `isError` result from automatic
instrumentation always has `$mcp_error_type: "Error"`, because the spec gives automatic instrumentation
no way to receive an explicit type. One production server reported 148,967 calls in 30 days, with
2,872 failures, all typed `"Error"`. By the owner's own classification, 81% of those failures were
expected (PostHog/posthog#113659).

## What Changes

- Add an error classifier option to the Configuration table. It is a server callback that, for a failed
tool call, may return an explicit error type and whether the failure is expected.
- Add `$mcp_error_expected`, a JSON boolean on failed calls, to the Property reference.
- Modify **Failure capture**. Automatic instrumentation takes an explicit type from the classifier.
An expected failure is still a failed call: `$mcp_is_error` stays `true`, and `$exception` is
unchanged. Add `not_found` to the shared type vocabulary.
- Modify **Manual capture**. The result step and the lower-level method take whether the failure is
expected, next to the explicit error type. The manual API never calls the classifier.
- An empty or blank explicit type is no type, and every explicit type is sanitized as free text.
- Add scenarios for a classified `isError` result and a manually captured expected failure.

## Capabilities

### New Capabilities

_None._

### Modified Capabilities

- `mcp-analytics`: the **Failure capture** and **Manual capture** requirements, the Configuration
table, and the Property reference.

## Impact

Nothing changes for a host that sets no classifier and passes no expected flag. A host that sets one
gets explicit types on failures automatic instrumentation captures, and `$mcp_error_expected` on the
failures it marks. Every SDK that implements MCP analytics adds the option and the manual field.
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
## MODIFIED Requirements

### Requirement: Failure capture

A failed call (an `isError` result or a thrown error) SHALL set `$mcp_is_error: true` and:

- `$mcp_error_type`: a low-cardinality label, never containing the message. An explicit type wins: one
the caller passes to the manual API, one the error classifier returns, or one the SDK assigns (such as
`validation` for a framework validation error it recognizes), in that order; explicit types SHOULD
come from the shared vocabulary `validation`, `permission`, `missing_context`, `not_found`, `timeout`,
`rate_limited`, `api_4xx`, `api_5xx`, `cancelled`, `input_required`, and `internal`, so failures group
the same way across servers. An empty or blank type from any source is no type, so the next source
applies, and an explicit type is sanitized as free text per "Payload privacy" like any other captured
string. Otherwise it is the thrown error's own type name (in JavaScript, its `name` unless that is the
generic `"Error"`, then its constructor's name), looking past wrappers the SDK, standard library, or
MCP framework add; otherwise `"Error"`, which is also the type of an `isError` result without an
explicit type.
- `$mcp_error_message`: the thrown message or the text of the tool's own `isError` content (never
the SDK's appended handle), sanitized per "Payload privacy".

The SDK MAY set `$mcp_error_status` to an upstream HTTP status code.

A failure can be expected: an outcome the server returns by design for the agent to act on, such as a
rate limit, a record that does not exist, or arguments the agent must fix. Only the server knows which
failures are expected, so automatic instrumentation SHALL offer the error classifier option (see
"Configuration"): a server callback that receives one value holding the tool name, the `$mcp_error_type`
and `$mcp_error_message` the call would carry without a classifier, and the failure itself when the SDK
holds it. The failure is the tool's own thrown error, looking past the same wrappers as
`$mcp_error_type`, else the `isError` result as the tool produced it, before the SDK appends a
conversation handle. The classifier may return an explicit error type, whether the failure is expected,
both, or nothing. The SDK SHALL call it exactly once for each failed call that ends in a thrown error or
an `isError` result, and for nothing else: never for a failed or cancelled task or an `input_required`
sequence the framework ends, whose types "Tool call event" and "Optional events" fix. The manual API
never calls it: its caller holds the failure and passes the type and the expected flag itself. When the
classifier or the manual API's caller says whether the failure is expected, the SDK SHALL set
`$mcp_error_expected` to that JSON boolean; otherwise it SHALL NOT set it. An expected failure is still
a failed call: it keeps `$mcp_is_error: true` and every other rule of this requirement. A classifier
that fails or returns nothing leaves the call as it would be without one, and SHALL NOT change the
response.

When the SDK emits `$exception`, the event SHALL follow the `exception-event-metadata` capability, with
`$exception_source: "mcp.tool_call"`, `mechanism.handled: true` (the framework turns the failure into a
response to the client), and `mechanism.synthetic: true` for an `isError` result that carried no thrown
error; the exception's type SHALL be the thrown error's own type (`"Error"` for an `isError` result),
never the explicit label, so error tracking groups by real types; and the event SHALL carry
`$exception_level: "error"` and the call's `$session_id` and `$mcp_tool_name`. Failure analytics SHALL
NOT depend on `$exception`, which can be disabled.

#### Scenario: A thrown error is a failed call and still reaches the client
- **GIVEN** an MCP server instrumented by the SDK with a tool "query" whose handler throws an error of a custom type named "UpstreamTimeout" with message "upstream timed out"
- **WHEN** an MCP client calls tool "query"
- **AND** the SDK is flushed
- **THEN** the "$mcp_tool_call" event's property "$mcp_is_error" should be the boolean true
- **AND** its property "$mcp_error_type" should name the "UpstreamTimeout" type (platform spelling, such as "UpstreamTimeout" or "errors.UpstreamTimeout")
- **AND** the client should receive the same failure it would receive without instrumentation

#### Scenario: The error classifier labels an isError result as an expected failure
- **GIVEN** an MCP server instrumented by the SDK with conversation anchoring disabled, an error classifier that returns error type "not_found" and expected true, and a tool "get_record" that returns a result with isError true and the text "record not found"
- **WHEN** an MCP client calls tool "get_record"
- **AND** the SDK is flushed
- **THEN** the "$mcp_tool_call" event's property "$mcp_is_error" should be the boolean true
- **AND** its property "$mcp_error_type" should be "not_found"
- **AND** its property "$mcp_error_expected" should be the boolean true
- **AND** its property "$mcp_error_message" should be "record not found"
- **AND** the client should receive the same result it would receive without instrumentation

#### Scenario: A failed call without a classification has no expected flag
- **GIVEN** an MCP server instrumented by the SDK with no error classifier and a tool "get_record" that returns a result with isError true and the text "record not found"
- **WHEN** an MCP client calls tool "get_record"
- **AND** the SDK is flushed
- **THEN** the "$mcp_tool_call" event should not have property "$mcp_error_expected"

### Requirement: Manual capture

An SDK SHALL offer a manual API for servers it cannot instrument automatically, such as a custom
dispatcher, a gateway, or a framework without an adapter. Given the same traffic, its step-based path
SHALL produce the same `$mcp_tool_call` and `$exception` events as automatic instrumentation; for the
events of "Optional events" it MAY offer capture methods. It SHALL work in three steps. A list step
takes the server's tools and returns them with the injected arguments advertised. A call step takes the
tool's registered definition (input schema, description, `_meta`) when the tool exists, the arguments
as the agent sent them, the request's `_meta`, the HTTP headers when there are any, the transport
session when there is one, whether the tool exists (known to exist, known not to, or unknown for a
server with no registry), and the time the request started; it returns the arguments for the handler
and a call context, and for a tool known not to exist it emits `$mcp_unknown_tool` where the SDK does.
The transport session is either a raw id, derived per rule 2 of "Session identity", or a key the caller
keeps for the lifetime of a one-client connection, which the SDK tracks and rotates. A result step
takes the call context and the outcome (the result or the error, an explicit error type and whether the
failure is expected if the caller knows them, and when it ended); it sends the events and returns the
result to deliver, with a minted handle appended. From these inputs the SDK applies every other rule of
this spec itself: the `$mcp_parameters` shape, intent and model, `$session_id`, person and client
identity, failure capture (apart from the error classifier), privacy, and size bounds.

A lower-level method MAY also accept a finished call's fields directly, for a server that already holds
them; it is exempt from producing the same events as automatic instrumentation. It sends
`$mcp_parameters` and `$mcp_response` as the caller passes them, takes whether a failure is expected
next to its explicit error type, and follows the rest of this spec for everything else. A client
conversation id applies only when the caller passes the request's `_meta` together with the `User-Agent`
or client name that recognizes its client. A `$session_id` the caller passes is sent unchanged as the
transport session unless a client conversation id or a conversation handle applies; without any of
these, each call gets a fresh `ses_<UUIDv7>`, as rule 4 of "Session identity" gives a stateless
request, so a server on a one-client connection SHOULD pass its connection's id.

#### Scenario: A manually captured call matches the automatic one
- **GIVEN** an MCP server that dispatches tool "search_docs" itself through the SDK's manual API
- **WHEN** an MCP client calls tool "search_docs" with arguments `{"query": "flags", "context": "find the flags docs"}`
- **AND** the SDK is flushed
- **THEN** the "$mcp_tool_call" event's properties "$mcp_tool_name", "$mcp_intent", "$mcp_parameters", and "$mcp_is_error" should equal those automatic instrumentation sends for the same call
- **AND** its property "$session_id" should be a non-empty string

#### Scenario: A manually captured failure can be marked expected
- **GIVEN** an MCP server that records a failed call to tool "query" through the SDK's manual API with error type "rate_limited" and expected true
- **WHEN** the SDK is flushed
- **THEN** the "$mcp_tool_call" event's property "$mcp_is_error" should be the boolean true
- **AND** its property "$mcp_error_type" should be "rate_limited"
- **AND** its property "$mcp_error_expected" should be the boolean true
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
## 1. Spec delta

- [x] 1.1 Modify the **Failure capture** requirement as a full copied-and-edited block
- [x] 1.2 Add the classifier, `$mcp_error_expected`, and `not_found` to it
- [x] 1.3 Modify the **Manual capture** requirement so the result step and the lower-level method take
the expected flag
- [x] 1.4 Add scenarios for a classified `isError` result and a manual expected failure

## 2. Prose alignment (applied at archive, outside the requirement-delta mechanism)

- [x] 2.1 Add the error classifier row to the Configuration table
- [x] 2.2 Add `$mcp_error_expected` to the Property reference
- [x] 2.3 Add both scenarios to `acceptance/public/mcp-analytics.feature`

## 3. Validation

- [x] 3.1 Run `openspec validate --specs --strict`
- [x] 3.2 Run `openspec archive` to sync the delta and archive the change

## 4. Downstream follow-ups (separate changes, not this one)

- [ ] 4.1 posthog-sdk-test-harness: step bindings for the new scenarios
- [ ] 4.2 Implement the option and the manual field in every SDK that implements MCP analytics
Loading
Loading