Skip to content

Feature request: mcp analytics - don't count expected tool errors as failures #113659

Open

Description

@posthog

Is your feature request related to a problem?

MCP Analytics counts every tool call that throws or returns isError: true as a failure. Some tools return an error on purpose when a request can't be served, for example when the agent asks for a resource that doesn't exist or an option that is unavailable. The tool works as intended in these cases, but the calls still raise the tool's error rate and show up as failures in MCP Analytics. There is no SDK option or product setting to mark a specific error as expected.

Describe the solution you'd like

A way to tell MCP Analytics that a given error is expected, so those calls don't count as tool failures.

Describe alternatives you've considered

  • Return a normal result instead of an error. This changes the result the agent receives.
  • Use the SDK's beforeSend hook to set $mcp_is_error to false, remove $mcp_error_type and $mcp_error_message, and drop the paired $exception event. This works in @posthog/mcp today, but each server must match on its own error text, and the event no longer shows that the call returned an error.

Additional context

From: https://us.posthog.com/project/2/support/tickets/76831

  • Reported on EU Cloud, project 58216, TypeScript SDK (@posthog/mcp).
  • The dashboard failure count reads $mcp_is_error (products/mcp_analytics/backend/hogql_queries/dashboard_series.py).
  • captureToolCall accepts an errorType label, but the SDK only sets $mcp_error_type when isError is true, so the call still counts as a failure.

Debug info

  • PostHog Cloud, Debug information: EU Project 58216

Activity

  1. posthog commented on Oct 7, 2026

    @posthog
    ContributorAuthor

    PostHog self-driving picked up this issue and researched it. Somebody may already be working on it, so ask before you start a fix.

    Report (team access only): https://us.posthog.com/project/2/inbox/reports/01a111af-e754-7ea0-bff0-51f5694f7a51

  2. Src-Bhavesh commented on Oct 8, 2026

    @Src-Bhavesh

    Is this available for a community contribution, or has the team/self-driving work already picked it up? I would first confirm how an expected tool error should be represented while preserving the original error event. Would a focused change to the MCP Analytics failure calculation and its tests be useful, or does this need an SDK contract change first? I would wait for that scope decision before implementing.

  3. mkushner commented on Nov 3, 2026

    @mkushner

    @posthog-bot +1, with a few production numbers for the context.

    We run an MCP server with instrument() (stateless - Cloudflare Workers).
    Over 30 days (Oct 4 – Nov 3): 148,967 tool calls, 2,872 returned isError: true, and all 2,872 have $mcp_error_type = "Error". For an isError result the SDK only sees the envelope, so the type carries no signal either.

    We classify by message prefix in HogQL. By that classifier:

    • 81% (2,318 errors, 140 callers) are expected or the agent's to fix: rate limits (1,410), an ambiguous workspace argument (394), not found (222), validation and schema rejections (250), permission and auth (42).
    • 13% (365, 41 callers) are ours to fix, almost all upstream timeouts.
    • 7% our prefixes don't match.

    So the failure count on the dashboard is about 8× what we can act on.
    A possible shape, as an optional instrument() option:

    classifyError?: (toolName: string, result: CallToolResult) =>
      { errorType?: string; expected?: boolean } | undefined

    errorType fills $mcp_error_type. expected: true sets a property (say $mcp_error_expected) that the dashboard excludes from the failure rate. $mcp_is_error stays true, which keeps the "event no longer shows it was an error" problem from the beforeSend workaround out of it. captureToolCall could take the same expected field next to its existing errorType.

    If that works, happy to open the posthog-js PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions