Skip to content

ai(observability): $ai_model reports the discarded Workers-AI placeholder on every failed call, hiding which model actually broke #10186

Description

@JSONbored

Part of epic #8286 (Phase 3 — AI observability).

Problem

On the failure path, $ai_model reports a model that never ran.

resolveModel (src/selfhost/ai.ts:141) exists precisely because the core passes a Workers-AI model id that is meaningless off-Workers — it discards any @cf/-prefixed value and substitutes the provider-specific self-host model.

runProviderWithOtel then captures the two paths differently:

  • Success (src/selfhost/ai.ts:1506): model: usage?.model ?? (model || "default")usage.model is the real resolved model. Correct.
  • Failure (src/selfhost/ai.ts:1525): model: model || "default" — there is no usage when the call failed, so this reports the raw passed-in value, which is exactly the @cf/ placeholder resolveModel threw away.

Impact, measured

On the live project, every one of the 33 failed AI events is mislabelled:

Reported $ai_model Reported $ai_provider Calls Errors
@cf/openai/gpt-oss-120b claude-code 11 11
default claude-code 12 12
@cf/baai/bge-m3 ai_embed 8 8
visual-vision ai_vision 2 2

Every successful call is correctly labelled (claude-sonnet-5, qwen3:8b, bge-m3:latest, qwen3-vl:8b-instruct). The result is a perfect false signal: a breakdown by model shows a set of Cloudflare model ids with a 100% error rate that this deployment does not use and has never called, while the real failing models — self-hosted ollama and the claude-code/codex CLIs — show a 0% error rate because their failures were attributed elsewhere.

The underlying failures are real and correctly captured in $ai_error: claude_code_error_429 (15), claude_stalled_no_output — no stdout within firstOutputTimeoutMs (8), ai_embed_http_400: the input length exceeds the context length (8).

This is the same class of bug the codebase already fixed once in a different sink — src/services/ai-review.ts:3525: "self-host claude-code reviews were mis-logged as the Workers-AI model ids (@cf/openai/gpt-oss-120b+...), which hid outages."

Deliverables

  • Resolve the model label the same way on both paths, so a failure reports the model that was actually going to run. resolveModel's output is what the provider used; the raw model argument is a request-layer placeholder and should never reach telemetry.
  • Never emit a @cf/-prefixed value as $ai_model from the self-host path — Workers AI has no live binding anywhere (src/services/ai-summaries.ts:32), so any such value in telemetry is by definition wrong.
  • "default" is equally unhelpful as a model label; prefer the provider's own resolved default.
  • Regression test asserting the error path and the success path report the same model for the same configured provider.

Related

The same fallback shape appears in withAiGenerationCapture (src/selfhost/ai.ts:1693), which additionally passes no repo context at all — worth checking in the same pass.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions