Skip to content

feat: stable error catalog with machine-readable error codes - #1148

Merged
sarahxsanders merged 8 commits into
mainfrom
error-catalog
Aug 27, 2026
Merged

sarahxsanders merged 8 commits into
mainfrom
error-catalog

Conversation

@fercgomes

Copy link
Copy Markdown
Contributor

Problem

The wizard runs headless inside cloud-worker sandboxes. When a run dies, the backend and web UI cannot tell why — every failure surfaces as a generic message, because thrown errors are not propagated with any stable classification. Hosts have to parse human-readable text to guess the failure class.

Changes

A stable, append-only error catalog (PHW_<GROUP>_<NAME>) covering every terminal failure path, propagated through three channels:

  • New module src/lib/errors/ — 40 codes (codes.ts), per-code metadata with retry advice (catalog.ts), classifiers (auth 401 → stored-login/settings/key-type/scope/region/invalid; detectError kinds; skill-install failures; AgentErrorType), and the phw-error: stderr line emitter.
  • Propagation — wizardAbort({ code, detail }) resolves the code (explicit param → WizardError.code), stamps OutroData.errorCode/errorDetail, tags analytics with error_code, and — for non-interactive hosts — prints one machine-readable line to stderr just before exit:
    phw-error: {"code":"PHW_DETECT_NO_FRAMEWORK","message":"Could not auto-detect your framework for this project."}
    
    This line is the only channel that works even when the task stream itself is down; sandboxes scrape the last phw-error: line on stderr.
  • Task stream — TaskStreamError now carries code/detail, so the PostHog backend receives the failure class on the terminal RunPhase.Error push.
  • ~30 exit sites stamped — non-interactive validation, env/outage checks, detect prerequisites (ciPreRun + onReady), skill install, the full agent error taxonomy (linear + orchestrator), auth 401 classification, CI install/signup, doctor, skill registry, CLI guards, and fatal catches.

Also closes two headless gaps found while mapping exits:

  1. Unsupported framework version was silently ignored in non-interactive runs — the setUnsupportedVersion stub dropped it; now aborts with PHW_DETECT_UNSUPPORTED_VERSION.
  2. Unfixable Claude settings conflicts continued silently in non-interactive runs (LoggingUI no-ops the override screen); now fail closed with PHW_SETTINGS_UNFIXABLE_CONFLICT.

Incidentally fixes three pre-existing type errors in orchestrator-runner.ts (docsUrl lookup, missing requestRemark task inputs).

Exit codes stay 1/130 — the code rides structured channels, not the exit byte. Full reference: docs/error-catalog.md (catalog table, stability contract, propagation spec, sandbox integration recipe).

Test plan

  • New unit tests enforce catalog completeness, code pattern, and uniqueness (src/lib/errors/__tests__/codes.test.ts — 6 tests).
  • Full suite: pnpm build ✓, 2020/2021 tests pass. The single failure (skill-cli.test.ts yargs locale message) is pre-existing on main — verified by stashing the diff and re-running.
  • tsc --noEmit: zero new errors vs main; net −3.
  • Covered by CI.

Introduces PHW_<GROUP>_<NAME> error codes (src/lib/errors/codes.ts),
per-code metadata with retry advice (catalog.ts), classifiers for auth
401s, detectError kinds, skill-install failures, and AgentErrorType,
plus the machine-readable phw-error stderr line emitter. Unit tests
enforce catalog completeness and code uniqueness.
WizardError carries an optional code; wizardAbort({ code, detail })
stamps OutroData.errorCode/errorDetail, tags analytics with error_code,
and emits the phw-error stderr line when running under LoggingUI/
HeadlessUI. TaskStreamError carries code/detail so the backend sees
the failure class on the terminal error push.
Wires catalog codes at every terminal failure site: non-interactive
validation, local-services and outage checks, detect prerequisites
(both ciPreRun and onReady paths), skill install, the full agent error
taxonomy (linear + orchestrator), auth 401 classification, CI install
signup, doctor, skill registry, CLI guards, and interactive fatal
catches.

Also closes two headless gaps found while mapping exits: an
unsupported framework version now aborts non-interactive runs instead
of being silently dropped, and unfixable Claude settings conflicts now
fail closed instead of continuing into the redirect.

Fixes three pre-existing type errors in orchestrator-runner
(docsUrl lookup, missing requestRemark task inputs).
@fercgomes
fercgomes requested review from a team as code owners August 26, 2026 18:26
@fercgomes
fercgomes requested review from TueHaulund, arnohillen, fasyy612 and ksvat and removed request for a team August 26, 2026 18:26
@github-actions

Copy link
Copy Markdown

🧙 Wizard CI

Run the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands:

Test all apps:

  • /wizard-ci all

Test all apps in a directory:

  • /wizard-ci ai-observability
  • /wizard-ci basic-integration
  • /wizard-ci mcp-analytics
  • /wizard-ci replay-vision
  • /wizard-ci revenue
  • /wizard-ci self-driving

Test an individual app:

  • /wizard-ci ai-observability/anthropic
  • /wizard-ci ai-observability/groq
  • /wizard-ci ai-observability/manual-capture
Show more apps
  • /wizard-ci ai-observability/openai
  • /wizard-ci ai-observability/openai-agents
  • /wizard-ci ai-observability/vercel-ai
  • /wizard-ci basic-integration/android
  • /wizard-ci basic-integration/angular
  • /wizard-ci basic-integration/astro
  • /wizard-ci basic-integration/django
  • /wizard-ci basic-integration/fastapi
  • /wizard-ci basic-integration/flask
  • /wizard-ci basic-integration/flutter
  • /wizard-ci basic-integration/javascript-node
  • /wizard-ci basic-integration/javascript-web
  • /wizard-ci basic-integration/laravel
  • /wizard-ci basic-integration/next-js
  • /wizard-ci basic-integration/nuxt
  • /wizard-ci basic-integration/python
  • /wizard-ci basic-integration/rails
  • /wizard-ci basic-integration/react-native
  • /wizard-ci basic-integration/react-router
  • /wizard-ci basic-integration/sveltekit
  • /wizard-ci basic-integration/swift
  • /wizard-ci basic-integration/tanstack-router
  • /wizard-ci basic-integration/tanstack-start
  • /wizard-ci basic-integration/vue
  • /wizard-ci mcp-analytics/custom-dispatcher
  • /wizard-ci mcp-analytics/typescript-sdk
  • /wizard-ci replay-vision/javascript-node
  • /wizard-ci replay-vision/next-js
  • /wizard-ci replay-vision/react-vite
  • /wizard-ci revenue/stripe
  • /wizard-ci self-driving/astro
  • /wizard-ci self-driving/fastapi
  • /wizard-ci self-driving/nuxt
  • /wizard-ci self-driving/react-router
  • /wizard-ci self-driving/sveltekit

Results will be posted here when complete.

An abort case that matches a known [ABORT] reason now carries its own
errorCode, so dynamic precondition failures classify correctly instead
of all surfacing as PHW_AGENT_ABORT. Wired: audit / web-analytics-doctor
/ source-maps no-SDK cases (PHW_DETECT_NO_POSTHOG_SDK), source-maps bare
React Native + mcp-analytics unsupported language
(PHW_DETECT_UNSUPPORTED_PLATFORM), web-analytics-doctor insufficient
permissions (PHW_AUTH_MISSING_SCOPE), self-driving interactive mode
(PHW_CLI_INTERACTIVE_REQUIRED).

Fixes the headless audit run reporting PHW_AGENT_ABORT for a repo with
no PostHog SDK installed.
@fercgomes
fercgomes requested review from a team as code owners August 26, 2026 20:11
Comment thread src/lib/runners/run-non-interactive.ts Resolved
@veria-ai

veria-ai Bot commented Aug 26, 2026 •

Copy link
Copy Markdown

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

@edwinyjlim edwinyjlim left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit: looks great

Comment thread src/lib/errors/emit.ts Resolved
Comment thread src/lib/errors/emit.ts Resolved
Remote channels (task-stream session row, phw-error stderr line) now
drop detail fields not on the allowlist (reason, detected, platform),
so detect failures no longer ship the absolute install path to
team-visible telemetry. Local surfaces (TUI error screen, debug log)
keep the full detail.

Addresses the local-path-disclosure review comment on PR #1148.
PhwErrorLine.code is now the closed ErrorCode union instead of a bare
string, so every emit site is compile-time checked against the catalog.
The type import in emit.ts is erased at compile time, keeping the
module runtime-dependency-free for the bin.ts Node preflight. bin.ts
and ci-install's failCI now pass catalog constants instead of raw
strings.

Addresses the typing review comment on PR #1148.
emitPhwError -> emitWizardError, formatPhwErrorLine ->
formatWizardErrorLine, PhwErrorLine -> WizardErrorLine, matching the
repo's Wizard* vocabulary. The PHW namespace stays on the wire
contract only (PHW_* codes, phw-error: prefix, PHW_ERROR_PREFIX
constant). No behavior change.

Addresses the naming review comment on PR #1148.
@sarahxsanders
sarahxsanders merged commit 25be176 into main Aug 27, 2026
17 checks passed
@sarahxsanders
sarahxsanders deleted the error-catalog branch August 27, 2026 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants