Skip to content

fix(host-service): classify an unreachable cloud API as 503, not a 500 - #6735

Merged
saddlepaddle merged 1 commit into
mainfrom
sentry/host-info-cloud-unreachable
Aug 21, 2026
Merged

saddlepaddle merged 1 commit into
mainfrom
sentry/host-info-cloud-unreachable

Conversation

@saddlepaddle

@saddlepaddle saddlepaddle commented Aug 21, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

host.info is a small procedure the desktop polls: it reports which host this is, and which organization it belongs to. Resolving the organization means an outbound tRPC call to our cloud API, and that call's transport failures were unhandled. Any network problem between the user's machine and us escaped getOrganization as a raw TRPCClientError: fetch failed, became an INTERNAL_SERVER_ERROR, and — under the #6164 contract, where a 500 means "we have a bug" — got reported to Sentry as a host-service defect.

The captured causes are unambiguously the user's network, not our software:

  • HOST-SERVICE-1B (51 events, escalating) — Error: read ETIMEDOUT on an established socket.
  • HOST-SERVICE-43 — undici HeadersTimeoutError.
  • HOST-SERVICE-46 — a captive wifi portal terminated TLS and presented its own certificate: Hostname/IP does not match certificate's altnames, for a hostname that is not ours.

Why fix rather than archive again. Two earlier issues in this family were archived on 2026-08-19. They came back, because the Sentry grouping key follows the innermost transport error — so each new flavour of network failure from this one throw site opens a new issue. Archiving does not converge; it is unbounded manual triage. And the classification is wrong on the merits: someone's captive portal is not an internal server error.

Reach. The failing builds are 1.22.0 / 1.23.0 / 1.24.1; the package is at 1.24.1, so this ships in the next host-service release and reaches machines as the desktop app auto-updates and respawns its bundled host-service. The existing issues will keep firing from older builds until the fleet rolls forward.

Root cause

getOrganization awaited api.organization.getByIdFromJwt.query(...) with no adapter at the boundary. The one-hour success cache means this only fires when the call actually goes out.

Fix

One adapter, cloud-api-error.ts, applied at that single call site.

The producer is a tRPC client over Node's HTTP stack — we do not own it, so the error has to be reconstructed rather than recognised. The most structural signal available is the machine-readable code Node and undici attach to exactly these failures, which sits three levels down the cause chain: TRPCClientError("fetch failed") → TypeError("fetch failed") → the coded error. The adapter walks that chain structurally (duck-typed, never instanceof — the links are not ours) and matches code against a closed set. No message regexes.

Matched: ECONNREFUSED, ECONNRESET, ETIMEDOUT, ENOTFOUND, EAI_AGAIN, EHOSTUNREACH, ENETUNREACH, UND_ERR_CONNECT_TIMEOUT, UND_ERR_HEADERS_TIMEOUT, and the certificate-validation codes a TLS-intercepting portal or proxy produces: ERR_TLS_CERT_ALTNAME_INVALID, UNABLE_TO_VERIFY_LEAF_SIGNATURE, UNABLE_TO_GET_ISSUER_CERT, UNABLE_TO_GET_ISSUER_CERT_LOCALLY, SELF_SIGNED_CERT_IN_CHAIN, DEPTH_ZERO_SELF_SIGNED_CERT.

A match becomes SERVICE_UNAVAILABLE — already this package's idiom for "the thing you need to reach is not reachable" — carrying the underlying message, which is the informative one (fetch failed is not). Behaviour is otherwise untouched: same cache, same PRECONDITION_FAILED when the organization is genuinely missing, no retries or backoff added.

What deliberately still reports as a 500

  • HOST-SERVICE-41 — an HTML page where JSON was expected. It surfaces as a SyntaxError with no transport code attached, so the adapter does not match it and it stays a 500. This is intentional: at this layer a non-JSON response body is indistinguishable from our own cloud genuinely returning something broken, and silencing it would hide a real server-side bug. It is being archived separately.
  • Application errors the cloud API returned (auth failures, not-found). These arrive as a TRPCClientError built from an error response, with no transport code anywhere in the chain — they keep reporting.
  • Any failure with no cause chain at all.
  • CERT_HAS_EXPIRED / CERT_NOT_YET_VALID — excluded on purpose, and the one judgement call here. Interception announces itself as an untrusted issuer or the wrong hostname; an expired certificate is at least as likely to be ours, and if our certificate lapses this procedure should be one of the things that screams.

Worth naming the residual over-match: ECONNRESET / ECONNREFUSED could in principle be our own edge misbehaving rather than the user's network. From the host's vantage point that still is "could not reach the cloud", and cloud-side monitoring owns that failure — it is not something a desktop host-service should be diagnosing.

No typed cause

Thrown as a plain new TRPCError({ code, message }) with no cause: { kind }. Nothing reads one: the only desktop callers of host.info are useRemoteHostStatus, which branches on isSuccess alone and falls through to ready otherwise, and the tray menu, which does catch { return null }. A kind nobody reads is code maintained for nothing; it can be added when a renderer branch wants it.

Scope

Deliberately confined to this one procedure's outbound call. Other host-service procedures also call the cloud API — workspaces.ts:456 (task.byId), workspaces.ts:501 (v2Workspace.trackCreated), project.ts:453 (v2Project.findByGitHubRemote), plus fire-and-forget calls in workspace.ts, notifications.ts, and local-workspace-store.ts that already swallow their own rejections. They are left alone; each deserves its own look at what its caller should see. Classifying inside createApiClient's link chain would have covered them all in one move, and was rejected for exactly that reason — it would silently reclassify call sites nobody has reviewed.

Verification

The negative tests are the point of this change, so both directions were checked against a strawman first.

Strawman 1 — message regex on fetch failed (the last-resort approach this task rules out). Two negatives fail, and every positive fails too, because the client error's message is fetch failed and the real message is unreachable from there:

(fail) rethrowCloudUnreachable > read timeout on an established socket (HOST-SERVICE-1B)
(fail) rethrowCloudUnreachable > undici headers timeout (HOST-SERVICE-43)
(fail) rethrowCloudUnreachable > captive portal presents its own certificate (HOST-SERVICE-46)
... 12 more positives ...
(fail) rethrowCloudUnreachable > an error with no cause chain still reports
(fail) rethrowCloudUnreachable > an expired certificate still reports
 4 pass
 17 fail

Strawman 2 — blanket "anything this call throws is unreachable" (the tempting wrong fix). All five negatives fail:

(fail) rethrowCloudUnreachable > an HTML body where JSON was expected still reports (HOST-SERVICE-41)
(fail) rethrowCloudUnreachable > an application error from the cloud API still reports
(fail) rethrowCloudUnreachable > a not-found from the cloud API still reports
(fail) rethrowCloudUnreachable > an error with no cause chain still reports
(fail) rethrowCloudUnreachable > an expired certificate still reports
(fail) rethrowCloudUnreachable > our own typed error passes through untouched
 4 pass
 17 fail

With the shipped adapter — fixtures built with the real TRPCClientError and real captured shapes, not mocks:

$ bun test src/trpc/router/host
bun test v1.3.14 (0d9b296a)

 21 pass
 0 fail
 37 expect() calls
Ran 21 tests across 1 file. [34.00ms]
$ cd packages/host-service && bun run typecheck
$ tsc --noEmit --emitDeclarationOnly false
$ bunx biome check packages/host-service/src/trpc/router/host/host.ts \
    packages/host-service/src/trpc/router/host/cloud-api-error.ts \
    packages/host-service/src/trpc/router/host/cloud-api-error.test.ts
Checked 3 files in 5ms. No fixes applied.

(--write was run once for formatting, then re-checked clean as above. Repo-wide bun run lint was skipped per the known cross-worktree bun-lock hang.)

Refs HOST-SERVICE-1B
Refs HOST-SERVICE-46
Refs HOST-SERVICE-43

https://claude.ai/code/session_01MLQT5zzzaRZgrnU4Tbz27Y


Summary by cubic

Classifies cloud API transport failures in host.info as 503 Service Unavailable instead of 500 Internal Server Error, so Sentry stops flagging user network issues as host-service bugs. Previously, a TRPCClientError: fetch failed bubbled up as a 500.

  • Adds a small adapter that walks the TRPCClientError → TypeError → coded error cause chain and maps Node/undici transport codes (timeouts, DNS, connection resets/refusals, and untrusted TLS issuer/hostname) to SERVICE_UNAVAILABLE, preserving the underlying message; no regex.
  • Applies the adapter only at the host.info organization lookup call site; other host-service cloud calls are unchanged by design.
  • Keeps 500s for non-JSON responses, cloud application errors, unknown failures with no code, and expired/not-yet-valid certificates.
  • Behavior otherwise unchanged: same cache, no retries/backoff, no API surface changes. Includes focused tests for the adapter.
  • Rollout: ships with the next host-service release bundled in the desktop app; older builds will continue emitting 500s until auto-update advances.

Written for commit 57f7e4c. Summary will update on new commits.

Review in cubic

Summary by CodeRabbit

  • Bug Fixes
    • Improved handling of cloud connectivity failures during organization lookup.
    • Network, DNS, timeout, connection, and certificate trust errors now report a clear service-unavailable message.
    • Application errors, malformed responses, and unrelated certificate errors continue to be reported normally.
  • Tests
    • Added coverage for supported unreachable-cloud scenarios and errors that should remain unchanged.

host.info calls the cloud API to resolve the organization. Transport
failures from that call were unhandled, so a captive portal, a dropped
connection, or broken DNS on the user's machine surfaced as
INTERNAL_SERVER_ERROR from a procedure the desktop polls, and Sentry
reported the user's network as a host-service bug.

Translate at the boundary: walk the tRPC client error's cause chain for
Node/undici transport codes and rethrow as SERVICE_UNAVAILABLE, keeping
the underlying message. Anything without a matching code -- an
application error the cloud returned, a response body we could not
parse -- keeps reporting as a 500.

Refs HOST-SERVICE-1B
Refs HOST-SERVICE-46
Refs HOST-SERVICE-43

Claude-Session: https://claude.ai/code/session_01MLQT5zzzaRZgrnU4Tbz27Y
@coderabbitai

coderabbitai Bot commented Aug 21, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 6419c13a-1e08-4ae4-9a69-964b21178a69

📥 Commits

Reviewing files that changed from the base of the PR and between 2a48a49 and 57f7e4c.

📒 Files selected for processing (3)
  • packages/host-service/src/trpc/router/host/cloud-api-error.test.ts
  • packages/host-service/src/trpc/router/host/cloud-api-error.ts
  • packages/host-service/src/trpc/router/host/host.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review.


📝 Walkthrough

Walkthrough

The host service now detects selected cloud transport failures, converts them to SERVICE_UNAVAILABLE, and applies this handling to organization lookup. Tests cover recognized network and TLS failures, application errors, parse errors, and pre-typed tRPC errors.

Changes

Cloud unreachable handling

Layer / File(s) Summary
Transport error classification and validation
packages/host-service/src/trpc/router/host/cloud-api-error.ts, packages/host-service/src/trpc/router/host/cloud-api-error.test.ts
rethrowCloudUnreachable traverses nested causes and maps known network, timeout, DNS, connection, and TLS trust failures to SERVICE_UNAVAILABLE. Tests verify matching and non-matching errors.
Organization lookup integration
packages/host-service/src/trpc/router/host/host.ts
The organization lookup catches errors, invokes rethrowCloudUnreachable, and rethrows the resulting error.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 57f7e

This localized change classifies specific cloud transport failures as 503 while preserving existing behavior for application and unknown errors; no actionable merge-blocking risk remains after normal checks and review.

Sequence Diagram(s)

sequenceDiagram
  participant hostRouter
  participant getOrganization
  participant cloudApi
  participant rethrowCloudUnreachable
  hostRouter->>getOrganization: request organization
  getOrganization->>cloudApi: getByIdFromJwt.query
  cloudApi-->>getOrganization: organization or error
  getOrganization->>rethrowCloudUnreachable: classify caught error
  rethrowCloudUnreachable-->>getOrganization: SERVICE_UNAVAILABLE or unchanged error
  getOrganization-->>hostRouter: organization or rethrown error
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 6 functions across 3 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title uses conventional commit format and clearly describes classifying unreachable cloud API failures as 503 instead of 500.
Description check ✅ Passed The description clearly explains the problem, fix, scope, and verification, although it does not use the template headings or include the checklist.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch sentry/host-info-cloud-unreachable

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 21, 2026 •

Copy link
Copy Markdown
Contributor

🧹 Preview Cleanup Complete

The following preview resources have been cleaned up:

  • ✅ Neon database branch

Thank you for your contribution! 🎉

@saddlepaddle
saddlepaddle merged commit fbb93e9 into main Aug 21, 2026
20 checks passed
@saddlepaddle
saddlepaddle deleted the sentry/host-info-cloud-unreachable branch August 22, 2026 22:26
saddlepaddle added a commit that referenced this pull request Sep 20, 2026
…being unreachable (#7680)

## Problem

`host.info` calls the cloud API to resolve the organization. When that request dies in transport, the host-service already knows not to report it: #6735 added `rethrowCloudUnreachable`, which walks the cause chain for a transport code and rethrows as `SERVICE_UNAVAILABLE` so the tRPC Sentry middleware leaves it alone.

One transport code was missing from that set. A machine whose socket to the API was torn down mid-request surfaced as a fresh host-service `INTERNAL_SERVER_ERROR` issue, i.e. reported as our bug.

## Root cause

The failure arrives as the usual three-link chain — `TRPCClientError("fetch failed")` → `TypeError("fetch failed")` → a Node socket error carrying `code: "ENOTCONN"`. That is the identical shape to the `read ECONNRESET` case the classifier already covers (and which its tests already name), but `ENOTCONN` was never in `UNREACHABLE_CODES`, so `findUnreachableCause` returned `null` and the error fell through to the middleware as a 500.

The middleware's contract is explicit: anything still `INTERNAL_SERVER_ERROR` at that boundary is a bug, and the remedy is to fix the missing translation at the throw site rather than filter at the middleware. This is that missing translation.

## Fix

One entry added to the existing code set, plus the matching row in the existing table-driven test. Nothing else: only the code this issue actually produced is added. Sibling codes that have never been observed here (`EPIPE`, `ECONNABORTED`, …) are deliberately left out — a classifier that matches more than the condition it names would silence real failures, which is why the set is grown from observed evidence one code at a time.

`ENOTCONN` on a read means the socket was not connected when the read happened: the peer, a proxy, or the network between the user and the API dropped it. It cannot describe a defect in host-service's own call, and the only errors that reach this classifier are the ones thrown by that single cloud API call.

No behaviour change for any consumer: the desktop's `readHostServiceInfo` already catches a failing `host.info` and omits the optional fields, and `isHostServiceConnectionError` already treats `SERVICE_UNAVAILABLE` as retryable/connection-level.

## Verification

Test row added first, and confirmed failing against unchanged source:

```
(fail) rethrowCloudUnreachable > ENOTCONN is the cloud being unreachable [0.09ms]
Expected: "SERVICE_UNAVAILABLE"
Received: undefined
 21 pass
 1 fail
```

After the one-line source change:

```
$ bun test src/trpc/router/host/
 22 pass
 0 fail
 39 expect() calls
Ran 22 tests across 1 file. [23.00ms]
```

```
$ bunx biome check packages/host-service/src/trpc/router/host/cloud-api-error.ts packages/host-service/src/trpc/router/host/cloud-api-error.test.ts
Checked 2 files in 15ms. No fixes applied.

$ cd packages/host-service && bun run typecheck
$ tsc --noEmit --emitDeclarationOnly false
(clean)
```

No user-facing strings touched, so `check:i18n` does not apply.

Refs HOST-SERVICE-6S

This branch was previously deployed

1 inactive deployment
preview — 57f7e4c3 Deployed Aug 21, 2026 by saddlepaddle via Deploy API #8234
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant