fix: retry backend connection failures and improve error logs

.changeset/transport-failure-not-user-error.md

@@ -0,0 +1,6 @@
+---
+'@workflow/core': patch
+'@workflow/world-vercel': patch
+---
+
+Route unrecognized backend connection and stream failures through existing retry policies, preserving event-write retries and rebuilding shared event connections after repeated HTTP/2 failures. Keep invalid request headers and caller cancellations out of event-write retries. Include error cause chains in run-failure logs to expose underlying socket, DNS, and TLS failures.

docs/content/docs/v5/foundations/errors-and-retries.mdx

@@ -164,6 +164,16 @@ export async function myWorkflow(input: unknown) {
 
 Uncaught, the run fails immediately with the `USER_ERROR` code, without retrying. See [serialization-failed](/docs/errors/serialization-failed) for common causes and fixes.
 
+## Backend connection failures
+
+On Vercel, backend connection failures and interrupted event streams use the existing retry policies, even when their error codes are unrecognized. This lets workflows recover from network failures instead of immediately failing with `USER_ERROR`. Persistent failures can still exhaust the retry budget.
+
+The SDK also replaces its shared events connection pool after repeated HTTP/2 session failures. Invalid backend URLs, including unsupported protocols and embedded credentials, fail immediately. Fetch requests to blocked ports or with unsupported headers (such as `Expect`) also fail without retrying. Interrupted event writes retain their existing retries; caller cancellations do not trigger another write.
+
+A connection failure does not prove that the backend rejected a write: it may have accepted it before the response was lost. Continue to make step side effects [idempotent](/docs/foundations/idempotency).
+
+Failed-run logs include the underlying error causes and their codes, exposing socket, DNS, or TLS errors behind messages such as `TypeError: fetch failed`. If a cause cannot be read, the log includes `[unavailable cause]` and the run can still be recorded as failed.
+
 ## Error codes
 
 When a workflow run fails, the error includes an `errorCode` that classifies the failure, alongside the original thrown value (preserved as `cause`):

packages/core/README.md

@@ -2,6 +2,8 @@
 
 Core runtime package for [Workflow SDK](https://workflow-sdk.dev).
 
+Failed-run logs include underlying error causes and codes to help diagnose failures such as socket, DNS, and TLS errors. Unreadable causes are marked without preventing the run from being recorded as failed.
+
 Queued step messages carry immutable run identity in `runContext`, so step
 execution skips the initial `runs.get` and fetches the run row only when
 continuing into workflow replay. Messages from older producers without

packages/core/src/log-format.test.ts

@@ -142,6 +142,48 @@ describe('composeLogLine', () => {
     `);
   });
 
+  test('renders the errorCause chain under the message it explains', () => {
+    // The failure this exists for: undici's wrapper has an empty message and
+    // a stack of frames the trimmer drops, so without the chain the log says
+    // "fetch failed" and nothing about what failed.
+    const out = composeLogLine(PREFIX, 'Error while running workflow', {
+      workflowRunId: 'wrun_01ABC',
+      errorCode: 'WORLD_CONTRACT_ERROR',
+      errorName: 'WorkflowWorldError',
+      errorMessage:
+        'POST /v4/runs/wrun_01ABC/events/step_completed transport failure after 2103ms (UND_ERR_SOCKET)',
+      errorCause: [
+        'TypeError: fetch failed',
+        'SocketError: other side closed (UND_ERR_SOCKET)',
+      ].join('\n'),
+      errorStack: [
+        'WorkflowWorldError: POST /v4/runs/wrun_01ABC/events/step_completed transport failure after 2103ms (UND_ERR_SOCKET)',
+        '    at makeRequest (../../packages/world-vercel/src/utils.ts:470:19)',
+      ].join('\n'),
+    });
+    expect(out).toMatchInlineSnapshot(`
+      "[workflow-sdk] Error while running workflow
+        run    wrun_01ABC
+        code   WORLD_CONTRACT_ERROR
+        cause  TypeError: fetch failed
+               SocketError: other side closed (UND_ERR_SOCKET)
+      WorkflowWorldError: POST /v4/runs/wrun_01ABC/events/step_completed transport failure after 2103ms (UND_ERR_SOCKET)
+          at makeRequest (../../packages/world-vercel/src/utils.ts:470:19)"
+    `);
+  });
+
+  test('omits the cause row when the stack body already spells it out', () => {
+    const out = composeLogLine(PREFIX, 'Error while running workflow', {
+      errorCause: 'SocketError: other side closed',
+      errorStack: [
+        'TypeError: fetch failed',
+        '    at node:internal/deps/undici/undici:13502:13',
+        '  [cause]: SocketError: other side closed',
+      ].join('\n'),
+    });
+    expect(out).not.toMatch(/^\s+cause\s+/m);
+  });
+
   test('does not duplicate a stack the message already embeds', () => {
     const stack = [
       'Error: boom',

packages/core/src/log-format.ts

@@ -15,6 +15,7 @@ import {
  *       user error · FatalError
  *       run    wrun_…
  *       step   step_… · add (./workflows/x)
+ *       cause  TypeError: fetch failed
  *       hint:  Move the call to a step function.
  *     FatalError: …
  *         at … (trimmed stack, internals collapsed)
@@ -90,6 +91,7 @@ function renderStructuredFields(
     'errorName',
     'errorMessage',
     'errorStack',
+    'errorCause',
     'hint',
     'attempt',
     'retryCount',
@@ -159,6 +161,15 @@ function renderStructuredFields(
     lines.push(`  ${kvKey('error')} ${formatPassthroughValue(errorMessage)}`);
   }
 
+  // What the error was wrapping. Sits directly under the message it explains,
+  // one link per line (see `formatErrorCauseChain`). Skipped when the stack
+  // body already spells the chain out, which happens when the stack came off
+  // an already-inspected error and carries `[cause]:` lines of its own.
+  const errorCause = pickString(metadata, 'errorCause');
+  if (errorCause && !body.includes(errorCause)) {
+    lines.push(`  ${kvKey('cause')} ${formatPassthroughValue(errorCause)}`);
+  }
+
   const hint = pickString(metadata, 'hint');
   if (hint) {
     lines.push(`  ${Ansi.hint(hint)}`);

packages/core/src/runtime.test.ts

@@ -3759,4 +3759,73 @@ describe('workflowEntrypoint run-failure logging', () => {
     expect(runFailureLog).toContain('user workflow blew up');
     expect(runFailureLog).toMatch(/^\s+at /m);
   });
+
+  it('records run_failed when a cause code getter throws', async () => {
+    const createdEvents: any[] = [];
+
+    await runWorkflowHandlerWithEvents(
+      `async function workflow() {
+        const cause = new Error('inner');
+        Object.defineProperty(cause, 'code', {
+          get() { throw new Error('code getter failed'); }
+        });
+        throw new Error('original failure', { cause });
+      };globalThis.__private_workflows = new Map();
+      globalThis.__private_workflows.set("workflow", workflow);`,
+      await failingRun(),
+      [],
+      { createdEvents }
+    );
+
+    expect(createdEvents).toContainEqual(
+      expect.objectContaining({
+        eventType: 'run_failed',
+        eventData: expect.objectContaining({
+          errorCode: RUN_ERROR_CODES.USER_ERROR,
+        }),
+      })
+    );
+    const runFailureLog = errorSpy.mock.calls
+      .map((call) => String(call[0]))
+      .find((line) => line.includes('Error while running workflow'));
+    expect(runFailureLog).toContain('original failure');
+    expect(runFailureLog).toContain('[unavailable cause]');
+  });
+
+  /**
+   * Message and stack are not enough when the terminal error is a wrapper.
+   * A world-layer transport failure names the request it was making, and the
+   * `TypeError: fetch failed` under it names nothing at all — the socket/DNS
+   * failure that actually happened is only reachable through `cause`, which
+   * nothing in the log pipeline used to read.
+   */
+  it('emits the cause chain of a wrapped terminal error', async () => {
+    const workflowRun = await failingRun();
+
+    await runWorkflowHandlerWithEvents(
+      `async function workflow() {
+        throw new Error('POST /v4/runs/wrun_1/events transport failure (UND_ERR_SOCKET)', {
+          cause: new TypeError('fetch failed', {
+            cause: Object.assign(new Error('other side closed'), {
+              name: 'SocketError',
+              code: 'UND_ERR_SOCKET',
+            }),
+          }),
+        });
+      };globalThis.__private_workflows = new Map();
+      globalThis.__private_workflows.set("workflow", workflow);`,
+      workflowRun,
+      []
+    );
+
+    const runFailureLog = errorSpy.mock.calls
+      .map((call) => String(call[0]))
+      .find((line) => line.includes('Error while running workflow'));
+    expect(runFailureLog).toBeDefined();
+
+    expect(runFailureLog).toContain('TypeError: fetch failed');
+    expect(runFailureLog).toContain(
+      'SocketError: other side closed (UND_ERR_SOCKET)'
+    );
+  });
 });

packages/core/src/runtime.ts

@@ -135,7 +135,12 @@ import {
   withTraceContext,
   withWorkflowBaggage,
 } from './telemetry.js';
-import { getErrorName, getErrorStack, normalizeUnknownError } from './types.js';
+import {
+  formatErrorCauseChain,
+  getErrorName,
+  getErrorStack,
+  normalizeUnknownError,
+} from './types.js';
 import { buildWorkflowSuspensionMessage } from './util.js';
 import {
   compileWorkflowBundle,
@@ -5110,6 +5115,16 @@ export function workflowEntrypoint(
                           // throw produced no stack to carry it.
                           errorMessage,
                           errorStack,
+                          // Neither of those reaches a wrapped error's
+                          // reason: `TypeError: fetch failed` carries an
+                          // empty message by design and a stack of pure
+                          // `node:internal/` frames, and the world layer's
+                          // own wrappers name the request that failed
+                          // rather than what failed about it. Undefined
+                          // when there is no cause, so the row disappears
+                          // for an ordinary user throw.
+                          errorCause:
+                            formatErrorCauseChain(terminalError) || undefined,
                         });
 
                         // Apply the source-map-remapped stack to the thrown

packages/core/src/types.test.ts

@@ -1,7 +1,11 @@
 import { runInNewContext } from 'node:vm';
 import { FatalError } from '@workflow/errors';
 import { describe, expect, it } from 'vitest';
-import { isAbortError, promoteAbortErrorToFatal } from './types.js';
+import {
+  formatErrorCauseChain,
+  isAbortError,
+  promoteAbortErrorToFatal,
+} from './types.js';
 
 describe('isAbortError', () => {
   it('recognizes an AbortError from another realm', () => {
@@ -58,3 +62,136 @@ describe('promoteAbortErrorToFatal', () => {
     expect(promoteAbortErrorToFatal(error)).toBe(error);
   });
 });
+
+describe('formatErrorCauseChain', () => {
+  it.each([
+    'cause',
+    'name',
+    'message',
+    'code',
+    'errors',
+  ])('tolerates a throwing %s getter in the cause chain', (property) => {
+    const cause = new Error('inner');
+    Object.defineProperty(cause, property, {
+      get() {
+        throw new Error('getter failed');
+      },
+    });
+
+    expect(formatErrorCauseChain(new Error('outer', { cause }))).toContain(
+      '[unavailable cause]'
+    );
+  });
+
+  it('preserves readable links before an inaccessible cause', () => {
+    const { proxy, revoke } = Proxy.revocable({}, {});
+    revoke();
+    const middle = new Error('middle', { cause: proxy });
+
+    expect(formatErrorCauseChain(new Error('outer', { cause: middle }))).toBe(
+      'Error: middle\n[unavailable cause]'
+    );
+  });
+
+  it('tolerates a throwing getter on the outer cause', () => {
+    const error = new Error('outer');
+    Object.defineProperty(error, 'cause', {
+      get() {
+        throw new Error('getter failed');
+      },
+    });
+
+    expect(formatErrorCauseChain(error)).toBe('[unavailable cause]');
+  });
+
+  it('returns an empty string when there is no cause', () => {
+    expect(formatErrorCauseChain(new Error('boom'))).toBe('');
+    expect(formatErrorCauseChain('not an error')).toBe('');
+    expect(formatErrorCauseChain(undefined)).toBe('');
+  });
+
+  it('renders the wrapped reason a `fetch failed` hides', () => {
+    // The whole point: the outer error says nothing, the cause says
+    // everything. The outer link is skipped — the log already prints it.
+    const cause = Object.assign(new Error('other side closed'), {
+      name: 'SocketError',
+      code: 'UND_ERR_SOCKET',
+    });
+    const wrapper = new TypeError('fetch failed', { cause });
+
+    expect(formatErrorCauseChain(wrapper)).toBe(
+      'SocketError: other side closed (UND_ERR_SOCKET)'
+    );
+  });
+
+  it('renders each link of a multi-level chain, outermost first', () => {
+    const inner = Object.assign(
+      new Error('getaddrinfo ENOTFOUND ai-gateway.vercel.sh'),
+      { code: 'ENOTFOUND' }
+    );
+    const wrapper = new Error('POST /v4/… transport failure (ENOTFOUND)', {
+      cause: new TypeError('fetch failed', { cause: inner }),
+    });
+
+    expect(formatErrorCauseChain(wrapper)).toBe(
+      [
+        'TypeError: fetch failed',
+        // The code is already in the message, so it is not repeated.
+        'Error: getaddrinfo ENOTFOUND ai-gateway.vercel.sh',
+      ].join('\n')
+    );
+  });
+
+  it('summarizes the attempts an AggregateError collects', () => {
+    // A happy-eyeballs connect reports every address it tried on `errors`
+    // and leaves the AggregateError itself blank.
+    const aggregate = new AggregateError(
+      [
+        Object.assign(new Error('connect ECONNREFUSED 10.0.0.1:443'), {
+          code: 'ECONNREFUSED',
+        }),
+        Object.assign(new Error('connect ECONNREFUSED [::1]:443'), {
+          code: 'ECONNREFUSED',
+        }),
+      ],
+      ''
+    );
+
+    expect(
+      formatErrorCauseChain(new TypeError('fetch failed', { cause: aggregate }))
+    ).toBe(
+      [
+        'AggregateError',
+        'Error: connect ECONNREFUSED 10.0.0.1:443',
+        'Error: connect ECONNREFUSED [::1]:443',
+      ].join('\n')
+    );
+  });
+
+  it('caps a long chain', () => {
+    let error = new Error('innermost');
+    for (let i = 0; i < 8; i++) {
+      error = new Error(`level ${i}`, { cause: error });
+    }
+
+    const lines = formatErrorCauseChain(error).split('\n');
+    expect(lines).toHaveLength(5);
+    expect(lines.at(-1)).toBe('…');
+  });
+
+  it('stops on a cyclic chain', () => {
+    const inner = new Error('inner') as Error & { cause?: unknown };
+    const outer = new Error('outer', { cause: inner });
+    inner.cause = outer;
+
+    expect(formatErrorCauseChain(outer)).toBe(
+      ['Error: inner', 'Error: outer'].join('\n')
+    );
+  });
+
+  it('renders a non-error cause', () => {
+    expect(
+      formatErrorCauseChain(new Error('boom', { cause: 'a string' }))
+    ).toBe('a string');
+  });
+});