A step argument with a custom [WORKFLOW_SERIALIZE] discards the retained VM, so agent loops replay the full log every iteration (WorkflowAgent is affected by default) #4316

Checked against @workflow/core 5.0.0-beta.53 and 5.0.0-beta.55 (identical code path), @ai-sdk/workflow 2.0.33 and 2.0.43.

Summary

When a step argument is an instance of a class with a custom [WORKFLOW_SERIALIZE]
method, dehydrating that step's input records a guest-code execution
(serialization/reducers/class.js, the Instance reducer calls
recordGuestCode('method', '[WORKFLOW_SERIALIZE] (...)') unconditionally).
getRetentionDecision in runtime.js then returns
{ retain: false, reason: 'serialization_executed_workflow_code' } and the
retained VM session is discarded. The next pass of the inline loop is a full
replayWorkflow over the entire event log.

A workflow that calls such a step once per loop iteration therefore pays one
full replay per iteration. Each replay is O(events), so total non-step time is
O(iterations²), and every replay is charged to the per-invocation replay budget
(REPLAY_TIMEOUT_MS, 240 s). Nothing surfaces it outside DEBUG.

Passing a model wrapper to a step is the natural shape for this: a custom
language-model class has to implement [WORKFLOW_SERIALIZE] to cross into the
step, and an agent loop calls the model step once per iteration.

Repro (clean npx workflow scaffold, local world)

// workflows/loop.ts
import { WORKFLOW_DESERIALIZE, WORKFLOW_SERIALIZE } from '@workflow/serde';

export class Model {
  constructor(public init: { id: string }) {}
  static [WORKFLOW_SERIALIZE](instance: Model) { return instance.init; }
  static [WORKFLOW_DESERIALIZE](init: { id: string }) { return new Model(init); }
}

async function call(model: Model | { id: string }, i: number) {
  'use step';
  return { i, payload: 'x'.repeat(20_000) };
}
async function after(i: number) {
  'use step';
  return i;
}

export async function loop(iterations: number, useClass: boolean) {
  'use workflow';
  for (let i = 0; i < iterations; i++) {
    const model = useClass ? new Model({ id: 'm' }) : { id: 'm' };
    await call(model, i);
    await after(i);
  }
}

Run loop(160, true) and loop(160, false) with
DEBUG=workflow:runtime:debug, and compare the gap between each call
completing and the following after starting (step_completed to step_started
in the event log):

The gap is the full replay. On a larger workflow body with bigger step outputs
the per-iteration replay reaches seconds by a few hundred steps.

This affects WorkflowAgent out of the box

The AI SDK's own provider language models implement [WORKFLOW_SERIALIZE] (for example @ai-sdk/gateway's language model class has static [WORKFLOW_SERIALIZE](model)), and @ai-sdk/workflow's WorkflowAgent passes the model instance straight into its model step (doStreamStep(conversationPrompt, currentModel, ...)). So every WorkflowAgent loop demotes the retained VM once per model call and replays the whole log before the next step, with no custom classes involved.

In production (@workflow/core 5.0.0-beta.53, @ai-sdk/workflow 2.0.33, Vercel), on an 837-step agent run the gap between the model step completing and the next tool step starting grew from about 0.36 s early in the run to about 3.7 s late in the same invocation. That added up to about 290 s of the run's wall time, and the per-invocation non-step time climbed toward the 240 s replay budget. The same pattern likely explains the linear per-transition growth reported in #3026.

We worked around it by reducing the model argument to a plain { classId, data } envelope in the replayed workflow body and reviving it in the step from the class registry. Retention then holds across model calls (1 replay instead of 161 in the repro above).

Why the demotion is avoidable

The hazard being guarded is real: a serializer runs only on the live path, so if
it perturbs VM state (a PRNG draw, a write to reachable state) the retained VM
drifts from what a cold replay would produce. But:

1. The documented contract already forbids that. foundations/serialization
   says WORKFLOW_SERIALIZE runs "inside the workflow context and [is] subject
   to the same constraints as "use workflow" functions" and should be "focused
   on data transformation only".
2. The hazard exists only because the serializer runs off the replayed path. If
   the runtime reduced custom-class step arguments to { classId, data } at the
   step CALL site (inside the workflow body, which replays), the serializer would
   run identically on every replay and dehydration would execute no guest code.
   We confirmed this shape works on the current runtime by doing that reduction
   in user code: retention holds and the per-iteration gap stays flat.

Possible fixes, in order of preference:

• Reduce custom-class step arguments in the replayed body (above), which keeps
  the determinism guarantee without trusting the serializer.
• Or let a class declare its serializer pure (a static brand read as an own data
  property), and skip recordGuestCode for it.
• At minimum, log the demotion above debug: today a loop silently becomes
  quadratic.