Repository navigation
Workflow submission resolves floating transitive deps at run time, so a fresh npm publish can kill a generation #377
Description
What happened
R54 (aave-v4-v0012-main-issue368fix-r54-invariant-only, image built from merged main 0088f80) lost two
recovery generations without a single node failing. The node-error channel saw nothing, because the failure
was above the node layer:
workflow-submit-failed 2026-08-07T23:52:39.078Z WORKFLOW_SUBMISSION_FAILED
Command failed: npm install --prefix .../.workflow runner --ignore-scripts --package-lock=false \
--registry=https://registry.npmjs.org --no-audit --no-fund --loglevel=error
npm error code E404
npm error 404 Not Found - GET https://registry.npmjs.org/@ai-sdk/provider/-/provider-4.0.7.tgz
Root cause: a publish race, not a broken package
@ai-sdk/provider@4.0.7was published at 2026-08-07T23:48:58.306Z.- R54's
npm installran at 23:51:06Z — 2 minutes 8 seconds later. - The packument already listed 4.0.7 and advertised it as
dist-tags.latest, but the tarball had not
propagated to the CDN, so npm selected a version it could not download. - Re-checked at 23:57Z, the identical URL returns
200.
So the immediate condition was external and self-clearing. The monitor classified it correctly on its own
(transient-operational-failure -> relaunch) and the overseer backed off and retried. This issue is not
about that transient. It is about why a two-minute-old npm publish is on the critical path of a run at all.
The actual defect
packages/runtime/src/smithers.ts:1684 and packages/modal/src/node-worker.ts:84 both install the generated
workspace with --package-lock=false. Every workflow submission therefore re-resolves floating semver ranges
to whatever is newest on npm at that instant.
This repo already knows this hazard. packages/runtime/src/smithers-package.ts:19 documents it precisely:
@effect/platform-bunasks for@effect/platform-node-shared: ^4.0.0-beta.102, an open caret over
prereleases, and the generated workspace is installed with--package-lock=false. Unpinned, two cloud
containers resuming the same run at different times install different@effect/*builds, and a newer beta
that needs Effect APIs absent from the pinned core breaks every run including in-flight resumes.
REQUIRED_SMITHERS_OVERRIDES answers that hazard for effect and four @effect/* packages — the ones that
had already bitten. @ai-sdk/provider is not in the pin set, and it reached the run through an open range.
Two consequences, the second worse than the first:
- Availability. A publish race turns into a dead generation, as here.
- Reproducibility. An "immutable image" is not hermetic. Two runs from the same image, launched hours
apart, can install different dependency trees. For a benchmark whose entire purpose is comparing runs
against a fixed ground truth, that silently undermines the comparison — and unlike a 404, it leaves no
error to notice.
Why it is filed rather than fixed right now
R54 does not need this fix to proceed: the transient has provably cleared, and rebuilding the image to widen
the pin set would discard R54's progress and restart the clock. Filing without fixing follows the same
recorded discipline as #285, #348 and #361.
Acceptance criteria
- A regression test that fails without the fix: given a manifest whose transitive closure reaches
@ai-sdk/providerthrough an open range, the generated workspace pins it to a known version. - Decide, and write down, the general rule rather than adding one more name to a list. Adding
@ai-sdk/providertoREQUIRED_SMITHERS_OVERRIDESfixes today's package and leaves the mechanism
intact; the third occurrence of this bug should not require a fourth issue. Options worth weighing:
ship a lockfile with the generated workspace, or resolve-and-freeze once at image build time. - Whatever is chosen must keep working for in-flight resumes, which is the constraint that motivated
--package-lock=falsein the first place — establish that reason before removing the flag.
Evidence
Full payload preserved at
/home/ubuntu/ultrafuzz-private/aave-v4-v0012-n1-20260802/r54/r54-terminal-20260807T235540Z-workflow-log.txt
and in the run volume's events.index/type/workflow-submit-failed.jsonl.
Activity
aviggiano commentedon Aug 8, 2026
PR #384 implements this:
npm install --before=<cutoff>on both installers, so the whole transitive closure resolves from the manifest plus one constant instead of the wall clock.Against the acceptance criteria:
startRun resolves the generated workspace as of a fixed instant, not the launch clock, verified red onorigin/main; the red diagnostic is R54's argv verbatim.--beforecovers every open range below the pins, not just@ai-sdk/provider. And it is enforced rather than documented:SMITHERS_PIN_PUBLISH_TIMES+assertSmithersResolutionCutofffail the suite if a pin is raised without moving the cutoff past its publish instant.--package-lock=falsefirst, as asked:git log -Sputs its origin at 9a4a3ad ("Apply lean security audit remediations", Apply lean security audit remediations #45), added in one hunk with--ignore-scripts,--no-audit,--no-fundand the explicit--registry. It is install hardening, not a resume requirement. Either way fix(runtime): resolve the generated workspace as of a fixed instant, not the launch clock #384 does not remove it — that is the main reason it takes--beforeover the lockfile option, sincenpm cihard-errors on lock/manifest drift and would turnmigrateLegacySmithersPackageManifestfrom a forward migration into a launch failure for any project root written by an older Ultrafuzz.Cutoff chosen as
2026-08-02T00:00:00Z— the next UTC midnight after the newest pin,smithers-orchestrator@0.32.0(published2026-08-01T05:00:25.735Z).Note #384 conflicts textually with #380 (#379's retry) at the same two call sites; they are complementary and whichever merges second needs a trivial rebase.
fix(runtime): resolve the generated workspace as of a fixed instant (#…