Repository navigation
[workers-auth] Recover from a partially installed keyring native binding - #15223
Conversation
🦋 Changeset detectedLatest commit: 7f09943 The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Codeowners approval required for this PR:
Show detailed file reviewers |
|
✅ All changesets look good |
|
Now I have a solid understanding. Let me consider one important concern about Critical concern — In But there's a subtle real-world risk that the test doesn't catch: the However — one genuinely important issue: the recovery test rewrites Let me verify there isn't an issue with the memoization interaction — specifically whether a successful Let me double-check one thing: the Both changes are well-reasoned, correct, and thoroughly documented. The tests are appropriate. Let me verify there are no edge cases with the The I've reviewed both changes carefully:
No logic bugs, security issues, backward-compat violations, or incorrect API behavior found. The author was appropriately transparent about the untested mock-npm-registry path, and the reasoning is valid. LGTM |
@cloudflare/autoconfig
@cloudflare/build-output-utils
@cloudflare/config
create-cloudflare
@cloudflare/deploy-helpers
@cloudflare/kv-asset-handler
miniflare
@cloudflare/pages-functions
@cloudflare/pages-shared
@cloudflare/unenv-preset
@cloudflare/vite-plugin
@cloudflare/vitest-pool-workers
@cloudflare/workers-auth
@cloudflare/workers-editor-shared
@cloudflare/workers-utils
wrangler
commit: |
NuroDev
left a comment
There was a problem hiding this comment.
Apart from a few comments from Devin that need looking at this looks good to me
workers-devprod
left a comment
There was a problem hiding this comment.
Codeowners reviews satisfied
Applies the same principle as the miniflare Chrome install fix: the presence of files is not evidence of a working install. `findKeyringBinding` accepted any `@napi-rs/keyring` whose `index.js` existed. That `index.js` requires a platform-specific `.node` binary delivered as a separate package, so an interrupted or partially-failed `npm install` leaves it on disk with nothing loadable beside it. The binding was then reported as available, so the install was never retried, and every credential read and write failed with a raw module-resolution error. The verdict was memoised per process, making the breakage sticky. A candidate is now only accepted once it has actually been required — the same thing `resolveKeyringEntryFactory` does with it moments later — so a half-installed binding is reported absent, reinstalled, and then works. Also fixes `mock-npm-registry`'s `stop()`, which resolved on `treeKill`'s callback. That fires when the signals have been delivered, not when the process tree has exited, and callers remove the storage directory and may restart on the same port straight afterwards. It now waits for the real `exit` event, bounded by a timeout so a wedged process cannot hang teardown. `packages/wrangler/e2e/helpers/command.ts` already did this correctly.
… as a failure Review feedback: reporting a half-installed binding as absent fixed the permanent breakage, but left the retry unbounded in one case. `npm` exits 0 when an optional dependency cannot be fetched, and the platform-specific `.node` binary is exactly that. On a machine where it can never be fetched the install therefore "succeeded", so `handleNeedsInstall()` never latched `installFailedThisSession` — that only happens when the install throws — and the resolver, which re-resolves per credential operation, took the `needs-install` branch again on the very next read or write. The user would sit through a fresh multi-second install before every single credential access, each one ending in an unhelpful internal error. The install now verifies its own work: after npm exits 0 it checks that the binding actually loads, and throws a `UserError` when it does not. That routes the failure through the existing remediation path — one install attempt per session, a single explanation, then a fallback to the plaintext credentials file (or a hard error when `CLOUDFLARE_AUTH_USE_KEYRING=true` demanded the keyring). `resolveKeyringEntryFactory()` also no longer throws a bare `Error` when the binding is unavailable. That is close to an internal invariant violation now that resolution verifies loadability, but it is still the user's machine that needs fixing, so it carries the same remediation hint. Also rewrites the changeset to drop the forbidden `fix:` prefix and describe user impact rather than internals, per REVIEW.md.
ed11d1e to
7f09943
Compare
Two instances of the same class of bug found while auditing the codebase after #15206, which fixed a partially-downloaded Chrome install in miniflare. The shared principle: the presence of files is not evidence of a working install, and a delivered signal is not evidence of an exited process.
workers-auth: a half-installed keyring binding was trusted foreverOn Windows, opting into keyring-backed credential storage lazily
npm installs@napi-rs/keyring.findKeyringBinding()accepted any candidate whoseindex.jsexisted:That
index.jsrequires a platform-specific.nodebinary which npm delivers as a separate package. An install that is interrupted, runs out of disk, or fails to fetch the platform package leavesindex.json disk with nothing loadable beside it.The consequences compounded:
resolveKeyProvider()saw a non-nullresult and returnedkind: "available", so theneeds-installbranch was never taken and the install was never retried.createRequire(...)inresolveKeyringEntryFactory()— no remediation hint, no fallback.cachedBindingPathByDir, which is only invalidated by a successful install orsetNpmRunner(). Since step 1 means no install is ever attempted, the breakage was permanent for that config dir.A candidate is now only accepted once it has actually been
required — precisely what the credential path does with it moments later, so this costs nothing extra (requirecaches on success) and only ever runs on the Windows opt-in path. A half-installed binding is reported absent, so the resolver reinstalls it and it then works.Bounding the retry
Reporting the wreck as absent fixes the permanent breakage but, as Devin spotted, leaves the retry unbounded in one case.
npmexits 0 when an optional dependency cannot be fetched, and the platform-specific.nodebinary is exactly that — so on a machine where it can never be fetched (no network route to the platform package, or an npm configured to skip optional dependencies), the install "succeeds" while producing nothing loadable.handleNeedsInstall()only latchesinstallFailedThisSessionwhen the install throws, so nothing latched, and because the resolver re-resolves on every credential operation it took theneeds-installbranch again on the very next read or write. The user would sit through a fresh multi-second install before every single credential access, each one still ending in an internal error.The install now verifies its own work: after npm exits 0 it checks the binding actually loads, and throws a
UserErrorwhen it does not. That routes the failure through the existing remediation path — one attempt per session, a single explanation, then a fallback to the plaintext credentials file, or a hard error whenCLOUDFLARE_AUTH_USE_KEYRING=truedemanded the keyring. A transient failure is still retried on the next invocation, since the latch is per-process.resolveKeyringEntryFactory()also no longer throws a bareErrorwhen the binding is unavailable. With resolution now verifying loadability that is close to an internal invariant violation, but it is still the user's machine that has to be fixed, so it carries the same remediation hint.mock-npm-registry:stop()resolved before the process had exitedtreeKill's callback fires when the signals have been delivered, not when the process tree has finished exiting. CallersremoveDir()the registry's storage directory immediately afterwards and may restart on the same port, both of which race a Verdaccio that is still shutting down —ENOTEMPTY/EBUSYon directory removal (notably on Windows, where an open file blocks deletion) andEADDRINUSEon restart.It now waits for the real
exitevent, bounded by a 10s timeout so a wedged process cannot hang test teardown. The listener is attached atfork()time rather than insidestop(), so the event cannot be missed.packages/wrangler/e2e/helpers/command.tsalready used this pattern correctly; this brings the registry helper in line.Testing notes
The
workers-authchange has six new tests, each mutation-verified — removing the corresponding guard fails exactly the intended test and nothing else:treats a binding whose native binary is missing as absentignores a global binding that cannot be loadedrecovers when a reinstall repairs a half-installed bindingthrows a UserError when npm exits 0 without leaving a loadable bindingstops retrying when npm exits 0 but the binding is still unloadableexpected 3 to be 1— three credential operations, three installs)hard-errors when CLOUDFLARE_AUTH_USE_KEYRING=true and the install lands no loadable bindingthrows a UserError with remediation when no binding is availableresolveKeyringEntryFactory()reverts to a bareErrorThe fixtures reproduce the real failure exactly, with an
index.jscontainingrequire("./keyring.win32-x64-msvc.node")and no such binary. Two pre-existing tests whose fake npm runner exited 0 without writing anything now have to land a loadable binding, since the install verifies itself.I did not add a test for the
mock-npm-registrychange, and want to be upfront about why: I wrote one, and it passed against the unfixed code too. Verdaccio exits fast enough after being signalled on macOS that the port is already free by the time the assertion runs, so the test proved nothing. The race only bites where an exiting process still holds its files or the port is reclaimed before the listener is released, which I cannot reproduce on this platform. Rather than commit a test that would give false confidence, I've left it out. That code path is exercised (though not asserted on) by thecreate-cloudflare,vitest-pool-workers, andvite-plugin-cloudflaree2e global setups.pnpm checkpasses: 0 lint errors, all typecheck tasks green,workers-auth16 test files / 173 tests passing, pluswrangler's 93user.test.tstests.Note
This is a contribution from an AI agent: OpenCode, claude-opus-5.