[Bug]: a local Windows ACL hardening failure is returned to the client as `401 authentication_error`

[brunoflma]

Client or integration

Codex CLI

Area

Authentication and account pool

Summary

A local filesystem problem — Windows ACL hardening of a secret path exceeding its budget — reaches the client as 401 authentication_error. The user sees what looks like an expired or rejected credential, and the actual message can be the literal ACL hardening skipped - previous attempt timed out.

Two properties combine to produce this:

1. hardenSecretPath(..., { required: true }) throws on a genuine timeout, and the timeout memo timedOutPaths is process-global (src/lib/windows-secret-acl.ts:47). Once a path times out, subsequent writes to it keep failing for the life of the process — one authorized recovery attempt aside.
2. src/server/responses/core.ts passes the raw exception message straight into an auth error in three branches (:880, :883, :886):

return { ok: false, response: formatErrorResponse(401, "authentication_error", err.message) };

When the poisoned path is auth.json, an OAuth refresh cannot persist, and the resulting failure is reported as a credential error rather than as a local storage/permission error. src/server/management-auth.ts is more careful on its own surface — it says management token directory ACL hardening did not complete (:76, :95) — but the data-plane path has no such classification.

Expected: an ACL/filesystem failure should surface with its own classification (a 5xx, or at minimum an error type that is not authentication_error), naming the local cause. Whatever the status code, it should not be indistinguishable from "your token is bad", because the two have opposite remediations — one is chmod/ACL/disk, the other is re-login.

This is a diagnosability defect rather than a data-loss one, but it cost me a long misdiagnosis: I went looking at provider credentials while the actual cause was a directory tree under ~/.opencodex.

Reproduction

Honest status first: I no longer have a live reproduction, because I fixed the local trigger. I am reporting it because the mechanism is unchanged in 2.11.0 and the misclassification is structural, not incidental. The template invites this case, so here is exactly what I observed and under which conditions.

What produced it originally, on 2.10.x:

1. ~/.opencodex had grown to ~94,700 descendants (git worktrees and backups under it).
2. hardenSecretDir applies /grant:r <principal>:(OI)(CI)(F) followed by /inheritance:r. (OI)(CI) makes Windows propagate the ACE to every descendant, so the write cost is linear in the number of descendants — measured on this host: 88 ms at 10 files, 9,438 ms at 20,000, extrapolating to ~45 s at 94,715.
3. That exceeded the harden budget, hardenSecretPath(required: true) threw, and the path entered the global timeout memo.
4. POST /v1/responses then returned 401 authentication_error carrying the literal text ACL hardening skipped - previous attempt timed out. The service log had 319 occurrences of the ACL failure up to the day I fixed it.
5. The dashboard surface failed differently and more usefully, with GET /api/logs → 503 {"error":"management API unavailable"} and reason management token directory ACL hardening did not complete.

To recreate the condition synthetically on Windows, without waiting for a tree to grow:

1. Place a large subtree under ~/.opencodex (tens of thousands of files) with ACL inheritance enabled.
2. Lower the envelope: OPENCODEX_ACL_TIMEOUT_MS=1000.
3. Restart the proxy and send POST /v1/responses for any routed provider.
4. Observe a 401 authentication_error whose message describes ACL hardening rather than a credential.

Reading the code path alone is enough to see the classification issue: windows-secret-acl.ts throws a sanitized ACL error, and responses/core.ts:880-886 re-labels whatever message arrives as authentication_error.

Note on 2.11.0: #1156 raised HARDEN_DEADLINE_DEFAULT_MS from 5,000 to 30,000 (src/lib/windows-secret-acl.ts:253), which widens the margin — but it does not change the classification, and the process-global memo is unchanged.

Version

2.11.0

Operating system

Windows 11 Home Single Language 26H2 (build 10.0.26200)

Provider and model

Not provider-specific — the failure is in credential persistence, before provider selection.

Logs or error output

# data plane — the misleading one
POST /v1/responses -> 401 authentication_error
  message: "ACL hardening skipped - previous attempt timed out"

# management plane — same root cause, correctly classified
GET /api/logs -> 503 {"error":"management API unavailable"}
  reason: "management token directory ACL hardening did not complete"

# measured propagation cost of `/grant:r (OI)(CI)` on this host
10 files      ->     88 ms
20,000 files  ->  9,438 ms
94,715 files  -> ~45 s (extrapolated; above the 2.10.x budget)

Screenshots and supporting files

Not attached.

Redacted configuration

{
  "note": "Not configuration-dependent. Relevant environment only:",
  "OPENCODEX_ACL_TIMEOUT_MS": "5000",
  "platform": "win32"
}

Checks

• I searched existing issues and documentation.
• I removed secrets, tokens, account details, request credentials, and personal data.

[github-actions]

_{Automated translation bookkeeping — detected language: English.}

[lidge-jun]

@brunoflma Traced this on current dev and I am not patching it yet — but not because the report is wrong. Here is exactly what I found, so the next attempt starts further along than mine did.

Your analysis of the misclassification shape is correct. src/server/responses/core.ts does pass err.message straight into a 401 authentication_error in three branches, and src/server/management-auth.ts:76 does classify the same underlying cause correctly on its own surface. The asymmetry you describe is real and worth closing.

What I could not confirm is that anything currently reaches those branches with an ACL error. Working backwards on dev:

• CodexDirectAuthenticationError takes no message at all — it is a fixed missing-bearer string (auth-context.ts:132).
• ForwardAdmissionCredentialError is likewise a fixed proxy-bearer message (auth-cors.ts:359).
• CodexPoolAuthenticationError accepts only a message and preserves neither code nor cause (auth-context.ts:108).

And upstream of all three, read-path hardening is now required: false (config.ts:1426, codex/account-store.ts:88), which returns {ok: false} instead of throwing. grep -rn 'required: true' finds it only on write paths — lock files, the profile manager, the catalog writer, the prompt journal, the spill store, and management-auth. The OAuth store does not harden with required: true at all.

So the chain you described — poisoned auth.json → refresh cannot persist → 401 carrying the ACL text — does not appear to be reachable through those branches on dev. I drafted a fix that pushed a synthetic ACL-coded error through them, and abandoned it: that would have tested a state production does not create, and gone green while proving nothing.

Deliberately not closing this as disproven. "I found no current producer-to-401 path" is a weaker claim than "the path is gone", and I have not traced every write-path ACL failure to its surface. If you still have the 2.10.x service log, a stack trace or the exact version alongside one of those 319 occurrences would settle which producer it was — and if it is a write path, the classification work still needs doing, just somewhere other than where I looked.

One thing worth noting for whoever picks this up: the ACL errors already carry usable codes (ETIMEDOUT, EICACLS, EACCES, EPERM, EACLIDENTITY, EACLRETRYEXHAUSTED), so classification never needs to match on message text.

For what it is worth, the diagnosis you did here was the valuable part — measuring the (OI)(CI) propagation cost at 88 ms / 10 files versus 9.4 s / 20,000 is the kind of evidence that makes a performance claim checkable instead of arguable.

[brunoflma]

Still applies to 2.12.0, verified rather than assumed — src/lib/windows-secret-acl.ts is unchanged between v2.11.0 and v2.12.0, timedOutPaths is still the process-global map, and responses/core.ts still passes err.message straight into formatErrorResponse(401, "authentication_error", ...) in the same three branches.

Two things worth adding, both from measurements on this host since I filed.

I still cannot hand you a live reproduction, and I want to be precise about why. The trigger here was topology, not latency: ~/.opencodex had grown to ~94,700 descendants, and /grant:r (OI)(CI) propagates to every one of them. I have since pruned that tree to 14,123 entries, and the harden now costs 31 ms for the DACL read and 27 ms for a findsid. So the local condition that produced the 401 is gone on my machine, and I cannot re-trigger it without deliberately re-growing the tree. The misclassification itself is structural and unchanged — that part does not depend on my host.

#1156 is a margin, not a fix, and it is easy to lose by accident. Raising HARDEN_DEADLINE_DEFAULT_MS from 5,000 to 30,000 buys room, but an operator carrying an explicit OPENCODEX_ACL_TIMEOUT_MS=5000 — which was a no-op before 2.11.0, since it merely repeated the default — silently keeps the old envelope and gives back the margin. I was in exactly that state without noticing until I re-read the constant. Not a bug in your code; worth knowing that the mitigation can be inert on hosts that pinned the old value back when pinning it meant nothing.

What I would still ask for is the classification, independent of any envelope: an ACL/filesystem failure reaching the client as authentication_error sends the operator to re-authenticate when the fix is a directory ACL. The management surface already gets this right (management token directory ACL hardening did not complete); the data plane is where the signal is lost.

[Wibias]

Thanks — the additional measurements and the note about an explicitly pinned OPENCODEX_ACL_TIMEOUT_MS=5000 are useful. I agree with the narrow point that #1156 is only additional margin; it does not itself provide error classification.

I do want to keep one distinction precise, though: the fact that timedOutPaths and the generic 401 authentication_error mappings still exist does not by itself establish that this specific bug is reachable in 2.12/current dev.

I re-traced this after the original report. The particular path described in the issue — ACL failure around the OAuth/auth store reaching those core.ts branches with the ACL message intact — does not appear to exist on current dev: the relevant read-side hardening is non-required, two of the authentication errors have fixed messages, and CodexPoolAuthenticationError does not preserve an ACL code/cause.

So I don’t want to “fix” the three generic 401 branches by injecting a synthetic ACL-coded error that production currently cannot produce. That would give us a green regression for an invented state.

The next useful step is narrower: identify one actual required: true ACL failure on current dev that can propagate into a client-visible authentication error, then trace that producer all the way to its response surface. If that path exists, I agree the classification should be fixed there — preferably by preserving the typed ACL cause/code and returning a local-storage/server error rather than authentication_error, consistently across every surface using that wrapper.

If no such producer-to-surface path exists anymore, then the honest conclusion is that the 2.10.x failure was real but the original route became unreachable through later hardening changes, while the generic error-classification shape remains something worth guarding against.

If you still have even a small excerpt from one of those old 2.10.x occurrences — exact patch version plus surrounding stack/log context — that could identify the original producer and settle which path actually produced the 401.

And yes: the pinned-5000ms observation is worth keeping. An old explicit value can silently cancel the extra margin introduced by #1156, even though that is separate from the classification question.

[lidge-jun]

Fixed on dev (1e816ee): Windows ACL hardening failures (icacls timeout, NTFS permission errors) are now classified as 503 server_error instead of 401 authentication_error. Users will see local remediation guidance instead of credential error messages. 다음 배포에서 해결될 예정입니다.