Runtime credential rotation for subscription CLI providers (no restart, single-box + fleet)

[JSONbored]

Problem

Rotating the subscription-CLI credentials on an ORB review host (CLAUDE_CODE_OAUTH_TOKEN, and the codex auth.json) is a manual, multi-step, footgun-laden process. A live rotation on the hosted box surfaced two failure modes, both of which are silent — the container stays healthy and the status stays green while reviews quietly degrade:

1. Content shape. loadFileSecrets (src/selfhost/load-file-secrets.ts) only .trim()s the file. It does not strip comments or select a line. A human-added label line above the value (# some-account) becomes part of the credential. With AI_PROVIDER=claude-code,ollama the result is a failed auth that silently falls through to the Ollama fallback — no error surfaced, reviews just get worse.

2. Inode pinning. A Compose secrets: entry is a plain bind mount pinned to the inode. Writing the file in place propagates to the running container immediately; writing via write-new-then-rename (mv, and the default save behaviour of several editors) leaves the container serving the old content. docker compose up -d does not fix this — it reports Container Running and changes nothing, because the Compose config is unchanged. Only --force-recreate re-establishes the mount.

On top of that, the rotation currently requires a container recreate at all, which drops in-flight reviews for a credential swap that should be a no-op.

Why a restart shouldn't be needed

The two providers are structurally different, and one is already doing the right thing:

	Claude	Codex
Storage	docker secret file → env var, materialised once at boot	auth.json in the loopover-data volume
Read	parentEnv.CLAUDE_CODE_OAUTH_TOKEN inside run() (src/selfhost/ai.ts)	path re-resolved per call (resolveCodexAuthPath)
Restart to rotate?	Yes	No

Claude is nearly hot already: src/server.ts passes the live process.env object into createSelfHostAi, and the token is read inside run() per call, not captured at construction. The only missing piece is that nothing ever re-reads the file after boot.

Scope

Both deployment shapes need to be covered — a single self-hosted box and a multi-instance fleet.

1. Call-time credential resolution

Resolve the Claude token at call time with an explicit precedence:

1. Fleet — a DB-backed, encrypted-at-rest credential, injected via a Workers-safe registry slot.
2. Single box — a fresh read of CLAUDE_CODE_OAUTH_TOKEN_FILE, but only when the boot loader actually sourced the value from that file, so the documented "an inline .env value always wins" precedence in secrets/README.md is preserved exactly.
3. Fallback — the boot-time env value.

Degrade to the previous rung on an empty or unreadable file rather than hard-failing a review.

2. Host-side rotation (single box)

• scripts/rotate-secret.sh — one command that validates the value's shape (rejects multi-line, comments, whitespace, and a wrong prefix), backs up the old value, writes in place to preserve the inode, and verifies the running container's own view matches. Handles both the Compose secret files and codex's auth.json.
• A rotate-secret verb on the redeploy companion (scripts/redeploy-companion.ts), so the rotation can be driven remotely over the existing Unix-socket protocol rather than requiring an SSH session. The app container cannot write these files itself — the secrets mount is rw=false — so the host-side companion is the correct home for this.
• A loopover_admin_rotate_secret MCP admin tool on top of it, matching the existing loopover_admin_trigger_redeploy shape.

3. Fleet rotation

• A provider_credentials table reusing the existing BYOK envelope (AES-256-GCM via TOKEN_ENCRYPTION_SECRET, per-record salt, key_version), mirroring repository_ai_keys.
• INTERNAL_JOB_TOKEN-gated routes under /v1/internal/provider-credentials/* to set / inspect / clear a credential. GET must never return the value — only configured, last4, and timestamps, matching the BYOK status shape.
• Audited on every lifecycle change, and read fresh per call so a rotation takes effect on the very next review with no restart anywhere.

4. Operational hardening

• Support a sidecar *.label file so an operator can annotate which account a credential belongs to without putting a comment inside the secret file itself. (Teaching the shared loader to strip # comments is deliberately rejected: it is shared with the PEM key and every other secret, and a secret value may legitimately begin with #.)
• Document the in-place-write requirement and the inode trap in secrets/README.md.

Notes

claude_code_error_429 is a quota signal, not an auth signal — rotating a credential does not clear it. Worth keeping the two visibly distinct so an operator doesn't rotate a token that was never the problem.

[linear-code]

JSO-6593