Skip to content

fix(orb): cache brokered installation tokens to stop GitHub token-mint throttling - #1634

Merged
JSONbored merged 1 commit into
mainfrom
fix/orb-token-cache
Jun 28, 2026
Merged

fix(orb): cache brokered installation tokens to stop GitHub token-mint throttling#1634
JSONbored merged 1 commit into
mainfrom
fix/orb-token-cache

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

The brokered token endpoint (POST /v1/orb/token) minted a fresh GitHub installation token on every call — no server-side cache. GitHub installation tokens are valid ~1h, so minting that often throttles GitHub's token-creation endpoint (measured 16-20s responses from the live VPS, with the 3rd sequential mint timing out entirely). The engine aborts the broker call at 10s, so it can never complete a throttled mint → falls back to its in-memory cache (orb_broker_degraded_serving_cached_token) or, once that lapses (e.g. after a container restart), throws orb_broker_unavailable — which blocks reviews. (Diagnosed live: secret valid, relay registered, /health 0.57s, but /v1/orb/token 16-20s.)

Fix:

  1. Server-side token cache (brokerOrbToken): cache the minted token on the enrollment row (cached_token_json, migration 0081), encrypted at rest (AES-256-GCM via TOKEN_ENCRYPTION_SECRET — the same scheme as the relay secret), folded into the SELECT the broker already does (no extra read). Serve a still-fresh cached token (≥10m before expiry) instead of minting → GitHub is minted at most once per install per ~hour → throttling clears, and the cache survives engine restarts. Skipped (mint-every-call, as before) when no encryption key is set.
  2. Timeout headroom for the cold transition mint: the broker's GitHub mint (createOrbInstallationToken) and the engine's broker call (BROKER_TIMEOUT_MS) get a generous timeout so the one uncached mint during the throttled transition completes and populates the cache; steady-state cache hits return in well under a second.

Closes #1633.

Scope

  • Conventional Commit title; focused (broker + its mint/timeout + a 1-column migration + tests).
  • No site//CNAME/Pages; follows CONTRIBUTING.md.
  • DB change → contiguous migrations/0081_*.sql committed (Phase 4).

Validation

  • git diff --check · actionlint · db:migrations:check · typecheck
  • test:coverage — new broker tests cover every arm: cache hit (GitHub minted once across two exchanges; stored ciphertext, never plaintext), near-expiry re-mint, unparseable-cache fallthrough (read catch), and a fail-safe cache-write error (mint still succeeds + warns). Existing no-encryption-key tests cover the cache-skip branches.
  • test:workers · build:mcp · test:mcp-pack · ui:* · npm audit --audit-level=moderate

If any required check was skipped, explain why:

  • No OpenAPI/cf-typegen: response shape + bindings unchanged (token-mint internals only).

Safety

  • No secrets/wallets/trust-scores in code/tests/PR text. The cached token is a short-lived (~1h) GitHub credential stored encrypted at rest on the enrollment row (mirrors the existing encrypted relay secret); the cache test asserts the ciphertext never contains the plaintext token.
  • Auth/eligibility unchanged — the enrollment-secret validation + install eligibility gate still run on every exchange (the cache only skips the GitHub mint), and the negative-path tests (invalid/revoked enrollment, ineligible install) are unchanged + green.
  • No public GitHub text change.

Notes

  • brokerOrbToken + the mint run on the cloud Orb (gittensory-api, auto-deploys on merge); BROKER_TIMEOUT_MS runs on the self-host engine (ships on the next rebuild).

…t throttling

POST /v1/orb/token minted a fresh GitHub installation token on every call (no
cache). GitHub tokens last ~1h, so minting that often throttles GitHub's
token-creation endpoint — measured 16-20s live, exceeding the engine's 10s
broker timeout, surfacing as orb_broker_degraded_serving_cached_token and (once
the engine's in-memory cache lapses) orb_broker_unavailable, which blocks reviews.

- Cache the minted token on the enrollment row (cached_token_json, migration
  0081), encrypted at rest (AES-256-GCM via TOKEN_ENCRYPTION_SECRET, same scheme
  as the relay secret), folded into the existing SELECT. Serve a still-fresh
  cached token (>=10m before expiry); GitHub is minted at most once per install
  per ~hour and the cache survives engine restarts. Skipped (mint-every-call)
  when no encryption key is configured.
- Give the cold transition mint timeout headroom (createOrbInstallationToken +
  the engine BROKER_TIMEOUT_MS) so the one uncached mint completes and populates
  the cache; steady-state cache hits return in well under a second.

Closes #1633.
@dosubot dosubot Bot added the size:M label Jun 28, 2026
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jun 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.53%. Comparing base (b1631fb) to head (c374210).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1634   +/-   ##
=======================================
  Coverage   95.52%   95.53%           
=======================================
  Files         204      204           
  Lines       22063    22084   +21     
  Branches     7968     7972    +4     
=======================================
+ Hits        21076    21097   +21     
  Misses        412      412           
  Partials      575      575           
Files with missing lines Coverage Δ
src/orb/app-auth.ts 100.00% <100.00%> (ø)
src/orb/broker-client.ts 100.00% <100.00%> (ø)
src/orb/broker.ts 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JSONbored
JSONbored merged commit 758e524 into main Jun 28, 2026
19 checks passed
@JSONbored
JSONbored deleted the fix/orb-token-cache branch June 28, 2026 00:15
@JSONbored JSONbored self-assigned this Jun 28, 2026
JSONbored added a commit that referenced this pull request Jun 28, 2026
…umps (#1640)

The self-host engine already caches brokered tokens in Redis (redis-token-cache.ts,
wired in server.ts) — the right layer (no broker round-trip on a hit, survives
restarts, mints ~hourly). #1634's cloud-side cache in brokerOrbToken was therefore
redundant, did a wasteful per-call D1 write, and its read never hit in prod
(always fell through to a fresh mint). Restore brokerOrbToken to mint-on-call.

KEEP the useful parts of #1634: BROKER_TIMEOUT_MS=25s (broker-client.ts) +
createOrbInstallationToken's 25s mint timeout (app-auth.ts). The migration-0081
cached_token_json column stays (already applied; an applied migration can't be
dropped without a gap) — now an unused, harmless column.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

No open projects
Status: Done

Development

Successfully merging this pull request may close these issues.

orb_broker_unavailable: /v1/orb/token mints on every call (no cache) → GitHub throttles to 16-20s → reviews blocked

1 participant