fix(orb): cache brokered installation tokens to stop GitHub token-mint throttling - #1634
Merged
Conversation
…t throttling POST /v1/orb/token minted a fresh GitHub installation token on every call (no cache). GitHub tokens last ~1h, so minting that often throttles GitHub's token-creation endpoint — measured 16-20s live, exceeding the engine's 10s broker timeout, surfacing as orb_broker_degraded_serving_cached_token and (once the engine's in-memory cache lapses) orb_broker_unavailable, which blocks reviews. - Cache the minted token on the enrollment row (cached_token_json, migration 0081), encrypted at rest (AES-256-GCM via TOKEN_ENCRYPTION_SECRET, same scheme as the relay secret), folded into the existing SELECT. Serve a still-fresh cached token (>=10m before expiry); GitHub is minted at most once per install per ~hour and the cache survives engine restarts. Skipped (mint-every-call) when no encryption key is configured. - Give the cold transition mint timeout headroom (createOrbInstallationToken + the engine BROKER_TIMEOUT_MS) so the one uncached mint completes and populates the cache; steady-state cache hits return in well under a second. Closes #1633.
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1634 +/- ##
=======================================
Coverage 95.52% 95.53%
=======================================
Files 204 204
Lines 22063 22084 +21
Branches 7968 7972 +4
=======================================
+ Hits 21076 21097 +21
Misses 412 412
Partials 575 575
🚀 New features to boost your workflow:
|
9 tasks
JSONbored
added a commit
that referenced
this pull request
Jun 28, 2026
…umps (#1640) The self-host engine already caches brokered tokens in Redis (redis-token-cache.ts, wired in server.ts) — the right layer (no broker round-trip on a hit, survives restarts, mints ~hourly). #1634's cloud-side cache in brokerOrbToken was therefore redundant, did a wasteful per-call D1 write, and its read never hit in prod (always fell through to a fresh mint). Restore brokerOrbToken to mint-on-call. KEEP the useful parts of #1634: BROKER_TIMEOUT_MS=25s (broker-client.ts) + createOrbInstallationToken's 25s mint timeout (app-auth.ts). The migration-0081 cached_token_json column stays (already applied; an applied migration can't be dropped without a gap) — now an unused, harmless column.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The brokered token endpoint (
POST /v1/orb/token) minted a fresh GitHub installation token on every call — no server-side cache. GitHub installation tokens are valid ~1h, so minting that often throttles GitHub's token-creation endpoint (measured 16-20s responses from the live VPS, with the 3rd sequential mint timing out entirely). The engine aborts the broker call at 10s, so it can never complete a throttled mint → falls back to its in-memory cache (orb_broker_degraded_serving_cached_token) or, once that lapses (e.g. after a container restart), throwsorb_broker_unavailable— which blocks reviews. (Diagnosed live: secret valid, relay registered,/health0.57s, but/v1/orb/token16-20s.)Fix:
brokerOrbToken): cache the minted token on the enrollment row (cached_token_json, migration 0081), encrypted at rest (AES-256-GCM viaTOKEN_ENCRYPTION_SECRET— the same scheme as the relay secret), folded into the SELECT the broker already does (no extra read). Serve a still-fresh cached token (≥10m before expiry) instead of minting → GitHub is minted at most once per install per ~hour → throttling clears, and the cache survives engine restarts. Skipped (mint-every-call, as before) when no encryption key is set.createOrbInstallationToken) and the engine's broker call (BROKER_TIMEOUT_MS) get a generous timeout so the one uncached mint during the throttled transition completes and populates the cache; steady-state cache hits return in well under a second.Closes #1633.
Scope
site//CNAME/Pages; followsCONTRIBUTING.md.migrations/0081_*.sqlcommitted (Phase 4).Validation
git diff --check·actionlint·db:migrations:check·typechecktest:coverage— new broker tests cover every arm: cache hit (GitHub minted once across two exchanges; stored ciphertext, never plaintext), near-expiry re-mint, unparseable-cache fallthrough (read catch), and a fail-safe cache-write error (mint still succeeds + warns). Existing no-encryption-key tests cover the cache-skip branches.test:workers·build:mcp·test:mcp-pack·ui:*·npm audit --audit-level=moderateIf any required check was skipped, explain why:
Safety
Notes
brokerOrbToken+ the mint run on the cloud Orb (gittensory-api, auto-deploys on merge);BROKER_TIMEOUT_MSruns on the self-host engine (ships on the next rebuild).