feat(control-plane): persist provisioning and failed tenant lifecycle states for the customer status surface

[cleanjunc]

Summary

Implements #7677's ratified decision (2026-07-21) — the provisioning-status state machine behind the customer-facing polling surface:

• "failed" added to TenantLifecycleState (control-plane/src/tenant-provisioning-driver.ts) — the spec's bolded gap: today a provision-step failure pages and rethrows but persists nothing, so a polling customer would watch a record stuck at "provisioning" forever with no signal.
• provisionTenant transitions the record to "failed" before rethrowing (control-plane/src/provisioning.ts): a new optional onFailure seam runs first in the existing catch — best-effort by design (its own rejection is swallowed), so a KV write outage can never mask the provisioning error, which still pages (Wire real PagerDuty alerting into control-plane provisioning failures #7667) and rethrows exactly as before.
• The record is now written as "provisioning" BEFORE the standup starts (POST /v1/tenants, control-plane/src/http-app.ts): without this pre-write, the transitional state the decision's polling loop watches ("Customer dashboard polls every 3-5s while in provisioning") was never observable at all — the old flow only ever wrote the record after success. Per the ratified decision, the polling surface is the already-decided GET /v1/tenants transport (Stand up control-plane's real HTTP transport (POST/GET/DELETE /v1/tenants) matching the already-merged client #7654) — no new endpoint is added.
• "failed" is re-creatable, like "torn down" (isRecreatableState, both conflict sites — name+product and the Route incoming GitHub webhooks to the correct tenant's hosted ORB container #7181 installation-ID claim): before this PR a failure left NO record so a retry always worked; persisting the failure must not turn "Setup failed" into a permanent squat on the tenant name. An active tenant still 409s both checks, pinned by test. DELETE of a failed tenant needs no change — deprovisionTenant is idempotent by driver contract, so cleanup works as-is.

Deliberate scope line, from the spec's own text: the customer dashboard polling UI and its three-state copy ("Setting up your instance" / "Ready" / "Setup failed") live "alongside #4926/#4927" — both of which are open and owner-assigned; that surface is the maintainer's in-flight redesign, not contributor territory. This PR delivers the state machine and its observability through the exact read path that dashboard will poll — the acceptance test below is worded directly from the spec's own test deliverable: "a provisioning failure transitions the record to failed and is observable via the same read path a customer's dashboard uses." The state→copy mapping is preserved verbatim in the lifecycle type's doc comment for that follow-up to consume.

Closes #7677

Scope

• The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
• This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
• This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
• I linked a currently open issue this PR resolves (e.g. Closes #123) — a linked open issue is required for every contributor PR.

Validation

• git diff --check
• npm run actionlint
• npm run typecheck
• npm run test:coverage locally; codecov/patch requires ≥99% coverage of the lines AND branches you changed (aim for 100% on your diff so CI variance does not fail near the threshold). Global coverage is a non-blocking trend with a loose 90% backstop, not the gate.
• npm run test:workers
• npm run build:mcp
• npm run test:mcp-pack
• npm run ui:openapi:check
• npm run ui:lint
• npm run ui:typecheck
• npm run ui:build
• npm audit --audit-level=moderate
• New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries

If any required check was skipped, explain why:

• The diff is entirely inside the self-contained control-plane/ package (own npm package, own build, own node:test suite, own Codecov flag), so the checks that actually exercise it were run in full, not the root chain whose suites never import it:
  • npm --prefix control-plane test → build + 187/187 tests pass (182 existing + the 5 added below).
  • npm --prefix control-plane run cf:typecheck (the Worker tsconfig) → clean.
  • npm run control-plane:coverage → package 99.94% lines / 99.05% functions; empirically intersecting lcov.info with this diff's changed lines: 48/48 changed instrumented lines covered, zero uncovered branch arms — patch = 100% lines and branches (measured per line/arm with max-merged DA/BRDA records, not inferred from file totals).
  • npm audit --audit-level=moderate → found 0 vulnerabilities; git diff --check clean.
• The unchecked root-chain boxes (typecheck, test:coverage, workers/mcp/ui/openapi) cover surfaces this PR does not touch — no src/**, packages/**, or apps/** file changes; root tsconfig does not include control-plane/ (it typechecks under its own two tsconfigs, both run above). Also swept every repo-wide consumer of TenantLifecycleState before widening the union: no exhaustive switch/mapping exists (the miner's tenant-client passes states through verbatim with deliberately non-exhaustive docs), so nothing outside the package can be broken by the new member.

Safety

• No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
• Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
• Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests.
• API/OpenAPI/MCP behavior is updated and tested where needed.
• UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks.
• Visible UI changes include a UI Evidence section below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository.
• Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs.

Unchecked boxes above, and why — all N/A rather than skipped:

• Auth/cookie/CORS/session: no auth behavior changes — the routes keep their existing Bearer/503/401 gating, and the existing auth-branch tests pass unchanged. (The registry's own no-secret-values posture is untouched; the new states carry no payload.)
• UI / UI Evidence: no UI files are touched — the dashboard surface belongs to owner-assigned Redesign the setup-wizard for true multi-instance self-serve provisioning #4926/Decouple "admin of a GitHub install" from "has a running container" #4927, per the Summary's scope line. There is nothing visual to screenshot in a lifecycle-state persistence change.

Notes

• Tests added (control-plane/test/http-app.test.ts, +5, mirroring the existing driver-failure/onError idioms):
  1. The spec's acceptance test, verbatim in intent: a provision failure → 500 (still pages + rethrows), registry shows "failed", and GET /v1/tenants — the dashboard's read path — returns the terminal failed state, not a record stuck at "provisioning".
  2. The polling premise: with the driver gated mid-standup, the record is observable as "provisioning" while provisioning is in flight, then "active" after release — the transitional state the 3-5s poll watches now actually exists on the wire.
  3. Retry: a "failed" tenant (including one holding an installation-ID claim) is re-creatable — 201 on retry with a healthy driver; before Spec: customer-facing repo-provisioning status surface (provisioning/ready/failed) #7677 a failure left no record, so this pins that persisting failures doesn't regress retryability.
  4. Conflicts unchanged for live tenants: an "active" tenant still 409s both the name+product check and the installation-claim check.
  5. The best-effort seam: a registry that rejects the "failed" write still surfaces the ORIGINAL provisioning error (500 internal_error), with the record left at its pre-written "provisioning" state — the swallowed-rejection arm of the new catch path, covered explicitly.
• State→copy mapping for the Redesign the setup-wizard for true multi-instance self-serve provisioning #4926/Decouple "admin of a GitHub install" from "has a running container" #4927 dashboard follow-up (from the ratified decision, recorded in the lifecycle type's doc comment): "provisioning" → "Setting up your instance" (poll every 3-5s) · "active" → "Ready" (terminal, stop polling) · "failed" → "Setup failed" (terminal, stop polling) · "suspended"/"torn down" → post-provisioning operator states, out of the provisioning flow's scope.

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.51%. Comparing base (ad1877e) to head (f80c060).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8154      +/-   ##
==========================================
- Coverage   92.04%   91.51%   -0.54%     
==========================================
  Files         765      765              
  Lines       77668    77701      +33     
  Branches    23472    23474       +2     
==========================================
- Hits        71493    71111     -382     
- Misses       5062     5524     +462     
+ Partials     1113     1066      -47     

Flag	Coverage Δ
control-plane	99.84% <100.00%> (+<0.01%)	⬆️
shard-1	51.48% <ø> (-0.65%)	⬇️
shard-2	53.61% <ø> (-0.54%)	⬇️
shard-3	55.81% <ø> (-0.77%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
control-plane/src/http-app.ts	100.00% <100.00%> (ø)
control-plane/src/provisioning.ts	100.00% <100.00%> (ø)
control-plane/src/tenant-provisioning-driver.ts	100.00% <100.00%> (ø)

... and 3 files with indirect coverage changes

[loopover-orb]

Tip

✅ LoopOver review result - approve/merge recommended

_{Review updated: 2026-07-23 06:35:22 UTC}

4 files · 1 AI reviewer · no blockers · readiness 100/100 · CI green · clean

✅ Suggested Action - Approve/Merge

• safe to merge

Review summary
This PR adds a persisted "failed" lifecycle state, pre-writes a "provisioning" record before the slow standup begins (so GET /v1/tenants can observe it mid-flight), and adds a best-effort onFailure seam in provisionTenant that flips the record to "failed" before the existing page-and-rethrow. isRecreatableState correctly extends the pre-existing "torn down" retry allowance to "failed" at both conflict sites (name+product and orbInstallationId), while an active tenant still 409s, and the failure-write itself is swallowed so a KV outage can never mask the real provisioning error. I traced the write-order end to end (pending upsert before driver calls, onFailure closure capturing the same `pending` object, final upsert overwriting with the terminal state) and it's correct and well covered by the new tests, including the mid-flight-poll and swallowed-failure-write scenarios.

Nits — 4 non-blocking

• control-plane/src/http-app.ts:238-244 (POST /v1/tenants/rollout) still only excludes `state === "torn down"` from pinning — a newly-"failed" tenant (no container ever stood up) can now be pinned by rollout, which is likely meaningless; worth a follow-up to also exclude `"failed"` there for consistency with this PR's own conflict-check pattern.
• control-plane/src/http-app.ts: the get-then-upsert conflict check for name+product and orbInstallationId is still a classic check-then-act race under concurrent POSTs (pre-existing, not introduced here) — not a regression from this diff, just worth noting since the window is now slightly larger (two upserts instead of one).
• control-plane/src/http-app.ts:238 — add `failed` alongside `torn down` in the rollout torn-down guard for symmetry with `isRecreatableState`.
• Consider a short comment on `pending` (http-app.ts) clarifying it's intentionally reused unmutated inside the `onFailure` closure, since that's the crux of why the failed-state write can't drift from the provisioning-state write.

Decision drivers

• ✅ Code review — No blockers (1 reviewer)
• ✅ Gate result — Passing (No configured blocker found.)

Context & advisory signals — never blocks the verdict

Signal	Result	Evidence
Linked issue	✅ Linked	#7677
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 53 registered-repo PR(s), 19 merged, 33 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	cleanjunc; Gittensor profile; 53 PR(s), 33 issue(s).
Improvement	✅ Minor	risk: clean · value: minor · LLM: significant

Linked issue satisfaction

Partially addressed
The PR fully implements the backend deliverables: adding "failed" to TenantLifecycleState, transitioning provisionTenant's catch block to persist the failed state before rethrowing, pre-writing "provisioning" so it's observable during standup, and thorough tests confirming failures are observable via the GET /v1/tenants read path. However, the issue's second deliverable explicitly requires the cus

Review context

• Author: cleanjunc
• Role context: outside_contributor
• Public audience mode: oss maintainer
• Lane context: Repository is configured for direct PR review.
• Public profile languages: not available
• Official Gittensor activity: 53 PR(s), 33 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Keep the PR focused and include validation evidence before maintainer review.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

• @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
• @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
• A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

• Re-run LoopOver review

[loopover-orb]

LoopOver approves — the gate is satisfied and CI is green.