docs(server): document provider turn startup recovery by patroza · Pull Request #249 · patroza/t3code

patroza · 2026-07-31T09:45:31Z

What Changed

add an operator runbook for diagnosing and recovering stuck provider turns
document the global command-worker head-of-line failure mode and startup reconciliation behavior
record the 2026-07-31 incident pattern, safe SQLite diagnostics, and normal/emergency systemd recovery
define prevention priorities and acceptance tests for timeouts, per-thread isolation, sweeping, and readiness
cross-link the runbook from the docs index and composer lifecycle analysis

Why

One wedged provider session start can currently block unrelated turn starts while the process and HTTP listener remain alive. Recovery is reliable only through service restart and startup replay. This documents the present operator path and the product work needed to make recovery automatic.

Downstream Fork Relationship

Base: fork/changes. Affected area: documentation only. No dependency on another open PR. Deployment-specific hostnames, credentials, monitoring, and restart automation remain assigned to the private operations repository.

Verification

git diff --check
vp check (zero errors; existing repository warnings only)
ELECTRON_SKIP_BINARY_DOWNLOAD=1 vp run -r --cache --log labeled typecheck

Checklist

This PR is small and focused
I explained what changed and why
UI screenshots are not applicable
Animation/interaction video is not applicable

Add custom "Open with" applications Source: tim-smart#4 Source head: 8c4bdfbc5b57f6b600233244d330f9efa41dc498 Source commits: 08e1a4fb949585c3c441d6d00455fe904f72cd7b,cd43a401c6c148f1fe26cff72104ac527ea189f3,a8370e7502c552ebb064436e42e1c00f86f0946b,8c4bdfbc5b57f6b600233244d330f9efa41dc498 Imported: complete product delta from the source PR. (cherry picked from commit 9fae005)

Load direnv environments for provider sessions Source: tim-smart#5 Source head: 8f5fc87c13f4628c179cda44d4f32f7fe4d316b2 Source commits: e4f07014d39964fde2498bcb35588974cc5e6232,0d1463af61e0bd174f698b2519ebf3b207a2eaca,a66e4160d5f4b79140ec8fbcbc6aa66af750a991,8f5fc87c13f4628c179cda44d4f32f7fe4d316b2 Imported: complete product delta from the source PR. (cherry picked from commit 0da8bfe)

Add unsigned retry for commit signing failures Source: tim-smart#6 Source head: 7d65c5a224e97a6b811b0a84892f1fda065c5963 Source commits: 18ee567ecfdb11c9372153127b26b5cf57213a76,72a6fae23c86708080c4fed346d5bf0f136f0221,6614b28239ed2330a8f601357a413f2d50da195a,ec169369daa554541511aa28f551b36f3dd26485,7d65c5a224e97a6b811b0a84892f1fda065c5963 Imported: complete product delta from the source PR. (cherry picked from commit 03671a2)

Add /new command for contextual threads Source: tim-smart#7 Source head: 2051a8003041fe2806fcb4bc7a0d8940579fc543 Source commits: 2051a8003041fe2806fcb4bc7a0d8940579fc543 Imported: complete product delta from the source PR. (cherry picked from commit 4d94f31)

Add session dashboard board Source: tim-smart#8 Source head: d9f8e4d0a8dc22231ca315f3c595c3597f3b13e5 Source commits: 268fb8df9863ffbda51b975a8dbe68f11c41500c,dde20f271f674da22dd8f3a08201c2acf5e58ee5,df4a145e7b2cd2dc17a7a595267d2d8eb0a2a3f0,ce5723ddb0bf630a18d4cb8227b5344d12626e72,ad8c1a6af41161e1fc38a52f681b306517c7b918,6281887e6125317da0c7b4252d59bfd41c9bf35e,550db6316c634febdbe1cb27334d1347c23c7b2a,d9f8e4d0a8dc22231ca315f3c595c3597f3b13e5 Imported: complete product delta from the source PR. (cherry picked from commit cd0e281)

Recover interrupted provider turns after server restarts Source: tim-smart#9 Source head: b181832560177250b90bbfe07b0882c9e5b93493 Source commits: 7f69028a25be21f1882ecba14b62f387ad60cf2a,1d52bce1376766d804ef884d7d50b8b6d1b48cf7,b181832560177250b90bbfe07b0882c9e5b93493 Imported: complete product delta from the source PR. (cherry picked from commit 83de8f5)

Avoid repeated thread snapshot loads during subscription retries Source: tim-smart#10 Source head: c8c9eadb9de3026706bc3a403ca05b12d0da8dd5 Source commits: c8c9eadb9de3026706bc3a403ca05b12d0da8dd5 Imported: complete product delta from the source PR. (cherry picked from commit 9e400c3)

…fixes Identity product layer based on current fork/changes: session claims, source attribution, participant UI (#247), and Jira queue/inline replies (#248), with mobile typecheck fixes for stage label and toolbar clearance. (cherry picked from commit d3872ea747f7e0cce78c26668ff939759cccff13)

) Mine/Ours now keeps threads with no person tags (legacy, channel-only stamps like desktop, identity-disabled servers). Theirs is only threads that have person attribution excluding the session claim. Co-authored-by: omegent-app[bot] <306514130+omegent-app[bot]@users.noreply.github.com> Co-authored-by: Patrick Roza <42661+patroza@users.noreply.github.com> (cherry picked from commit 31ea163eccd6d37c52e3d064133262da81d1424d)

…pply Identity product reapply had dropped the mobile model-selection helper while new-task-flow still imported it, breaking mobile typecheck. (cherry picked from commit 1080cb3cd9f50966becaed898a4bc04c6dd71915)

…#256) Remember Mine/Theirs across mobile restarts via device preferences, and let Mine/Theirs refine by created, participated, or both (default). Co-authored-by: T3 Code PR Stack <41898282+github-actions[bot]@users.noreply.github.com> (cherry picked from commit 50a7ffd2d4c28a1b50ceb36d2cebd432290c27ad)

Created looked only at origin, so Theirs+Created included threads you joined (origin ≠ you) while Theirs+Both required you to be absent from every person tag — Created could show more than Both. Classify involvement first, then narrow by role so Both is always the superset. Co-authored-by: T3 Code PR Stack <41898282+github-actions[bot]@users.noreply.github.com> (cherry picked from commit 35af60589c25f51352a3f3221dd7192cd9719f65)

Keep mobile-showcase landscape harness typing from fork/changes when rebasing the identity overlay; identity reapply had dropped the field. (cherry picked from commit 59dd846e757ed1a406557432b6270c67347d9f7e)

Keep checkForAppUpdateOnLaunch wired after product-merging snooze actions into the identity HomeRouteScreen. (cherry picked from commit 00fd93a46a87ee9bad39a4f0b323b6bcd97ee025)

Production showcase deep links use t3code://; identity reapply left t3code-dev:// expectations that fail after restack compose. (cherry picked from commit 4a3b314800b89ae9e7165a76d29fa78bb0463fdd)

Jira and GitHub treat an unset/empty identity map like an unmapped actor: no agent turns. Only mapped people may drive the host via those webhooks. Co-authored-by: omegent-app[bot] <306514130+omegent-app[bot]@users.noreply.github.com> Co-authored-by: Patrick Roza <42661+patroza@users.noreply.github.com> (cherry picked from commit 8be7e5ba7ad70c08f23bb1f0424b83826210249f)

The header said claims were a process-local Ref with "Persistence later". `layerPersisted` has since made server claims SQLite-backed (SessionIdentityClaimRepository + migration 037), with the Ref demoted to a read-through cache. Only the residual-free `layer` used by the CLI and tests is still memory-only. Co-authored-by: T3 Code PR Stack <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 26e478f938afd102763b932c00226278fb91f9df)

The server read T3_IDENTITY_MAP_PATH once at layer construction, so adding or removing a person meant restarting t3code-server. The map is now re-checked on a 60s TTL and applies in place. Polls rather than watches: the map arrives over virtiofs from the host, where inotify propagation is not something to depend on. An ino/size/mtime fingerprint keeps an untouched file from being re-parsed every TTL. Two safety rules, because a reload can now fail in production where startup could not: - A re-read that yields no people never disables an already-enabled map. `enabled === false` turns the operate gate off entirely, so a truncated or unparseable file would have failed open. The last good map keeps serving, marked unhealthy, and the next TTL retries. - requireOperateClaim no longer deletes the persisted claim of a person who is absent from the map. A half-written file can still parse as a valid map with a subset of people, and that delete is not reversible. Refusing operate is the gate; membership is re-checked on every operate, so a stale row grants nothing. Startup behaviour is unchanged: a missing or empty map still means the feature is off, and removing T3_IDENTITY_MAP_PATH remains the way to disable the gate. Co-authored-by: T3 Code PR Stack <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 3b2203f961ac31ba10ac7289b0a9cef6100441f2)

…292) Port the Board Ownership filter (Anyone/Mine/Theirs + relation sub-filter) into classic Sidebar v1, sharing localStorage with v2. Default the filter to Mine on web v1/v2 and mobile so the inbox starts on your work; unattributed threads still count as Mine. Co-authored-by: omegent-app[bot] <306514130+omegent-app[bot]@users.noreply.github.com> Co-authored-by: Patrick Roza <42661+patroza@users.noreply.github.com> (cherry picked from commit aa2b21f364ff1e4b4822e495059b69eddc7e27d4)

Co-authored-by: T3 Code PR Stack <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Patrick Roza <42661+patroza@users.noreply.github.com>

Identity reapply dropped forward-compat wrappers for config issues and available editors while keeping ServerProviders on ForwardCompatibleArray.

fork/dev had no CI path at all: fork-ci.yml listed only the rebased stack layers and the registered overlays as pull_request bases, and had no push trigger anywhere. That blocks both halves of the cutover — PRs into fork/dev could not satisfy a required check, and no run would ever exist for a merge SHA. Add fork/dev as a pull_request base, and add a push trigger for it. The push run matters because fork/dev is never rebased, so its merge commits are the release candidates: deployment promotes an exact green SHA and keys on a successful run of this workflow for that SHA. A green PR tip is not enough when the merge SHA differs. Push runs are keyed by SHA and exempt from cancel-in-progress. Cancelling one as superseded would leave that merge SHA permanently unapprovable for deployment, which is not a state the poller can recover from on its own. Extend the release-tip jobs the same way. deployment_scope and dispatch_mobile_releases were gated on workflow_dispatch + fork/integration, so after the cutover mobile releases would have stopped silently — nothing errors, EAS simply never gets dispatched again. Both now accept a fork/dev push as well. The previous-successful-CI lookup that feeds classification hardcoded fork/integration + workflow_dispatch. Left alone it would diff every fork/dev push against an unrelated tip and classify every component as changed, so it now follows the running branch and event. The mobile dispatch likewise targets the branch being validated instead of a fixed fork/integration ref, so the mobile workflows come from the same tip that passed. Both fork/integration paths are retained unchanged; the two tips coexist until fork/integration is retired. Co-authored-by: omegent-app[bot] <306514130+omegent-app[bot]@users.noreply.github.com> Co-authored-by: Patrick Roza <42661+patroza@users.noreply.github.com>

@patroza

Found by the first real `fork/dev` merge. [#343](#343) merged, the push run went green, `Dispatch Mobile Releases` fired — and then EAS production [failed](https://github.com/patroza/t3code/actions/runs/31074271669) at **Resolve approved integration source**. ## Cause Both mobile workflows check out a hardcoded ref and then assert containment: ```yaml ref: fork/integration # <- hardcoded ... git merge-base --is-ancestor "$target_sha" "$integration_sha" ``` A `fork/dev` SHA is not contained by `fork/integration`, so the assertion rejected it. Same class of hardcoding #343 fixed on the dispatch side — just one workflow further along, and only observable once a real `fork/dev` merge dispatched a release. ## Change - `release_branch` input on both mobile workflows, **defaulting to `fork/integration`** so any manual dispatch that omits it behaves exactly as before. - Checkout uses `${{ inputs.release_branch }}`; the "overlay deploy tooling" condition compares against it instead of the literal. - `fork-ci.yml` passes `release_branch` alongside the `--ref` it already passed. Passing one without the other is the trap worth naming: the workflow file would come from `fork/dev` while the product checkout stayed on `fork/integration` — precisely the failure above. ## Validation - All three workflow files parse; `release_branch` default confirmed as `fork/integration`. - `vp fmt --check` clean across `.github/workflows/`. - **The EAS path itself is not re-run by this PR.** Proof is the next `fork/dev` merge that classifies `mobile=true` — this PR's own merge should do it, since it touches `.github/workflows/**`. Worth watching that run rather than assuming. ## Note on the first dispatch That run classified every component as changed because no previous successful `fork/dev` push run existed to diff against — the documented conservative fallback. Subsequent merges diff against the prior green `fork/dev` SHA and should scope normally. Co-authored by [@patroza](https://github.com/patroza) opened by [Patrick Roza](https://discord.com/users/95218063095377920) in chat thread **Discord** · [Discord](https://discord.com/channels/1083767712431480922/1534783738322485399/1534783738322485399) · [T3](https://t3vm/?thread=584a9ad3-243e-4308-8a13-49acdd758b17) Co-authored-by: omegent-app[bot] <306514130+omegent-app[bot]@users.noreply.github.com> Co-authored-by: Patrick Roza <42661+patroza@users.noreply.github.com>

@patroza

First run of the provenance synchronization from [#342](#342) — and the answer to "get the latest upstream onto `fork/dev`". Five upstream commits enter the product: | | | | --- | --- | | `a2ca89aa` | feat: native subagent & workflow observability (pingdotgg#5219) | | `990bb0b6` | fix: reconnect faster after remote server updates (pingdotgg#5404) | | `7251f1a1` | Prevent terminal loading flash (pingdotgg#5432) | | `30e47153` | fix(web): preserve terminal font size when splitting (pingdotgg#5444) | | `de592a00` | Enrich terminal font previews (pingdotgg#5428) | 54 files, +7,055 / −175. ## Checkpoint ```json { "importedCandidatesCommit": "9655a9ba955197361044ef6f8f97e35841ff779e", "importedCandidatesTree": "50f9bfab717c30a8ea90d52060349e209502d116", "importedUpstreamCommit": "a2ca89aa10f13a2222e08afd98c66285121d5ba2", "previousCandidatesTree": "9b4cd3e1c774c3c436e43305c151edf596b2936a" } ``` `previousCandidatesTree` is C1 from tag `fork-dev/2026-08-06.1`. Tag the merge commit `fork-dev/2026-08-06.2` with the values above once this lands. ## Two resolutions worth reviewing Most of the 7 conflicted files are independent additions on both sides and resolve as unions — upstream's `backgroundLiveness` beside identity's `originSource`/`participantSummaries`, upstream's agent-spawn CTA rows beside the imported user-input Q&A timeline. Two were not unions: **`apps/web/src/components/Sidebar.logic.ts`** — the 3-way merge welded upstream's `hasPlanReadyPrompt` condition onto the `"Wake Required"` return body. Left alone, a plan-ready thread would render as **Wake Required** and no thread could ever reach **Plan Ready**. Both branches are restored with their own bodies. The status rank map is merged onto upstream's new scale with `Wake Required` at the `Working`/`Connecting` tier — its relative position before the import. **That tier placement is a judgement call; say if you want it ranked differently.** **`apps/mobile/src/lib/threadActivity.ts`** — upstream's `isAgentInternalActivity` skip guard must run *before* identity's resolved-user-input enrichment. The other order enriches and pushes exactly the agent-internal rows upstream means to drop. ## Validation - All 12 `fork/candidates` commits replayed onto the rebuilt `fork/tim`; `a2ca89aa1` confirmed an ancestor of the new candidates tip. - No residual conflict markers; brace balance checked on the hand-edited files. - **Typecheck and tests have not run locally** — the rebase workspace has no `node_modules`. Fork CI on this PR is the first real verification. Do not merge on the strength of this description. ## Provenance branches not yet pushed `fork/base` → `4a73589` and `fork/tim` → `b1c5fa5` are rebased and `fork/candidates` → `9655a9ba` is rebuilt, but all three are **local only**. The ruleset *Protect fork/tim, candidates, integration* sets `non_fast_forward` with no bypass actor and the app token has no `administration` scope, so I cannot force-push them. Until they are pushed, `importedCandidatesCommit` refers to a commit that exists nowhere on the remote. The tree is what the delta depends on, but the checkpoint is not fully honest until that push happens. Co-authored by [@patroza](https://github.com/patroza) opened by [Patrick Roza](https://discord.com/users/95218063095377920) in chat thread **Discord** · [Discord](https://discord.com/channels/1083767712431480922/1534783738322485399/1534783738322485399) · [T3](https://t3vm/?thread=584a9ad3-243e-4308-8a13-49acdd758b17) Co-authored-by: omegent-app[bot] <306514130+omegent-app[bot]@users.noreply.github.com> Co-authored-by: Patrick Roza <42661+patroza@users.noreply.github.com>

@patroza

The first `fork/dev` sync merge (`ae4719b1c`, #345) passed **every required check** and still ended as a **failed run**: ``` Check ✅ Test ✅ Mobile Native Static Analysis ✅ Release Smoke ✅ Classify Deployment Scope ✅ Dispatch Mobile Releases ❌ HTTP 422: Unexpected inputs provided: ["release_branch"] ``` Because the deploy poller only promotes a SHA whose `fork-ci` run **concluded success**, that one failure blocked server, Discord, desktop and VS Code deployment of `ae4719b1c` entirely. The guest is still sitting on `21badd04e`. ## Cause [#344](#344) added the `release_branch` input **declaration** to `mobile-eas-production.yml`, but for `mobile-eas-development.yml` it only rewrote the *usages* — leaving the file referencing `inputs.release_branch` without declaring it. Production dispatch succeeded; development was rejected. The validation in that PR printed the default for production and nothing for development. That was the evidence, and it was read past. ## Change Declare the input with the same `fork/integration` default, so a manual dispatch that omits it behaves exactly as before. ## Validation Rather than eyeball it again, all three workflows are now checked for `inputs.*` references with no matching declaration: ``` fork-ci.yml | declared: checkout_ref | undeclared: none mobile-eas-production.yml | declared: mode,platform,message,runtime_version,sha,release_branch | undeclared: none mobile-eas-development.yml | declared: platform,runtime_version,sha,release_branch | undeclared: none ``` ## Separate design question, not fixed here **Should a release-dispatch failure invalidate a validation verdict?** `Dispatch Mobile Releases` performs a *release action*; the other five jobs *validate the SHA*. Mixing them in one run means any dispatch hiccup — a 422, a transient API error — marks the SHA unapprovable and stalls the whole fleet, which contradicts the handover doc's own rule that per-target release status is recorded independently and that one target's failure must not hold back the others. Two options, both one-liners: - `continue-on-error: true` on `dispatch_mobile_releases` — run concludes success, the failed job stays visible, mobile status is tracked by the EAS workflows anyway. - Move the dispatch into its own `push`-triggered workflow so `fork-ci`'s conclusion means "this SHA is valid" and nothing else. I did not apply either, because weakening the deploy-approval signal is a policy call. Say which you want and I'll do it. Co-authored by [@patroza](https://github.com/patroza) opened by [Patrick Roza](https://discord.com/users/95218063095377920) in chat thread **Discord** · [Discord](https://discord.com/channels/1083767712431480922/1534783738322485399/1534783738322485399) · [T3](https://t3vm/?thread=584a9ad3-243e-4308-8a13-49acdd758b17) Co-authored-by: omegent-app[bot] <306514130+omegent-app[bot]@users.noreply.github.com> Co-authored-by: Patrick Roza <42661+patroza@users.noreply.github.com>

The C1..C2 tree delta in #345 imported the content of upstream commits 2a04db1..a2ca89a but not their commit objects, so GitHub measured fork/dev as 5 commits behind pingdotgg/t3code:main while being fully current. -s ours keeps the tree byte-for-byte and records only the parent link, which is honest because #345 already landed the content and its checks passed. Recorded in tag fork-dev/2026-08-06.2 as importedUpstreamCommit. Co-authored-by: Patrick Roza <42661+patroza@users.noreply.github.com>

patroza force-pushed the fork/changes branch 22 times, most recently from a3c8507 to 364f3ad Compare August 5, 2026 07:55

github-actions Bot and others added 8 commits August 5, 2026 15:48

chore(fork-base): fork workflows, stack tools, and docs

0c4fd9a

github-actions Bot and others added 24 commits August 5, 2026 16:46

fix(identity): keep resolveSelectableModelSelection after overlay rea…

b7ca076

…pply Identity product reapply had dropped the mobile model-selection helper while new-task-flow still imported it, breaking mobile typecheck. (cherry picked from commit 1080cb3cd9f50966becaed898a4bc04c6dd71915)

fix(identity): restore showcase iOS orientation field after restack

66ea845

Keep mobile-showcase landscape harness typing from fork/changes when rebasing the identity overlay; identity reapply had dropped the field. (cherry picked from commit 59dd846e757ed1a406557432b6270c67347d9f7e)

fix(identity): restore app-update launch import after restack

d90796f

Keep checkForAppUpdateOnLaunch wired after product-merging snooze actions into the identity HomeRouteScreen. (cherry picked from commit 00fd93a46a87ee9bad39a4f0b323b6bcd97ee025)

fix(identity): align showcase URL tests with production scheme

df3106c

Production showcase deep links use t3code://; identity reapply left t3code-dev:// expectations that fail after restack compose. (cherry picked from commit 4a3b314800b89ae9e7165a76d29fa78bb0463fdd)

fix(identity): attribute commits from every client (#314)

f9efba3

Co-authored-by: T3 Code PR Stack <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Patrick Roza <42661+patroza@users.noreply.github.com>

fix(identity): dedupe connector attribution (#315)

d82a8e7

Co-authored-by: T3 Code PR Stack <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Patrick Roza <42661+patroza@users.noreply.github.com>

fix(identity): restore ForwardCompatibleArray after restack

aa830e8

Identity reapply dropped forward-compat wrappers for config issues and available editors while keeping ServerProviders on ForwardCompatibleArray.

fix(identity): move migrations to fork ledger

803d3a6

test(server): cover deployed migration ledgers

4f113db

chore(integration): regenerate pnpm-lock.yaml after overlay compose

21badd0

docs(server): add provider turn recovery runbook

e6e3429

omegent-app Bot changed the base branch from fork/changes to fork/dev August 6, 2026 06:42

omegent-app Bot force-pushed the docs/provider-turn-recovery-runbook branch from bfde6c0 to e6e3429 Compare August 6, 2026 06:42

patroza force-pushed the fork/dev branch 4 times, most recently from 80434dc to fc6701e Compare October 3, 2026 10:00

Navigation Menu

Repository navigation

docs(server): document provider turn startup recovery - #249

docs(server): document provider turn startup recovery#249
patroza wants to merge 95 commits into
fork/devfrom
docs/provider-turn-recovery-runbook

patroza commented Jul 31, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

patroza commented Jul 31, 2026

What Changed

Why

Downstream Fork Relationship

Verification

Checklist

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants