Repository navigation
Interruptions, resumes and the time-limit resume - #81
Merged
Merged
Conversation
Port softfac's "Resuming after a time limit" tests: the launch message paragraphs, the resume decision in start, and the pull request binding of a resumed run in advance and the gate. The fake tracker gains branchTip and textEditedSince. Red on purpose: 10 tests in start.test.ts, 3 in messages.test.ts, 1 in building.test.ts and 1 in gating.test.ts fail because launchMessage takes no resume, start never looks at the previous run, and checkBinding only accepts the session's own branch. The plain-start cases already pass. Issue: #25
When an issue is set to ready again, start looks at the previous run. If it stopped at its time limit, or was relaunched after an interruption, and its branch is a plain ref name, the issue text is unedited since that run's approval, and the branch is on origin, the session launches from that branch at its tip on origin with one fixed paragraph added to the first message. A 404 starts from base; any other lookup failure is an error that launches nothing. The decision is recorded in the launch-sending step and reused when the launch is sent again. session_launched records resumed_from, and a resumed run's pull request may come from its own branch or the one it resumed from, in advance and in the gate. Issue: #25
Port softfac's interruption tests: advance says a session that ended failed or cancelled, or whose thread is gone, ended badly and writes nothing; interrupted decides a resume, a relaunch from the run's branch or a stop; a decided relaunch is finished by advance and skipped by the time limit, the watch and forward; the pass collects the sessions that ended badly, reads the machine log once and decides them together. The pull request feedback tests of a paused thread belong to issue #24. Red on purpose: 48 tests in interruption.test.ts, 12 in pass.test.ts, 7 in machineLog.test.ts, 5 in progress.test.ts and 6 in building.test.ts fail because advance still sends these runs to stuck, interrupted, t3TroubleSince and the interruption lines in Seen do not exist yet, and the pass has no t3Trouble or interrupted. The pause tests that need no new behaviour already pass. Issue: #25
A session whose run ended failed or cancelled with no done line, or whose thread T3 no longer has, is reported by advance as ended badly with nothing written. After the pass has been through every run it reads the machine log once, only when one ended badly, and interrupted decides each with the shared evidence: a T3 error in the last ten minutes, or two or more sessions ended badly in the pass, with resumes left under limits.interruption_resumes, is an interruption. A thread that still exists gets one fixed message under softfac:<run>:interrupted:<n>, recorded by a session_interrupted line before it and the step interruption-sent <n> after it; a send cut short is sent again by advance. A thread that is gone is relaunched from the run's branch: the next attempt's ledger takes the same approval, the issue goes back to triage, the old run finishes as stuck with the reason session-interrupted, and resume launches the next attempt from the branch with the interrupted paragraph. A relaunch cut short is finished by advance from the ledger, and the time limit, the text watch and forward leave a run with a relaunch decided alone. Anything else goes stuck as before, and a stop by hand always wins. Rolled back runs and runs with no ordinal still go stuck at once. Issue: #25
advance and carryOutRelaunch take the one richer context, so a caller that cannot start a session no longer compiles instead of failing at run time. resumesUsed finds the previous attempt through previousRun, the reader continuation already uses. Test rigs that call advance go through a shared helper that adds the claim's fields and fakes that fail loudly if called. Issue: #25
github-actions
Bot
added
vouch:trusted
rawphp
marked this pull request as ready for review
October 16, 2026 22:44
Author
|
Factory gate: passed (attempt 2, run by the factory in a fresh checkout of the pull request's head
|
…one, never relaunched
…at launches nothing
… branch it pushes to
…ough one helper, with its instructor
…a t3 error in the machine log
…hine log, the sign T3 trouble is read from
…at the first entry out of the window
… stop by hand ends a cut-short relaunch's next attempt A T3 restart ends every running turn as `cancelled` and the next status read answers as usual, so no error reached the machine log and the run went stuck. `t3TroubleSince` now also takes the server's own start (`performance.timeOrigin`, passed to `liveWorld`) as a sign within the ten-minute window, newest wins. Only a sessions call that is T3 itself sets `t3Down`: a failed `remove-worktree` is git on this machine. A `MessageRejectedError` never did, and a test now pins that. When a relaunch was decided and cut short after the next attempt's ledger was made, and the operator then stopped the run, `advance` now finishes that unbegun ledger too, so it no longer counts as an active run holding a slot. A next attempt that already began is left alone. Review-gate round 4: f11, f12, f14, f15.
… launch failures are not T3 trouble, and a stop by hand on a begun relaunch writes nothing on the issue A server start counted as T3 trouble for every session that ended badly in the ten minutes after it, so a run a person cancelled after a redeploy was resumed. The start is now part of the evidence the pass hands `interrupted`, which counts it only for a run whose T3 run started before the server did and ended after it: the shape only a restart's recovery leaves. The status now carries the run's start and end from the projection. A thread that is gone never counts the start. `t3TroubleSince` reads the log alone again. `liveWorld` takes the server's start as a required argument (`serverStartedMs()`), so a refactor cannot drop it silently. Only a sessions call on T3 itself (find-project, status, thread, send, archive) sets `t3Down`: a launch also fails on the worktree, origin and bad ids. A stop by hand on a run whose relaunch had already started the next attempt finished the run through the stuck path, labelling the next attempt's issue stuck. It now finishes the run as the relaunch would have, with nothing written on the issue. Review-gate round 5: f14, f16, f17, f18.
…t by any of them is an interruption The restart evidence was this process's start alone, compared with a run's end as "at or after": a graceful stop marks the turn cancelled seconds before the next start, and a second restart before a pass decided the run lost the evidence. The server now writes a `server_started` line to the machine log at boot (`serverStartedMs()`, the process start), the pass reads every start of the last day with the T3 trouble, and `interrupted` names the start the run spanned: begun before it, ended at or after it or within ten minutes before it. A thread that is gone still never counts a start. A stop by hand on a run whose relaunch already started the next attempt now carries the operator's `stop_requested` into that attempt's ledger, so the next pass stops the issue's work instead of only finishing the old run. The set of sessions operations that are T3 trouble is typed over `SessionsError["operation"]`, and every member is pinned to reach the machine log. Review-gate round 6: f19, f20, f21, f22.
…a server that stayed down is still an interruption A graceful stop marks the running turn cancelled as the old process shuts down; with only the next start on record and a ten-minute grace, a server unloaded for longer sent that run to stuck. The factory now writes a `server_stopping` line as its scope closes, the pass reads the stops with the starts, and `interrupted` pairs each start with the newest stop after the start before it: a run that began before the start and ended at or after that stop (with the grace before it) was cut by the restart. A start with no stop line keeps the grace alone. The design doc row now names the grace and the one-day window. Review-gate round 7: f23, f24.
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #25
Contract
failed/cancelledrun, or a thread T3 no longer has, is an interruption when the machine log has a t3 error in the last 10 minutes or two or more sessions ended badly in the pass, and resumes are left (limits.interruption_resumes, carried over a relaunch): resumed undersoftfac:<run>:interrupted:<n>with softfac's fixed message, or relaunched from its branch into the next attempt when the thread is gone; otherwise stuck with the old words;rolled_backalways stuck. (2) Aninterruptedrun stays paused and is never decided as an interruption. (3) After a time-limit stop the next attempt resumes from the previous run's branch when it is a plain ref name, on origin, and the issue text is unchanged since approval:launch-sending resumed-from <branch>,session_launched.resumed_from, the fixed paragraph; a resumed run accepts a pull request from that branch. (4) softfac'sinterruption.rs, the relaunch cases ofstop.rs, the pass-leveldecide_interruptionscases and the time-limit-resume cases oftests.rsare ported and green.commits_aheadcounting,branch_bound/pr_adoptedlines, the Factory page, any upstream file.vp test run apps/server/src/factory/runner apps/server/src/factory/pass.test.ts apps/server/src/factory/index.test.ts→ pass (702 tests); whole factory suitevp test run apps/server/src/factory→ pass (1636 tests)Acceptance criteria
failedorcancelledrun counts as an interruption under softfac's rules, is resumed withsoftfac:<run>:interrupted:<n>or relaunched from its branch when the thread is gone, up tolimits.interruption_resumes—runner/interruption.test.ts("resumes a cancelled session after a T3 timeout too", "resumes two sessions that ended badly in one pass without any T3 trouble", "after a T3 outage is relaunched from its branch and not stuck", "the cap is two resumes and the third ended badly goes stuck", "the counter carries over a relaunch");pass.test.ts("is decided with the T3 error from the machine log", "still decides two sessions when the machine log cannot be read").interruptedis paused by hand and waits for an instructor comment —runner/interruption.test.ts("does not decide a paused run as an interruption even with T3 trouble", "lifts the pause with an instructor's issue comment through forward").runner/start.test.tsdescribe "start: the resume after a time limit" ("starts from base when the branch is not on origin", "is not made when the issue text changed since the earlier approval", "is not made from a branch name that could be taken for a git option");runner/building.test.ts("is accepted from the branch the run resumed from").src/runner/tests/interruption.rsandstop.rsinrunner/interruption.test.ts, the pass-level cases inpass.test.ts, the machine-log window cases inledger/machineLog.test.ts, 17 time-limit-resume cases fromtests.rsinstart.test.ts,messages.test.ts,building.test.tsandgating.test.ts. Eachtest:commit was run at its own sha: red (13 and 67 failures), then green at thefeat:commit that follows it.Gate
factory/scripts/check.sh→ exit 0 on87d7cb9c2(1636 tests) and again on the merged tip4f13a1bdd(1758 tests): seam check,vp test run apps/server/src/factory,vp run --filter t3 typecheck,vp run --filter @t3tools/web typecheckModels
modelsblock in.dw/config.yml, so the default roster)Review
advanceandcarryOutRelaunchtakeInterruptContextinstead of a runtime duck-type guard;resumesUsedreads the previous attempt throughstart.tspreviousRun, the one readercontinuationalready uses.Notes
origin/main(fix(factory): build.sh smoke test passes on a clean build #72, Forward instructor comments and edits into a running session #73); it was fast-forwarded tob6c45aba7at Isolate before any write.apps/server/src/factory/.origin/mainmerged Gate again when a pr-open pull request's head or base moves #23 (the pr-open re-gate) while this was in review; the factory's gate could not run on the conflict.origin/mainwas merged in with one merge commit: oneResumeFromunion now serves the time-limit, interrupted and failed-regate launches, both sides'Seenfields and tests are kept, and the interruption relaunch rule (relaunchDecided) stays separate from the re-gate'spendingRelaunch, as in softfac.branch_bound/pr_adoptedhandling, the workspace commit count on the stuck comment, and the paused-run pull-request feedback reads (Relaunch a session for instructor review feedback #24).Lessons
git log -2 -pfirst; build on it"; each slice gets its own redtest:and greenfeat:pair.node_modules, no install. Symptom: proving atest:commit red needs the suite at an old sha, andvp ithere costs two minutes per sha. Default:git worktree add --detach <worktree>/.factory/red-check <sha>, symlink the run worktree's rootnode_modulesand eachapps/*andpackages/*node_modulesinto it, runvp test run <files>,git checkoutthe next sha in place, and remove the worktree from outside it when done.Commits
72b556542test: the resume after a time limit— Make slice A (red)3f6babf72feat: resume a new run from the branch a time-limited run left— Make slice Ab888a03e0test: interruptions and relaunches— Make slice B (red)02f7a6302feat: interruptions and relaunches— Make slice B87d7cb9c2fix: type the relaunch path and read the previous attempt in one place— Check cycle 14f13a1bddMerge origin/main into t3/factory-issue-25— Ship: the base moved (Gate again when a pr-open pull request's head or base moves #23), gate re-run greenFiles
advancereports ended-badly with nothing written;interrupteddecides a resume, a relaunch or a stop;carryOutRelaunch; relaunch-decided skips inwatch,forward,timeLimit;checkBindingaccepts the resumed-from branchresumeFrom,continuation,previousRun,isPlainRefName; the launch from the branch on origin with the recorded decision and the fixed paragraphSeengains the interruption, relaunch-decided, resumed-from and paused-run facts, and the interruption wait markerinterruptedandt3Troubleadvancetakesrunner/interruption.test.ts