SPLIT 2026-09-01 — do not implement this issue directly

Split into three slices at the maintainer's request. This issue stays open as the tracker; the work happens in the slices.

slice	issue	depends on
1 — emit structured per-run timing	#1658	—
2 — report the distribution + near-miss alerting	#1659	#1658
3 — checkpoint-push before the budget expires	#1660	independent

#1660 is the highest-value slice and needs nothing else first. The expensive part of a timeout is not the timeout — it is that all the work dies with the ephemeral runner. Knowing the ideal budget would not have saved any of the three timeouts, because each consumed its entire budget.

Split because this issue as written asks for measurement infrastructure across a broad surface, and releasing it whole would very likely produce a fourth timeout — the same failure it exists to study. #1609 and #1047 both landed in ~7 minutes once narrowed to a single slice.

Do not tune ACTION_TIMEOUT_SEC before #1659 produces evidence. See the correction below: "one second over budget" is a reporting artifact, not a near-miss.

Maintainer decision 2026-09-01: rather than raise ACTION_TIMEOUT_SEC blind or keep splitting issues by hand forever, measure this behaviour and tune it on evidence.

The trigger

.github-private vars.ACTION_TIMEOUT_SEC = 2700. Issue #1609 hit the action budget at elapsed 2701s against a 2700s budget — one second over — and escalated dev-lead:needs-human with reason=timeout (run 33433995544).

A timeout is treated as first-class and non-retryable (correctly — a same-budget retry just times out again). But the cost is total:

the engine had produced changes before the timeout. They were on branch dev-lead/issue-1609-20260831-2004 on the runner and were not pushed, so they are not recoverable from this run (the runner is ephemeral).

So a run that was ~99.96% of the way through the budget threw away all of its work and cost a full human re-scoping cycle.

This is the second occurrence, so it is a pattern, not an incident

• review-one-pr: metadata-digest re-review path has no atomic claim — concurrent triggers can both launch the cascade #1589 — re-scoped after a timeout (recorded in the prior session's ledger).
• Follow-up to #1567: threads resolved with NO commit at all — the shipped evidence bar gates status=applied, not thread resolution (near-miss: fleet-breaking merge on .github#1024) #1609 — re-scoped after a timeout; split into Follow-up to #1567: threads resolved with NO commit at all — the shipped evidence bar gates status=applied, not thread resolution (near-miss: fleet-breaking merge on .github#1024) #1609 (slice 1) + Follow-up to #1609 (slice 2): gate dev-lead thread resolution on the pass having advanced the PR head #1617 (slice 2). Slice 1 then completed in ~7 minutes.

Splitting demonstrably works and produces better-scoped work. The open question is whether we are paying for it more often than we should, and whether 2700 is the right number at all.

Why not just raise the budget

Raising it blind trades one unknown for another: a larger budget burns more engine quota per failure and delays the escalation signal, and we have no data on where real runs actually land in the distribution. #1609 landing at 2701s is either a wild coincidence or evidence that runs commonly expand to fill the budget — and those two readings imply opposite fixes. We should not guess.

Acceptance criteria

• Measure. Emit per-run stage elapsed time and the configured budget as structured output, and collect it across dev-lead runs (both repos, all intents). At minimum: intent type, issue/PR number, stage, elapsed, budget, outcome.
• Report the distribution. A p50/p90/p99 of elapsed-vs-budget for successful runs, and a separate bucket for timeouts. The specific question to answer: do runs cluster just under the budget (⇒ work expands to fill it, so raising it will not help) or is there a clean tail (⇒ raising it recovers real work)?
• Alert on near-misses. A run finishing within ~10% of budget is a warning today and a lost run tomorrow. #1609 at 2701/2700 would have been caught by this. Surface it the way engine-token-liveness surfaces token outages — a non-failing signal, not silence.
• Never lose the work. The largest cost is not the timeout, it is that partial work dies with the ephemeral runner. Investigate pushing the working branch before the budget expires (e.g. a checkpoint push at ~80% of budget, or trapping the timeout and pushing what exists) so a timeout costs a re-dispatch rather than a full re-implementation. This is likely the highest-value item here and is worth doing even if the budget number never changes.
• Then tune ACTION_TIMEOUT_SEC on the evidence and record the rationale, so the next person does not re-litigate it from scratch.
• Document when to split vs. when to raise, so the choice stops being ad-hoc.

Explicitly not in scope

Changing ACTION_TIMEOUT_SEC as a first move. The decision was to investigate, monitor and tune — not to bump the number and hope.

Context

DEEP_TIMEOUT_SEC = 1800 is a separate budget and should be included in the measurement, since a deep stage plus other stages is one of the ways the action budget gets consumed.

Related: #1609, #1617, #1589, #1587 (engine-token-liveness — the model for a monitor that reports a real signal without crying wolf), #1600.

⚠️ CORRECTION (maintainer, 2026-09-01): "one second over" is a reporting artifact, not a near-miss

The framing above leans on #1609 having died at 2701s against a 2700s budget — one second over — which reads as a near-miss that a small budget bump would have rescued. A second data point shows that reading is wrong.

petry-projects/.github#1047 timed out the same night:

Two different budgets, both reporting elapsed = budget + 1. That is not coincidence — it is how a timeout is detected and reported. The stage is killed at the budget and the elapsed figure is the budget plus the tick that noticed.

(Note the budgets differ because .github has no repo-level ACTION_TIMEOUT_SEC and inherits a 2100s default, while .github-private sets 2700.)

What this changes

• A small budget increase would not have saved either run. Both consumed their entire budget. We have no idea how much more they needed — 10 more seconds or 10 more minutes are equally consistent with the evidence.
• "Alert on near-misses" must not be built on this number. A near-miss detector keyed on elapsed vs budget would fire on every timeout at exactly +1 and never on a genuine near-miss, because a run that finishes just under budget reports its real elapsed while a timeout always reports budget+1. The two populations are not comparable.
• The AC "alert when a run finishes within ~10% of budget" is still right, but it must measure successful runs' real elapsed times. Timeouts are a separate bucket and tell you only "≥ budget".

Consequences for the ACs above

1. The measurement AC becomes more important, not less: we currently have no data on how long real runs take, and the one number we thought we had is meaningless.
2. The checkpoint-push AC is now clearly the highest-value item, independent of any budget decision — three timeouts (review-one-pr: metadata-digest re-review path has no atomic claim — concurrent triggers can both launch the cascade #1589, Follow-up to #1567: threads resolved with NO commit at all — the shipped evidence bar gates status=applied, not thread resolution (near-miss: fleet-breaking merge on .github#1024) #1609, ci(canary-rollout): list all registry agents in the dispatch input (#1045 part a) #1047) have each destroyed all their work, and knowing the true required budget would not have prevented any of them.
3. Do not tune ACTION_TIMEOUT_SEC off the +1 figure. Tune it off the successful-run distribution, once that exists.
4. Add: .github should set ACTION_TIMEOUT_SEC explicitly rather than inheriting a default that is 600s tighter than .github-private's, on a repo doing comparable work. That asymmetry is undocumented and is a plausible contributor to ci(canary-rollout): list all registry agents in the dispatch input (#1045 part a) #1047 timing out where a similarly-sized issue in .github-private would not have.

Evidence: #1609 escalation comment (run 33433995544), #1047 escalation comment (run 33458972975).