dev-lead engine repeatedly times out (engine-error) on larger issues — exhausts retries without delivering (e.g. .github#518)

Summary

The dev-lead engine repeatedly fails with engine-error (timeout / transient engine error) on larger, feature-sized issues, exhausting its retry budget without ever producing a PR. The retry mechanism now works (auto re-dispatch, "attempt N/3") — but the underlying per-attempt engine budget is too small for substantial issues, so retries just repeat the same timeout. Net: bigger issues silently never get delivered, even though the agent correctly posts an implementation plan first.

Live example — `petry-projects/.github#518`

#518 ("add-to-project: add a reconcile/backfill mode") is a real feature build (a scan script + workflow_dispatch/cron triggers + bats + docs). dev-lead:

posted a correct implementation plan, then
attempt 1/3 failed engine-error (run 27908018949),
auto-retried → attempt 2/3 failed engine-error (run 27909956774),
~2.5h elapsed, no PR. Status comment: status=failed attempt=2 reason=engine-error.

This matches earlier instances this session:

#478 — single attempt, Engine failed to implement (no retry then; pre-dev-lead retry doesn't cover failed initial issue implementations (only rate-limited PRs) — issues silently stall #781-fix).
#500 — attempt 1 failed engine-error, attempt 2 succeeded (but over-engineered the fix → introduced a CodeQL finding that needed manual rework in [Phase 2] Replace PUT-contents clobber deploy with versioned, ring-staged, health-gated promotion #501).

So the throughline: the engine times out on issues above some size/complexity, and either fails outright or produces a rushed/over-engineered result.

Why it matters

Larger initiatives labeled dev-lead don't get delivered — they consume 3 attempts of compute and then stall, with only failure comments to show.
It's silent: no escalation/hand-off, so a human only notices if watching.

Suggested directions

Right-size the budget: raise the per-attempt engine timeout (and/or use a higher-capacity model/tier) for issues above a complexity threshold.
Detect oversize + decompose or hand off: if an issue is too big for one attempt, have dev-lead split it (epic → stories, via the planner) or label dev-lead:needs-human and notify, instead of burning 3 identical timeouts.
Escalate on retry-exhaustion: when attempt 3/3 fails, apply a clear dev-lead:failed/needs-human label + ping, so it doesn't just go quiet.

References

Live: petry-projects/.github#518 (runs 27908018949, 27909956774).
Related: .github-private#781 (retry gap — the N/3 retry now exists, but the engine-error root cause remains); .github#500→[Phase 2] Replace PUT-contents clobber deploy with versioned, ring-staged, health-gated promotion #501 (over-engineered success).

Thanks for the detailed report and for including live run IDs — that makes diagnosis much easier. To narrow down the right fix, could you share the current per-attempt timeout (and/or token/step budget) configured for the dev-lead engine? Knowing that baseline will clarify whether a simple budget increase covers issues like #518, or whether a decompose/escalate path is needed for this size of work.

Root-cause research (new evidence + code trace)

The timeouts are deterministic budget-exceeded events, mislabeled as transient engine-error and then retried ~6× — pure waste. Traced end-to-end in scripts/engine.sh + scripts/dev-lead-fix-issue.sh + dev-lead-reusable.yml:

1. The knob and the walls. Each model tier is wrapped in GNU timeout (scripts/engine.sh):

DEEP_TIMEOUT_SEC="${DEEP_TIMEOUT_SEC:-600}"   # deep implementation tier
ACTION_TIMEOUT_SEC="${ACTION_TIMEOUT_SEC:-600}"

dev-lead-reusable.yml sets DEEP_TIMEOUT_SEC: ${{ vars.DEEP_TIMEOUT_SEC || '' }} → empty falls through to the engine default 600s. So:

petry-projects/.github: vars.DEEP_TIMEOUT_SEC unset → 600s wall.
petry-projects/.github-private: vars.DEEP_TIMEOUT_SEC=1800 → 1800s wall.
Job backstop: timeout-minutes: 60 (3600s).

2. Timeout (exit 124) is classified as retryable. scripts/engine.sh:

is_transient_failure() {  # 0 = retry
  case "$rc" in 124|137|143) return 0 ;; *) return 1 ;; esac
}

The comment assumes 124 means "flaky network/process state." In practice a per-tier timeout firing at exactly DEEP_TIMEOUT_SEC means the deep model genuinely needed more than the budget — deterministic, not flaky. Retrying with the same budget just repeats it.

3. Retried at two layers. Engine-level RETRY_MAX_ATTEMPTS retries the model call on 124, and the workflow-level MAX_ATTEMPTS=3 re-dispatches the whole run. A single too-big issue burns up to ~6 full-budget attempts.

4. The reason is collapsed. handle_engine_failure reads reason from the /tmp/dev-lead-failure-reason sidecar, defaulting to engine-error. The engine never writes timeout on a 124, so every timeout surfaces as reason=engine-error — indistinguishable from a genuine transient error, which is why the retry path treats them the same.

Fresh evidence from the PR-limits initiative (combsmith/.github#505)

Issue	Repo	Wall	Runs	Outcome
#507	.github	~10m23s each	4×	all `engine-error` at the 600s wall; delivered only after splitting into #507+#561
#510	.github	exactly 10m	2×	same 600s wall; hand-authored (PR #573)
#1003	.github-private	exactly 30m00s	1×	engine finished + posted "Implementation Complete" with passing tests, then died at the 1800s wall before pushing → work lost; hand-authored (PR #1004)

The identical per-issue durations are the signature of a deterministic wall. #1003 also exposes a distinct failure mode: the engine can complete the implementation and still time out in the post-implementation phase (self-review / commit / PR-create), losing the finished work.

Answers to the open design questions

Does time = cost, or is system slowness a factor? Both. Wall-clock maps to (a) GitHub Actions runner-minutes and (b) LLM token spend (the deep model runs the whole window). But wall-clock is not pure "work" — it's inflated by model/API latency, provider throttling, MCP/tool round-trips, and runner slowness. So a lone timeout is ambiguous: genuinely-too-much-work vs. a transiently-slow run. The discriminator we have is empirical: repeated timeouts at ~identical durations = deterministic (too big); a one-off = possibly slowness. That argues for at most one budget-escalated retry (to absorb slowness), never same-budget retries.

Variable timeouts by complexity? Worth it, but secondary. The first-order fix is to stop same-budget retries. Two complementary levers: (a) adaptive retry — if we retry a timeout at all, give it a larger budget (e.g. 1.5×, capped by the job), not the same one that just failed; (b) complexity-scaled initial budget — size DEEP_TIMEOUT_SEC at dispatch from an issue-size heuristic (a size:* label, body length, # acceptance criteria / referenced files).

What is the maximum timeout? Hard ceiling = the job timeout-minutes (currently 60 → 3600s; GitHub's absolute max is 360 min/job). A single deep call must leave room for the other tiers in the same job — triage 300 + audit 600 + duck 300 + git/PR/overhead — so within a 60-min job the practical DEEP_TIMEOUT_SEC max is ~1800–2100s (1800 is already in use in .github-private). Going higher requires raising timeout-minutes too.

Plan

Classify timeouts distinctly. On a per-tier timeout (exit 124) in the deep/action tier, the engine writes reason=timeout to /tmp/dev-lead-failure-reason (mechanism already exists; just populate it) so it is no longer collapsed into engine-error.
Stop the wasteful retry (the core fix). Do not retry a deep/action timeout with the same budget at either layer: remove 124 from the engine model-call retry for these tiers, and add a timeout branch to handle_engine_failure that escalates immediately (mirroring the existing missing-binary immediate-escalate path) instead of the engine-error → will retry path.
At most one adaptive-budget retry (optional, absorbs slowness). Before escalating, allow exactly one retry at an escalated DEEP_TIMEOUT_SEC (e.g. 1.5×, only if it fits under the job cap). If it still times out → deterministic → escalate.
Actionable escalation. The needs-human comment states the tier, elapsed seconds, and the budget, plus guidance: "hit the {N}s deep budget — split this issue or raise vars.DEEP_TIMEOUT_SEC," not the current generic "engine-error — will retry."
Don't lose completed work. Address the [PR-limits Phase 3] Wire the admission gate into dev-lead-fix-issue.sh (enforce the org-wide cap) #1003 mode: if the implementation phase completed (commit exists / changes staged) but a later phase times out, prefer pushing the branch + opening the PR over discarding — or at minimum surface that the work was completed-but-not-pushed.
Instrument. Log per-tier elapsed seconds alongside the existing TOKEN_LOG_FILE token counts, and surface elapsed-vs-budget in the escalation, so timeout-vs-slowness can be told apart over time.
Sane fleet defaults. Set vars.DEEP_TIMEOUT_SEC explicitly (petry-projects/.github is currently unset → 600, too small for feature-sized stories); recommend a documented fleet default with verified job-cap headroom.

Acceptance criteria

Evidence, code trace, and the design-question answers gathered while shepherding epic combsmith/.github#505; those stories were unblocked by splitting/rescoping the oversized issues and hand-authoring two of them. Related mislabel-of-timeout note lives in the shepherd's session memory.

Direction update (per maintainer) — maximize per-stage runtime; don't use timeouts for cost

Revised strategy, superseding the "tight budget + adaptive-retry" emphasis in the plan above:

Timeouts are a hang/stuck-run backstop, not a cost control. We will maximize the allowed runtime per stage so a legitimately large issue rarely hits a wall in the first place, and we will not size timeouts to bound token/runner cost. Cost, if it needs bounding, is handled elsewhere (token budgets / model tiering / issue sizing) — never by starving a stage of time.

What changes vs. the plan above

Keep (still correct): classify a per-tier timeout (exit 124) as reason=timeout distinct from engine-error; do not retry a timeout with the same budget. But the retry framing simplifies: at a maximized budget a timeout is definitively deterministic (there is no larger budget to escalate to), so a timeout → escalate to human immediately ("this issue needs splitting"), no retry. Drop the "adaptive-budget retry" step — it's unnecessary once the initial budget is already the max.
New primary lever: max runtime per stage.
New: timeout observability in monitoring.

Max runtime per stage — model

Per-tier timeouts must fit inside the job timeout-minutes backstop, alongside the other tiers that run in the same job + fixed overhead (checkout, CLI install, git/PR ops). So:

DEEP budget  =  job_cap  −  (Σ other per-tier budgets)  −  fixed_overhead_margin

Goal: raise the job timeout-minutes well above today's 60, and set each stage's timeout to consume its available share — deep/action (the implementation stages) get the lion's share; triage/duck stay small (they're lightweight reads). Concrete proposal (tune during implementation):

Knob	Today	Proposed
job `timeout-minutes` (backstop)	60 (3600s)	120 (7200s) — raise toward the practical max; GitHub's hard ceiling is 360 min/job
`DEEP_TIMEOUT_SEC`	600 (.github) / 1800 (.github-private)	≈ job_cap − reserved → ~5400s at a 120-min cap
`ACTION_TIMEOUT_SEC`	600	same share as deep (implementation-class work)
`AUDIT_TIMEOUT_SEC`	600	keep modest (900–1200s)
`TRIAGE_TIMEOUT_SEC` / `DUCK_TIMEOUT_SEC`	300	keep small (lightweight read/classify)

Two ways to set the deep budget; implementation picks one:

Static per-repo: set vars.DEEP_TIMEOUT_SEC explicitly per repo (petry-projects/.github is currently unset → 600, the root of the .github walls) to the raised value, and verify it fits under timeout-minutes.
Derived: compute DEEP_TIMEOUT_SEC at startup as job_cap − Σ(other tiers) − overhead_margin so it always tracks the job cap and can't exceed it.

Guardrail: Σ(all per-tier budgets) + overhead_margin ≤ job_cap must hold (a CI/lint check), so the sum can never silently exceed the backstop.

Timeout observability (new)

Raise the visibility of this failure mode in the monitoring fleet so we can see whether the bigger budgets actually eliminate walls:

Emit a timeout counter/rate for dev-lead (and the other engine callers) in scripts/fleet_monitor.sh / actions-fleet-monitor.yml — count runs whose failure is reason=timeout (keyed off the new classification), broken out from generic engine-error, over the reporting window.
Surface it in the fleet-monitor summary/issue and (optionally) the daily status, so a rising timeout count is an actionable signal rather than buried in engine-error.

Revised acceptance criteria

Cost is intentionally out of scope for timeout sizing per maintainer direction; timeouts exist only to catch genuinely hung/stuck runs, and should otherwise be as large as the job budget allows.

Dev-Lead Implementation Plan

Issue: #901 — dev-lead engine repeatedly times out (engine-error) on larger issues

Root cause

run_writer in scripts/engine.sh enforces a fixed per-attempt writer timeout
(ACTION_TIMEOUT_SEC, default 600s / 10min). Feature-sized issues need longer, so
the engine is killed by GNU timeout (exit 124 → classified engine-error), and because
every retry uses the same 600s budget, retries just repeat the identical timeout — exactly
the #518 symptom (attempt 1/3 and 2/3 both engine-error, no PR).

The retry mechanism (attempt N/3) and retry-exhaustion escalation (dev-lead:needs-human
label + comment) already exist and work — the missing piece is right-sizing the per-attempt
budget.

Scope

Right-size the dev-lead writer engine timeout: scale it by issue complexity and grow it
per attempt, bounded by a cap that stays under the reusable workflow's 60-min job cap.
No change to the common small-issue path (stays 600s), and an explicit operator override
(repo var ACTION_TIMEOUT_SEC) is honored verbatim. Retry-exhaustion escalation is already
implemented, so this focuses on direction #1 ("right-size the budget").

Implementation Checklist

scripts/dev-lead-fix-issue.sh: add pure issue_complexity_tier (0/1/2 from issue
body length + task-list checkbox count) and compute_engine_timeout <attempt>
(base + tier*step + (attempt-1)*step, capped).
scripts/dev-lead-fix-issue.sh: before the run_writer_with_fallback call, set
ACTION_TIMEOUT_SEC to the computed budget — but only when the operator did not
set it explicitly (capture the raw env value before engine.sh collapses empty→600).
Emit a ::notice:: recording the chosen budget/attempt/tier for observability.
scripts/dev-lead-fix-issue.sh: convert the bottom main "$@" to the repo-standard
source-guard so the pure functions are unit-testable.

Tests to Write

tests/dev-lead/unit/test_fix_issue.bats: issue_complexity_tier → 0 for a small
body, 1 for medium, 2 for a large feature body with many checkboxes.
compute_engine_timeout: tier-0/attempt-1 == 600 (preserves default), grows with
attempt (attempt 2 → 1200), grows with complexity (tier-2/attempt-1 → 1800), and is
capped (tier-2/attempt-3 → cap).
End-to-end: a large issue logs the right-sized ::notice::; an explicit
ACTION_TIMEOUT_SEC override suppresses auto-sizing.

Test Command

bats tests/dev-lead/unit/test_fix_issue.bats (full: bats tests/**/*.bats + bash .dev-lead/scripts/dev-lead-lint.sh)

Dev-Lead: issue #901 implementation failed — will retry

The engine failed (engine-error — e.g. a timeout or transient engine error). This was attempt 1 of 3; dev-lead will retry automatically. You can also re-apply the dev-lead label to retry now.

Cause: engine-error
Run: https://github.com/petry-projects/.github-private/actions/runs/28598293351

Meta-timeout: this run hit the exact wall it fixes

Dispatching #901 to dev-lead timed out at the 1800s deep-implementation budget (run 28598293351, dur=30m, reason=engine-error, attempt 1/3) — the change is too large for a single engine pass, which is precisely the bug this issue tracks. Halting the auto-retry (dev-lead:needs-human) to avoid burning two more 30-min walls.

Delivery approach — split into budget-sized stories (dogfooding this issue's own "split large issues" guidance). Implement order chosen so the first piece enlarges the budget for the rest:

Max runtime per stage (config/workflow only) — raise the job timeout-minutes + per-tier timeout defaults to their max viable share + a Σ(tier budgets) ≤ cap guardrail; set vars.DEEP_TIMEOUT_SEC. Lands first — makes every subsequent dev-lead run less likely to wall.
Catch + escalate timeouts — engine writes reason=timeout on exit 124; handle_engine_failure routes it to the immediate escalate-to-human path (no same-budget retry); don't discard completed-but-unpushed work; tests.
Timeout observability — fleet monitor surfaces a reason=timeout count/rate, broken out from engine-error.

Child stories to follow.

test(timeout): assert no fallback to copilot OR gemini on a 124 timeout

Follow-ups complete — fully closed out

The three deferred follow-ups noted at closure are now all merged and closed, completing this epic end-to-end:

[dev-lead timeout follow-up 1/3] Don't retry exit-124 at the engine level (in-run same-budget retry) #1028 → PR feat: implement issue #1028 — [dev-lead timeout follow-up 1/3] Don't retry exit-124 at the engine level (in-run same-budget retry) #1033 — engine-level: exit-124 no longer retried by is_transient_failure (last same-budget-retry gap closed).
[dev-lead timeout follow-up 2/3] Dedup handle_engine_failure escalation branches into a shared helper #1029 → PR feat: implement issue #1029 — [dev-lead timeout follow-up 2/3] Dedup handle_engine_failure escalation branches into a shared helper #1035 — handle_engine_failure escalation branches folded into a shared escalate_needs_human helper.
[dev-lead timeout follow-up 3/3] Report timeout wall-rate as prevalence (over total dev-lead runs) #1030 → PR feat: implement issue #1030 — [dev-lead timeout follow-up 3/3] Report timeout wall-rate as prevalence (over total dev-lead runs) #1057 — timeout wall-rate reported as true prevalence (timeout / all fleet dev-lead runs).

All work (A/B/C split #1017/#1018/#1019 + these 3 follow-ups) is on main, activating via the normal dev-lead channel soak. Removing the stale dev-lead:needs-human / needs-human-review labels — resolved, not awaiting a human.

feat: implement issue #1018 — [dev-lead timeout B/3] Catch exit-124 a…

Navigation Menu

Repository navigation

dev-lead engine repeatedly times out (engine-error) on larger issues — exhausts retries without delivering (e.g. .github#518) #901

Summary

Live example — `petry-projects/.github#518`

Why it matters

Suggested directions

References

Root-cause research (new evidence + code trace)

Fresh evidence from the PR-limits initiative (combsmith/.github#505)

Answers to the open design questions

Plan

Acceptance criteria

Direction update (per maintainer) — maximize per-stage runtime; don't use timeouts for cost

What changes vs. the plan above

Max runtime per stage — model

Timeout observability (new)

Revised acceptance criteria

Dev-Lead Implementation Plan

Root cause

Scope

Implementation Checklist

Tests to Write

Test Command

Dev-Lead: issue #901 implementation failed — will retry

Meta-timeout: this run hit the exact wall it fixes

3 remaining items

Follow-ups complete — fully closed out

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

dev-lead engine repeatedly times out (engine-error) on larger issues — exhausts retries without delivering (e.g. .github#518) #901

Description

Summary

Live example — petry-projects/.github#518

Why it matters

Suggested directions

References

Activity

github-actions commented on Jun 22, 2026

don-petry commented on Jul 2, 2026

Root-cause research (new evidence + code trace)

Fresh evidence from the PR-limits initiative (combsmith/.github#505)

Answers to the open design questions

Plan

Acceptance criteria

don-petry commented on Jul 2, 2026

Direction update (per maintainer) — maximize per-stage runtime; don't use timeouts for cost

What changes vs. the plan above

Max runtime per stage — model

Timeout observability (new)

Revised acceptance criteria

don-petry commented on Jul 2, 2026

Dev-Lead Implementation Plan

Root cause

Scope

Implementation Checklist

Tests to Write

Test Command

don-petry commented on Jul 2, 2026

Dev-Lead: issue #901 implementation failed — will retry

don-petry commented on Jul 2, 2026

Meta-timeout: this run hit the exact wall it fixes

3 remaining items

don-petry commented on Jul 4, 2026

Follow-ups complete — fully closed out

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Live example — `petry-projects/.github#518`