bug(dev-lead): engine.sh defines multi-tier model pipeline but all tasks always invoke ENGINE_ACTION_MODEL (sonnet-4-6) #195

Problem

The engine.sh abstraction layer defines a multi-tiered model pipeline with four distinct roles:

It also defines corresponding functions: run_triage(), run_agentic(), run_duck(), and run_writer().

However, none of the dev-lead action scripts use this pipeline. Every intent — regardless of complexity — routes through a single call:

run_writer_with_fallback "$prompt_file"  # always ENGINE_ACTION_MODEL = sonnet-4-6

This affects all four scripts:

• dev-lead-fix-reviews.sh → build_and_run() → run_writer_with_fallback()
• dev-lead-fix-ci.sh → run_writer_with_fallback()
• dev-lead-fix-issue.sh → run_writer_with_fallback()
• dev-lead-fix-reviews.sh (also handles human, human-pr, fix-bot-comment, rebase) → run_writer_with_fallback()

Impact

This was observed on 2026-05-16 when 6 PRs were opened simultaneously across all org repos. Each triggered a Dev-Lead Agent run, all using claude-sonnet-4-6 unconditionally — including a human-pr run on a 1-file CI-only PR (chore(dev-lead): remove claude.yml) that had zero open review threads, which was confirmed in the logs:

The GitHub API confirms there are zero review comments on PR #277

That entire run could have been handled by haiku-4-5 in the triage step, or skipped entirely. The burst hit the daily usage cap (You've hit your limit · resets 5pm (UTC)) across the org.

The ENGINE_LABEL printed at startup — triage: haiku 4.5 → deep: sonnet 4.6 + duck: o4-mini → audit: opus 4.7 — is misleading: it describes a pipeline that does not execute. The run_triage(), run_agentic(), and run_duck() functions in engine.sh appear to be dead code within the dev-lead system (they may serve a separate PR review workflow via review-one-pr.sh).

Additional Problem: Engine Failover Is Broken

The run_writer_with_fallback() function intends to failover from Claude → Gemini → Copilot when a provider is rate-limited, but the detection mechanism is silently broken in multiple ways.

Bug 1: Rate-limit output is never captured

run_writer() attempts to detect rate-limits by reading /tmp/dev-lead-writer-stderr, but nothing writes to that file:

# engine.sh — run_writer()
timeout "$ACTION_TIMEOUT_SEC" claude --print \
  --model "$model" \
  < "$prompt_file" || rc=$?          # stdout goes to terminal, not captured

# Then:
output_check=$(cat /tmp/dev-lead-writer-stderr 2>/dev/null)  # always empty
is_rate_limited "$output_check"                               # never fires

The rate-limit text (e.g. You've hit your limit · resets 5pm (UTC)) appears on stdout, not in the file that is_rate_limited checks. So run_writer() always returns the raw non-zero exit code, never code 2, and run_writer_with_fallback() never advances to the next engine. Gemini and Copilot are never tried.

This was directly confirmed in the 2026-05-16 incident: broodly's Dev-Lead run failed with exit code 1 and the rate-limit message appeared in the step log — but no fallback to Gemini or Copilot occurred.

Bug 2: Copilot write path falls back to Claude

When REVIEW_ENGINE=copilot, run_writer() silently substitutes Claude for write operations:

copilot)
  echo "::warning::Copilot engine is text-only; falling back to Claude for write operations"
  REVIEW_ENGINE="claude" timeout ... claude --print ...   # back to the rate-limited provider

So even if the fallback to Copilot were reached, it would immediately call Claude again — cycling back to the same rate-limited endpoint.

Bug 3: No proactive usage monitoring

The agent only reacts to rate-limit errors after they occur. By that point, a full model invocation has already been attempted (and charged), the job fails, and the PR is left in a broken state.

There is no mechanism to check remaining usage before invoking a provider and switch to an alternative proactively.

Proposed Fix

The goal is a clean, simple, linear decision path: pick the right model for the task complexity, confirm the provider has headroom, run it, and fall back gracefully if needed. No implicit pipelines, no dead code.

Fix 1 — Intent-based model dispatch (addresses Problem 1)

Add a single model_for_intent function to engine.sh and call it from each script:

# engine.sh — replace ENGINE_ACTION_MODEL hardcoding everywhere
model_for_intent() {
  case "${1:-}" in
    human-pr|fix-bot-comment)   echo "$ENGINE_TRIAGE_MODEL" ;;   # haiku: lightweight check
    fix-reviews|fix-ci|rebase)  echo "$ENGINE_ACTION_MODEL" ;;   # sonnet: write operations
    fix-issue|human)            echo "$ENGINE_DEEP_MODEL"   ;;   # sonnet: full feature work
    *)                          echo "$ENGINE_ACTION_MODEL" ;;
  esac
}

# Each script — one change per call site
run_writer_with_fallback "$prompt_file" "$(model_for_intent "$INTENT_TYPE")"

Fix 2 — Repair rate-limit capture in run_writer() (addresses Bug 1)

Capture stdout to a temp file so is_rate_limited can inspect it:

run_writer() {
  local prompt_file="$1" model="${2:-$ENGINE_ACTION_MODEL}"
  local _out _rc=0

  _out=$(mktemp)
  case "$REVIEW_ENGINE" in
    claude)
      timeout "$ACTION_TIMEOUT_SEC" claude --print \
        --model "$model" --permission-mode acceptEdits \
        --allowed-tools "Bash,Read,Write,Edit,Grep,Glob" \
        < "$prompt_file" | tee "$_out" || _rc=$? ;;
    gemini)
      timeout "$ACTION_TIMEOUT_SEC" gemini --prompt "" \
        --model "$model" --approval-mode auto_edit \
        --output-format text \
        < "$prompt_file" | tee "$_out" || _rc=$? ;;
    copilot)
      copilot_chat "$prompt_file" "$ACTION_TIMEOUT_SEC" | tee "$_out" || _rc=$? ;;
  esac

  if [ "$_rc" -ne 0 ] && is_rate_limited "$(cat "$_out")"; then
    rm -f "$_out"; return 2   # signal fallback
  fi
  rm -f "$_out"; return "$_rc"
}

This also removes the broken copilot-falls-back-to-claude path (Bug 2): the copilot case now calls copilot_chat directly.

Fix 3 — Proactive usage check before invocation (new requirement, addresses Bug 3)

Add a check_provider_headroom function that queries each provider's usage API and skips to the next engine if usage is at or above 75%:

# engine.sh
USAGE_THRESHOLD="${DEV_LEAD_USAGE_THRESHOLD:-75}"   # percent; override via repo var

# check_provider_headroom <engine>
# Returns 0 (ok to proceed) or 1 (at/above threshold — skip to next engine).
# Falls back to 0 (proceed) on any query failure so a missing API never blocks work.
check_provider_headroom() {
  local engine="$1"
  local used_pct=0

  case "$engine" in
    claude)
      # Claude usage via Anthropic API header probe (X-RateLimit-Remaining-Tokens)
      local remaining_tokens limit_tokens
      local _resp
      _resp=$(curl -sI -X POST https://api.anthropic.com/v1/messages \
        -H "x-api-key: ${ANTHROPIC_API_KEY:-}" \
        -H "anthropic-version: 2023-06-01" \
        -H "content-type: application/json" \
        --data-raw '{"model":"claude-haiku-4-5","max_tokens":1,"messages":[{"role":"user","content":"."}]}' \
        2>/dev/null || true)
      remaining_tokens=$(echo "$_resp" | grep -i 'x-ratelimit-remaining-tokens' | awk '{print $2}' | tr -d '\r' || echo "")
      limit_tokens=$(echo "$_resp" | grep -i 'x-ratelimit-limit-tokens' | awk '{print $2}' | tr -d '\r' || echo "")
      if [ -n "$remaining_tokens" ] && [ -n "$limit_tokens" ] && [ "$limit_tokens" -gt 0 ]; then
        used_pct=$(( 100 - (remaining_tokens * 100 / limit_tokens) ))
      fi ;;

    gemini)
      # Gemini does not expose a usage header; skip headroom check (always proceed)
      return 0 ;;

    copilot)
      # GitHub Models rate-limit headers from a lightweight probe
      local _resp _remaining _limit
      _resp=$(curl -sI https://models.github.ai/inference/chat/completions \
        -H "Authorization: Bearer ${COPILOT_GITHUB_TOKEN:-}" \
        -H "X-GitHub-Api-Version: 2022-11-28" 2>/dev/null || true)
      _remaining=$(echo "$_resp" | grep -i 'x-ratelimit-remaining-requests' | awk '{print $2}' | tr -d '\r' || echo "")
      _limit=$(echo "$_resp" | grep -i 'x-ratelimit-limit-requests' | awk '{print $2}' | tr -d '\r' || echo "")
      if [ -n "$_remaining" ] && [ -n "$_limit" ] && [ "$_limit" -gt 0 ]; then
        used_pct=$(( 100 - (_remaining * 100 / _limit) ))
      fi ;;
  esac

  if [ "$used_pct" -ge "$USAGE_THRESHOLD" ]; then
    echo "  [headroom] $engine usage ${used_pct}% >= threshold ${USAGE_THRESHOLD}% — skipping" >&2
    return 1
  fi

  echo "  [headroom] $engine usage ${used_pct}% — ok" >&2
  return 0
}

Then wire it into run_writer_with_fallback():

run_writer_with_fallback() {
  local prompt_file="$1" model="${2:-$ENGINE_ACTION_MODEL}"
  local engines=("$REVIEW_ENGINE")
  for e in claude gemini copilot; do
    [ "$e" != "$REVIEW_ENGINE" ] && engines+=("$e")
  done

  for engine in "${engines[@]}"; do
    if ! check_provider_headroom "$engine"; then
      echo "::warning::$engine at/above usage threshold, trying next engine" >&2
      continue
    fi
    export REVIEW_ENGINE="$engine"
    local rc=0; run_writer "$prompt_file" "$model" || rc=$?
    [ "$rc" -eq 0 ] && return 0
    if [ "$rc" -eq 2 ]; then
      echo "::warning::$engine rate-limited mid-run, trying next engine" >&2
      continue
    fi
    return "$rc"
  done

  echo "::error::All engines exhausted (rate-limited or above usage threshold)" >&2
  return 2
}

The logic is now flat and explicit: check headroom → run → check exit → fall back or fail. No hidden branching.

Acceptance Criteria

Model dispatch (Problem 1)

• Simple intents (human-pr, fix-bot-comment) use ENGINE_TRIAGE_MODEL (haiku-4-5)
• Medium intents (fix-reviews, fix-ci, rebase) use ENGINE_ACTION_MODEL (sonnet-4-6)
• Complex intents (fix-issue, human) use ENGINE_DEEP_MODEL (sonnet-4-6, or configurable)
• ENGINE_LABEL accurately reflects which model(s) are actually invoked per intent
• run_triage() / run_agentic() / run_duck() are documented as review-one-pr.sh-only or removed from engine.sh scope

Failover (Problem 2)

• Rate-limit output is captured from all three provider invocations and correctly detected
• Copilot write path does not fall back to Claude; it uses copilot_chat directly
• When Claude is rate-limited, Gemini is attempted next (requires GOOGLE_API_KEY)
• When Gemini is unavailable/rate-limited, Copilot is attempted next (requires COPILOT_GITHUB_TOKEN)
• All-engines-exhausted path emits a clear error and exits non-zero

Proactive usage monitoring (new requirement)

• check_provider_headroom queries each provider before invocation
• Provider is skipped (falls to next) when usage ≥ 75% (configurable via DEV_LEAD_USAGE_THRESHOLD)
• Query failure does not block execution (fail-open)
• Threshold and per-engine headroom is logged to the step summary

Code quality

• run_writer_with_fallback is a flat loop: check headroom → run → handle exit — no nested engine-specific branching
• All three provider paths in run_writer are symmetric: capture output, detect rate-limit, return standard codes
• Existing unit tests (tests/dev-lead/unit/) pass; new tests added for headroom check and fallback path

References

• scripts/engine.sh — model tier definitions, run_writer(), run_writer_with_fallback()
• scripts/dev-lead-fix-reviews.sh — handles fix-reviews, human-pr, human, fix-bot-comment, rebase
• scripts/dev-lead-fix-ci.sh — handles fix-ci
• scripts/dev-lead-fix-issue.sh — handles issue
• .github/workflows/dev-lead-reusable.yml — intent routing