Skip to content

fix(dev-lead): multi-label issue creation can cancel the dev-lead dispatch (race) #443

Closed

Description

@don-petry

Summary

When an issue is created (or updated) with the dev-lead label plus one or more other labels in the same API call, the dev-lead agent can silently fail to be dispatched. GitHub fires a separate issues:labeled event per label; the dispatch job only acts when the triggering label is dev-lead, while a per-issue cancel-in-progress concurrency group lets a later (non-dev-lead) labeled event cancel the very run that would have acted. Net effect: the workflow reports success, but the agent never runs and no PR is opened.

Observed behaviour (concrete evidence)

While working issue #438 (created via gh issue create --label "dev-lead,bug,ci,automation"):

  • Four issues:labeled events fired at ~21:10, all sharing concurrency group dev-lead-issue-438.
  • Runs 27306552438 and 27306552128 completed success but the dispatch job classified intent as:
    INTENT_TYPE: skip
    INTENT_REASON: label-not-dev-lead
    
    i.e. they were triggered by the bug/ci/automation label events and correctly skipped.
  • The other two runs were cancelled by cancel-in-progress — and the run triggered by the dev-lead label was among them, so it never dispatched the agent.
  • Result after 25 min: issue OPEN, 0 comments, no in-progress label, no branch, no PR. A "successful" workflow that did nothing.
  • Recovery: cycling the label alone (remove + re-add dev-lead, no competing events) fired run 27308171856 →
    INTENT_TYPE: issue
    INTENT_REASON: issue-labeled-dev-lead
    
    agent ran (Pre-flight → Install engine CLIs → Run issue, all success) → PR feat: implement issue #438 — fix(compliance): remove dev-lead workflow manipulation from re-trigger sweep (Step 1 is out of scope) #442 opened.

This also means such issues are not self-healing: the daily compliance-retrigger sweep only re-triggers issues carrying the compliance-audit label, so a plain dev-lead issue stuck this way stays stuck indefinitely.

Root Cause

Per standards/ci-standards.md:421-428, the issue-intent job is gated and serialised as:

if: >-
  github.event_name == 'issues' && github.event.action == 'labeled' &&
    github.event.label.name == 'dev-lead'
concurrency:
  group: dev-lead-issue-${{ github.event.issue.number }}
  cancel-in-progress: true

Two facts combine into the race:

  1. Gate keys on the triggering label (github.event.label.name), not on whether the issue currently has the dev-lead label. So only the single event for the dev-lead label can act.
  2. cancel-in-progress collapses all same-issue events, and the cancellation happens at the workflow/concurrency layer before the if: decides a run is a no-op. So a non-acting run (wrong label) can cancel the one acting run (dev-lead label) purely based on arrival order.

When dev-lead is not the last label applied, the acting run loses the race.

Reproduction

  1. gh issue create --repo <org>/<repo> --label "dev-lead,bug" --title t --body b
  2. Observe the dev-lead runs: the dev-lead-triggered run is cancelled; survivors skip with label-not-dev-lead.
  3. Issue receives no agent dispatch (no plan comment, no PR).
  4. Confirm recovery by cycling dev-lead alone.

Recommended Fix

Gate on the issue's current label set rather than the triggering label, and let per-issue concurrency collapse the duplicate events onto a single run that still acts:

if: >-
  github.event_name == 'issues' && github.event.action == 'labeled' &&
    contains(github.event.issue.labels.*.name, 'dev-lead')
concurrency:
  group: dev-lead-issue-${{ github.event.issue.number }}
  cancel-in-progress: true

With this, every one of the N labeled events passes the gate; cancel-in-progress collapses them to the last-arriving run, which now acts (because the gate no longer depends on which label triggered it). Pair with an idempotency guard so re-entrant events can't double-dispatch (skip if a dev-lead/issue-<n>-* PR or fresh in-progress label already exists — the same check dl_dev_lead_active already encodes).

Apply the same fix to the org template (standards/workflows/dev-lead.yml / the reusable) and roll it to the fleet.

Acceptance Criteria

  • Creating an issue in a single API call with dev-lead plus ≥1 other label dispatches the agent exactly once (not zero).
  • Dispatch does not depend on dev-lead being the last label applied.
  • Adding dev-lead to an issue that already carries other labels (single add) still dispatches.
  • At most one concurrent agent run per issue; simultaneous events never produce duplicate PRs.
  • Label cycling / re-trigger recovery path (relied on by compliance-retrigger.sh) still works.
  • Regression test simulates multiple near-simultaneous issues:labeled events where dev-lead is not last, and asserts exactly one agent dispatch.
  • Standard (standards/ci-standards.md) and template updated to match.

Context

Discovered while monitoring #438 (whose fix, PR #442, is unrelated to this race). Part of the Compliance program / dev-lead reliability work. Sibling to #431/#432 (lost-labeled-event recovery).

Note: when assigning this to dev-lead, add the dev-lead label on its own (or cycle it last) to avoid triggering the very race described here.

Activity

  1. added
    bugBug reports
    ciCI/CD pipeline issues
    automationAutomation improvements and gaps
    dev-leadFor dev-lead agent pickup
    on Jun 11, 2026
  2. don-petry commented on Jun 11, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #443 — fix(dev-lead): multi-label issue creation can cancel the dev-lead dispatch (race)

    Scope

    Fix the event classifier (dev-lead-intent.sh) to gate on whether the issue's current label set contains dev-lead rather than solely on the triggering label, so any one of the N issues:labeled events passes the gate and dispatches the agent exactly once.

    Root Cause (confirmed)

    In .dev-lead/scripts/dev-lead-intent.sh (lines 339-361), the issues routing checks .label.name (the single triggering label) instead of .issue.labels[] (the issue's current label set). With concurrent events and cancel-in-progress: true (the old claude.yml pattern), the acting dev-lead-triggered run could be cancelled. The concurrency race was already fixed (#402, cancel-in-progress: false in the reusable), but the intent classifier still reads the wrong field — so when dev-lead is not the last label applied, the surviving run emits skip:label-not-dev-lead even though the issue carries the label.

    Implementation Checklist

    • Fix .dev-lead/scripts/dev-lead-intent.sh — change issues handler to use has_label "dev-lead" (reads .issue.labels[]) instead of comparing .label.name to "dev-lead"
    • Create test/scripts/dev-lead-intent/helpers/setup.bash — test harness helpers
    • Create test/scripts/dev-lead-intent/dev-lead-intent.bats — bats regression tests
    • Create .github/workflows/dev-lead-intent-tests.yml — CI workflow that checks out .github-private and runs the intent test suite
    • Update standards/ci-standards.md — document that the intent classifier gates on current label set, not triggering label; add note about multi-label race fix

    Tests to Write

    • Issue dispatch when dev-lead is the triggering label — baseline: INTENT_TYPE=issue
    • Issue dispatch when dev-lead is NOT the triggering label but IS in the label set — the core race scenario; must emit INTENT_TYPE=issue, not skip
    • Skip when dev-lead is not in the current label set — no regression: events for unrelated labels on issues without dev-lead still skip
    • Skip when issue action is not labeled — unchanged baseline
    • Idempotency: check_existing_pr guard still active — handled by dev-lead-fix-issue.sh; intent script emits issue intent regardless, duplicate dispatch prevented downstream

    Test Command

    sudo apt-get install -y bats && bats --print-output-on-failure test/scripts/dev-lead-intent/

    Notes

    The idempotency guard (check_existing_pr in dev-lead-fix-issue.sh, dl_dev_lead_active in scripts/lib/dev-lead-retrigger.sh) already prevents double-dispatch; no change needed there. The fix is a one-line change in the intent classifier's issues case.

  3. added
    dev-leadFor dev-lead agent pickup
    and removed
    dev-leadFor dev-lead agent pickup
    on Jun 11, 2026
  4. don-petry commented on Jun 11, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #443 — fix(dev-lead): multi-label issue creation can cancel the dev-lead dispatch (race)

    Scope

    Fix the event classifier (dev-lead-intent.sh) to gate on the issue's current label set rather than the triggering label, so any issues:labeled event dispatches the agent when dev-lead is among the issue's labels — regardless of which specific label triggered the event.

    Root Cause (confirmed)

    In .dev-lead/scripts/dev-lead-intent.sh lines 350–360, the issues routing checks .label.name (the triggering label):

    label_name=$(jq -r '.label.name // empty' "$EVENT_PATH")
    case "$label_name" in
      dev-lead) emit_intent "issue" ... ;;
      *)        emit_skip "label-not-dev-lead" ;;
    esac

    The existing has_label helper already reads .issue.labels[]?.name (the full label set). Using it instead eliminates the race.

    Implementation Checklist

    • test/dev-lead/helpers/setup.bash — test helper (TDD first)
    • test/dev-lead/dev-lead-intent.bats — bats test suite covering race scenario and regression cases
    • .dev-lead/scripts/dev-lead-intent.sh — change issues routing from triggering-label check to has_label "dev-lead"
    • .github/workflows/dev-lead-intent-tests.yml — CI gate: shellcheck + bats on every PR touching the intent classifier
    • standards/ci-standards.md — document the label-set-based gate in the Dev-Lead Agent section

    Tests to Write

    • dev-lead is the only label applied (single-label happy path — regression)
    • bug is the triggering label but dev-lead is already in the issue's label set → dispatches (the race scenario)
    • dev-lead is the triggering label but other labels are also present → dispatches
    • Non-dev-lead triggering label, dev-lead not in issue labels → skips (correct skip preserved)
    • issues_action != labeled → skips (existing behavior preserved)
    • Missing event path → skips (existing behavior preserved)
    • Idempotency: check_existing_pr in dev-lead-fix-issue.sh already guards against double-dispatch when multiple events reach the agent

    Test Command

    bats --print-output-on-failure test/dev-lead/
    
  5. don-petry commented on Jul 14, 2026

    @don-petry
    ContributorAuthor

    Closing as already resolved by #450 (per-issue/per-PR concurrency lanes, merged 2026-06-06 — 5 days before this was filed on 2026-06-11 from the pre-fix #438 incident evidence).

    The reusable now uses cancel-in-progress: false with separate dev-lead-issue-<n> / dev-lead-pr-<n> lanes: same-lane issues:labeled events queue behind the active run instead of cancelling it, so the dev-lead-triggered pickup always finishes regardless of label order. All runtime acceptance criteria hold:

    • multi-label create → exactly one dispatch (the dev-lead event acts; others skip label-not-dev-lead);
    • dispatch independent of which label is last;
    • single dev-lead add on an already-labelled issue still dispatches;
    • ≤1 concurrent run per issue; label-cycling recovery intact.

    Regression guard added in combsmith/.github-private#1221.

    Doc-hygiene follow-up (not blocking): standards/ci-standards.md still carries a legacy claude-issue example (~L417) showing github.event.label.name == 'claude' + cancel-in-progress: true — the retired claude label (#456) and the exact anti-pattern this issue warned about. That example block should be updated to the current dev-lead pattern or removed as part of the claude-deprecation doc cleanup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    automationAutomation improvements and gapsbugBug reportsciCI/CD pipeline issuesdev-leadFor dev-lead agent pickup

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions