Skip to content

fix(compliance): daily re-trigger sweep broken (search 422) + fleet-wide burst risk #431

Closed

Description

@don-petry

Summary

The daily compliance re-trigger sweep (.github/workflows/compliance-retrigger.yml → scripts/compliance-retrigger.sh) — the mechanism that re-engages dev-lead on stale compliance issues whose original issues:labeled event was lost (agent token limits, rate-limits, transient failures) — has been failing every day since at least 2026-06-05, leaving the fleet's stuck compliance work unrecovered.

Evidence

  • Last 5 scheduled runs all failure, each dead in ~10–15s before re-triggering anything.
  • Failing run 27220764926:
    [error] search/issues API call failed (exit 1). Response:
    {"message":"Query must include 'is:issue' or 'is:pull-request'", "status":"422"}
    
  • 107 open compliance-audit+dev-lead issues piled up across the fleet, some ~2 weeks old (created 2026-05-26).

Root Cause

GitHub tightened the search/issues REST endpoint to require is:issue (or is:pull-request) in the query. scripts/compliance-retrigger.sh omitted it, so the sweep aborted before re-triggering. The weekly audit is unaffected (it uses the gh search issues CLI, which injects is:issue).

Secondary risk: fleet-wide burst

Once unbroken, the sweep would cycle the dev-lead label on all stale issues at once (dry-run: 63 in one shot), queuing many concurrent dev-lead runs in the same repo — rebase storms and token exhaustion, the very failure mode that stranded the work.

Resolution

  1. Add is:issue to the search query (unbreak the sweep).
  2. Throttle to one engagement per repo per run, oldest-first, so the most-stuck finding per repo is re-engaged and the daily cadence drains the rest one-at-a-time. A repo already active (open dev-lead PR / in-progress) consumes its slot.
  3. Move the sweep to 12:00 AM Central (05:00 UTC) so it runs off-peak.

Validated via workflow_dispatch dry-run on the fix branch: query returns the backlog (106), throttle yields 5 re-triggered / 3 skipped / 92 deferred instead of a 63-wide burst.

Context

Part of the Compliance program initiative (GH Project #1 Initiatives) / "Compliance Blitz" effort.

Activity

  1. added
    bugBug reports
    ciCI/CD pipeline issues
    automationAutomation improvements and gaps
    dev-leadFor dev-lead agent pickup
    and removed
    dev-leadFor dev-lead agent pickup
    on Jun 10, 2026
  2. added 2 commits that reference this issue on Jun 10, 2026
    1b8730a
    d6fb9c1
  3. added 2 commits that reference this issue on Jun 11, 2026
    817cbf5
    180b9f9
  4. added a commit that references this issue on Aug 7, 2026
    063acbc
  5. added a commit that references this issue on Aug 31, 2026
    2533bf9
  6. added a commit that references this issue on Sep 7, 2026
    67e9c1c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    automationAutomation improvements and gapsbugBug reportsciCI/CD pipeline issues

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions