Skip to content

ci: make stalled PRs and hung tests visible (shepherd:reapprove, pytest --timeout) - #4865

Merged
squid-protocol merged 1 commit into
mainfrom
ci/shepherd-visible-reapproval
Oct 10, 2026
Merged

squid-protocol merged 1 commit into
mainfrom
ci/shepherd-visible-reapproval

Conversation

@squid-protocol

Copy link
Copy Markdown
Owner

Part of #4840

Two quiet failures from 2026-10-10. Each had a fix in place on the spot, but nothing would have made either one visible the next time. This makes both visible.

1. A PR that lost its approval now carries a label: shepherd:reapprove

What happened: #4858 was green but didn't merge. A push dropped shepherd:merge (by design: an approval is bound to the head it was given on). With no request label left, the bot also cleared every status label, so the only trace was a comment. Nobody re-added the label until a person noticed.

Now: when the bot drops shepherd:merge because the head moved, it also sets shepherd:reapprove.

  • It stays after the request labels are gone, so is:open label:shepherd:reapprove lists every PR waiting for a maintainer to re-approve.
  • It clears when shepherd:merge comes back (or shepherd:hold is set), or when someone removes it by hand. A manual removal sticks.
  • Reconcile already visits PRs that carry any status label, so these stay in its sweep.
  • The security model is unchanged: nothing is re-approved automatically.
  • The "awaiting marker" case (push or label time unknown) stays quiet as before. That state is undecided and can resolve on the next pass, so a label there would flap.

2. A hung test now fails by name instead of hitting the job time limit

What happened: every macOS matrix leg on main ran into the 45-minute timeout-minutes on a hung test (#4862: no flock on macOS). GitHub reports that as "cancelled", which names nothing, and it read as noise all day.

Now:

  • full-suite-gate.yml and smoke-test.yml run pytest with --timeout=600 (pytest-timeout==2.4.0). The slowest real test is ~60 s (.test_durations), so this is ~10× headroom and well under the 30 and 45-minute job limits. Checked locally with pytest 9.1.1 + xdist 3.8.0: a hung test fails as FAILED …::test_hangs - Failed: Timeout, and the rest of the run carries on.
  • main_watch.py gives each cancelled leg its run time ("… after 45 min"). It also no longer calls a cancel at the time limit "not a verdict on the code".

Also

docs/ci.md: the new label in the status table and its lifecycle, the timeout, and a pair of paragraphs that appeared twice is removed.

Checks

  • test_shepherd_gh.py + test_main_watch.py: 69 pass, including 4 new tests.
  • pr_gates --fast PASS. --ratchets all PASS except ports (the local CardDemo corpus is not at its pinned ref; same on main).
  • Both workflows parse as YAML. actionlint is not installed here.
  • main_watch.py reconcile --dry-run against the live repo: no actions. That's correct, since the latest verdict is already on main is red: Full Suite Gate (All OS x Python) #4840.

After merge

  • Run shepherd-reconcile.yml once with ensure_labels: true, so the label exists with its colour and description. Until then, the first add creates it with GitHub's default colour.

Not done here (worth a look)

  • The macOS legs were split into 3 shards on 2026-10-10 because they "timed out". The real cause was the hang: the shards together take ~18 min, so one macOS leg per Python would fit in 45 min again and save the extra setup time per shard on the costliest runners.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AUWeQX7v4QBdErFbghEEEX

…st --timeout)

Two quiet failures from 2026-10-10:

- #4858 sat green-but-unmerged: a push dropped shepherd:merge (an approval
  is bound to its head, by design) and, with no request label left, every
  status label was cleared too, so nothing marked the PR. The bot now sets
  shepherd:reapprove when it drops the label. It outlives the request labels
  (is:open label:shepherd:reapprove finds every PR that lost its approval)
  and clears when shepherd:merge or shepherd:hold comes back, or by hand.

- Every macOS matrix leg on main ran into the 45-minute job limit on a hung
  test (#4862) and read as "cancelled", naming nothing. full-suite-gate.yml
  and smoke-test.yml now run pytest with --timeout=600 (pytest-timeout
  2.4.0; slowest real test ~60 s), so a hang fails as a named test.
  main_watch.py reports each cancelled leg's run time ("after 45 min") and
  no longer calls a time-limit cancel "not a verdict on the code".

docs/ci.md: the new label, its lifecycle, the timeout; drops a duplicated
pair of paragraphs.

Part of #4840

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUWeQX7v4QBdErFbghEEEX
@squid-protocol squid-protocol added ci-cd Continuous integration, delivery, and automated runners enhancement New feature, sensor, or structural signature labels Oct 10, 2026
@squid-protocol

Copy link
Copy Markdown
Owner Author

Note for merging: this changes shepherd_gh.py itself, so per the shepherd's own rule it merges through the local shepherd (python tests/tools/box/ci_shepherd.py add 4865) or a manual gh pr merge --squash after review, not by its own label path.

@github-actions

Copy link
Copy Markdown
Contributor

🐦‍⬛ Muninn Security Scan

✅ No security issues found.

🐦‍⬛ Powered by Muninn · Skald Lab

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-cd Continuous integration, delivery, and automated runners enhancement New feature, sensor, or structural signature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant