Motivation
During the personal PR-reviewer evaluation (kept outside this repo), a session had to reconstruct
purely from PR evidence (zero Copilot reviews across every active repo, for days) that Copilot's
silence was a self-configured budget/quota stop, not a vendor-side malfunction — and had to
retroactively correct its own prior "non-functional fleet-wide" / "outage" phrasing once pieter
clarified the actual cause (a $20/mo Premium Request SKU budget cap with Stop usage: Yes, hit at
a specific timestamp, expected to reset on the account's own monthly cycle). That reconstruction
work is avoidable.
Proposal
Add a small, cheap check/principle to pr-review-conduct (or agent-conduct, wherever fits
best): before reporting a reviewer bot (Copilot, CodeRabbit, Qodo, or any future one) as
"broken," "down," or "non-functional," check whether it's actually a self-configured budget/quota
stop first (GitHub Settings -> Billing and plans -> Budgets for Copilot; each vendor's own
plan/usage dashboard for CodeRabbit/Qodo) rather than assuming a vendor incident from silence
alone.
Suggested terminology to standardize on, since this doc found the ambiguity itself worth naming
explicitly: "broken" = the vendor's product is malfunctioning, nothing pieter did causes it
and nothing pieter does fixes it; "turned off" = a self-imposed or account-configured limit
was hit on purpose, expected and resolves on a known schedule, no incident to chase.
Why this matters beyond just Copilot
This is a recurring pattern, not a one-off: pr_review.py already had to build dedicated
detection for the "Copilot's refusal doesn't reliably surface as an in-PR message" case
(ProjectTemplate #969, exit codes 46/47). The same investigative cost will recur for any reviewer
bot silence, on any repo, unless checking budget/quota status first becomes a standing step rather
than something reconstructed after the fact each time.
Motivation
During the personal PR-reviewer evaluation (kept outside this repo), a session had to reconstruct
purely from PR evidence (zero Copilot reviews across every active repo, for days) that Copilot's
silence was a self-configured budget/quota stop, not a vendor-side malfunction — and had to
retroactively correct its own prior "non-functional fleet-wide" / "outage" phrasing once pieter
clarified the actual cause (a $20/mo Premium Request SKU budget cap with
Stop usage: Yes, hit ata specific timestamp, expected to reset on the account's own monthly cycle). That reconstruction
work is avoidable.
Proposal
Add a small, cheap check/principle to
pr-review-conduct(oragent-conduct, wherever fitsbest): before reporting a reviewer bot (Copilot, CodeRabbit, Qodo, or any future one) as
"broken," "down," or "non-functional," check whether it's actually a self-configured budget/quota
stop first (GitHub Settings -> Billing and plans -> Budgets for Copilot; each vendor's own
plan/usage dashboard for CodeRabbit/Qodo) rather than assuming a vendor incident from silence
alone.
Suggested terminology to standardize on, since this doc found the ambiguity itself worth naming
explicitly: "broken" = the vendor's product is malfunctioning, nothing pieter did causes it
and nothing pieter does fixes it; "turned off" = a self-imposed or account-configured limit
was hit on purpose, expected and resolves on a known schedule, no incident to chase.
Why this matters beyond just Copilot
This is a recurring pattern, not a one-off:
pr_review.pyalready had to build dedicateddetection for the "Copilot's refusal doesn't reliably surface as an in-PR message" case
(ProjectTemplate #969, exit codes 46/47). The same investigative cost will recur for any reviewer
bot silence, on any repo, unless checking budget/quota status first becomes a standing step rather
than something reconstructed after the fact each time.