PR steward's generated-only check fails open to a full review when the GitHub App installation hits its API rate limit — no retry/backoff on 403

What is failing

The PR steward's generated-only check (GeneratedOnlyPullRequest.Exempt in the PlatformBuildInbox watcher) reads a candidate settle PR through the GitHub App to decide whether it owes a language-model review. When that read returns GitHub's 403 installation API rate limit (API rate limit exceeded for installation ID 161076693), the check throws, logs [Steward] … the generated-only check failed — reviewed as usual, and fails open: the head goes down the ordinary review path and occupies a language-model round — the exact dependency the exemption exists to remove for settle-locks / stamp-floors PRs (the #2860 deadlock).

The concrete case is the meshweaver-cloud[bot] PR Systemorph/MeshWeaver.Plugins#3031 at head 2a079e4.

Probable cause

High confidence the proximate cause is a genuine GitHub-side rate limit: the exception carries GitHub's verbatim installation-rate-limit body with a request ID, not a parse or auth failure. Medium confidence on the underlying cause: the steward's GitHub client has no rate-limit handling at all — no retry/backoff honoring x-ratelimit-reset on 403/429 and no call budgeting — while each PR admission costs several API reads (pulls, commits pages, per-page files, check-runs) across many repositories, so a busy window can exhaust the installation's hourly quota. The Catch in Exempt is by design fail-open ("never exempt on a failure"), but that design assumes read failures are rare and isolated; during an installation-wide rate-limit burst the exemption is unavailable across every PR at once.

Impact

Judged from the evidence: a single burst on one pod, one PR affected at the moment captured — not an outage. However, the same rate-limit window produced sibling failures in the same watcher (ReviewCarryOver's check-runs read on the same PR, and the changed-files read — see incidents ce4f2cc9053ed862 and 04059b2bfbeec521), so a sustained rate limit degrades the whole steward pipeline, not just the exemption. The fallback is safe (the PR still gets an ordinary review), which keeps this below blocking severity.

Where to look

Hosting/Deployment/Source/GeneratedOnlyPullRequest — Exempt's read chain: the throwing branch is the ParsePull null-check on the pulls/{n} reply (<Exempt>b__4 in the top frame); the Catch that fails open to Observable.Return(null).
PullRequestIntake.Rest / GetFiles — the GitHub REST wrapper; the natural place for rate-limit awareness (honoring x-ratelimit-remaining / x-ratelimit-reset, retrying a 403 rate limit, or at least throttling the sweep's read volume).
Not duplicates: MeshWeaver#5727 covers a stale head sha cause of the sibling "changed files could not be read" site; Plugins fix(test): the inline-resumption probe compared thread IDs, which proves nothing #2788/Recompile-set derivation times out and abandons the affected recompiles with no retry or failure detail #2887 covered the model-provider quota outage — neither covers the GitHub App installation rate limit.

Evidence


Fingerprint	`c3a9154f3c0f3d32`
Category	`PlatformBuildInboxWatcher`
Severity	Error
Exception	`System.InvalidOperationException`
Top frame	`GeneratedOnlyPullRequest.<>c__DisplayClass14_2.<Exempt>b__4(Nullable`1 files)`
Namespace	`memex`
Pods	`memex-portal-deployment-6bcdbb54fc-g9gh4`
Occurrences	1
First seen	2026-10-06 21:45:43Z
Last seen	2026-10-06 21:45:43Z
Routing	not determined — no configured route matches the category `PlatformBuildInboxWatcher`. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.

Recent log lines

2026-10-06 21:45:43Z memex-portal-deployment-6bcdbb54fc-g9gh4 fail: PlatformBuildInboxWatcher[0]
      [Steward] Systemorph/MeshWeaver.Plugins#3031 at 2a079e4: the generated-only check failed — reviewed as usual
      System.InvalidOperationException: the pull request could not be read (403): { 	"message": "API rate limit exceeded for installation ID 161076693. If you reach out to GitHub Support for help, please include the request ID 1819:99D9:7A8C8:91B5B:6AC56C06 and timestamp 2026-10-06…
         at GeneratedOnlyPullRequest.<>c__DisplayClass14_2.<Exempt>b__4(Nullable`1 files) in /tmp/MeshWeaver/.mesh-cache/Hosting_PlatformBuildInbox.cs:line 8288
         at System.Reactive.Linq.ObservableImpl.Select`2.Selector._.OnNext(TSource value)

_{Opened automatically from Admin/_LogIncident/c3a9154f3c0f3d32. Recurrences are folded into this issue rather than opening new ones.}
_{It also stands for the whole log site f90833a19637ffd7: other fingerprints of this site fold in here as comments rather than opening tickets of their own.}

Fix merged: Systemorph/MeshWeaver.Plugins#3145 (merge c385307855). Not yet verified on the control instance, so this stays open.

Root cause. PullRequestIntake.SendAsync dropped the response headers. A 403 API rate limit exceeded for installation … therefore reached every caller as an ordinary refusal, indistinguishable from a permission 403. Two checks then fell open to a language-model round:

ReviewCarryOver.Unread (Review carry-over check aborts on a GitHub API rate-limit 403 instead of backing off until the reset window #6215);
GeneratedOnlyPullRequest.Exempt (this issue).

Fix.

RestReply carries x-ratelimit-remaining / x-ratelimit-reset / retry-after.
GitHubRateLimit.WaitFor classifies a reply: until the reset; retry-after; or GitHub's minute for a secondary limit that names no wait. A permission 403 is not a rate limit.
Both checks raise GitHubRateLimitedException and pass it through. Every other failure falls back as before.
AdmittedReview defers the head: no round, no slot, the reset named on the item's note, and Warning in the log. The head records no round, so PullRequestSweep kicks it again after the window refills.
Nothing waits inline. A first cut that waited in the seam was rejected in review: it would have held the inbox's serial .Concat() slot, and callers' 60 s / 2 min / 20 min deadlines would have cancelled it.

Tests. GitHubRateLimitTests (5 cases) replays the incident's reply through the checks' real REST seams. mw-plugin-test build: all 16 packages green, Hosting/Deployment 448 passed / 0 failed. Negative control: with the pass-through removed, the carry and generated-only cases FAIL (446/2).

To close it (not done yet):

the merged module reaches the control instance;
Hosting/PlatformBuildInbox is recycled, because a running activation keeps the old build;
the next rate-limited window logs [Steward] … Deferred at … rate limit … at Warning, instead of the review carry-over check failed — reviewed as usual at Error.

What I did NOT establish:

the installation's hourly consumption;
whether the other GitHub clients in the module (PrBabysitter, PrFixer, PlatformPinUpdater, each with its own SendAsync) meet the same refusal. Those are not covered by this fix.

Closing: the fix is merged and running, and the code path it fixes no longer runs on any portal.

The fix. Systemorph/MeshWeaver.Plugins#3145 (c385307855) merged on 2026-10-08. The control instance's portal image (3.0.0-ci.10322, booted 2026-10-09 13:57Z) was built from Plugins main 0b8f875478, which contains c385307855 (compare → ahead).

The path is dormant, measured read-only through governed Logs actions on control.systemorph.com:

query	window	lines
`{namespace="memex"} \|~ "\[Steward\]"` (`Ops/Actions/logs-memex-20261009-steward-24h`)	2026-10-08 14:40Z → 2026-10-09 14:40Z	0
`{namespace="control"} \|~ "\[Steward\]"` (`Ops/Actions/logs-control-20261009-steward-24h`)	same 24 h	0
positive control, memex: same query over 7 days (`Ops/Actions/logs-memex-20261009-steward-7d`)	2026-10-02 → 2026-10-09	20, cut at the limit. For example `[Steward] review of Hosting/Triage/pull-request/systemorph-meshweaver-plugins-30…` at ~2026-10-07 12:29Z
positive control, control: `Follow the run` over 48 h	2026-10-07 → 2026-10-09	20

So the query reads both namespaces, and the steward stopped admitting reviews before the last 24 h. That matches the deployed configuration. The control record renders Hosting__Triage__PullRequestReview__Enabled: "false" and Hosting__PrBabysitter__Enabled: "false". /health on both control.systemorph.com and memex.systemorph.com reads pr_babysitter: state=not-armed. PR reviews now go through Copilot (policy copilot-code-review).

What this does NOT establish. The deferral itself ([Steward] … Deferred at … rate limit … at Warning) has never been seen live, because no steward admission has run since the fix was deployed. If the steward is re-enabled, the first rate-limited window is the reading to take. The other GitHub clients named above (PrBabysitter, PrFixer, PlatformPinUpdater) are also off on both portals.

Refs Systemorph/MeshWeaver.Plugins#3145

Navigation Menu

Repository navigation

PR steward's generated-only check fails open to a full review when the GitHub App installation hits its API rate limit — no retry/backoff on 403 #6216

What is failing

Probable cause

Impact

Where to look

Evidence

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

PR steward's generated-only check fails open to a full review when the GitHub App installation hits its API rate limit — no retry/backoff on 403 #6216

Description

What is failing

Probable cause

Impact

Where to look

Evidence

Activity

rbuergi commented on Oct 8, 2026

rbuergi commented on Oct 9, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions