Skip to content

test(ai-review): pin isRateLimitError's 429-shape contract - #5506

Closed
carlh7777 wants to merge 1 commit into
JSONbored:mainfrom
carlh7777:test/rate-limit-error-contract
Closed

test(ai-review): pin isRateLimitError's 429-shape contract#5506
carlh7777 wants to merge 1 commit into
JSONbored:mainfrom
carlh7777:test/rate-limit-error-contract

Conversation

@carlh7777

Copy link
Copy Markdown
Contributor

Summary

Adds a direct contract test for isRateLimitError (src/services/ai-review.ts), the shared 429 short-circuit introduced in the recently-merged #5481 (#5385-sentry / GITTENSORY-K/8).

That PR exported isRateLimitError and wired it into five independent AI-calling retry loops — runWorkersOpinion + runDualAiTieBreakJudgeCall (services/ai-review.ts), review/planner.ts, services/ai-slop.ts, and services/linked-issue-satisfaction-run.ts — each of which now breaks out of its same-model retry on a 429 instead of burning the remaining per-model budget on a rate-limit window that won't have cleared a few hundred ms later. The helper shipped with no test of its own, so a future tweak to its /_(?:http|error)_429$/ matcher could silently regress the short-circuit across all five call sites at once.

The suite pins the exact contract:

  • matches every 429 shape src/selfhost/ai.ts actually throws — ai_http_429, anthropic_http_429, claude_code_error_429, ai_embed_http_429;
  • does not match a non-429 status (ai_http_400/500/503, claude_code_error_404) that a retry might legitimately clear;
  • does not match the sibling non-transient errors that have their own separate guard (subscription_cli_timeout, codex_exit_1: …);
  • respects the $ anchor (a longer status or trailing detail like ai_http_4291 / ai_http_429: too many requests is not a 429) and the _http/_error delimiter (a bare 429 / http_429 substring never trips it);
  • returns false for any non-Error value (the error instanceof Error guard).

No production code changes.

Closes #

Scope

  • The PR title follows type(scope): short summary Conventional Commit format.
  • This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
  • This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
  • I linked a currently open issue this PR resolves.

Validation

  • git diff --check
  • npm run typecheck
  • npm run test:coverage — not applicable to coverage: this is a test-only change (no src/** lines), which Codecov ignores; npx vitest run test/unit/ai-review-rate-limit-error.test.ts passes (6/6), asserting behavior verified empirically against the live regex.
  • npm run test:workers · build:mcp · test:mcp-pack · ui:* — N/A (no worker/MCP/UI/OpenAPI surface touched).
  • New tests cover the matcher's positive, negative, anchor, delimiter, and non-Error branches.

If any required check was skipped, explain why:

Test-only change adding one test/unit/** file; it touches no src/**, workflow, MCP, OpenAPI, or UI surface, so the build/UI/MCP checks are N/A and there are no coverable lines for Codecov.

Safety

  • No secrets, wallet details, hotkeys, coldkeys, PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
  • Public GitHub text stays sanitized and low-noise.
  • Auth/CORS/session negative-path tests — N/A (no such surface touched).
  • API/OpenAPI/MCP behavior updated/tested — N/A.
  • UI changes use real states — N/A (no UI).
  • Visible UI changes include UI Evidence — N/A.
  • Public docs/changelogs updated where needed — N/A.

UI Evidence

N/A — no UI/frontend/docs changes.

Notes

  • All assertions were checked against the real /_(?:http|error)_429$/ matcher before being written, so the suite documents the actual contract rather than an assumed one.

Introduces a new test suite for the isRateLimitError function to ensure it correctly identifies rate limit errors (HTTP 429) across various provider error messages. The tests cover matching valid 429 errors, rejecting non-429 errors, and validating the function's behavior with non-Error values. This addition helps prevent future regressions in error handling related to rate limits.
@carlh7777
carlh7777 requested a review from JSONbored as a code owner July 12, 2026 23:48
@superagent-security

Copy link
Copy Markdown
Contributor

🚨 Contributor flagged. Click here for more info: Superagent Dashboard

@superagent-security superagent-security Bot added the contributor:flagged Contributor flagged for review by trust analysis. label Jul 12, 2026
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.75%. Comparing base (68ce986) to head (dfbfdf1).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #5506   +/-   ##
=======================================
  Coverage   94.75%   94.75%           
=======================================
  Files         564      564           
  Lines       44896    44896           
  Branches    14675    14675           
=======================================
  Hits        42541    42541           
  Misses       1621     1621           
  Partials      734      734           
Flag Coverage Δ
shard-1 43.80% <ø> (-0.50%) ⬇️
shard-2 35.41% <ø> (+0.04%) ⬆️
shard-3 32.10% <ø> (+0.09%) ⬆️
shard-4 31.28% <ø> (-0.63%) ⬇️
shard-5 33.18% <ø> (+0.46%) ⬆️
shard-6 43.81% <ø> (+0.27%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 12, 2026
@loopover-orb

loopover-orb Bot commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Caution

🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥

🛑 Gittensory review result - reject/close recommended

Review updated: 2026-07-12 23:56:11 UTC

1 file · 2 AI reviewers · 2 blockers · readiness 100/100 · CI green · unstable

🛑 Suggested Action - Reject/Close

Review summary
This is a pure test-only addition pinning the contract of isRateLimitError, the shared 429 short-circuit used across five retry loops per the description. The test cases are well-targeted and cover the actual regex behavior described (`/_(?:http|error)_429$/`) — anchoring, delimiter requirement, non-Error guard, and disambiguation from the sibling subscription-timeout guard — with no production code touched. The PR lacks a linked issue number (the template's "Closes #<...>" placeholder was left unfilled), which is a scope-policy gap for this repo.

Blockers

  • No issue is linked — the description leaves the 'Closes #<open the issue above and put its number here>' placeholder unfilled, and this repo requires every contributor PR to close or clearly link an eligible open issue.
  • PR description: the closure line is still `Closes #<open the issue above and put its number here>`; replace it with a real eligible issue reference such as `Closes feat(orb): add POST /v1/orb/ingest central collector endpoint #1234`, or explain why this maintainer-authorized test-only PR is exempt.
Nits — 5 non-blocking
  • The file-level comment block (lines 3-11) duplicates most of what's already in the PR description; consider trimming to a one-line pointer to fix(review): stop retrying an AI provider 429 immediately with zero backoff #5481.
  • Per-test inline comments (e.g. citing ai.ts line numbers) will silently rot if ai.ts is refactored — consider referencing the exporting function name instead of line numbers.
  • Fill in the actual issue number this PR closes before merge, per repo policy.
  • Consider adding one case for an Error subclass (e.g. a custom AiHttpError) to confirm the `instanceof Error` guard covers subclasses too, if any AI-calling code throws a custom Error subtype.
  • test/unit/ai-review-rate-limit-error.test.ts:4 says the helper is shared by four independent AI-calling retry loops, but the same comment lists five call sites, so make the count match the actual claim.

Why this is blocked

  • No issue is linked — the description leaves the 'Closes #<open the issue above and put its number here>' placeholder unfilled, and this repo requires every contributor PR to close or clearly link an eligible open issue.
  • No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.
📋 Copy for AI agents — paste into your coding agent
Fix the following blocker(s) from this PR review:

1. No issue is linked — the description leaves the 'Closes \#&lt;open the issue above and put its number here&gt;' placeholder unfilled, and this repo requires every contributor PR to close or clearly link an eligible open issue.

2. No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.
Flagged checks (non-blocking)
  • Contributor trust — Contributor flagged for review
Signal Result Evidence
Code review ❌ 2 blockers 2 reviewers, synthesized
Linked issue ✅ No-issue rationale PR body explains why no issue is linked.
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (no linked issue context).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 181 registered-repo PR(s), 117 merged, 17 issue(s).
Contributor context ✅ Confirmed Gittensor contributor carlh7777; Gittensor profile; 181 PR(s), 17 issue(s).
Gate result ❌ Blocking Repo-configured hard blocker found.
Improvement ⚠️ ℹ️ Insufficient signal risk: clean · value: insufficient-signal — Nothing measurable for the structural-improvement analyzers on this PR (e.g. no code files changed). LLM value judgment: minor — It's a well-constructed, test-only contract pin for existing shared logic with no functional risk, but it's narrow in scope and doesn't move any feature or bug forward on its own.
Review context
  • Author: carlh7777
  • Role context: outside_contributor
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: Python, JavaScript, TypeScript, Cuda, HTML, MDX, Rust
  • Official Gittensor activity: 181 PR(s), 17 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Keep the PR focused and include validation evidence before maintainer review.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
[BETA] Chat with Gittensory

Ask Gittensory a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @gittensory ask &lt;question&gt; answers contribution-quality Q&A with source citations and freshness.
  • @gittensory chat &lt;question&gt; answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @gittensory mention with a real question is routed to the closest matching read-only command automatically -- no exact syntax required.

Full command reference: https://gittensory.aethereal.dev/docs/gittensory-commands

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@loopover-orb

loopover-orb Bot commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Gittensory is closing this pull request on the maintainer's behalf (No linked issue detected; AI reviewers agree on a likely critical defect: No issue is linked — the description leaves the 'Closes #' placeholder unfilled, and this repo requires every contributor PR to close or clearly link an eligible open issue.). This is an automated maintenance action — to pursue this change, please open a new pull request with the issues resolved. Closed PRs may be analyzed later to improve review accuracy, but they are not automatically reopened or re-reviewed.

@loopover-orb loopover-orb Bot closed this Jul 12, 2026
@carlh7777
carlh7777 deleted the test/rate-limit-error-contract branch July 13, 2026 04:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contributor:flagged Contributor flagged for review by trust analysis. gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant