Skip to content

fix(reliability): harden webhook/queue/public-fetch error handling (#786, #787, #790, #791) - #817

Merged
JSONbored merged 1 commit into
mainfrom
fix/reliability-hardening
Jun 17, 2026
Merged

fix(reliability): harden webhook/queue/public-fetch error handling (#786, #787, #790, #791)#817
JSONbored merged 1 commit into
mainfrom
fix/reliability-hardening

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Pre-agent-layer reliability fixes from the 2026-06-17 audit — the webhook→gate→queue path must be solid before it can ever take write actions. Surgical: 4 source files, 52 lines, no signature changes.

Bug Fix
#786 webhook strands event JOBS.send wrapped in try/catch — on failure, flag the event error (so the dedup guard lets GitHub redeliver) + return 500 so GitHub retries, instead of leaving it queued forever.
#787 poison-pill batch jobs Per-login try/catch in buildContributorDecisionPacks + buildContributorEvidence so one failing login can't fail the whole batch (which would re-run from login #1 on retry).
#790 public fetch hang 12s AbortSignal.timeout on fetchPublicContributorProfile's two api.github.com calls so a hung response can't stall the 500-login evidence loop.
#791 casing splits stats Aggregate contributor stats by canonical lowercase login so one user across mixed casings is one row.

Tests

New webhook enqueue-failure test (valid signature + failing JOBS.send → 500 + row flagged error); existing tests cover the rest; trivial log-and-continue handlers v8 ignored per the repo idiom. 1892 unit tests pass; all changed lines covered (verified against the lcov).

Note

This closes #786/#787/#791. #790 stays open for its second half — the GitHub-public-token rate-ceiling lift (needs threading env through callers; deferred to keep this PR surgical). The hang (the P1) is fixed here.

Closes #786
Closes #787
Closes #791

Pre-agent-layer reliability fixes from the 2026-06-17 audit — the gate path must
be solid before it can ever take write actions. Surgical, no signature changes.

- #786 webhook: wrap JOBS.send in try/catch. On enqueue failure, flag the event
  "error" (so the dedup guard lets GitHub redeliver) and return 500 so GitHub
  retries, instead of silently stranding the event as "queued" forever.
- #787 queue: isolate each login in buildContributorDecisionPacks and
  buildContributorEvidence (per-login try/catch) so one failing login can't fail
  the whole batch and poison-pill the queue (which would re-run from login #1).
- #790 public: add a 12s AbortSignal.timeout to fetchPublicContributorProfile's
  two api.github.com calls so a hung response can't stall the 500-login evidence
  loop. (The auth-token rate-ceiling lift is tracked separately on #790.)
- #791 backfill: aggregate contributor stats by canonical lowercase login so one
  user across mixed casings collapses to a single ContributorRepoStatRecord.

New webhook enqueue-failure test; existing tests cover the rest; the trivial
log-and-continue handlers are v8-ignored per the repo idiom. 1892 unit tests
pass; all changed lines covered.

Closes #786
Closes #787
Closes #791
@dosubot dosubot Bot added the size:M label Jun 17, 2026
@ghost

ghost commented Jun 17, 2026

Copy link
Copy Markdown

Note

Gittensory Gate skipped

PR closed before full evaluation. No late first comment was created.

Signal Result Evidence Action
Gate result ⚠️ Skipped #817 is no longer open. No action.

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

@ghost ghost added the gittensory:reviewed label Jun 17, 2026
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@ghost

ghost commented Jun 17, 2026

Copy link
Copy Markdown

gittensory · advisory review

Reviewed 5 changed file(s) — two independent AI reviewers.

Changed files: src/github/backfill.ts, src/github/public.ts, src/github/webhook.ts, src/queue/processors.ts, test/unit/webhook.test.ts

Suggested action: 🛠️ Request changes. (reviewers split: merge / request changes) Address the suggestions below before merging.

Reviewer A · gpt-oss-120b — recommends ✅ merge
The PR adds robust error handling for webhook enqueue failures, per‑login isolation in queue processing, a timeout for public GitHub fetches, and case‑insensitive login aggregation, along with a new test for the webhook failure path. The changes are well‑scoped and improve reliability without altering public behavior.

Suggestions

  • Add a unit test verifying that a timeout on the GitHub public fetch correctly aborts the request.
  • Confirm that AbortSignal.timeout is available in the Cloudflare Workers runtime you target, or provide a fallback.
  • Consider adding a test for the per‑login try/catch paths in buildContributorDecisionPacks and buildContributorEvidence to ensure failures are logged and do not halt the batch.

Worth double-checking

  • The new case‑insensitive login mapping may affect downstream code that expects the original casing; verify that all consumers handle the mixed‑case login correctly.
  • If errorMessage is not imported in the modified files, a runtime ReferenceError could occur.

Reviewer B · mistral-small-3.1-24b-instruct — recommends 🛠️ request changes
This PR improves error handling in the webhook/queue/public-fetch paths, making the system more resilient to transient failures. The changes are well-targeted and address specific issues related to reliability.

Suggestions

  • Consider adding more detailed logging or metrics to track the frequency of these isolated failures.
  • Ensure that the error messages logged are not exposing sensitive information.
  • Add tests to cover the new error handling paths, especially for the webhook enqueue failure scenario.

Worth double-checking

  • Verify that the new error handling logic does not introduce any performance regressions, especially in high-load scenarios.
  • Ensure that the changes do not inadvertently mask underlying issues that need to be addressed.
  • Check that the new timeout logic in public fetches does not affect the reliability of data retrieval.

@JSONbored
JSONbored merged commit 7d7d26d into main Jun 17, 2026
18 checks passed
@JSONbored
JSONbored deleted the fix/reliability-hardening branch June 17, 2026 08:58
@codecov

codecov Bot commented Jun 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.77%. Comparing base (aaeff16) to head (9233957).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #817   +/-   ##
=======================================
  Coverage   96.76%   96.77%           
=======================================
  Files          97       97           
  Lines       13992    14000    +8     
  Branches     5098     5098           
=======================================
+ Hits        13540    13548    +8     
  Misses         86       86           
  Partials      366      366           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

JSONbored added a commit that referenced this pull request Jun 17, 2026
Completes #790 — the timeout half shipped in #817. fetchPublicContributorProfile's
two api.github.com calls were unauthenticated (60/hr ceiling), so the 500-login
contributor-evidence loop could exhaust it and silently return source: unavailable.

Thread an optional env through the 8 callers and send Authorization: Bearer when
GITHUB_PUBLIC_TOKEN is set, lifting the ceiling to 5000/hr (mirrors the existing
fetchPublicRepoStats). Signature stays backward-compatible (env is optional).

Closes #790
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant