Skip to content

fix(rees): standardize bounded external fetches #1812

Description

@JSONbored

Context

Some REES analyzers already have strong timeout/fanout controls, but the behavior is uneven across GitHub, npm, PyPI, OSV, deps.dev, endoflife, and other external calls.

This issue creates one bounded external I/O layer so every analyzer degrades consistently instead of hanging, over-fetching, or surfacing generic timeout failures.

Parent: #1499

Deliverables

  • Add a shared bounded fetch helper for REES external calls.
  • Support parent abort signal, per-subcall timeout, max response bytes, endpoint category, and safe diagnostics.
  • Add request-scoped in-flight de-duplication for identical URLs/payloads where safe.
  • Add per-host/category call counters and caps.
  • Migrate existing external-call analyzers to the shared helper.
  • Ensure slow or oversized responses degrade the affected analyzer only.

Requirements

  • Endpoint categories should be safe labels such as github-commits, github-search, npm-packument, pypi-json, osv-query, deps-dev, and endoflife.
  • Diagnostics may include counts, elapsed time, caps, endpoint category, and reason codes.
  • Diagnostics must not include tokens, full URLs with sensitive query params, request bodies, diffs, raw prompts, comments, or private config values.
  • Avoid unbounded retries inside review-time budgets.
  • Preserve existing analyzer output compatibility.

Acceptance criteria

  • Every analyzer that calls the network uses the shared bounded fetch layer or has an explicit documented exception.
  • Slow external calls produce degraded analyzer status instead of request-level failure.
  • Oversized external responses are capped and reported through safe reason codes.
  • Sentry context can attribute REES failures to analyzer, phase, subcall, endpoint category, and capped/degraded reason.

Validation

  • Unit tests for timeout, abort, max bytes, safe diagnostics, and in-flight de-duplication.
  • Focused analyzer regression tests for at least two migrated external analyzers.
  • Sentry/safe-context test proving sensitive payload fields are excluded.

Ownership

Maintainer-only. No gittensor:* labels.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

Status
Done

Relationships

None yet

Development

No branches or pull requests

Issue actions