Skip to content

orb(upstream): the raw-github fallback is the only fetch in the file with no timeout, so a stalled CDN connection hangs the whole drift refresh #9165

Description

@JSONbored

Problem

fetchTrackedSource's raw-GitHub fallback is the only fetch in the file without a timeout
(src/upstream/ruleset.ts:520):

const response = await fetch(rawUrl(config, source.path), { headers: githubHeaders({ token: env.GITHUB_PUBLIC_TOKEN, accept: "text/plain" }) });

The primary contents-API read at :490 uses timeoutFetch, which carries
AbortSignal.timeout(GITHUB_FETCH_TIMEOUT_MS) (12s, src/github/client.ts:18, :466). The other six
call sites in this file (:33, :490, :1093, :1123, :1136, :1153) all use timeoutFetch too.
Node's fetch has no default timeout.

Trigger

raw.githubusercontent.com accepts the TCP connection but never responds — a common CDN failure mode, and
precisely the case this fallback exists to handle. refreshUpstreamSourceSnapshots awaits Promise.all
over all six TRACKED_SOURCES (:110-112), so one stalled source hangs the whole refresh. And it only
reaches the fallback because the primary already failed — i.e. exactly when GitHub is degraded and all
six sources are likely to take the same path.

Impact

The scheduled refreshUpstreamDrift job hangs indefinitely instead of degrading to a status: "error"
snapshot with previous?.parsed retained. Upstream drift silently stops being detected, and the hung job
occupies a worker slot.

Requirements

  • Use timeoutFetch at :520 (or add signal: AbortSignal.timeout(GITHUB_FETCH_TIMEOUT_MS)), matching
    the primary path.
  • Sweep for any other bare fetch( in src/ outside the client helpers and bring them under the same
    timeout, so this class cannot recur.
  • Confirm the Promise.all fan-out degrades per-source rather than all-or-nothing.

Test Coverage Requirements

99%+ patch coverage, branch-counted; the timeout arm asserted to produce an error snapshot rather than
hanging.

Links & Resources

  • src/upstream/ruleset.ts ~110-112, ~490, ~520; src/github/client.ts ~18, ~466

maintainer-only — liveness.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions