Skip to content

queue: generateSignalSnapshots has no per-repo failure isolation, unlike its job-dispatch.ts sibling (#8355) #9293

Description

@JSONbored

⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.

Context

generateSignalSnapshots in src/queue/signal-snapshot.ts (~line 65) iterates repos with a plain
for (const repo of repositories) loop and no per-repo error isolation. If any repo's DB
reads/writes throw partway through (the Promise.all batch of listIssueSignalSample /
listOpenPullRequests / etc., or the later persistSignalSnapshot call, has no .catch), every
subsequent repo in that same invocation's batch silently never gets its snapshot generated, and
the caller just sees one rejected promise for the whole run — there's no way to tell which repo
failed or that the rest were skipped as a side effect.

This is reachable with a multi-repo batch via /v1/internal/jobs/generate-signal-snapshots/run
(src/api/routes.ts, repoFullName is optional there) and via the requestedBy === "test" bypass
in job-dispatch.ts's "generate-signal-snapshots" case. In normal production the sibling
fanOutRepoSignalSnapshotJobs fans this out into one isolated queue job per repo, so this
multi-repo path is rarely exercised with more than one repo — but it's a real, reachable gap.
test/unit/queue-trends.test.ts only exercises single-repo calls, no multi-repo/partial-failure
scenario.

The precedent for the correct fix already exists in the same file family:
src/queue/job-dispatch.ts's "backfill-registered-repos" case (#8355) explicitly switched from
Promise.all to Promise.allSettled with per-item failure tracking, for the identical reason —
one repo's failure must never silently block or skip its siblings in the same fan-out.

Requirements

  • generateSignalSnapshots must isolate per-repo failures: one repo's error (from either the
    Promise.all data-gathering step or the persistSignalSnapshot write) must not prevent any
    other repo in the same invocation from being processed.
  • Failures must be surfaced, not silently swallowed — collect the failing repo full names and
    throw a single aggregate error listing them (matching fix(queue): backfill-registered-repos cron fan-out has no per-repo isolation, risking duplicate dispatch on partial failure #8355's pattern: attempt every repo
    exactly once regardless of an earlier one's outcome, then report which ones failed) so the
    invocation is still marked failed for observability, and log a structured error per failed repo
    (e.g. console.error(JSON.stringify({ level: "error", event: "generate_signal_snapshots_repo_failed", repoFullName, reason }))).
  • Do not change fanOutRepoSignalSnapshotJobs or the per-repo queue-job fan-out path — this issue
    is only about isolating failures within a single generateSignalSnapshots invocation's own
    multi-repo loop.

Deliverables

  • generateSignalSnapshots in src/queue/signal-snapshot.ts isolates per-repo failures using
    the same Promise.allSettled + collected-failures + aggregate-throw pattern as
    job-dispatch.ts's "backfill-registered-repos" case (fix(queue): backfill-registered-repos cron fan-out has no per-repo isolation, risking duplicate dispatch on partial failure #8355).
  • A new test in test/unit/queue-trends.test.ts (or a new adjacent test file) covers a
    multi-repo call where one repo's data-gathering or persistence throws, asserting: (a) the
    other repo(s) in the same batch still get their snapshot generated/persisted, and (b) the
    function still surfaces an error identifying the failed repo.

Both deliverables are required in this single PR.

Test Coverage Requirements

This repo enforces 99%+ Codecov patch coverage, branch-counted, on every changed line/branch in
src/**. The new isolation logic and its failure/success branches must be covered by the new test
above.

Expected Outcome

A multi-repo generateSignalSnapshots invocation where one repo's data-gathering or persistence
throws no longer silently skips every subsequent repo in the batch — each repo is attempted
independently, successes persist regardless of a sibling's failure, and failures are identifiable
in the surfaced error.

Links & Resources

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions