Skip to content

fix(queue): coalesce scheduled RAG full re-index over pending webhook incrementals #1949

Description

@JSONbored

Parent: #1936

Problem

The scheduled six-hourly RAG full re-index (fanOutRagIndexJobs → per-repo rag-index-repo, src/queue/processors.ts:1068-1107) can run for a repo that already has a pending webhook-driven incremental re-index of changed paths for the same repo (runRagIndexJob with paths, :1124-1146). The two are not coalesced against each other (src/selfhost/queue-common.ts:652-657 keys them separately), so the same repo is re-embedded twice — wasting the free-tier vector budget and piling scheduled work on top of live webhook work.

Requirements

  • A scheduled full re-index for a repo should not run while a webhook incremental re-index for that same repo is pending/processing (and vice-versa the full supersedes queued incrementals).
  • Preserve correctness: a full re-index must still eventually run; an incremental must not be silently dropped if no full is pending.

Deliverables

  • Coalesce-scope the two rag-index-repo variants by repo so overlapping full/incremental work collapses.
  • Tests for full-over-incremental supersession and incremental-preserved-when-no-full.

Acceptance criteria

  • No double re-embedding of the same repo in one window; vector-budget spend drops on active repos.
  • RAG retrieval freshness is unchanged.

Expected outcome

Scheduled RAG indexing stops duplicating webhook-driven indexing, cutting redundant embedding cost. (Net-new from the #1936 rate-limit/resource audit, rank 14.)

Metadata

Metadata

Assignees

No one assigned

    Labels

    maintainer-onlyOwner-only work — yields no Gittensor points.roadmapOn the Wave-2 agent-layer roadmap board (project 9)

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions