Skip to content

fix(queue): bound rag path coalesce keys - #1934

Merged
JSONbored merged 1 commit into
mainfrom
codex/fix-unbounded-rag-path-coalesce-key
Jul 1, 2026
Merged

fix(queue): bound rag path coalesce keys#1934
JSONbored merged 1 commit into
mainfrom
codex/fix-unbounded-rag-path-coalesce-key

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Motivation

  • The rag-index-repo coalesce key embedded a JSON-serialized list of changed paths directly into the indexed job_key, which can grow unbounded and cause PostgreSQL btree index tuple insertion failures for self-hosted Postgres deployments.
  • The goal is to keep the same semantic coalescing (same normalized path scope should coalesce) while ensuring the stored key is length-bounded and safe to index.

Description

  • Hash sorted, deduplicated RAG path scopes with SHA-256 in normalizedPathScope() and import createHash from node:crypto so the coalesce key uses a bounded sha256:<hex> representation for path scopes.
  • Preserve the empty-path fallback and existing rag-index-repo coalescing semantics (use full when no valid paths are present).
  • Update unit tests in test/unit/selfhost-queue-common.test.ts and test/unit/selfhost-sqlite-queue.test.ts to assert hashed keys, distinguish comma-containing paths, verify long attacker-influenced path lists produce a bounded SHA key, and verify empty-path fallback behavior.

Testing

  • Ran the targeted unit tests with npx vitest run test/unit/selfhost-queue-common.test.ts test/unit/selfhost-sqlite-queue.test.ts, which passed.
  • Ran npx vitest run test/unit/selfhost-pg-queue.test.ts and npm run typecheck, both of which passed for the modified surfaces.
  • Attempted the full gate npm run test:ci and npm run test:coverage, but the full-suite run encountered unrelated infrastructure/timeouts (actionlint binary download/DNS, several long-running test timeouts); these failures are not caused by the bounded key change and the targeted queue tests passed.
  • npm audit --audit-level=moderate could not complete due to the registry audit endpoint returning 403 Forbidden in this environment.

Codex Task

@codecov

codecov Bot commented Jul 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.56%. Comparing base (725831b) to head (46d51c4).
⚠️ Report is 5 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1934   +/-   ##
=======================================
  Coverage   95.56%   95.56%           
=======================================
  Files         217      217           
  Lines       24139    24140    +1     
  Branches     8749     8749           
=======================================
+ Hits        23068    23070    +2     
  Misses        438      438           
+ Partials      633      632    -1     
Files with missing lines Coverage Δ
src/selfhost/queue-common.ts 90.70% <100.00%> (+0.48%) ⬆️
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@loopover-orb

loopover-orb Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Tip

🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩

✅ Gittensory review result - approve/merge recommended

Review updated: 2026-07-01 00:30:32 UTC

3 files · 1 AI reviewer · no blockers · readiness 86/100 · CI green · clean

✅ Suggested Action - Approve/Merge

  • safe to merge

Review summary
The change correctly bounds `rag-index-repo` coalesce keys by hashing the same sorted, trimmed, deduplicated path scope that previously went directly into the key. Empty or invalid path lists still fall back to `full`, so the visible coalescing contract is preserved while avoiding oversized indexed job keys. The added tests cover duplicate normalization, comma-containing paths, long path lists, and the empty-path fallback.

Nits — 4 non-blocking
  • nit: `test/unit/selfhost-queue-common.test.ts:563` hard-codes SHA-256 literals without documenting the normalized preimage, so future failures will be harder to diagnose when normalization changes intentionally.
  • nit: `src/selfhost/queue-common.ts:652` embeds the digest prefix inline; extracting a small helper or constant would make future key-versioning less error-prone if the representation changes again.
  • In `test/unit/selfhost-queue-common.test.ts:563`, add a short local helper or comment that ties each expected digest to the normalized sorted JSON preimage being hashed.
  • In `src/selfhost/queue-common.ts:652`, consider a `hashedPathScope(paths)` helper so the `normalizedPathScope` flow stays focused on validation/normalization and the key format is centralized.
Signal Result Evidence
Code review ✅ No blockers 1 reviewer
Linked issue ⚠️ Missing No linked issue or no-issue rationale found.
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (no linked issue context).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 68 registered-repo PR(s), 58 merged, 242 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 68 PR(s), 242 issue(s).
Gate result ✅ Passing No configured blocker found.
Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository registration is not available in the local Gittensory cache.
  • Public profile languages: not available
  • Official Gittensor activity: 68 PR(s), 242 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Treat this as maintainer-lane context rather than normal contributor-lane activity.
  • Explain no-issue PR.
  • Triage stale or unlinked PRs.
  • No action.
  • Link the issue being solved, or explicitly explain why this is a no-issue PR.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@loopover-orb loopover-orb Bot added gittensor gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. labels Jul 1, 2026
@JSONbored JSONbored self-assigned this Jul 1, 2026
@JSONbored
JSONbored merged commit 9e2f533 into main Jul 1, 2026
13 checks passed
@JSONbored
JSONbored deleted the codex/fix-unbounded-rag-path-coalesce-key branch July 1, 2026 00:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

No open projects
Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant