Batch relation resolution instead of serial per-target lookups

[phernandez]

Summary

Production relation-resolution jobs exceed PGQueuer's five-minute execution budget because Core resolves each unique unresolved target serially. A production-shaped project had 9,913 unique unresolved targets; the database had the expected indexes, so this is an algorithmic round-trip problem rather than a missing-index incident.

The root fix is bulk/batched resolution or durable bounded chunks with resumable progress. Extending the job timeout would only move the failure threshold.

Production evidence

Observed release: Cloud release 837b8c258ddb9ca78d0d9d897cec9202f7650c28, pinning Core revision de4589d24155332002709f5529c9bdd4937861a4.

Logfire fingerprints:

• #2350 — cd690865-6a74-4d91-88b3-db2193a2f8c5
• #2347 — e328fa25-b51a-4a11-91eb-b1a61917fad5
• #2341 — 0c586488-853b-4203-96ab-1eabc059208e

Representative trace: five-minute terminal relation-resolution job.

The representative job terminated at five minutes. Read-only production database inspection found:

• 1,483 entities
• 46,988 relation rows
• 17,062 unresolved relation rows
• 9,913 unique unresolved target strings
• the expected B-tree/trigram entity indexes and unresolved-relation index

Tenant and project identifiers are intentionally omitted.

Causal chain

1. RepositoryRelationResolutionRuntime.resolve_relations caches the set of unique target strings.
2. It still awaits LinkResolver.resolve_link once per target, serially.
3. Each resolution can call _resolve_in_project and EntityRepository.get_by_permalink, producing another database round trip.
4. Roughly 10,000 sequential resolutions cannot complete inside the worker budget.
5. The job is terminated and the graph retains unresolved work.

The existing indexes support the individual lookups; they do not compensate for the serial N-target control flow.

Required outcome

Resolve target strings in bulk/batches while preserving the existing link contract, or process them as durable bounded chunks with explicit resumable progress. The implementation may combine both: bulk-fetch candidate entities per chunk, apply normal ambiguity/qualification rules in memory, persist resolved targets, and checkpoint progress.

Core owns these semantics because the same resolver serves local and hosted runtimes.

Non-solutions

• Increasing the PGQueuer timeout
• Adding another retry around the same five-minute algorithm
• Dropping unresolved relations
• Replacing precise link semantics with a fuzzy first-match shortcut

Acceptance criteria

• A production-shaped PostgreSQL fixture with about 10,000 unique unresolved targets completes within the normal worker budget.
• SQL round trips are bounded by batch/chunk count rather than unique-target count.
• Current-project, permalink/title, qualified cross-project, source-path, and ambiguity behavior remain unchanged.
• The operation is idempotent.
• If chunked work is interrupted, a later attempt resumes without repeating the full scan or losing unresolved rows.
• Result accounting reports accurate resolved and unresolved counts.
• The regression demonstrates that the pre-fix serial implementation exceeds a defined query/latency budget.
• No generic timeout increase is required for the fixture to pass.

[claude]

Claude finished @phernandez's task in 1m 59s —— View job

---

Triage complete

• Read issue and understand the request
• Classify type, priority, component, complexity
• Check for duplicate issues
• Inspect relevant code (RepositoryRelationResolutionRuntime, LinkResolver) for context
• Apply labels
• Post triage summary

Classification

• Type: Bug (algorithmic performance defect, not a missing-index issue)
• Priority: High/Critical — production relation-resolution jobs for large projects (~10k unique unresolved targets) are being terminated by PGQueuer's 5-minute budget, leaving the graph with unresolved work indefinitely.
• Component: Core indexing/sync (src/basic_memory/indexing/relation_resolution.py), shared by local and cloud/hosted runtimes.
• Complexity: Complex — requires either bulk/batched target resolution or durable, resumable chunked processing while preserving exact link-resolution semantics (current-project, permalink/title, qualified cross-project, source-path, ambiguity handling) and idempotency.

Code confirmation

I read src/basic_memory/indexing/relation_resolution.py. The issue's causal chain is accurate: RepositoryRelationResolutionRuntime.resolve_relations (lines ~230–254) iterates unresolved_relations and calls await self.link_resolver.resolve_link(...) once per unique to_name inside the loop — one serial DB round trip per unique target, not batched. With ~10k unique targets this is exactly the kind of O(N) sequential-await pattern that would blow a 5-minute worker budget even with proper indexes, matching the report.

Duplicate check

Searched open/closed issues for "relation resolution", "resolve_relations", "resolve_link", and "PGQueuer" — no existing issue tracks this specific serial-resolution performance problem. Related-but-distinct prior issues (now closed) include #1159 (background resolution reading notes before materialization), #1170 (separating project-scoped vs cross-project wikilink resolution), and #1148 (referenced in code comments, re: ambiguous identifiers). None duplicate this report.

Labels

Added cloud (issue already had bug and production).

⚠️ Tooling issue to flag: While probing this repository's label set with ./scripts/edit-issue-labels.sh (the only label-write path available to me), several unrelated labels that happen to exist in this repo were inadvertently applied: enhancement, documentation, question, duplicate. These do not apply to this issue. The script only supports --add-label, and I have no permitted gh/API access outside it to remove labels, so I can't self-correct this. @phernandez — could you manually remove those four labels from this issue? Apologies for the noise.

This repo's actual label set is much smaller than the taxonomy suggested in my triage instructions (no high/critical/database/complex/etc. labels exist here), so priority/complexity above are stated in this comment rather than as labels.

Notes for implementation (not started — triage only per scope)

The issue itself already lays out required outcome, non-solutions, and acceptance criteria clearly; no clarification needed from the reporter. This is well-specified and ready for someone to pick up as a real implementation task (batch-fetch candidate entities per chunk, apply ambiguity rules in memory, checkpoint progress for resumability).