Make unknown semantic and hybrid search totals explicit in structured responses

[phernandez]

Summary

search_notes returns total: 0 for semantic and hybrid queries even when the current page contains results. The implementation intentionally uses zero as an unknown-total sentinel to avoid a second vector retrieval, but the structured response does not tell clients that the value is unknown.

This caused the 2026-07-16 development MCP acceptance run to classify working pagination as a regression until the source contract was inspected.

Reproduction

1. Create at least two searchable notes.
2. Call search_notes(search_type="semantic", page=1, page_size=1, min_similarity=0, output_format="json").
3. Call page 2 with the same query.
4. Observe distinct results and has_more=true, while both responses report total: 0.

The same behavior occurs for hybrid search.

Investigation

The behavior is currently intentional:

• src/basic_memory/api/v2/routers/search_router.py skips the exact count for non-FTS retrieval, fetches one probe row, and sets total = 0.
• src/basic_memory/mcp/tools/search.py documents that vector and hybrid callers should use has_more.
• Existing CLI tests describe the value as an unknown total.

The problem is therefore the machine-readable contract, not retrieval or pagination.

Suggested fix

Represent unknown totals explicitly. Options include:

• Change total to int | null and return null for non-exact modes.
• Add total_is_exact: bool or total_status: "exact" | "unknown" while preserving total: 0 temporarily for compatibility.
• Update MCP descriptions, API schemas, CLI rendering, and regression tests together.

Prefer an additive compatibility field first if changing total would break existing clients.

Acceptance criteria

• A structured client can distinguish an exact zero from an unknown total without reading prose documentation.
• Semantic and hybrid pagination continues to use the probe-row strategy and does not add a second vector retrieval.
• Text, title, and permalink searches continue to report exact totals.
• MCP, HTTP API, and CLI tests cover both exact and unknown totals.

Source

Development MCP acceptance run on 2026-07-16. Deployment matched basic-memory revision 796607fdfd0db7a5978d3d07fb71417938ec6bf9.

[claude]

Claude finished @phernandez's task in 7m 9s —— View job

---

Triage & Investigation

Todo list:

• Gather context and read issue
• Investigate search_router.py / search.py unknown-total behavior
• Check for duplicate issues (none found)
• Decide triage labels
• Implement fix (additive total_is_exact field) across API schema, MCP tool, CLI rendering
• Add/update tests
• Run lint/typecheck/tests — blocked, see note below
• Push branch and provide PR link — blocked, see note below

Triage

• Type: Enhancement
• Priority: Medium
• Component: API, MCP
• Complexity: Medium

Implementation

Added an additive total_is_exact: bool = True field to SearchResponse (src/basic_memory/schemas/search.py) so structured clients can distinguish an exact zero from an unknown probe-pagination total without reading docs. total itself is unchanged for backward compatibility.

• api/v2/routers/search_router.py — sets total_is_exact=exact_count_available (True for text/title/permalink FTS, False for vector/hybrid probe pagination).
• mcp/tools/search.py — _result_total and the multi-project aggregator (_search_all_projects) now key off total_is_exact instead of the old total > 0 heuristic; the merged multi-project total is marked unknown if any searched project's total was unknown. Docstring for search_notes updated to document the new field.
• cli/commands/tool.py — _search_result_summary prefers the explicit total_is_exact flag (falling back to the legacy heuristic only for payloads that predate the field, e.g. cached fixtures), so bm tool search-notes shows a real "page X of Y" count for exact totals and the existing safe fallback otherwise.
• Added/updated tests in tests/schemas/test_search.py, tests/api/v2/test_search_router.py (FTS + vector + new hybrid case), tests/mcp/tools/test_search_notes_multi_project.py (including a new test for cross-project total-exactness propagation), and tests/cli/test_cli_tool_rich_output.py (rich + plain, exact and unknown cases).

⚠️ Two blockers I could not resolve in this session

1. Verification tooling is sandboxed: every attempt to run just fast-check, ruff, pytest, or even a bare python3 -c ... in this environment returned "This command requires approval" with no user available to grant it. I was not able to run lint, typecheck, or the test suite. I did a careful manual review of the diff (syntax, blank-line formatting, logic) but this needs to be verified by CI or a maintainer running just fast-check && just fast-test locally.
2. Push denied: git commit -s succeeded locally (commit a0ce94b on claude/issue-1068-20260716-0302), but pushing failed:

   remote: Permission to basicmachines-co/basic-memory.git denied to github-actions[bot].
   fatal: unable to access '...': The requested URL returned error: 403
   

   The token available in this session identifies as github-actions[bot] rather than the Claude app installation, and lacks write access to this repo. This looks like a workflow/token configuration issue rather than something fixable from within the sandbox — someone with repo admin access will need to check the GitHub App installation permissions or the workflow's token source. Once push access is restored, the branch claude/issue-1068-20260716-0302 already has the fix committed locally in this run and just needs git push (or the workflow re-run) to land it.