Skip to content

fix(explorer): stop the file filter claiming it searched everything - #21612

Closed
nplez1 wants to merge 1 commit into
stablyai:mainfrom
nplez1:fix/explore-file-filter-partial-results
Closed

nplez1 wants to merge 1 commit into
stablyai:mainfrom
nplez1:fix/explore-file-filter-partial-results

Conversation

@nplez1

@nplez1 nplez1 commented Sep 19, 2026

Copy link
Copy Markdown

ELI5

The file filter in the file browser was reading only the first page of a very long list of files, and it never said so — so a file further down that list came back as "no files match this filter". On a local workspace it was worse than that: the filter threw away its own results and returned nothing for any search.

Now the filter asks the computer for the files that match what you typed, wherever they are in the workspace. When there are more matches than it will show, it says how many it found and how many it is showing, and asks you to type a bit more.

What Changed

Before, on a local workspace:

  • Typing anything into the Explore pane's Names filter returned zero rows and "No files match this filter" — every query, regardless of whether the file existed.
  • Filtering ran over a full workspace listing capped at 20 001 paths, truncated in ripgrep's traversal order, with the truncation flag never rendered. On a large workspace a real file could read as "no match" with no hint that the search was partial.
  • Quick Open ranked that capped listing in the renderer.

After:

  • The local filter returns matches from anywhere in the workspace. With more matches than the page holds it shows "Showing the first N of M matches — add more of the name to narrow it down".
  • When the host cannot count matches (SSH, older remote hosts) it shows "Only part of this workspace was searched — add more of the name to narrow it down". "No files match this filter" is now reachable only for a scan that completed.
  • Quick Open takes the same host-side path and renders the flag the host reports, instead of inferring truncation from files.length >= 20 001.

Mechanism

  • Matching moves to the execution host, which is where SSH and remote runtimes already did it; the local window was the only one filtering a truncated page in the renderer.
  • QuickOpenPathRanker gains a sibling, NameFilterPathMatcher, in src/shared/quick-open-path-search.ts. The Explore filter keeps its documented substring-AND semantics (it is deliberately not pointed at the fuzzy ranker), counts every match, and retains the lexicographically first page so the tree stays a stable sorted prefix rather than an arbitrary subset.
  • searchQuickOpenFilePaths takes a mode (quick-open | name-filter) over one ripgrep pass; the shared tokenizer/predicate is now used by both the host matcher and the renderer projection, so the two cannot drift.
  • New fs:searchFilePaths IPC returns { files, totalCount, truncated } and shares the fs:listFiles cancellation registry, so a superseded keystroke kills the previous scan. searchRuntimeFilePaths uses it for a local workspace; an empty query keeps the unscoped listing, so Quick Open still lists files before you type.
  • Remote runtimes and legacy inventories now pass through the totalCount their replies already carried; their wire format is unchanged.

QUICK_OPEN_LISTING_MAX_RESULTS is untouched.

Why

The reported symptom had two independent causes, and the tempting fix — remove the 20 001 limit — is wrong:

  1. The filter's projection was gated on a query-scoped listing (resolvedQuery === query). A local listing is unscoped (resolvedQuery === undefined), so every local query resolved to []. This alone explains zero results for any query.
  2. Filtering then depended on a truncated listing. The cap is an OOM bound from fix(memory): bound OOM-prone accumulators across Orca #10179: QUICK_OPEN_READDIR_MAX_FILES aliases it and the readdir fallback throws past it, the relay clamps to it via resolveQuickOpenResultLimit, and the runtime command path imports it. Raising it would also re-introduce the cost it exists to prevent — the renderer fuzzy-ranks the whole retained array per keystroke. It changes what remote hosts publish to clients that have already shipped.

So the bound stays, and a filtered query stops using the bounded listing at all: it gets its own query-scoped search with a smaller page. That is the design the remote paths already used, now shared with local.

Alternatives considered: raising the cap (rejected above); client-side paging with cursors (rejected — paging over a mutating filesystem needs snapshot tokens and host-side caches, which is the memory the bound exists to bound, and typing one more character is better UX than a "load more" button); streaming results over a new RPC opcode (rejected for now — docs/reference/remote-wire-compatibility.md Rule 2 says a new stream opcode must be capability-negotiated, which is the follow-up stage rather than this one).

Design, contract, staging, and the test strategy are written up in docs/reference/file-path-filtering-at-scale.md (linked from AGENTS.md, and allow-listed in .gitignore the way that file's own convention requires for a tracked reference doc).

Linked Issue

Fixes #21611

Visual Proof

Pending — no before/after captured. This change is visible in the Explore pane (a partial-result notice replaces a false "No files match"), but this worktree has no rendered-app harness, so it is verified at the unit and integration level instead: see Testing. Screenshots for the two states are easy to produce for anyone with a workspace over ~20 000 files, and I will attach them on request.

Testing

  • pnpm tc — clean (node/cli/web projects).

  • 9 780 tests pass across src/renderer/src/components/right-sidebar, src/renderer/src/runtime, src/renderer/src/web, src/main/ipc, src/relay (1 073 files, 18 skipped).

  • src/main/ipc/filesystem-search-file-paths-real.test.ts is new and drives real ripgrep over a temp workspace: 20 002 filler files plus src/a/b/drover.eve_schema, asserting the query-scoped name-filter search returns that file with totalCount: 1 — the reported scenario. Plus a broad-query case (bounded sorted page + exact count + truncated) and a fuzzy-mode regression case. The suite skips itself when no rg binary is present.

  • check:code-quality:changed, check:max-lines-ratchet, verify:localization-catalog, verify:localization-extraction, check:readme-local-links — all pass.

  • Platforms: macOS only. The SSH/relay and runtime paths are deliberately unchanged, and the added code is path-agnostic (no new shell, shortcut, or path handling). Windows/Linux and a real SSH host were not run locally — CI covers the unit matrix.

  • Not run locally: pnpm lint in full, pnpm build.

  • I manually tested these changes locally

  • Automated tests added/updated, or explained why not below

AI Disclosure

Implemented with an agent (DeepSeek v4.1 Flash in the pi harness), with a design consultation from Kimi K3 (Fireworks) and a review pass from a stronger advisor model. The human maintainer reported the bug, chose the approach among the presented options, and reviewed the result.

Review

Agent skill upstream boundary

  • Not applicable, or this change follows docs/reference/agent-skill-sharing-upstream-boundary.md and copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.

Notes

  • Security: the new IPC channel authorizes through the same resolveAuthorizedPath as fs:listFiles, honours the same excludePaths (nested worktrees) and the same hidden-dir/node_modules blocklist, and reuses the sender-scoped cancellation registry. No new file access is granted.
  • Cross-platform: verified only on macOS. wslAwareSpawn and the existing path normalization are untouched; the new matcher compares lowercased /-separated relative paths, as the renderer already did.
  • Remote SSH / mobile: the SSH path is unchanged (it still returns truncated without a count, which the pane now reports honestly). Paired-web gained a files.searchPaths implementation for the new API method. An exact name-filter total over RPC, with version negotiation and a legacy fallback, is stage 2 in the design doc.
  • Backwards compatibility: fs:searchFilePaths is a new channel (same-version local IPC). Remote wire format unchanged; totalCount was already in the runtime reply.
  • Performance: a local keystroke now triggers a debounced (120 ms) ripgrep scan that is aborted on the next keystroke, instead of filtering one capped listing client-side. Host memory is bounded by the page size, not the workspace size — 5 000 paths for the filter, 32 for Quick Open — and the renderer no longer holds or fuzzy-ranks a 20 000-path array per keystroke.

Checklist

  • This PR is small and focused
  • I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
  • Before/after screenshots or videos attached for UI changes, or N/A with reason
  • Self-reviewed for correctness, security, and performance
  • Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
  • pnpm lint, pnpm typecheck, pnpm test, and pnpm build pass (or CI will cover; local preferred)

The Explore pane's filename filter and Quick Open searched local files by scanning
the whole workspace and stopping at 20 001 paths in ripgrep's traversal order.
Whether a file was inside that prefix was arbitrary, and the Explore pane never
showed the truncation flag, so a filter could answer a file that exists with "No
files match this filter" — the reported bug, on a repository large enough that a
deep file fell outside the prefix. A local filter was also broken outright: its
projection was gated on a query-scoped listing, and a local listing is unscoped,
so every local query resolved to zero rows.

Both panes now ask the execution host for the files that match the query, which is
where SSH and remote runtimes already did it:

- `searchQuickOpenFilePaths` takes a matcher mode: the existing fuzzy ranker for
  Quick Open, and a new `NameFilterPathMatcher` for the Explore filter. The matcher
  keeps the pane's substring-AND semantics, counts every match, and retains the
  lexicographically first page, so the tree stays a stable sorted prefix.
- `fs:searchFilePaths` exposes that to the window as
  `{ files, totalCount, truncated }`, sharing the `fs:listFiles` cancellation
  registry so a superseded keystroke kills the previous scan.
- A local workspace takes that path once the query is non-empty; an empty query
  keeps the unscoped listing, so Quick Open still shows files before you type.
  SSH and remote runtimes are unchanged, and now also carry the `totalCount` their
  replies already contained.
- The Explore pane reports a bounded page honestly: "Showing the first N of M
  matches" when the host counted matches, "Only part of this workspace was
  searched" when it could not. "No files match this filter" is now reachable only
  for a scan that completed.

`QUICK_OPEN_LISTING_MAX_RESULTS` is untouched. It is an OOM bound from stablyai#10179,
aliased by the readdir fallback and clamped by the relay, so raising it would
re-open that hole instead of fixing the filter.

Design, contract, and staging live in docs/reference/file-path-filtering-at-scale.md,
linked from AGENTS.md.

`pnpm tc` is clean, 9 780 tests pass across right-sidebar/runtime/web/main-ipc/relay
(`filesystem-search-file-paths-real.test.ts` drives real ripgrep over 20 003 files),
and `check:code-quality:changed` reports no findings.
@nplez1 nplez1 closed this Sep 19, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

The Explore name filter now sends a limit the runtime RPC contract rejects. On a runtime-environment (or paired-web) workspace the files.searchPaths request fails schema validation and the pane shows an error instead of matches. Details inline.

Reviewed changes

  • Host-side name-filter matcher — NameFilterPathMatcher counts every substring-AND match and retains a sorted page; searchQuickOpenFilePaths selects it via a new mode.
  • New local search IPC — fs:searchFilePaths wraps searchQuickOpenFilePaths, shares the fs:listFiles cancellation registry, and returns { files, totalCount, truncated }.
  • Renderer wiring — useRuntimeFileListForWorktree gains queryMode/queryLimit/totalCount; the Explore filter asks for name-filter with a 5 000-path page and fixes the resolvedQuery === undefined gate that made every local query empty.
  • Honest empty state — getFileExplorerNameFilterEmptyMessageKind never says "No files match" for a truncated listing; a new FileExplorerNameFilterTruncationNotice reports the count.
  • Refactor/docs/tests — runtime-file-list-scan-target.ts extraction, docs/reference/file-path-filtering-at-scale.md, and new unit/integration tests including a real-ripgrep suite.

ℹ️ Legacy SSH name filter can still claim "No files match" after a partial scan

For an SSH host that lacks quick-open search, fs:listFiles answers a searchQuery from a ≤33-path legacy inventory, yet the renderer derives truncated from files.length > limit (here 5 000) and sets totalCount: null. Both stay false/absent, so getFileExplorerNameFilterEmptyMessageKind returns 'no-match' and the pane can still tell the user their file does not exist after scanning at most 33 paths — the exact claim this PR is built to remove, and the opposite of docs/reference/file-path-filtering-at-scale.md:116-117.

Technical details
# Legacy SSH path reports no truncation

## Affected sites
- `src/main/ipc/filesystem/filesystem-search-handlers.ts:201-218` — legacy branch caps `provider.listFiles` at `QUICK_OPEN_SSH_LEGACY_RESULT_LIMIT` (33) and returns `ranker.result().paths` (no count, no flag).
- `src/renderer/src/runtime/runtime-file-search-client.ts:112-120` — renderer sets `truncated: files.length > limit`; with the Explore limit of 5 000 that is never true for ≤33 paths.
- `src/renderer/src/components/right-sidebar/file-explorer-name-filter-projection.ts:31-44` — `getFileExplorerNameFilterEmptyMessageKind` keys only off `truncated`.

## Required outcome
- The pane must not show "No files match this filter" when the host answered from the 33-path legacy inventory. Signal partial-ness independently of `files.length > limit` (e.g. force `truncated` whenever the legacy branch ran, or carry a count/flag out of `fs:listFiles`).

## Open questions for the human
- Is this branch intentionally deferred to stage 2, or is the doc's "SSH reports `truncated` without a count" the intended behavior? If deferred, the PR body and doc should say the guarantee does not hold for pre-quick-open-search hosts.

ℹ️ Runtime-environment name filter still runs the fuzzy matcher

searchRuntimeFilePaths's environment branch hardcodes mode: 'quick-open', so a runtime environment answers the Explore filter with fuzzy QuickOpenPathRanker results while the tree then re-filters with substring-AND. The totalCount/shownCount in the new notice are therefore fuzzy-matcher values, and the multi-token case (which the fuzzy path does not tokenize) can under-count badly. The design doc defers exact remote name-filter totals to stage 2, but the numeric "Showing the first N of M matches" message is not flagged as fuzzy.

Technical details
# Fuzzy count presented as the name-filter match count

## Affected sites
- `src/renderer/src/runtime/runtime-file-search-client.ts:138-151` — RPC call hardcodes `mode: 'quick-open'`, ignoring `args.mode`.
- `src/main/runtime/rpc/methods/files.ts:38-49` — host branches on that literal and runs `searchQuickOpenFilePaths` (fuzzy).
- `src/renderer/src/components/right-sidebar/file-explorer-name-filter-projection.ts:178` — tree re-filters the fuzzy page with substring-AND.
- `src/renderer/src/components/right-sidebar/FileExplorer.tsx:248-253` — `shownCount` is the fuzzy page length.

## Required outcome
- Either forward the pane's `mode` (once the wire schema accepts it) so the host counts substring-AND matches, or suppress the numeric banner for runtime environments until stage 2 lands and show the "Only part of this workspace was searched" form instead.

ℹ️ Nitpicks

  • When truncated && files.length === 0 (reachable via the runtime legacy inventory), FileExplorer.tsx renders the truncation notice while the tree pane renders the partial-scan empty message — two copies of the same sentence.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using DeepSeek Flash (free via Pullfrog for OSS) | 𝕏

searchRuntimeFilePaths(requestContext, {
query: remoteQuery,
limit: 32,
limit: queryLimit,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

queryLimit is forwarded unchanged to searchRuntimeFilePaths, and for a runtime-environment workspace that becomes limit: 5000 on the files.searchPaths RPC. The wire schema caps limit at 32, so the dispatcher rejects the request with invalid_argument, and searchRuntimeFilePaths only falls back on method_not_found — the Explore pane shows a load error instead of matches. Paired web additionally forwards mode: 'name-filter', which the same schema rejects.

Technical details
# Name-filter params violate the `files.searchPaths` wire contract

## Affected sites
- `src/renderer/src/components/quick-open-file-list.ts:230` — `limit: queryLimit` sends the Explore page size (5 000) over the RPC.
- `src/shared/rpc-contract/files-params.ts:10-15` — `FilePathSearch` is `limit: z.number().int().positive().max(32)` and `mode: z.literal('quick-open').optional()`.
- `src/main/runtime/rpc/dispatcher-request-parsing.ts:21-26` — schema failure returns `invalid_argument`.
- `src/renderer/src/runtime/runtime-file-search-client.ts:138-151` — environment branch forwards `limit` (5 000) and hardcodes `mode: 'quick-open'`.
- `src/renderer/src/runtime/runtime-file-search-client.ts:152-172` — only `method_not_found` is recovered, so `invalid_argument` propagates to the pane's `loadError`.
- `src/renderer/src/web/preload-api/web-filesystem-api.ts:114-132` — paired web forwards both `limit: 5 000` and `mode: 'name-filter'` to the same method.

## Verification
Parsing the real schema:
- `limit: 5 000` -> `success: false`, `too_big`, `maximum: 32`.
- `mode: 'name-filter'` -> `success: false`, `invalid_value`, expected `"quick-open"`.

Pre-PR the hook hardcoded `limit: 32`, so runtime-environment workspaces regress from working to erroring. The existing renderer tests mock `searchRuntimeFilePaths`, so CI does not exercise the wire schema.

## Required outcome
- Clamp/omit the limit for the RPC path and extend the schema for a `name-filter` mode (per `docs/reference/remote-wire-compatibility.md` Rule 1, a new optional field), or keep the Explore filter off the RPC until stage 2.
- Add a test that drives the real `files.searchPaths` dispatcher with the Explore pane's params so this cannot regress silently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Explore filename filter says "No files match this filter" for files that exist in large workspaces

1 participant