Skip to content

Return relative names from DirFileSystem glob and find - #2127

Merged
martindurant merged 2 commits into
fsspec:masterfrom
rioyu123:codex/fix-dirfs-detailed-names
Sep 11, 2026
Merged

martindurant merged 2 commits into
fsspec:masterfrom
rioyu123:codex/fix-dirfs-detailed-names

Conversation

@rioyu123

@rioyu123 rioyu123 commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

DirFileSystem.glob(..., detail=True) and find(..., detail=True) return relative dictionary keys but leave each entry's name prefixed with the wrapped directory. Passing that name back to the same filesystem adds the root again, so listing a file and then reading it by its returned name fails.

This change sets each entry's name to its relative dictionary key in both synchronous and asynchronous listings, matching the existing info() and ls() convention. Each metadata dictionary is copied so the wrapped filesystem's results are left unchanged. Entries without a name, including the empty metadata returned for a single TAR file, now receive the relative name as well. Non-detailed results are unchanged.

Like the earlier info() fix in #1798, this keeps names usable through the wrapper. This PR is limited to glob() and find(); walk(), path joining and root-path conventions are unchanged. Code that used these metadata names directly with the underlying filesystem should instead use the wrapper or explicitly add the root.

Tests

  • Updated the existing detailed-listing tests to include name and check that the wrapped metadata is not modified.
  • Added synchronous and asynchronous tests that list a real nested file, read it using the returned name, and compare with the non-detailed listing and info() name.
  • The original-name regression checks fail against the unfixed code (10 failures on both Linux and Windows). The 7 missing-name cases fail against the previous guarded implementation and pass after the follow-up change. The real TAR case checks that the added name can be used for reading and that the backend's empty metadata is unchanged.
  • Final Windows DirFileSystem, TAR and ZIP selection: 226 passed, 1 skipped.
  • Final Linux DirFileSystem, local, memory, async-wrapper, spec, mapping, TAR and ZIP selection: 682 passed, 26 skipped, 1 xfailed.
  • Repository pre-commit checks pass.

The full Docker/FUSE/cloud/downstream matrix was not run locally.

Comment thread fsspec/implementations/dirfs.py Outdated
for path, info in paths.items():
path = self._relpath(path)
info = info.copy()
if "name" in info:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should probably always do this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that makes sense. I removed the guard so every entry gets a relative name, including when the backend returns empty metadata. The metadata is still copied, leaving the backend's result unchanged. Updated the sync/async tests and the TAR case, which also checks that the returned name can be used to read the file.

@martindurant
martindurant merged commit 8c9bded into fsspec:master Sep 11, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants