Skip to content

MemoryFileSystem: create the file when appending to a missing path - #2062

Merged
martindurant merged 2 commits into
fsspec:masterfrom
Sreekant13:fix/memoryfs-append-create-missing
Jul 6, 2026
Merged

martindurant merged 2 commits into
fsspec:masterfrom
Sreekant13:fix/memoryfs-append-create-missing

Conversation

@Sreekant13

Copy link
Copy Markdown
Contributor

Opening a path that doesn't exist yet in append mode behaves differently on MemoryFileSystem than on LocalFileSystem and the builtin open:

import fsspec
fsspec.filesystem("file").open("/tmp/new.txt", "ab")   # creates the file
fsspec.filesystem("memory").open("/new.txt", "ab")     # FileNotFoundError

open(path, "ab") creates the file if it's missing, and LocalFileSystem inherits that, so having the memory backend raise FileNotFoundError here is surprising. This makes _open create an empty file when the path isn't in the store yet for append modes (a/ab/a+b), then position at the end as before. Read modes (rb/r+b) are unchanged and still raise.

Added a regression test covering the three append modes.

Opening a non-existent path in append mode ("a"/"ab"/"a+b") on
MemoryFileSystem raised FileNotFoundError, unlike builtin open() and
LocalFileSystem, which create the file. Create it in that case, and add
a regression test.
Comment on lines +203 to +205
m = MemoryFile(self, path, kwargs.get("data"))
if not self._intrans:
m.commit()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should make sure to seek to the end of the file. In the test, it is already at the end, but this is not guaranteed.

Per review: explicitly position the append-created file at the end, matching
the existing-file append path, instead of relying on the new file being empty.
Assert the position in the regression test.
@Sreekant13

Copy link
Copy Markdown
Contributor Author

Good point @martindurant, thanks! Pushed a commit that seeks to the end explicitly after creating the file, so it matches the existing-file append path and no longer relies on the new file being empty. Also updated the regression test to assert the position is at the end.

@martindurant
martindurant merged commit 0907962 into fsspec:master Jul 6, 2026
21 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants