Repository navigation
Conversation
Reading ContiguousMemmapArray caches held up to two file descriptors and a mapping per cache file: the per-file np.memmap cached in each DumpContext (pinned by every loaded proxy) plus the unbuffered read handle. - ContiguousMemmap keeps the path and the view geometry only (on a shared zero skeleton buffer that is never read); ContiguousMemmapArray loads through ContiguousMemmap.from_file, without creating a memmap. - Read handles live in a per-thread LRU shared by all loads, closed beyond EXCA_CONTIGUOUS_MEMMAP_MAX_OPEN_FILES (default: half the soft RLIMIT_NOFILE, 0 for no limit). - np.asarray of a 0-d proxy keeps its shape (was (1,)).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why. Reading
ContiguousMemmapArraycaches holds up to two file descriptors and one mapping per cache file for as long as the process lives: the per-filenp.memmapcached in eachDumpContext(itsmmapkeeps a dup'd fd), which every loaded proxy also pins (so clearing the resource cache frees nothing while items are kept in RAM), plus the unbuffered read handle per file (#347). A DataLoader worker reading ~40k cache files goes past a 65,536 soft fd limit and fails withOSError: [Errno 24] Too many open files.How.
ContiguousMemmapkeeps only the path, a file-offset shift and the view geometry. The geometry lives on a shared zero "skeleton" buffer that is never read: basic indexing and view ops run on it as before, scalars are read through file I/O, and a view op that would copy raisesTypeError(it did before as well, through the memmap-backed check).ContiguousMemmapArray.__load_from_info__uses the newContiguousMemmap.from_file(path, offset, shape, dtype). It creates no memmap and opens nothing at load time.EXCA_CONTIGUOUS_MEMMAP_MAX_OPEN_FILES. The default is half the softRLIMIT_NOFILE(0 means no limit). That default keeps every file open whenever the old code still had enough fds (it needed two per file), so throughput is unchanged there. Beyond it, reads reopen files instead of failing.np.asarrayon a 0-d proxy, which returned shape(1,)because ofnp.ascontiguousarray.Tests.
test_open_files_boundedchecks that no file is open after loading, that at most the cap is open while reading, and that reads stay correct after eviction. Added a nonzero scalar read, a 0-d read, and a check that a copying view op raises. Full suite (Python 3.11, as CI runs it): mypy, ruff andtest_safeguardclean. pytest: 685 passed, 1 skipped, 2 failed. Both failures also fail on main on our Slurm cluster:test_inflight.py::test_is_alive[liveness4-700-False](Slurm answers for the fake job id) and the timing-sensitivetest_map.py::test_map_infra_cache_dict_calls(fails 14 runs out of 15 on main here, 9 out of 15 on this branch).Numbers. Real EEG and iEEG neuro-extractor caches on Lustre: one entry from each of 25,495 cache files, items kept alive, 3 passes of random 2 s windows, warm page cache.
Window checksums were identical in every run.
Caveat: a cap below the number of files read at random costs one
open()per miss, about 0.8 ms on Lustre. A forced cap of 256 gave ~800 windows/s (1 process) and ~12,500 (16 processes). Hence the limit-relative default instead of a small fixed one.