Skip to content

ContiguousMemmap: no memmap per instance, bounded file handles - #360

Draft
sdascoli wants to merge 1 commit into
mainfrom
sdascoli/bounded-file-handles
Draft

sdascoli wants to merge 1 commit into
mainfrom
sdascoli/bounded-file-handles

Conversation

@sdascoli

Copy link
Copy Markdown
Contributor

Why. Reading ContiguousMemmapArray caches holds up to two file descriptors and one mapping per cache file for as long as the process lives: the per-file np.memmap cached in each DumpContext (its mmap keeps a dup'd fd), which every loaded proxy also pins (so clearing the resource cache frees nothing while items are kept in RAM), plus the unbuffered read handle per file (#347). A DataLoader worker reading ~40k cache files goes past a 65,536 soft fd limit and fails with OSError: [Errno 24] Too many open files.

How.

  • ContiguousMemmap keeps only the path, a file-offset shift and the view geometry. The geometry lives on a shared zero "skeleton" buffer that is never read: basic indexing and view ops run on it as before, scalars are read through file I/O, and a view op that would copy raises TypeError (it did before as well, through the memmap-backed check).
  • ContiguousMemmapArray.__load_from_info__ uses the new ContiguousMemmap.from_file(path, offset, shape, dtype). It creates no memmap and opens nothing at load time.
  • Read handles: a per-thread LRU in the module-level cache, shared by all loads, which closes handles beyond EXCA_CONTIGUOUS_MEMMAP_MAX_OPEN_FILES. The default is half the soft RLIMIT_NOFILE (0 means no limit). That default keeps every file open whenever the old code still had enough fds (it needed two per file), so throughput is unchanged there. Beyond it, reads reopen files instead of failing.
  • Also fixes np.asarray on a 0-d proxy, which returned shape (1,) because of np.ascontiguousarray.

Tests. test_open_files_bounded checks that no file is open after loading, that at most the cap is open while reading, and that reads stay correct after eviction. Added a nonzero scalar read, a 0-d read, and a check that a copying view op raises. Full suite (Python 3.11, as CI runs it): mypy, ruff and test_safeguard clean. pytest: 685 passed, 1 skipped, 2 failed. Both failures also fail on main on our Slurm cluster: test_inflight.py::test_is_alive[liveness4-700-False] (Slurm answers for the fake job id) and the timing-sensitive test_map.py::test_map_infra_cache_dict_calls (fails 14 runs out of 15 on main here, 9 out of 15 on this branch).

Numbers. Real EEG and iEEG neuro-extractor caches on Lustre: one entry from each of 25,495 cache files, items kept alive, 3 passes of random 2 s windows, warm page cache.

exca 0.6.1 this PR
open fds / mappings 50,990 / 25,495 25,495 / 0
at soft limit 8,192 crash (EMFILE) after 4,094 items 4,096 fds, completes
windows/s, 1 process ~2,760 ~2,890
windows/s, 16 processes (aggregate) ~43,800 ~45,000
load time 57–63 s 48–53 s
RSS 0.83 GB 0.78 GB

Window checksums were identical in every run.

Caveat: a cap below the number of files read at random costs one open() per miss, about 0.8 ms on Lustre. A forced cap of 256 gave ~800 windows/s (1 process) and ~12,500 (16 processes). Hence the limit-relative default instead of a small fixed one.

Reading ContiguousMemmapArray caches held up to two file descriptors and a
mapping per cache file: the per-file np.memmap cached in each DumpContext
(pinned by every loaded proxy) plus the unbuffered read handle.

- ContiguousMemmap keeps the path and the view geometry only (on a shared
  zero skeleton buffer that is never read); ContiguousMemmapArray loads
  through ContiguousMemmap.from_file, without creating a memmap.
- Read handles live in a per-thread LRU shared by all loads, closed beyond
  EXCA_CONTIGUOUS_MEMMAP_MAX_OPEN_FILES (default: half the soft
  RLIMIT_NOFILE, 0 for no limit).
- np.asarray of a 0-d proxy keeps its shape (was (1,)).
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Nov 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant