You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
cachedict inflight registry: sqlite 'file is not a database' under concurrent job-array access #304
When a grid of experiments is submitted as a single slurm job array (via TaskInfra.job_array()), and each job independently accesses a shared cached step (same folder) that computes a large set of items, exca's cachedict inflight registry sqlite database gets corrupted:
sqlite3.DatabaseError: file is not a database
The corruption then cascades into FileNotFoundError on full-uid.yaml and submitit.core.utils.UncompletedJobError for the sub-tasks, failing the whole run.
~18 experiments submitted together via TaskInfra.job_array().
Each job calls into the same shared cached pipeline (a Cached step backend + a map that computes ~432 items) pointing at the same cache folder.
All ~18 jobs start ~simultaneously, so many concurrently query/populate the same inflight registry.
Pre-warming the cache once (single builder) before submitting the array does not prevent it: the concurrent jobs still re-open/validate the shared registry and corrupt it.
Stacktrace
WARNING - exca.cachedict.inflight:349 - Waiting for 432 in-flight items (of 432 requested) held by: <jobid> [slurm:RUNNING] x432
WARNING - exca.cachedict.registry:234 - Inflight registry query: file is not a database
Traceback (most recent call last):
File ".../exca/cachedict/registry.py", line 222, in _safe_execute
return self._retry_on_lock(run)
File ".../exca/cachedict/registry.py", line 130, in _retry_on_lock
return fn()
File ".../exca/cachedict/registry.py", line 211, in run
return fn(conn)
File ".../exca/cachedict/inflight.py", line 316, in _do
rows = registry.select_in_chunks(
File ".../exca/cachedict/registry.py", line 49, in select_in_chunks
out.extend(conn.execute(sql, batch).fetchall())
sqlite3.DatabaseError: file is not a database
Also observed alongside it:
FileNotFoundError: [Errno 2] No such file or directory: '.../<cache-uid>/full-uid.yaml'
submitit.core.utils.UncompletedJobError: Job <jobid> (task: 0) with path .../logs/<jobid>/<jobid>_0_result.pkl
Expected
Concurrent readers/writers of a shared cachedict (typical when a job array shares one cache folder) should not corrupt the sqlite inflight registry. Either serialize/guard registry writes robustly across processes, tolerate concurrent access, or fail loudly with a clear message instead of silently corrupting the DB.
Notes
Happy to help repro. Workaround we used: disable the Slurm timelines sub-infra so items are computed in-process (removing the concurrent writers to that registry).
Pre-warm worked — the study built once cleanly and the 18-job sweep launched ("study cache ready; submitting 18 jobs"). But the run still failed because each of the 18 concurrent jobs re-runs study.run(), which re-accesses the study's sqlite Cached backend + Slurm timelines builder concurrently → sqlite3.DatabaseError: file is not a database. Pre-warm alone isn't enough: exca's sqlite study cache isn't safe under 18-way concurrent access (the workers re-validate/rebuild it, not just read).
#220 — replace the flaky JobChecker with an SQLite advisory registry (journal_mode=DELETE, BEGIN IMMEDIATE claims). #224 — general robustness hardening of the inflight path. #226 — batch the worker-info updates (_QUERY_BATCH_SIZE, single BEGIN) to fix slow DB updates under many items. #235 — add a second advisory DB (errors.db) for Step error caching, same pattern. #263 — factor the shared SQLite plumbing into a reusable AdvisoryRegistry base. #273 — best-effort chmod on transient DB files (avoid crashing on permission races). #276 — better logging around inflight DB operations. #290 — fix an infinite poll in the wait loop. #298 — fix the update_worker_info freeze (per-row autocommit → NFS fcntl/-journal/fsync storm parking in D-state): batch it into one BEGIN…COMMIT and switch DELETE→WAL to kill the per-transaction journal-sidecar churn. The WAL half ignored the "unsafe on NFS" caveat, which is #304.
Activity
jrapin commentedon Jul 13, 2026
For the record,
neuralsetconfigand coment
Pre-warm worked — the study built once cleanly and the 18-job sweep launched ("study cache ready; submitting 18 jobs"). But the run still failed because each of the 18 concurrent jobs re-runs study.run(), which re-accesses the study's sqlite Cached backend + Slurm timelines builder concurrently → sqlite3.DatabaseError: file is not a database. Pre-warm alone isn't enough: exca's sqlite study cache isn't safe under 18-way concurrent access (the workers re-validate/rebuild it, not just read).
jrapin commentedon Jul 13, 2026
#220 — replace the flaky JobChecker with an SQLite advisory registry (journal_mode=DELETE, BEGIN IMMEDIATE claims).
#224 — general robustness hardening of the inflight path.
#226 — batch the worker-info updates (_QUERY_BATCH_SIZE, single BEGIN) to fix slow DB updates under many items.
#235 — add a second advisory DB (errors.db) for Step error caching, same pattern.
#263 — factor the shared SQLite plumbing into a reusable AdvisoryRegistry base.
#273 — best-effort chmod on transient DB files (avoid crashing on permission races).
#276 — better logging around inflight DB operations.
#290 — fix an infinite poll in the wait loop.
#298 — fix the update_worker_info freeze (per-row autocommit → NFS fcntl/-journal/fsync storm parking in D-state): batch it into one BEGIN…COMMIT and switch DELETE→WAL to kill the per-transaction journal-sidecar churn. The WAL half ignored the "unsafe on NFS" caveat, which is #304.
Advisory registries: reconnect instead of resetting on mid-op file co…