Skip to content

cachedict inflight registry: sqlite 'file is not a database' under concurrent job-array access #304

Open

Description

@sdascoli

Summary

When a grid of experiments is submitted as a single slurm job array (via TaskInfra.job_array()), and each job independently accesses a shared cached step (same folder) that computes a large set of items, exca's cachedict inflight registry sqlite database gets corrupted:

sqlite3.DatabaseError: file is not a database

The corruption then cascades into FileNotFoundError on full-uid.yaml and submitit.core.utils.UncompletedJobError for the sub-tasks, failing the whole run.

Environment

  • exca 0.5.27
  • Python 3.12, Linux, slurm cluster (submitit backend)

Scenario

  • ~18 experiments submitted together via TaskInfra.job_array().
  • Each job calls into the same shared cached pipeline (a Cached step backend + a map that computes ~432 items) pointing at the same cache folder.
  • All ~18 jobs start ~simultaneously, so many concurrently query/populate the same inflight registry.
  • Pre-warming the cache once (single builder) before submitting the array does not prevent it: the concurrent jobs still re-open/validate the shared registry and corrupt it.

Stacktrace

WARNING - exca.cachedict.inflight:349 - Waiting for 432 in-flight items (of 432 requested) held by: <jobid> [slurm:RUNNING] x432
WARNING - exca.cachedict.registry:234 - Inflight registry query: file is not a database
Traceback (most recent call last):
  File ".../exca/cachedict/registry.py", line 222, in _safe_execute
    return self._retry_on_lock(run)
  File ".../exca/cachedict/registry.py", line 130, in _retry_on_lock
    return fn()
  File ".../exca/cachedict/registry.py", line 211, in run
    return fn(conn)
  File ".../exca/cachedict/inflight.py", line 316, in _do
    rows = registry.select_in_chunks(
  File ".../exca/cachedict/registry.py", line 49, in select_in_chunks
    out.extend(conn.execute(sql, batch).fetchall())
sqlite3.DatabaseError: file is not a database

Also observed alongside it:

FileNotFoundError: [Errno 2] No such file or directory: '.../<cache-uid>/full-uid.yaml'
submitit.core.utils.UncompletedJobError: Job <jobid> (task: 0) with path .../logs/<jobid>/<jobid>_0_result.pkl

Expected

Concurrent readers/writers of a shared cachedict (typical when a job array shares one cache folder) should not corrupt the sqlite inflight registry. Either serialize/guard registry writes robustly across processes, tolerate concurrent access, or fail loudly with a clear message instead of silently corrupting the DB.

Notes

Happy to help repro. Workaround we used: disable the Slurm timelines sub-infra so items are computed in-process (removing the concurrent writers to that registry).

Activity

  1. jrapin commented on Jul 13, 2026

    @jrapin
    Contributor

    For the record, neuralset config

    study = {
                    "name": "Lebel2023Natural",
                    "path": DATADIR,
                    "timelines": {
                        "infra": {
                            "backend": "Slurm",
                            "min_items_per_job": 20,  # ~22 jobs for 432 timelines
                            "max_jobs": 2048,
                            "folder": CACHEDIR,
                            "partition": SLURM_PARTITION,
                            "cpus_per_task": 10,
                            "timeout_min": 60 * 4,  # Lebel TextGrid/audio loading is slow
                        },
                    },
                    "infra": {
                        "backend": "Cached",
                        "folder": CACHEDIR,
                    },
                }

    and coment

    Pre-warm worked — the study built once cleanly and the 18-job sweep launched ("study cache ready; submitting 18 jobs"). But the run still failed because each of the 18 concurrent jobs re-runs study.run(), which re-accesses the study's sqlite Cached backend + Slurm timelines builder concurrently → sqlite3.DatabaseError: file is not a database. Pre-warm alone isn't enough: exca's sqlite study cache isn't safe under 18-way concurrent access (the workers re-validate/rebuild it, not just read).

  2. jrapin commented on Jul 13, 2026

    @jrapin
    Contributor

    #220 — replace the flaky JobChecker with an SQLite advisory registry (journal_mode=DELETE, BEGIN IMMEDIATE claims).
    #224 — general robustness hardening of the inflight path.
    #226 — batch the worker-info updates (_QUERY_BATCH_SIZE, single BEGIN) to fix slow DB updates under many items.
    #235 — add a second advisory DB (errors.db) for Step error caching, same pattern.
    #263 — factor the shared SQLite plumbing into a reusable AdvisoryRegistry base.
    #273 — best-effort chmod on transient DB files (avoid crashing on permission races).
    #276 — better logging around inflight DB operations.
    #290 — fix an infinite poll in the wait loop.
    #298 — fix the update_worker_info freeze (per-row autocommit → NFS fcntl/-journal/fsync storm parking in D-state): batch it into one BEGIN…COMMIT and switch DELETE→WAL to kill the per-transaction journal-sidecar churn. The WAL half ignored the "unsafe on NFS" caveat, which is #304.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions