Skip to content

repo-clone.js's ensureRepoCloned lock is in-process only; races across fleet-mode's sibling containers #7084

Description

@JSONbored

Context

packages/loopover-miner/DEPLOYMENT.md documents two supported deployment shapes: "laptop mode (single
machine, zero Docker) and fleet mode (containerized workers with a shared data volume)". Fleet mode's own
docker-compose setup runs multiple separate containers — each its own OS process — against the SAME bind-mounted
LOOPOVER_MINER_CONFIG_DIR/state volume. packages/loopover-miner/lib/claim-ledger.js's own comment (around
its claimIssueWithinCap function) independently confirms this exact concurrency model already matters
elsewhere in the package: "two sibling miner processes racing the same repo."

packages/loopover-miner/lib/repo-clone.js's ensureRepoCloned (lines 106-124) serializes concurrent
git fetch/checkout/reset --hard calls against the same per-repo base clone via withRepoCloneLock
(lines 83-104) — but that lock is a plain JavaScript Map (repoCloneLocks) living in ONE Node process's
memory. Its own header comment (lines 75-83) explicitly frames the problem it fixed (#6762) as same-repo
concurrency in general — "two attempts for the SAME repo... without this two same-repo attempts can interleave
git subprocesses on the same .git dir and corrupt the index/HEAD/refs or trip .git/index.lock" — without ever
distinguishing "two attempts in the same process" from "two attempts in two sibling fleet-mode containers", even
though the latter is the exact scenario DEPLOYMENT.md documents as a first-class, shipped deployment mode and
claim-ledger.js already treats as real elsewhere in this same package. An in-process Map provides zero
serialization between two separate container processes sharing the same clone directory: two fleet workers
racing the same target repo can still both reach ensureRepoCloned at the same instant, each with an empty
local repoCloneLocks Map (since Maps aren't shared across processes), and both run git checkout/
git reset --hard against the same working tree concurrently.

Requirements

  • ensureRepoCloned (packages/loopover-miner/lib/repo-clone.js) MUST serialize concurrent mutations against
    the same repoPath across SEPARATE OS PROCESSES, not only within one Node process's event loop — e.g. via an
    OS-level exclusive lockfile (open(lockPath, 'wx')-style create-and-hold, or an equivalent proper-lockfile-
    style advisory lock) taken on a deterministic path derived from the resolved repoPath before any git mutation
    runs, held for the duration of the clone/fetch/checkout/reset sequence, and released (including on error) once
    it completes.
  • The existing in-process repoCloneLocks Map-based serialization MUST be retained (or subsumed by the new
    cross-process lock) so same-process concurrent calls remain at least as safe as they are today — this is an
    additive fix, not a regression of No concurrency guard on the shared per-repo base clone lets concurrent attempts race git fetch/checkout/reset --hard #6762's existing protection.
  • A losing process MUST wait for the lock (bounded, with a timeout that fails closed with a clear
    repo_clone_lock_timeout-style error) rather than either racing ahead or hanging indefinitely.
  • The lockfile/lock resource MUST be cleaned up (or self-expire) so a process that crashes mid-clone does not
    permanently wedge every future attempt against that repo — mirror this package's existing crash-safety
    convention (local-store.js's registerCleanupResource on SIGINT/SIGTERM, worktree-allocator.js's stale-
    allocation reclaim) rather than inventing an unbounded, unrecoverable lock.

Deliverables

  • ensureRepoCloned in packages/loopover-miner/lib/repo-clone.js takes a real cross-process lock (in
    addition to the existing in-process one) before mutating the shared base-clone directory.
  • A regression test simulating two concurrent "processes" (two independent calls with no shared in-memory
    state, e.g. by clearing/bypassing the in-process repoCloneLocks Map between them, or by spawning the
    exported lock helper twice against a real temp lockfile) against the same repoPath, asserting the second
    caller's git mutations only begin after the first's complete rather than interleaving.
  • A test covering the crash-recovery path: a stale/orphaned lock (simulating a killed process) does not
    permanently block a subsequent ensureRepoCloned call.

Test Coverage Requirements

99%+ Codecov patch coverage on every changed line and branch in repo-clone.js, including the lock-acquired,
lock-contended/wait, lock-timeout, and stale-lock-recovery branches, plus the regression test above proving the
fix actually serializes across two independent (non-shared-memory) callers, not just within one process's Map.

Expected Outcome

Two fleet-mode miner containers (or any two separate loopover-miner OS processes) racing an attempt against
the same target repo can no longer interleave git fetch/checkout/reset --hard on the shared base clone —
the second process waits for the first to finish instead of corrupting the shared working tree or getting an
opaque .git/index.lock failure, closing the gap #6762's in-process-only fix left open for the package's own
documented fleet-mode deployment.

Links & Resources

  • packages/loopover-miner/lib/repo-clone.js:75-124withRepoCloneLock/ensureRepoCloned, the in-process-only
    fix to extend.
  • packages/loopover-miner/DEPLOYMENT.md — "fleet mode (containerized workers with a shared data volume)".
  • packages/loopover-miner/lib/claim-ledger.js (claimIssueWithinCap's own comment) — this exact "sibling miner
    processes" concurrency model already documented as real elsewhere in the package.
  • No concurrency guard on the shared per-repo base clone lets concurrent attempts race git fetch/checkout/reset --hard #6762 — the prior in-process fix this issue extends to the cross-process case.

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions