investigate: unexpected checkpoint lock contention in isolated revert test

[DecisionNerd]

Release disposition

WriterBusy is the documented nonblocking contention/backpressure result. It means this mutation could not immediately obtain exclusive ownership of the relevant writer or checkpoint lock, so the operation failed visibly before completing the requested mutation.

This issue is not a v0.5.0 release blocker. The observed intermittent test failure did not demonstrate data corruption, loss of an acknowledged write, partial mutation, silent success, or a violation of the public WriterBusy contract. Issue #275 remains open to identify the unexpected lock owner, lock lifetime, test interaction, or environmental condition.

Observation

The exact-SHA Rust Non-Cypher Surface Gate failed on 99ae08bae2496a5aff4b1e5ecf10f64cdfd0d5fd in run 30682040773.

repository::tests::repository_snapshot_checkpoint_revert_reopens_and_replays failed at crates/graphforge-api/src/repository.rs:3121 when revert_to_checkpoint returned:

Project { code: WriterBusy, message: "checkpoint mutation could not acquire checkpoints.lock" }

The same source passed:

• the exact focused test 20/20 times locally with one test thread;
• the full graphforge-api library suite locally in parallel: 459 passed, 0 failed;
• prior ordinary exact-head CI.

Investigation outcomes

• A deterministic regression test reproduces unexpected WriterBusy from checkpoint/revert under the relevant same-process or cross-process schedule, or deterministic evidence rules out the suspected schedule.
• The lock owner/lifetime and failure mechanism are identified with deterministic evidence, if the condition can be reproduced.
• Any fix preserves the public WriterBusy contention/backpressure contract and does not conceal failures with retries, sleeps, skips, or test-suite serialization.
• Focused checkpoint/revert tests and the full graphforge-api --lib --no-fail-fast suite pass for any code change.

Non-goals

• Treating WriterBusy itself as a defect or weakening its nonblocking behavior.
• Publication, tags, GitHub Release creation, or registry writes.
• Unrelated checkpoint API changes.

[coderabbitai]

🔗 Related PRs

#208 - ci: minimize Blacksmith cache storage [merged]
#217 - feat(api): add embedded write modes [merged]
#220 - feat(bindings): expose embedded write modes [merged]
#234 - feat: add portable project interchange [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[DecisionNerd]

Diagnostic evidence

On exact source 99ae08bae2496a5aff4b1e5ecf10f64cdfd0d5fd:

• Focused exact test with --test-threads=1: 20/20 passed.
• Full cargo test -p graphforge-api --lib --no-fail-fast: 459 passed, 0 failed in 138.73s with normal parallelism.
• The test uses a unique tempdir() repository at .graphforge/state; no shared fixed project path was found.
• open_checkpoint_generation holds its shared checkpoint lock only through registry/generation verification and drops it on return.
• The production mutation path deliberately uses nonblocking try_lock_exclusive for writer.lock then checkpoints.lock; changing this to retry/blocking or serializing the test suite would weaken/hide the contention contract and is not proposed.

A single controlled CI reproduction attempt is being run on the unchanged SHA to establish whether the Blacksmith failure repeats. A green reproduction does not itself prove a root-cause fix.