Skip to content

docs(miner): document and load-test shared store concurrency model - #8002

Closed
andriypolanski wants to merge 1 commit into
JSONbored:mainfrom
andriypolanski:docs/4942-shared-store-concurrency-model
Closed

docs(miner): document and load-test shared store concurrency model#8002
andriypolanski wants to merge 1 commit into
JSONbored:mainfrom
andriypolanski:docs/4942-shared-store-concurrency-model

Conversation

@andriypolanski

Copy link
Copy Markdown
Contributor

Summary

Closes #4942

Change

  • Doc packages/loopover-miner/docs/ams-shared-store-concurrency-model.md: what SQLite BEGIN IMMEDIATE / busy_timeout, miner createD1Adapter.batch, and ORB createPgAdapter (READ COMMITTED) guarantee — and what they do not (SERIALIZABLE, installation admission, two loops on one dir, miner-already-on-Postgres). Contrasts installation-concurrency-admission / mapWithConcurrency. Notes control-plane one-DB-per-tenant isolation.
  • Links from operations-runbook.md and ams-storage-abstraction-research.md.
  • Load / race tests test/unit/miner-shared-store-concurrency.test.ts + claim-within-cap-child.mjs:
    • Cross-process claimIssueWithinCap (cap=1, 8 distinct issues → exactly one active row)
    • 32× concurrent UPDATE … SET n = n + 1 via createD1Adapter.batch → exact count
    • Naive interleaved RMW schedule pins the documented lost-update non-guarantee
    • Optional Postgres suite (PG_TEST_URL) for the same atomic increment via createPgAdapter

Validation

  • git diff --check clean.
  • npx vitest run test/unit/miner-shared-store-concurrency.test.ts test/unit/miner-operations-runbook.test.ts — green (PG case skipped without PG_TEST_URL).
  • Docs / fixtures / unit tests only — no src/** or miner lib/** executable changes; Codecov patch has no graded diff lines.

UI Evidence

N/A — documentation + concurrency correctness tests only.

@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@loopover-orb loopover-orb Bot added the gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. label Jul 22, 2026
@loopover-orb

loopover-orb Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Caution

🛑 LoopOver review result - fixes required

Review updated: 2026-07-22 06:59:15 UTC

6 files · 1 AI reviewer · no blockers · CI failing · blocked

🛑 Suggested Action - Fix Blockers

Review summary
This PR adds a maintainer doc (`ams-shared-store-concurrency-model.md`) restating SQLite/Postgres concurrency guarantees for the shared-service model, links it from the runbook and storage-abstraction research doc, and backs it with a cross-process load-test suite (claimIssueWithinCap race, SqliteDriver/createD1Adapter atomic increments, an optional Postgres suite, and a deliberately-sequential 'lost update' demonstration). The diff is scoped correctly to docs+tests with no src/lib changes, closes #4942 as required, and the doc content matches what the new assertions check for (BEGIN IMMEDIATE, READ COMMITTED, SqliteDriver, installation-concurrency-admission, map-with-concurrency.ts substring). The most notable wrinkle is that the 'concurrent' SqliteDriver increment test drives `Promise.all` over calls into a synchronous driver, so it can't produce genuine interleaving in Node's single-threaded model — it mainly re-verifies sequential additive correctness rather than a real race, unlike the cross-process child-process test which does.

Nits — 5 non-blocking
  • test/unit/miner-shared-store-concurrency.test.ts's 'N concurrent atomic UPDATE' SqliteDriver test wraps synchronous `adapter.batch()` calls in `Promise.all`, which cannot produce genuine thread interleaving in Node — it's effectively sequential execution dressed as concurrency, unlike the real cross-process `claimIssueWithinCap` test above it; consider noting this distinction in the test description or doc's 'Load / race verification' section so readers don't over-read it as a true race test.
  • test/unit/miner-shared-store-concurrency.test.ts unconditionally imports `pg` and `../../src/selfhost/pg-adapter` at module scope even though the Postgres suite is skipped without `PG_TEST_URL` — confirm `pg` is already a resolvable devDependency for this workspace so the whole file (including the always-run SQLite tests) doesn't fail to load in environments without it.
  • The two failing 'validate-tests' CI checks and 'validate' check are listed with no detail provided, so their cause could not be verified from what's given here — worth checking the actual CI logs before merge since the PR description claims a green local run.
  • Consider asserting the doc's `mapWithConcurrency` guarantee text with a more literal match (currently the test's `map-with-concurrency` substring is satisfied only by the file path `map-with-concurrency.ts`, not the camelCase symbol name used in prose) so the assertion tracks the intended prose claim rather than an incidental file-path substring.
  • In the 'naive app-level RMW' test, a short comment noting this is a deterministic single-threaded demonstration (not a true race) rather than empirical timing-based interleaving would make the non-guarantee documentation clearer for future readers.

CI checks failing

  • validate
  • validate-tests (1)
  • validate-tests (3)
  • validate-tests (2)

Decision drivers

  • ✅ Code review — No blockers (1 reviewer)
  • ✅ Gate result — Passing (No configured blocker found.)
Context & advisory signals — never blocks the verdict
Signal Result Evidence
Linked issue ✅ Linked #4942
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 199 registered-repo PR(s), 128 merged, 22 issue(s).
Contributor context ✅ Confirmed Gittensor contributor andriypolanski; Gittensor profile; 199 PR(s), 22 issue(s).
Improvement ℹ️ Insufficient signal risk: clean · value: insufficient-signal · LLM: moderate
Linked issue satisfaction

Addressed
The PR adds an explicit concurrency-model doc distinguishing SQLite/D1-adapter and Postgres pg-adapter guarantees (and non-guarantees) versus the old single-file assumptions, and includes cross-process/concurrent load tests (claim cap, atomic increments, a naive RMW lost-update demonstration) verifying no lost updates under real concurrent access, matching both deliverables in the issue.

Review context
  • Author: andriypolanski
  • Role context: outside_contributor
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: Python, TypeScript, JavaScript, Rust, Cuda, Kotlin, MDX, Scala
  • Official Gittensor activity: 199 PR(s), 22 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Keep the PR focused and include validation evidence before maintainer review.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
  • @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

  • Re-run LoopOver review

@loopover-orb

loopover-orb Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

LoopOver is closing this pull request on the maintainer's behalf (CI is failing (validate, validate-tests (1), validate-tests (3), validate-tests (2))). This is an automated maintenance action — to pursue this change, please open a new pull request with the issues resolved. Closed PRs may be analyzed later to improve review accuracy, but they are not automatically reopened or re-reviewed.

@loopover-orb loopover-orb Bot closed this Jul 22, 2026
andriypolanski pushed a commit to andriypolanski/gittensory that referenced this pull request Jul 22, 2026
…JSONbored#4942)

Keep only the three coverage-hitting split test files so scoped
--coverage.all=false shards each produce non-empty lcov (closes the

Co-authored-by: Cursor <cursoragent@cursor.com>
JSONbored#8002 empty-shard failure mode).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Re-evaluate the local concurrency model for a shared-service context

2 participants