Skip to content

docs(selfhost): verify + document the shared backend's concurrency model - #8071

Closed
davion-knight wants to merge 1 commit into
JSONbored:mainfrom
davion-knight:doc-test-backend-concurrency-model
Closed

docs(selfhost): verify + document the shared backend's concurrency model#8071
davion-knight wants to merge 1 commit into
JSONbored:mainfrom
davion-knight:doc-test-backend-concurrency-model

Conversation

@davion-knight

Copy link
Copy Markdown
Contributor

What & why

The AMS local-store concurrency guarantees were originally designed for two local processes sharing one SQLite file. #7175 migrated that layer onto the shared SelfHostD1Database seam (src/selfhost/backend-contracts.ts, #4010), which now has two interchangeable adapters — SQLite (d1-adapter.ts) and Postgres (pg-adapter.ts). This PR verifies and documents what concurrency those adapters actually guarantee, so the hosted service's assumptions are stated explicitly instead of inherited implicitly from the old local-file design.

Deliverables

1. Documented concurrency modelsrc/selfhost/backend-concurrency-model.md. Records, per adapter, what is guaranteed (batch() all-or-nothing atomicity; no lost updates for atomic single-statement writes; committed batches apply in order; reads never see an uncommitted intermediate) and what is not (non-atomic read-modify-write across awaits; cross-process SQLite sharing — out of scope for the single-process-per-deployment topology the admission system already assumes). Every claim cites the exact code it describes.

2. A deterministic verification testtest/unit/selfhost-d1-concurrency.test.ts. Exercises concurrent read/write against the real createD1Adapter(nodeSqliteDriver(...)) seam (opened with the production PRAGMAs) and asserts: N concurrent atomic increments lose no updates; batch() rolls back fully on a failing statement; a committed batch applies in order; a read concurrent with a batch never observes a rolled-back intermediate; and — as the documented hazard — a non-atomic read-modify-write does lose updates.

Why this test shape

The SQLite backend's real topology is single-process with a synchronous driver, so its guarantees are verified deterministically in-process — no external dependency, no flakiness. Real multi-connection Postgres concurrency needs a live server, so it stays behind the existing PG_TEST_URL-gated test/integration/selfhost-pg.test.ts rather than a scripted mock that cannot exhibit real races. No production source changes.

Closes #4942

The AMS local-store concurrency guarantees were designed for two local
processes sharing one SQLite file. JSONbored#7175 migrated that layer onto the shared
SelfHostD1Database seam (SQLite + Postgres adapters), so the guarantees the
hosted service now relies on need to be verified against the real seam and
stated explicitly rather than inherited implicitly from the old local-file
design.

Adds src/selfhost/backend-concurrency-model.md documenting what each adapter
guarantees (batch() atomicity, no-lost-update for atomic writes) and what it
does not (non-atomic read-modify-write, cross-process SQLite sharing), with the
guarantees pinned by a deterministic in-process test of the SQLite seam
(test/unit/selfhost-d1-concurrency.test.ts) and cross-referencing the existing
PG_TEST_URL-gated integration suite for the Postgres side. The SQLite backend's
real topology is single-process/synchronous, so its guarantees are verified
in-process with no external dependency and no flakiness; real multi-connection
Postgres concurrency stays behind the existing gated integration test rather
than a scripted mock that cannot exhibit real races.

Closes JSONbored#4942
@davion-knight
davion-knight requested a review from JSONbored as a code owner July 22, 2026 18:38
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@loopover-orb loopover-orb Bot added the gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. label Jul 22, 2026
@loopover-orb

loopover-orb Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Caution

🛑 LoopOver review result - fixes required

Review updated: 2026-07-22 18:49:35 UTC

2 files · 1 AI reviewer · no blockers · CI failing · blocked

🛑 Suggested Action - Fix Blockers

Review summary
This PR is pure documentation plus a new deterministic unit test — no production code is touched. The test exercises the real createD1Adapter/nodeSqliteDriver seam and correctly reasons about the synchronous-driver/async-wrapper interleaving to prove atomic single-statement writes lose no updates, batch() is all-or-nothing, and non-atomic read-modify-write does lose updates; the doc's atomicity claims are cited to d1-adapter.ts/pg-adapter.ts line references that match the reference code shown. Since the diff adds no runtime code paths, there is no reachable-path defect to flag.

Nits — 5 non-blocking
  • test/unit/selfhost-d1-concurrency.test.ts:16-19: `PRAGMA journal_mode = WAL;` is a no-op on a `:memory:` database (SQLite ignores WAL for in-memory DBs), so the comment claiming this 'matches the deployed configuration' is slightly misleading — worth a one-line caveat or drop the pragma from the in-memory case.
  • src/selfhost/backend-concurrency-model.md cites specific line numbers (e.g. `d1-adapter.ts:75-90`, `~171 data-access call sites`) that will silently go stale as the adapter evolves — consider a lighter-weight anchor (function name only) or accept the doc will need periodic re-verification.
  • The linked issue Re-evaluate the local concurrency model for a shared-service context #4942 is noted as only partially covered by this PR per the external brief — worth confirming with the maintainer whether remaining scope is intentionally deferred or should be called out in the PR description.
  • Consider adding a short 'last verified against commit/PR' note at the top of backend-concurrency-model.md so future adapter changes have an obvious trigger to re-check the doc.
  • The CI failures on validate/validate-tests (no detail provided) are most plausibly explained by this branch being 1 commit behind the current default branch rather than a defect in this diff — rebasing would help confirm.

CI checks failing

  • validate
  • validate-tests (3)
  • validate-tests (2)
  • validate-tests (1)

Decision drivers

  • ✅ Code review — No blockers (1 reviewer)
  • ✅ Gate result — Passing (No configured blocker found.)
Context & advisory signals — never blocks the verdict
Signal Result Evidence
Linked issue ✅ Linked #4942
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 35 registered-repo PR(s), 19 merged, 0 issue(s).
Contributor context ✅ Confirmed Gittensor contributor davion-knight; Gittensor profile; 35 PR(s), 0 issue(s).
Improvement ℹ️ Insufficient signal risk: clean · value: insufficient-signal · LLM: moderate
Linked issue satisfaction

Addressed
The PR adds an explicit concurrency-model document covering both SQLite and Postgres adapters (guarantees and non-guarantees) and a concurrency test exercising concurrent atomic writes, batch atomicity/rollback, ordering, and read-during-write isolation, directly matching both deliverables. The Postgres load test is deferred to an existing PG_TEST_URL-gated integration suite rather than a new one,

Review context
  • Author: davion-knight
  • Role context: outside_contributor
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: Python, Rust
  • Official Gittensor activity: 35 PR(s), 0 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Keep the PR focused and include validation evidence before maintainer review.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
  • @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

  • Re-run LoopOver review

@loopover-orb

loopover-orb Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

LoopOver is closing this pull request on the maintainer's behalf (CI is failing (validate, validate-tests (3), validate-tests (2), validate-tests (1))). This is an automated maintenance action — to pursue this change, please open a new pull request with the issues resolved. Closed PRs may be analyzed later to improve review accuracy, but they are not automatically reopened or re-reviewed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Re-evaluate the local concurrency model for a shared-service context

2 participants