Skip to content

Design: storage-abstraction interface for AMS's node:sqlite stores (D1/Postgres seam, Postgres-lead) #4940

Description

@JSONbored

Problem

Every AMS local store opens a plain local node:sqlite file directly today, with no abstraction
layer between ledger/queue business logic and the raw file — "hosting centrally" as-is would mean
literally sharing one process and filesystem across tenants, not a real multi-instance design.

This issue predates #7173's ratified control-plane architecture and was under-specified against
it — refreshed now that the shared design is settled, rather than left to drift further out of
sync.

Area

AMS / Cloud architecture — packages/loopover-miner/lib/* (event-ledger.js, claim-ledger.js,
portfolio-queue.js, governor-state.js, and the other node:sqlite-backed stores).

Proposal

Adopt ORB's existing storage seam rather than inventing a new one, per #7173's ratified direction:
src/selfhost/d1-adapter.ts + pg-adapter.ts already abstract D1-or-Postgres behind one interface
(SqliteDriver/Statement), with a SQL dialect translator. AMS's own prior research
(packages/loopover-miner/docs/ams-storage-abstraction-research.md, written for the now-closed
#5216) independently reached the same conclusion, Postgres-lead: AMS's batchClaim-style
interactive transactions (claim-ledger.js's claimIssueWithinCap, portfolio-queue.js's
batchClaim, both currently BEGIN IMMEDIATE/COMMIT/ROLLBACK against node:sqlite) need real
transactional Postgres — D1's batch() can't do read-then-conditional-write, so a D1-first design
would force AMS's atomic-claim logic to be rewritten around a weaker primitive.

Self-host behavior must stay byte-identical by default: the existing node:sqlite backend remains
the default for a local/self-hosted miner; the Postgres backend is additive, selected only when a
hosted-mode connection string is configured (mirrors how d1-adapter.ts/pg-adapter.ts already
coexist for the main app today).

Deliverables

  • A storage-abstraction design doc naming the specific interface AMS's stores adopt (matching
    or extending SqliteDriver/Statement from d1-adapter.ts/pg-adapter.ts), with both
    backends addressed: existing node:sqlite (self-host, unchanged default) and Postgres
    (hosted, additive)
  • An explicit list of which AMS store modules need interface changes (event-ledger.js,
    claim-ledger.js, portfolio-queue.js, governor-state.js, and any others using raw
    DatabaseSync/BEGIN IMMEDIATE transactions) vs. which stay untouched
  • A statement on migration shape: whether AMS reuses src/selfhost/pg-adapter.ts directly (via
    the control-plane's HTTP boundary per Design: shared hosting control-plane for ORB + AMS SaaS (provisioning, secrets, billing, health) #7173, not a source import — AMS's package boundary
    stays separate from the main Worker's src/) or needs its own thin Postgres driver mirroring
    the same interface

Test Coverage Requirements

Design doc only in this issue — no code changes. A follow-on implementation issue inherits
Codecov's normal 99% patch-coverage gate on packages/loopover-miner/lib/** once the interface is
built.

Expected Outcome

AMS's local stores have a settled, documented storage-abstraction plan consistent with #7173's
ratified architecture, ready for a follow-on implementation issue — instead of a plan written
before that architecture existed.

Boundaries

  • Design/planning only: no changes to the actual store modules in this issue.
  • Must not invent a storage backend or interface shape that diverges from d1-adapter.ts/
    pg-adapter.ts's existing pattern without an explicit, justified reason — the whole point of
    Design: shared hosting control-plane for ORB + AMS SaaS (provisioning, secrets, billing, health) #7173's "core package + wrapper" direction is one shared seam, not a second one AMS maintains in
    parallel.
  • Self-host behavior (the node:sqlite default) must stay byte-identical; this is additive-only.

Links & Resources

  • Design: shared hosting control-plane for ORB + AMS SaaS (provisioning, secrets, billing, health) #7173 (ratified control-plane architecture this issue implements against)
  • src/selfhost/d1-adapter.ts, src/selfhost/pg-adapter.ts, src/selfhost/pg-queue.ts — the
    seam to extend
  • packages/loopover-miner/docs/ams-storage-abstraction-research.md — AMS's own prior research,
    already Postgres-lead
  • packages/loopover-miner/lib/event-ledger.js, claim-ledger.js, portfolio-queue.js,
    governor-state.js — the node:sqlite-backed stores this abstracts

Metadata

Metadata

Assignees

Labels

gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.help wantedExtra attention is neededmaintainer-onlyOwner-only work — yields no Gittensor points.

Projects

Status
Done

Relationships

None yet

Development

No branches or pull requests

Issue actions