Storage backend seam (step 1): async operation-level contract + sqlite re-wired through it — MySQL stays a registered stub (#3388)

Parent: #3388 (pluggable storage backends — MySQL/Greenplum for distributed deployment). This issue tracks step 1 only, as scoped by the owner: land the abstraction seam, keep every test green, re-wire sqlite through it, leave MySQL for step 2.

Goal

The storage layer today is UnifiedStore — 1k+ lines of synchronous SQLite SQL, consumed directly by persist/shadow/export/web/admin. Step 1 extracts the seam without changing any behavior:

StorageBackend contract (operation-level, async) — src/storage/backend.ts. The contract is a set of operations (ingest, message/block reads, session_ext, watermark, reap journal, archive/restore/gc, checkpoints, lookup, delete, close), NOT SQL: each backend implements the same operations natively so transaction/atomicity semantics stay per-backend (the [PR] refactor(storage): extract StorageBackend contract behind UnifiedStore (#3388 Phase A) #3432 lesson — a dialect shim cannot preserve ingestSession's single-transaction refcount discipline).
SqliteBackend — a real adapter wrapping the current UnifiedStore (sync under the hood, Promise-wrapped surface). Zero behavior change; no schema change (stays v7); no migration.
Registry + config — storage.backend file key + BILI_STORAGE_BACKEND env (default sqlite). mysql is a registered stub: selecting it fails at open time with one clear, doctor-readable line ("registered, ships in step 2 — Extend the storage layer to support additional backends beyond SQLite (e.g., MySQL, Greenplum), while maintaining extens | 存储层增加支持除了sqlite之外的其他存储，比如mysql，greenplum等，并保持扩展性 #3388"), never mid-flight. Unknown names fail with the valid-names list.
Seam proof — export.ts switches to consuming the StorageBackend interface (async) end-to-end; a contract test suite (tests/storage-backend-contract.test.ts) runs the same assertions against any backend instance — against sqlite today, against mysql unchanged in step 2.

Non-goals (step 2+, tracked on #3388)

No real MySQL implementation, no mysql2 dependency (optional-dependency + self-download discipline is already established by better-sqlite3's engine bootstrap and will be reused).
No async conversion of the persist/shadow hot path (the interface being async is enough for step 1; callers convert per-surface starting step 2).
No dynamic third-party backend loading (built-in registry only; the interface shape does not preclude opening registration later).
No multi-writer/concurrency rework (CAS refcount + watermark under network MVCC — ships with the mysql backend).

Acceptance

Full test suite green, goldens untouched, zero schema/version change.
storage.backend knob matrix pinned: default sqlite; explicit sqlite works; mysql → the documented stub error at open; unknown → error listing valid names.
Contract suite passes against the sqlite backend via the registry path.
export.ts goes through the interface (its behavior pinned by existing export tests, unmodified).
Docs: CONFIGURATION (via yaml seeds) + CHANGELOG.

老规矩: 打桩先行 (contract suite + knob matrix staked red first), then implementation.

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。 检索 open+closed 的 issue 与 PR（storage backend seam / StorageBackend / UnifiedStore 抽象）：唯一相关项是 PR #3432（同步版 StorageBackend 契约），已于今日关闭未合并——关闭理由正是被本 issue 重划的两步计划取代（接口清单落后于 schema v6/v7）。无其他在途条目触碰这条缝；父 issue #3388 作主要索引，不另开新 issue。

标签: scope/storage, p3 — 计划内功能步骤（非缺陷），波及面为存储层。

工作计划（按 issue 四项验收，打桩先行）：

打桩（红）：tests/storage-backend-contract.test.ts（契约套件 runner——同一组断言对任意 backend 实例可跑，step 2 的 mysql 原样复用）+ tests/storage-backend-knob-matrix.test.ts（storage.backend 旋钮矩阵：默认 sqlite / 显式 sqlite / mysql → open 时 stub 错误 / unknown → 列有效名 / env 优先于 file）。
契约：src/storage/backend.ts —— 操作级 async 契约（ingest、watermark、message/block 读、session_ext、reap journal、archive/restore/gc、checkpoints、lookup、delete、close；不是 SQL，事务/原子性语义留各 backend 自持——[PR] refactor(storage): extract StorageBackend contract behind UnifiedStore (#3388 Phase A) #3432 的教训）。
适配器：src/storage/sqlite-backend.ts —— SqliteBackend 原样包现有 UnifiedStore（sync 在内核、Promise 面在外），零行为变更、零 schema 变更（v7 不动）、无迁移。
注册表 + 旋钮：src/storage/registry.ts —— 内置注册表 [sqlite, mysql]；mysql 已注册但 open 时即报一行清晰错误（registered, ships in step 2 — Extend the storage layer to support additional backends beyond SQLite (e.g., MySQL, Greenplum), while maintaining extens | 存储层增加支持除了sqlite之外的其他存储，比如mysql，greenplum等，并保持扩展性 #3388），不碰磁盘、绝不 mid-flight 失败；unknown 名报有效名清单。配置面：storage.backend file key + BILI_STORAGE_BACKEND env（env 优先，默认 sqlite）。
缝证明：export.ts 端到端改走 registry 打开 index.db（async）；stub/unknown/索引不可读全部落入既有的 legacy-tail renderer 回退（stderr 一行提示），行为由既有 export 测试钉住不动。
文档：CONFIGURATION yaml seed（storage.backend）+ CHANGELOG Unreleased 条目。

非目标严格守住：不引 mysql2 依赖、不做 persist/shadow 热路径 async 化（接口 async 即够，调用方 step 2 起按面转换）、不做第三方 backend 动态加载（仅内置注册表）、不做多写并发改造。

交付：分支 2026-10-20_storage-backend-seam，三个 commit（test 打桩红 → feat 实现 → docs），PR 带 Closes #3931。全量套件判定以远程 CI 为准。

Decision for step 2 — mysql2 ships as a bundled default dependency (owner call, 2026-10-20):

mysql2 goes into dependencies (not optional, not self-download). The package is pure JS, ~498KB unpacked / ~1.2MB with its runtime closure — negligible next to the ~30MB install.
The code path still uses await import("mysql2") inside openBackend, so:
- startup cost stays zero until storage.backend = "mysql" is configured;
- nothing else (client/plugin lanes) can reach the module transitively — only server-side src/storage/backend.ts imports it;
- if a future packaging pipeline wants to strip it, a dynamic import is trivially external-izable.
The sqlite-style self-download machinery is not used for mysql — the dependency itself is the distribution mechanism (version pinned via lockfile like every other dep).
The resolution order at open time collapses to: installed dependency (always, once shipped) → documented error. The host-override / registry-fallback tiers from the earlier sketch are unnecessary.

Everything else in step 2 is unchanged: MysqlBackend implements StorageBackend, the contract test suite runs the same assertions against a real MySQL in CI (skipped locally without one), and multi-writer semantics (CAS refcount / watermark under MVCC) land in the same PR.

Step 2 delivered in #3946 (stacked on #3932, step 1):

MysqlBackend implements the full StorageBackend contract against MySQL 8.0 (mysql:8.0.46 verified in docker): ingest with atomic CAS refcounts, reads, session_ext/kv, reap journal, checkpoints, conversations, archive/restore/gc, diagnostics.
mysql2 bundled + lazily loaded (the m39104 inline decision); DSN = env BILI_MYSQL_DSN > storage.mysql.dsn.
export speaks mysql today; the proxy hot path (shadow ingest) stays on sqlite until the step-3 async swap — the remaining work in this issue.

Remaining for step 3: persist/shadow async-ification (loadSync sync kernel contract), replicas/checkpoints/gc surfaces on the backend interface, multi-writer hardening (MVCC CAS discipline from the #3388 plan).

Step 3 delivered in #3955 (stacked on #3946): mysql now carries the proxy hot path — serial async queue for saves (watermark deltas, reap journal, digest sampling, 3-strike), JSON envelope as the sync read path with lazy disk-index seeding, replicas/ext/GC via async twins, writerOff sqlite-only, closeQuietlyAsync for pool teardown. Verified against real MySQL 8.0.46 (hot-path e2e + contract 7/7); sqlite families byte-identical (fast path untouched).

fix(storage): enforce storage.backend selection at export open time (#…

storage: MySQL backend (step 2 of #3388) — MysqlBackend implements th…

storage: wire mysql into the proxy hot path (#3388 step 3, #3931)

storage: StorageBackend contract (step 1) — sqlite re-wired through t…

fix(storage): enforce storage.backend selection at export open time (#…

Merge pull request #3932 from ranxianglei/2026-10-20_storage-backend-…

storage: MySQL backend (step 2 of #3388) — MysqlBackend implements th…

storage: wire mysql into the proxy hot path (#3388 step 3, #3931)

Merge pull request #3946 from ranxianglei/2026-10-20_mysql-backend

fix: adapt the #3928 pass-throughs to the storage-backend seam (#3946)…

fix(ci): build kernel dist before mysql-contract lane (#3931)

fix(persist): summary facade misses on network backends instead of le…

Navigation Menu

Repository navigation

Storage backend seam (step 1): async operation-level contract + sqlite re-wired through it — MySQL stays a registered stub (#3388) #3931

Goal

Non-goals (step 2+, tracked on #3388)

Acceptance

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Storage backend seam (step 1): async operation-level contract + sqlite re-wired through it — MySQL stays a registered stub (#3388) #3931

Description

Goal

Non-goals (step 2+, tracked on #3388)

Acceptance

Activity

ranxianglei commented on Oct 20, 2026

ranxianglei commented on Oct 20, 2026

ranxianglei commented on Oct 20, 2026

ranxianglei commented on Oct 20, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions