Skip to content

test(benchmarks): vendor benchmark suite and add write-load harness - #1021

Merged
phernandez merged 19 commits into
mainfrom
benchmark-suite
Jun 25, 2026
Merged

test(benchmarks): vendor benchmark suite and add write-load harness#1021
phernandez merged 19 commits into
mainfrom
benchmark-suite

Conversation

@phernandez

@phernandez phernandez commented Jun 25, 2026

Copy link
Copy Markdown
Member

Vendors the basic-memory-benchmarks suite into benchmarks/ and adds a new
write-load benchmark (driver + justfile recipes + findings doc), so we can
measure write-path performance against any branch/SHA and re-run it easily.

Most of the diff is the upstream retrieval-benchmark suite (locomo / longmemeval /
convomem, converters, runners). The new work is:

  • benchmarks/scripts/write_load_bench.py — drives write_note over a warm
    bm mcp stdio session at increasing concurrency. Measures caller-perceived
    accept latency (p50/p95/p99), throughput, error rate, and the async drain
    stages: time_to_materialized_ms, time_to_searchable_ms (FTS),
    time_to_embedded_ms (semantic). Supports SQLite and a throwaway Postgres
    testcontainer, with per-ref venvs.
  • benchmarks/justfile recipes: bench-venv, bench-write-load,
    bench-write-scale, bench-write-backend.
  • benchmarks/docs/write-load-benchmark.md — the full run log + findings.

What the benchmark found (fixes live in #1002 / #1018, not here)

  • The accepted-note refactor's deferred materialization needed resource bounds:
    a bounded worker pool (fixed the unbounded-queue collapse: C=64 p99 122s→9.9s,
    throughput 1.5→11.9/s, errors→0) and Postgres connection pooling (NullPool
    collapse: C=32 p99 478s→2s, 21%→0% errors).
  • Controlled main-vs-branch 2×2: async writes beat direct writes on both backends
    at every concurrency level (~3–4× lower p50, ~2× throughput at C=64); Postgres
    scales better than SQLite once pooled.
  • Local embedding keeps pace; SQLite PRAGMA tuning (synchronous=OFF, mmap, …)
    shows no measurable gain     — we're bound by per-note work, not the engine.

This PR is just the harness + findings; the corresponding code changes are in
#1002 (async write path, worker pool, PG pool, PRAGMA knobs) and #1018 (the
standalone local-Postgres fixes).

🤖 Generated with Claude Code

phernandez and others added 19 commits June 23, 2026 22:46
@phernandez @claude
chore(benchmarks): vendor basic-memory-benchmarks suite into benchmarks/
521bd91 
Copy the standalone basic-memory-benchmarks package (github main 2f42fb0,
tracked tree only) into benchmarks/ as a self-contained monorepo sub-package.
It has its own pyproject/justfile/uv.lock, so its heavy competitor deps
(mem0ai[nlp], etc.) never touch a normal basic-memory install, and the root
pyproject has no uv workspace that would pull it in.

This brings the warm `bm mcp` stdio harness (bm-local provider) and the
generic JSONL run-compare tooling in-repo so we can benchmark across
branches/SHAs easily and often. Per-ref venv setup + a write-path load
workload follow in subsequent commits.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez @claude
feat(benchmarks): add write-path load benchmark with per-ref venv har…
69e5b7f 
…ness

A standalone driver (scripts/write_load_bench.py) drives write_note over a warm
`bm mcp` stdio session at increasing concurrency and records caller-perceived
accept latency (p50/p95/p99), throughput, error rate, and time-to-visible
(materialized on disk, FTS-searchable) as standard compare-format JSONL.

Branch/SHA comparison runs through per-ref venvs: `just bench-venv <ref>`
installs basic-memory@<ref> from GitHub, `just bench-write-load <ref>` runs the
sweep against that venv's basic-memory, and `just bench-write-compare A B`
diffs the two JSONL outputs via the existing generic compare tool. The harness
spawns bm mcp so the real watcher + background schedulers run (env is kept out
of test mode), and BASIC_MEMORY_CONFIG_DIR isolates a fresh runtime per run.

docs/write-load-benchmark.md records the methodology, decisions, and the first
main baseline (accept latency grows ~28x from C=1 to C=32 while throughput
plateaus ~6.7/s — the synchronous-write pile-up the accepted-note refactor
should beat).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez @claude
docs(benchmarks): record main vs PR-1002 write-load comparison
16258dc 
First head-to-head: on local SQLite the accepted-note branch is consistently
slower per write accept (p50 +41-252%) and lower throughput (-21-54%) than main
at every concurrency level, with 0 errors on both. The async-accept hypothesis
did not hold locally; the accept path got heavier. Leading suspect is the
per-write full-project relation resolution scheduled for #1015 (no debounce),
plus extra note_content DB writes per accept. The offline-queue benefit is
likely cloud/PGQ-specific. Single run each - directional, needs repetition and
isolation of the #1015 contribution.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez @claude
docs(benchmarks): coalescing fix did not close the write-load gap
c075bca 
Re-ran after pushing the relation-resolution coalescing (7ecb672): the gap is
unchanged within noise. At C=1 (no contention) the branch is still ~3x slower
per write (92->294ms), so the cost is the synchronous accept path itself, not
the background schedulers (which never sit on accept latency). Revised
conclusion: the accepted-note accept path is heavier locally (DB-first accept +
inline materialization both), and the cloud queue benefit does not exist on the
local SQLite runtime. Next: profile the C=1 accept path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez @claude
docs(benchmarks): materialization deferral validates the async-accept…
4ee15d3 
… hypothesis

After deferring local materialization off the accept path (696d71b), the
branch beats main at concurrency >=16 (accept p50 -45% at C=16, -23% at C=32)
and throughput now scales with concurrency (3.5->7.6/s) instead of plateauing
like main's synchronous path (~6.7/s). Crossover ~C=8; residual +21ms at C=1 is
the intrinsic DB-first accept overhead. The accepted-note refactor is a net win
for concurrent local writes once materialization is deferred.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez @claude
feat(benchmarks): add postgres backend + high-concurrency scaling to …
472e4e1 
…write-load

Add --backend {sqlite,postgres} to the write-load driver: postgres spins up a
throwaway Postgres testcontainer, points the bm server at it via
BASIC_MEMORY_DATABASE_BACKEND/_URL, and tears it down (Ryuk reaper on error
paths). Recipes: bench-write-scale (concurrency 1..128 to find the SQLite
single-writer choke point) and bench-write-backend (one ref on sqlite vs
postgres). bench-write-load now exposes --drain-timeout.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez @claude
docs(benchmarks): high-concurrency scaling + SQLite-vs-Postgres findings
d018eb1 
Two results, one conclusion: the write path needs resource bounds.

Scaling (concurrency to 128): the materialization deferral improves the median
accept at every level but the unbounded background queue collapses the tail and
throughput under heavy sustained load (C>=64: p99 122s, throughput 1.5/s, SQLite
lock errors) because there is no backpressure. Main's synchronous accept is
self-throttling. The deferral reorders work; on one SQLite it adds no capacity.

SQLite vs Postgres (async writes, local): Postgres is the opposite of on-par --
~4x slower at C=1 and collapsing at C>=32 (p99 478s, 0.2/s, 21% write failures)
because _create_postgres_engine uses NullPool (fresh connection per request), so
concurrent writes + per-materialization connections storm max_connections.

Both point to: bounded background materialization + Postgres connection pooling.

Driver: pgvector/pgvector:pg16 image (Postgres search-index init needs CREATE
EXTENSION vector) and BASIC_MEMORY_DEFAULT_PROJECT for the Postgres path (fresh
local Postgres seeds no default project).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez @claude
docs(benchmarks): worker pool fixes the high-concurrency wall
2c74ef5 
The bounded materialization worker pool (materialization_workers, default 4)
eliminates the unbounded-queue collapse: at C=64 p99 drops 121771->9870ms (12x),
throughput 1.5->11.9/s (8x), and SQLite lock errors go to zero. Throughput now
plateaus at ~11/s instead of collapsing, and the pooled branch beats main at
high concurrency. Tuning shows 2 workers is the SQLite sweet spot (single writer
is the bottleneck); Postgres will want more once pooled.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez @claude
docs(benchmarks): Postgres pool + default-project seed fix the collapse
8cadadf 
With the connection pool (AsyncAdaptedQueuePool wired to db_pool_* config) and
the default-project seed for local Postgres, Postgres no longer collapses:
C=32 p99 478612->2021ms (237x), throughput 0.2->21.9/s (100x), errors 21%->0,
and it runs without the BASIC_MEMORY_DEFAULT_PROJECT shim (removed from the
driver). SQLite was noisy this run (machine load), so the head-to-head is
indicative; the validated result is the eliminated Postgres collapse.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez @claude
docs(benchmarks): 3x variance bounds confirm the fixes are stable
3b117f7 
3 fresh runs each, sqlite vs postgres (notes=100): zero failed writes across all
24 cells, the earlier SQLite slow run was noise (clean C=1 p50 ~95ms), the two
backends are on par at C=1, and Postgres scales better at concurrency with
non-overlapping ranges (C=64: ~20.5 vs ~12.2/s throughput, ~3.5s vs ~7.5s p99).
The worker pool + Postgres connection pool hold up reproducibly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez @claude
docs(benchmarks): controlled 2x2 main (direct) vs branch (async)
8e77219 
With the 3 enabling fixes on main (PR #1018), ran a true apples-to-apples 2x2:
main's direct/inline writes vs the branch's async (deferred + worker-pooled)
writes, on SQLite and Postgres, interleaved in one session. The async path beats
direct writes on both backends at every concurrency level (~3-4x lower p50, ~2x
throughput at C=64); main plateaus while branch scales; Postgres gives the async
path the most headroom (C=64 18.5 vs SQLite 10.3/s). Zero write failures in all
16 cells. Supersedes the earlier cross-run '+21ms at C=1' note.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez @claude
docs(benchmarks): split the 2x2 into readable per-metric rows
1ac8777 
Replace the cramped 'main -> branch' arrow cells with explicit main/branch rows
plus a bold speedup row per metric (p50, p99, throughput), concurrency across
the columns. Same data, easier to scan.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez @claude
docs(benchmarks): flip 2x2 to C-rows with main/branch/speedup columns
790b7be 
Concurrency down the rows; each metric (p50, p99, throughput) gets a main
column, a branch column, and the speedup. Easier to read across a single
concurrency level.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez @claude
docs(benchmarks): express the 2x2 improvement as percentages
67ae6b1 
Replace the x-multiplier improvement columns with percent: latency 'faster' =
% lower (main-branch)/main, throughput 'higher' = % more (branch-main)/main.
p50 ~47-75% faster on SQLite, ~41-67% on Postgres; throughput up to +160%.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez @claude
feat(benchmarks): measure time_to_embedded (semantic vector-sync drain)
657fc13 
Add a time_to_embedded_ms stage to the write-load driver: after each burst,
poll the SQLite vector store (search_vector_chunks JOIN entity, filtered to the
benchmark's notes by path) until all N notes are embedded, so we can see whether
the embedding pipeline (the heaviest background follow-up: fastembed ONNX per
note) keeps up under write load. The warmup doubles as a semantic-enabled probe
(short timeout) so the run doesn't hang if semantic is off; SQLite only (the
peek reads sqlite-vec; Postgres stores vectors in pgvector). Also isolate
BASIC_MEMORY_HOME to an empty scratch dir so the index+embed pipeline only
processes the benchmark's notes, not the user's real ~/basic-memory.

Result (notes=100, C up to 64): t_embedded stays ~1-3ms while t_materialized
grows to ~9s, i.e. embedding keeps pace and is NOT the bottleneck with the local
model; materialization is the dominant async stage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez @claude
docs(benchmarks): time_to_embedded result — embedding keeps pace
0d3eb74 
Document the new embedded-drain metric and the finding: t_embedded stays ~1-3ms
while t_materialized climbs to ~9s at C=64, so the local embedding pipeline keeps
up and materialization is the dominant async stage. Note the incremental
(per-stage) semantics of the drain metrics and the caveat that a remote
embedding provider would likely flip this.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez @claude
docs(benchmarks): scope embedding to local fastembed
bccda12 
Local fastembed keeps pace with the write path, so it's the optimization target;
remote embedding providers are a caveat-emptor tradeoff, explicitly out of scope
for write-path tuning (no bounded embed pool planned).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez @claude
docs(benchmarks): SQLite PRAGMA sweep — no measurable gain
5cdee17 
synchronous=OFF, mmap, wal_autocheckpoint, page_size show no reliable throughput
improvement (3x means 17.6/18.6/19.4/s, ranges overlap). WAL+NORMAL already skips
the per-commit fsync so OFF buys nothing; default stays NORMAL. We're not commit/
fsync/read-IO bound -- the ~17-19/s ceiling is per-note work (parse, index, FTS,
file write, watcher refresh), not the database. Group-commit skipped (hairy, and
its payoff vanishes with no fsync to save).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez @claude
ci: add benchmarks scope to PR title checker
48887f5 
Allows test(benchmarks)/chore(benchmarks)/... titles for the vendored
benchmark subpackage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed Jun 25, 2026
View reviewed changes

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 48887f53eb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread benchmarks/src/basic_memory_benchmarks/providers/bm_cloud.py
Comment thread benchmarks/.github/workflows/benchmark-smoke.yml
@phernandez

phernandez commented Jun 25, 2026

Copy link
Copy Markdown
Member Author

Both Codex suggestions are in the vendored upstream basic-memory-benchmarks suite (the cloud provider's corpus loading, and the suite's own benchmarks/.github/workflows/), not in the write-load harness this PR adds. They're imported as-is here; the suite's internal CI workflows don't auto-run from a subdirectory (GitHub only runs root .github/workflows/) so they're inert, and rewiring/refactoring the vendored provider is out of scope for this vendor PR. Resolving these threads — they can be revisited upstream.

@phernandez
phernandez merged commit cb6bc32 into main Jun 25, 2026
18 checks passed
@phernandez
phernandez deleted the benchmark-suite branch June 25, 2026 17:17
@phernandez phernandez mentioned this pull request Jul 27, 2026
Closed
7 tasks
@phernandez phernandez mentioned this pull request Aug 12, 2026
Open
14 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@phernandez