test(benchmarks): vendor benchmark suite and add write-load harness

.github/workflows/pr-title.yml

@@ -44,5 +44,6 @@ jobs:
             plugins
             skills
             integrations
+            benchmarks
           # Allow breaking changes (needs "!" after type/scope)
           requireScopeForBreakingChange: true

benchmarks/.github/workflows/benchmark-nightly.yml

@@ -0,0 +1,46 @@
+name: benchmark-nightly
+
+on:
+  schedule:
+    - cron: "0 8 * * *"
+  workflow_dispatch:
+
+jobs:
+  nightly:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: astral-sh/setup-uv@v4
+
+      - uses: extractions/setup-just@v2
+
+      - name: Install dependencies
+        run: uv sync --group dev --extra judge
+
+      - name: Fetch LoCoMo
+        run: just bench-fetch-locomo
+
+      - name: Convert LoCoMo
+        run: just bench-convert-locomo
+
+      - name: Run retrieval benchmark
+        run: |
+          uv run bm-bench run retrieval \
+            --providers bm-local,bm-cloud,mem0-local \
+            --dataset-id locomo \
+            --dataset-path benchmarks/datasets/locomo/locomo10.json \
+            --corpus-dir benchmarks/generated/locomo/docs \
+            --queries-path benchmarks/generated/locomo/queries.json \
+            --allow-provider-skip
+
+      - name: Run judge benchmark (best effort)
+        run: |
+          LATEST_RUN=$(ls -1t benchmarks/runs | head -n1)
+          uv run bm-bench run judge --run-dir "benchmarks/runs/${LATEST_RUN}" || true
+
+      - name: Upload benchmark artifacts
+        uses: actions/upload-artifact@v4
+        with:
+          name: benchmark-nightly-artifacts
+          path: benchmarks/runs

benchmarks/.github/workflows/benchmark-smoke.yml

@@ -0,0 +1,32 @@
+name: benchmark-smoke
+
+on:
+  pull_request:
+  workflow_dispatch:
+
+jobs:
+  smoke:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: astral-sh/setup-uv@v4
+
+      - uses: extractions/setup-just@v2
+
+      - name: Install dependencies
+        run: uv sync --group dev
+
+      - name: Run smoke benchmark
+        run: just bench-smoke
+
+      - name: Validate latest artifacts
+        run: |
+          LATEST_RUN=$(ls -1t benchmarks/runs | head -n1)
+          uv run bm-bench validate-artifacts --run-dir "benchmarks/runs/${LATEST_RUN}"
+
+      - name: Upload benchmark artifacts
+        uses: actions/upload-artifact@v4
+        with:
+          name: benchmark-smoke-artifacts
+          path: benchmarks/runs

benchmarks/.gitignore

@@ -0,0 +1,27 @@
+__pycache__/
+*.pyc
+.venv/
+.pytest_cache/
+.ruff_cache/
+.idea/
+
+# Generated benchmark outputs
+benchmarks/runs/
+benchmarks/results/public/
+benchmarks/generated/
+benchmarks/logs/
+
+# Downloaded datasets (source distribution may be restricted)
+benchmarks/datasets/locomo/locomo10.json
+benchmarks/datasets/locomo/locomo10.provenance.json
+.env
+benchmarks/bm-home/
+benchmarks/datasets/longmemeval/longmemeval_s.json
+benchmarks/datasets/longmemeval/longmemeval_s.provenance.json
+benchmarks/datasets/locomo-audit/
+benchmarks/.mem0-qdrant/
+benchmarks/datasets/convomem/
+benchmarks/.bm-homes/
+.supermemory/
+.venvs/
+.scratch/

benchmarks/.python-version

@@ -0,0 +1 @@
+3.13

benchmarks/AGENTS.md

@@ -0,0 +1,124 @@
+# AGENTS.md - basic-memory-benchmarks Guide
+
+## Project Overview
+
+`basic-memory-benchmarks` is a standalone benchmark harness for comparing Basic Memory against other memory systems.
+
+Primary goals:
+- Deterministic retrieval benchmarks
+- Optional LLM-as-a-judge benchmarks
+- Public, reproducible artifact publication (including provenance metadata)
+
+This repo is intentionally isolated from `basic-memory` so benchmark dependencies do not pollute the product repo.
+
+## Build / Test Commands
+
+- Install: `uv sync --group dev`
+- Install judge extras: `uv sync --group dev --extra judge`
+- Run tests: `uv run pytest -q`
+- Lint: `uv run ruff check .`
+- Type check: `uv run pyright`
+
+Recommended local gate before pushing:
+1. `uv run pytest -q`
+2. `uv run ruff check .`
+3. `uv run pyright`
+
+## Benchmark Commands
+
+CLI entrypoint: `uv run bm-bench ...`
+
+Dataset and conversion:
+- `uv run bm-bench datasets fetch --dataset locomo`
+- `uv run bm-bench convert locomo --dataset-path benchmarks/datasets/locomo/locomo10.json --output-dir benchmarks/generated/locomo`
+
+Run retrieval:
+- `uv run bm-bench run retrieval --providers bm-local,mem0-local --dataset-id locomo --dataset-path benchmarks/datasets/locomo/locomo10.json --corpus-dir benchmarks/generated/locomo/docs --queries-path benchmarks/generated/locomo/queries.json --output-root benchmarks/runs --allow-provider-skip`
+
+Run judge (optional):
+- `uv run bm-bench run judge --run-dir benchmarks/runs/<run-id>`
+
+Validate and publish:
+- `uv run bm-bench validate-artifacts --run-dir benchmarks/runs/<run-id>`
+- `uv run bm-bench publish --run-dir benchmarks/runs/<run-id> --destination benchmarks/results/public`
+
+`just` shortcuts:
+- `just bench-smoke`
+- `just bench-fetch-locomo`
+- `just bench-convert-locomo`
+- `just bench-run-bm-local`
+- `just bench-run-mem0-local`
+- `just bench-run-full`
+- `just bench-judge RUN_DIR=benchmarks/runs/<run-id>`
+- `just bench-publish RUN_DIR=benchmarks/runs/<run-id>`
+
+## Repository Layout
+
+- `src/basic_memory_benchmarks/cli.py` - CLI surface
+- `src/basic_memory_benchmarks/runner.py` - run orchestration
+- `src/basic_memory_benchmarks/providers/` - provider adapters (`bm-local`, `bm-cloud`, `mem0-local`, `zep-reference`)
+- `src/basic_memory_benchmarks/scoring/` - retrieval + judge scoring
+- `src/basic_memory_benchmarks/reporting/` - artifact writers / comparison helpers
+- `src/basic_memory_benchmarks/converters/` - dataset conversion logic
+- `src/basic_memory_benchmarks/datasets/` - dataset fetch/load helpers
+- `benchmarks/datasets/` - source metadata + download helpers
+- `benchmarks/generated/` - generated corpus/query outputs
+- `benchmarks/runs/` - raw run artifacts
+- `benchmarks/results/public/` - published bundles
+- `tests/`, `test-int/` - unit and integration tests
+
+## Benchmark Integrity Rules
+
+These are non-negotiable for headline comparisons:
+
+1. Use the same query set and same `top_k` across providers.
+2. Do not apply provider-specific query rewriting for headline runs.
+3. Keep official categories and adversarial breakout separate.
+4. Record provider `SKIPPED(reason)` explicitly; do not silently drop providers.
+5. Always capture provenance in `manifest.json`:
+   - benchmark repo SHA
+   - BM source + resolved BM SHA
+   - provider versions
+   - dataset source + checksum
+   - runtime metadata
+
+## Provider Notes
+
+### Basic Memory (`bm-local`)
+
+- Interact via external `bm` CLI contract, not internal imports from `basic-memory`.
+- Repeated runs against the same corpus path may reuse an existing BM project name.
+- The benchmark command is typically invoked through `uv run ...` so `.venv/bin/bm` is used.
+
+### Mem0 (`mem0-local`)
+
+- Requires `OPENAI_API_KEY` (or equivalent configured model creds).
+- Ingest and search use a stable benchmark `user_id` namespace per run.
+- Store source metadata (`source_doc_id`, `source_path`, `conversation_id`, `dataset_id`) for grounding.
+
+## Environment / Secrets
+
+- Keep secrets in `.env` (already gitignored).
+- Avoid exporting unrelated `BASIC_MEMORY_*` environment variables into benchmark runs unless intended.
+- Prefer setting only required credentials for run reproducibility.
+
+## Dataset Policy
+
+- If redistribution is allowed: publish snapshot + checksum.
+- If restricted: publish canonical source link + downloader + checksum verification.
+- Always publish conversion code and run artifacts.
+
+## Coding Guidelines
+
+- Python 3.12+ style with type hints.
+- Keep diffs focused and minimal.
+- Fail fast; do not silently swallow benchmark-critical failures.
+- Use `apply_patch` for targeted edits when practical.
+- Add tests for behavior changes in adapters, scoring, or artifact schemas.
+
+## Git / Collaboration
+
+- Use non-interactive git commands.
+- Sign commits: `git commit -s`.
+- Do not commit `.env`, generated runs, or local editor state.
+- If changing benchmark behavior, include a brief note in PR/commit describing fairness or reproducibility impact.