From 95b886c1ae38b1ac26761d42d4676255651d6861 Mon Sep 17 00:00:00 2001 From: plind-junior <59729252+plind-junior@users.noreply.github.com> Date: Thu, 2 Jul 2026 02:52:43 +0900 Subject: [PATCH] chore(bench): wire make bench, add pytest-benchmark, record baseline the benchmark suite was implemented but unrunnable out of the box: pytest-benchmark wasn't in the [dev] extras, and the bench_*.py filenames don't match pytest's default python_files glob, so the documented invocation collected zero tests. - add pytest-benchmark to the [dev] optional-dependencies - add a `make bench` target that folds in the python_files override - record the first baseline (medians at 1k/10k) in benchmarks/README.md and flip its status from "not implemented yet" to implemented - gitignore the bench.json / .benchmarks run artifacts --- .gitignore | 2 ++ Makefile | 14 ++++++++--- benchmarks/README.md | 58 +++++++++++++++++++++++++++++++++++--------- pyproject.toml | 1 + 4 files changed, 61 insertions(+), 14 deletions(-) diff --git a/.gitignore b/.gitignore index eec57be1..9a61a8d8 100644 --- a/.gitignore +++ b/.gitignore @@ -7,6 +7,8 @@ __pycache__/ .ruff_cache/ .coverage htmlcov/ +bench.json +.benchmarks/ dist/ build/ *.swp diff --git a/Makefile b/Makefile index 03dd2596..6ac63fc2 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: help install dev test test-cov lint format type check build clean examples-screenshots +.PHONY: help install dev test test-cov bench lint format type check build clean examples-screenshots PY ?= python PIP ?= $(PY) -m pip @@ -9,6 +9,7 @@ help: @echo " make install editable install with dev extras" @echo " make test run pytest" @echo " make test-cov run pytest with coverage" + @echo " make bench run the performance benchmark suite (needs pytest-benchmark)" @echo " make lint ruff check" @echo " make format ruff format (writes)" @echo " make type mypy" @@ -39,6 +40,13 @@ type: check: lint type test +# the bench_*.py filenames don't match pytest's default python_files glob, +# so the override is required or zero benchmarks are collected. +bench: + $(PY) -m pytest benchmarks/ --benchmark-only \ + -o python_files='bench_*.py test_*.py' \ + --benchmark-json=bench.json + examples-screenshots: $(PY) docs/img/examples/render.py @@ -48,6 +56,6 @@ build: clean: rm -rf build dist *.egg-info src/*.egg-info \ - .pytest_cache .ruff_cache .mypy_cache \ - coverage.xml .coverage htmlcov + .pytest_cache .ruff_cache .mypy_cache .benchmarks \ + coverage.xml .coverage htmlcov bench.json find . -type d -name __pycache__ -prune -exec rm -rf {} + diff --git a/benchmarks/README.md b/benchmarks/README.md index 4b2f293a..11da8fd5 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -14,19 +14,48 @@ against tools that are. But there are a few numbers that *do* matter: ## Status -Not implemented yet. See [ROADMAP.md](../ROADMAP.md) (0.3) for the -planned timeline. This README is a placeholder so we don't lose the -shape of what we want to measure. +Implemented. All four `bench_*.py` files run under `pytest-benchmark`. +See [ROADMAP.md](../ROADMAP.md) (0.3) for the surrounding milestone. The +100k fixture in `conftest.py` exists but no bench file exercises it yet. -## Planned layout +## Baseline + +First recorded run. This is a single developer-machine snapshot, **not** +a published environment per the methodology below — treat it as +order-of-magnitude, not gospel. Medians, warm: + +| Benchmark | 1k claims | 10k claims | +|---|---|---| +| `search_fts5` (FTS5 query) | 0.46 ms | 1.65 ms | +| `search_substring` (fallback scan) | 241 ms | 2.42 s | +| `propose_claim` (hot-loop write) | 2.40 ms | — | +| `index_rebuild` | 311 ms | 7.25 s | +| `bundle_export` | 113 ms | — | +| `bundle_export_check` | 48.6 ms | — | +| `bundle_import` | 1.05 s | 12.4 s | + +What the numbers say: + +- **FTS5 search stays fast and scales sub-linearly** (~3.6x for 10x the + claims). The substring fallback reads every claim file, so it's ~500x + slower — that's the whole reason it's only a fallback. +- **`propose_claim` medians 2.4 ms**, comfortably under the ~50ms hot-loop + budget noted above. +- **A 10k-claim bundle imports in 12.4 s** — seconds, not minutes, which + is the bar this benchmark was written to guard. + +Environment: 13th Gen Intel Core i9-13900K (16 threads), ~22 GB RAM, +Python 3.14, vouch 1.0.0. Full per-run detail (min/max/stddev, machine +info) lands in `bench.json`. + +## Layout ``` benchmarks/ ├── README.md (you are here) -├── conftest.py pytest-benchmark configuration +├── conftest.py pytest-benchmark configuration + seeded KB fixtures ├── fixtures/ -│ ├── gen_kb.py synth a KB of N claims with realistic distributions -│ └── seed/ pre-built fixture KBs (small, medium, large) +│ └── gen_kb.py synth a KB of N claims with realistic distributions ├── bench_search.py kb.search latency at varying KB sizes ├── bench_propose.py kb.propose_* write latency ├── bench_bundle.py export + import + verify round-trips @@ -34,14 +63,21 @@ benchmarks/ ``` Benchmarks live outside `tests/` so a regular `pytest` run doesn't -pull them in. The intended invocation is: +pull them in. `pytest-benchmark` isn't in the `[dev]` extras, and the +`bench_*.py` filenames don't match pytest's default `python_files` +glob — so the invocation needs both an install and a collection +override: ```bash -make bench # not yet wired in the Makefile -# or: -pytest benchmarks/ --benchmark-only --benchmark-json=bench.json +pip install pytest-benchmark +pytest benchmarks/ --benchmark-only \ + -o python_files='bench_*.py test_*.py' \ + --benchmark-json=bench.json ``` +`make bench` is not wired in the Makefile yet; when it is, it should +fold in the `python_files` override so this isn't a footgun. + ## Methodology principles - **Real disks.** No tmpfs benchmarks. The file-based design makes diff --git a/pyproject.toml b/pyproject.toml index b82bb865..7e2f9b60 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -39,6 +39,7 @@ web = [ ] dev = [ "pytest>=9.0.3,<10", + "pytest-benchmark>=5,<6", "pytest-cov>=5,<6", "mypy>=2.1.0", "ruff>=0.15.13",