Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ __pycache__/
.ruff_cache/
.coverage
htmlcov/
bench.json
.benchmarks/
dist/
build/
*.swp
Expand Down
14 changes: 11 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: help install dev test test-cov lint format type check build clean examples-screenshots smoke-capture smoke-recall
.PHONY: help install dev test test-cov bench lint format type check build clean examples-screenshots smoke-capture smoke-recall

PY ?= python
PIP ?= $(PY) -m pip
Expand All @@ -9,6 +9,7 @@ help:
@echo " make install editable install with dev extras"
@echo " make test run pytest"
@echo " make test-cov run pytest with coverage"
@echo " make bench run the performance benchmark suite (needs pytest-benchmark)"
@echo " make lint ruff check"
@echo " make format ruff format (writes)"
@echo " make type mypy"
Expand Down Expand Up @@ -41,6 +42,13 @@ type:

check: lint type test

# the bench_*.py filenames don't match pytest's default python_files glob,
# so the override is required or zero benchmarks are collected.
bench:
$(PY) -m pytest benchmarks/ --benchmark-only \
-o python_files='bench_*.py test_*.py' \
--benchmark-json=bench.json

examples-screenshots:
$(PY) docs/img/examples/render.py

Expand All @@ -56,6 +64,6 @@ build:

clean:
rm -rf build dist *.egg-info src/*.egg-info \
.pytest_cache .ruff_cache .mypy_cache \
coverage.xml .coverage htmlcov
.pytest_cache .ruff_cache .mypy_cache .benchmarks \
coverage.xml .coverage htmlcov bench.json
find . -type d -name __pycache__ -prune -exec rm -rf {} +
58 changes: 47 additions & 11 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,34 +14,70 @@ against tools that are. But there are a few numbers that *do* matter:

## Status

Not implemented yet. See [ROADMAP.md](../ROADMAP.md) (0.3) for the
planned timeline. This README is a placeholder so we don't lose the
shape of what we want to measure.
Implemented. All four `bench_*.py` files run under `pytest-benchmark`.
See [ROADMAP.md](../ROADMAP.md) (0.3) for the surrounding milestone. The
100k fixture in `conftest.py` exists but no bench file exercises it yet.

## Planned layout
## Baseline

First recorded run. This is a single developer-machine snapshot, **not**
a published environment per the methodology below — treat it as
order-of-magnitude, not gospel. Medians, warm:

| Benchmark | 1k claims | 10k claims |
|---|---|---|
| `search_fts5` (FTS5 query) | 0.46 ms | 1.65 ms |
| `search_substring` (fallback scan) | 241 ms | 2.42 s |
| `propose_claim` (hot-loop write) | 2.40 ms | — |
| `index_rebuild` | 311 ms | 7.25 s |
| `bundle_export` | 113 ms | — |
| `bundle_export_check` | 48.6 ms | — |
| `bundle_import` | 1.05 s | 12.4 s |

What the numbers say:

- **FTS5 search stays fast and scales sub-linearly** (~3.6x for 10x the
claims). The substring fallback reads every claim file, so it's ~500x
slower — that's the whole reason it's only a fallback.
- **`propose_claim` medians 2.4 ms**, comfortably under the ~50ms hot-loop
budget noted above.
- **A 10k-claim bundle imports in 12.4 s** — seconds, not minutes, which
is the bar this benchmark was written to guard.

Environment: 13th Gen Intel Core i9-13900K (16 threads), ~22 GB RAM,
Python 3.14, vouch 1.0.0. Full per-run detail (min/max/stddev, machine
info) lands in `bench.json`.

## Layout

```
benchmarks/
├── README.md (you are here)
├── conftest.py pytest-benchmark configuration
├── conftest.py pytest-benchmark configuration + seeded KB fixtures
├── fixtures/
│ ├── gen_kb.py synth a KB of N claims with realistic distributions
│ └── seed/ pre-built fixture KBs (small, medium, large)
│ └── gen_kb.py synth a KB of N claims with realistic distributions
├── bench_search.py kb.search latency at varying KB sizes
├── bench_propose.py kb.propose_* write latency
├── bench_bundle.py export + import + verify round-trips
└── bench_index_rebuild.py kb.index_rebuild at varying sizes
```

Benchmarks live outside `tests/` so a regular `pytest` run doesn't
pull them in. The intended invocation is:
pull them in. `pytest-benchmark` isn't in the `[dev]` extras, and the
`bench_*.py` filenames don't match pytest's default `python_files`
glob — so the invocation needs both an install and a collection
override:
Comment on lines +66 to +69

```bash
make bench # not yet wired in the Makefile
# or:
pytest benchmarks/ --benchmark-only --benchmark-json=bench.json
pip install pytest-benchmark
pytest benchmarks/ --benchmark-only \
-o python_files='bench_*.py test_*.py' \
--benchmark-json=bench.json
```

`make bench` is not wired in the Makefile yet; when it is, it should
fold in the `python_files` override so this isn't a footgun.
Comment on lines +78 to +79

## Methodology principles

- **Real disks.** No tmpfs benchmarks. The file-based design makes
Expand Down
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ web = [
]
dev = [
"pytest>=9.0.3,<10",
"pytest-benchmark>=5,<6",
"pytest-cov>=5,<6",
"mypy>=2.1.0",
"ruff>=0.15.13",
Expand Down