Skip to content

bench(efficiency): reusable import, hot-path, Postgres and cold-start benchmark harness - #6357

Closed
time-attack wants to merge 2 commits into
masterfrom
capy/reusable-import-hot-path
Closed

time-attack wants to merge 2 commits into
masterfrom
capy/reusable-import-hot-path

Conversation

@time-attack

Copy link
Copy Markdown
Collaborator

Opt-in benchmark scripts under scripts/bench/efficiency/ so perf fix threads can measure before/after on the same machine. Nothing here runs in CI, ships in the CLI, or touches product code. Brain data, synthetic brains, engine homes and results stay under $BENCH_WORK (default ~/.capy/work/brain) and are never committed; result rows carry counts, sizes and timings only.

What's included:

  • Data: synth-brain.ts derives a shape (per-source/type counts, size samples, timeline and wikilink density) from a pull manifest and generates a deterministic synthetic markdown brain at any --scale (generated words, alice-example style names). materialize-pull.ts turns a local pull into an importable directory.
  • Import: bench-import.ts builds a fresh isolated brain per engine (PGLite, or Postgres+pgvector in a gbra66-bench-pg container) and times init, per-source import, extract all and embed --stale, with engine-method, HTTP and CPU splits from preload-instrument.ts (a bun --preload hook; it records provider token usage, never bodies) and USD from provider-reported tokens.
  • Hot paths: bench-hot.ts covers search, query with and without expansion, get_page, list_pages, recall, doctor, MCP stdio and HTTP startup plus one call, and one-page sync, both cold (a fresh CLI process per sample) and warm (one long-lived gbrain serve). Each row records machine, engine, brain size, command, warm/cold, N, p50 and p95.
  • Probes: probe-pg.ts resets pg_stat_statements around N warm calls of one tool and prints the top statements (placeholders only), with a --jit off A/B. probe-cold.ts splits a cold CLI call into bun start, module graph, --version and real reads, each with and without the non-TTY exit-flush grace.
  • report.ts renders results to markdown, run-all.sh orchestrates everything, and README.md explains usage and caveats.

bun run verify passes, typecheck included (76 checks). Requires Bun >= 1.4 (the repo's engines.bun) and Docker for the Postgres engine.

Open in Capy

… benchmark harness

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@trunk-io

trunk-io Bot commented Oct 9, 2026

Copy link
Copy Markdown

Merging to master in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

… percentiles

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@capy-ai

capy-ai Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Superseded by #6390, which carries this harness current with master. Closing this draft.

@capy-ai capy-ai Bot closed this Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant