Repository navigation
bench(efficiency): reusable import, hot-path, Postgres and cold-start benchmark harness - #6357
Closed
time-attack wants to merge 2 commits into
Closed
time-attack wants to merge 2 commits into
time-attack wants to merge 2 commits into
Conversation
… benchmark harness Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
… percentiles Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This was referenced Oct 9, 2026
Contributor
|
Superseded by #6390, which carries this harness current with master. Closing this draft. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Opt-in benchmark scripts under
scripts/bench/efficiency/so perf fix threads can measure before/after on the same machine. Nothing here runs in CI, ships in the CLI, or touches product code. Brain data, synthetic brains, engine homes and results stay under$BENCH_WORK(default~/.capy/work/brain) and are never committed; result rows carry counts, sizes and timings only.What's included:
synth-brain.tsderives a shape (per-source/type counts, size samples, timeline and wikilink density) from a pull manifest and generates a deterministic synthetic markdown brain at any--scale(generated words,alice-examplestyle names).materialize-pull.tsturns a local pull into an importable directory.bench-import.tsbuilds a fresh isolated brain per engine (PGLite, or Postgres+pgvector in agbra66-bench-pgcontainer) and times init, per-source import,extract allandembed --stale, with engine-method, HTTP and CPU splits frompreload-instrument.ts(abun --preloadhook; it records provider token usage, never bodies) and USD from provider-reported tokens.bench-hot.tscovers search, query with and without expansion, get_page, list_pages, recall, doctor, MCP stdio and HTTP startup plus one call, and one-page sync, both cold (a fresh CLI process per sample) and warm (one long-livedgbrain serve). Each row records machine, engine, brain size, command, warm/cold, N, p50 and p95.probe-pg.tsresetspg_stat_statementsaround N warm calls of one tool and prints the top statements (placeholders only), with a--jit offA/B.probe-cold.tssplits a cold CLI call into bun start, module graph,--versionand real reads, each with and without the non-TTY exit-flush grace.report.tsrenders results to markdown,run-all.shorchestrates everything, andREADME.mdexplains usage and caveats.bun run verifypasses, typecheck included (76 checks). Requires Bun >= 1.4 (the repo'sengines.bun) and Docker for the Postgres engine.