Repository navigation
feat(bench): migrate benchmarks to a GQT fixture and workload catalog - #894
Merged
azimafroozeh merged 2 commits intoOct 7, 2026
Merged
Conversation
azimafroozeh
force-pushed
the
codex/0224-1-gqt-external-store
branch
from
October 7, 2026 23:05
c77c411 to
bcecdcc
Compare
azimafroozeh
enabled auto-merge
October 7, 2026 23:05
This was referenced Oct 8, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
This PR uses GQT to define benchmark fixtures, workloads and correctness checks. It extracts shared parsing and execution into
omnigraph-gqt-core, then adds a benchmark adapter that times one selected engine operation and verifies results outside that interval. Verified fixture templates are restored between repetitions.Fixtures and workloads live in separate folders. One YAML catalog connects them to benchmark settings, with no separate case or suite YAML needed for new scenarios.
The catalog includes 73 scenarios and 12 groups, runnable by name or custom YAML. It adds 33 bounded end-to-end merge, load, branch, search, query and traversal scenarios with exact results and checks after reopening. The CLI provides config generation, JSON discovery and help, and read-only fixture cache inspection with optional byte verification. GQT also supports deterministic seed/load generation, while standalone GQT can execute against existing file, S3 and Azure stores.
Former Rust acquisition entry points are retired. Unsupported cleanup, optimization, concurrency, HTTP and physical/scale experiments are preserved with source checksums in
benchmarks/deferred/, outside Cargo discovery. Cleanup and optimization await GQ support; GQT gains no separate maintenance commands. Ordinary correctness and deterministic cost tests remain.Backing issue / RFC
Checklist
docs/dev/invariants.md: no Hard Invariant weakened or deny-list item hit (engine operations retain their publication boundary; fixture caches remain derived state).Local verification
cargo test -p omnigraph-bench --locked --no-fail-fast -q: 368 passed; one ignored subprocess helper.cargo test -p omnigraph-gqt --locked --lib --test runner_dispatch --test gq_logic_tests -q: 316 passed; the migration adds no GQT runtime changes.cargo check --workspace --all-targets --locked -q --message-format=short: passed.cargo clippy --workspace --all-targets --locked -q --message-format=short -- -D warnings -W clippy::dbg_macro: passed.cargo fmt --all --check,typos,python3 scripts/check-ci-cells.py --self-test,bash scripts/check-agents-md.shand the comment gate against the assigned base: passed.python3 -m unittest discover -s scripts -p 'test_http_perf_analysis.py' -q: five passed.cargo test --workspace --exclude omnigraph-gqt --exclude omnigraph-dst --locked --no-fail-fast --features omnigraph-engine/failpoints,omnigraph-cluster/failpoints -q: three existing managed CLI session-lock permission failures; other selected targets passed.scripts/check-docs.py(documentation virtualenv): fails the frozen v0.12.0 release-note snapshot check.cargo run -q -p omnigraph-vocabulary-guard -- check --surface all --base 420d742457fe3d8acf3495a4e37d298c257acde2 --inventory tools/omnigraph-vocabulary-guard/graph-vocabulary-inventory.tsv --openapi openapi.json: OpenAPI passed; merge-base Rust-string inventory errors prevent completion.tokio_unstable. Live S3/Azure execution and full-scale history/FinBench campaigns were not run. No performance figures are claimed.Notes for reviewers
fixture run-graphand.run-v1.yamlexecution are retired.