Skip to content

feat(bench): migrate benchmarks to a GQT fixture and workload catalog - #894

Merged
azimafroozeh merged 2 commits into
ModernRelay:mainfrom
azimafroozeh:codex/0224-1-gqt-external-store
Oct 7, 2026
Merged

azimafroozeh merged 2 commits into
ModernRelay:mainfrom
azimafroozeh:codex/0224-1-gqt-external-store

Conversation

@azimafroozeh

Copy link
Copy Markdown
Collaborator

What & why

This PR uses GQT to define benchmark fixtures, workloads and correctness checks. It extracts shared parsing and execution into omnigraph-gqt-core, then adds a benchmark adapter that times one selected engine operation and verifies results outside that interval. Verified fixture templates are restored between repetitions.

Fixtures and workloads live in separate folders. One YAML catalog connects them to benchmark settings, with no separate case or suite YAML needed for new scenarios.

The catalog includes 73 scenarios and 12 groups, runnable by name or custom YAML. It adds 33 bounded end-to-end merge, load, branch, search, query and traversal scenarios with exact results and checks after reopening. The CLI provides config generation, JSON discovery and help, and read-only fixture cache inspection with optional byte verification. GQT also supports deterministic seed/load generation, while standalone GQT can execute against existing file, S3 and Azure stores.

Former Rust acquisition entry points are retired. Unsupported cleanup, optimization, concurrency, HTTP and physical/scale experiments are preserved with source checksums in benchmarks/deferred/, outside Cargo discovery. Cleanup and optimization await GQ support; GQT gains no separate maintenance commands. Ordinary correctness and deterministic cost tests remain.

Backing issue / RFC

Checklist

  • Change is focused (one workload language across correctness tests and benchmarks).
  • Tests added/updated for behavior changes (execution, generated inputs, cache boundaries, process supervision, catalog identity and compatibility).
  • Public docs updated (benchmark and GQT guides, RFCs and changelog).
  • Reviewed against docs/dev/invariants.md: no Hard Invariant weakened or deny-list item hit (engine operations retain their publication boundary; fixture caches remain derived state).

Local verification

  • cargo test -p omnigraph-bench --locked --no-fail-fast -q: 368 passed; one ignored subprocess helper.
  • Before the catalog migration, cargo test -p omnigraph-gqt --locked --lib --test runner_dispatch --test gq_logic_tests -q: 316 passed; the migration adds no GQT runtime changes.
  • cargo check --workspace --all-targets --locked -q --message-format=short: passed.
  • cargo clippy --workspace --all-targets --locked -q --message-format=short -- -D warnings -W clippy::dbg_macro: passed.
  • cargo fmt --all --check, typos, python3 scripts/check-ci-cells.py --self-test, bash scripts/check-agents-md.sh and the comment gate against the assigned base: passed.
  • python3 -m unittest discover -s scripts -p 'test_http_perf_analysis.py' -q: five passed.
  • All 22 archived sources match their original bytes and checksums. Cargo metadata confirms retired targets are absent. Independent reviews checked all 33 new result oracles and archive closure.
  • Before the catalog migration, cargo test --workspace --exclude omnigraph-gqt --exclude omnigraph-dst --locked --no-fail-fast --features omnigraph-engine/failpoints,omnigraph-cluster/failpoints -q: three existing managed CLI session-lock permission failures; other selected targets passed.
  • scripts/check-docs.py (documentation virtualenv): fails the frozen v0.12.0 release-note snapshot check.
  • cargo run -q -p omnigraph-vocabulary-guard -- check --surface all --base 420d742457fe3d8acf3495a4e37d298c257acde2 --inventory tools/omnigraph-vocabulary-guard/graph-vocabulary-inventory.tsv --openapi openapi.json: OpenAPI passed; merge-base Rust-string inventory errors prevent completion.
  • Qualified release timing remains unverified: the release binary refuses inherited tokio_unstable. Live S3/Azure execution and full-scale history/FinBench campaigns were not run. No performance figures are claimed.

Notes for reviewers

  • Legacy case/suite readers and historical record encodings remain supported. fixture run-graph and .run-v1.yaml execution are retired.
  • New bounded scenarios have different sizes, topology or measurement preparation from the retired instruments, so their timings are not interchangeable. The deferred optimize allocation-bound instrument is preserved but no longer executable.
  • The D50 warm workload now runs 24 authored aggregate queries. Its timings are not interchangeable with the previous warm program.
  • Cache treatment describes process and workload state; it does not establish a cold OS page cache or cold object storage.

@azimafroozeh
azimafroozeh force-pushed the codex/0224-1-gqt-external-store branch from c77c411 to bcecdcc Compare October 7, 2026 23:05
@azimafroozeh
azimafroozeh enabled auto-merge October 7, 2026 23:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant