Conversation
02b1033 to
c8bc43a
Compare
|
derek bench run-pairs=1 run-side=feature preset=public-mix |
|
cc @shekhirin ❌ Benchmark failed. View logs |
1 similar comment
|
cc @shekhirin ❌ Benchmark failed. View logs |
bc16f39 to
f56a26e
Compare
|
cc @decofe 🚀 Benchmark started! View job ⏳ Status: Running benchmark... Config: mode: |
|
cc @onbjerg ⚪ Benchmark complete: No Difference View job ⚪ Bench Comparison: No DifferenceRefs: a6d5349 vs 75fb3a9 Configuration
Tempo Metrics
Builder
Builder details
Validator
Observability |
|
cc @onbjerg ⚪ Benchmark complete: No Difference View job ⚪ Bench Comparison: No DifferenceRefs: a6d5349 vs 75fb3a9 Configuration
Tempo Metrics
Builder
Builder details
Validator
Observability |
#214) Setup currently inserts receipt barriers whenever consecutive transactions change sender or nonce lane, serializing independent accounts. Replace that heuristic with explicit `setup.steps[].depends_on` receipt dependencies: independent lanes submit concurrently, ordered nonce lanes still pipeline, and every setup transaction must succeed before workload starts. This is a breaking scheduling change. Existing workloads must declare deployment, configuration, funding, and authorization prerequisites explicitly. Dependencies on expanded steps wait for every emitted transaction. Generation and `bench send` validate the complete setup graph before output/submission, rejecting missing/duplicate IDs, unknown prerequisites, and cycles including nonce-lane ordering. Setup now also honors `--max-pending`. Forward receipt dependencies are supported by `generate | bench send`. Online scenario initialization remains serial and rejects forward dependencies before submission. Setup is buffered for validation; workload streaming is unchanged. Validation: core/CLI/Tempo suites passed (456 tests, one ignored), plus 12 setup sender tests including fan-in, independent submission, failures, and forward dependencies exceeding the bounded buffer under a pending cap. Workspace Clippy and nightly formatting passed. The unrelated `bench-cli` RPC replay integration test fails identically on the base commit with the installed Anvil (four successful responses instead of five). All GitHub CI checks pass. Companion workload migration: tempoxyz/tempo#7569. The matched multiregion benchmark reduced mean setup time from **729.81 s to 145.66 s**: **5.01× faster**, an **80.0% reduction**, saving **584.15 s (9m 44s)** per setup. | Scheduler | Repetition 1 | Repetition 2 | Mean setup time | | --- | ---: | ---: | ---: | | [Heuristic baseline](https://github.com/tempoxyz/tempo/actions/runs/35210688461) | 729.69 s | 729.94 s | 729.81 s | | [Explicit dependencies](https://github.com/tempoxyz/tempo/actions/runs/35215421489) | 145.75 s | 145.57 s | 145.66 s | All four measurements completed **3,451 setup transactions successfully, with zero setup failures**. Both workflows and their infrastructure teardown completed successfully. Timings come from `Setup transactions completed ... elapsed=...` in each phase's `txgen.out`; they include setup input reading, validation, submission, and receipt waits, and exclude infrastructure provisioning and subsequent workload execution. Inputs: 10 validators across `us-east-1,eu-central-1`, 1,000 users, 715 zones, `bloat=1` (reported as 1,000 MiB), target 50k TPS, 5,000 concurrent requests, a 30 s workload generation limit, and `--defer-signing`. Both runs use Tempo node commit `8e4c039b96c3e8feb7a59f171d17504de909679f` and benchmark runner `67ab5444bf0fdf4ff2d4b28eab46e72ead38d3a9`. Baseline txgen is `466fca39ca26d276c917878b7c34d25523772baa`; explicit-dependency txgen is `c70b70230f37ca5fee2851224b48170a5ca9d269`. Artifact verification confirmed identical rendered transaction definitions and workload weights after removing dependency metadata and normalizing the per-run temporary include directory. Each repetition sent 75–76 workload transactions after setup, all successful, so these measurements establish setup-time improvement for this configuration; they do not establish sustained throughput.
Declare the four receipt dependencies between the deployer and owner in the shared vault setup used by `vault-deposit` and `vault-withdraw`, so both remain correct when [txgen tempoxyz#214](tempoxyz/txgen#214) removes automatic sender-change barriers. These are the same vault edges exercised in [tempoxyz#7569](tempoxyz#7569), split out against main. Validated generation for both presets with 3 and 1,000 users (23 and 1,020 setup transactions), including emitted dependency IDs and graph validation; only dependency metadata changes.
…empoxyz#7698) Make withdrawal-position seeding wait for `wrapper`, the final deployment on the shared neobank deployer nonce lane, before the user calls the deployed fixture. This replaces reliance on automatic sender-change barriers removed by [txgen tempoxyz#214](tempoxyz/txgen#214), splitting the migration already included in [tempoxyz#7569](tempoxyz#7569) into a standalone change against main. Validated generation of all 12 setup transactions with the emitted `setup.seed_neobank_withdrawals → setup.wrapper` dependency; transaction definitions are otherwise unchanged.
Voight-Kampff: dismissing this approval review — it was never confirmed through a Voight-Kampff prompt, so it must not count toward merging this pull request. Re-submit the review and complete the Voight-Kampff prompt if you want this verdict to count.
Co-authored-by: Derek Cofausper <256792747+decofe@users.noreply.github.com>
Co-authored-by: Derek Cofausper <256792747+decofe@users.noreply.github.com>
Co-authored-by: Derek Cofausper <256792747+decofe@users.noreply.github.com>
Co-authored-by: Derek Cofausper <256792747+decofe@users.noreply.github.com>
Co-authored-by: Derek Cofausper <256792747+decofe@users.noreply.github.com>
Use 25-second expiring nonces for expanded zone and vault workload templates, matching transfers, mints, and MPP. Co-authored-by: Derek Cofausper <256792747+decofe@users.noreply.github.com>
9b96ad9 to
0b84953
Compare
…nes-vaults-1788961139
|
@decofe two findings from running the multi-region bench with this preset today (
|
|
derek bench preset=public-mix baseline=d1e08849d6168caac3d7c7413b0b2c068a674ee8 feature=mattsse/builder-precompile-cache txgen-ref=9d2124ad5831e848381b369dea3875c3e7999a38 |
|
cc @decofe
|
| Metric | Baseline | Feature | Delta |
|---|---|---|---|
| TPS Mean | 11036 | 11179 | +1.30% ⚪ (+/-0.97/floor 0.55) |
| Gas Throughput [Mgas/s] | 1816.8 | 1833.4 | +0.91% ⚪ (+/-0.82/floor 0.50) |
| Block Time Mean [ms] | 401.0 | 399.1 | -0.47% ⚪ (+/-0.70/floor 0.40) |
| Block Time P50 [ms] | 417.0 | 433.0 | +3.84% ❌ (+/-2.00/floor 0.70) |
| Block Time P90 [ms] | 536.0 | 530.0 | -1.12% ⚪ (+/-1.37/floor 0.70) |
| Block Time P99 [ms] | 603.0 | 599.0 | -0.66% ⚪ (+/-4.01/floor 1.60) |
Builder
| Metric | Baseline | Feature | Delta |
|---|---|---|---|
| Gas Throughput [Mgas/s] | 2882.1 | 2939.9 | +2.01% ✅ (+/-0.76/floor 0.95) |
| P50 [ms] | 249.3 | 245.6 | -1.48% ✅ (+/-0.36/floor 0.45) |
| P90 [ms] | 284.5 | 276.0 | -2.99% ✅ (+/-0.71/floor 0.90) |
| P99 [ms] | 329.8 | 310.0 | -6.00% ⚪ (+/-6.25/floor 1.25) |
Builder details
| Metric | Baseline | Feature | Delta |
|---|---|---|---|
| Finish P50 [ms] | 5.0 | 5.3 | +6.00% |
| Finish P90 [ms] | 20.8 | 25.1 | +20.67% |
| Finish P99 [ms] | 70.2 | 69.9 | -0.43% |
| Pool Fetch P50 [ms] | 5.6 | 5.8 | +3.57% |
| Pool Fetch P90 [ms] | 9.9 | 10.2 | +3.03% |
| Pool Fetch P99 [ms] | 18.0 | 15.6 | -13.33% |
| Reverted Txs | 259270 | 261832 | +0.99% |
| Invalid Tx Skips | 6 | 15 | +150.00% |
| Stop Reason — Build Budget | 654 | 659 | +0.76% |
| Serialized Block Size P50 [KiB] | 1367.8 | 1410.6 | +3.13% |
| Serialized Block Size P90 [KiB] | 2585.4 | 2555.5 | -1.16% |
| Serialized Block Size P99 [KiB] | 3235.9 | 3089.0 | -4.54% |
| Serialized Block Size / Tx P50 [B/tx] | 349.8 | 345.3 | -1.29% |
| Serialized Block Size / Tx P90 [B/tx] | 507.1 | 506.1 | -0.20% |
| Serialized Block Size / Tx P99 [B/tx] | 513.1 | 514.6 | +0.29% |
| Fill Idle P50 [ms] | 0.0 | 0.0 | 0.00% |
| Fill Idle P90 [ms] | 0.0 | 0.0 | 0.00% |
| Fill Idle P99 [ms] | 0.0 | 0.0 | 0.00% |
Validator
| Metric | Baseline | Feature | Delta |
|---|---|---|---|
| Gas Throughput [Mgas/s] | 3488.0 | 3510.5 | +0.64% ⚪ (+/-0.80/floor 0.65) |
| P50 [ms] | 208.1 | 207.4 | -0.34% ⚪ (+/-1.63/floor 1.55) |
| P90 [ms] | 256.9 | 255.8 | -0.43% ⚪ (+/-1.10/floor 1.55) |
| P99 [ms] | 315.2 | 298.7 | -5.23% ⚪ (+/-4.88/floor 2.05) |
Observability
Warn/Error Logs
| Run type | Total lines |
|---|---|
| Baseline | 36 |
| Feature | 36 |
Counts by message
| Message | Baseline | Feature |
|---|---|---|
| context request for block `0x589f5a3c363fedfadd7151f08c62fca9b67f920fee89209193786777befa017c` with no consensus context | 12 | 12 |
| failed to read dealer log from block extraData header field | 12 | 12 |
| requested buffer capacity is too low, increasing it to floor | 6 | 6 |
| executor could not verify the block; abstaining | 2 | 2 |
| executor encountered fatal execution-layer update error; shutting down to prevent consensus-execution divergence | 2 | 2 |
| [failed delivering block `0x1cc707d71562bd33537adfe3c6501e3e786b4d25c3256f600cae1ed5a7944c3d` for verification (0, 374), failed sending new-payload request to execution layer, beacon consensus engine task stopped] | 0 | 1 |
| [failed delivering block `0x2d80c70ffbb4ee771679f7a88088109d59bb997e5ed8d2b8f8459bcfed0e5b37` for verification (0, 356), failed sending new-payload request to execution layer, beacon consensus engine task stopped] | 1 | 0 |
| [failed delivering block `0x472e0c749e9a10df07f7899af9b06370ea8f9f4c9276a0e9e84f2a581f6114c5` for verification (0, 351), failed sending new-payload request to execution layer, beacon consensus engine task stopped] | 1 | 0 |
| [failed delivering block `0x8769d0691f15efaa5ee8e70154da498404aa37066154cd3c04341f630313523c` for verification (0, 322), failed sending new-payload request to execution layer, beacon consensus engine task stopped] | 0 | 1 |
Co-authored-by: Derek Cofausper <256792747+decofe@users.noreply.github.com>
Adds zone deposits/withdrawals at 5%/4% and vault deposits/withdrawals at 6%/4% to
public-mix, matching the requested table; transfers, memo transfers, mints, and open-only MPP remain at 25%/40%/1%/15%. Reuses the zone fixtures merged in #7537 and adds the fixture setup, funding, and per-user/portal weighting needed to compose the workloads.Also leaves the zone account-range placeholder unquoted so environment expansion yields an integer when the multi-region runner loads the preset directly. This corrects the
invalid type: string "1000", expected u32error in the failed comparison.Updated against main to include the merged compact metadata producer from #7663. Preserves this PR’s existing allocation, including memo transfers, as requested. Metadata is extracted after fixture expansion and reports aggregate category weights; the merged perf reader and multi-region sender already support these categories, so no companion changes are needed.
Declares explicit vault and neobank setup dependencies for txgen #214. Independent Zone deployment/funding and per-user approvals can overlap vault initialization, while receipt dependencies protect registry publication, engine binding, factory calls, and neobank position seeding. Use
txgen-ref=9d2124ad5831e848381b369dea3875c3e7999a38, pinned to txgen main as of 2026-09-24. This revision includes the merged explicit-dependency scheduler and the--warmup-validatorssupport required by the multi-region runner.The existing
bench-e2e.nu render-txgen-speccommand composes public-mix using the supplied accounts, TPS, and duration; preparation reuses that rendered file unchanged. Other presets retain their existing rendering and pipeline behavior.Validation: rendered and generated setup for 3 users/2 zones and 1,000 users/715 zones (3,451 setup transactions). Compared the original and migrated renderings: transaction definitions, senders, ordering, templates, and exact eight-category weights are unchanged; only dependency metadata differs. The public-mix render command produces identical specifications at both sizes (31 and 3,451 setup transactions), and already-rendered files are reused unchanged. Two mocked-RPC public-mix pipeline repetitions preserve the input and generate workload successfully.
The matched multiregion benchmark reduced mean setup time from 729.81 s to 145.66 s: 5.01× faster, an 80.0% reduction, saving 584.15 s (9m 44s) per setup.
All four measurements completed 3,451 setup transactions successfully, with zero setup failures. Both workflows and their infrastructure teardown completed successfully. Timings come from
Setup transactions completed ... elapsed=...in each phase'stxgen.out; they include setup input reading, validation, submission, and receipt waits, and exclude infrastructure provisioning and subsequent workload execution.Inputs: 10 validators across
us-east-1,eu-central-1, 1,000 users, 715 zones,bloat=1(reported as 1,000 MiB), target 50k TPS, 5,000 concurrent requests, a 30 s workload generation limit, and--defer-signing. Both runs use Tempo node commit8e4c039b96c3e8feb7a59f171d17504de909679fand benchmark runner67ab5444bf0fdf4ff2d4b28eab46e72ead38d3a9. Baseline txgen is466fca39ca26d276c917878b7c34d25523772baa; explicit-dependency txgen isc70b70230f37ca5fee2851224b48170a5ca9d269.Artifact verification confirmed identical rendered transaction definitions and workload weights after removing dependency metadata and normalizing the per-run temporary include directory. Each repetition sent 75–76 workload transactions after setup, all successful, so these measurements establish setup-time improvement for this configuration; they do not establish sustained throughput.
Prompted by: @shekhirin