Skip to content

feat(mainnet): include all benchmarks#478

Merged
al3mart merged 28 commits into
mainfrom
al3mart/feat-mainnet-all-benchmarks
Mar 11, 2025
Merged

feat(mainnet): include all benchmarks#478
al3mart merged 28 commits into
mainfrom
al3mart/feat-mainnet-all-benchmarks

Conversation

@al3mart

@al3mart al3mart commented Feb 20, 2025

Copy link
Copy Markdown
Member

Adds to mainnet runtime benchmarks for all pallets.

  • Remove default weight configuration.
  • Include pallets in define_benchmarks
  • Script to easily run benchmarks across machines

Depends on:


On introducing the benchmarks, these are also refactored into pop_runtime_common, such that they can be used from any of the runtimes. This refactor is pushed in: da0aedf


Closes: #435

[sc-2757]

@codecov-commenter

codecov-commenter commented Feb 20, 2025

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 4.55779% with 3518 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.51%. Comparing base (a51ea5f) to head (c9aa96b).
⚠️ Report is 27 commits behind head on main.

Files with missing lines Patch % Lines
runtime/common/src/weights/pallet_revive.rs 8.41% 588 Missing ⚠️
runtime/common/src/weights/pallet_nfts.rs 0.00% 379 Missing ⚠️
runtime/common/src/weights/pallet_assets.rs 0.00% 315 Missing ⚠️
...n/src/weights/xcm/pallet_xcm_benchmarks_generic.rs 0.00% 265 Missing ⚠️
runtime/mainnet/src/config/xcm_weights.rs 0.00% 221 Missing ⚠️
runtime/common/src/weights/pallet_xcm.rs 0.00% 191 Missing ⚠️
runtime/common/src/weights/pallet_collective.rs 0.00% 171 Missing ⚠️
runtime/common/src/weights/pallet_scheduler.rs 0.00% 158 Missing ⚠️
runtime/common/src/weights/pallet_preimage.rs 0.00% 126 Missing ⚠️
...me/common/src/weights/pallet_collator_selection.rs 11.45% 116 Missing ⚠️
... and 20 more
@@             Coverage Diff             @@
##             main     #478       +/-   ##
===========================================
- Coverage   74.69%   63.51%   -11.18%     
===========================================
  Files          98      125       +27     
  Lines       19424    23099     +3675     
  Branches    19424    23099     +3675     
===========================================
+ Hits        14508    14672      +164     
- Misses       4649     8160     +3511     
  Partials      267      267               
Files with missing lines Coverage Δ
runtime/common/src/lib.rs 85.24% <ø> (ø)
runtime/common/src/weights/block_weights.rs 81.81% <ø> (ø)
runtime/common/src/weights/extrinsic_weights.rs 81.81% <ø> (ø)
runtime/common/src/weights/paritydb_weights.rs 73.33% <ø> (ø)
runtime/common/src/weights/rocksdb_weights.rs 73.33% <ø> (ø)
runtime/mainnet/src/config/assets.rs 100.00% <ø> (ø)
runtime/mainnet/src/config/collation.rs 100.00% <100.00%> (ø)
runtime/mainnet/src/config/governance.rs 100.00% <100.00%> (ø)
runtime/mainnet/src/config/proxy.rs 92.30% <ø> (ø)
runtime/mainnet/src/config/revive.rs 94.50% <ø> (ø)
... and 34 more
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@al3mart al3mart changed the title feat(benchmakrs): include all benchmarks for mainnet feat(benchmarks): include all benchmarks for mainnet Feb 21, 2025
@al3mart al3mart changed the title feat(benchmarks): include all benchmarks for mainnet feat(mainnet): include all benchmarks Feb 24, 2025
@al3mart al3mart force-pushed the al3mart/feat-mainnet-all-benchmarks branch from ec2ffde to 1c65204 Compare February 24, 2025 23:01
Comment thread pallets/motion/src/benchmarking.rs
Comment thread runtime/mainnet/src/config/monetary.rs
@al3mart al3mart marked this pull request as ready for review February 25, 2025 15:45
@al3mart al3mart requested a review from a team February 25, 2025 15:46
Comment thread scripts/benchmarks-ci.sh
Comment thread runtime/mainnet/src/config/collation.rs

@peterwht peterwht left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incredibly clean PR. I've got no concerns. Will wait to approve until after XCM is merged

Comment thread runtime/mainnet/src/benchmarks.rs
@Daanvdplas Daanvdplas self-requested a review February 28, 2025 08:20

@Daanvdplas Daanvdplas left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The weight results are a little bit concerning to me as there are many differences when comparing it to asset hub or the other project that we work on. Also, have you thought about putting the weight functions we use for the runtimes in a common folder?

Comment thread pallets/motion/src/benchmarking.rs
Comment thread runtime/mainnet/src/benchmarks.rs Outdated
Comment thread runtime/mainnet/src/config/collation.rs
Comment thread runtime/mainnet/src/config/collation.rs
Comment thread runtime/mainnet/src/config/governance.rs
Comment thread runtime/mainnet/src/weights/mod.rs
Comment thread runtime/mainnet/src/weights/frame_system_extensions.rs Outdated
Comment thread runtime/mainnet/src/weights/pallet_assets.rs Outdated
Comment thread runtime/mainnet/src/weights/pallet_balances.rs
Comment thread runtime/mainnet/src/weights/pallet_collator_selection.rs Outdated
@al3mart

al3mart commented Feb 28, 2025

Copy link
Copy Markdown
Member Author

I've re-excetued the benchmarks now and using a configuration with more steps and repetitions. I can see complexity parameters being included in the results now. cc @Daanvdplas

Although, I can still see some divergence between the #read/writes in these results and what we have seen in other places. I believe some related to balances are influenced by the existence of MaintenanceAccount.

@al3mart al3mart force-pushed the al3mart/feat-mainnet-all-benchmarks branch from 437de12 to fd64662 Compare March 4, 2025 15:58
Comment thread scripts/benchmarks-ci.sh
Comment thread scripts/benchmarks-ci.sh
al3mart added 22 commits March 10, 2025 18:12
* feat(benchmark): add pallet_xcm extrinsic benchmarks

* refactor(benchmkars): amend xcm transfer_assets benchmark

fmt

* style(benchmarks): pretty comments.

* refactor(benchmarks): fund SA of AH on pop in reserve_transfer benchmark

* feat(mainnet): add generic & fungible xcm benchmarks

* refactor(benchmarks): don't define CheckAccount for pallet_xcm_benchmarks::fungible

* refactor(benchmarks): amend benchmarks for pallet_xcm_benchmarks::generic::unsubscribe_version

* feat(mainnet): configure xcm Weigher as WeightInfoBounds

* docs(xcm): missing doc

* refactor(weights): MAX_ASSETS is MaxAssetsIntoHolding

* refactor(mainnet): remove default weights from queue pallets

* docs(xcm): credit source for WeighAssets impl

* docs(xcm): reminders for updating benchmarks

* refactor(mainnet): DeliveryHelper considers delivery to parent

* docs(benchmarks): clarify xcm NonFungible AllOf weight comments

* refactor(api): reorg imports

* style(benchmark): refactor import syntax

* docs(bencmark): better docs for XcmBalances

* refactor(benchmark): define locations via xcm config definitions

* test(xcm): ensure xcm encoding size

* refactor(benchmark): improve worst_case_response

* chore(benchmark): update results for query_response bench

* style(benchmarks): remove RelayAsset

* docs(benchmarks): explain ExistentialDepositAsset

* docs(xcm): note worst_case_holding needs updating

* style(benchmark): better naming

* chore(benchmarks): update xcm benchmarks

visibility

* test(xcm): amend router_uses_ump_for_relay_and_xcmp_for_para

* test(xcm): amend router_uses_ump_for_relay_and_xcmp_for_para

* refactor(benchmarks): unnecessary ParaId::from

* refactor(xcm): unnecessary u64 cast

* test(xcm): msg_length factor is > 1 in price_for_parent_delivery~
@al3mart al3mart force-pushed the al3mart/feat-mainnet-all-benchmarks branch from 3eff2d3 to c9aa96b Compare March 11, 2025 08:23
@al3mart al3mart merged commit b1e0fa9 into main Mar 11, 2025
@al3mart al3mart deleted the al3mart/feat-mainnet-all-benchmarks branch March 11, 2025 10:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(mainnet): include pallets in define_benchmarks and remove default configs

4 participants