Rationale
For PoV benchmarking, we need some way to estimate what the maximum encoded size of an arbitrary storage item can be. We've settled on a new trait, MaxEncodedLen.
While Encode::size_hint already exists, we can't rely on it. One fatal flaw: it's an optional, "if possible" method which for many structs simply returns the default size of 0. Another: Encode is implemented for certain types such as VecDeque<T>, which we'd want to intentionally exclude from storage, as they aren't bounded. We need a trait which we can use to exclude types.
Design
pub trait MaxEncodedLen: Encode {
fn max_encoded_len() -> usize;
}Roadmap
Notes
This always explicitly computes an upper bound. While a u128 can potentially be compact-encoded in only one byte, Compact<u128>::max_encoded_len() will always return 17.
While implementations of BoundedVec<T> and friends impose a hard cap on their length, we do not use BoundedEncodedLen to impose a hard cap on the size of StorageMap and friends. While potentially a parachain author could misrepresent the bounds on its maps to attempt to pack a parachain block with more transactions, that behavior is self-defeating. Eventually they will try to pack too many transactions into a block, causing that block to become invalid because its actual size is too large. User documentation for the storage::map_bound attribute should emphasize that it is in the chain author's best interest to use a reasonable value here.
We'd like to offer logging warnings that could be noted by Prometheus etc when the size of StorageMap exceeds certain thresholds (eg. info at 80%, warn at 100%), but that feature is out of scope of this tracking issue. It requires additional design work because currently the only way to determine a map's size is to iterate over its members, which is not performant.
max_encoded_len is a function instead of an associated constant for pragmatic rather than theoretical reasons. Currently, BoundedVec's limit is not a constant, so we can't use its value in a constant context.
Rationale
For PoV benchmarking, we need some way to estimate what the maximum encoded size of an arbitrary storage item can be. We've settled on a new trait,
MaxEncodedLen.While
Encode::size_hintalready exists, we can't rely on it. One fatal flaw: it's an optional, "if possible" method which for many structs simply returns the default size of0. Another:Encodeis implemented for certain types such asVecDeque<T>, which we'd want to intentionally exclude from storage, as they aren't bounded. We need a trait which we can use to exclude types.Design
Roadmap
BoundedEncodedLentraitEncodeis implemented. Mostly defers tosp_std::mem::size_of::<T>().BoundedVec<T>Option<T>Result<T, E>PhantomData<T>Compact<T>Boundedvariants for other container types for whichEncodeis implemented. ImplementBoundedEncodedLenfor these bounded variants.BoundedBTreeMaptoframe_support::storage#8745BTreeMap<K, V>BoundedBTreeSet#8750BTreeSet<T>BinaryHeap<T>LinkedList<T>VecDeque<T>#[derive(MaxEncodedLen)]#8737 Derive macro:#[derive(Encode, BoundedEncodedLen)]for non-container structs, enums. Requires all fields to implementEncodeandBoundedEncodedLen.StorageMapand friends are actually implemented as newtypes which implement the full API of the parent type, plusBoundedEncodedLen#[storage::map_bound(300_000)]. The value specified in this attribute is used to determine the bounded length of the map.frame::storage: all storage items must implementBoundedEncodedLen.Notes
This always explicitly computes an upper bound. While a
u128can potentially be compact-encoded in only one byte,Compact<u128>::max_encoded_len()will always return 17.While implementations of
BoundedVec<T>and friends impose a hard cap on their length, we do not useBoundedEncodedLento impose a hard cap on the size ofStorageMapand friends. While potentially a parachain author could misrepresent the bounds on its maps to attempt to pack a parachain block with more transactions, that behavior is self-defeating. Eventually they will try to pack too many transactions into a block, causing that block to become invalid because its actual size is too large. User documentation for thestorage::map_boundattribute should emphasize that it is in the chain author's best interest to use a reasonable value here.We'd like to offer logging warnings that could be noted by Prometheus etc when the size of
StorageMapexceeds certain thresholds (eg. info at 80%, warn at 100%), but that feature is out of scope of this tracking issue. It requires additional design work because currently the only way to determine a map's size is to iterate over its members, which is not performant.max_encoded_lenis a function instead of an associated constant for pragmatic rather than theoretical reasons. Currently,BoundedVec's limit is not a constant, so we can't use its value in a constant context.