Repository navigation
Refresh and pare down benchmarks game benchmarks #1452
Description
Some of our Benchmark Games benchmarks are a bit old and don't match what they're using now. We also have several copies of 7 of them, which seems wasteful.
- spectralnorm winner is # 5 (uses intrinsics). We should pick that up and drop # 1 and possibly the other.
- k-nucloetide winner is # 1 (uses DictionarySlim). We should probably pick that up and drop the different # 1 that we have and possibly the other.
- binarytrees winner is # 6. We could replace our # 5 with that (both use Task; we also have # 2 that does not and could drop that)
- fannkuch-redux winner is # 9. We could replace our # 5, which is very similar and possibly drop the other too.
- fasta winner is # 5. We probably should replace our our # 2 which is quite different and possibly drop the other.
- Mandelbrot winner is # 9. We could replace our # 7 as it's a progression and possibly drop the other.
- regex-redux # 5 which we already have. We could drop the other one.
I'm not sure what our criteria are for dropping benchmarks but perhaps it would make sense after the 6.0 snap.
cc @AndyAyersMS
Activity
AndyAyersMS commentedon Aug 14, 2020
Initially I just wanted to select versions that were good tests for codegen, and not worry so much about which were the fastest. A few years back we augmented the initial set with the then-fastest variants, which is why we have two or more of some of the tests.
I'm ok with updating any or all of these, provided that for a given test, we don't exclusively pick variants that dive into native code for the interesting bits of computation (pcre or gmp). If want to run these native variants to keep track of any perf gap, we should also pick the fastest version that's all our own codegen.
danmoseley commentedon Aug 14, 2020
OK, well let's not remove any until 5.0 is shipped, then we can think about it if we want.
adamsitnik commentedon Aug 14, 2020
Perhaps we should just add new benchmarks and keep the old ones?
danmoseley commentedon Aug 14, 2020
Not sure how short perf machine time is though.
adamsitnik commentedon Aug 14, 2020
This repo is configured in a way that every benchmark iteration should take 250ms and there should be no more than 20 iterations. So on average, every benchmark takes 20x0.25s = 5s to execute.
The benchmarks are split into partitions and run using multiple machines at the same time.
So in general adding a few benchmarks that make sense is never a problem.
danmoseley commentedon Aug 14, 2020
We might consider having just one variant for each scenario and using the official parameters. I think they take 1-8 secs an iteration but we’d be sure of measuring the same dynamics. Don’t know whether it matters.
billwert commentedon Aug 14, 2020
Right now we are a long way from being machine bound. When we get there we'll take a much more holistic approach to trimming.
billwert commentedon Sep 22, 2021
Closing this issue as we still aren't really machine bound.
danmoseley commentedon Oct 18, 2021
Given we aren't machine bound, this issue is worth keeping open. If we were, we could close it.
danmoseley commentedon Oct 20, 2021
A few of the winners have changed, roughly:
k-nuc # 1 (different to our # 1 -- replace ours)
fasta # 5 (replace our similar # 1)
mandelbrot # 1 (replacing our # 7 here)
reverse complement # 7 (possibly replace one of ours, they are quite different)
BruceForstall commentedon Oct 26, 2021
cc @kunalspathak