Repository navigation
A benchmark's runtime can depend on the presence/absence of other benchmarks #60
Description
Activity
I am observing a similar behavior. When I add a second benchmark the first one takes longer time. The two pieces of code I am benchmarking are completely unrelated. Also have tried this with many combinations and consistently seeing the same result.
The measurement should measure only the function under test and should be independent of the surrounding code if that's the case. I do not think the generated code of the two benchmarked functions per se is different in the two cases. They are completely unrelated.
Its hard to trust the benchmarking results because of this.
Here is a simplified test code:
import qualified Data.Text.ICU as TI ... main :: IO () main = do str <- readFile "data/English.txt" let str' = take 1000000 (cycle str) txt = T.pack str' str' `deepseq` txt `deepseq` defaultMain [ bgroup "text-icu" $ [bench "1" (nf (TI.normalize TI.NFD) txt)] , bgroup "just-count" $ [bench "1" (nf (show . length) str')] ]The first benchmark is measuring
text-icunormalize function. When run it with only the first benchmark alone it reports:benchmarking text-icu/1 time 2.830 ms (2.777 ms .. 2.913 ms)When I add the second one it becomes:
cueball:/vol/hosts/cueball/workspace/play/criterion$ ./Benchmark1 benchmarking text-icu/1 time 3.709 ms (3.570 ms .. 3.846 ms) benchmarking just-count/1 time 2.677 ms (2.516 ms .. 2.872 ms)A 30% degradation by just adding a line. The difference is even more marked in several other cases.
This problem is forcing me to run criterion with only one benchmark at a time. Also note that the benchmark result is wrong even when only one benchmark is chosen out of many via command line. Just the presence of another benchmark is enough irrespective of the runtime selection.
I am running criterion-1.1.1.0 and ghc-7.10.3.
It seems my problem was due to the sharing of input data across benchmarks which caused undue memory pressure for the later benchmarks. The problem got resolved by using
env. I rewrote the above code like this:setup = fmap (take 1000000 . cycle) (readFile "data/English1.txt") main :: IO () main = do defaultMain [ bgroup "text-icu" $ [ env (fmap T.pack setup) (\txt -> bench "1" (nf (TI.normalize TI.NFD) txt)) ] , bgroup "just-count" $ [ env setup (\str -> bench "1" (nf (show . length) str)) ] ]One possible enhancement could be to strongly recommend using
envin the documentation when using multiple benchmarks or even better if possible detect the case when env is not being used and issue a warning at runtime.Yes, being cognizant of working set is hard with Haskell's lazy semantics and GHC's optimizations. I don't know what would be detected here for a warning though -- what would the check for a linter be?
The original issue is a tricky one of compilation units. You can always put benchmarks in separate modules but that's a pain. I'm not sure what would be a good solution to alleviate this pain. TH doesn't seem sufficient.
It kind of seems like you'd want something like fragnix to create minimal compliation units that isolate your benchmarks.
FWIW, the
hspectest suite has a tool calledhspec-discoverthat automates the process of discovering other modules in a directory that contain tests (modules that end with the suffix-Spec). If isolating benchmarks into other modules is the recommended approach to solving this particular issue, we could consider implementinghspec-discover-style functionality to automate discover of benchmarks in other modules.See #166 for another example.
The following code compares two functions for summing over a vector:
But suppose I change the last few lines to the following:
This, surprisingly, affects the runtime of the
sumVbenchmark. It makes it about 20% faster. Similarly, if we remove thesumVbenchmark and leave theVU.sumbenchmark, theVU.sumbenchmark becomes about 20% slower. Tests were run with the patched criterion-1.0.0.2 I sent on ghc-7.8.3 with the-O2 -fllvmflags.What's going on is that different core is generated for the
sumVandVU.sumbenchmarks depending on whether the other benchmark is present. Essentially, the common bits are being factored out and placed in a function, and this function is getting called in both benchmarks. This happens to make both benchmarks faster.I'm not sure if this should be considered a "proper bug," but it confused me for a an hour or so. It's something that criterion users (especially those performing really small benchmarks) probably should be aware of.