Skip to content

How can I do shorter benchmarks? #210

Open
Open
How can I do shorter benchmarks?#210
Labels
Documentation

Description

@chrissound
chrissound
opened on Feb 1, 2019
module Main where

import Criterion.Types
import Criterion.Main

myConfig :: Config
myConfig = defaultConfig {
              resamples = 1
           , timeLimit = 0.01
           }

main :: IO ()
main = do
  let f = (\x -> bench (show x)  $ nf xyz x)
  defaultMainWith myConfig [
    bgroup "fib" [
                  env (pure 400000) f
                , env (pure 100) f
                , env (pure 200) f
                ]
    ]

xyz :: Int -> [Double]
xyz 0 = []
xyz _ = [fromIntegral 1000]

I was hoping I could quickly benchmark an example like this - but it seems to be taking around 4 seconds - to do numerous(?) runs.

Have I misunderstood the resamples / timeLimit values?

Activity

  1. RyanGlScott
    added
    Documentation
    on Feb 1, 2019

  2. RyanGlScott commented on Feb 1, 2019

    @RyanGlScott
    RyanGlScott
    on Feb 1, 2019
    Member

    In order to ensure that criterion generates enough data for us to have some trust in the raw measurements, criterion sets a minimum time threshold that a benchmark must run for:

    https://github.com/bos/criterion/blob/642bea6d15b9a1737c82ffd051f663edcb89d280/criterion-measurement/src/Criterion/Measurement.hs#L240-L250

    This is where the threshold is used in the inner loop of benchmarking:

    https://github.com/bos/criterion/blob/642bea6d15b9a1737c82ffd051f663edcb89d280/criterion-measurement/src/Criterion/Measurement.hs#L300-L315

    If you do the math here, it will likely add up to the time you're seeing.

    Admittedly, the Haddocks for timeLimit don't make mention of this fact, although they probably should.

  3. jberryman commented on Dec 12, 2019

    @jberryman
    jberryman
    on Dec 12, 2019

    I have some benchmarks where a single iteration takes ~5min. Is there any alternative for me other than rewriting them outside of criterion? I essentially want to run them just once and have the time reported.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Documentation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions