Repository navigation
Logarithmic scale when plotting results #213
Description
Activity
criterioncurrently doesn't support configuring logarithmic scales to my knowledge, no.As for how difficult it would be to implement such a feature, I don't know off the top of my hand. That being said, the plotting functionality in this
criterionmakes use of theflotJavaScript library, so ifflotsupports logarithmic scales, there may be a way to incorporate that functionality intocriterionas well.According to flot documentation you can use something like this:
xaxis: { transform: function (v) { return Math.log(v); }, inverseTransform: function (v) { return Math.exp(v); } }
in the javascript (in the html template).
Note that I haven't tested this.
I also think while the criterion report looks pretty it should have more features / be more easily extendable. E.g able to split the overview graph at the top into groups. But maybe that is better implemented in another package and not in criterion directly.
Thanks for the tip, @Lythenas. A next step will be figuring out what kind of interface we should offer for tweaking the axes' scales in the HTML report itself. One option is to accept extra command-line arguments for configuring this. Alternatively, we could try offering this UI in the report itself, although that would be much more work to implement.
I not sure what the best approach is. I think a general interface to write custom report exporters would be the easiest way to expand criterion. But I would keep the default exporter as is (by default) because for basic benchmarks it is fine IMO.
I wouldn't make this configurable by command line because it will likely get very complicated very fast. Maybe a list of presets would be ok as command line arguments.
I'm thinking criterion could offer something like this:
class ReportExporter where formatReport :: [Report] -> Text
(and maybe helpers to extend the default flot template)
And I could write my own exporter that generates HTML or custom JSON or whatever I want.
FYI: hvega looks like it has a nice DSL to generate data for Vega-Lite (which is an alternative to flot). Maybe we could build something similar for flot or replace flot with vega lite (maybe in an alternative exporter).
There should be a simple heuristic to choose between a linear scale or a log scale when the bar plot would otherwise be useless, which is most of the time in my experience
i think log scale should probably be the default ... maybe wiht the base "ofset" at the low end at pico sends? or we shift to box and whisker plots? on log scale https://en.wikipedia.org/wiki/Box_plot
maybe @Hasufel or @Bodigrim have thoughts? (theres also the issue that default rendering js last i looked doesn't just consume summary statistics, but rather all the points, last i looked? or some other insane algorithmic issue )
There should be a simple heuristic to choose between a linear scale or a log scale when the bar plot would otherwise be useless, which is most of the time in my experience
What kind of heuristic could it be? I cannot think of anything simple and one-size-fits-all.
i think log scale should probably be the default
Use cases for benchmarking are quite diverse. Like, if we benchmark different implementations of an algorithm, then log-scale usually looks very sensible, because it nicely captures differences in asymptotical growth without making plot unreadable. But if I benchmark the very same function on different inputs, I would definitely prefer a normal scale.
or we shift to box and whisker plots?
Absolutely, it would be nice to have a box-and-whisker plot. But altogether it looks to me that this is not a job for
criterion. There are many ways to transform a statistical data, far more than it is sensible to configure fromcriterion's command line.I remember that
criterioncan dump results as JSON. Isn't it enough? External tools are free to analyse and visualise it in any imaginable way.I think the linear scale should be the default, I think it's kind of a general assumption regarding any plot
I respectfully 100% disagree and I am more than happy to have an educational resource collectiong/back and forth email/whatever with both of you to make this case.
I realize that people dont see log scale plots till relatively late in their educations, but for the way people use criterion: namely to compare different algorithms, we care about the multiplicative difference between two algorithms on the same input (2x or 10x or whatever speed difference). Default linear scales makes the emphasis on whatever algorithm is slowest. and complicate actually visually seeing multiplicative difference in perf.
(who using criterion hasn't commented out the slowest algorithm on their machine so the visual scale "focuses" on the actual contenders?). I'm more than happy to also provide clear opinion references from statistics/applied statistics/public health where correct interpretable data plotting is part of the core decision making loop.
I'm wiling to be convinced i'm wrong, but I ask that you give me the opportunity to convince you and address whatever grounded reasons you have in mind.
(also i'm interested in making this happen :) )
but for the way people use criterion: namely to compare different algorithms, we care about the multiplicative difference between two algorithms on the same input (2x or 10x or whatever speed difference).
If we use criterion to compare different algorithms, then yes. My argument is that this is not always the case. For instance, I often test the same function on different sizes of input (like,
[100,110..200]), for which I would very much prefer a normal scale.who using criterion hasn't commented out the slowest algorithm on their machine so the visual scale "focuses" on the actual contenders?
:D Nice!
I totally agree that having an option to switch to log-scale would be awesome. I would not make it a default however.
Maybe we can have embed a switch directly into HTML page? So that users can change between normal and log scales without rerunning benchmarks? It should not be that difficult; I am happy to revive my webdev skills if we choose this avenue.
thats an even better idea! (on page switch/toggle). In that case for log scale probably should have it start a "picoseconds" so there some margin things start with ? idk
(also last i checked years ago) : I do think that the default html template has(?) some algorithm issues (either DOM related or because the raw data set is dumped into the page instead of just order statistics + mean + variance + min + max).
@Bodigrim you, as usual, make great points!
The ability to toggle between linear and log scales via a switch on the HTML page itself sounds like a promising way forward. This avoids one of the problems that a command-line–only switch would have, since it would allow toggling the scales on a per-chart basis.
I have a (horribly hacked together) proof of concept for switching between linear and log axes by clicking the x axis. Recent versions of flot support log axes nicely, but the version in js-flot on hackage and stackage doesn't, is 7 years old, and not very forwards compatible. I'd be happy to clean this up, but the issue is mostly figuring out what we want to depend on. What do?
Please upgrAde us to a better reality ! The universe will thank you.
Thanks for picking this up, @jonascarpay! The maintainer of the
js-flotlibrary is pretty responsive, so I imagine that if you submitted a PR to the upstreamjs-flotrepo, he'd be willing to incorporate it.The issue there is that it would cause a ton of breaking changes in 3 fairly high-profile packages, which I don't want to take responsibility for. Major CDNs are also still on 0.8.3 for some reason, so there's a good chance Neil will decide it's not worth it just for having log axes in criterion. I'll open an issue on js-flot, but I was mostly suggesting that we think about other options, like
- packaging a newer version of flot ourselves
- relying on a CDN
- moving to a more modern plotting library
- decide it's not worth it
Unfortunately none of these sound very good..
The issue there is that it would cause a ton of breaking changes
Ah, I didn't realize that the upgrade would be that significant! Although I know nothing about
flot, going from 0.8.3 to 4.2.1 does sound like a pretty sharp increase.I'll open an issue on js-flot
Thanks. I'll keep an eye on ndmitchell/js-flot#4.
I was mostly suggesting that we think about other options, like
- packaging a newer version of flot ourselves
- relying on a CDN
- moving to a more modern plotting library
- decide it's not worth it
Judging from the interest on this issue alone, I'd say that this is worth it! So I'm going to be bold and rule out the fourth option. As far as the other options go:
- packaging a newer version of flot ourselves
This is definitely within the realm of possibility. In fact,
criterionuses to bundleflotitself beforejs-flotcame along; see #72. As far as I can tell,criterion's dependency on thejs-flotlibrary is purely for the convenience of not having to bundle it itself, so there should be no loss in functionality by dropping our dependency on it. Moreover, ifjs-flotdoes decide to upgrade to 4.2.1 in the future, we could always reinstate the dependency.- relying on a CDN
I'm not sure I understand what this options entails.
- moving to a more modern plotting library
I'd also be open to this idea, although it's unclear to me how much more work this would require.
Reacted by Jonas CarpayStatus update; the bad news is that it turns out
flot4.2.1 is kind of a mess, or at the very least, does not fit nicely into what we're trying to do. Switching to it is very frustrating, and at this point, I don't recommend we do so (also see ndmitchell/js-flot#5).
The good news is that we (@considerate and I) tried outchart.js, which was much more pleasant, doesn't require JQuery and is smaller than flot 4.2.1. You can find the latest proof of concept here. It starts out with a log axis, and clicking a bar will give you a linear scale with the clicked bar as the limits, as an example of the kind of interactivity thatchart.jsmakes easy.So, the question is whether we want to commit to this rewrite. If you're OK with it, I'll open a new PR and we can move the discussion there, since the log axis is only a small part of it at this point. The main questions are how to package it, plus other feature requests for the reports in general.
If you don't think a major overhaul like this is worth it, that's obviously also completely fine.
- Do it! :) I’m not the maintainer of course. But the point stands, the charting piece of criterion is crusty and slow and wrong :)…On Thu, Nov 5, 2020 at 12:54 PM Jonas Carpay ***@***.***> wrote: Status update; the bad news is that it turns out flot 4.2.1 is kind of a mess, or at the very least, does not fit nicely into what we're trying to do. Switching to it is very frustrating, and at this point, I don't recommend we do so. The good news is that we ***@***.*** <https://github.com/considerate> and I) tried out chart.js, which was much more pleasant, doesn't require JQuery and is smaller than flot 4.2.1. You can find the latest proof of concept here <https://jsfiddle.net/dxcbfps5/1/>. It starts out with a log axis, and clicking a bar will give you a linear scale with the clicked bar as the limits, as an example of the kind of interactivity that chart.js makes easy. So, the question is whether we want to commit to this rewrite. If you're OK with it, I'll open a new PR and we can move the discussion there, since the log axis is only a small part of it at this point. The main questions are how to package it, plus other feature requests for the reports in general. If you don't think a major overhaul like this is worth it, that's obviously also completely fine. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#213 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAABBQX2AOGATFHRBBPRJSDSOLRGHANCNFSM4HOHEDQQ> .
As a maintainer, I'm tentatively in support of switching to a different charting library if it will make visualizing the data easier. I do think we'll need to make sure that we can replicate the existing functionality in the current
flot-based UI inchart.js(to whatever extend we can classify the functionality of the current UI), but I'm hopeful that that won't be too difficult.In short, please continue with your rewriting efforts! I'm eager to see what this will look like when the rewrite is closer to completion.
Hi!
I was wondering if there was such a feature in criterion, and if not how would I go about implementing it in the library?