Is there a nice library where I can time a piece of code and then create a regression test with it that alerts if that piece of code gets slower (or faster)?
Would be an amazing thing to hook into a CI test.
1 Like
I don’t know about this kind of library but it would be great to have one.
Do you have any examples of this kind of libraries in other languages/infrastructures?
core_bench might have what you’re looking for.
Hmm, actually, some cool guy told me once that the reason Chrome is so fast is because they have regression tests like this that prevent you from checking code in that makes stuff slower.
Never found confirmation though but it certainly sounds plausible. Uh, now that I’m saying this out loud I realize it wouldn’t actually be hard to look up since Chrome is an open source project?
EDIT: might have been talking about this thing they call Perf Trybots which benchmarks both the base of your patch and tip against a collection of devices and tells you what the performance impact of the patch is. Not quite what I was talking about but perhaps a more sensible approach.
1 Like
You may also be interested by lucene’s approach. I like the fact that label the graph to explain some of the changes.
1 Like
I don’t think anyone answered @mbacarella’s question here.
Does anyone have references (libraries, papers, etc. in whatever language) on how to perform (reasonably) machine independent automated performance regression testing ?
For timings, I guess a naive idea is to establish some form of baseline timing (Say List.map Fun.id data) l and then compare runs of your computation (say List.map f data) to this baseline in order to get a relative number which you store and compare to.
In my humble opinion, that should be the job of the CI tool to time things and detect regressions.
That’s just shifting the problem :–) (That being said I don’t mind having one number per platform/architecture).