Leibniz pi calculation test - OCaml ranking

Community
1

We don’t seem to be doing too well here :slight_smile:

I wonder why?

Cheers,

Eray

1 Like
2

If I understand right, the source code for OCaml should be this one here:

3

I wonder why you would think that OCaml is not doing well. There is a clear difference between programs that use vectorized instructions and programs that do not, and OCaml programs are in the second category. In this category, I think OCaml is doing very fine.
The only annoying part is that to reach this level, we have to write code in a very specific way, that does not feel very idiomatic for a functional language.
It might be interesting to see what OxCaml can do here: on one hand, Flambda2 should be able to reach the same level of performance with programs that look more functional, and on the other hand the support for vectorized operations should allow us to reach the same level of performance as C (although there is no automatic vectorization currently, and I’m not sure the rules of the contest allow hand-written vectorization).

3 Likes
4

Are there even rules? Clearly some examples have been vectorized by hand:

And some versions use heavily optimized variants of Leibniz formula:

This is also the case for the OCaml version. It does not implement an unadulterated version of Leibniz formula, but the reason eludes me.

5

See OCaml speed comparison - calculating pi with Leibniz - optimize? - #2 by copy

6

Slightly off-topic, but is there another good benchmark website where you can see how OCaml performs with a variety of computing problems? (I know it’s often difficult to compare languages, decide what’s idiomatic code, etc.)

7
8

With OCaml to C++ comparison here. I’ve seen the site in past when it was a bit easier to find a good overview.

Ideally, it would be nice to have some sort of “reasonable” weighing of different problems to find a suitable comparison. I’ve not found something like that yet (but I haven’t done a deep research either).

9

I guess I have already seen this benchmark.

But when the C proposition uses

#pragma omp for schedule(dynamic) ordered

or even

_mm_loadl_pd(...)

and other things like this, can it be considered idiomatic C ?

I guess, it would be more useful with an “strict ANSI C”.

When I choose a language, it doesn’t matter that it can be very fast if I spend hours of multithread or SSE optimizations I am not willing to do.

In some use cases, an application server paralelizes multiple query handlings, then it is interesting to have a benchmark with monothread proposition only. Which language can be scaled the most. Multithread propositions make it hard to know (or know by which dregree).

1 Like
10

I really would like to know how OCaml (roughly) compares to C. I can look at specific benchmarks, but they don’t tell me much. Sometimes it competes well, sometimes not. Sometimes it depends on whether crazy things are done in C, or whether we replace recursion with imperative control flow in OCaml.

I feel like the usual answer is, “you can’t compare programming languages”. But we do know that “generally” CPython is slower (w.r.t. execution time) than OCaml, and that OCaml is slower than C, don’t we?

The key task when comparing languages, I guess, is not to write specific benchmarks, but to invest the work in finding a scientific method to justify how to build a reasonable benchmark that covers multiple aspects of computational problems, i.e. which aspects of computation challenges should be tested in which ratio, and which rules should apply (and why) for each language with regard to “idiomatic” programming. Simply giving the answer, “we cannot compare languages” seems to be a bit lazy to me.

Is there some prior work on that which goes beyond the Computer Language Benchmarks Game website? I would be surprised if there wasn’t, but I didn’t find any.

Scientific quantification aside, what speed ratio, compared to C, should we roughly expect from OCaml?

I looked into the Koka programming language, which aims to get close to C using Perceus Reference Counting, and they aim to be approx twice as slow as C according to this graphic, though I’m not entirely sure how to read the illustration. Some benchmarks are here, including OCaml in the comparison (where OCaml is close to C, but I didn’t really look at the specific benchmarks used).

So what speed to expect from OCaml when compared to C? 2× to 5× slower? Shouldn’t we ask the question? Is there any scientific work on quantification of the speed that goes beyond putting together a couple of benchmarks?

11

I don’t know if that has changed since then but 21 years ago the stated goal by @xavierleroy was:

My performance goals have always been “never more than twice as slow as C”.

1 Like
12

Pethaps How 27 programming languages differ in energy consumption | App Developer Magazine is adequate. We don’t see a large gap between C and Rust or Ada. (Then no gcc-specific optimizations). But the text indicates it uses the same CLBG set of problems.

However, comparing OCaml could be interesting when separing float and other types since the float boxing can make OCaml less efficient.

1 Like
13

The one where TypeScript is ~7x slower than JavaScript?

14

Again, that attempt goes back to the Computer Language Benchmarks Game. The study in full seems to be available here. I didn’t fully read it, but it didn’t seem to do this:

Instead, (if I understand right), existing benchmarks were collected and used without deeper reasoning (and without any weighing) other than deciding which benchmarks are in and out, based on coverage regarding existing implementations/availability. If I understand right, those results were then averaged? (I’m not sure.) But that is no weighing (or reasoning).

As far as I see, they did describe how they combined multiple runs of the same problem, but not how they merged different problems into single “Normalized global results for Energy, Time, and Memory” in Figure 4.

The findings may still be of interest, but I don’t think they are suitable for a “global” (i.e. overall) comparison.

15

This is still reasonably accurate today, provided that “C” means “C without automatic or manual vectorization or parallelization”. Of course, for any program that performs lots of dynamic memory allocation, OCaml (and other functional languages optimized for this kind of code) will be significantly faster than C.

This said, programming languages and their implementations target specific application areas, so comparing the performance of languages that target widely different areas is rather meaningless. If all you care is tight numerical loops like this silly “Leibniz pi” calculation, just use FORTRAN: it’s a fine language for this code, and it will deliver the best FP performance you can get from your hardware, even better than C. If you care about symbolic computation, on the other hand, you’ll find OCaml to be hard to beat.

5 Likes
16

Why? We still could compare the performance for specific areas (or an “average” use case, or several specialized use cases). Whether we want to use those languages for those areas then, is yet another issue. But isn’t speed/performance one of the decision criteria used when selecting a language?

In other words:
Is measuring the speed really meaningless because we wouldn’t use those languages anyway? Or is it rather the opposite: Because of the low speed in practice, we don’t use some languages in those application areas? (Of course there are other criteria too.)

(Edit: Also, sometimes there are reasons to use a specific language even if it is not targeted for the given use-case. Also in those cases, it would be interesting to know how big the penalty is.)

I assume C and OCaml do target similar application areas, right?

So I guess it’s perfectly fine to say, “we would like OCaml to not be more than twice as slow as C”. Or, (hypothetically speaking, if floating-point calculation isn’t the main target of OCaml): “We would like OCaml to not be more than twice as slow as C, except for floating point arithmetic.”

Maybe it’s indeed impossible to find a single measure for speed, but we could at least try to carve out realistic profiles as a basis for scientific benchmarking other than “let’s calculate pi” or “let’s throw in these 10 benchmarks from the Computer Language Benchmarks Game and average the results”.

17

Umm, not really. C is supposed to be used for low-level, systems applications or services that cannot afford a garbage collector pause. OCaml is supposed to be used for general-purpose applications where a GC pause would not really be noticeable. Fortran is supposed to be used for calcaulation-intensive code.

Trying to find a scientific way to say ‘Language X should be at most N times slower than Language Y’ seems like an interesting problem but I think in practice you won’t really get anything useful out of it. If OCaml is supposed to be at most twice as slow as C, would you actually use it for applications where C should be used? I don’t think anyone would seriously do that. Like, what’s the use of writing a robotics control program in OCaml to make an autonomous vehicle that’s twice as slow as C at reacting to a child suddenly running out in front of the car?

You’re better off writing an OCaml program that outputs formally verified, fast C code–something OCaml is awesome at.

1 Like
18

I didn’t say C and OCaml target the same application areas but similar application areas. At least I see some overlap.

Maybe nobody would want to program a kernel module in OCaml (really?), but consider the following examples:

  • Parsers
  • Calculations (not necessarily as in numerically heavy, but with regard to complex formulas, possibly where precision is relevant, e.g. using zarith) including calculations on graphs or other complex data structures that go beyond arrays
  • Creation of domain-specific languages
  • Microservices
  • …

I feel like I can do these in C or in OCaml, with different advantages and disadvantages of course. Knowing a speed difference of typical application fields or a rough general estimation (that is based on scientific research rather than mere feelings or averaging abitrary benchmarks) would be nice.

Getting back to the subject of this thread: A Leibniz pi calculation test or the Computer Language Benchmarks Game, while it is fun, doesn’t give reliable quantified numbers w.r.t. practical problems. Does it mean it’s impossible to do better benchmarks that resemble problems in reality better? I don’t think it’s impossible. But it seems like we simply don’t have any as of yet.

19

I looked at that thread again and found the argument that floats are boxed at function boundaries (including recursive functions). So I tried a hybrid version with a recursive function that takes only an int and updates a float ref:

let calc2 rounds =
  let double = 2 * rounds in
  let upto = double + 1 in
  let sum = ref 0. in
  let[@inline always] rec loop curr =
    if curr >= upto
    then ()
    else
      let _ = sum := !sum +. 1. /. float_of_int curr in
      loop (curr + 4) in
  loop (1 - double); !sum

but to my surprise that ran at the same speed as the fully recursive function which takes the current sum as a float argument, which is about twice slower than the while look. Does anyone known why?

20

The sum reference is not local anymore (it is captured by the closure of loop), so it is not optimised away and has to hold boxed floats.
By the way, [@inline always] on a recursive function is not that useful (it just inlines the first few iterations). What you probably want is the Flambda2 [@loop] attribute, but it’s going to take some time before it lands upstream.