In OCaml, external is used in implementations to declare that a function is implemented by a foreign function call (presumably to C but sometimes Javascript or Rust).
The same external keyword can be used in interface. According to the documentation:
External functions thus defined can be specified in interface files or sig…end signatures either as regular values
val name : type
thus hiding their implementation as C functions, or explicitly as “manifest” external functions
external name : type = C-function-name
The latter is slightly more efficient, as it allows clients of the module to call directly the C function instead of going through the corresponding OCaml function. On the other hand, it should not be used in library modules if they have side-effects at toplevel, as this direct call interferes with the linker’s algorithm for removing unused modules from libraries at link-time.
So far, so good.
It turns out that there is a little bit more to that. E.g.,
# module B : sig
val ( && ) : bool -> bool -> bool
end = struct
let ( && ) = Stdlib.( && )
end ;;
module B : sig val ( && ) : bool -> bool -> bool end
# ( && ) (print_endline "one"; false) (print_endline "two"; true) ;;
one
- : bool = false
# B.( && ) (print_endline "one"; false) (print_endline "two"; true) ;;
two
one
- : bool = false
So external can, in some cases, influence the evaluation strategies of function parameters. This lead me to the following questions:
- Are there cases other than
( && ) and ( || ) where external-vs-val influences the semantic of the language?
- Are there cases where
external-vs-val influences the compilation process? (Wild guess: the "%identity" external is always inlined?)
- How much more efficient is the
external version when it calls a real C function vs a val that hides an implementation external?
- Are the
external for ( || ) and ( && ) actual C externals or are they builtin compiler primitives?
- Are there other gotchas that are worth learning about?
- Should this be mentioned in the doc? (Probably in this section; currently it only points at C externals which the manual excerpt above is from.) I’m willing to contribute some of this doc, by summarising whatever comes from this here discussion.
- And now from the fun-and-profit part of the post: Is it possible to wrangle that whole special casing in order to provide lazy option operators:
( || ) : 'a option -> 'a option -> 'a option?
2 Likes
It might be worth looking at the bytecode (and maybe compiling with opt and looking at the assembler): I’m going to guess that the reason you’re seeing a difference is that in the first case, the compiler recognizes (&&) as the logical conditional-and operator, where in the second it is a function.
I don’t remember what the documentation says about it, but basically there are two kinds of primitives:
- The ones that the compiler understands, which (almost) always start with a
% character.
- The ones that the compiler doesn’t know about, which resolve to a C-like call to an external function (or a JS function in the case of js_of_ocaml)
The % primitives can do funny things with your code, and in particular the (||) and (&&) operators are not regular calls with the call-by-value convention but instead shortcuts for if-then-else constructs.
If you export them as regular values, then a stub function is generated and this stub is called with the regular calling convention.
8 Likes
I hadn’t noticed the % marker. This will help me narrow down my search. Thank you.
I had a look and here’s a quick summary of my finds:
Some constructors are just optimised away:
-
"%identity" and "%opaque" are not just inlined, they are simply erased. That is external f : 'a -> 'a = “%identity” ;; f somethingis compiled to justsomething`.
-
"%apply" and "%revapply" are inlined. That is x |> f/f @@ x is compiled to just f x.
-
"%boolnot" is optimised when used inside a conditional. That is if not b then foo else bar is compiled to if b then bar else foo.
And then those are semantically different than when exposed as a val.
-
"%sequand" is both
- evaluated lazily (roughly,
e1 && e2 compiles to if e1 then if e2 then true else false else false; it’s not strictly like that, there’s a notion of labels and continuation to represent the control flow), and
- optimised when used in a conditional (roughly,
if e1 && e2 then foo else bar compiles to if not e1 then bar else if e2 then foo else bar; except again it’s not strictly like that and there’s sharing via labels)
-
"%sequor" is the same but flipped on its head.
And the rest seems to be mostly:
- Small primitives which can be optimised in some cases (e.g., arithmetic can be optimised when literals are involved, e.g., polymorphic comparison can be optimised when the argument type is known).
- Ways to interact with the runtime representation of values (e.g.,
"%string_safe_get") which compiles down to runtime functions (caml_string_get). I’m not sure why those are not just declared as externals to the eventual runtime function. A guess would be that the byte-code and native-code backends treat them widely differently? Another guess is historical baggage?
I did not spot any other external that seem to have special semantic meaning that is erased when exposing them as val. Have I missed any?
Note: these are inferred from reading bytecomp/bytegen.ml. Maybe there are differences in the native code generator. However, because I’m more interested in the semantics of external (and less in the performance) I think it’s ok.
2 Likes
I think you should consider that %-primitives are not function calls, but AST node constructors.
It’s rather obvious with the %sequand and %sequor primitives, but there are other good examples:
-
|>, bound to %revapply: x |> f is not “inlined”, it is syntactic sugar for f x.
- The difference between
%identity and %opaque is the place in the backend where they’re simplified. The %identity primitive is simplified early, to allow optimisations to see through it, while %opaque is simplified much later, so it acts as a barrier to optimisation.
For bytecode (and in a few cases in the native backend too) the compiler can choose to compile the constructions associated to these primitives to C calls when it’s simpler, but you shouldn’t assume that there is a C function associated to each of those primitives.
2 Likes
To give a bit more context, I was not wondering how to use externals, but how to pass them through to a specialised environment.
In the Tezos project, the economic protocol of the blockchain can be upgraded via an on-chain operation. It’s kind of an higher-order thing where the function that does state transitions is part of the state. In order to guarantee some properties about the economic protocol (doesn’t read files from the machine it’s running on, doesn’t create randomness, these kinds of things) we limit the access that the protocol code has of the standard library. This is achieved by compiling the protocol code against the environment built out of these mlis.
Recently, an external dependency (I think it was zarith but maybe it was another one) released a new version where one of its external is now exposed as a val. Because you can expose an external as a val but not the other way around, we cannot use the newer version of the dependency through the environment. (There’s a newer environment to account for these updates and to add new features.)
Still, to avoid having to make new environments too often, we considered changing all the externals into vals. Some forward-compatibility! But this is unsound because of the few externals that are not actual externals but actually internals of sorts. In addition to being unsound, there are some efficiency considerations that I haven’t even touched upon in this thread.
Anyway, I needed a deeper understanding of the external-vs-val and the special casing or internals. I’ll leave this thread as is for searchability. I may also make a PR against ocaml trunk to mention the %-externals in the doc. Thanks for the pointers @vlaviron
2 Likes