Using Core_kernel.List.mem. And why usingBase&Core instead of stdlib?

[RogerT]

Hi!
I’m coming back to OCaml and I’m having some trouble using the Jane Street stuff (Base, Core_kernel, Core). It was already the same story when studying RWO 1 (open Core.Std instead of open Core, etc.)
This situation is recurrent and it badly interrupts the pleasure of OCaml coding.

1/ Using List.mem
I’m fine with using stdlib:
val List.mem : 'a → 'a list → bool).
See https://caml.inria.fr/pub/docs/manual-ocaml/libref/List.html#VALmem

And I’m fine too with the JS version, even if I don’t need it:
Core_kernel
type 'a t = 'a list`
val List.mem : 'a t ‑> 'a ‑> equal:('a ‑> 'a ‑> bool) ‑> bool
See https://ocaml.janestreet.com/ocaml-core/latest/doc/core_kernel/Core_kernel/List/

Let’s take a very simple example with List.mem:

$ cat test.ml
open Core_kernel (* *)
let m = Core_kernel.List.mem [1; 2; ] 1 ~equal:(=)

Here, we see that OCaml compiler in emacs is not happy with “open Core_kernel” which seems the cause of the non compilation:

Error: This function has type
'a Core_kernel.List.t → 'a → equal:('a → ‘a → bool) → bool
It is applied to too many arguments; maybe you forgot a `;’.

As the signature seems to be respected, I first see in emacs that the following directive is seen as an error:
! open Core_kernel (! is added by emacs in the margin)

It happens, whether I use Base or Core_kernel or Core (in the open directive and in the function ; who knows…): I get exactly the same error.
So what is the right way for using JS stdlib replacement for such a simple List.mem function? (I’ve lost many hours with that)
And why our “simple OCaml” is becoming progressively unusable?

2/ Usage: stdlib vs. JS Base/Core_kernel/Core
I roughly understand that JS Base&Core provide more functions (I don’t have to develop) and optimized implementations of functions compared to OCaml stdlib, which makes sense for “industrial strength” programming language.
But for what really motivated reasons should I use JS Base/Core instead of stdlib (and some well chosen and useful libraries)?
With stdlib I can simply and fastly code reliable programs in (O)Caml.
What do I really lose if I keep on coding with stdlib+libs instead of JS Base/Core? (or, what are my real benefits in using JS Base/Core stdlib replacement?)
Thanks

[perry]

@RogerT

But for what really motivated reasons should I use JS Base/Core instead of stdlib (and some well chosen and useful libraries)?

I started with the Jane Street libraries because I learned for my first month from RWO, and rapidly fell back to using other things. It’s a matter of taste, I think.

[RogerT]

@perry

I started with the Jane Street libraries because I learned for my first month from RWO, and rapidly fell back to using other things. It’s a matter of taste, I think.

Can you elaborate a little bit about what motivated you to “fall back using other things” (than JS libs), about those things that meet your needs, and about your context?

[perry]

Honestly, most of it was that the documentation web site for Core is awful and the manual for the standard library isn’t. I actually find some things in Core’s philosophy (like not throwing exceptions in the normal case but using option types instead) far more sympatico, but every time I wanted to look something up I hit a maze of twisty links leading after a bunch of effort to something that’s barely more than a list of type signatures.

[perry]

I’m also fond of Containers, fwiw.

[Leonidas]

@RogerT

open Core_kernel (* *)
let m = Core_kernel.List.mem [1; 2; ] 1 ~equal:(=)

I am not sure what the issue here is, because to get it to work you just have to write:

open Core_kernel
let m = List.mem [1; 2] 1 ~equal:Int.equal

If you open Core_kernel there is no need to refer to Core_kernel.List.map, because it is List.map now. Or you can not use the open. Opening modules without using them can trigger a warning and Dune uses warnings as errors, so maybe this is the issue you’re running into? There is a variant of open, open! which does not trigger this warning, but its use is a bit controversial, personally I rather avoid it.

Also I am using Int.equal instead of the polymorphic equality. That’s a nice thing about Core that it never assumes polymorphic equality. Most OCamlers argree that non-polymorphic equality is preferable.

In general I find Core/Base much closer to the way OCaml in 2018 (monomorphic compare, avoiding exceptions, result types) would be written than the Stdlib, so that’s why I use it.

[RogerT]

1/Thank you for your clear opinion about the features of JS Base/Core. Polymorphism is/was a real source of unpredictable issues (just have to see the message of the compiler when it fails!)
As @perry mentioned, the documentation web site for Core is awful, and it would be appreciated to be in phase with the ambition of JS team and their impressive work to improve OCaml “coding safety”.
/

2/ Regarding the simple case with List.mem, I used exactly your code and (within Emacs) got the same Error “Unbound module Core_kernel” (or already described type error).

I used the shell to compile with dune and it failed:

$ dune build test.exe
File “test.ml”, line 8, characters 5-16:
Error: Unbound module Core_kernel

Obviously, I forgot to mention core_kernel in packages. I admit here that mistake assuming it can help other people being aware of dune (formerly jbuilder) requirements.
I could compile correctly from the shell without any error or warning after fixing dune file as follows:

$ cat dune
(executable
(name test)
(libraries core_kernel))

So:
good source file (edited with nano, gedit or Emacs…) and good dune file => good compilation

BTW, it raises the subject of the process for safely editing OCaml code:
We must first define all required dependencies (on paper or in our mind), then we must realize that by editing the dune file AND use open directives in the source file. This is redundant and error-prone (things are changing and it’s possible to forget something).
How can we setup dune so it generate packages dependencies in dune file from open directives in source file (as unique specification of package dependencies)?

BTW2 regarding dune and merlin
When we are incited to edit .merlin file to tell it where to look for its prediction, in fact this .merlin file is now generated by dune as soon as we use it to compile source code (so any manual .merlin configuration file is erased and replaced by dune generated text).

Now coming back to Emacs, it’s hell again:
Emacs displays an exclamation mark besides open Core_kernel (or open Core).
When doing Tuareg/Evaluate Buffer, now it displays the correct answer with or without the open Core directive (even with new Ocaml
Tuareg/Evaluate Buffer:

Tuareg/Evaluate Phrase:

 let m = List.mem [1; 2] 1 ~equal:Int.equal;;
val m : bool = true 

Formely it told the type error I’ve already described in my first post.

I even tried to use dune to modify .merlin to see if it has an influence. It hasn’t.

So the issue seems in fact to be Setting up a consistent and efficient OCaml dev environment (with Emacs) .
Because we are not all old/expert OCaml programmers used to Emacs-vi-tuareg-merlin-etc. in the dark!

What is your ~/.emacs file and emacs configuration (auto-complete, etc.) so you can edit Ocaml code, see it’s type, check if it’s compile, send it to the interpreter; compile it, debug it?
What are your other settings out of Emacs in addition to the dune file? (in ~/.ocamlinit or in other files)

PS1 :
I tried with Int.equal instead of (=). It works in this case with integers. This monomorphic vs. polymorphic discussion is not the issue.

PS2 :
Having a look at topfind messages (with RWO recommended ~/.ocamlinit), I saw just one error message regarding ocamltoplevel.cma:

/home/lm/.opam/4.06.1/lib/ocaml/compiler-libs/ocamltoplevel.cma: loaded
Exception:
Invalid_argument
“The ocamltoplevel.cma library from compiler-libs cannot be loaded inside the OCaml toplevel”.

Could that somehow explain the present issue?
How to fix that load issue?
Pls. notice that ocaml-top clearly display this Exception after the bunch of “load & added to search path” when ocaml toplevel doesn’t (within Emacs or not). It helped me to see it.

Thanks

[Leonidas]

@RogerT

We must first define all required dependencies (on paper or in our mind), then we must realize that by editing the dune file AND use open directives in the source file. This is redundant and error-prone (things are changing and it’s possible to forget something).
How can we setup dune so it generate packages dependencies in dune file from open directives in source file (as unique specification of package dependencies)?

The problem here is that there is no mapping between the name of the module and the findlib name. In the case of Core_kernel, Core, Async, Base there is, but how should dune know that if you’re referencing a module named Bcrypt that the library to be added is called safepass?

Also, one single library might introduce multiple modules into your compilation scope (this is a thing that dune tries to deprecate by making wrapped mode the default where all modules are inside the module that is named the same as the library).

I can’t help you with the Emacs problems, I am using Vim, where there is not much setup required.

Regarding the documentation of Core, I think it is alright for the most part, my main issue is discoverability, especially in Async, where I sometimes accidentally implement things that are already provided by Async.

[anuragsoni]

@RogerT

So the issue seems in fact to be Setting up a consistent and efficient OCaml dev environment (with Emacs) .
Because we are not all old/expert OCaml programmers used to Emacs-vi-tuareg-merlin-etc. in the dark!

What is your ~/.emacs file and emacs configuration (auto-complete, etc.) so you can edit Ocaml code, see it’s type, check if it’s compile, send it to the interpreter; compile it, debug it?
What are your other settings out of Emacs in addition to the dune file? (in ~/.ocamlinit or in other files)

GitHub - ocaml-opam/opam-user-setup: Simplify the configuration of editors for using OCaml tools should help in setting up the proper configuration for your editors. It supports emacs/vim. Once you install merlin,ocp-indent etc its as easy as opam user-setup install --editors=emacs

I’ll also add that i’ve heard good things about vscode with the reason-language-server.

[RogerT]

The problem here is that there is no mapping between the name of the module and the findlib name. In the case of Core_kernel, Async, Base there is, but how should dune know that if you’re referencing a module named Bcrypt that the library to be added is called safepass?

This is a good question!
In the general situation, where can I (not dune) know which are the labels to use for using a module :
for the open directive, for topfind and for compilation?
This a beginner (and comebacker) important question.

Regarding using vim, I tried to setup it instead of Emacs. But I got lost and confused and tired with recommended installation instructions (vim, sublime). So I switched to this Emacs monster.
My first experience with vi(m) was being in front of a wall for quite a moment before I discovered magic keys such as Ins :w :q :q!
Then, this was silent efficiency.
So I would be very happy if you could share your vim setup instructions for OCaml.

My only wish today is to be able to focus on coding in OCaml!
Thanks

[anuragsoni]

If you prefer emacs there is no reason to switch Try using opam user-setup (https://github.com/OCamlPro/opam-user-setup) as I mentioned above.

[lindig]

Here is a recent explanation about how modules (OCaml) and packages (Opam) and libraries (findlib) fit (not so much) together: Connection between libraries in Opam, Dune, and Findlib

[Yaron_Minsky]

@RogerT

As @perry mentioned, the documentation web site for Core is awful, and it would be appreciated to be in phase with the ambition of JS team and their impressive work to improve OCaml “coding safety”.

Not that I’m overly prideful of the result, but I’m curious what specific aspects of the Core website you’re sad about.

core

There are clear problems in the generated docs, which we very much want to fix, but in part await some coming improvements in the odoc toolchain. You might also want to check out Base:

base

Which is the part of Core that is most stable, and probably best suited for adoption in a wide variety of projects. Also, we’ve put some more attention on the quality of documentation there, though there’s more to go.

[lindig]

Regarding OCaml support in Vim: OCaml is supported by default except for Merlin, which offers a lot of IDE-like support and which I strongly suggest to install. I stay away from installing most Vim plugins and I only have 4 lines to configure OCaml in my .vimrc:

autocmd FileType ocaml set ts=2|set sw=2
let g:opamshare = substitute(system('opam config var share'),'\n$','','''')
execute "set rtp+=" . g:opamshare . "/merlin/vim"
execute "set rtp+=" . g:opamshare . "/ocp-indent/vim"

The purpose of the code above is to set up Vim’s runtime path such that Vim finds code that is installed by Opam packages.

[RogerT]

perry told us his reasons and maybe will enter into more details and examples.

In my opinion, behind the nice web pages you mention, there is a bunch of stuff where we find deprecated modules or functions and many new functions. This is obviously there to guarantee rich and reliable new functions as well as for keeping a modular and evolutive system. And that kind of linear presentation may be just acceptable for OCaml stdlib but it seems to me not adapted for all the Base/Core_kernel/Core stuff.
I found no example (for the moment) about how to use it and especially in which context. So, this generated documentation is not sufficient (for me).
I have a little bit the same feeling as with the big OMG stuff (UML, Corba, etc.) and its various implementations in Java (with additional “free” NullPointerException hidden in many corners…).

Honestly, for the moment, I feel that I/we can find some useful tools within Base/Core_kernel/Core libs, but I really really fear that I/we’ll need to spend a lot of time to fully understand the functions and to make a choice within all these functions/modules and in order to make Real World programs (certainly much more time than for stdlib I’ve been studying progressively in details, in the context of different programs - and there are traps in stdlib because each feature/module has pros and cons and it must be chosen in fact with great care regarding performance for Real World programs).

How would you recommend me/us to fully apprehend Base/Core_kernel/Core and be able to mainly focus on making Ocaml programs?

[RogerT]

Thanks.
I used user-setup but it did not fixed my problems.

$ opam user-setup install

We are currently looking for a solution with a more recent version of Emacs (I use 24.5.1).
A detailed manual about how using Emacs or vim for coding in OCaml would be very appreciated. In fact, there are many traps.

EDIT : FYI, upgrading to Emacs 26.1 solved immediately my problem with utop I didn’t mention here (with the same emacs init file ; utop was just working within shell)

[RogerT]

Thanks for this URL.
This confirms that there is a real issue regarding the Ocaml Toolchain/dev environment. I suspected it without being able to explain it, and it’s much more important than I believed.

I notice the solution from @Leonidas:

@Leonidas

This is true. As a stopgap solution I recommend using clean switches per project and only install packages via dependencies declared in the opam file and avoiding to call opam install … directly.

[perry]

@Yaron_Minsky

Not that I’m overly prideful of the result, but I’m curious what specific aspects of the Core website you’re sad about.

The following is not intended to make you feel bad or to claim that you are bad. It’s really intended to explain what the experience is like for someone who doesn’t know what they’re doing — which is why I’m reading the docs, right?

I just did my usual experiment and tried to read the documentation for Hashtbl in Core.

First I went to:

core

Then I clicked through to API docs: Jane Street packages documentation moved.

which told me “the full API is browsable here”: https://ocaml.janestreet.com/ocaml-core/latest/doc/core/Core/index.html

I don’t see Hash there (note, doing it I made the mistake of looking for Hash instead of Hashtbl for a bit, but see below, this actually leads me to a slightly better result than otherwise), so I try clicking through to Core_kernel as suggested at the top: https://ocaml.janestreet.com/ocaml-core/latest/doc/core_kernel/Core_kernel/

Which immediately tells me “Deprecated [since 2018-03] Use Core_kernel directly instead” and who knows what this means, but never mind, I fearlessly scroll forward looking for Hash, and find module Hash = Core_kernel__.Import.Hash, which I click through on: https://ocaml.janestreet.com/ocaml-core/latest/doc/core_kernel/Core_kernel__/Import/#module-Hash

This brings me to the Module Core_kernel__.Import where there’s an entry for module Hash = Base.Hash which I click on: https://ocaml.janestreet.com/ocaml-core/latest/doc/base/Base/#module-Hash

Now I’m on the page for Module Base, and I realize, looking at it, that I really want Hashtbl but it is listed as module Hashtbl : sig ... end and I can click through to: https://ocaml.janestreet.com/ocaml-core/latest/doc/base/Base/Hashtbl/

Note that I’m now seven levels of clicks in.

But this is finally promising, I’m on the page with Module Base.Hashtbl. Sadly, as I scan down it, it’s a mess. Mostly it’s stuff like

val create : ?⁠growth_allowed:bool ‑> ?⁠size:int ‑> (module Base__.Hashtbl_intf.Key with type t = 'a) ‑> ('a, 'b) t

with no doc string afterwards. What’s a Base__.Hashtbl_intf.Key? I have no idea. What does growth_allowed do? What’s the size, is it the initial size? Who knows. I can guess, being an old timer I know a bit about hash tables. Maybe growth_allowed means that the table can expand the array and maybe size is the initial array, but I don’t want to guess, I want to know. Knowing is why you read the docs, they’re not there so you can learn about some options and start conducting experiments.

A large fraction of what I’ve hit after seven pages of clicks is type signatures with word-salad module names involved and no doc strings. I suppose I could read the source but I find the maze of twisty little module includes hard to navigate, and besides, the point of documentation is so I don’t have to read the code.

I could guess some of the time, but for some things, well, what’s a filter_mapi_inplace do? I’m sure someone smarter than me just looks at the type signature and knows, but I have no clue.

And yes, I understand, the documentation tools aren’t very friendly to Core’s rather intricate module inclusion structure, but as a user that really is cold comfort to me when I can’t find anything quickly if I can find it at all. And once I find stuff, often it’s just type signatures with no documentation strings at all.

Now, I could have taken a right turn at Albuquerque instead of a left turn if I hadn’t mistakenly clicked on Hash instead of Hashtbl a couple of times and ended up here two clicks earlier: https://ocaml.janestreet.com/ocaml-core/latest/doc/core_kernel/Core_kernel/Hashtbl/

…which would seem to be the more correct set of clicks, but having ended up here, at the docs for Module Core_kernel.Hashtbl, I find literally nothing but type signatures, so maybe the other page was better after all even though it was seven clicks down and not five. What’s the difference between the two? Why does one have some docs and the other has none? Which should I be reading? As a naive user, I have no clue, and I really am a naive user.

So I gave it a reasonable shot and I couldn’t find adequate documentation for the Hashtbl module’s create function, or much else, and it was a lot of effort to get there.

Where should I have clicked instead? I have no idea. I started on something that claimed to be the way to browse the full API. If there’s some other way to look at it, how would I know what it is without being told?

Note that this experience today is different from previous experiences trying to read the Core documentation only in that the page I started from is a bit different and the long but ultimately unrewarding series of clicks was a bit different. Maybe I’m not doing it right, but no one has made it particularly clear what right would actually be.

And so, even though I suspect Core would be a big help to me — from what I can tell the design is cleaner, more consistent, and much closer to my “only use optional and result, avoid exceptions” way of thinking, I don’t use it.

[jeffsco]

I don’t mean to pile on, but my experience has been very similar to what @perry describes. There are many modules with names like Core, Std, and others I can’t remember. It takes many clicks through nested modules to find any documentation of a particular function I might care about. By the time I find the function I’m not at all sure it’s even a function I’m allowed to call. Maybe it’s an internal function used to implement some other module that is the one I should be using. To this day I don’t understand the structure and naming of the modules of Core (though I haven’t looked in a while).

[perry]

BTW, I don’t want to be mean here. Jane Street has done amazing things for the OCaml community, and from what I can tell, its tools are great. You guys are fabulous, and I don’t want to make it sound like I’m hating on you. However, the documentation for Core, Base, etc., do not seem to be in a usable state for me, and that’s been the primary reason I haven’t used them much. After doing exercises in Real World OCaml when I started learning the language, I immediately wanted to start using the language for real for my research work, and turned to the docs for Core, and hit a brick wall. The docs for the official stdlib, though not ideal, were a lot better, and so I tend to work with it and with Containers.