Using Core_kernel.List.mem. And why usingBase&Core instead of stdlib?

1/Thank you for your clear opinion about the features of JS Base/Core. Polymorphism is/was a real source of unpredictable issues (just have to see the message of the compiler when it fails!)
As @perry mentioned, the documentation web site for Core is awful, and it would be appreciated to be in phase with the ambition of JS team and their impressive work to improve OCaml “coding safety”.
/

2/ Regarding the simple case with List.mem, I used exactly your code and (within Emacs) got the same Error “Unbound module Core_kernel” (or already described type error).

I used the shell to compile with dune and it failed:

$ dune build test.exe
File “test.ml”, line 8, characters 5-16:
Error: Unbound module Core_kernel

Obviously, I forgot to mention core_kernel in packages. I admit here that mistake assuming it can help other people being aware of dune (formerly jbuilder) requirements.
I could compile correctly from the shell without any error or warning after fixing dune file as follows:

$ cat dune
(executable
(name test)
(libraries core_kernel))

So:
good source file (edited with nano, gedit or Emacs…) and good dune file => good compilation

BTW, it raises the subject of the process for safely editing OCaml code:
We must first define all required dependencies (on paper or in our mind), then we must realize that by editing the dune file AND use open directives in the source file. This is redundant and error-prone (things are changing and it’s possible to forget something).
How can we setup dune so it generate packages dependencies in dune file from open directives in source file (as unique specification of package dependencies)?

BTW2 regarding dune and merlin
When we are incited to edit .merlin file to tell it where to look for its prediction, in fact this .merlin file is now generated by dune as soon as we use it to compile source code (so any manual .merlin configuration file is erased and replaced by dune generated text).

Now coming back to Emacs, it’s hell again:
Emacs displays an exclamation mark besides open Core_kernel (or open Core).
When doing Tuareg/Evaluate Buffer, now it displays the correct answer with or without the open Core directive (even with new Ocaml
Tuareg/Evaluate Buffer:
Emacs_Screenshot_2018-09-24%2015-35-37
Tuareg/Evaluate Phrase:

 let m = List.mem [1; 2] 1 ~equal:Int.equal;;
val m : bool = true 

Formely it told the type error I’ve already described in my first post.

I even tried to use dune to modify .merlin to see if it has an influence. It hasn’t.

So the issue seems in fact to be Setting up a consistent and efficient OCaml dev environment (with Emacs) .
Because we are not all old/expert OCaml programmers used to Emacs-vi-tuareg-merlin-etc. in the dark!

What is your ~/.emacs file and emacs configuration (auto-complete, etc.) so you can edit Ocaml code, see it’s type, check if it’s compile, send it to the interpreter; compile it, debug it?
What are your other settings out of Emacs in addition to the dune file? (in ~/.ocamlinit or in other files)

PS1 :
I tried with Int.equal instead of (=). It works in this case with integers. This monomorphic vs. polymorphic discussion is not the issue.

PS2 :
Having a look at topfind messages (with RWO recommended ~/.ocamlinit), I saw just one error message regarding ocamltoplevel.cma:

/home/lm/.opam/4.06.1/lib/ocaml/compiler-libs/ocamltoplevel.cma: loaded
Exception:
Invalid_argument
“The ocamltoplevel.cma library from compiler-libs cannot be loaded inside the OCaml toplevel”.

Could that somehow explain the present issue?
How to fix that load issue?
Pls. notice that ocaml-top clearly display this Exception after the bunch of “load & added to search path” when ocaml toplevel doesn’t (within Emacs or not). It helped me to see it.

Thanks

The problem here is that there is no mapping between the name of the module and the findlib name. In the case of Core_kernel, Core, Async, Base there is, but how should dune know that if you’re referencing a module named Bcrypt that the library to be added is called safepass?

Also, one single library might introduce multiple modules into your compilation scope (this is a thing that dune tries to deprecate by making wrapped mode the default where all modules are inside the module that is named the same as the library).

I can’t help you with the Emacs problems, I am using Vim, where there is not much setup required.

Regarding the documentation of Core, I think it is alright for the most part, my main issue is discoverability, especially in Async, where I sometimes accidentally implement things that are already provided by Async.

GitHub - ocaml-opam/opam-user-setup: Simplify the configuration of editors for using OCaml tools should help in setting up the proper configuration for your editors. It supports emacs/vim. Once you install merlin,ocp-indent etc its as easy as opam user-setup install --editors=emacs

I’ll also add that i’ve heard good things about vscode with the reason-language-server.

The problem here is that there is no mapping between the name of the module and the findlib name. In the case of Core_kernel, Async, Base there is, but how should dune know that if you’re referencing a module named Bcrypt that the library to be added is called safepass?

This is a good question!
In the general situation, where can I (not dune) know which are the labels to use for using a module :
for the open directive, for topfind and for compilation?
This a beginner (and comebacker) important question.

Regarding using vim, I tried to setup it instead of Emacs. But I got lost and confused and tired with recommended installation instructions (vim, sublime). So I switched to this Emacs monster.
My first experience with vi(m) was being in front of a wall for quite a moment before I discovered magic keys such as Ins :w :q :q!
Then, this was silent efficiency.
So I would be very happy if you could share your vim setup instructions for OCaml.

My only wish today is to be able to focus on coding in OCaml!
Thanks

If you prefer emacs there is no reason to switch :slight_smile: Try using opam user-setup (https://github.com/OCamlPro/opam-user-setup) as I mentioned above.

Here is a recent explanation about how modules (OCaml) and packages (Opam) and libraries (findlib) fit (not so much) together: Connection between libraries in Opam, Dune, and Findlib

Not that I’m overly prideful of the result, but I’m curious what specific aspects of the Core website you’re sad about.

There are clear problems in the generated docs, which we very much want to fix, but in part await some coming improvements in the odoc toolchain. You might also want to check out Base:

Which is the part of Core that is most stable, and probably best suited for adoption in a wide variety of projects. Also, we’ve put some more attention on the quality of documentation there, though there’s more to go.

Regarding OCaml support in Vim: OCaml is supported by default except for Merlin, which offers a lot of IDE-like support and which I strongly suggest to install. I stay away from installing most Vim plugins and I only have 4 lines to configure OCaml in my .vimrc:

autocmd FileType ocaml set ts=2|set sw=2
let g:opamshare = substitute(system('opam config var share'),'\n$','','''')
execute "set rtp+=" . g:opamshare . "/merlin/vim"
execute "set rtp+=" . g:opamshare . "/ocp-indent/vim"

The purpose of the code above is to set up Vim’s runtime path such that Vim finds code that is installed by Opam packages.

2 Likes

perry told us his reasons and maybe will enter into more details and examples.

In my opinion, behind the nice web pages you mention, there is a bunch of stuff where we find deprecated modules or functions and many new functions. This is obviously there to guarantee rich and reliable new functions as well as for keeping a modular and evolutive system. And that kind of linear presentation may be just acceptable for OCaml stdlib but it seems to me not adapted for all the Base/Core_kernel/Core stuff.
I found no example (for the moment) about how to use it and especially in which context. So, this generated documentation is not sufficient (for me).
I have a little bit the same feeling as with the big OMG stuff (UML, Corba, etc.) and its various implementations in Java (with additional “free” NullPointerException hidden in many corners…).

Honestly, for the moment, I feel that I/we can find some useful tools within Base/Core_kernel/Core libs, but I really really fear that I/we’ll need to spend a lot of time to fully understand the functions and to make a choice within all these functions/modules and in order to make Real World programs (certainly much more time than for stdlib I’ve been studying progressively in details, in the context of different programs - and there are traps in stdlib because each feature/module has pros and cons and it must be chosen in fact with great care regarding performance for Real World programs).

How would you recommend me/us to fully apprehend Base/Core_kernel/Core and be able to mainly focus on making Ocaml programs?

Thanks.
I used user-setup but it did not fixed my problems.

$ opam user-setup install

We are currently looking for a solution with a more recent version of Emacs (I use 24.5.1).
A detailed manual about how using Emacs or vim for coding in OCaml would be very appreciated. In fact, there are many traps.

EDIT : FYI, upgrading to Emacs 26.1 solved immediately my problem with utop I didn’t mention here (with the same emacs init file ; utop was just working within shell)

Thanks for this URL.
This confirms that there is a real issue regarding the Ocaml Toolchain/dev environment. I suspected it without being able to explain it, and it’s much more important than I believed.

I notice the solution from @Leonidas:

The following is not intended to make you feel bad or to claim that you are bad. It’s really intended to explain what the experience is like for someone who doesn’t know what they’re doing — which is why I’m reading the docs, right?

I just did my usual experiment and tried to read the documentation for Hashtbl in Core.

First I went to:

Then I clicked through to API docs: Jane Street packages documentation moved.

which told me “the full API is browsable here”: https://ocaml.janestreet.com/ocaml-core/latest/doc/core/Core/index.html

I don’t see Hash there (note, doing it I made the mistake of looking for Hash instead of Hashtbl for a bit, but see below, this actually leads me to a slightly better result than otherwise), so I try clicking through to Core_kernel as suggested at the top: https://ocaml.janestreet.com/ocaml-core/latest/doc/core_kernel/Core_kernel/

Which immediately tells me “Deprecated [since 2018-03] Use Core_kernel directly instead” and who knows what this means, but never mind, I fearlessly scroll forward looking for Hash, and find module Hash = Core_kernel__.Import.Hash, which I click through on: https://ocaml.janestreet.com/ocaml-core/latest/doc/core_kernel/Core_kernel__/Import/#module-Hash

This brings me to the Module Core_kernel__.Import where there’s an entry for module Hash = Base.Hash which I click on: https://ocaml.janestreet.com/ocaml-core/latest/doc/base/Base/#module-Hash

Now I’m on the page for Module Base, and I realize, looking at it, that I really want Hashtbl but it is listed as module Hashtbl : sig ... end and I can click through to: https://ocaml.janestreet.com/ocaml-core/latest/doc/base/Base/Hashtbl/

Note that I’m now seven levels of clicks in.

But this is finally promising, I’m on the page with Module Base.Hashtbl. Sadly, as I scan down it, it’s a mess. Mostly it’s stuff like

val create : ?⁠growth_allowed:bool ‑> ?⁠size:int ‑> (module Base__.Hashtbl_intf.Key with type t = 'a) ‑> ('a, 'b) t

with no doc string afterwards. What’s a Base__.Hashtbl_intf.Key? I have no idea. What does growth_allowed do? What’s the size, is it the initial size? Who knows. I can guess, being an old timer I know a bit about hash tables. Maybe growth_allowed means that the table can expand the array and maybe size is the initial array, but I don’t want to guess, I want to know. Knowing is why you read the docs, they’re not there so you can learn about some options and start conducting experiments.

A large fraction of what I’ve hit after seven pages of clicks is type signatures with word-salad module names involved and no doc strings. I suppose I could read the source but I find the maze of twisty little module includes hard to navigate, and besides, the point of documentation is so I don’t have to read the code.

I could guess some of the time, but for some things, well, what’s a filter_mapi_inplace do? I’m sure someone smarter than me just looks at the type signature and knows, but I have no clue.

And yes, I understand, the documentation tools aren’t very friendly to Core’s rather intricate module inclusion structure, but as a user that really is cold comfort to me when I can’t find anything quickly if I can find it at all. And once I find stuff, often it’s just type signatures with no documentation strings at all.

Now, I could have taken a right turn at Albuquerque instead of a left turn if I hadn’t mistakenly clicked on Hash instead of Hashtbl a couple of times and ended up here two clicks earlier: https://ocaml.janestreet.com/ocaml-core/latest/doc/core_kernel/Core_kernel/Hashtbl/

…which would seem to be the more correct set of clicks, but having ended up here, at the docs for Module Core_kernel.Hashtbl, I find literally nothing but type signatures, so maybe the other page was better after all even though it was seven clicks down and not five. What’s the difference between the two? Why does one have some docs and the other has none? Which should I be reading? As a naive user, I have no clue, and I really am a naive user.

So I gave it a reasonable shot and I couldn’t find adequate documentation for the Hashtbl module’s create function, or much else, and it was a lot of effort to get there.

Where should I have clicked instead? I have no idea. I started on something that claimed to be the way to browse the full API. If there’s some other way to look at it, how would I know what it is without being told?

Note that this experience today is different from previous experiences trying to read the Core documentation only in that the page I started from is a bit different and the long but ultimately unrewarding series of clicks was a bit different. Maybe I’m not doing it right, but no one has made it particularly clear what right would actually be.

And so, even though I suspect Core would be a big help to me — from what I can tell the design is cleaner, more consistent, and much closer to my “only use optional and result, avoid exceptions” way of thinking, I don’t use it.

6 Likes

I don’t mean to pile on, but my experience has been very similar to what @perry describes. There are many modules with names like Core, Std, and others I can’t remember. It takes many clicks through nested modules to find any documentation of a particular function I might care about. By the time I find the function I’m not at all sure it’s even a function I’m allowed to call. Maybe it’s an internal function used to implement some other module that is the one I should be using. To this day I don’t understand the structure and naming of the modules of Core (though I haven’t looked in a while).

BTW, I don’t want to be mean here. Jane Street has done amazing things for the OCaml community, and from what I can tell, its tools are great. You guys are fabulous, and I don’t want to make it sound like I’m hating on you. However, the documentation for Core, Base, etc., do not seem to be in a usable state for me, and that’s been the primary reason I haven’t used them much. After doing exercises in Real World OCaml when I started learning the language, I immediately wanted to start using the language for real for my research work, and turned to the docs for Core, and hit a brick wall. The docs for the official stdlib, though not ideal, were a lot better, and so I tend to work with it and with Containers.

2 Likes

Good to hear the feedback (though in my own documentation-related request: fewer words! I think that could have been conveyed more clearly in 1/4 the space.)

Seems to me like the core of the issues you raise are the things we know, and that are related to the mismatch between the techniques used for constructing Base (and our libraries more generally) and the current state of the doc-generation tools. A sad but true fact is that the mli’s in our libraries are generally easier to read than the generated docs:


https://ocaml.janestreet.com/ocaml-core/latest/doc/base/Base/Float/

(Which is not to say that the docs themselves couldn’t use some more detail. They surely could. But it’s the double-underscores and extra indirections that are really deadly.)

It’s perhaps no surprise that within Jane Street, we read the mli’s far more often than we look at the generated docs. (And we have nice shortcuts in our tools for bringing those mli’s up.)

These are all things that we want to see fixed, and I’m hopeful that the work coming out of OCaml Labs on that front will solve the tooling issues, at which point we can do more to integrate the generated docs into our workflows, and improve the docs yet more.

y

In my own personal experience, I have found Jane Street’s documentation to be fabulously great, when one can find the documentation. However, my day to day experience often mirrors Perry’s experience; and as a working developer, I have found that typically the best way to find exactly the documentation that is relevant, I just have to look directly at .mli and *_intf.ml files. In some cases, when the documentation is not sufficient, I often end up needing to search for examples of a particular idiom. This involves checking out all of Jane Street’s publicly available open source code, and grepping through it until I have found an example which uses a particular feature idiomatically.

2 Likes

Probably, but I’ve said as much more compactly before and I don’t think it got as much attention.

That’s probably true. On the other hand, I suspect that pushing hard on the doc generating tools for a month or two would also fix the problem, or at least, substantially improve it.

(And we have nice shortcuts in our tools for bringing those mli’s up.)

These shortcuts you refer to, are they internal Jane Street tools which one would only be able to use at Jane Street?

An honest question, and I’m only asking because for a while, ppx_expect was publicized but not really easy to use until dune added the ability to run inline tests. As far as I am aware, expect testing was available internally through something provided by jenga(?) (some comments referred to an “inline_test_runner” script which was supposed to be available in the local directory, but I was never able to generate, and essentially had to reverse engineer what it was supposed to be doing). However, once dune added support for it, expect testing became awesome and super easy to use.

A wild thought, but could it be that some of those shortcuts and tools are only available internally? Merlin gets me pretty far most of the time, but I still invariable fall back to grep more often than I’d like to admit.

I think you’re mistaken. Long emails where short ones would do will not increase the set of people who will want to hear from you.

Far more effort than that has already gone in. The problems are unfortunately non-trivial, since it requires really understanding the module structure fairly well to generate the docs correctly. I don’t think we’re incredibly far from a solution, but I wouldn’t trivialize it.

Internal. The gap between Jane Street’s internal tools and what’s available publicly keeps on shriking (Dune, ocp-indent, ocamlformat, Merlin, all help.), but a lot of the editor customizations we have are only internal for the moment, mostly because they’re tied to purely internal infrastructure.

I don’t understand your last point, but we’re definitely converging as we move towards flipping from Jenga to Dune internally. I’m sure there will always be some gap, but hopefully it will get smaller and smaller over time. Dune in particular should make navigating through the tree with Merlin better, so if you check out and build your code and your dependencies with Dune, you should be able to navigate with Merlin to the file you need quite seamlessly.

I expect more of the tooling to make that practical to come out in the next 3-6 months.

y

5 Likes