Using Core_kernel.List.mem. And why usingBase&Core instead of stdlib?

Not that I’m overly prideful of the result, but I’m curious what specific aspects of the Core website you’re sad about.

There are clear problems in the generated docs, which we very much want to fix, but in part await some coming improvements in the odoc toolchain. You might also want to check out Base:

Which is the part of Core that is most stable, and probably best suited for adoption in a wide variety of projects. Also, we’ve put some more attention on the quality of documentation there, though there’s more to go.

Regarding OCaml support in Vim: OCaml is supported by default except for Merlin, which offers a lot of IDE-like support and which I strongly suggest to install. I stay away from installing most Vim plugins and I only have 4 lines to configure OCaml in my .vimrc:

autocmd FileType ocaml set ts=2|set sw=2
let g:opamshare = substitute(system('opam config var share'),'\n$','','''')
execute "set rtp+=" . g:opamshare . "/merlin/vim"
execute "set rtp+=" . g:opamshare . "/ocp-indent/vim"

The purpose of the code above is to set up Vim’s runtime path such that Vim finds code that is installed by Opam packages.

2 Likes

perry told us his reasons and maybe will enter into more details and examples.

In my opinion, behind the nice web pages you mention, there is a bunch of stuff where we find deprecated modules or functions and many new functions. This is obviously there to guarantee rich and reliable new functions as well as for keeping a modular and evolutive system. And that kind of linear presentation may be just acceptable for OCaml stdlib but it seems to me not adapted for all the Base/Core_kernel/Core stuff.
I found no example (for the moment) about how to use it and especially in which context. So, this generated documentation is not sufficient (for me).
I have a little bit the same feeling as with the big OMG stuff (UML, Corba, etc.) and its various implementations in Java (with additional “free” NullPointerException hidden in many corners…).

Honestly, for the moment, I feel that I/we can find some useful tools within Base/Core_kernel/Core libs, but I really really fear that I/we’ll need to spend a lot of time to fully understand the functions and to make a choice within all these functions/modules and in order to make Real World programs (certainly much more time than for stdlib I’ve been studying progressively in details, in the context of different programs - and there are traps in stdlib because each feature/module has pros and cons and it must be chosen in fact with great care regarding performance for Real World programs).

How would you recommend me/us to fully apprehend Base/Core_kernel/Core and be able to mainly focus on making Ocaml programs?

Thanks.
I used user-setup but it did not fixed my problems.

$ opam user-setup install

We are currently looking for a solution with a more recent version of Emacs (I use 24.5.1).
A detailed manual about how using Emacs or vim for coding in OCaml would be very appreciated. In fact, there are many traps.

EDIT : FYI, upgrading to Emacs 26.1 solved immediately my problem with utop I didn’t mention here (with the same emacs init file ; utop was just working within shell)

Thanks for this URL.
This confirms that there is a real issue regarding the Ocaml Toolchain/dev environment. I suspected it without being able to explain it, and it’s much more important than I believed.

I notice the solution from @Leonidas:

The following is not intended to make you feel bad or to claim that you are bad. It’s really intended to explain what the experience is like for someone who doesn’t know what they’re doing — which is why I’m reading the docs, right?

I just did my usual experiment and tried to read the documentation for Hashtbl in Core.

First I went to:

Then I clicked through to API docs: https://ocaml.janestreet.com/ocaml-core/latest/doc/core/index.html

which told me “the full API is browsable here”: https://ocaml.janestreet.com/ocaml-core/latest/doc/core/Core/index.html

I don’t see Hash there (note, doing it I made the mistake of looking for Hash instead of Hashtbl for a bit, but see below, this actually leads me to a slightly better result than otherwise), so I try clicking through to Core_kernel as suggested at the top: https://ocaml.janestreet.com/ocaml-core/latest/doc/core_kernel/Core_kernel/

Which immediately tells me “Deprecated [since 2018-03] Use Core_kernel directly instead” and who knows what this means, but never mind, I fearlessly scroll forward looking for Hash, and find module Hash = Core_kernel__.Import.Hash, which I click through on: https://ocaml.janestreet.com/ocaml-core/latest/doc/core_kernel/Core_kernel__/Import/#module-Hash

This brings me to the Module Core_kernel__.Import where there’s an entry for module Hash = Base.Hash which I click on: https://ocaml.janestreet.com/ocaml-core/latest/doc/base/Base/#module-Hash

Now I’m on the page for Module Base, and I realize, looking at it, that I really want Hashtbl but it is listed as module Hashtbl : sig ... end and I can click through to: https://ocaml.janestreet.com/ocaml-core/latest/doc/base/Base/Hashtbl/

Note that I’m now seven levels of clicks in.

But this is finally promising, I’m on the page with Module Base.Hashtbl. Sadly, as I scan down it, it’s a mess. Mostly it’s stuff like

val create : ?⁠growth_allowed:bool ‑> ?⁠size:int ‑> (module Base__.Hashtbl_intf.Key with type t = 'a) ‑> ('a, 'b) t

with no doc string afterwards. What’s a Base__.Hashtbl_intf.Key? I have no idea. What does growth_allowed do? What’s the size, is it the initial size? Who knows. I can guess, being an old timer I know a bit about hash tables. Maybe growth_allowed means that the table can expand the array and maybe size is the initial array, but I don’t want to guess, I want to know. Knowing is why you read the docs, they’re not there so you can learn about some options and start conducting experiments.

A large fraction of what I’ve hit after seven pages of clicks is type signatures with word-salad module names involved and no doc strings. I suppose I could read the source but I find the maze of twisty little module includes hard to navigate, and besides, the point of documentation is so I don’t have to read the code.

I could guess some of the time, but for some things, well, what’s a filter_mapi_inplace do? I’m sure someone smarter than me just looks at the type signature and knows, but I have no clue.

And yes, I understand, the documentation tools aren’t very friendly to Core’s rather intricate module inclusion structure, but as a user that really is cold comfort to me when I can’t find anything quickly if I can find it at all. And once I find stuff, often it’s just type signatures with no documentation strings at all.

Now, I could have taken a right turn at Albuquerque instead of a left turn if I hadn’t mistakenly clicked on Hash instead of Hashtbl a couple of times and ended up here two clicks earlier: https://ocaml.janestreet.com/ocaml-core/latest/doc/core_kernel/Core_kernel/Hashtbl/

…which would seem to be the more correct set of clicks, but having ended up here, at the docs for Module Core_kernel.Hashtbl, I find literally nothing but type signatures, so maybe the other page was better after all even though it was seven clicks down and not five. What’s the difference between the two? Why does one have some docs and the other has none? Which should I be reading? As a naive user, I have no clue, and I really am a naive user.

So I gave it a reasonable shot and I couldn’t find adequate documentation for the Hashtbl module’s create function, or much else, and it was a lot of effort to get there.

Where should I have clicked instead? I have no idea. I started on something that claimed to be the way to browse the full API. If there’s some other way to look at it, how would I know what it is without being told?

Note that this experience today is different from previous experiences trying to read the Core documentation only in that the page I started from is a bit different and the long but ultimately unrewarding series of clicks was a bit different. Maybe I’m not doing it right, but no one has made it particularly clear what right would actually be.

And so, even though I suspect Core would be a big help to me — from what I can tell the design is cleaner, more consistent, and much closer to my “only use optional and result, avoid exceptions” way of thinking, I don’t use it.

6 Likes

I don’t mean to pile on, but my experience has been very similar to what @perry describes. There are many modules with names like Core, Std, and others I can’t remember. It takes many clicks through nested modules to find any documentation of a particular function I might care about. By the time I find the function I’m not at all sure it’s even a function I’m allowed to call. Maybe it’s an internal function used to implement some other module that is the one I should be using. To this day I don’t understand the structure and naming of the modules of Core (though I haven’t looked in a while).

BTW, I don’t want to be mean here. Jane Street has done amazing things for the OCaml community, and from what I can tell, its tools are great. You guys are fabulous, and I don’t want to make it sound like I’m hating on you. However, the documentation for Core, Base, etc., do not seem to be in a usable state for me, and that’s been the primary reason I haven’t used them much. After doing exercises in Real World OCaml when I started learning the language, I immediately wanted to start using the language for real for my research work, and turned to the docs for Core, and hit a brick wall. The docs for the official stdlib, though not ideal, were a lot better, and so I tend to work with it and with Containers.

2 Likes

Good to hear the feedback (though in my own documentation-related request: fewer words! I think that could have been conveyed more clearly in 1/4 the space.)

Seems to me like the core of the issues you raise are the things we know, and that are related to the mismatch between the techniques used for constructing Base (and our libraries more generally) and the current state of the doc-generation tools. A sad but true fact is that the mli’s in our libraries are generally easier to read than the generated docs:


https://ocaml.janestreet.com/ocaml-core/latest/doc/base/Base/Float/

(Which is not to say that the docs themselves couldn’t use some more detail. They surely could. But it’s the double-underscores and extra indirections that are really deadly.)

It’s perhaps no surprise that within Jane Street, we read the mli’s far more often than we look at the generated docs. (And we have nice shortcuts in our tools for bringing those mli’s up.)

These are all things that we want to see fixed, and I’m hopeful that the work coming out of OCaml Labs on that front will solve the tooling issues, at which point we can do more to integrate the generated docs into our workflows, and improve the docs yet more.

y

In my own personal experience, I have found Jane Street’s documentation to be fabulously great, when one can find the documentation. However, my day to day experience often mirrors Perry’s experience; and as a working developer, I have found that typically the best way to find exactly the documentation that is relevant, I just have to look directly at .mli and *_intf.ml files. In some cases, when the documentation is not sufficient, I often end up needing to search for examples of a particular idiom. This involves checking out all of Jane Street’s publicly available open source code, and grepping through it until I have found an example which uses a particular feature idiomatically.

2 Likes

Probably, but I’ve said as much more compactly before and I don’t think it got as much attention.

That’s probably true. On the other hand, I suspect that pushing hard on the doc generating tools for a month or two would also fix the problem, or at least, substantially improve it.

(And we have nice shortcuts in our tools for bringing those mli’s up.)

These shortcuts you refer to, are they internal Jane Street tools which one would only be able to use at Jane Street?

An honest question, and I’m only asking because for a while, ppx_expect was publicized but not really easy to use until dune added the ability to run inline tests. As far as I am aware, expect testing was available internally through something provided by jenga(?) (some comments referred to an “inline_test_runner” script which was supposed to be available in the local directory, but I was never able to generate, and essentially had to reverse engineer what it was supposed to be doing). However, once dune added support for it, expect testing became awesome and super easy to use.

A wild thought, but could it be that some of those shortcuts and tools are only available internally? Merlin gets me pretty far most of the time, but I still invariable fall back to grep more often than I’d like to admit.

I think you’re mistaken. Long emails where short ones would do will not increase the set of people who will want to hear from you.

Far more effort than that has already gone in. The problems are unfortunately non-trivial, since it requires really understanding the module structure fairly well to generate the docs correctly. I don’t think we’re incredibly far from a solution, but I wouldn’t trivialize it.

Internal. The gap between Jane Street’s internal tools and what’s available publicly keeps on shriking (Dune, ocp-indent, ocamlformat, Merlin, all help.), but a lot of the editor customizations we have are only internal for the moment, mostly because they’re tied to purely internal infrastructure.

I don’t understand your last point, but we’re definitely converging as we move towards flipping from Jenga to Dune internally. I’m sure there will always be some gap, but hopefully it will get smaller and smaller over time. Dune in particular should make navigating through the tree with Merlin better, so if you check out and build your code and your dependencies with Dune, you should be able to navigate with Merlin to the file you need quite seamlessly.

I expect more of the tooling to make that practical to come out in the next 3-6 months.

y

5 Likes

You can disregard my last point, as the previous point answered my question.

I’m glad to hear an acknowledgment about what I always suspected regarding internal vs external tooling. However, as always, I’d like to reiterate my personal gratitude to the team at Jane Street and all the incredibly valuable resources you have given and continue to give away to the community. Looking forward to improved tooling as it becomes available!

1 Like

You might like batteries then (excellent documentation).

Extended standard libraries, like Jane Stree Core or Base and Batteries only/mostly have tail recursive functions.
Hence, your program will not stack overflow while in production when you hit that 100M elements list.
And, when you program, you don’t need to check the documentation to see if the implementation is tail-rec; which you should do if you are using the stdlib…

2 Likes

I only use my own minimalist extensions, like this.

Your short sentence looks important to me (I would love to work only with a secured version of the stdlib, and optionally with a strictly minimal set of libraries).

Can you tell us what kind of OCaml programs you can create with your minimalist extensions?
Do you think that this choice prevents you from writing some kind of programs?
If you want to write any programs in OCaml, I understand that you should be forced to reinvent what is already in those external libraries you don’t rely on. Then do you evaluate the cost for you?

I don’t think there is much that is missed, TBH. Though I’m usually keen on data structures and will implement them separately. I prefer the stdlib whenever I can. These few changes were enough, there are a few more elsewhere. Maybe I should split that stuff make a small util library. There isn’t a real need for a stdlib replacement unless there is a radical re-design IMO. The only thing that’s problematic is that some functions are slow and some common functional programming functions are absent, which were fixed in my util.ml’s. :slight_smile: