Good to hear the feedback (though in my own documentation-related request: fewer words! I think that could have been conveyed more clearly in 1/4 the space.)
Seems to me like the core of the issues you raise are the things we know, and that are related to the mismatch between the techniques used for constructing Base (and our libraries more generally) and the current state of the doc-generation tools. A sad but true fact is that the mli’s in our libraries are generally easier to read than the generated docs:
(Which is not to say that the docs themselves couldn’t use some more detail. They surely could. But it’s the double-underscores and extra indirections that are really deadly.)
It’s perhaps no surprise that within Jane Street, we read the mli’s far more often than we look at the generated docs. (And we have nice shortcuts in our tools for bringing those mli’s up.)
These are all things that we want to see fixed, and I’m hopeful that the work coming out of OCaml Labs on that front will solve the tooling issues, at which point we can do more to integrate the generated docs into our workflows, and improve the docs yet more.
In my own personal experience, I have found Jane Street’s documentation to be fabulously great, when one can find the documentation. However, my day to day experience often mirrors Perry’s experience; and as a working developer, I have found that typically the best way to find exactly the documentation that is relevant, I just have to look directly at .mli and *_intf.ml files. In some cases, when the documentation is not sufficient, I often end up needing to search for examples of a particular idiom. This involves checking out all of Jane Street’s publicly available open source code, and grepping through it until I have found an example which uses a particular feature idiomatically.
Probably, but I’ve said as much more compactly before and I don’t think it got as much attention.
That’s probably true. On the other hand, I suspect that pushing hard on the doc generating tools for a month or two would also fix the problem, or at least, substantially improve it.
(And we have nice shortcuts in our tools for bringing those mli’s up.)
These shortcuts you refer to, are they internal Jane Street tools which one would only be able to use at Jane Street?
An honest question, and I’m only asking because for a while, ppx_expect was publicized but not really easy to use until dune added the ability to run inline tests. As far as I am aware, expect testing was available internally through something provided by jenga(?) (some comments referred to an “inline_test_runner” script which was supposed to be available in the local directory, but I was never able to generate, and essentially had to reverse engineer what it was supposed to be doing). However, once dune added support for it, expect testing became awesome and super easy to use.
A wild thought, but could it be that some of those shortcuts and tools are only available internally? Merlin gets me pretty far most of the time, but I still invariable fall back to grep more often than I’d like to admit.
I think you’re mistaken. Long emails where short ones would do will not increase the set of people who will want to hear from you.
Far more effort than that has already gone in. The problems are unfortunately non-trivial, since it requires really understanding the module structure fairly well to generate the docs correctly. I don’t think we’re incredibly far from a solution, but I wouldn’t trivialize it.
Internal. The gap between Jane Street’s internal tools and what’s available publicly keeps on shriking (Dune, ocp-indent, ocamlformat, Merlin, all help.), but a lot of the editor customizations we have are only internal for the moment, mostly because they’re tied to purely internal infrastructure.
I don’t understand your last point, but we’re definitely converging as we move towards flipping from Jenga to Dune internally. I’m sure there will always be some gap, but hopefully it will get smaller and smaller over time. Dune in particular should make navigating through the tree with Merlin better, so if you check out and build your code and your dependencies with Dune, you should be able to navigate with Merlin to the file you need quite seamlessly.
I expect more of the tooling to make that practical to come out in the next 3-6 months.
You can disregard my last point, as the previous point answered my question.
I’m glad to hear an acknowledgment about what I always suspected regarding internal vs external tooling. However, as always, I’d like to reiterate my personal gratitude to the team at Jane Street and all the incredibly valuable resources you have given and continue to give away to the community. Looking forward to improved tooling as it becomes available!
Extended standard libraries, like Jane Stree Core or Base and Batteries only/mostly have tail recursive functions.
Hence, your program will not stack overflow while in production when you hit that 100M elements list.
And, when you program, you don’t need to check the documentation to see if the implementation is tail-rec; which you should do if you are using the stdlib…
Your short sentence looks important to me (I would love to work only with a secured version of the stdlib, and optionally with a strictly minimal set of libraries).
Can you tell us what kind of OCaml programs you can create with your minimalist extensions?
Do you think that this choice prevents you from writing some kind of programs?
If you want to write any programs in OCaml, I understand that you should be forced to reinvent what is already in those external libraries you don’t rely on. Then do you evaluate the cost for you?
I don’t think there is much that is missed, TBH. Though I’m usually keen on data structures and will implement them separately. I prefer the stdlib whenever I can. These few changes were enough, there are a few more elsewhere. Maybe I should split that stuff make a small util library. There isn’t a real need for a stdlib replacement unless there is a radical re-design IMO. The only thing that’s problematic is that some functions are slow and some common functional programming functions are absent, which were fixed in my util.ml’s.
One of the stdlib problem - prevalence of the exceptions, which is always a pain if you try to stick with functional approach. While Base/Core still has plenty of exceptions-throwing functions, it tries to move them in “<func_name>_exn” naming, which is very useful. I think at some point all those functions should be rewritten to Result-like or just sum-type return instead of using exceptions.
AFAIK, with OCaml you can wrap a function that may raise an exception in order to make it use of values with relevant return types. Right ?
And you can also filter entries to only accept values of some type (pre-conditions).
This is probably not sufficient for a pure functionnal style but it should help.
What do you think about that?
This is just a thought from a simple OCaml user. No more.
Exceptions are great for things that are actually exceptional, like rare errors. For something that’s expected (say, end of file when reading, or hitting the end of a list or what have you), option is a lot better as choices go.
Typically I use exceptions for things like likely internal errors in my code, or, say, in a parser, signaling that the thing being parsed is in fact malformed. Otherwise I use option.
The problem is that functions don’t say which exceptions are being thrown, so you either need to read the documentation or even worse, run the code to discover what exceptions are thown in which cases. The compiler can’t help you at all in these cases.
I in particularly don’t enjoy legacy codebases which suddenly throw Illegal_argument in production for who knows what reason or where.
Sometimes I think Java was onto something with checked exceptions, it was just lacking the proper (language-level) tools to deal with checked exceptions properly. That said, I haven’t used it much and there is a strong community disapproval of checked exceptions.
Checked exceptions get very irritating fast, but I agree with your comments about how annoying exceptions can be. My attitude is you use them only when something really bad has happened.
I disagree that option is a better choice for that, with exception handling the code becomes so much easier to read and it becomes sensible because exceptions are also an event driven programming mechanism. Though I’d say that instead using a functor-ial lib for explicit reactive programming would be better.
