What are some libraries you almost always use?

Any interesting project needs tests, and qcheck is my favorite way to test. I think it’d be neat if a standard library encouraged property based testing (with perhaps a section on using constant generators for unit testing, as a degenerate case of PBT).

Expect tests or cram style tests are a nice informal complement.

4 Likes

Since moving to dune, I have no use for ocamlfind; what makes it indispensable for you? pcre binds C libraries - I’d prefer re because it is a pure OCaml package and hence likely to be more portable.

1 Like

Qcheck is remarkably easy to port, btw. I ported qcheck-core to BuckleScript once with very minimal changes.

2 Likes
  • Menhir. There’s no parser generator that compares to it.
  • re. Regular expressions without a weird syntax and pathological performance edge cases.
  • Markus Mottl’s numerical libraries (lacaml, gsl). Highly stable and a great resource for understanding how to write C bindings.
  • Jane street’s libraries deserve a special mention. They’re rarely usable in practice because of portability and dependency concerns, but they’re the best reference when one needs to figure out how to do something correctly and efficiently in OCaml.
9 Likes

Ah. So: I don’t use Dune, for three reasons:

  1. I’ve already got a bunch of Makefile-based projects, and they work fine, really fine. I see no gain in dune
  2. Dune is “opaque”: I’ve seen too many questions “why is dune doing this?” for me to feel comfortable with it.
  3. I build projects from time-to-time with significant C/C++ components: Dune doesn’t support these well – frankly, nothing will other than Make.
  4. I wrote a little “wrapper” for ocamlfind called “not-ocamlfind” that provides a little extra function, and with that, building multi-directory projects with Make is actually really sweet. To wit, each directory’s makefile installs its final product into a “local-install” directory at the top of the project, but using a “reinstall-if-diff” operation. During “make depend” processing, it looks at the relevant packages in “local-install” and puts in dependencies on the META files in those packages.

The effect is that, when you have a nontrivial graph of directories-and-dependences, you just run thru them doing “make” (from your toplevel makefile) and only those directories that depend on other directories that got recomipiled, will need to be recompiled.

What I’m trying to say is: the “composition model” of findlib packages, is equally applicable to the internal organization of a large project into subdirectories and sub-packages. As an added plus, when writing tests you just assume the packages being tested are already installed, and findlib will find them. So writing/running tests -inside- a project, is the same as doing so -outside- the project.

But whatever: I get that people like dune. It’s all good.

2 Likes

I use cmdliner for user-facing executables, but there’s no other library that I use systematically. The other tools I use systematically are not libraries (ocamlfind, opam, dune, …).

1 Like

I use all those very often: batteries, dolog, minicli, parany.

2 Likes

Since many people mentioned it, I’d just like to say that looking at cmdliner's API makes me feel slightly nauseous these days.

A separate redesign would be a good idea but somehow that does not put food on the table and I still manage to cope with the current one.

Also, refined and better designs of the other libraries you mention can be found here.

9 Likes

Cmdliner’s applicative style looks really nice with let operators though.

1 Like

Some feedback on cmdliner from a late adopter (I know it’s off-topic, but we’ll be ok):

I don’t understand the cmdliner type machinery, but I get by by using past projects as templates and sources of examples aka copy-paste. Also, I avoid using & completely (I forgot what it does except that it’s trivial).

Once I have a template for a CLI implementation, most of the time is spent figuring out how to express the type/format of a command-line argument. Maybe a cheatsheet would be beneficial.

The feature-completeness and the output of cmdliner are amazing.

7 Likes

I forgot to mention alcotest, for unit tests. I’m a very happy user.

9 Likes

Since @ostera gave his interpretation of this here. I’ll make that statement more precise as I find myself quite in disagreement with what he wrote under what I find to be the wrong lenses (or rules) to assess the situation.

I won’t comment on the docs aspect since I don’t find cmdliner's docs to be particularly good – nor particularly bad either but I would certainly write them differently now that the feature set of the library has changed (more on this below).

However I’d like to address two impressions that come out of that article before they start to become myths. These are:

  1. The API is hard to use.
  2. You need to understand what an applicative is to be able to use the library and that’s a high bar for usage.

Regarding 1. I don’t think the API is hard to use. As far as I’m concerned a hard to use API would be an API that allows you to easily shoot yourself in the foot, that makes it hard to define, tweak or evolve your command lines, or makes it hard to understand what is going on when you come back to the code or when you have a bug. I personally find none of that to be true.

Regarding 2. You absolutely do not need to understand what an applicative is to be able to use the library. With the time I even evicted that fact from documentation and renamed Term.pure to Term.const. I think that any working OCaml programmer should be able to start from the short basics and gradually tweak that example to get to what s.he needs without ever having to understand what an applicative is by simply following the type mechanics.

So if it’s not 1. what it is ? One of the reasons why a lot of cmdliner code gets cut and pasted to be modified is that the API induces a lot boilerplate and that it became slightly messier over the years. Here are few reasons for why this is the case:

  1. It started simply as a cli parsing library. However over time, it gradually evolved towards an “os process” interface library. Environment variable lookup was integrated as well as a formalization of program exits and their documentation. This was bolted on top of the API without breaking it. This means that the current API is unlikely to be the best way of structuring and exposing the feature set. That of course hampers its usability.
  2. It failed to capture one important pattern that became widespread as command line
    tools grew in complexity over the last decade which is to have specific command lines syntaxes for tool object verb not just tool cmd. Many people, myself included, ended up manually encoding this pattern in an unsatisfactory manner too many times. (This PR is meant to fix that, but that may not help with streamlining the API).
  3. The Arg.t to Term.t mechanics could likely be streamlined by using less applications (in fact the new design I have in my head eschews it entirely).
  4. A few defaults that became clearer as more and more cmdliner programs have been written could likely be changed and/or integrated to cut on some of the boilerplate.
  5. The library was written 10 years ago and OCaml and its stdlib were different. Newcomers often evaluate program sources and designs with respect to the current state of the art rather than in the light of the era in which they were created; that leads to misunderstandings. For example there was no result type in the stdlib. Its integration both at the Arg.conv and Term level was added later, perhaps hastily, which muddled the API; nowadays it would certainly play a central role in the design, both for parsing and managing exits. Another example is the & operator which confuses a lot people but would nowadays simply not exist since @@ does, or not be needed at all (see 3.).

Finally to come back to the use of applicative which in the particular case of cmdliner was discovered, not applied. I have seen many people over time claiming that this was non-obvious, needlessly bureaucratic for “just parsing an array of string” or that what they wanted is just “simple” and direct Arg-like mutations to get their parse result (these people should be forced to go work on the ocaml drivers…). These thoughts largely miss the point in my opinion. I think the current design is a good one because:

  1. It scales. In a term you can encapsulate non-trivial cli interaction that eventually defines a whole immutable datastructure.
  2. It is composable. Your libraries can expose cli interaction terms to be reused.
  3. It precisely avoids the need for reference cells or mutations. These become tempting global mutable state or enable easy to do but hard to understand effectful contorsions; so it’s better to avoid them.

These points encourage you to have a good program structure where you define data structures and algorithms, and, cleanly separated from them, the cli interface and its logic to expose them to the shell.

20 Likes

@dbuenzli :wave:t4: thanks for taking the time to answer!

I think we should encourage more public discussions like this since we all can learn a lot from each other. Especially because I think your contributions have time and again forwarded the OCaml platform

I’ll say tho that the Rambling Machines mailing list is meant as a place for me to share less-polished ideas, with as little editing as possible, in contrast to the essays I’ve got on my website.

I even encourage replies like this one! :raised_hands:t4:


With that out of the way, on to your points!

before they start to become myths

I don’t think I’m starting any myths here. I’ve heard this from a lot of people coming from the Reason world, for well over 3 years now. :thinking:

Regarding 1. I don’t think the API is hard to use

Hard and easy are completely subjective and rely on your body of knowledge and experience. If you asked me how hard it is to contribute to Caramel, I’d say its super easy! But also, I wrote it, and I’ve become more familiar over time with the OCaml compilation toolchain, and AST traversals, etc. “Easy” is earned.

Sometimes a “hard api” is the one that only lets you do the right thing, or impose usage patterns that take a lot of time to figure out, or doesn’t use the metaphors you’re used to. This is, again, entirely subjective.

If this sounds confusing its because when people say easy or hard they mean very different things.

Sometimes in the same sentence.

Regarding 2. You absolutely do not need to understand what an applicative is to be able to use the library.

I absolutely agree!

What I meant with “actually understand how to use cmdliner” (emphasis as in the newsletter issue) is that to be able to think clearly about what the code is doing, you need an understanding of applicatives. Maybe only intuitively. That’s what the “actually” stands for in that sentence. Lack of editing gets you this lack of nuances :slight_smile: so I’ll agree with you that this could have been written more clearly.

Of course you can use cmdliner and get a cli running, but that doesn’t mean you understand how it does it :woman_shrugging: – just like I don’t have the palest clue how my inductive stove works, and I still can make food.

Here are few reasons for why this is the case:

This is good background story to understand the current state of things. Thanks for sharing!

These thoughts largely miss the point in my opinion

Perhaps! But the points you focused on when designing and evolving Cmdliner left gaps, by choice or accident, that I tried to highlight. Some of those gaps exist there because you are a proficient ocaml programmer focused on building composable and scalable libraries.

Which is why I really like that you took the time to reply and make your focus clearer as well.

Anyway, I’d be happy to continue this chat and explore some of your new ideas for the API :raised_hands:t4:

Maybe we can find ways to make it composable and scalable while remaining intuitive and convenient to a larger part of the ecosystem.

3 Likes

If it’s subjective then why do you say the API is hard to use ? It brings nothing to the discussion and becomes a baseless claim which is precisely the start of myth building.

Personally I don’t think hard and easy are subjective. It can be contextual but for each context you can specify criterions to assess the level of hardness which is precisely what I did in my reply to demonstrate why I think it’s not for the context of the working ocaml programmer.

3 Likes

Instead of ranting against Cmdliner

you can try some alternatives in opam.

I think Daniel has a style, and his style can be recognized in all his libraries.
Either you like his style of API design and heavy use of types, either you don’t.

The newcomer alternative:

My personal hack, for fast development of executables with a CLI that must be correctly used, but still easy to change:

Conservative people still use this one:
https://caml.inria.fr/pub/docs/manual-ocaml/libref/Stdlib.Arg.html
I think that once you have a working example, Arg is bearable.

If you don’t like any of those, then write your own, publish it in open-source and contribute to the fantastic and always changing OCaml ecosystem bazaar. :smiley:

1 Like

I’d be curious where you see heavy use of types ?

I do use types to design APIs because it is a strong organizing and compositional tool.

But most of the time I eschew more advanced type level programming tricks (e.g. phantom types) because they tend to hamper usability (e.g. unreadable error messages, unflexible code motion, etc.) for little returns.

That’s an interesting comment because I thought about it but forgot to mention it in my assessment above. Like Cmdliner when I need to use Arg I go back to a previous example, cut and paste it and start from there…

1 Like

There are multiple things one does when one “uses an API.”

One thing that is somewhat difficult is to figure out how to use multiple functions one-after-the-other to do what you want. Wait, how do I make this arg optional? Uh, the type system says I have only partially applied something?

One thing that is somewhat easy is to make sure you haven’t forgotten something, you haven’t mixed up two functions, etc. In short: that you haven’t misused the library. This is somewhat easy because you can lean on the type checker! The library defines a tightly typed playground within which there is one way to do one thing and anything else will get you a compiler error.

There are some libraries that are much easier on the first part (where you can write code that compiles easily) but much harder on the second part (where you can write code that fails at runtime in difficult to debug).

3 Likes

Boy, this thread’s gone really off-topic, but hey, at least I get to learn about great stuff like this! Looking forward to this!

I don’t use Dune

I’m glad not being alone :slight_smile:

Great and useful information. I appreciate your answers, guys!

1 Like