[ANN] Cmdliner 2.0.0

Community
,
1

Hello,

It is my pleasure to announce the release of cmdliner 2.0.0.

Cmdliner is a library that allows the declarative definition of command line interfaces with outstanding support for command line interface user conventions and standards.

The main points of this release are:

  • ANSI styled error and deprecation messages (details)
  • Support for manpage installation (details)
  • Support for shell auto-completion (details)

The latter was made possible by good initial ground work of @andreypopp who can now claim to have unblocked my mind and the very first and 11 years old Cmdliner issue. Many thanks to him!

This addition has the following consequences:

  1. The problematic feature that allowed you to specify command names, option names and enumerant values by a prefix if the prefix was unambiguous has now been removed. See this issue for the rationale. Set CMDLINER_LEGACY_PREFIXES=true in your environment if you find yourself in need of a quick backward compatibility fix because one of your scripts is failing due to a prefix being used (but do eventually correct the script!).

  2. It finally triggered making the type Arg.conv abstract as announced it would become in 2017. See this issue for details.

If you are a user of cmdliner based tools. You may want to have at a look how to configure your shell in order to benefit from their completion scripts, especially if said tools are installed via opam. After installing cmdliner you should be able to check that your configuration works correctly on the new cmdliner tool that now gets installed with cmdliner itself.

For other changes that may affect you or your users please head to the release notes which have many other details.

Other than that a full pass was made over the documentation to try to improve and bring it up-to-date with the latest style and additions. Notably the tutorial and examples were updated to make use of the binding operators; however obscure let punning may feel, these are less error prone as your number of cli arguments grow.

I also added a cookbook which tries to distill in shorter snippets some of cmdlinerā€™s features and the experience I gathered over the past 14 years of using cmdliner to define dozens of command line interfaces. It includes source code structure tips and a few bootstrapping blueprints to cut and paste for when you start your next command line tool.

For this release Iā€™m very thankful to a private one-time donation[1], a grant from the OCaml software foundation and, as always, my few but faithfull donors. All of which are essential for these releases to eventually get out. They do take quite a bit longer to devise that one would expect :ā€“)

Home page: https://erratique.ch/software/cmdliner
API docs & manuals: https://erratique.ch/software/cmdliner/doc/ or odig doc cmdliner
Install: opam install cmdliner (once the PR is merged, may take a few days)

Best,

Daniel

  1. Which are as nice as recurring donations ;ā€“) ā†©ļøŽ

28 Likes
2

To follow up on the completion feature. It should be stressed that I donā€™t consider it to be fully ā€œdoneā€ as it stands. Iā€™m pretty sure the completion API, protocol, features and the generic completion scripts can be improved. The main problem is that it seems shell programmers are more interested in cajoling the look of their prompts than defining sane cross-shell standard protocols for tool/shell interaction. The current completion mecanisms are broken beyond imagination.

Issues about completion are tagged accordingly in the issue tracker. Do not hesitate to chime in if you have ideas or more knowledge than I do for improvements. Iā€™m also happy to add support for more shells but itā€™s better if you help for that because working with shells makes me want to throw my computer out of the window.

Meanwhile Iā€™d like to show two completion feature that Iā€™m quite happy to have support for in this release.

Context sensitive completion

The idea here, suggested by @andreypopp, is that completion can depend on a context that is specified via a cmdliner term itself. This is typically useful for configuration dependent completions: you have a cmdliner term that represents your configuration and you access it when doing a completion.

For example in the next release of odig you can autocomplete package names on odig doc [PKG]. The available packages depend on looking up a libdir which can be specified with a command line argument itself. So for example this completes according to the automatic libdir lookup

# Auto discovered libdir
> odig doc cā‰
camlp-streams   checkseum  cmarkit  camlpdf  cairo2  
containers      cmdliner   cpuid    cppo     cpdf    
cstruct-lwt     cstruct    ctypes   crunch   csexp   
ctypes-foreign

But the following looks for packages in another switch:

# Explicit libdir
> odig doc cā‰ --lib-dir $(opam var lib --switch=myswitch) 
capitalization  cerberus-lib  calendar  charon        
core_kernel     core_unix     cmdliner  cppo    core  
cstruct         csexp

The commit that implements this in odig is rather straightforward, it simply reuses the existing conf term for the completion context. Also the cookbook has a simple self-contained example to start from.

Compositional completion (restart and raw)

The second completion feature is to retain completion on the tools that another tool invokes ā€“ commands like sudo.

For example in the b0 tool, the build system Iā€™m using for all my developments. The b0 vcs [OPTION]ā€¦ -- VCS [ARG]ā€¦ command allows to bulk operate the VCSs of the projects you included in a B0.ml build description file.

Using appopriate cmdliner completion directives the completion of this command first completes the VCS enum (which can be git or hg) and then gracefully drops back to the completion of
your VCS:

> b0 vcs -- ā‰
git  hg  --
> b0 vcs -- git shā‰
shell       -- restricted login shell for GIT-only SSH access
shortlog    -- summarize git log output
show        -- show various types of objects
show-branch -- show branches and their commits
show-index  -- show packed archive index
show-ref    -- list references in a local repository

This is a restart completion type. It restarts the completion context as if the cli started after the -- token.

It is the kind of behaviour you want from e.g. opam exec, though arguably in the case of opam exec it will be sligthly misleading since completions will occur using the outer environment rather than the one setup by opam exec -- TOOL [ARG]ā€¦. Still, sometimes inaccurate completion is better than no completion.

Note that this would be quite easy to solve with a good cross-shell completion standard: just invoke TOOL in the environment setup by opam exec according to the completion standard (e.g. the cmdliner completion protocol) and propagate the result back in the completion for opam exec, but we do not live in that world. Still the API is ready for such a technique to be used, by using a raw completion type (I use this in b0 to complete custom, library and user-defined, actions like b0 -- .opam or b0 -- .ocaml).

11 Likes
3

This looks amazing, and I canā€™t wait to try it out. Thanks for all the hard work, and thanks for the great documentation! The cookbook is especially useful.

1 Like
4

Iā€™m curious about this part:

Avoid default option values

Optional arguments with values can have a default value, that is be of the form --opt[=VALUE]. In general it is better to avoid them as they lead to context sensitive command lines specifications and surprises when users refine invocations. For examples suppose you have the synopsis

tool --opt[=VALUE] [FILE]

Trying to refine the following invocation to add a FILE parameter is error prone and painful:

tool --opt

There is more than one way but the easiest way is to specify:

tool --opt -- FILE

which is not obvious unless you have toolā€™s cli hard wired in your brain. This would have been a careless refinement if --opt did not have a default option value.

Optional arguments with default values are often useful. Or at least I often use them in things Iā€™ve made with Python argparse.

Optional arguments as flags whose presence or absence are treated as setting a value, usually true/false, are also useful.

But if the optional arg takes a value then the user entering tool --opt seems meaningless, and tool --opt <file> I would expect it to take as meaning opt=<file>.

Is the caveat above saying that if the user enters something meaningless (opt with no value) then the parse is ambiguous and itā€™s hard to give an error? Or that the useful cases are also problematic?

5

As mentioned in the synopsis it takes an optional value (--opt[=VALUE]), so itā€™s not meaningless.

You canā€™t give an error. If you specify a non-optional argument (does not start with -) after an --opt token this will be taken as VALUE.

Technically this arises from being nice to users and allow option values to be specified as a separate argv argument rather than force them to be glued to the option after an = (or to the option name in the case of short options). Itā€™s a usability tradeoff.

6

Is there a way for me to provide manpages or completion scripts to end users who are downloading my tool as a binary (i.e., will not have an opam environment or cmdliner installed themselves)?

1 Like
7

Sure you can generate all the support file beforehand yourself and add them to your tarball, see cmdliner --help and here for manpages and there for completion scripts.

2 Likes
8

The python equivalent of the thing weā€™re talking about is setting nargs to '?' for an optional argument which I suspect you donā€™t actually use a lot. (Btw if you want to use it you need to set two defaults, default and const.)

Whereas you seem to be talking about simply setting default for actions that take exactly one argument.

9

I guess what I really meant is tool --opt[=VALUE] [FILE] seems fine to me

  • Iā€™d assume just tool with no args would mean opt is set to its default value
  • tool --opt=something would be explicitly setting it to a new value
  • but tool --opt by itself doesnā€™t seem meaningful, user is using it wrong
  • ā€¦so interpreting tool --opt FILE as opt=FILE seems fine and expected

And then in other cases you have optional args which are just flags (store_true in argparse)

like tool --verbose might set verbose=true, with false as the default value if omitted

and for args like that tool --verbose FILE is no prob because verbose doesnā€™t expect a value

10

But then the synopsis is tool --opt=VALUE [FILE] which is something else and not what the cookbook is talking about.

In what Iā€™m talking about the synopsis is --opt[=VALUE] [FILE] and tool --opt is meaningful so the user ā€œis not using it wrongā€.

(I think you are confusing the notion of default value taken in your program if the option is not specified on the cli and the ability to have on the cli an option name that takes a value but that can be left unspecified).

11

Thanks a lot for this release. I notice some ecosystem utilities are not yet up-to-date with cmdliner 2.0, but fortunately lots of them have cmdliner 2.0.0 support merged in their development branches. When I come across these (ready, but not released), I open issues to ask whether they could cut a release.

So far:

(Iā€™m aware thereā€™s still the unsolved issue of cmdliner 2.0.0 support Ā· Issue #1602 Ā· mirage/mirage Ā· GitHub - I lost steam when trying to work on it again.)

3 Likes
12

Thanks for the summary.

If you are dying to use cmdliner.2.0.0 then donā€™t die for that but you can always use the repo version for those package that support it. I have been living with opam pin add ocp-indent.dev --dev for a couple of months now. ocp-indent has seen a bit of activity by @NathanReb recently so Iā€™m hopeful weā€™ll get a release soon.

For alcotest I see a release is on the way (thanks @dinosaure) but technically alcotest can be unblocked without a release: one can also PR the opam-repository to modify the opam file not to run-test: (thatā€™s the case for a few other packages). See this discussion.

13

If it is just tests that are failing maybe the opam file in opam-repository could be updated to make run-test only execute if the alcotest version is < 2.0? That would unblock a lot of older versions as well.

14

Yes this can be done. But not by me.

I decided that I would not deal with such a granularity of details to maintain the OCaml opam-repository health on cmdliner releases (which are not the most pleasant thing to do given the amount of reverse dependencies). If your package fails because of the release, it gets a constraint.

Iā€™m also a bit annoyed in this in case since Iā€™m pretty sure I already warned the project on an earlier cmdliner release that they should not run expect tests in the repo on error message they donā€™t control.

This wastes everyoneā€™s time. I repeat, be mindful about the tests you run in the OCaml opam-repository.

2 Likes
15

One more seems to be:

This one seems again to be a run-test: issue. A work around is:

opam install js_of_ocaml --ignore-constraints-on=cmdliner
16

Js_of_ocaml doesnā€™t run test with opam AFAIK..

17

You are right. These was a kerfuffle during the cmdliner merge due to the unexpected interference of an experimental tool. This has now been resolved.

18

With the new release of ocamlformat it seems that all the packages that @hannes mentioned no longer block use of cmdliner.2.0.0.

Is there anything still preventing people from using the release ?