Plans to choose an official package/project manager?

[yawaramin]

We’re not far from the organizational hierarchy, but in terms of DX it’s quite different

Btw, package names in Go are hierarchically namespaced, e.g. net/http. Or for packages coming from modules, e.g. github.com/google/uuid. Importing these brings a package with the name http or uuid into scope. And running go mod tidy resolves any imported modules to their sources, downloads them if necessary, updates the go.mod (manifest) file, updates the lockfile, and caches the module. It feels very smooth and integrated.

[rgrinberg]

What if a module github.com/foo/bar contains two packages x and y? How would those be visible in the source?

[yawaramin]

The go tool enforces that each package must be in a directory of the same name. So the project root would contain directories named x and y. In each subdirectory, you would put the files for the corresponding package.

[rgrinberg]

I don’t think the popularity you had in mind was ever on the cards for OCaml - even with the kind of slick tooling you had in mind.

I’m not saying that unified tooling isn’t desirable, but a topdown mandate is not the way to go. We (at least me) are not keeping everything siloed on purpose. Unifying things in a coherent way is something that takes enormous effort. I’m confident that we’ll get there eventually.

[shonfeder]

Is poetry “official”? Tool recommendations - Python Packaging User Guide suggests pipenv, then pip, then pip-tools, then poetry, in that order. It sounds like your team has adopted poetry, and maybe the python ecosystem at large is moving in that direction? But iiuc, this is something that is emerging from a plethora of tools targeted at this space in Python-land.

I don’t think I fully understand the diversity in tooling argument

As I read it, avsm was not being prescriptive. They just offered a description of the current state of things, along with an explanation of why things are in that state.

iiuc, essential to avsm’s reply is this idea: (fortunately!) the OCaml ecosystem doesn’t currently have an authoritarian “them” who lays down the law governing which tools everyone should use. There is a core group who shoulders the responsibility of helping to curate, document, and maintain core tools and infrastructure. But it is we all, the members of this community, converging on an overwhelming agreement on tools that makes them “core”. For my part, this is a virtuous and healthy way to cultivate the ecosystem.

Excellent development tooling is of course a boon to developers! But we don’t end up with such tooling just by saying we want it, or asking “them” to decide that something is excellent. Someone has to do the work to make the excellent thing, and then the thing’s excellence has to be recognized by the developers. People here are communicating very clear and concrete ways we can help contribute to this outcome.

if the goal is to make OCaml more popular

Not everything boils down to a popularity contest. Of course those contributing to and curating the ecosystem want to make the various tools, documentation, communities, and infrastructure more inclusive, accessible, inviting, excellent, etc. etc. But this doesn’t mean the language popularity contest has to dominate every concern and overrule all other interests.

I do not think it is the right approach

You do not think what is the right approach? What exactly do you want to see happen? And what kinds of contributions and engagements are you planning to make to help improve the chances of that outcome?

[shonfeder]

Speaking of this low-hanging fruit for automation (and coincidentally ending up with a nice round issue number): Add a default @sync build rule to update opam deps · Issue #5000 · ocaml/dune · GitHub

[yawaramin]

This is partway there. Ideally dune should automatically keep the dune-project file updated by traversing all the dune files in the directory structure, adding all dependencies it finds there, and also downloading and building the packages as needed. I think it can find all the package manifests in the local copy of opam-repository?

Since this may be somewhat less than instant, it’s reasonable to do it with a specific command e.g. dune sync.

[Gopiandcode]

@yawaramin

Ideally dune should automatically keep the dune-project file updated by traversing all the dune files in the directory structure, adding all dependencies it finds there, and also downloading and building the packages as needed. I think it can find all the package manifests in the local copy of opam-repository?

Doesn’t the dune-lint plugin do this? For projects I intend to publish on opam, I often use opam dune-lint to make sure all my dependencies are accounted for and up to date.

[yawaramin]

Oh cool, looks like it does some of that. But it’s only part of the workflow. Take a few minutes and check out the Go install/setup links I posted above, you’ll see what I mean, it gets you to the point where not only is the project manifest file updated, but all the dependencies are downloaded as needed, and the lockfile is updated.

[bluddy]

Some thoughts I have based on the discussion so far (I’m not certain about all of them):

1. OCaml’s limited manpower means it’s very hard to come up with a tool outside of opam and dune that will be built to the same standards. Ideally, one of them could take over the task of running the show, similar to what other languages have. As an illustration, we see how esy was able to replace the opam executable while still using the opam repository. I could see dune being expanded to fill the same role.
2. Having tools be opinionated is very good and is the only way to achieve parity with other ecosystems in terms of ease of use. A good example of this is dune’s wrapped option (which is on by default), or determining a set folder layout - people really don’t want flexibility in this area anymore. They want to stop needing to make trivial decisions that require time to learn and configure.
3. However, before making an opinionated choice, it’s crucial to first check out what the standards are in other language ecosystems and to try and copy the best practices as much as possible. Otherwise, we could accidentally make an esoteric choice that could turn people away. A good example of this is dune's config file format, which I’m sure scares people away from OCaml. I know it fills me with dread every time I have to deal with it, simply because I never deal with s-expressions in my daily life. I would guess that 90% of potential newcomers to OCaml will never have seen s-expressions, and now they have to learn to use this foreign-looking thing to configure what is an already foreign language.
4. It’s far more important to think about ease of use in future projects than it is about compatibility with legacy projects. dune succeeded because it supplied a much better new story than any existing tool, even though it didn’t cover all the bases for existing projects. If a new version of dune was planned that was more opinionated, but supplied more intuitive, complete solutions that competed better with other ecosystems, it would dominate as well IMO.

[Leonidas]

@shonfeder

Is poetry “official”? Tool recommendations — Python Packaging User Guide suggests pipenv, then pip, then pip-tools, then poetry, in that order. It sounds like your team has adopted poetry, and maybe the python ecosystem at large is moving in that direction? But iiuc, this is something that is emerging from a plethora of tools targeted at this space in Python-land.

That is exactly my experience. When I used Python there first was no tool whatsoever, you downloaded tarballs from wherever and used python setup.py install which for the most part used distutils. Then came easy_install (which famously couldn’t uninstall things). Then virtualenv came by, but it was annoying to use so virtualenvwrapper was made then replace by venv or so. Then pip appeared and actually listing your dependencies in requirements.txt became a thing. At some point npm in Node became really popular so people decided that lockfiles are neat (they are!) so pipenv appeared which seems to have its own config file. Then there was some drama, and apparently poetry exists now? On the packaging side, you have good ol’ setup.py and setup.cfg but people have rightfully realized that not every installation needs to have custom code running (very similar to our .install files which are seemingly also on the way out) so there is setup.ini to do things declaratively.

I think the packaging story in Python is, if anything, even worse than in OCaml and people complain rightfully so. Big part of it is that there was no good (enough) solution from the getgo.

Both Node, Go and Rust came with a package manager from the beginning (or close enough to that to guarantee universal adoption), whereas OPAM came may years later (but has become the de-facto standard for supplying package metadata) and dune even later and while it has taken over a big part of the ecosystem there still exist considerable amount of packages that don’t use it for some kind of reason (it would be interesting to see some statistics from OPAM repository on that). So saying “it should be all unified” is hard, when the ecosystem is split.

This is all mostly an explanation of why things are as they are. I agree with @rgrinberg that the fact that things are two binaries isn’t bad per se (see cargo and rustup) but I also agree with @shonfeder that the integration between opam and dune is a bit rickety (part is of course because there’s no general mapping between modules, libraries and packages). And I have my own set of complaints that opam’s default commands are nearly never doing what I want. Fortunately this is in my opionion mostly an UI thing, not an architectural issue, so it is perfectly possible to build an opam plugin that acts more in the way I would envision it.

[dbuenzli]

Personally I see the separation between my build tool and the one that is in charge of setting up the build environment (the package manager) as a good thing – which obviously doesn’t mean you shouldn’t strive for a good integration between the two.

Especially if your package manager is generic like opam is, it provides a good point for integrating foreign dependencies not written in your language. Integrated language specific package manager or build tools fail to see the larger computing picture in which programming languages are embedded.

Speaking about the good things :–) Which other languages follow the practice of having an insanely curated package repository that rarely explodes in your hands (at least in mines :–) ? The rare time I tried to install python software not only was I confused at which tool I should be using, it also generally failed spectacularly.

[kanishka]

I started using OCaml relatively recently, coming from Scala and Haskell. I have enjoyed using esy and dune so far. I haven’t used opam, so my experience may be different from the typical ocaml developer.

I will probably switch to golang or python next, because the language decision will be made for me wherever I land. My hesitation about returning will be the small library ecosystem. I don’t think I will hesitate on tooling.

I would note that there are different onboarding scenarios. If OCaml was widely used commercially, more people might be onboarding as part of a job. In that case, experienced devs will already have setup the tooling and conventions. Other people might onboard through hobby programming alone.

[Maelan]

(Apologies for replying to an early message of this thread:)

@rgrinberg

I didn’t know about poetry either. I don’t use pip and usually use conda when I need a python environment. The fact that didn’t prevent Python from becoming the most popular language is contradictory to the premise of this entire thread.

That’s a false dichotomy. Languages become popular because of a combination of reasons (many of them circumstantial), different from language to language. OCaml obviously didn’t benefit from the same factors that made Python popular. That good tooling was not one of them doesn’t mean it wouldn’t help. Any advantage, even if not a single definitive argument, is good to take.

[rgrinberg]

I’m certainly not arguing that better tooling (more unified actually, because I think our tooling is pretty good) does not increase adoption. I just don’t think it’s essential to adoption as others are saying. I can see plenty of other issues that hinder adoption just as much if not more than having a single binary.

[lambda_foo]

I like the scaffolding with spin that makes it really simple to get started. Personally I was using dune init project <name> since it came out of the box with dune.

The main complaint I have is the workflow for:

1. Installing the correct dependencies eg when you add cmdliner it really should be driven off the dune-project file
2. Setting up an OCaml compiler and/or switch, again this should come from the dune-project file.

From the cargo docs:

$ cargo new grrs
     Created binary (application) `grrs` package
$ cd grrs/
$ ]$ tree -L 2  
.
├── Cargo.lock
├── Cargo.toml
└── src
    └── main.rs

1 directory, 3 files
$ cargo run
   Compiling grrs v0.1.0 (/Users/pascal/code/grrs)
    Finished dev [unoptimized + debuginfo] target(s) in 0.70s
     Running `target/debug/grrs`
Hello, world!
# Add cli library `structopt = "0.3.13"` to Cargo.toml
$ cargo build
    Updating crates.io index
  Downloaded unicode-width v0.1.9
  Downloaded proc-macro2 v1.0.30
  Downloaded syn v1.0.80
  Downloaded quote v1.0.10
  Downloaded libc v0.2.103
  Downloaded 5 crates (860.5 KB) in 0.75s
   Compiling proc-macro2 v1.0.30
   Compiling version_check v0.9.3
   Compiling unicode-xid v0.2.2
   Compiling libc v0.2.103
   Compiling syn v1.0.80
   Compiling unicode-segmentation v1.8.0
   Compiling unicode-width v0.1.9
   Compiling vec_map v0.8.2
   Compiling strsim v0.8.0
   Compiling bitflags v1.3.2
   Compiling ansi_term v0.11.0
   Compiling lazy_static v1.4.0
   Compiling textwrap v0.11.0
   Compiling heck v0.3.3
   Compiling proc-macro-error-attr v1.0.4
   Compiling proc-macro-error v1.0.4
   Compiling atty v0.2.14
   Compiling clap v2.33.3
   Compiling quote v1.0.10
   Compiling structopt-derive v0.4.16
   Compiling structopt v0.3.23
   Compiling grss v0.1.0 (/Users/tim/code/rust/grss)
    Finished dev [unoptimized + debuginfo] target(s) in 32.54s

The same workflow I use for OCaml is:

$ dune init project grss
$ cd grss
$ opam switch create 4.12.1-grss 4.12.1
....
$  echo "use ocaml 4.12.1-grss" > .envrc
$ direnv allow
$ opam install . --deps-only -ty
...
$ dune build @all

# Edit dune-project to generate the opam file and add the stanza for the cli into there.

(generate_opam_files true)

(package
 (name grss_ocaml)
 (depends
  (ocaml (>= 4.12.1)))
 (synopsis "cli"))

$ dune build @all
...
# Add cli library (cmdliner (>= 0.9.8)) to the dune-project
$ dune build @all
# Required to generate the updated *.opam files
$ opam install . --deps-only -ty
# Install the new dependency
$ dune build @all
# To get a build

That process is longer than it should be. But the good thing is it’s easily fixed with the tools we already have. For the switches/direnv there are local switches opam switch create ./ --deps-only. The mismatch between adding a library and installing it, could be a tool that watches for changes in dune-project regenerates the *.opam files and issues the opam install command. With both of those you’d get to a similar place as cargo.

I’m obviously overlooking more complicated processes where dune-project doesn’t generate the right opam files or other edge cases.

For the record, I’ll add that I haven’t done significant Rust development to know where the tooling doesn’t work well. I’m sure there are places, there always are.

[yawaramin]

There are a few other missing pieces. E.g. right now every local switch on your computer will reinstall and duplicate dependencies, taking up N times the space. I think binary installs are being worked on but I’m not sure if they’re shipped or how to use them.

There is also the opam pin feature which seems to do something similar to locking package versions Not sure how it interacts with locking though.

It’s difficult to track all this as the opam documentation isn’t really updated in sync with functionality. E.g. try to find the word ‘lock’ in opam - Manual or opam - Usage …

Anyway, the other issue with opam lockfiles is they don’t have checksums. Reproducibility is more a suggestion than a guarantee.

I hope dune eventually solves these issues. To your point about the dune-project file though, I actually think it would be better to treat dune files as the source of truth of project dependencies, due to dune’s modular design. As shown in a previous message in this thread, it’s possible to walk through the dune files and figure out a complete list of deps. These can then be compiled in the dune-project file, downloaded, cached, checksummed, etc.

Finally, there is also the issue of dealing with an esoteric s-expression syntax and a Byzantine set of stanzas to set up dune components. But I think we are committed to that for the foreseeable future, and it does have some benefits.

[lefessan]

I completely agree that newcomers need tools to help them with their first projects in OCaml (and actually, even after…). It’s not because most people here have become experts in opam and dune that we should expect it to be easy for other ones. The 2 tools I have written to help are drom (for project scafolding and more) and opam-bin (for binary packages):

• drom (github.com/ocamlpro/drom) can be used to create an OCaml project from scratch (there are several templates, including the possibility to use remote templates from github), as spin would do, but also to manage the project:

  • It is possible to update the drom.toml and package.toml files to add/remove dependencies, change the description, etc. and automatically update the opam and dune files, and other generated files (Github actions, etc.)
  • drom build will automatically perform many steps, i.e. create a local opam switch, install dependencies, and build the package

• opam-bin (github.com/ocamlpro/opam-bin) can be used to create binary packages for all packages in the opam repository. It also shares files between switches. If you get used to creating local switches in many projects, opam-bin will allow you to create them very fast (by using binary packages previously generated by other switches), and save a lot of space (by using hard links for identical files in the different switches)

I would be very interested to get some feedback from users about these tools, and any contribution, including improving the (existing but need-some-love) documentation. opam-bin was presented at the last OCaml workshop (unfortunately, the talk about drom was not accepted).

[bluddy]

@lefessan

I completely agree that newcomers need tools to help them with their first projects in OCaml (and actually, even after…). It’s not because most people here have become experts in opam and dune that we should expect it to be easy for other ones.

I’m very grateful for both these tools and I’d love for them to be converged on as the main entry point to OCaml. Do you accept contributions readily from other developers?

I think a very important point is that very few people are exclusively OCaml developers, and even those that are, don’t get to tweak dune files very often. If code needs to be made as clear as possible so that other developers can read it because even the same developer is effectively “another developer” after a couple of months (which I think is true from experience), then build tools need to be as intuitive and cross-compatible with other languages as possible concept-wise because they are the aspects most developers rarely touch and so most of us are constant “newbies”.

[lefessan]

Do you accept contributions readily from other developers?

Yes, of course! There are actually many ways to contribute to these tools, for example for drom, the tool itself can be improved, but there are also many ways to extend the templates (for example, they are mostly designed for hosting on Github, so support for Gitlab and other ones would be very welcome…)