What are maintained vs non-maintained packages? Also, what is opam-overlays?
I’ll point out that for pinned packages, in other words, packages where the user explicitly selected the source, we can indeed omit the build instructions from the lock directory.
However, even for such packages, we still have a strong incentive to copy the effective list of dependencies. This allows us to construct the package build graph without downloading anything thus allowing us to optimally execute build plans for commands such as dune build foo.install. We’d like to immediately start downloading all the dependencies for foo concurrently rather than waiting for sources to finish downloading before constructing the entire build graph.
EDIT: I suppose if the locking stop downloaded all the sources, we could just copy the build scripts only for packages where the script differs from what is defined in the sources. Doesn’t sound like it’s worth the hassle though.
On running dune init proj hello_world for the first time ever, it took ~19s for dune to complete its work. On subsequent runs, it’s immediate. I assume dune is downloading and building necessary tools.
Can we print something to let users know that dune is actively working?
Maintained packages have an opam file in the package (with up-to-date build/depends fields, but not necessarily upper bounds), which could be checked when submitting to opam-repository. As mentioned previously, this is desirable regardless of dune package management, and I would assume most it’s already the case for most packages.
In the end, it seems to be a trade-off. The current design keeps dune pkg lock simple, and makes dune build (somewhat) faster. Is that really worth having a bloated dune.lock directory in every project though? (think about running git log -p dune.lock after a year of active development)
Okay, but I’m still not confident I understand your suggestion. I will try to rephrase it, and you tell me if I’ve understood.
To recap, I’ll just focus on the build instructions for now - as mentioned earlier, storing deps has other uses. The only reason that we save the build instructions is to make sure that we run package builds exactly as defined in our package database (opam-repository). With the exception of pinned packages, the package database is the ultimate source of truth of this metadata. It follows that if we omit this metadata, we must faithfully regenerate it through some other means. For example, there’s these two possibilities:
We can detect that a package is built by dune by looking at the build instructions. If it’s the familiar dune template, we can just say (build dune) instead of copying the build action.
When a package is submitted to opam-repository, a separate lint step could perhaps record that the equivalence between the opam file being submitted and the one present in the repository. After solving in dune, we can remove the build instructions for all packages where the opam file is claimed to be equivalent to what’s present in the package sources.
My opinion on these two suggestions: We should do 1. indeed. For 2., I would like to follow the lead of the opam-repository devs.
You might not like to hear this, but I consider that even one of the two benefits you’ve pointed out as more valuable than keeping git log -p dune.lock concise. With that being said, there’s explicit versioning the lock directory for a reason. So we could reconsider in the future if enough users ask for this and we’ve addressed the more serious issues.
Only the subset of the database that is used for the build is copied. The reasoning is that this omits a rather expensive database download step (think how slow opam update is) when trying to build the workspace in an another environment (CI for example)
Not that I care much but honestly this feels rather untidy to me.
I don’t find the argument very compelling, especially since this is only for the initial step. First you can likely cache the opam repo at the machine level (in the sense across projects) and I think we should rather focus on making opam update faster rather than asking everyone to start commit cruft to their repos (at that rate we could ask people to commit their reproducible builds to repos so that it builds faster).
We do have an alternative workflow where nothing but the opam repository’s git hash is saved. It needs a little more polish, but it’s even tidier than storing names, version numbers and checksum sources. Once running opam update isn’t such a penalty, we can reconsider the defaults.
Just wanted to add my perspective on this since I faced a similar dilemma when building onix.
I aimed to generate a cache-efficient, standalone lock context for building with nix that would not rely on the opam repository. In the end I gave up because opam files can get really complicated and require partial evaluation of host variables to obtain the build instructions, among other things.
Additionally, from what I remember, to get the full build instructions you need to download all package sources during the lock step. This is because some packages have extra config files that require variable resolution. To make matters worse, IIRC, the config files may be generated by the build process itself…
In the end, I decided to cache the opam repo’s hash and use it as the canonical source of truth for build instructions (except for pins, of course). Caching with nix is trivial and I don’t think that waiting a bit to fetch the latest opam repo when locking is a big deal.
Finally, there’s something I wanted to ask: I noticed that the lock context generated by dune does not seem to distinguish runtime and build-time dependencies. Are there any plans to change this in the future? What was the rationale behind this decision?
A new version of the vscode-ocaml-platform was just released which fixes a few issues with ocamllsp. You’ll probably have to update your install of the Dune Developer Preview (just rerun the command on this page). You’ll need to configure a custom sandbox for vscode by putting this in your settings.json file as otherwise the plugin assumes you’re using opam to launch ocamllsp:
My guess is the extension uses whichever dune is first in your $PATH, so just make sure that /home/nojaf/.dune/bin appears earlier that /home/nojaf/.opam/5.2.0/bin in $PATH.