I raised an issue on GitHub a little while ago about not being able to use MerlinLocate from inside installed libraries (I can jump to them from my project but then I hit a dead end from there). @trefis advised moving the discussion here.
From what I understand, to jump to definition Merlin needs .merlin files, which Dune auto-generates for your project but are not available for libraries installed through Opam. Excluding them seems to be a deliberate choice, which confuses me, since jumping through definitions and implementations of libraries is important to understand and work with them. My questions are:
Am I correct that this is a design choice as opposed to a technical challenge, and if so, can this choice be revisited?
If .merlin files will not be released, what is the best way to get jump to definition working locally? Should I create .merlin files in the opam directory tree (and if so what should they look like and where should they be placed)? Or should I symlink to the installed libraries from my project – or copy them to my project – so that Dune is able to generate Merlin files for them?
Is this something that the Duniverse project will resolve?
In the general case, the source code of an installed library is not available. So there is no definition to jump to, except maybe the declaration in an MLI file but even that is not required for the compiler to link the library. Are you assuming a different setup where the library source code is available?
Yes, I am assuming source is available, though that’s a fair point. The source does seem to be available for libraries like core and async, and I imagine for several others as well. So I suppose my questions should be qualified with that assumption.
Note that for topkg packages you can set TOPKG_CONF_DEBUGGER_SUPPORT=true in your environment before opam installing topkg packages. This should install sources for these packages.
Like I said, in the general case a package in Opam does not install its source code and therefore you can’t jump to a definition. However, in a project built with Dune or Jbuilder you can unpack the code of selected libraries in your project, where they will be built and visible for Merlin. See Dune Scopes.
Can you elaborate on how to get this to work? What my directory structure needs to be / what files I need (dune, dune-project, core.opam, etc. what needs to be created and placed where)? I’ve tried several iterations but none of them seem to be respected and the documentation on this is sparse.
I am working an a small webapp that uses Opium, Moustache and Ezjsonm as libraries. When I unpack them in the top-level of my project (opam source moustache, …) and build the project as before, now the unpacked sources will be used and I can jump to any code inside them using Merlin. My own code, dune files and so on are completely unchanged.
Does this work for libraries that were not themselves built with dune? When I try opam source-ing core, core_kernel, base, etc. and their dependencies into a vendor/ directory, dune builds them fine but MerlinLocate seems to ignore them. I suspect (though am not sure) this is because these libraries are missing dune files, as they have jbuild files instead. If this is the issue, is there a way to reliably translate the jbuild file into the needed dune file (they follow similar formats but I don’t think they are 1:1)?
@lindig Do you mind showing a minimal working setup with core? I’m retrying this again (I never was able to get it working) and dune build starts complaining about conflicts between the libraries in my vendor/ directory and libraries in the opam directory. I started moving more and more of these conflicts into vendor/ but then ran into a problem with ppx_compare.runtime-lib, a base_bigstring dependency that still wasn’t being recognized even after I added ppx_compare to vendor/. Fwiw I also added (vendored_dirs vendor) to a dune file in the root of my dune workspace.
@diml Do you know if there’s a conflict arising here between vendoring and the usage of (preprocess (pps ppx_jane)) in the dune file for base_bigstring that’s causing it to look in the opam directory instead of the vendored directory?
I know it’s from a while ago but did you have any joy with this @ddickstein?
I find it a little odd there appears to be no straightforward way to move around installed libraries in the cases where the source is available, and also that nobody seems to require this functionality, which makes me think I’m doing something wrong.
Jumping Mtime only brings you to the .mli file, since mtime doesn’t install its sources (which is quite useful in itself for browsing documentation).
In order to jump from libraries to other libraries, libraries would need to install .merlin files. I’m not sure if that’s a good idea, but I’ve opened an issue here: https://github.com/ocaml/dune/issues/3276
Thank you for opening that issue! It is indeed the final scenario I’m after. Just as a concrete example I came across today, if I have the below in a file in my Dune project:
(* example.ml *)
let _ = Async.with_timeout
Then I can jump to Async.with_timeout ok. However it turns out this is defined as:
(* async_unix.ml *)
let with_timeout = Clock.with_timeout
So it would be quite handy to be able to jump directly to Clock.with_timeout, though it’s not currently possible.
I think I’ve found a way to achieve this so thought I’d mention it here, it also required a bit of manual work so further thought it’d be good to post in case there’s a better way. Steps I took were:
Run dune build followed by opam monorepo lock && opam monorepo pull to pull any dependencies into the local duniverse folder.
After once again running dune build, it should be possible to jump around within dependencies, for example if I add the below to bin/main.ml, then I can jump to Async.with_timeout, and from there to Clock.with_timeout (and onwards if I wish).
open Async
let () =
let _ = Async.with_timeout in
print_endline "Hello, World!"
It’s not too painful but if there are any steps which could be streamlined I’d grateful to hear how! Particularly I’m wondering if there’s a better approach than manually adding the dependencies to the dune-project package stanza - is there a way to generate this section automatically somehow, maybe from the libraries section in the dune files?
For those who might still visit this thread in 2021: Here are some workarounds to the problem of not being navigate via merlin/ocaml-lsp in the installed libraries / seeing all kinds of merlin errors in opam switch installed ocaml library code.
Use dune vendoring If you’re particularly interested in the source code of a particular library and how it interoperates with your own code, you can vendor the library code. (Look up dune vendoring for how to do it and limitations). Once you have vendored the library code in your opam switch you can navigate to your hearts content. You will be able to see both ml and mli’s as the source code will be in your vendor directory. There is some mention on vendoring in the above discussion – my experience is that (temporary) vendoring can be quite useful in solving merlin/ocaml-lsp issues. Its also very easy to do
Use the opam monorepo plugin People coming from rust may expect to be able to totally navigate in the codebase everywhere. (Dependency of a dependency etc). You could vendor every library used by your code but that quickly becomes exhausting. The closest thing that exists in ocaml for this is the GitHub - ocamllabs/opam-monorepo: Duniverse is the spice of OCaml build life plugin. That plugin will (dune) vendor every transitive dependency into the duniverse folder automatically. Once you do that you should be able to again navigate using merlin/ocaml-lsp in such a way that you wont hit any dead ends while navigating. One of the limitations here is that every dependency of yours should be build-able by dune. (This isn’t a problem in most cases because there is a custom duniverse overlay repository that provides dune buildable versions of most popular non-dune repos – see the opam monorepo documentation)
There are some caveats and limitations but I urge you read the dune readthedocs and ocaml monorepo plugin details for more information. Also, if I have made any technical errors in my above recommendations, please let me know too.
The workarounds are impressive, but I think we should be able to solve this inside dune and make it possible to jump through installed sources once and for all.
When merlin queries dune for the .merlin config, dune should be able to generation information that makes it possible to navigate through installed projects. I just need to confirm this is possible with Ulysse - the main developer of merlin/dune integration.
BTW I notice that @brocaml already had suggested the opam monorepo approach in his comment above a few months ago – props for that. Though his example is about a fresh project. I want to indicate that you can do this for any exisiting project also.
So lets say you want to get the full ocaml-lsp experience while looking at the dream source code and its libraries. (I chose dream just as a random example that works).
$ mkdir dream-switch
$ cd dream-switch
$ opam switch create . 4.12.0
# Adds the ability to dune build ocaml projects that are dream dependencies
# but dont use `dune` in their default repos
$ opam repository add dune-universe git+https://github.com/dune-universe/opam-overlays.git
$ opam update
$ git clone https://github.com/aantron/dream.git
$ cd dream
$ opam monorepo lock
$ opam monorepo pull
$ dune build -p dream
# Now you can install ocaml-lsp
$ opam install ocaml-lsp-server