I noticed that the --working-dir option on opam install (2.2.1) still does not do anything after all these years… Is anyone else having this issue, or is it one of those things that a fix is still making its way through the process for? Sorry if I’ve already asked about this… But I have been thinking that if a fix is not forthcoming, then the option should be removed.
To be clear, the issue i’m experiencing is: I want to build my project with uncommitted changes, and no matter what I do,
opam install --working-dir .
results in the following output:
[NOTE] Package forester is already installed (current version is 5.0).
Even if this does get fixed, I would suggest that most users probably want --working-dir to be the default (once it works).
I’m a bit confused at this post as you already know the answer to your question. It is tracked by your own ticket in the opam bugtracker and there is a PR open to fix it with timeframe for the merge and release, all mentioned in the latest comment to date:
This option was introduced after long discussion in ocaml/opam#2156. The default has been discussed at great length over the years since but feel free to open a feature request, we can always change this behaviour if a solution exists that satisfies everyone.
For example, “storing the working-dir state” by using file:// instead of git+file:// by default for the resulting pinned package urls when using opam install and not changing the pin url on subsequent opam install could be a possibility, and would mean that you’d only need to use --working-dir the first time.
When looking at the bigger picture though, i think the UX of opam install <local dir> could be improved, i’m not sure exactly how quite just yet, maybe by decoupling it from the notion of pinning. For example in the example fix above this can be done already by using opam pin add -k path . instead of opam install -w .. With that any subsequent opam upgrade <your package> would pick up the uncommitted changes and install it.
@kit-ty-kate Thank you very much for responding — I apologise that I seem to have forgotten my own ticket on this matter, which is embarrassing. I see from your comment on the original ticket that a fix is implemented and planned for application sometime this year. Thanks also for expanding on the subtleties of the ‘default behaviour’ that I suggested — I see that it is not easy to design a behaviour that makes sense in all cases.
I would like to just comment on one thing. Although I understand that it can be very annoying to hear repeatedly about bugs that you know about and have even prepared patches to fix, I wish to suggest that when there is a highly visible feature that appears to in fact be a no-op from the user’s perspective, if the process of fixing the issue is likely to take upwards of a year, it would be far better to remove the feature from the external interface until the fix is in place — or to take five minutes to add a warning to the output of the command to tell the user that it won’t actually do anything.
I can’t be the only one who is surprised by the fact that for at least a year, maybe more, --working-dir has been a no-op, and although it is a reasonable expectation that the person who reported a bug be able to remember its status (sorry again, that’s my fault), I think it is not a reasonable expectation that an arbitrary new user of opam have to find out that an option documented on the man page actually doesn’t do anything by searching through forum posts and 570 open tickets on the opam repository. I think this is a qualitatively different situation from a bug that triggers in some edge case. It is one of many ‘broken stairs’ that contributes to the current reputation of our ecosystem being much harder to understand than other ones — and although one (very expensive and time consuming) solution to these broken stairs is to fix them, another solution that could be compatible with the current circumstances of mostly unfunded volunteer labour is to remove or document non-working surface area proactively.
I sincerely hope that my suggestions are not taken as criticism of a volunteer project. I am very grateful for everything you do to develop opam and work for this community. I would also like to find (and contribute to) a solution that limits the impact of these ‘broken stairs’.
Note that if you had to live with that default you would quickly see that in the end it may not be what you want. Please note that current default was established after experience of having --working-dir as a default and having suffered the consequences of it. It’s not the first time we have to explain why this is a bad idea perhaps this should be a FAQ.
The reason is this: with working-dir pins your switch updates becomes dependent on brittle global mutable state (read below). Having repo pins by default enables you to mindlessly opam upgrade without fearing of having the upgrade silently picking up unwanted changes while always having a precise idea of what you have in your switch (cue opam info | grep source-hash). That’s generally what you want, hence the default.
Once you start juggling with multiple pins, mutiple branches but still want to be able to mindleslly opam upgrade you will find --working-dir to be a terrible default.
This problem typically goes like that. You go to a working-dir pin repo1 make a small broken edit or checkout an older commit to lookup things or switch to another branch. Now you realise that in fact you need to make a change in another pinned repo repo2 you go there make the change and happy with it you do an opam ugprade at that point either:
You are “lucky”. The install of repo1 and all its dependents fail spectacturaly here and destroys your switch because of your mindless edit in repo1 which you never intended to be picked up.
You are unlucky, it silently installs and older commit without leaving a trace of what you actually have in your switch. If you like obscure debugging sessions that’s what you want.
Basically once you start using mutiple working-dir pins you need to keep in your brain and be mindfull about the checkout state and edits in all of these repositories, possibly over multiple days. Now:
I’m pretty sure your brain has more interesting things to keep track of.
That’s not what repository checkouts are made for: they are made for messing up around.
Thanks very much for this additional explanation… I think I see your point as to why having (a working version of) --working-dir pins by default would be pretty bad.
In case it helps to understand a user’s perspective on this, I think that the issue is not that I want to pin things to my working directory status, but rather that I want an easy way to globally install an executable that is built from uncommitted changes. I understand that these are very different goals, but part of the problem is that opam’s architecture seems to treat them as the same — and the discussion here illustrates how these two things ought to be very different, so I wholeheartedly agree with @kit-ty-kate’s comment that perhaps installation should be decoupled from pinning.
In case it wasn’t clear, the reason I (and probably the majority of other people who are developing executable tools in OCaml) want to just “install the dang current version without committing” is that we are developing a feature at the same time as testing it out by running the program. It is absurd to have to do things like git commit -m 'foo'; opam install . every ten seconds, and then remember to rebase all my foo committs once I’ve managed to get the functionality I want implemented.
I understand that I could call directly the executable in dune’s build directory, but this is a lot of friction (especially when my way of running my executable factors through a bunch of scripts — imagine developing a compiler and then testing it out by building an external project, which has its own infrastructure that calls the compiler).
I would just like to be able to write and run my code without the needless friction caused by identifying pinning with what most users would almost certainly have expected to be as simple as make install.
I understand your use case, but in my opinion it’s not something you should solve through opam. Rather provide an easy way in your repo to extend your environment so that it looks up your built executable by default.
In the past I have used this kind of scripts to solve this problem. I would likely do things differently nowadays (e.g. ensure that the build is up to date) as my build tools are more evolved nowadays and I don’t think using aliases would work in scripts (?). But you get the idea.
I understand your use case, but in my opinion it’s not something you should solve through opam . Rather provide an easy way in your repo to extend your environment so that it looks up your built executable by default.
Thanks! I appreciate your delineating your position clearly and for providing some suggestions for scripts. I think I disagree pretty strongly about whether something equivalent to make install should be a core feature of opam, but I respect that there can be differing opinions — and that your opinion is informed by a lot more experience and understanding of opam than mine.
With that said, it sounds like you are arguing that the --working-dir feature should be removed rather than fixed (as, if this feature were no longer a no-op, it would be precisely a way to do the thing you said shouldn’t be done through opam). As I mentioned, I’m open to this idea—but I did not realise it was on the table. If this were done, most OCaml programmers would probably ultimately switch to using Makefiles that call opam and dune commands but ultimately provide install clauses that physically copy the executable, in the same way that C developers use Makefiles. I think this is potentially OK, though it does introduce another layer of friction to be overcome.
I don’t understand what you mean by make install and what we are disagreeing about: opam is all about install.
I think that if you do an opam install . it should behave as if you did a opam pin add .. And if you do an opam install -w ., it should behave as if you did an opam pin add -k path .. It seems it doesn’t which I find very strange.
However I do think that for your use case, which I ran into more than once, going through opam is just not the right workflow (notably: it’s too slow). Extending your environment to include your build artefact and dynamically rebuilding them if they are not up-to-date when you use them, is.
No, I wouldn’t be arguing about removing something other people use that I don’t use, does not otherwise get into my way or does not provides a bad user experience by default.
What I’m arguing about is --working-dir not being the default on opam install/pin dir if dir is tracked by a VCS. Nothing more. Because we have already been there.
the option works fine when it’s used in opam pin or opam install --deps-only (when not pinned) for example. The part that was buggy was when it’s used in opam install and the package was pinned already (either by opam pin or a previous opam install). It also works fine when used with opam upgrade -w <pkg> or opam install -w <pkg> (<pkg> instead of .).
There are plenty of cases where it works so claiming it is a no-op because your use-case doesn’t work is a bit strong. I’ve seen plenty of people use that option successfully despite this bug and removing it doesn’t seem fair to them.
However you’re more than welcome to add some documentation about this bug to the FAQ: opam/doc/pages/FAQ.md at master · ocaml/opam · GitHub (which is displayed on opam.ocaml.org/doc/FAQ.html)
As noted in the ticket, with the new opam release schedule, it’s only 6 months. I don’t think it’s that long of a wait.
This is very interesting and helpful, thanks! Does this mean that I could have achieved my desired behaviour all along by simply typing opam install -w <mypackagename> after I had pinned <mypackagename> to whatever directory? Or is this non-equivalent?
Thanks for the invitation, I will indeed do so once I better understand the point above.
Understood, but I meant “upwards of a year from when I reported it” — which has been 530 days. I’m indeed very glad that the release is coming as soon as April.
Perhaps I misunderstand something, but couldn’t you just run dune install to do this? I suppose this may be modifying the opam switch without opam’s involvement or knowledge, which may be problematic.
This makes me think it might be nice to have a dune shell command which builds @install and runs $SHELL with PATH and OCAMLPATH extended with the staged install tree in _build.
Oh wow, I did not know that dune install exists, thank you very much for pointing me to it! I will give that a try
EDIT: I can confirm that dune install does exactly what I want. Huge thanks to @greedy for pointing this out.
With this in hand, I can now address the apparent confusion herein the quoted text: when I spoke of the difference between what make install does in a typical C project, vs. what opam install does, I can now be seen to have been speaking precisely of the difference between the functionality of opam install . and dune install in a typical OCaml project that combines dune and opam. I hope that clarifies any possible confusions.
Now that I understand how to use dune install, I am becoming a lot more sympathetic to the viewpoint of @dbuenzli that this is not the kind of problem that opam should be solving — given that there does indeed appear to be a reasonable way to achieve it without invoking opam.
Oh wow. I didn’t realize that dune exec populated the environment that way. I treated it just as a way to run something built by dune without manually finding it in the _build directory. Thanks for the enlightenment.
