Is eval $(opam env --switch={switch} --set-switch) equivalent to opam switch set switch?

brando90 · December 14, 2022, 1:46am

I need to change opam envs within python due to my applicaiton (no way around this 100%).

Usually I do:

eval $(opam env --switch={switch} --set-switch)

but this gives an issue (see end).

Thus, going to try:

opam switch set {switch}

are these truly equivalent?

For context error:

Traceback (most recent call last):
  File "/lfs/ampere4/0/brando9/iit-term-synthesis/iit-term-synthesis-src/data_pkg/data_gen.py", line 510, in <module>
    main()
  File "/lfs/ampere4/0/brando9/iit-term-synthesis/iit-term-synthesis-src/data_pkg/data_gen.py", line 497, in main
    asyncio.run(create_dataset(path_2_save_new_dataset_all_splits=args.path_to_save_new_dataset,
  File "/dfs/scratch0/brando9/anaconda/envs/iit_synthesis/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/dfs/scratch0/brando9/anaconda/envs/iit_synthesis/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/lfs/ampere4/0/brando9/iit-term-synthesis/iit-term-synthesis-src/data_pkg/data_gen.py", line 437, in create_dataset
    coq_proj_data: DataCoqProj = await get_coq_proj_data(coq_proj, split)
  File "/lfs/ampere4/0/brando9/iit-term-synthesis/iit-term-synthesis-src/data_pkg/data_gen.py", line 194, in get_coq_proj_data
    path2filenames_raw: list[str] = strace_build_coq_project_and_get_filenames(coq_proj)
  File "/afs/cs.stanford.edu/u/brando9/pycoq/pycoq/opam.py", line 706, in strace_build_coq_project_and_get_filenames
    activate_opam_switch(switch)
  File "/afs/cs.stanford.edu/u/brando9/pycoq/pycoq/opam.py", line 892, in activate_opam_switch
    raise e
  File "/afs/cs.stanford.edu/u/brando9/pycoq/pycoq/opam.py", line 886, in activate_opam_switch
    res = subprocess.run(command.split(), check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  File "/dfs/scratch0/brando9/anaconda/envs/iit_synthesis/lib/python3.9/subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/dfs/scratch0/brando9/anaconda/envs/iit_synthesis/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/dfs/scratch0/brando9/anaconda/envs/iit_synthesis/lib/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'eval'

I think it has something to do with calling subprocesses from within python don’t fully understand, refs:

I think an alternative would be to get the output of $(…) then run that with subprocess (without bash’s eval I think).

Kakadu · December 14, 2022, 11:00am

You could easily make an experiment for two approaches:

Change the switch in one tab
Check the switch there (it should be changed)
Open a new terminal tab. Check switch again

One of the commands changes switch globally, another one only for the current environment.

emillon · December 14, 2022, 1:54pm

To execute a command in a switch from a python program, prefer opam exec --switch {switch} -- command args - it will exec command args without you having to mess with shell eval etc.

brando90 · December 14, 2022, 7:22pm

I still want to know why opam switch set SWITCH and the eval $(opam env --switch={switch} --set-switch) methods exist and the difference.

brando90 · December 15, 2022, 2:17am

so what do I put as args in my context? @emillon

brando90 · December 15, 2022, 2:20am

would args be args := opam env --switch={switch} --set-switch?

brando90 · December 15, 2022, 2:41am

this doesn’t work:

def run_eval_opam_env(switch: str,
                      ):
    """
Tries to run through opam exec command arg https://opam.ocaml.org/doc/man/opam-exec.html.
eval $(opam env --switch=switch --set-switch)

    note:
        - command substituion: replace the command ran here $(cmd) with its output (string).
        It turns out, $() is called a command substitution. The command in between $() or backticks (“) is run and the
        output replaces $().
    ref:
        - command sub: https://blog.wplauncher.com/what-is-in-linux/#:~:text=Example%20of%20command%20substitution%20using%20%24()%20in%20Linux%3A&text=Again%2C%20%24()%20is%20a%20command,another%20context%E2%80%9D%20(Source).
        - eval $(opam env): https://stackoverflow.com/questions/30155960/what-is-the-use-of-eval-opam-config-env-or-eval-opam-env-and-their-differen?noredirect=1&lq=1
        - there is a way with opam exec: https://discuss.ocaml.org/t/is-eval-opam-env-switch-switch-set-switch-equivalent-to-opam-switch-set-switch/10957/4
    """
    logging.info(f'{run_eval_opam_env=}')
    # - get cmd sub output
    try:
        cmd_sub_output: str = subprocess.check_output(f'opam env --switch={switch} --set-switch'.split())
    except Exception as e:
        logging.critical(f'Error: {e=}')
        raise e
    # - run command to eval
    # command: str = f'eval {cmd_sub_output}'  # doesn't seem possible https://stackoverflow.com/questions/53950225/python-check-output-call-to-eval-with-arguments-fails
    command: list[str] = f'opam exec --switch {switch}'.split()
    # command: list[str] = command + cmd_sub_output.split()
    command: list[str] = command + [cmd_sub_output]
    logging.info(f"-> {command=}")
    try:
        # res = subprocess.run(command.split(), check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        res = subprocess.run(command, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        logging.info(f'{res.stdout.decode()=}')
        logging.info(f'{res.stderr.decode()=}')
    except Exception as e:
        logging.critical(f'Error: {e=}')
        raise e

current attempts linux - How does one run `eval $(opam env)` when the command substitution is changes to the environment - all ran from within **python**? - Stack Overflow

emillon · December 15, 2022, 10:35am

These are the arguments you want to pass to your command. Say, if you want to run dune build under a switch named 4.14.0, you’d call opam exec -- --switch 4.14.0 dune build.

In a nutshell: running a command under an opam switch means running a command with certain environment variables set. There are 2 techniques to do that:

for non-interactive use (in scripts, etc), wrap the commands you want to run under opam exec --switch switch_name, which will set the environment variables for the underlying command
for interactive use (development shells), there’s the opam env command that outputs a bit of shell that needs to be evaluated. That’s why the full command to run in the shell is eval $(opam env). But it that case, which switch should be used? Opam maintains that as a piece of global data that you can change using opam switch set. So opam switch set changes what the following opam env command will return.

For completeness, be aware that there’s a notion of local switches in opam that are stored in specific directories and get priority when computing the “active switch” (either in the shell or in opam exec when --switch is not passed) but it does not seem that you’re using these in your setup.

brando90 · February 17, 2023, 7:47pm

general question on distinction btw eval $(opam env) vs opam switch set: If I want to switch opam switch, should I use eval $(opam env --switch=<switch_name> --set-switch) or opam switch set <switch_name>? why are there two ways to do this? - #7 by brando90

brando90 · February 20, 2023, 7:14pm

doing eval $(opam env --switch={switch} --set-switch) in python question: What is the difference between eval $(opam env --switch={switch} --set-switch) and opam switch set SWITCH? - Stack Overflow

Frederic_Loyer · February 20, 2023, 8:16pm

The idea behind opam env is to feed your shell with variable change sentences. And then, afterward, from the same shell, since the PATH is changed, you can trigger the right ocaml programs.

With str = subprocess.check_output(f'opam env --switch={switch} --set-switch'.split()), you have a set of sentences you should split and execute like a shell would do. But the execution must be done in the Python process. If you start the sentences in a subprocess, the variable change will only affect this process.

Changing the current switch with opam switch set $switch will not change your environment (the PATH will still point to the previous ocaml). It will only change the following answer of opam env. It is handy with a PROMPT_COMMAND which call automatically an eval opam env command

brando90 · February 20, 2023, 8:37pm

I guess it’s confusing what “a change of environment means for opam” but what I did do is change it with

        command: str = f'opam switch set {switch}'
        result = subprocess.run(command.split(), check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

and then after that is done, I have another python function (running in python’s main) that calls opam switch to check the switch was made right. e.g.

        result = subprocess.run('opam switch'.split(), capture_output=True, text=True)

and it all looked right. The later subprocess checking the earlier switch was done prints what I’d expect. Is there something missing?

Frederic_Loyer · February 20, 2023, 8:52pm

Your environment is what is displayed by the env command from a shell. It contains (normally) multiple variables useful for Ocaml. The main one is the PATH which must point to the right ocaml program set.

To change it in a shell, you have the eval $(opam env) command, but it is a Shell command.

If you want to do the same in Python, you have to rewrite an equivalent of eval in Python. (Hopefully, opam env will only output a list of VAR=value lines, not to difficult to parse. Given a VAR and value strings, just type os.environ[VAR]=value).

Note : you should be precise about your intents. Do you want Python to launch the right ocaml interpreter from a subcommand ?

brando90 · February 20, 2023, 9:05pm

great question. My intent is to build individual ocaml (coq) projs/pkgs which I have git cloned so I have the source code. My intent is to run make in the location of the ocaml (coq) project/pkg.

More precisely due to me using this set up I actually want to build it from python ny running the following script:

#!/usr/bin/env bash
eval "OPAMSWITCH='coq-8.10'; export OPAMSWITCH;
OPAM_SWITCH_PREFIX='/lfs/ampere1/0/brando9/.opam/coq-8.10'; export OPAM_SWITCH_PREFIX;
CAML_LD_LIBRARY_PATH='/lfs/ampere1/0/brando9/.opam/coq-8.10/lib/stublibs:/lfs/ampere1/0/brando9/.opam/coq-8.10/lib/ocaml/stublibs:/lfs/ampere1/0/brando9/.opam/coq-8.10/lib/ocaml'; export CAML_LD_LIBRARY_PATH;
OCAML_TOPLEVEL_PATH='/lfs/ampere1/0/brando9/.opam/coq-8.10/lib/toplevel'; export OCAML_TOPLEVEL_PATH;
MANPATH=':/usr/kerberos/man:/usr/share/man:/usr/local/man:/usr/man:/opt/puppetlabs/puppet/share/man:/lfs/ampere1/0/brando9/.opam/coq-8.10/man'; export MANPATH;
PATH='/lfs/ampere1/0/brando9/.opam/coq-8.10/bin:/lfs/ampere1/0/brando9/miniconda/envs/metalearning_gpu/bin:/lfs/ampere1/0/brando9/miniconda/condabin:/lfs/ampere1/0/brando9/miniconda/bin:/lfs/ampere1/0/brando9/.local/bin:/lfs/ampere1/0/brando9/.ruby-build/bin:/lfs/ampere1/0/brando9/.rbenv/shims:/lfs/ampere1/0/brando9/.rbenv/bin:/usr/local/cuda-11.7/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/afs/cs/software/sbin:/afs/cs/software/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/puppetlabs/bin'; export PATH;"
make

I see I might have to parse it and do your os.env[var] = new_var suggestion. Idk if it’s needed but I wish I could have replaced the final make with an opam pin . or something like that. Unfortunately, everything has to be done through python.

Frederic_Loyer · February 20, 2023, 9:17pm

The relationship between your bash script and python is not clear.

If you want to have your Python script do what the bash script do, you will have to change the os.environ from Python.

If not (python and bash are complementary). Calling opam env from the bash script will be the simplier.

Nota: the separator is a ; and you have export command to be ignored.

brando90 · February 20, 2023, 9:38pm

apologies, I don´t understand what this means. Does it mean if I did ‘bash make.sh’ from the terminal as I previously wrote won’t actually modify the bash env as expected?

brando90 · February 20, 2023, 9:39pm

happy to clarify. I run python -u main.py once from bash and never run bash again. I then switch dynamically which opam and ocaml project/pkg I am building from within python. Or at least that is what I need to work.

Frederic_Loyer · February 20, 2023, 9:44pm

I mean opam env outputs something like this:

OPAM_SWITCH_PREFIX='/home/loyer/.opam/default'; export OPAM_SWITCH_PREFIX;
CAML_LD_LIBRARY_PATH='/home/loyer/.opam/default/lib/stublibs:/usr/local/lib/ocaml/4.11.1/stublibs:/usr/lib/ocaml/stublibs'; export CAML_LD_LIBRARY_PATH;
OCAML_TOPLEVEL_PATH='/home/loyer/.opam/default/lib/toplevel'; export OCAML_TOPLEVEL_PATH;
PKG_CONFIG_PATH='/home/loyer/.opam/default/lib/pkgconfig:/home/loyer/.opam/default/lib/pkgconfig'; export PKG_CONFIG_PATH;
MANPATH=':/home/loyer/.opam/default/man:/home/loyer/.opam/default/man'; export MANPATH;
PATH='/home/loyer/.opam/default/bin:/home/loyer/.opam/default/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games'; export PATH;

Only the VAR=value part should be process, the ; and the remaining export command must be ignored.

yawaramin · February 20, 2023, 9:46pm

I actually had been wondering why opam doesn’t just export the variable definitions directly, instead of defining and exporting separately in two steps.

brando90 · February 20, 2023, 9:47pm

@yawaramin great question! Might it be that if it did that it would run it in it’s own process and the parent process (called it main process) doesn’t get ther right env vars changed? but if you ran eval from your main process actually sets the env vars right?

That is my guess. Thoughts?

Topic		Replies	Views
If I want to switch opam switch, should I use eval $(opam env --switch=<switch_name> --set-switch) or opam switch set <switch_name>? why are there two ways to do this? Learning opam	6	608	February 17, 2023
How are opam switches supposed to be used? Learning	17	1928	January 7, 2023
Stop opam blurb about environment sync Ecosystem opam , script	4	712	October 24, 2020
What does eval $(opam env) do, does it activate a opam environment? Learning	11	3801	June 16, 2022
Local opam switch and emacs Learning opam , emacs	4	784	May 29, 2023

Is eval $(opam env --switch={switch} --set-switch) equivalent to opam switch set switch?

Related topics