AFAIK the main issue is that the compiler and compiler artifacts are not relocatable; they contain absolute paths (e.g. the path to ocamlrun for bytecode objects, or, in the case of the compiler, the path to its stdlib).
That’s why moving a local switch will not work properly, and there’s no easy way to setup a cache. Eventually this is something that we would like to improve on the compiler side, so that opam can provide by default a binary cache that works well…