I look at this as someone who spent a good bit of time debugging segfaults when trying to upgrade the compiler, which turned out to be caused by uses of Obj.magic in a third-party library that used to be "ok" but the compiler had just gotten smart enough to catch the lie. From this point of view, I'm very wary of taking dependencies on libraries that use Obj.magic, and in cases where there are compelling reasons to use such libraries despite that, I find myself looking at every occurrence and trying to reconstruct the safety argument. Maybe I'm paranoid, but I figure that it's a position that might be helpful for library authors to hear. That is, I think it would be good if whenever Obj.magic was used, the library author thought that every user of the library was going to read the code and reconstruct the safety argument.
With that said, I wonder if there would be significant value in some mechanism akin to Obj.magic that only allowed casting between types that the type checker can prove have the same representation. I'm not sure how feasible such a feature would be, but handling the particular case of magicking a phantom type does not seem too far out (although not expressible as a type of a value in the current system unless I'm confused).