open sig ... end does not make sense in general, you could not have an arbitrary module type in there. For example you couldn’t introduce new types in there and then reference them afterward.
For your specific case, you can go the other way :
include (sig
type t
val x : t
val y : t
end with type t := SomethingVeryLong.t)
You can even use destructive substitutions at toplevel directly:
(* file.mli *)
type t := SomethingVeryLong.t
val x : t
val y : t
On the original error:
Unlike include, which take either a structure or a signature depending on the context, open always takes a structure (i.e. a module, not a module type) wherever it appears, so open struct type t = ... end would be required. In addition, some contexts allow arbitrary structures, but some others (typically, in signatures) only allow a subset of module expressions (only paths and functor applications). So even with open struct type t = ... end you would get a syntax error.