In select, you’d like b to have bytes t type in the Bytes case and string t type in the String case. The two types are incompatible, so this can’t work without a bit of magic, in this case GADTs.
My suggestion is to replace your type definitions with:
type _ data = Bytes : bytes -> bytes data | String : string -> string data
type 'a t = { data : 'a data; size : int; mutable pos : int; mutable len : int }
You’ll also need some type annotations on select (let select (type a) (b : a t) =), and maybe a few more things will need to be adapted, but this should get you at least a bit further.