The short version is that you need an explicit universal annotation when defining a recursive function over GADTs.
And the syntax
let rec map: type size.
f:('a -> 'b) -> ('a, size) t -> ('b, size) t = ...
makes the type size both locally abstract and universally quantified.
More precisely, those explicit universal quantification annotations are needed when defining recursive functions which are used inside their own body with different types.
This happens with your map function since it is defined as working on ('a,n) t and then called inside its definition on a value of type ('a, n-1) t.