Binary tree error

I can only agree with the compiler. Try replacing (f(f)) with

(fun x -> f (f x))