In bytecode, all versions should be more or less equivalent (closures are never optimised away).
In native code, the first version allocates a closure everytime you call my_public_function, while the second will have its closure statically allocated if it doesn’t depend on other non-static variables.
The performance of my_private_function itself could be impacted too: the first version has one less parameter so it should be slightly faster to call, but that is offset by an optimisation on the second version: the extra parameter passed to each function that stores its closure can be removed for functions that do not need it, so in the end they will both take two arguments. The first version also requires an extra load from the closure to get some_argument (the load occurs once with Flambda, at each use otherwise).
Finally, a non-trivial question is how all of this interacts with the compiler’s optimisations. The function my_public_function is rather small, so it could be considered for inlining, but without Flambda only the third version (with my_private_function out of the body of my_public_function) can be inlined. With Flambda, the second and third version are equivalent, and the first one could be actually easier to optimise, in particular if you call my_public_function with a constant or statically-allocated argument.