This framework looks yummy.
It’s Java, so creating the bindings can be painful. Another approach would be to use the normal FFI as to any C library to the Rust alternative - Native Spark.
Because Rust allows exporting the functions with C ABI it should be fairly easy. You can read more in the blogpost “FastSpark: A New Fast Native Implementation of Spark from Scratch”.
And it is obviously faster than Java version:
Rust implementation CPU utilization during the execution (8 cores)
Spark CPU utilization during the execution (8 cores)
They have a Gitter channel if you are interested.