Library to schedule Unix processes?

I’m looking for library recommendations to do the following:

  • run Unix processes, redirect and collect their output and status
  • schedule these processes: some depend on others and thus must run after those that they depend on
  • run processes in parallel when possible, up to a given upper limit

There are several libraries to run Unix processes but I haven’t seen the part where their dependencies can be expressed. Dune’s fiber sub-system is surely capable of this but I would prefer a dune package when possible. Roughly, this would be equivalent to what can be expressed in a Makefile and using make -p.

1 Like

I know you want a library, but this really sounds like a job for systemd which comes with Linux out of the box with all these features.

The application is to trigger all kinds of ffmpeg video transformation jobs and finally to assemble the resulting video. Each ffmpeg job takes minutes to complete. Obviously I could try to implement this as a Makefile that expresses these dependencies. But at the same time, ffmpeg has complicated command line arguments which I would prefer to assemble in OCaml. I’m currently doing this in Python (without clever scheduling) but would like to do a re-implementation in OCaml. The compromise would be a mixture: implement a driver in OCaml for a single job and task Make with doing the scheduling. I don’t see how systemd would fit in here.

I suppose if you don’t need scheduling in the sense of ‘run this job at a specific time’ then systemd isn’t needed, yes. Generating a Makefile or a ninja file in OCaml seems like a good approach to me. That’s how ReScript (fka BuckleScript) build system works.

1 Like

That’s similar to what I expect in the neighbor post.

Abstractly, I do need a dependency layer(framework) where I can specify their relation and priority, and there is a low-level running engine whether it’s UNIX processes as you expect or just function applications.

1 Like

I think you should take a look on minifiber which seems to be what you want :+1:. Such small code is widely used on my project when I need true parallelism - of course, I use fork inside…

The code comes from a look on the dune's codebase. It’s not released mostly because I use it only for tests.

4 Likes

:white_check_mark: This is the solution, I think. Thanks for writing this.

This is not exactly what you’re looking for, but if you have things that needs to be setup before the “main” job you could have a look at this: GitHub - anmonteiro/archi: An OCaml library for managing the lifecycle of stateful components and their dependencies

2 Likes