Library to schedule Unix processes?

Community
#1

I’m looking for library recommendations to do the following:

  • run Unix processes, redirect and collect their output and status
  • schedule these processes: some depend on others and thus must run after those that they depend on
  • run processes in parallel when possible, up to a given upper limit

There are several libraries to run Unix processes but I haven’t seen the part where their dependencies can be expressed. Dune’s fiber sub-system is surely capable of this but I would prefer a dune package when possible. Roughly, this would be equivalent to what can be expressed in a Makefile and using make -p.

1 Like
#2

I know you want a library, but this really sounds like a job for systemd which comes with Linux out of the box with all these features.

#3

The application is to trigger all kinds of ffmpeg video transformation jobs and finally to assemble the resulting video. Each ffmpeg job takes minutes to complete. Obviously I could try to implement this as a Makefile that expresses these dependencies. But at the same time, ffmpeg has complicated command line arguments which I would prefer to assemble in OCaml. I’m currently doing this in Python (without clever scheduling) but would like to do a re-implementation in OCaml. The compromise would be a mixture: implement a driver in OCaml for a single job and task Make with doing the scheduling. I don’t see how systemd would fit in here.

#4

I suppose if you don’t need scheduling in the sense of ‘run this job at a specific time’ then systemd isn’t needed, yes. Generating a Makefile or a ninja file in OCaml seems like a good approach to me. That’s how ReScript (fka BuckleScript) build system works.

1 Like
#5

That’s similar to what I expect in the neighbor post.

Abstractly, I do need a dependency layer(framework) where I can specify their relation and priority, and there is a low-level running engine whether it’s UNIX processes as you expect or just function applications.

1 Like
#6

I think you should take a look on minifiber which seems to be what you want :+1:. Such small code is widely used on my project when I need true parallelism - of course, I use fork inside…

The code comes from a look on the dune's codebase. It’s not released mostly because I use it only for tests.

4 Likes
#7

:white_check_mark: This is the solution, I think. Thanks for writing this.

#8

This is not exactly what you’re looking for, but if you have things that needs to be setup before the “main” job you could have a look at this: GitHub - anmonteiro/archi: An OCaml library for managing the lifecycle of stateful components and their dependencies

2 Likes