Hi all, glad you are OK :slight_smile: Let’s say I want to create and fill with some data a really large Bigarray, like 80% of all available physical memory. Now I want to fork 40 children to process data in this BA. Theoretically, Linux has COW (copy on write) so as long as I don’t write in this B…

Just to let you know, guys: COW works as expected and I did literally nothing to make it work :slight_smile: Thanks for your time and attention :slight_smile:

[image] zbroyar: Will they? :wink: According to COW, I think they will, since it’s mostly an OS thing，

COW will help you share the array data itself, but keep in mind each process will still need its own page tables to index the data. Depending on what else needs to run in the 20% of RAM left, it could be a problem. – Ian

[image] zbroyar: Let’s say I want to create and fill with some data a really large Bigarray, like 80% of all available physical memory. Now I want to fork 40 children to process data in this BA. Theoretically, Linux has COW (copy on write) Even if it is read-only access, I’d be careful about…

Good point, thanks. If somehow you’ll be able to remember at least approach to the solution, please let me know.

The point about the GC modifying data is valid—the GC can update two bits in the headers of OCaml values—however the values in a Bigarray don’t have a header and are not scanned by the GC, so I think you are safe in this case.

Thanks. I also expect that SOME of BA pages will be copied but the majority will be shared.

If somehow you’ll be able to remember at least approach to the solution, please let me know. @otini mentioned you should be safe. For completeness my Python solution was to make use of Internal memory layout of an ndarray : in the parent process concatenate many large tensors’ ndarray.data into…

Thank you for the explanation. From the description I can see why GC was the issue: the architecture of your data is much more complicated than mine. In my case, - as I work with basically plain single chunk of memory, - GC effects must be much less severe. Thanks again.

I think you’re not the only one who got surprised by this. :slight_smile:

Your approach should work. Try with a rather small bigarray at first, just to check things in a simple setting (e.g. can all processes read it, or parts of it, correctly). Then you can try processing your “big data” taking almost all RAM. Do you really need mprotect if you mmap a file which …

Readonly Bigarray

Ecosystem

zbroyar August 15, 2022, 9:15pm 14

Just to let you know, guys: COW works as expected and I did literally nothing to make it work

Thanks for your time and attention

4 Likes

Topic		Replies	Views
Using a bigarray as a shared memory for parallel programming Community parallel , shm	18	2558	December 18, 2019
Memmap a bigarray, then Unix.fork() (parallel initialization of a numerical array) Learning shm , parallel-programming	1	413	March 29, 2023
[ANN] carray.0.0.1 Ecosystem opam , announce , ffi , c , array	16	1827	June 16, 2022
Is it possible to mmap a regular OCaml array to a file? Learning mmap	12	1635	February 7, 2019
What happens in runtime memory when calling `Unix.fork`? Learning memory	16	2303	June 9, 2020

Readonly Bigarray

Related topics