Memory usage improvements

[qkaiser]

This branch contains multiple optimizations around unblob's memory consumption, specifically RSS usage when used on Linux.

unblob makes heavy use of mmap'ed files through the File class but always defaults to the MADV_NORMAL policy. This policy applies moderate read-ahead and is conservative about reclaiming pages, which can cause RSS to grow to the full size of large input files.

While this is generally okay on analysts machines, this can become problematic on constrained environment where parallel extractions are competing for memory to the risk of being killed by OOM.

We address this in two ways:

• use MADV_SEQUENTIAL when scanning file with HyperScan. Since we expect pages to be read in sequential order, pages in the given range can be aggressively read ahead, and may be freed soon after they are accessed.
• explicitly set MADV_DONTNEED on pages that were just read through File.read(), it's a generic approach so that every unblob component reading from a File can take advantage of it.

On top of that, we noticed a syscall optimization around File.size(), avoiding millions of fstat syscalls per run on complex firmware.

On a 7GB gzip multi-chunk stream containing a 14GB extfs filesystem, the initial consumption observed by memray follows this trend:

With the modifications in this branch, we end up with this trend instead:

Since memray was used to perform memory profiling, I added it as a profiling dev dependency.

I would also like to point out that these optimizations were already spotted years ago. At the time we chose to move to streaming mode but never implemented the madvise calls. See #477 (comment)

[vlaci]

I don't question the decrease in memory usage by this MR, I just don't really understand how each part affects it in isolation. Also it remains a question for me how aggressive we need to be reclaiming memory. Also how does the memory usage profile change between before and after this change when under memory pressure?

[qkaiser]

I don't question the decrease in memory usage by this MR

noted

I just don't really understand how each part affects it in isolation.

I'll provide traces before/after each commit so we can make informed decisions.

Also it remains a question for me how aggressive we need to be reclaiming memory.

I'd argue we can be pretty aggressive in reclaiming memory as long as the impact on speed is limited.

Also how does the memory usage profile change between before and after this change when under memory pressure?

Do you have ideas on how we can test that ? Any tools that could simulate memory pressure to see how unblob behaves ?

[vlaci]

Do you have ideas on how we can test that ? Any tools that could simulate memory pressure to see how unblob behaves ?

When we first experimented with mmap, I don't recall if I tested it only in a memory constrained docker container or a VM, when we did the mmap transition. I was looking at two things: scanning through a file bigger than the available memory works, and that memory gets reclaimed from unblob when a process external to unblob puts additional pressure on the kernel

[martonilles]

When a memory area is madvised with DONTNEED, the RSS usage drops:

Note that, when applied to shared mappings, MADV_DONTNEED might not lead to immediate freeing of the pages in the range.  The kernel is free to delay freeing the pages until an appropriate moment.  The resident set size (RSS) of the calling process will be immediately reduced however.

However in case of a mmap-ed memory backed by a file, under memory pressure the kernel can "reclaim" the memory, so this is not causing OOM errors. However in this particular case the process is running in a docker container with a memory limit. In case a docker container (cgroup) memory limit is reached the kernel tries to reclaim and only goes into OOM if after the reclaim it still fails.

So, in normal cases, DONTNEED is just a hint (advise) and should have no actual impact. Visible the RSS drops, but from a kernel standpoint it does not matter. On the other hand the docker memory limit calculation the RSS might matter, however the reclaim should handle it.

[martonilles]

On the other hand calling too many madvise could be slow.

I would look at what exactly happens in a docker container+mem limit scenario with mmap-ed files.

[martonilles]

• https://serverfault.com/questions/1192733/actual-sequence-of-events-from-memory-pressure-to-oom-for-cgroups-v2
• https://www.michal-drozd.com/en/blog/container-page-cache-thrashing/

Would be nice to double check if indeed the files are mapped readonly and shared

[qkaiser]

On the other hand calling too many madvise could be slow.

yes, time impact will be measured. When I talked with @vlaci I explained that the regained syscalls by removing fstat calls could be spent calling madvise instead 🙃

I would look at what exactly happens in a docker container+mem limit scenario with mmap-ed files.

that's the plan, objective is to see if we can actually reproduce OOM

[qkaiser]

I have been misled by memray that does not distinguish between RssFile and VmRss (see bloomberg/memray#778).

I'll keep:

• the MADVISE_SEQUENTIAL when we initialize the file, so we hint the kernel
• the syscall improvement around size() and fstat calls