Analyzer itself affects memory footprint

[mx781]

Hi @lisroach and maintainers, thanks for this useful tool!

A quick story. As I was profiling an application yesterday I kept running into an odd issue - the initial footprint seemed reasonable, but when profiling later into its lifecycle the snapshots showed a list of pd.Series objects. I do create some Series objects in my app, but to my knowledge all refs are discarded, so they get garbage collected. The profiling made me think there's an edge case where they do not get discarded, so I spent the day trying to pinpoint when and why that happens.

Along the way I realised that these extra references are actually getting created by the memory analyzer itself - so the sudden jump of pd.Series objects was a red herring. This seems like a huge oversight - the memory usage of the process being analyzed is being affected by the analyzer itself, so the metrics you get back aren't actually representative of the scanned process.

Here's a repro to support this:

1. Make a runnable and two scripts to measure+plot its memory usage using ps and gnuplot

# run.py
import time
import pandas as pd
import numpy as np


def clean_function():
    s1 = pd.Series(np.random.random((10000,)))
    s2 = pd.Series(np.random.random((10000,)))
    s3 = pd.Series(np.random.random((10000,)))
    total = s1.sum() + s2.sum() + s3.sum()
    print(total)
    time.sleep(0.5)
    return None


if __name__ == "__main__":
    time.sleep(5)
    for i in range(1000):
        print("iteration", i)
        clean_function()

#!/usr/bin/env bash
# watcher.sh; requires `gnuplot`
while true;
do
        ps -p $1 -o pid=,rss=,vsz= >> /tmp/mem.log
        gnuplot /tmp/show_mem.plt
        echo "Logged at $(date)"
        sleep 1
done

# /tmp/show_mem.plt
set ylabel "VSZ"
set y2label "RSS"

set ytics nomirror
set y2tics nomirror in

set yrange [0:*]
set y2range [0:*]

plot "/tmp/mem.log" using 3 with lines axes x1y1 title "VSZ", \
     "/tmp/mem.log" using 2 with lines axes x1y2 title "RSS"

2. Run, measure, plot

python run.py &
./script.sh $!
# Hit Ctrl+c when process is done

a) let it run to completion
b) run memory_analyzer run $PID periodically

Results
a:

b:
(I stopped running analyzer past the 100s mark)

First snapshot from profiler:

Last snapshot from profiler:

It feels like this could be fixed by piping the metrics retrieved from gdb into the analyzer process and then discarding them from the scanned process, though I don't know if there is an inherent limitation to the underlying tooling that prevents this. Though if this is at all expected, I think the README could use a clear warning sign!

EDIT: looks the like graphs suggest the process starts off using less memory when profiled than not - I think this is actually an issue with the plotting, so ignore the scale - the key aspect is that it grows vs staying static.