Speeding up CSV file download

Based on some rudimentary function speed testing, downloading the CSV file from Mountain Project is by far the slowest part of MPV, clocking in at over 6 seconds for a 2000+ entry tick list. With database operations sped up, I would like to take care of this last remaining speed bump.

I spent some time digging into downloading a file concurrently, with the hopes of speeding up how soon MPV has access to the data, but didn't have much luck. I suppose it is possible we are just limited by how quickly the CSV data comes over the network, but I'm not ready to give up just yet! :-)

[benjpalmer]

Can you provide a url for a 2000+ ticklist file that you would be willing to share? I would use my Mountain Project account except I don't have any ticks; Adding 2000+ seems pretty daunting and tedious too haha!

Anyway, this might or might not be helpful in your attempt to optimize:
http://masnun.com/2016/09/18/python-using-the-requests-module-to-download-large-files-efficiently.html

I would be interested in a side by side comparison of downloading the file the way we currently are vs setting streaming=True and parsing the file in chunks. This might shed some light on where any optimizations might be helpful. Is the process slow because we are storing a lot of content in memory? Os is the process slow because of the fact we have to download a file over http and are chained to those limitations?

@benjpalmer , you can use mine for a 2000+ ticklist. Yes, adding that many ticks to your own would be more than tedious. :-)

Thanks for pointing me in the direction of streaming=True. I had enabled that option while trying to get a threaded download to work (I never did), but using it on it's own definitely sped up MountainProjectHandler. Went from ~6.58 seconds to ~0.65 seconds for my ticklist. Just pushed out a commit with this change in effect. I think it closes this issue.

I'm going to open a new issue with some new performance data I discovered.

Given what I discovered in issue 33, I'm reopening this to continue exploring ways to improve the biggest performance bottleneck in MPV.