Memory issues

[willfig]

I've got the script up and running on one of the high performance computers and I think I'm having memory issues. I grabbed a few Lizard Island meshes and one from One Tree Island. The One Tree one worked but the other two failed with errors as in the screen shot. The computer I was using has 32GB of RAM. I'll see if I can upload them to cloudstore.

[shawes]

I cannot replicate this on my local machine, it seems to handle the mesh fine, albeit slowly. However, there could be differences between the way the OSX and Windows versions of the pypy interpreter handle the memory heap size. In any script, perhaps we might need to bump it up, especially when using large meshes such as this.

I will also look at limiting what is stored in memory for the vertices and faces, which is what is causing the error.

[willfig]

Did some reading. Pypy is 32 bit and as with all Windows 32bit apps, will only have access to max 2 GB ram. There is not 64bit version. I made sure to install 64bit python but it runs much slower as you know. Solutions:

1. Install VM running Linux on one of the machines and run from there. Not sure if this gets any round the RAM limitation though.
2. Accept we have to run big models in python and work to speed code up. Main way to do this is to index the faces as they load so we can then only search a small area to get info for each quadrat....I think. Best would be to index to the same spatial extent required for the quadrat size used but you don't really know where each quadrat will be until you load all the verticies and make the bounding box correct? So perhaps we just load them into a two dimensional array of list objects (or equivalent in python). So the array indices would be equivalent to cords of the bounding box at some fixed interval. Then faces get binned (put on the list) within the relevant x-y element combination. If you did this, once you know where the quadrat should be you just go search relevant array elements to see for sure if the face is in the quadrat. The step size for the array would just be some number a bit smaller than the desired quadrat size. So for 1x1, maybe you use indices that are 0.1 apart.

Would something like that work?

[shawes]

I'll have a think about the data structures we are using. Unless a file is ~2BG this should not really be an issue, unless the python garbage collection is causing issues.

[shawes]

My latest thoughts are to try and parallelise the code, which seems to work from my initial testing. We could add another input parameter to the script - the number of cores on the computer (default to say 4). Then we might be able to get away without using pypy and run 64-bit python. What do you think of this approach @willfig ?

[shawes]

I've been playing around with parallelising the code and it's a bit of a headache. Might need to restructure the code. Perhaps running Linux on a virtual machine might be easier?

[willfig]

Hold off on this. I talked with Mitch from ACFR about this a bit. He has a lot of code already that does various versions of this. He suggested we may be able to use some of his core bits, coded in C and made available to python. So we could still use python for the main execution. In any case, he was going to have a look at his code and send something through as a test.

In the meantime, I did ask for a linux server. They just got back to me. If I got this up and running, would you be able to log on and install what we need to run the code? Is that easy?

[shawes]

Yep, can do.

[willfig]

ok, i'll ask them to build it.