Prefect cache grows excessively while cmorizing

[mzapponi]

The prefect cache is storing a lot of data while using the tool.
I had to reset it three times, while cmorizing my data today, as it was building enough data to make me reach the /home quota on Levante.
It makes up to 20Gb of data while processing 5-6 decades of one 3D variable.

Is there a possible solution to that (beside moving the cache folder to /scratch) like reducing the data stored or adding a cleanup of the cache after using the file processing is complete?

[mandresm]

Hi @mzapponi.

Can you try this before you run pycmor?

export PREFECT_LOCAL_STORAGE_PATH="/path/to/your/cache"

And make that path point at a directory in your scratch?

20 GB is not a lot, but the caching should not end up in your home folder.

@siligam, I cannot find this in the documentation. Can you add it? I think it should be in one of the main pages on how to use pycmor because everyone will confront this problem on the HPCs if they don't setup their cache to for example the scratch.

[mzapponi]

Hi @mandresm!
Yes I already did it last week because at one point it was reaching the quota limit for every file that I was processing, now the caching is in the /scratch folder with a link to the home folder and it works fine.

I processed 2 3D variables for 100 years each and I have 66GB of data there now, so I agree that it is not a huge amount of data, but then definitely it can't be in the home folder by default.