Oversized github repository

[michaelchin]

The size of this repository is 914M as of 13 Jun 2023. Write down this problem lest I forgot.

du -sh -- * .[^.]* | sort -h

Ignore history:

git clone --depth 1 --branch master https://github.com/GPlates/gplately.git

[brmather]

All of this is taken up by graphics and animations in the notebooks. I would like to clear these from all notebooks in the repo and flush them from the git history as well.

I've looked at a few solutions in the past but none were fantastic. Running jupyter nbconvert in a git hook might be one way. Alternatively we could use jupytext to properly version control notebooks as markdown files (or regular python files).

[michaelchin]

Just a note to myself.

use BFG to purge history https://rtyley.github.io/bfg-repo-cleaner/#download

[michaelchin]

Put this on ice until it causes real problems or too painful to bear.

[brmather]

use BFG to purge history https://rtyley.github.io/bfg-repo-cleaner/#download

Most of the space is taken up by Jupyter notebook outputs (images, gifs, etc.) Does BFG help with this?

I've found a couple more resources which may help with Jupyter notebooks:

https://zhauniarovich.com/post/2020/2020-10-clearing-jupyter-output-p3/
https://www.scivision.dev/git-jupyter-strip-output/

[michaelchin]

use BFG to purge history https://rtyley.github.io/bfg-repo-cleaner/#download

Most of the space is taken up by Jupyter notebook outputs (images, gifs, etc.) Does BFG help with this?

I've found a couple more resources which may help with Jupyter notebooks:

https://zhauniarovich.com/post/2020/2020-10-clearing-jupyter-output-p3/ https://www.scivision.dev/git-jupyter-strip-output/

BFG just purges history. The history may take up some space. I think we can try your findings first. This issue is not too painful for now.

Anyway, people can always ignore history by

git clone --depth 1 --branch master https://github.com/GPlates/gplately.git

See the screenshot below. The history is not a big problem anymore.

du -sh -- * .[^.]* | sort -h

[jcannon-gplates]

Just commenting because you raised this issue again recently by email (since we're probably going to be adding a lot of notebooks as part of the documentation overhaul)...

From your screenshot it looks like the Notebooks directory is taking most of the space (compared to the .git history).

What Ben suggested here sounds good to me. I guess you are already using nbconvert to generate the documentation (ie, convert notebooks to html). Perhaps the documentation is the only place where output cells need to be visible? Does that mean the notebooks themselves would have no graphics or animations (that is, until the user actually runs them to generate the output)?

[jcannon-gplates]

This is what I get on a newly cloned repo (master branch at 7b267fd)...

...looks like the .git folder has increased dramatically from Michael's 80 MB to my 875 MB. And pretty much all of that is coming from .git/objects/.

So it looks like the history is becoming the biggest problem.

While the Notebooks/ sub-directory is reasonably big...

...the actual notebooks themselves only take up about 50 MB (of that 235 MB Notebooks/ sub-directory).

But committing notebooks with updated plots/animations is really accumulating in the git history.

For example, I just updated all the notebooks (including their plots/animations) in #337 and the size of .git/objects increased by 40 MB. If I were to do that again it would increase another 40 MB.

And, as a small test on the newly cloned repo, committing a notebook with a small change (just some code change) added less than 1 MB. But when I updated its plots/animations and committed, then it added about 5 MB. I guess that's pretty obvious, but just confirms that git is pretty good at delta-compressing small text changes in a notebook when all the binary bits (plots/animations) are the same.

So, in the short term, we could update only the code in the notebooks without updating plots/animations (if they don't change much), and save space that way. But probably looks like a longer term solution (like Ben and Michael have suggested) will be needed fairly soon.

[jcannon-gplates]

I just realized that the 80 MB of .git history in Michael's case was on a cloned repo of depth 1.

Still, it looks like the history size has increased from 700 MB in mid-2023 (top post above) to over 900 MB now.

[michaelchin]

I just realized that the 80 MB of .git history in Michael's case was on a cloned repo of depth 1.

Still, it looks like the history size has increased from 700 MB in mid-2023 (top post above) to over 900 MB now.

Thanks John. It seems the the history size only increased ~200MB over two years. In fact, it was better than I thought. Yes, the outputs(images/animations) in the notebooks are the problem. I think notebooks using base64 encoding to save those images make the situation even worse.

[jcannon-gplates]

Yeah, that's not too bad. Maybe that's because we haven't really modified the notebooks much over the last couple of years. As we start adding more and modifying it might accelerate a bit. We'll find out soon enough though.

[michaelchin]

convert to discussion until we decide some actionable plans.