Add compression

[thcrt]

Either batches or bundles would be compressed. I'm leaning towards compressing batches, because this would allow piecemeal transfer. However, compressing the entire bundle may lead to greater space savings. Compression would be documented in manifest.json files.

[thcrt]

@4censord you suggested zstd?

[4censord]

We should probably test different compression methods.
But the default for me seems to be zstd, yes.

[thcrt]

If you have a spare moment over the next few weeks, would you be able to put this together? I know very little about zstd besides some reading I did over lunch today, much of which flew over my head.

[4censord]

I will take a look.
Do you have an example dataset already that we can use to test how well it compresses with different algorithms?

[thcrt]

Thanks! You can import a sample dataset with ixporter sample "http://yoursolr.instance:8983/solr/stable". You may wish to let it run a while for a sufficiently large dataset or else your results may be biased towards whatever is on top of the sample set. I should probably change the code to pull sample entries randomly.

[thcrt]

Alternatively, you could just download the original corpus itself and remove the first two or three columns from the tab-separated CSV. Note that it uses an encoding I haven't been able to determine, but you can just use ISO and it should work fine.

[thcrt]

Okay, random sampling of the corpus is done, so you should be able to just run ixporter sample and then ixporter export.