Batch inference

Hi,

First of all, thanks a lot for this contribution–super cool.

I was wondering if you were planning to add batch inference? It seems like the memory consumption when running prediction is not so large, I wonder if running inference in batches might speed up the process quite a bit.

Happy to hear your thoughts!