Persist indexed documents to avoid cold starts

[danielfbnunes]

First of all, great work! 🎉

I'm anticipating that we would have a hard time with an app restart since we would have to restore the previous state (i.e. index the documents again). Could we persist this state? 💾

[alexmolas]

Hi @danielfbnunes

I'm not sure how we can efficiently persist the state between runs since all the documents are stored in memory. One option could be to store the state of the searcher in a pickle (or something similar) after indexing the documents, and for following runs, load the pickle and index additional documents. However, I suspect that this approach may not be very fast. If you test this option and compare its performance with the current implementation, we can determine if it's a viable solution for the package. Let me know if you plan to explore this track!