Serializing using protocol buffers

[crpietschmann]

Discussed in #75

^{Originally posted by dano1066 July 11, 2025}
I see we have a new method to serialize/deserialize to binary stream but its still the same json call under the hood. It's taking 3-4 seconds to load an index of around 4000 records for me at the moment. Not terrible but problematic. I was wondering if we could bypass the libraries built in serialization and use proto buffers to serialize the instance of the index/sharpvector in memory. This way, I could quickly switch between databases without the cost of the current cold start. Is it something anyone has tried. I pulled a fork of the code and gave it to a bot to pull apart to see if there would be any issues doing it and it did say it wouldnt work because not everything is designed to be serialized like this but I am not sure if that means everything needs to be or just the important parts?

[crpietschmann]

I asked GitHub Copilot to take a look at this. I need to review what it came up with and consider how to add a more performant serialization option.

The JSON serialization implemented today was to create a data persistent format that is easy to work with; as it's JSON files within a .ZIP file. This also gives the bonus of being able to use other tools to easily parse and work with the persisted data if desired. A binary format option would definitely give higher performance and would be a great additional option to the library in the future.