Reading FAISS generated indexes in compass

[sankha555]

Hi @Clive2312
I have been trying to generate faiss HNSW indexes using custom parameters and want to test them using compass. Can you please shed some light on the code that have been used to (i) generate HNSW indexes AND compressed PQ embeddings (ii) save them into files, and (iii) read the index and embeddings into Compass client?

[Clive2312]

Hi, sorry for the late reply. I hope you might have already figured this out by now. The HNSW index and PQ generation are handled using the Faiss library. You can use the write_index function from Faiss to save the HNSW and PQ data into files. A minimal example can be found here

For using custom parameters or datasets, you can follow these steps:

1. Add an entry in the configuration file and fill in the corresponding parameters and file paths.
2. Run ./compass_init with your chosen dataset to initialize the ORAM states.
3. Run ./test_compass_ring to test the search.

Let me know if you still run into any issues setting it up.