[Concept] : Weaviate Migration

[h0lybyte]

Core Concept/Theory
A clear and concise description of what the concept is. Ex. It would be cool if [...]

This ticket is for the Weaviate Migration and Cluster test casing!

---

Alternative Ideas
Is there any other way this concept could be used?

We are migrating away from ChromaDB as a vector storage and using Weaviate.

---

Alternative Examples/Sources
Are there any other references that you can provide?

No major examples or sources, but we could put notes up on Weaviate

---

Additional information
Add any other context or examples of this concept here.

I will add additional information here regarding any sources or material.

[h0lybyte]

We currently have three instances of Weaviate

1 - @ZachHandley Cluster through Peter from Weaviate - This will be our main / production cluster
2 - Local Swarm Cluster through Docker/Portainer Swarm - This is currently paused as the performance is really weak, storage is a bit too nested in emulation.
3 - Weaviate Test Instance under Appwrite's Test User - This is the one that @ernivani will mess around with.

This way we are not messing up anything when we are test casing the integration of weaviate during the dev cycle.

[h0lybyte]

We can utilize Unstructured -> https://github.com/Unstructured-IO to handle the _raw_data (raw) from HTML, PDFs, ect..

Blog Post from Weaviate -> https://weaviate.io/blog/ingesting-pdfs-into-weaviate
Repo for Weaviate + Unstructured -> https://github.com/weaviate/how-to-ingest-pdfs-with-unstructured

I am thinking we could have these be isolates, running outside of any main application then use a simple queue system to handle the data refinement.

_raw_data -> [ refinement node ] -> stores _data into the vector database ?

I am down to see if there could be any additional ways to handle it.

[ernivani]

Got Nodepy working with KBVE API, now setting up Weaviate to test case

[h0lybyte]

Additional Reference Material:

Quick Tour -> https://colab.research.google.com/drive/1U8VCjY2-x8c6y5TYMbSFtQGlQVFHCVIW
SEC Extraction Repo -> https://github.com/Unstructured-IO/pipeline-sec-filings

Known Issue with duel column format -> Unstructured-IO/unstructured#356