Project:

Package:

Development:

Contents

• Overview
• Installation
• Development
• Additional Information
• License

Overview

This project provides a modular benchmark pipeline for experimenting with different vector databases (FAISS, Qdrant, …).
It runs end-to-end:

1. Download → Hugging Face dataset (optionally export images + manifest)
2. Embed → Generate CLIP embeddings for images
3. Build → Construct indexes with multiple VectorDBs
4. Search → Profile queries (latency + Recall@K vs exact baseline)
5. Update → Test insertions & deletions (index maintenance)

All steps are run with uv as the package manager.

How to use iNatInqPerf

# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh

# Setup environment
uv venv .venv && source .venv/bin/activate
uv sync

# Run an end-to-end benchmark (FAISS IVF+PQ vectordb) on the INQUIRE dataset.
uv run python scripts/run_benchmark.py configs/inquire_benchmark.yaml

# Spin up a 3-node Weaviate cluster (shared Docker network + RAFT) and run the benchmark.
uv run python scripts/run_benchmark.py configs/inquire_benchmark_weaviate_cluster.yaml

# Spin up a 3-node Qdrant cluster (HTTP+gRPC+p2p) and run the benchmark.
uv run python scripts/run_benchmark.py configs/inquire_benchmark_qdrant_cluster.yaml

Distributed VectorDB Deployments

• Benchmark-managed clusters. The configs/inquire_benchmark_weaviate_cluster.yaml and configs/inquire_benchmark_qdrant_cluster.yaml files include the container descriptions that container_context will launch automatically before each run. Make sure no identically named containers are already running, otherwise Docker will raise a name-conflict error.

The benchmarking code will

1. Download the specified dataset from the HuggingFace website.
2. Embed the images using a CLIP model.
3. Build a vector database index.
4. Perform a search for given queries to obtain query latency, and compute Recall@K vs FAISS Flat baseline..
5. Update the index.

Dataset Output Structure

data/raw/
  dataset_info.json
  state.json
  data-00000-of-00001.arrow
  images/
    00000000.jpg
    00000001.jpg
    ...
  images/manifest.csv   # [index,filename,label]

Supported Vector Databases

• faiss.flat (exact)
• faiss.ivfpq (IVF + OPQ + PQ)

Profiling Outputs

• Latency statistics (avg, p50, p95)
• Recall@K vs baseline
• JSON metrics in .results/

Profiling with py-spy

Use py-spy to record flamegraphs during any step:

bash scripts/pyspy_run.sh search-faiss -- python src/inatinqperf/benchmark/benchmark.py search --vectordb faiss.ivfpq --hf_dir data/emb_hf --topk 10 --queries src/inatinqperf/benchmark/queries.txt

Outputs:

• .results/search-faiss.svg (flamegraph)
• .results/search-faiss.speedscope.json

---

Installation

Installation Method	Command
Via uv	uv add inatinqperf
Via pip	pip install inatinqperf

Development

Please visit Contributing and Development for information on contributing to this project.

Additional Information

Additional information can be found at these locations.

Title	Document	Description
Code of Conduct	CODE_OF_CONDUCT.md	Information about the norms, rules, and responsibilities we adhere to when participating in this open source community.
Contributing	CONTRIBUTING.md	Information about contributing to this project.
Development	DEVELOPMENT.md	Information about development activities involved in making changes to this project.
Governance	GOVERNANCE.md	Information about how this project is governed.
Maintainers	MAINTAINERS.md	Information about individuals who maintain this project.
Security	SECURITY.md	Information about how to privately report security issues associated with this project.

License

iNatInqPerf is licensed under the MIT license.