[FEATURE]: models via API

[qdrddr]

I'd like to be able to use models via API using LiteLLM Client SDK.

With providers like:
Ollama/LM Studio - Embeddings, LLM
OpenRouter - Embeddings, LLM
DeepInfra - OCR, reranker, Embeddings, LLM
LiteLLM Proxy - OCR, reranker, Embeddings, LLM

[selipso]

Same, this is the only feature making me look elsewhere. Want to run this in a Linuxbrew box without a GPU

[jamiequint]

#133

i forked a branch jamiequint/qmd that supports this now

[theol0403]

This makes running qmd on a raspberry pi practical!

[marcinbogdanski]

I did alternative/complimentary PR:

feat(llm): add API backend support with backend-aware session routing

• adds API-backed LLM support (embeddings, query expansion, rerank)
• backend-aware session/chunking paths so API mode does not initialize local Llama/tokenizer paths

Seem repo is exploding, for maintainer convenience, I found these duplicates/complimentary work:

• Add optional OpenRouter provider for embeddings/search pipeline#133
• feat: Add OpenAI embedding and query expansion support#116
• Add OpenAI-compatible remote LLM support and remote sanity-check scri…#106
• feat: add remote embedding providers (Voyage AI, OpenAI-compatible)#65
• possibly more

[JasonGiedymin]

I've closed my PR #106 as the maintainers have other priorities.

The app has improved performance but for those who still require remote calls (such as myself) my repo will still be available and I'll keep it up to date (within reason). Branch: remote-api: https://github.com/JasonGiedymin/qmd-remote/tree/remote-api

Local perf enhancements in latest mainline:
QMD now runs on both Node.js and Bun, with up to 2.7x faster reranking
through parallel GPU contexts. GPU auto-detection replaces the unreliable
gpu: "auto" with explicit CUDA/Metal/Vulkan probing.

Again, if this isn't possible for you (vps, no GPUs) I'll be maintaining my fork up to date (within reason) and you can continue using that (for now).

[qdrddr]

That's a pity.
I understand that the author wants to keep it air-gapped and fast.

Though some times user would prefer to make it a bit longer in favor of SotA models for maximum recall accuracy that you simply couldn't run locally.

Hope @tobi you could reconsider and could merge this PR.

[JasonGiedymin]

If it wasn’t clear I just want to point out that those notes were from the latest mainline release (not my fork). They did a good job improving performance. I just so happen to be running qmd on a raspberry pi so I need to use remote hosted LLMs.