Add support for local/open source LLM providers

[robinpaul85]

Can you please add support for open source local LLM providers such as Ollama?

[cacilhas]

I would like it too.

[MoonKraken]

Thanks for the suggestion! I've been planning on doing this, been mulling over the right way to do it.

One approach would be to just detect a listener on the default port ollama listens on. That would probably work for most folks, but its a little restrictive. The only way to handle alternate ports is to allow users to add custom providers, which requires building out that whole flow.

I guess ultimately we probably want both of these.

Ideas and PRs welcome!

[Dygear]

LM Studio (I know, I know) uses the 1234 port, and seems to be using the OpenAI API of

[LM STUDIO SERVER] Supported endpoints:
[LM STUDIO SERVER] ->	GET  http://192.168.0.251:1234/v1/models
[LM STUDIO SERVER] ->	POST http://192.168.0.251:1234/v1/responses
[LM STUDIO SERVER] ->	POST http://192.168.0.251:1234/v1/chat/completions
[LM STUDIO SERVER] ->	POST http://192.168.0.251:1234/v1/completions
[LM STUDIO SERVER] ->	POST http://192.168.0.251:1234/v1/embeddings

So you can run it on your local network, and have clients outside of local host query it for questions. Zed supports this interface fairly well. I can for example setup Zed on my MacBook Pro to query this interface that is running on my Framework Desktop running Windows. I guess the place to look for inspiration would be there.

[MoonKraken]

thanks, this is directional!