Where in code can you change the inference from Ollama to something else?

[zxcvxzcv-johndoe]

I have managed to get this to run with Windows and WSL otherwise but in the last part it fails with this error below.

Where in code can I try to change it from using Ollama to something else like OpenAI (koboldcpp in practice)?

Thanks!

[llm/error] [1:llm:Ollama] [6ms] LLM run errored with error: "Unexpected end of JSON input"

• error SyntaxError: Unexpected end of JSON input
  at JSON.parse ()
  at parseJSONFromBytes (node:internal/deps/undici/undici:4553:19)
  at successSteps (node:internal/deps/undici/undici:4527:27)
  at fullyReadBody (node:internal/deps/undici/undici:1307:9)
  at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
  at async specConsumeBody (node:internal/deps/undici/undici:4536:7)
  at async createOllamaStream (webpack-internal:///(sc_server)/./node_modules/langchain/dist/util/ollama.js:26:21)
  at async Ollama._call (webpack-internal:///(sc_server)/./node_modules/langchain/dist/llms/ollama.js:313:26)
  at async Promise.all (index 0)
  at async Ollama._generate (webpack-internal:///(sc_server)/./node_modules/langchain/dist/llms/base.js:311:29)
  at async Ollama._generateUncached (webpack-internal:///(sc_server)/./node_modules/langchain/dist/llms/base.js:174:22)
  at async Ollama.call (webpack-internal:///(sc_server)/./node_modules/langchain/dist/llms/base.js:242:34)
  at async POST (webpack-internal:///(sc_server)/./src/app/api/qa-pg-vector/route.ts:61:24)
  at async eval (webpack-internal:///(sc_server)/./node_modules/next/dist/server/future/route-modules/app-route/module.js:242:37)

[ykhli]

I think this error is usually because the service you are calling isn't up and running on the correct url/port - maybe double check you can curl the URL?

I'm not so sure if koboldcpp actually exposes their API endpoint (glanced through their README but couldn't find it). For OpenAI connection - it should just work if you delete Ollama-related env vars and fill in OPENAI_API_KEY

See here: https://github.com/ykhli/local-ai-stack/blob/578c87c02f850d94f072310312b4dc9d89c87ddd/src/app/api/qa-pg-vector/route.ts#L52C1-L78C4

[zxcvxzcv-johndoe]

Thanks, I removed the enviromental variables and changes the lines 63-68 to this because I want to run my local LLM which has OpenAI compatible API. So I added that BaseURL but that's not working, it will still connect to OpenAI and not my local computer.

model = new OpenAI({
      openAIApiKey: "YOUR_KEY_HERE",
      temperature: 0.9,
      streaming: true,
      modelName: "gpt-3.5-turbo-16k",
      configuration: {
        baseURL: "http://172.29.144.1:5001/v1",
      }, 
  
       callbackManager: CallbackManager.fromHandlers(handlers),
    });
    model.verbose = true;

Any ideas what could cause that? I was able to do the same change in https://github.com/langchain-ai/langchain-nextjs-template/blob/main/app/api/chat/route.ts and that connects properly to my local baseURL.