feat: Add LMStudio provider support #20
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I've added support for local inference through LMStudio. This new provider will allow you to use Séance with a local inference server. It's compatible with the OpenAI API, and you can configure it in the `config.ini` file—no API key is required. The endpoint can be set via the `endpoint` field in the provider's configuration section. I also included a fallback mechanism. If the LMStudio provider isn't configured in your `config.ini` file, it will use the default settings. Here's a summary of my changes: - Added `LMStudio` to the `Provider` enum. - Added default configuration for the `lmstudio` provider. - Created a new `lmstudio.nim` provider file. - Updated the configuration loading to handle the `endpoint` field. - Updated the provider loading to handle the case where the provider is not configured. - Added tests for the new provider and the fallback logic.
I've added a new feature that checks the LM Studio server for available models before sending a chat request. If the model you've requested isn't found, you will be notified with a warning. I also fixed a bug affecting both the OpenAI and LMStudio providers where the model returned by the API was not being correctly used. Here are the technical details of the implementation: - Added a `getRequestHandler` to the `ChatProvider` type to allow for mocking GET requests in tests. - Updated the `LMStudioProvider` to use this handler for fetching the list of available models. - Added a new test case to verify the model check logic.
I've added a new feature to the LMStudio provider. If you request a model that is available but not loaded, I will now prompt you to either continue (which will load the model) or to use an already loaded model instead. This improves your experience by giving you more control over model loading. To accomplish this, I: - Updated the `LMStudioProvider` to check the `state` of the model. - Added an interactive prompt to the `chat` method. - Added a test case to verify the new logic in a non-interactive session.
- **Interactive prompt for unloaded models**: If the requested model is available but not loaded, I will now prompt you to either continue (which will load the model) or to use an already loaded model instead. - **Robust model state parsing**: I've made the logic for parsing the `/v1/models` endpoint more robust, so it can now handle models that don't have a `state` field. - **Improved error parsing**: I have improved the error handling for API responses. I will now parse JSON error responses and extract the specific error message, providing a much cleaner and more informative error message to you. I have implemented this for both the OpenAI and LMStudio providers. - **Debug log for model parsing errors**: I have changed the warning about model parsing errors to a debug message to avoid confusing you.
This commit adds error handling to the `chat` command to gracefully handle API errors. Previously, if the API returned an error, the application would crash with an unhandled `IOError`. Now, the error is caught, and a user-friendly error message is displayed. - Added a `try...except IOError` block to the `chat` command in `src/seance/commands.nim`.
This commit removes the redundant error logging from the providers. The error is now only logged in the `chat` command, which prevents the error message from being displayed twice.
This commit fixes an issue where error messages from the API were being displayed twice. This was caused by the error being logged at the provider level and then again at the command level. I've removed the redundant logging from the providers and adjusted the exit call in the command to prevent the unhandled exception handler from printing the error again. The error is now displayed cleanly to you just once.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit introduces support for the LMStudio provider, allowing users to interact with locally hosted LLMs.
Key Changes:
lmstudio.nimmodule has been created to handle the specific API interactions for LMStudio.README.md: Updated to include LMStudio in the list of supported providers and added a configuration example for LMStudio.src/seance/config.nim:endpointtoProviderConfig.loadConfigto handle theendpointand to not require an API key for LMStudio.createConfigWizardto prompt for an LMStudio endpoint if selected.src/seance/defaults.nim: AddedDefaultLMStudioEndpointand a default model for LMStudio.src/seance/providers.nim:LMStudioProvider.newProviderto correctly instantiateLMStudioProviderand handle its specific configuration (e.g., no API key required, optional endpoint).getRequestHandlertoChatProviderinterface and implementations for passing it down.src/seance/providers/common.nim: AddeddefaultHttpGetHandlerfor providers that might need it.LMStudioProviderinsrc/seance/providers/lmstudio.nimimplements the chat functionality, including:/v1/chat/completionsendpoint./v1/modelsendpoint to check model loading status (with basic interactive prompts if not loaded).tests/t_providers_lmstudio.nimto verify the LMStudio provider's functionality, including request formatting, header inclusion, custom endpoint usage, and error handling.This enhancement significantly expands Séance's utility by enabling local LLM deployments.