Incomplete error handling when streaming response

[ebakkest]

I'm using the library with LM Studio's OpenAI compatible REST interface.

If I (for instance) I send too many tokens to the model in the chat API (using TOpenAIChat) and request streaming, the model right away sends back http status 400 and "error..." text, however the library doesn't catch the error condition and just appears to succeed instantly, but produces empty text for the completion.

If I turn the TOpenAIChat streaming off, the error is correctly reported.

[HemulGM]

Hi. Can you show me the place where the debugging error occurs? Or maybe you can provide a test case with this problem?

[ebakkest]

there is no debugging error raised. if you load a model in LM Studio with the context length set to 1 (for testing purposes) and attempt to stream back tokens from LM Studio using the library's chat API, you should see that the library raises no error but immediately reports work finished successfully. however, the first returned response contains the error text from LM Studio but this is not detected.

[HemulGM]

Yes, but could you please show me the place where the error response from the server comes from. So that I can add a result check

[ebakkest]

there is no error response! ;) the server returns http code 200 from the POST method when streaming, but instead of tokens, the text that is streamed contains error info formatted like below. so unfortunately, just checking the return code isn't enough.

I don't know if this is also the case when using the library against OpenAI. Perhaps the return code is a bug in LM Studio's API implementation?

event: error'#$A'data: {"error":{"message":"Trying to keep the first 6 tokens when context the overflows. However, the model is loaded with context length of only 1 tokens, which is not enough. Try to load the model with a larger context length, or provide a shorter input"},"message":"Trying to keep the first 6 tokens when context the overflows. However, the model is loaded with context length of only 1 tokens, which is not enough. Try to load the model with a larger context length, or provide a shorter input"}

[HemulGM]

Can you write down how the component is configured and what are you sending there? I can't repeat the error.