Use of Azure endpoints?

[iainmwallace]

This is really awesome. Is it possible to use with Azure endpoints?

https://learn.microsoft.com/en-us/azure/cognitive-services/openai/quickstart?pivots=programming-language-python&tabs=command-line

Thanks

[wch]

Yes, it should be possible, although I have not tested it with Azure myself.

When you call chatstream.chat_server(), you can pass it a url. See the docs here:
https://wch.github.io/chatstream/reference/chat_server.html#chatstream.chat_server

I believe you should still set the OPENAI_API_KEY environment variable. In the docs you linked to, they recommend setting AZURE_OPENAI_KEY, which you could do, but then you would need to get that env var's value manually and pass it like chatstream.chat_server(api_key=os.getenv("AZURE_OPENAI_KEY"), ...).

If you test it out, please let me know how it goes!

[iainmwallace]

I tried but it didn't seem to work - perhaps due to company firewall. I will dig in further

[trangdata]

Hi @wch, any chance you can add proxy and deployment_name as arguments to chat_server? There have been some issues regarding discrepancies between model and engine/deployment names on the Azure endpoint.

I modified my server function in basic/app.py as follows:

def server(input: Inputs, output: Outputs, session: Session):
    chatstream.chat_server(
        "mychat",
        model="Turbo", # also tried "gpt-35-turbo", didn't work
        url=os.getenv("OPENAI_API_BASE"),
        api_key=os.getenv("OPENAI_API_KEY"),
    )

and getting

Unhandled error: 'Could not automatically map Turbo to a tokeniser. Please use tiktok.get_encoding to explicitly get the tokeniser you expect.'

When I set model="gpt-3.5-turbo", I got

Unhandled error: Please set the OPENAI_API_KEY environment variable to your OpenAI API key.
which sounds like it's still using the default Open AI url, not my Azure endpoint.

I'm not sure if we're using langchain but I wanted to include a few issues in case they're relevant:

• Must provide an 'engine' when using ChatCompletion api openai/openai-python#318
• AttributeError: 'OpenAIEmbeddings' object has no attribute 'deployment' langchain-ai/langchain#3251
• https://github.com/hwchase17/langchain/pull/6697/files
• AzureOpenAI InvalidRequestError: Too many inputs. The max number of inputs is 1. langchain-ai/langchain#4575
• fix: separate model and deployment for OpenAIEmbeddings langchain-ai/langchain#3076

[iainmwallace]

In case it is useful information gptstudio provides the ability to connect to an Azure endpoint.

https://github.com/MichelNivard/gptstudio

[wch]

@trangdata I don't have access to Azure OpenAI, but I just applied for access. Hopefully I will be given access in the near future, but until then I'm not able to test anything with Azure.

I'll try adding proxy and deployment_name in a pull request.

For reference, this is their documentation on the differences between using OpenAI's endpoints vs. Azure's endpoints.
https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/switching-endpoints

[wch]

@trangdata I don't see proxy as a possible parameter to .create() or .acreate(). Taking a quick look at the openai source code, I think you will have to set it like this:

import openai

openai.proxy = "..."

https://github.com/openai/openai-python/blob/b82a3f7e4c462a8a10fa445193301a3cefef9a4a/openai/__init__.py#L56

[wch]

I've created a PR for Azure support, #10. Please see that PR and test it out. There is more information and instructions in the PR.

[trangdata]

Thank you for the quick response! I installed the new chatstream version in the PR, added some openai configs (below) to my app.py, and uses endpoint_type = "azure" in chat_server, and it doesn't error out but the app just kinda stalls with ... in the response text box.

openai.api_type = os.getenv("OPENAI_API_TYPE")
openai.api_base = os.getenv("OPENAI_API_BASE")
openai.api_key = os.getenv("OPENAI_API_KEY")
openai.proxy = os.getenv("OPENAI_PROXY")
openai.api_version = "2023-03-15-preview"  # "2022-12-01"

When I set model="Turbo" or model="gpt-35-turbo", which is what I would do for other projects using the Azure endpoint, I received the same error as before.

Unhandled error: 'Could not automatically map gpt-35-turbo to a tokeniser. Please use tiktok.get_encoding to explicitly get the tokeniser you expect.'

[wch]

Quick answer because I’m on my phone: shouldn’t there be a period in gpt-3.5-turbo?

[trangdata]

Actually no. Azure is very odd with their engine/deployment name and model name. See langchain-ai/langchain#1577 (comment) and example: langchain-ai/langchain#3251 (comment)

[wch]

In the PR I just pushed some code to map from Azure model names to OpenAI model names.
@trangdata Please give it another shot. Thanks!

[trangdata]

Thanks, Winston!

So this time, instead of erroring out immediately, the app just kinda stalls forever. And when I Ctrl C, I see:

Task exception was never retrieved
future: <Task finished name='Task-9' coro=<stream_to_reactive..task_main() done, defined at /Users/let20/Documents/chatstream/chatstream/init.py:547> exception=InvalidRequestError(message="Must provide an 'engine' or 'deployment_id' parameter to create a <class 'openai.api_resources.chat_completion.ChatCompletion'>", param='engine', code=None, http_status=None, request_id=None)>

Full message

(chatstream) ➜ chatstream shiny run examples/basic/app.py --launch-browser
INFO: Started server process [19238]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO: 127.0.0.1:58174 - "GET / HTTP/1.1" 200 OK
INFO: ('127.0.0.1', 58175) - "WebSocket /websocket/" [accepted]
INFO: connection open
^CINFO: Shutting down
INFO: connection closed
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [19238]
Task exception was never retrieved
future: <Task finished name='Task-9' coro=<stream_to_reactive..task_main() done, defined at /Users/let20/Documents/chatstream/chatstream/init.py:547> exception=InvalidRequestError(message="Must provide an 'engine' or 'deployment_id' parameter to create a <class 'openai.api_resources.chat_completion.ChatCompletion'>", param='engine', code=None, http_status=None, request_id=None)>
Traceback (most recent call last):
File "/Users/let20/Documents/chatstream/chatstream/init.py", line 550, in task_main
func = await func # type: ignore
^^^^^^^^^^
File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_resources/chat_completion.py", line 45, in acreate
return await super().acreate(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_resources/abstract/engine_api_resource.py", line 214, in acreate
) = cls.__prepare_create_request(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_resources/abstract/engine_api_resource.py", line 83, in __prepare_create_request
raise error.InvalidRequestError(
openai.error.InvalidRequestError: Must provide an 'engine' or 'deployment_id' parameter to create a <class 'openai.api_resources.chat_completion.ChatCompletion'>

[wch]

I received access to Azure OpenAI, and got it working; see #10 again for instructions and an example app. Please give it a shot!

The API in that PR is currently really awkward to accommodate OpenAI and Azure-OpenAI, and I will clean that up before merging.

[trangdata]

Hmm, so I installed the new changes, but the app hangs after I enter a prompt.

openai.error.APIConnectionError: Error communicating with OpenAI

Full message

Task exception was never retrieved
future: <Task finished name='Task-9' coro=<stream_to_reactive..task_main() done, defined at /Users/let20/Documents/chatstream/chatstream/init.py:547> exception=APIConnectionError(message='Error communicating with OpenAI', http_status=None, request_id=None)>
Traceback (most recent call last):
File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_requestor.py", line 668, in arequest_raw
result = await session.request(**request_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/client.py", line 536, in _request
conn = await self._connector.connect(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/connector.py", line 540, in connect
proto = await self._create_connection(req, traces, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/connector.py", line 899, in _create_connection
_, proto = await self._create_proxy_connection(req, traces, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/connector.py", line 1221, in _create_proxy_connection
proxy_req = ClientRequest(
^^^^^^^^^^^^^^
File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 305, in init
self.update_host(url)
File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 364, in update_host
raise InvalidURL(url)
aiohttp.client_exceptions.InvalidURL: proxy-server.bms.com:8080

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/Users/let20/Documents/chatstream/chatstream/init.py", line 550, in task_main
func = await func # type: ignore
^^^^^^^^^^
File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_resources/chat_completion.py", line 45, in acreate
return await super().acreate(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_resources/abstract/engine_api_resource.py", line 217, in acreate
response, _, api_key = await requestor.arequest(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_requestor.py", line 372, in arequest
result = await self.arequest_raw(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_requestor.py", line 685, in arequest_raw
raise error.APIConnectionError("Error communicating with OpenAI") from e
openai.error.APIConnectionError: Error communicating with OpenAI

[iainmwallace]

Is there any way of testing the api call is able to connect?

I am also getting the hanging issue but I get an SSL error saying that SSL:certificate_verify_failed for our Azure endpoint.

Thanks!