Enable OpenAI Realtime API

[grahac]

Describe the feature or improvement you're requesting

Would love to have openAIs realtime API which uses websockets added to the elixir library.

API description: https://platform.openai.com/docs/guides/realtime
You can see an example client in JS here: https://github.com/openai/openai-realtime-console?tab=readme-ov-file

Additional context

No response

[restlessronin]

@grahac I took a look at the spec, and I'm wondering about your use case. Since this is (at least initially) a voice-only api, it's hard to imagine it being used non-interactively, without a human on one side.

If it's a website where the user is on a browser anyway (using wavtools from the reference client), does the elixir app need to do anything beyond keeping the API key secret?

In this situation, we just need a Relay Server, forwarding websocket traffic both ways, like this app from the reference client.

https://github.com/openai/openai-realtime-console/blob/main/relay-server/index.js

[grahac]

Yeah my use case is likely going to hook it up to a phone line or web browser by linking the websockets.So maybe the answer is just to use a relay server and go direct HTTP calls?

[francoabaroa]

@grahac you could do relay server with Elixir, I tried it yesterday, worked fine.

Or

You can call the OpenAI Realtime API directly in your Elixir app using something like Websockex.

You can see an example of connecting to the OpenAI Realtime API using Websockets here: https://github.com/francoabaroa/openai_realtime_ex/blob/main/README.md.

It's still a WIP, but for now it successfully establishes the connection with OpenAI.

Just need to add support for capturing voice and sending it to OpenAI, getting the response and playing it back. To be updated in the coming days.