Calling Azure Open AI Endpoint, Getting API Rate Limiting Errors

[vuinguyen]

Hi friends and @MatthewCalder-msft !

My team and I are running into issues with an API rate limiting errors. We are calling our Azure Open AI endpoint for a RAG search chat application, and getting the following error: "An error occurred when calling Azure OpenAI: Server responded with status 429."

See attached screenshot.

We also researched the issue on Grok.com and received the following suggestion: “The error logs you provided show a 429 error with a suggestion to retry after long durations (e.g., 86400 seconds or 24 hours in some cases), which is common for free trial accounts when quotas are exhausted.”

Since we are using a free trial account for this hackathon and it looks like the quotas are exhausted, is there a way we can get a quota increase to continue development?

We're also looking into implementing load balancing via API management using this tutorial.

In case this load balancing doesn't work, can we get any other advice or assistance to unblock us?

Thanks so much for your help and advice!

[MatthewCalder-msft]

I will reach out to some others to see what they say. My first suggestion would be to point to the OpenAI API rather than Azure OpenAI. Would that be an option for you?
Thank you for posting this. It helps to know where people run into blockers using only the free trial. Also, remember that performance is only 25% of your score. Even if you can't get everything working as you would like, keep going! There are still lots of ways to impress the judges.

[vuinguyen]

Thanks @MatthewCalder-msft for your reply. Implementing the load balancing didn't quite work out so well, and we're afraid of spending too much time on this.

That being said, doing a search against the index on the Azure portal (index playground) does produce JSON output. We're thinking of saving off that JSON output from the index search and displaying that in the client application. Sure, the client application isn't calling the endpoint directly and we're hardcoding the answer we're getting from the index playground, but it's one workaround. And we know that the AI backend DOES work, we just can't call it.

Thanks for your suggestion about using Open AI API and if you hear of any others, please let us know here. In the meantime, I think we will go with this new plan so we can move forward and make the most of the time we still have to work on this.