Gemini 429 Error not being caught

[isaacrlevin]

Follow up from #144 (comment)

I am using Mscc.GenerativeAI.Microsoft to be able to map Gemini calls via the Microsoft.Extension.AI package and I am getting severely delayed responses due to the timeouts. Here is what I am doing per the sample

        try
        {
           var chatClient = new GeminiChatClient(apiKey: settings.GoogleApiKey, model: settings.GoogleApiModel);

            var messages = new List<Microsoft.Extensions.AI.ChatMessage>
            {
                new(ChatRole.User, "Hi")
            };

            var options = new ChatOptions
            {
                MaxOutputTokens = 10
            };

            var response = await chatClient.GetResponseAsync(messages, options);
            
            var text = response.Text;

        }
        catch (GeminiApiTimeoutException ex)
        {
            var error = $"Google Gemini API error: {ex.Message}";            
        }
        catch (TimeoutException ex)
        {
            var error = $"Google Gemini API error: {ex.Message}";            
        }
        catch (GeminiApiException ex)
        {
            var error = $"Google Gemini API error: {ex.Message}";
        }
        catch (Exception ex)
        {
            var error = $"Google Gemini API error: {ex.Message}";
        }

My question is how can I control the RequestOptions property if I am using IChatClient? Do I need to build a custom wrapper for GetResponseAsync() and etc? I would have assumed I can add RequestOptions directly to GeminiChatClient

[jochenkirstaetter]

Hello @isaacrlevin

Thanks for reporting this (again). Appreciate it.
Let me review the code base and I'll revert back to you. Right now, I'd assume that the RequestOptions are passed through to the underlying client. Also, regarding MEAI I assume it might be done a bit differently.

Please bear with me and I'll resolve this during the coming days.

Cheers

[jochenkirstaetter]

Hi @isaacrlevin

Please help me out regarding the expected default behaviour. Right now, I'm slightly undecided between two options.

• The GeminiChatClient uses its default retry mechanism and can be configured via ChatOptions.AdditionalProperties
• It returns instantly on any non 2xx status code and retry can be enabled via AdditionalProperties

Here's the mapping approach between ChatOptions and RequestOptions

public static RequestOptions? ToGeminiGenerateContentRequestOptions(mea.ChatOptions? options)
{
	if (options is null) return null;

	if (options.AdditionalProperties?.Any() ?? false)
	{
		var retry = new Retry();
		TimeSpan? timeout = null;
		TryAddOption<int?>(options, "RetryInitial", v => retry.Initial = v ?? retry.Initial);
		TryAddOption<int?>(options, "RetryMultiplies", v => retry.Multiplies = v ?? retry.Multiplies);
		TryAddOption<int?>(options, "RetryMaximum", v => retry.Maximum = v ?? retry.Maximum);
		TryAddOption<int?>(options, "RetryTimeout", v =>
			retry.Timeout = v.HasValue ? TimeSpan.FromSeconds((double)v.Value) : retry.Timeout);
		TryAddOption<int[]?>(options, "RetryStatusCodes", v => retry.StatusCodes = v ?? retry.StatusCodes);
		TryAddOption<TimeSpan?>(options, "Timeout", v => timeout = v);

		if (retry.Initial > 0 && timeout is not null)
		{
			return new RequestOptions(retry, timeout);
		}
		else if (timeout is not null)
		{
			return new RequestOptions(timeout: timeout);
		}
		else
		{
			return new RequestOptions(retry: retry);
		}
	}

	return null;
}

(assigning default values decides between the two options.)

What's your take on this weighing pros and cons. Thanks.

Cheers

[jochenkirstaetter]

Here's the relevant snippet for the first scenario

var additionalProperties = new AdditionalPropertiesDictionary { { "RetryStatusCodes", (int[]) [500, 503, 504] } };
var options = new ChatOptions
{
	MaxOutputTokens = 10,
	AdditionalProperties = additionalProperties
};

Cheers

[isaacrlevin]

IMOP the app should respond somewhat responsively by default. I would go with option 2 and allow folks the ability to facilitate retry (attempts and timeout) from there. But tbh I treat a 429 as a 500 from a user perspective