feat(ai-core): add provider health tracking and safe fallback routing #108
Conversation
Summary of ChangesHello @ShashankSaga, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the robustness of the AI provider system by integrating a lightweight health tracking and dynamic routing mechanism into the AIProviderRegistry. The changes enable the system to automatically detect and adapt to temporary provider unavailabilities, ensuring continuous operation by prioritizing healthy providers and implementing a graceful fallback strategy. This improvement is designed to be non-breaking, without altering existing API contracts or introducing new dependencies, while laying the groundwork for future resilience features. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
|
Hi @abhishek-nexgen-dev 👋 As discussed, I’ve implemented a minimal, non-breaking provider health tracking and safe fallback mechanism inside The changes are fully internal, don’t affect existing flows, and are designed to be extensible for future improvements (cooldowns, retries, metrics). Please let me know if you’d like any adjustments or additional safeguards added. Happy to iterate further. Thanks! |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a valuable feature for provider health tracking and fallback routing within the AIProviderRegistry. The overall approach is well-contained and the implementation of health status tracking (markSuccess, markFailure) is clear. However, I've identified a significant issue in the generateTextWithFallback method where the logic doesn't fully align with the stated goals in the PR description, and it loses critical error information. My review includes a suggested refactoring of this method to improve its robustness and align it with the intended behavior.
| async generateTextWithFallback( | ||
| capabilities: AICapability[], | ||
| input: { prompt: string; context?: string } | ||
| ): Promise<string> { | ||
| const candidates = this.findByCapabilities(capabilities) | ||
|
|
||
| for (const provider of candidates) { | ||
| if (!this.isHealthy(provider.name)) continue | ||
|
|
||
| try { | ||
| const result = await provider.generateText(input) | ||
| this.markSuccess(provider.name) | ||
| return result | ||
| } catch { | ||
| this.markFailure(provider.name) | ||
| } | ||
| } | ||
|
|
||
| throw new Error('No healthy AI provider available') | ||
| } |
There was a problem hiding this comment.
The current implementation of generateTextWithFallback has a couple of issues that deviate from the goals outlined in the PR description:
- Incorrect Fallback Logic: The PR description mentions that the system should "fall back gracefully if all providers are degraded." However, the code only attempts to use providers that are currently healthy (
if (!this.isHealthy(provider.name)) continue) and never tries degraded ones, even if they are the only option. - Error Swallowing: The
catchblock is empty, which means the actual error from the provider is lost. This makes debugging failures nearly impossible, as the caller only receives a generic "No healthy AI provider available" message without any context on why the provider failed (e.g., API error, timeout, etc.).
I suggest a refactoring to correctly implement the fallback logic and preserve error information for better diagnostics.
async generateTextWithFallback(
capabilities: AICapability[],
input: { prompt: string; context?: string }
): Promise<string> {
const candidates = this.findByCapabilities(capabilities);
const healthyProviders = candidates.filter(p => this.isHealthy(p.name));
const degradedProviders = candidates.filter(p => !this.isHealthy(p.name));
let lastError: unknown;
// First, try all healthy providers
for (const provider of healthyProviders) {
try {
const result = await provider.generateText(input);
this.markSuccess(provider.name);
return result;
} catch (err) {
lastError = err;
this.markFailure(provider.name);
}
}
// If all healthy providers failed, fall back to degraded ones
for (const provider of degradedProviders) {
try {
const result = await provider.generateText(input);
this.markSuccess(provider.name);
return result;
} catch (err) {
lastError = err;
this.markFailure(provider.name);
}
}
if (lastError) {
throw lastError;
}
throw new Error('No AI provider available for the specified capabilities.');
}There was a problem hiding this comment.
Thanks for the detailed review 👍
Good catch — I’ve refactored generateTextWithFallback to:
- Attempt all healthy providers first
- Gracefully fall back to degraded providers if needed
- Preserve and rethrow the last provider error for better diagnostics
This aligns the implementation with the PR description while keeping the behavior non-breaking. Please let me know if you’d like further adjustments.
1 participant
Summary
This PR introduces lightweight provider health tracking and safe fallback routing inside
AIProviderRegistry.The goal is to improve reliability when an AI provider becomes temporarily unavailable (timeouts, crashes, API errors), without changing any existing behavior or API contracts.
What’s Included
healthy/degraded)Design Notes
AIProviderRegistryMotivation
Currently all providers are treated as equally healthy. This PR adds resilience and prepares the system for real-world provider instability while keeping the implementation minimal and non-breaking.
Closes #105