fix: replace discontinued gemini preview model#399
fix: replace discontinued gemini preview model#399Miyoko076 wants to merge 3 commits intoGitlawb:mainfrom
Conversation
gnanam1990
left a comment
gnanam1990
left a comment
There was a problem hiding this comment.
Thanks for working on this. I don't think the discontinued Gemini default is fully removed yet.
The PR updates the model config, but there is still a hardcoded gemini-2.5-pro-preview-03-25 default path in src/utils/model/model.ts, and that path is still used by runtime model selection. That means users can still hit the discontinued model through the default-opus flow.
Please update the remaining runtime default path as well and add a focused test that covers actual model selection, not just config values.
Proposed changes to address review feedbackHere is a patch addressing the open review comments: 1. Update runtime default in - return process.env.GEMINI_MODEL || 'gemini-2.5-pro-preview-03-25'
+ return process.env.GEMINI_MODEL || 'gemini-2.5-pro'2. Add focused tests in
|
|
Here is the full test file ready to copy into providerModelDefaults.test.tsimport { afterEach, expect, test } from 'bun:test'
import {
getDefaultHaikuModel,
getDefaultOpusModel,
getDefaultSonnetModel,
} from './model.js'
const originalEnv = {
ANTHROPIC_DEFAULT_OPUS_MODEL: process.env.ANTHROPIC_DEFAULT_OPUS_MODEL,
ANTHROPIC_DEFAULT_SONNET_MODEL: process.env.ANTHROPIC_DEFAULT_SONNET_MODEL,
ANTHROPIC_DEFAULT_HAIKU_MODEL: process.env.ANTHROPIC_DEFAULT_HAIKU_MODEL,
ANTHROPIC_MODEL: process.env.ANTHROPIC_MODEL,
GEMINI_MODEL: process.env.GEMINI_MODEL,
OPENAI_MODEL: process.env.OPENAI_MODEL,
CLAUDE_CODE_USE_GEMINI: process.env.CLAUDE_CODE_USE_GEMINI,
CLAUDE_CODE_USE_OPENAI: process.env.CLAUDE_CODE_USE_OPENAI,
CLAUDE_CODE_USE_FIRST_PARTY: process.env.CLAUDE_CODE_USE_FIRST_PARTY,
}
function clearProviderEnv(): void {
delete process.env.CLAUDE_CODE_USE_GEMINI
delete process.env.CLAUDE_CODE_USE_OPENAI
delete process.env.CLAUDE_CODE_USE_FIRST_PARTY
}
function clearModelEnv(): void {
delete process.env.ANTHROPIC_DEFAULT_OPUS_MODEL
delete process.env.ANTHROPIC_DEFAULT_SONNET_MODEL
delete process.env.ANTHROPIC_DEFAULT_HAIKU_MODEL
delete process.env.ANTHROPIC_MODEL
delete process.env.GEMINI_MODEL
delete process.env.OPENAI_MODEL
}
afterEach(() => {
process.env.ANTHROPIC_DEFAULT_OPUS_MODEL = originalEnv.ANTHROPIC_DEFAULT_OPUS_MODEL
process.env.ANTHROPIC_DEFAULT_SONNET_MODEL = originalEnv.ANTHROPIC_DEFAULT_SONNET_MODEL
process.env.ANTHROPIC_DEFAULT_HAIKU_MODEL = originalEnv.ANTHROPIC_DEFAULT_HAIKU_MODEL
process.env.ANTHROPIC_MODEL = originalEnv.ANTHROPIC_MODEL
process.env.GEMINI_MODEL = originalEnv.GEMINI_MODEL
process.env.OPENAI_MODEL = originalEnv.OPENAI_MODEL
process.env.CLAUDE_CODE_USE_GEMINI = originalEnv.CLAUDE_CODE_USE_GEMINI
process.env.CLAUDE_CODE_USE_OPENAI = originalEnv.CLAUDE_CODE_USE_OPENAI
process.env.CLAUDE_CODE_USE_FIRST_PARTY = originalEnv.CLAUDE_CODE_USE_FIRST_PARTY
})
test('Gemini provider opus default uses stable gemini-2.5-pro (not discontinued preview)', () => {
clearProviderEnv()
clearModelEnv()
process.env.CLAUDE_CODE_USE_GEMINI = '1'
expect(getDefaultOpusModel()).toBe('gemini-2.5-pro')
})
test('Gemini provider does not reference discontinued preview model', () => {
clearProviderEnv()
clearModelEnv()
process.env.CLAUDE_CODE_USE_GEMINI = '1'
const opus = getDefaultOpusModel()
const sonnet = getDefaultSonnetModel()
const haiku = getDefaultHaikuModel()
expect(opus).not.toContain('preview-03-25')
expect(sonnet).not.toContain('preview-03-25')
expect(haiku).not.toContain('preview-03-25')
})
test('Gemini provider sonnet default is gemini-2.0-flash', () => {
clearProviderEnv()
clearModelEnv()
process.env.CLAUDE_CODE_USE_GEMINI = '1'
expect(getDefaultSonnetModel()).toBe('gemini-2.0-flash')
})
test('Gemini provider haiku default is gemini-2.0-flash-lite', () => {
clearProviderEnv()
clearModelEnv()
process.env.CLAUDE_CODE_USE_GEMINI = '1'
expect(getDefaultHaikuModel()).toBe('gemini-2.0-flash-lite')
})
test('Gemini provider respects explicit GEMINI_MODEL override for opus', () => {
clearProviderEnv()
clearModelEnv()
process.env.CLAUDE_CODE_USE_GEMINI = '1'
process.env.GEMINI_MODEL = 'gemini-2.5-flash'
expect(getDefaultOpusModel()).toBe('gemini-2.5-flash')
}) |
|
@gnanam1990 what do you think? |
gnanam1990
left a comment
gnanam1990
left a comment
There was a problem hiding this comment.
Hi @Miyoko076, thank you for catching this and putting up the fix! The deprecated preview model definitely needs to go. Here are my findings after reviewing the full codebase:
Critical: Missed runtime fallback in model.ts
The config updates in configs.ts look correct, but there is still a hardcoded reference to the discontinued model in the runtime code path that actually serves the model string to Gemini provider users:
src/utils/model/model.ts, line 130:
return process.env.GEMINI_MODEL || 'gemini-2.5-pro-preview-03-25'This is inside getDefaultOpusModel(), which is called by the opus alias, opusplan, Max/Team Premium defaults, and getBestModel(). Since this function does not read from configs.ts — it uses its own inline string — the config changes alone won't fix the runtime behavior for Gemini provider users without a GEMINI_MODEL env var set.
Fix: Change 'gemini-2.5-pro-preview-03-25' to 'gemini-2.5-pro' on that line.
Important: GEMINI_MODEL_DEFAULTS appears to be unused
The GEMINI_MODEL_DEFAULTS constant in configs.ts (lines 22–26) is exported but never imported anywhere in the codebase. The PR updates it (which is fine for consistency), but it might be worth noting this is effectively dead code. If it's intended as documentation, that's okay — just something to be aware of.
Suggestion: Centralize the Gemini Opus default
The same model string currently lives in two independent locations (configs.ts and model.ts:130), which is exactly how they fell out of sync in the first place. Having getDefaultOpusModel() read from GEMINI_MODEL_DEFAULTS.opus (or a shared constant) would prevent this class of bug from recurring. This is optional for this PR but would be a nice follow-up.
Suggestion: Add a focused test
There are currently no tests verifying what getDefaultOpusModel() returns when the Gemini provider is active. A simple test like the one proposed in the comments would have caught this automatically. The test file shared in the comments looks solid — would be great to include it.
Summary
| What | Status |
|---|---|
configs.ts updates (5 places) |
Looks good |
model.ts:130 runtime fallback |
Still has discontinued model — needs fix |
| Tests | Missing — would be nice to add |
The PR is very close! Just the one-line fix in model.ts is the blocker. Thank you again for working on this — really appreciate the effort to keep the defaults up to date.
Miyoko076
force-pushed
the
main
branch
2 times, most recently
from
344b98e to
f6f92ed
Compare
|
Thanks for the thorough review @gnanam1990! I've updated the PR to address your feedback:
|
This comment was marked as outdated.
This comment was marked as outdated.
|
Added providerModelDefaults.test.ts |
Testing
|
Miyoko076
force-pushed
the
main
branch
3 times, most recently
from
7bfe465 to
97a1da2
Compare
|
Fixed all nits. PTAL |
gnanam1990
left a comment
gnanam1990
left a comment
There was a problem hiding this comment.
Thanks for fixing this. I rechecked the latest head and the original blocker is resolved:
- the runtime Gemini fallback now uses
gemini-2.5-pro - the external defaults are centralized consistently
- focused coverage was added in
src/utils/model/providerModelDefaults.test.ts
I did not find any new blocking regression in the current head. This looks good to me.
|
@graham1990 Diff commandC:\Users\Miyoko\Desktop\openclaude>git diff 97a1da2 7026dc9
diff --git a/src/utils/model/providerModelDefaults.test.ts b/src/utils/model/providerModelDefaults.test.ts
index 8fdda62..7453c33 100644
--- a/src/utils/model/providerModelDefaults.test.ts
+++ b/src/utils/model/providerModelDefaults.test.ts
@@ -17,17 +17,11 @@ const originalEnv = {
OPENAI_MODEL: process.env.OPENAI_MODEL,
CLAUDE_CODE_USE_GEMINI: process.env.CLAUDE_CODE_USE_GEMINI,
CLAUDE_CODE_USE_OPENAI: process.env.CLAUDE_CODE_USE_OPENAI,
- CLAUDE_CODE_USE_CODEX: process.env.CLAUDE_CODE_USE_CODEX,
- CLAUDE_CODE_USE_GITHUB: process.env.CLAUDE_CODE_USE_GITHUB,
- CLAUDE_CODE_USE_FIRST_PARTY: process.env.CLAUDE_CODE_USE_FIRST_PARTY,
}
function clearProviderEnv(): void {
delete process.env.CLAUDE_CODE_USE_GEMINI
delete process.env.CLAUDE_CODE_USE_OPENAI
- delete process.env.CLAUDE_CODE_USE_CODEX
- delete process.env.CLAUDE_CODE_USE_GITHUB
- delete process.env.CLAUDE_CODE_USE_FIRST_PARTY
}
function clearModelEnv(): void {
@@ -48,9 +42,6 @@ afterEach(() => {
process.env.OPENAI_MODEL = originalEnv.OPENAI_MODEL
process.env.CLAUDE_CODE_USE_GEMINI = originalEnv.CLAUDE_CODE_USE_GEMINI
process.env.CLAUDE_CODE_USE_OPENAI = originalEnv.CLAUDE_CODE_USE_OPENAI
- process.env.CLAUDE_CODE_USE_CODEX = originalEnv.CLAUDE_CODE_USE_CODEX
- process.env.CLAUDE_CODE_USE_GITHUB = originalEnv.CLAUDE_CODE_USE_GITHUB
- process.env.CLAUDE_CODE_USE_FIRST_PARTY = originalEnv.CLAUDE_CODE_USE_FIRST_PARTY
})
Re-testC:\Users\Miyoko\Desktop\openclaude>bun test C:\Users\Miyoko\Desktop\openclaude\src\utils\model\providerModelDefaults.test.ts bun test v1.3.11 (af24e281) src\utils\model\providerModelDefaults.test.ts: ✓ Gemini provider loads expected default models (Fallback logic check) ✓ Gemini provider does not reference discontinued preview model ✓ Gemini provider correctly applies GEMINI_MODEL environment override [15.00ms] ✓ Gemini provider main loop setting defaults to sonnet ✓ Gemini provider main loop setting respects GEMINI_MODEL override ✓ OpenAI provider loads expected default models (Fallback logic check) ✓ OpenAI provider correctly applies OPENAI_MODEL environment override ✓ OpenAI provider main loop setting defaults to opus ✓ OpenAI provider main loop setting respects OPENAI_MODEL override ✓ Codex provider loads expected default models (Fallback logic check) ✓ Codex provider correctly applies OPENAI_MODEL environment override ✓ Codex provider main loop setting defaults to opus ✓ Codex provider main loop setting respects OPENAI_MODEL override 13 pass 0 fail 27 expect() calls Ran 13 tests across 1 file. [782.00ms] |
|
checking if all test pass and will be merging this. |
|
hello @Miyoko076 would love to include this in the next release. this looks good already please just fix the failing tests |
|
@kevincodex1 |
for i in {1..10}; do env -i PATH="$PATH" bun test --randomize 2>&1 | sed -n '/tests failed:/,$p'; doneHere is a summary of the 10 randomized test runs.
bun test --randomize result on wsl2miyoko@Legion7-16iax10:/mnt/c/Users/Miyoko/Desktop/openclaude$ for i in {1..10}; do env -i PATH="$PATH" bun test --randomize 2>&1 | sed -n '/tests failed:/,$p'; done
11 tests failed:
(fail) Gemini provider correctly applies GEMINI_MODEL environment override [2.44ms]
(fail) Codex provider main loop setting defaults to opus [0.75ms]
(fail) OpenAI provider main loop setting defaults to opus [0.83ms]
(fail) Codex provider main loop setting respects OPENAI_MODEL override [0.86ms]
(fail) Gemini provider loads expected default models (Fallback logic check) [0.96ms]
(fail) Gemini provider main loop setting respects GEMINI_MODEL override [0.94ms]
(fail) OpenAI provider loads expected default models (Fallback logic check) [1.08ms]
(fail) OpenAI provider main loop setting respects OPENAI_MODEL override [1.11ms]
(fail) Gemini provider main loop setting defaults to sonnet [1.07ms]
(fail) Windows clipboard fallback > uses PowerShell instead of clip.exe for local Windows copy [5.47ms]
(fail) clipboard path behavior remains stable > Windows clipboard fallback is skipped over SSH [3.21ms]
This issue was caused by Bun's single-process global state and environment.
Addressing this issue thoroughly will require further investigation later. |
|
ok so i looked at this pr because the gemini-2.5-pro-preview-03-25 got discontinued (thanks google very cool) and honestly the fix is pretty solid. commit f6d1528 swaps the dead model for gemini-2.5-pro in both configs.ts and model.ts including that hardcoded fallback at line 130 that the reviewer caught which was honestly kinda nasty ngl. then theres commit 6187733 which is actually a really clean refactor they made this EXTERNAL_PROVIDER_DEFAULTS thing so everything pulls from one place now instead of hardcoded strings scattered everywhere which means future updates will be way easier. and they added tests in 7026dc9 like 183 lines covering gemini openai and codex including a test that specifically checks the old preview model is GONE which is pretty smart. BUT and this is a big but the tests have this spy bug in providerModelDefaults.test.ts at lines 142 154 167 177 they use spyOn(providers, 'getAPIProvider') but the mockRestore only runs if the test passes so if an assertion fails the spy never gets restored and the next test gets the wrong provider and everything goes chaotic which is probably why theres random test failures happening. the fix would be wrapping those in try/finally so mockRestore always runs or just adding spy cleanup to afterEach. also i checked the whole codebase for the old model name and the only place that mentions gemini-2.5-pro-preview-03-25 now is that negative assertion in the test file which is literally checking it doesnt exist so thats fine. all production code uses the new stable model. verdict is the pr fixes the issue and the refactor is genuinely good architecture but that test bug needs fixing before merge or the random failures will keep happening. also someone please add a newline at the end of the test file my ocd is screaming. ok thats it time to crash gn |
The gemini-2.5-pro-preview-03-25 model has been discontinued. Replaced the opus mapping with the stable gemini-2.5-pro version to ensure continued functionality. Reference: https://ai.google.dev/gemini-api/docs/deprecations?hl=ko#gemini-2.5-pro-models
Updated model.ts to use EXTERNAL_PROVIDER_DEFAULTS instead of hardcoded values. Added CODEX_MODEL_DEFAULTS and unified provider mappings in configs.ts.
Following the centralization of provider defaults, added comprehensive unit tests to ensure fallback logic and env overrides (e.g., OPENAI_MODEL, GEMINI_MODEL) work correctly for all supported 3P providers (Gemini, OpenAI, Codex, GitHub). Uses `bun:test` spies to mock the Codex provider state to avoid auth dependency issues during testing.
This comment was marked as low quality.
This comment was marked as low quality.
This comment was marked as low quality.
This comment was marked as low quality.
|
import { expect, spyOn, test, type Mock } from 'bun:test'
import {
getDefaultHaikuModel,
getDefaultOpusModel,
getDefaultSonnetModel,
getDefaultMainLoopModelSetting,
} from './model.js'
import * as providers from './providers.js'
const ISOLATED_ENV_KEYS = [
'ANTHROPIC_DEFAULT_OPUS_MODEL',
'ANTHROPIC_DEFAULT_SONNET_MODEL',
'ANTHROPIC_DEFAULT_HAIKU_MODEL',
'ANTHROPIC_MODEL',
'GEMINI_MODEL',
'OPENAI_MODEL',
'CLAUDE_CODE_USE_GEMINI',
'CLAUDE_CODE_USE_OPENAI',
] as const
type IsolatedEnvKey = typeof ISOLATED_ENV_KEYS[number]
// Note: Synchronous execution only. test.concurrent and async/await are prohibited to prevent race conditions.
function runWithSandbox(
envOverrides: Partial>,
testFn: (cleanupSpies: Mock[]) => void | unknown
) {
const backupEnv: Record = {}
const spiesToRestore: Mock[] = []
for (const key of ISOLATED_ENV_KEYS) {
backupEnv[key] = process.env[key]
delete process.env[key]
}
for (const key of ISOLATED_ENV_KEYS) {
const overrideValue = envOverrides[key]
if (overrideValue !== undefined) {
process.env[key] = overrideValue
}
}
try {
const result = testFn(spiesToRestore)
if (result instanceof Promise) {
throw new Error('runWithSandbox: testFn must be synchronous.')
}
} finally {
for (const key of ISOLATED_ENV_KEYS) {
const originalValue = backupEnv[key]
if (originalValue === undefined) {
delete process.env[key]
} else {
process.env[key] = originalValue
}
}
for (const spy of spiesToRestore) {
spy.mockRestore()
}
}
}
// --- Gemini Provider Tests ---
test('Gemini provider loads expected default models (Fallback logic check)', () => {
runWithSandbox({}, (spies) => {
const providerSpy = spyOn(providers, 'getAPIProvider').mockReturnValue('gemini')
spies.push(providerSpy)
expect(getDefaultOpusModel()).toBe('gemini-2.5-pro')
expect(getDefaultSonnetModel()).toBe('gemini-2.0-flash')
expect(getDefaultHaikuModel()).toBe('gemini-2.0-flash-lite')
})
})
test('Gemini provider does not reference discontinued preview model', () => {
runWithSandbox({}, (spies) => {
const providerSpy = spyOn(providers, 'getAPIProvider').mockReturnValue('gemini')
spies.push(providerSpy)
expect(getDefaultOpusModel()).not.toContain('gemini-2.5-pro-preview-03-25')
expect(getDefaultSonnetModel()).not.toContain('gemini-2.5-pro-preview-03-25')
expect(getDefaultHaikuModel()).not.toContain('gemini-2.5-pro-preview-03-25')
})
})
test('Gemini provider correctly applies GEMINI_MODEL environment override', () => {
runWithSandbox({ GEMINI_MODEL: 'gemini-override' }, (spies) => {
const providerSpy = spyOn(providers, 'getAPIProvider').mockReturnValue('gemini')
spies.push(providerSpy)
expect(getDefaultOpusModel()).toBe('gemini-override')
expect(getDefaultSonnetModel()).toBe('gemini-override')
expect(getDefaultHaikuModel()).toBe('gemini-override')
})
})
test('Gemini provider main loop setting defaults to sonnet', () => {
runWithSandbox({}, (spies) => {
const providerSpy = spyOn(providers, 'getAPIProvider').mockReturnValue('gemini')
spies.push(providerSpy)
expect(getDefaultMainLoopModelSetting()).toBe('gemini-2.0-flash')
})
})
test('Gemini provider main loop setting respects GEMINI_MODEL override', () => {
runWithSandbox({ GEMINI_MODEL: 'gemini-override' }, (spies) => {
const providerSpy = spyOn(providers, 'getAPIProvider').mockReturnValue('gemini')
spies.push(providerSpy)
expect(getDefaultMainLoopModelSetting()).toBe('gemini-override')
})
})
// --- OpenAI Provider Tests ---
test('OpenAI provider loads expected default models (Fallback logic check)', () => {
runWithSandbox({}, (spies) => {
const providerSpy = spyOn(providers, 'getAPIProvider').mockReturnValue('openai')
spies.push(providerSpy)
expect(getDefaultOpusModel()).toBe('gpt-4o')
expect(getDefaultSonnetModel()).toBe('gpt-4o')
expect(getDefaultHaikuModel()).toBe('gpt-4o-mini')
})
})
test('OpenAI provider correctly applies OPENAI_MODEL environment override', () => {
runWithSandbox({ OPENAI_MODEL: 'openai-override' }, (spies) => {
const providerSpy = spyOn(providers, 'getAPIProvider').mockReturnValue('openai')
spies.push(providerSpy)
expect(getDefaultOpusModel()).toBe('openai-override')
expect(getDefaultSonnetModel()).toBe('openai-override')
expect(getDefaultHaikuModel()).toBe('openai-override')
})
})
test('OpenAI provider main loop setting defaults to opus', () => {
runWithSandbox({}, (spies) => {
const providerSpy = spyOn(providers, 'getAPIProvider').mockReturnValue('openai')
spies.push(providerSpy)
expect(getDefaultMainLoopModelSetting()).toBe('gpt-4o')
})
})
test('OpenAI provider main loop setting respects OPENAI_MODEL override', () => {
runWithSandbox({ OPENAI_MODEL: 'openai-override' }, (spies) => {
const providerSpy = spyOn(providers, 'getAPIProvider').mockReturnValue('openai')
spies.push(providerSpy)
expect(getDefaultMainLoopModelSetting()).toBe('openai-override')
})
})
// --- Codex Provider Tests ---
test('Codex provider loads expected default models (Fallback logic check)', () => {
runWithSandbox({}, (spies) => {
const providerSpy = spyOn(providers, 'getAPIProvider').mockReturnValue('codex')
spies.push(providerSpy)
expect(getDefaultOpusModel()).toBe('gpt-5.4')
expect(getDefaultSonnetModel()).toBe('gpt-5.4')
expect(getDefaultHaikuModel()).toBe('gpt-5.4-mini')
})
})
test('Codex provider correctly applies OPENAI_MODEL environment override', () => {
runWithSandbox({ OPENAI_MODEL: 'codex-override' }, (spies) => {
const providerSpy = spyOn(providers, 'getAPIProvider').mockReturnValue('codex')
spies.push(providerSpy)
expect(getDefaultOpusModel()).toBe('codex-override')
expect(getDefaultSonnetModel()).toBe('codex-override')
expect(getDefaultHaikuModel()).toBe('codex-override')
})
})
test('Codex provider main loop setting defaults to opus', () => {
runWithSandbox({}, (spies) => {
const providerSpy = spyOn(providers, 'getAPIProvider').mockReturnValue('codex')
spies.push(providerSpy)
expect(getDefaultMainLoopModelSetting()).toBe('gpt-5.4')
})
})
test('Codex provider main loop setting respects OPENAI_MODEL override', () => {
runWithSandbox({ OPENAI_MODEL: 'codex-override' }, (spies) => {
const providerSpy = spyOn(providers, 'getAPIProvider').mockReturnValue('codex')
spies.push(providerSpy)
expect(getDefaultMainLoopModelSetting()).toBe('codex-override')
})
})
|
|
@Miyoko076 you were first, his pr was a duplicate |
4 participants
Fixes #398
The gemini-2.5-pro-preview-03-25 model has been discontinued. Replaced the opus mapping with the stable gemini-2.5-pro version to ensure continued functionality.
Reference: https://ai.google.dev/gemini-api/docs/deprecations#gemini-2.5-pro-models
Summary
opustier mapping inGEMINI_MODEL_DEFAULTS(configs.ts) to usegemini-2.5-pro.Impact
opustier. Ensures stable model generation.Testing
bun run buildbun run smokebun test ./src/utils/model/providerModelDefaults.test.tsNotes