From fb859991c011af88273826fbcf223a9891f29d0a Mon Sep 17 00:00:00 2001 From: konard Date: Sat, 14 Feb 2026 21:30:36 +0100 Subject: [PATCH 1/4] Initial commit with task details Adding CLAUDE.md with task information for AI processing. This file will be removed when the task is complete. Issue: https://github.com/link-assistant/agent/issues/175 --- CLAUDE.md | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 CLAUDE.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..261e263 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,5 @@ +Issue to solve: https://github.com/link-assistant/agent/issues/175 +Your prepared branch: issue-175-6f5f5eeae855 +Your prepared working directory: /tmp/gh-issue-solver-1771101034402 + +Proceed. From 03f568b2b481aa0161e615762a539a2c7a42eab0 Mon Sep 17 00:00:00 2001 From: konard Date: Sat, 14 Feb 2026 21:47:40 +0100 Subject: [PATCH 2/4] fix: await models.dev cache refresh when stale or missing This fixes ProviderModelNotFoundError when using short model names like `kimi-k2.5-free` on fresh installations or after new models are added. The root cause was a race condition in ModelsDev.get() where the refresh() call was not awaited, causing the agent to use stale or empty cache data before the refresh completed. Changes: - Await refresh() when cache file doesn't exist (first run) - Await refresh() when cache is stale (> 1 hour old) - Only trigger background refresh when cache is fresh - Add detailed logging for cache state debugging - Add unit tests for ModelsDev module - Add case study documentation Fixes #175 Co-Authored-By: Claude Opus 4.5 --- docs/case-studies/issue-175/README.md | 250 ++++++++++++++++++ .../issue-175/data/issue-175.json | 66 +++++ js/src/provider/models.ts | 65 ++++- js/tests/models-cache.test.js | 129 +++++++++ 4 files changed, 509 insertions(+), 1 deletion(-) create mode 100644 docs/case-studies/issue-175/README.md create mode 100644 docs/case-studies/issue-175/data/issue-175.json create mode 100644 js/tests/models-cache.test.js diff --git a/docs/case-studies/issue-175/README.md b/docs/case-studies/issue-175/README.md new file mode 100644 index 0000000..b7b8858 --- /dev/null +++ b/docs/case-studies/issue-175/README.md @@ -0,0 +1,250 @@ +# Case Study: Issue #175 - `echo 'hi' | agent --model kimi-k2.5-free` Not Working on macOS + +## Summary + +When running `echo 'hi' | agent --model kimi-k2.5-free` on macOS, the agent throws a `ProviderModelNotFoundError` indicating that the model cannot be found, even though: +1. The `kimi-k2.5-free` model exists in the models.dev API +2. The OpenCode provider is correctly found +3. The model has `cost.input === 0` (free tier) + +## Timeline of Events + +### 2026-02-14T20:28:55.205Z - Agent Started +User installs and runs: +```bash +bun install -g @link-assistant/agent # version 0.13.1 +echo 'hi' | agent --model kimi-k2.5-free +``` + +### 2026-02-14T20:28:55.206Z - Instance Creation +```json +{"service":"default","directory":"/Users/konard","message":"creating instance"} +{"service":"project","directory":"/Users/konard","message":"fromDirectory"} +``` + +### 2026-02-14T20:28:55.214Z - Provider State Initialization +```json +{"service":"provider","status":"started","message":"state"} +``` + +### 2026-02-14T20:28:55.215Z - Config Loading +Multiple config files are checked: +- `/Users/konard/.config/link-assistant-agent/config.json` +- `/Users/konard/.config/link-assistant-agent/opencode.json` +- `/Users/konard/.config/link-assistant-agent/opencode.jsonc` + +### 2026-02-14T20:28:55.218Z - Models.dev Refresh (Non-blocking) +```json +{"service":"models.dev","file":{},"message":"refreshing"} +``` +**Note**: The `refresh()` call is not awaited, leading to potential race conditions with stale cache. + +### 2026-02-14T20:28:55.224Z - Providers Found +Both OpenCode and Kilo providers are discovered: +```json +{"service":"provider","providerID":"opencode","message":"found"} +{"service":"provider","providerID":"kilo","message":"found"} +``` + +### 2026-02-14T20:28:55.224Z - Model Resolution Failed +The short model name resolution failed: +```json +{ + "level": "warn", + "service": "provider", + "modelID": "kimi-k2.5-free", + "message": "unable to resolve short model name, using opencode as default" +} +``` + +This fallback behavior then led to: +```json +{ + "input": "kimi-k2.5-free", + "providerID": "opencode", + "modelID": "kimi-k2.5-free", + "message": "resolved short model name" +} +``` + +### 2026-02-14T20:28:55.246Z - Error Thrown +```json +{ + "level": "error", + "service": "session.prompt", + "providerID": "opencode", + "modelID": "kimi-k2.5-free", + "error": "ProviderModelNotFoundError", + "message": "Failed to initialize specified model - NOT falling back to default (explicit provider specified)" +} +``` + +## Root Cause Analysis + +### Primary Root Cause: Race Condition in Model Database Loading + +The `ModelsDev.get()` function has a race condition: + +```typescript +// js/src/provider/models.ts:70-77 +export async function get() { + refresh(); // NOT AWAITED - async refresh runs in background + const file = Bun.file(filepath); + const result = await file.json().catch(() => {}); + if (result) return result as Record; + const json = await data(); + return JSON.parse(json) as Record; +} +``` + +The `refresh()` function fetches the latest models from `https://models.dev/api.json` but is **not awaited**. This means: +1. If the cache file doesn't exist or is stale, old/missing data is used +2. The fresh data arrives after the model lookup has already failed +3. Users with new installations or outdated caches will encounter this error + +### Secondary Issue: No Fallback for New Installations + +When a user runs the agent for the first time: +1. The cache file at `~/.cache/link-assistant-agent/models.json` doesn't exist +2. The `data()` function is called which fetches from models.dev synchronously +3. However, if this fetch fails or is slow, the model lookup fails + +### Evidence from logs + +1. **Provider found but model not found**: The `opencode` provider was found (line: `"message":"found","providerID":"opencode"`), but when `resolveShortModelName("kimi-k2.5-free")` was called, it returned `undefined` because the model wasn't in `provider.info.models`. + +2. **Stale cache or missing model**: The `kimi-k2.5-free` model definitely exists in the current models.dev API (verified), but the user's cached data may not have contained it. + +### Verification + +Current models.dev API confirms `kimi-k2.5-free` exists: +```json +{ + "opencode": { + "id": "opencode", + "name": "OpenCode Zen", + "api": "https://opencode.ai/zen/v1", + "npm": "@ai-sdk/openai-compatible", + "env": ["OPENCODE_API_KEY"], + "models": { + "kimi-k2.5-free": { + "id": "kimi-k2.5-free", + "name": "Kimi K2.5 Free", + "cost": { "input": 0, "output": 0 } + } + } + } +} +``` + +## Related Issues in Upstream Repository + +### OpenCode (anomalyco/opencode) + +1. **[#12045](https://github.com/anomalyco/opencode/issues/12045)** - "ProviderModelNotFoundError when using GLM or Kimi" + - Status: OPEN + - Similar issue in GitHub Actions workflow + +2. **[#11591](https://github.com/anomalyco/opencode/issues/11591)** - "Kimi K2.5 Free OpenCode Zen throws an error" + - Status: OPEN + - Different error (JSON Schema validation) but same model + +3. **[#3046](https://github.com/sst/opencode/issues/3046)** - "ProviderModelNotFoundError when using Kimi K2 model in GitHub workflow" + - Status: CLOSED + - Confirmed similar root cause + +## Proposed Solutions + +### Solution 1: Await the Refresh Before Model Lookup (Recommended) + +Modify `ModelsDev.get()` to await the refresh when the cache is stale or missing: + +```typescript +export async function get() { + const file = Bun.file(filepath); + + // Check if cache exists and is recent (< 1 hour old) + const exists = await file.exists(); + const stats = exists ? await file.stat() : null; + const isStale = !stats || Date.now() - stats.mtime.getTime() > 60 * 60 * 1000; + + if (isStale) { + // AWAIT the refresh for stale/missing cache + await refresh(); + } else { + // Trigger background refresh for fresh cache + refresh(); + } + + const result = await file.json().catch(() => {}); + if (result) return result as Record; + const json = await data(); + return JSON.parse(json) as Record; +} +``` + +### Solution 2: Better Fallback Error Messages + +When a model can't be found, provide helpful suggestions: + +```typescript +export async function getModel(providerID: string, modelID: string) { + // ... existing code ... + + if (!info) { + const suggestion = await findSimilarModel(modelID); + throw new ModelNotFoundError({ + providerID, + modelID, + suggestion: suggestion || `Run 'agent models --refresh' to update model list`, + helpUrl: 'https://opencode.ai/docs/troubleshooting/#providermodelnotfounderror' + }); + } +} +``` + +### Solution 3: Add `--refresh` Flag for Model List + +Allow users to force a refresh of the model database: + +```bash +# Refresh models before running +agent --model kimi-k2.5-free --refresh-models + +# Or as a separate command +agent models --refresh +``` + +## Workarounds for Users + +### Workaround 1: Use Explicit Provider/Model Format +```bash +echo 'hi' | agent --model opencode/kimi-k2.5-free +``` + +### Workaround 2: Force Cache Refresh +```bash +rm -rf ~/.cache/link-assistant-agent/models.json +agent models --refresh +echo 'hi' | agent --model kimi-k2.5-free +``` + +### Workaround 3: Set OPENCODE_API_KEY +```bash +export OPENCODE_API_KEY="your-api-key" +echo 'hi' | agent --model kimi-k2.5-free +``` + +## References + +- [OpenCode Zen Documentation](https://opencode.ai/docs/zen/) +- [OpenCode Troubleshooting - ProviderModelNotFoundError](https://opencode.ai/docs/troubleshooting/) +- [Models.dev API](https://models.dev/api.json) +- [Related Issue: anomalyco/opencode#12045](https://github.com/anomalyco/opencode/issues/12045) +- [Related Issue: anomalyco/opencode#11591](https://github.com/anomalyco/opencode/issues/11591) + +## Files Involved + +- `js/src/provider/models.ts` - Model database loading with race condition +- `js/src/provider/provider.ts` - Provider state and model resolution logic +- `js/src/util/error.ts` - NamedError class for ProviderModelNotFoundError diff --git a/docs/case-studies/issue-175/data/issue-175.json b/docs/case-studies/issue-175/data/issue-175.json new file mode 100644 index 0000000..c9859e7 --- /dev/null +++ b/docs/case-studies/issue-175/data/issue-175.json @@ -0,0 +1,66 @@ +{ + "issue_number": 175, + "title": "`echo 'hi' | agent --model kimi-k2.5-free` is not working on macOS", + "reporter": "konard", + "reported_at": "2026-02-14T20:28:55Z", + "labels": ["bug"], + "environment": { + "os": "macOS", + "bun_version": "1.2.20", + "agent_version": "0.13.1" + }, + "command": "echo 'hi' | agent --model kimi-k2.5-free", + "error": { + "type": "ProviderModelNotFoundError", + "data": { + "providerID": "opencode", + "modelID": "kimi-k2.5-free" + }, + "stack_trace": [ + "at new NamedError (1:23)", + "at new ProviderModelNotFoundError (/Users/konard/.bun/install/global/node_modules/@link-assistant/agent/src/util/error.ts:28:9)", + "at getModel (/Users/konard/.bun/install/global/node_modules/@link-assistant/agent/src/provider/provider.ts:1253:13)" + ] + }, + "relevant_logs": [ + { + "type": "log", + "level": "info", + "service": "provider", + "providerID": "opencode", + "message": "found" + }, + { + "type": "log", + "level": "info", + "service": "provider", + "providerID": "kilo", + "message": "found" + }, + { + "type": "log", + "level": "warn", + "service": "provider", + "modelID": "kimi-k2.5-free", + "message": "unable to resolve short model name, using opencode as default" + }, + { + "type": "log", + "level": "info", + "service": "default", + "input": "kimi-k2.5-free", + "providerID": "opencode", + "modelID": "kimi-k2.5-free", + "message": "resolved short model name" + }, + { + "type": "log", + "level": "error", + "service": "session.prompt", + "providerID": "opencode", + "modelID": "kimi-k2.5-free", + "error": "ProviderModelNotFoundError", + "message": "Failed to initialize specified model - NOT falling back to default (explicit provider specified)" + } + ] +} diff --git a/js/src/provider/models.ts b/js/src/provider/models.ts index ae3a507..eabada3 100644 --- a/js/src/provider/models.ts +++ b/js/src/provider/models.ts @@ -67,11 +67,74 @@ export namespace ModelsDev { export type Provider = z.infer; + /** + * Cache staleness threshold in milliseconds (1 hour). + * If the cache is older than this, we await the refresh before using the data. + */ + const CACHE_STALE_THRESHOLD_MS = 60 * 60 * 1000; + + /** + * Get the models database, refreshing from models.dev if needed. + * + * This function handles cache staleness properly: + * - If cache doesn't exist: await refresh to ensure fresh data + * - If cache is stale (> 1 hour old): await refresh to ensure up-to-date models + * - If cache is fresh: trigger background refresh but use cached data immediately + * + * This prevents ProviderModelNotFoundError when: + * - User runs agent for the first time (no cache) + * - User has outdated cache missing new models like kimi-k2.5-free + * + * @see https://github.com/link-assistant/agent/issues/175 + */ export async function get() { - refresh(); const file = Bun.file(filepath); + + // Check if cache exists and get its modification time + const exists = await file.exists(); + + if (!exists) { + // No cache - must await refresh to get initial data + log.info(() => ({ + message: 'no cache found, awaiting refresh', + path: filepath, + })); + await refresh(); + } else { + // Check if cache is stale + const stats = await file.stat().catch(() => null); + const mtime = stats?.mtime?.getTime() ?? 0; + const isStale = Date.now() - mtime > CACHE_STALE_THRESHOLD_MS; + + if (isStale) { + // Stale cache - await refresh to get updated model list + log.info(() => ({ + message: 'cache is stale, awaiting refresh', + path: filepath, + age: Date.now() - mtime, + threshold: CACHE_STALE_THRESHOLD_MS, + })); + await refresh(); + } else { + // Fresh cache - trigger background refresh but don't wait + log.info(() => ({ + message: 'cache is fresh, triggering background refresh', + path: filepath, + age: Date.now() - mtime, + })); + refresh(); + } + } + + // Now read the cache file const result = await file.json().catch(() => {}); if (result) return result as Record; + + // Fallback to bundled data if cache read failed + log.warn(() => ({ + message: 'cache read failed, using bundled data', + path: filepath, + })); const json = await data(); return JSON.parse(json) as Record; } diff --git a/js/tests/models-cache.test.js b/js/tests/models-cache.test.js new file mode 100644 index 0000000..ab7b8eb --- /dev/null +++ b/js/tests/models-cache.test.js @@ -0,0 +1,129 @@ +import { test, expect, describe, setDefaultTimeout } from 'bun:test'; +// @ts-ignore +import { sh } from 'command-stream'; + +// Increase default timeout for network operations +setDefaultTimeout(60000); + +/** + * Test suite for models.dev cache handling - Issue #175 + * + * These tests verify that the ModelsDev.get() function properly handles: + * 1. Missing cache files (first run) + * 2. Stale cache files (> 1 hour old) + * 3. Fresh cache files (< 1 hour old) + * + * The fix ensures that when the cache is missing or stale, we await the refresh + * before proceeding, preventing ProviderModelNotFoundError for newly added models. + * + * @see https://github.com/link-assistant/agent/issues/175 + */ +describe('ModelsDev cache handling', () => { + const projectRoot = process.cwd(); + + test('echo provider works in dry-run mode', async () => { + // This tests that the echo provider works correctly in dry-run mode + // We use --dry-run to enable the echo provider + + const input = '{"message":"hi"}'; + const result = await sh( + `echo '${input}' | bun run ${projectRoot}/src/index.js --dry-run --no-always-accept-stdin` + ); + const output = result.stdout + result.stderr; + + // Should NOT get ProviderModelNotFoundError + expect(output).not.toContain('ProviderModelNotFoundError'); + + // Should see the model being used (echo provider outputs the input) + expect(output).toContain('echo') || expect(output).toContain('hi'); + }); + + test('models.dev refresh is triggered on startup', async () => { + // Test that the models.dev cache is being handled + // We use --dry-run to avoid API calls + + const input = '{"message":"hi"}'; + const result = await sh( + `echo '${input}' | bun run ${projectRoot}/src/index.js --dry-run --no-always-accept-stdin` + ); + const output = result.stdout + result.stderr; + + // Should see cache-related log messages + const hasCacheHandling = + output.includes('cache is fresh') || + output.includes('cache is stale') || + output.includes('no cache found') || + output.includes('refreshing') || + output.includes('models.dev'); + + expect(hasCacheHandling).toBe(true); + }); +}); + +/** + * Unit tests for ModelsDev module directly + */ +describe('ModelsDev module', () => { + test('ModelsDev.get() returns provider data', async () => { + const { ModelsDev } = await import('../src/provider/models.ts'); + + // Get the models database + const database = await ModelsDev.get(); + + // Should have providers + expect(database).toBeTruthy(); + expect(typeof database).toBe('object'); + + // Should have the opencode provider + expect(database['opencode']).toBeTruthy(); + expect(database['opencode'].id).toBe('opencode'); + + // opencode should have models + expect(database['opencode'].models).toBeTruthy(); + expect(Object.keys(database['opencode'].models).length).toBeGreaterThan(0); + }); + + test('opencode provider should have kimi-k2.5-free model', async () => { + const { ModelsDev } = await import('../src/provider/models.ts'); + + // Get the models database + const database = await ModelsDev.get(); + + // Check for kimi-k2.5-free model + const opencode = database['opencode']; + expect(opencode).toBeTruthy(); + + // The model should exist (this is the bug we're fixing) + const kimiModel = opencode.models['kimi-k2.5-free']; + expect(kimiModel).toBeTruthy(); + expect(kimiModel.name).toContain('Kimi'); + expect(kimiModel.cost.input).toBe(0); // Should be free + }); + + test('opencode provider should have other free models', async () => { + const { ModelsDev } = await import('../src/provider/models.ts'); + + // Get the models database + const database = await ModelsDev.get(); + + const opencode = database['opencode']; + expect(opencode).toBeTruthy(); + + // Check for other free models (cost.input === 0) + const freeModels = Object.entries(opencode.models) + .filter(([_, model]) => model.cost.input === 0) + .map(([id]) => id); + + // Should have multiple free models + expect(freeModels.length).toBeGreaterThan(0); + + // Some known free models that should exist + const expectedFreeModels = ['grok-code', 'gpt-5-nano', 'big-pickle']; + for (const modelId of expectedFreeModels) { + const model = opencode.models[modelId]; + if (model) { + expect(model.cost.input).toBe(0); + } + } + }); +}); From a58734d6954985847263b629ea823eb8ae9c2459 Mon Sep 17 00:00:00 2001 From: konard Date: Sat, 14 Feb 2026 21:49:40 +0100 Subject: [PATCH 3/4] chore: Add changeset for models.dev cache race condition fix Co-Authored-By: Claude Opus 4.5 --- .../fix-models-cache-race-condition.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 js/.changeset/fix-models-cache-race-condition.md diff --git a/js/.changeset/fix-models-cache-race-condition.md b/js/.changeset/fix-models-cache-race-condition.md new file mode 100644 index 0000000..2db055c --- /dev/null +++ b/js/.changeset/fix-models-cache-race-condition.md @@ -0,0 +1,17 @@ +--- +'@link-assistant/agent': patch +--- + +Fix ProviderModelNotFoundError for newly added models like kimi-k2.5-free + +When the models.dev cache was missing or stale, the refresh() call was not awaited, +causing the agent to use outdated/empty cache data. This led to ProviderModelNotFoundError +for models that exist in the remote API but weren't in the local cache. + +The fix ensures that: + +- When no cache exists (first run): await refresh() before proceeding +- When cache is stale (> 1 hour old): await refresh() to get updated model list +- When cache is fresh: trigger background refresh but use cached data immediately + +Fixes #175 From f40bbb27dbfd31952e954b15b9490152be25636d Mon Sep 17 00:00:00 2001 From: konard Date: Sat, 14 Feb 2026 21:52:12 +0100 Subject: [PATCH 4/4] Revert "Initial commit with task details" This reverts commit fb859991c011af88273826fbcf223a9891f29d0a. --- CLAUDE.md | 5 ----- 1 file changed, 5 deletions(-) delete mode 100644 CLAUDE.md diff --git a/CLAUDE.md b/CLAUDE.md deleted file mode 100644 index 261e263..0000000 --- a/CLAUDE.md +++ /dev/null @@ -1,5 +0,0 @@ -Issue to solve: https://github.com/link-assistant/agent/issues/175 -Your prepared branch: issue-175-6f5f5eeae855 -Your prepared working directory: /tmp/gh-issue-solver-1771101034402 - -Proceed.