-
Notifications
You must be signed in to change notification settings - Fork 63
Description
Description
When using evalite with vitest 4.x, if an eval test throws an error, the error rendering crashes with:
TypeError: Cannot read properties of undefined (reading 'pool')
This masks the actual underlying error, making debugging impossible.
Root Cause
In node_modules/vitest/dist/chunks/index.456_DGfR.js:2983:
if (options.task?.file.pool === "browser" && project.browser)The optional chaining on task doesn't protect against task.file being undefined. When task exists but task.file is undefined, accessing .pool throws.
This is triggered from evalite's reporter in renderErrorsSummary (node_modules/evalite/dist/reporter/rendering.js:75).
Stack Trace
TypeError: Cannot read properties of undefined (reading 'pool')
at Object.parseErrorStacktrace (file:///node_modules/vitest/dist/chunks/index.456_DGfR.js:2983:26)
at printErrorInner (file:///node_modules/vitest/dist/chunks/index.456_DGfR.js:3015:25)
at printErrorInner (file:///node_modules/vitest/dist/chunks/index.456_DGfR.js:3064:3)
at printError (file:///node_modules/vitest/dist/chunks/index.456_DGfR.js:2970:9)
at Logger.printError (file:///node_modules/vitest/dist/chunks/cli-api.C7sYjHmQ.js:6551:3)
at renderErrorsSummary (file:///node_modules/evalite/dist/reporter/rendering.js:75:24)
at EvaliteReporter.onTestRunEnd (file:///node_modules/evalite/dist/reporter.js:134:9)
Reproduction Steps
- Create a minimal evalite setup with vitest 4.x:
{
"devDependencies": {
"@modelcontextprotocol/sdk": "^1.24.3",
"evalite": "0.19.0",
"vitest": "4.0.15"
}
}- Create an eval that makes an HTTP request that fails (e.g., MCP client connecting to a server that isn't running):
// evals/example.eval.ts
import { evalite } from 'evalite'
import { Client } from '@modelcontextprotocol/sdk/client/index.js'
import { StreamableHTTPClientTransport } from '@modelcontextprotocol/sdk/client/streamableHttp.js'
evalite('mcp-eval', {
data: async () => [{ input: 'test' }],
task: async () => {
// This will throw: TypeError: fetch failed
// Caused by: Error: connect ECONNREFUSED 127.0.0.1:443
const client = new Client({ name: 'test', version: '1.0.0' })
const transport = new StreamableHTTPClientTransport(
new URL('https://localhost/mcp')
)
await client.connect(transport)
return 'done'
},
scorers: [],
})-
Run
npx evalite -
Instead of seeing the actual error:
TypeError: fetch failed
❯ StreamableHTTPClientTransport.send node_modules/@modelcontextprotocol/sdk/dist/esm/client/streamableHttp.js:315:30
Caused by: Error: connect ECONNREFUSED 127.0.0.1:443
Serialized Error: { errno: -61, code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 443 }
You get the unhelpful:
TypeError: Cannot read properties of undefined (reading 'pool')
Environment
- evalite: 0.19.0
- vitest: 4.0.15
- Node: 24.x
Expected Behavior
Evalite should properly display the actual error from the failing eval task (the ECONNREFUSED error), not crash while trying to render it.
Suggested Fix
When calling vitest's printError / parseErrorStacktrace, ensure the task object has the required file property, or pass undefined for the task if file information is not available.