Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion .changeset/config.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,10 @@
"linked": [],
"access": "public",
"baseBranch": "main",
"ignore": ["example", "evalite-tests", "evalite-docs", "evalite-ui"]
"ignore": [
"example",
"evaluhealth-tests",
"evaluhealth-docs",
"evaluhealth-ui"
]
}
5 changes: 0 additions & 5 deletions .changeset/tasty-parents-bathe.md

This file was deleted.

2 changes: 1 addition & 1 deletion .github/workflows/preview.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,4 +24,4 @@ jobs:
- run: pnpm build

- name: Publish preview releases
run: pnpx pkg-pr-new publish packages/evalite
run: pnpx pkg-pr-new publish packages/evaluhealth
6 changes: 3 additions & 3 deletions .prettierignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@ dist
.turbo
*.tsbuildinfo
coverage
.evalite
evalite.db
.evaluhealth
evaluhealth.db
pnpm-lock.yaml
**/tests/playground/**
build
vite.config.ts.timestamp*
*.d.ts
*.gen.ts
**/evalite-export
**/evaluhealth-export
94 changes: 47 additions & 47 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,24 +4,24 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co

## Overview

Evalite is a TypeScript-native, local-first tool for testing LLM-powered apps built on Vitest. It allows developers to write evaluations (evals) as `.eval.ts` files that run like tests.
Evaluhealth is a TypeScript-native, local-first tool for testing LLM-powered apps built on Vitest. It allows developers to write evaluations (evals) as `.eval.ts` files that run like tests.

## Configuration

The primary configuration method is `evalite.config.ts`. While `vitest.config.ts` is still supported for backward compatibility, it is not documented and `evalite.config.ts` should be used for all configuration needs.
The primary configuration method is `evaluhealth.config.ts`. While `vitest.config.ts` is still supported for backward compatibility, it is not documented and `evaluhealth.config.ts` should be used for all configuration needs.

## Development Commands

**Development mode** (recommended for working on Evalite itself):
**Development mode** (recommended for working on Evaluhealth itself):

```bash
pnpm run dev
```

This runs:

- TypeScript type checker on `evalite` package
- Tests in `evalite-tests` package
- TypeScript type checker on `evaluhealth` package
- Tests in `evaluhealth-tests` package
- Live reload for both packages

**Build all packages**:
Expand All @@ -30,7 +30,7 @@ This runs:
pnpm build
```

This builds `evalite` package first, then `evalite-ui`, copying UI assets to `packages/evalite/dist/ui`.
This builds `evaluhealth` package first, then `evaluhealth-ui`, copying UI assets to `packages/evaluhealth/dist/ui`.

**Run CI pipeline** (build, test, lint):

Expand All @@ -42,20 +42,20 @@ pnpm ci

```bash
pnpm run example
# Or: cd packages/example && pnpm evalite watch
# Or: cd packages/example && pnpm evaluhealth watch
```

**Run single package tests**:

```bash
cd packages/evalite && pnpm test
cd packages/evalite-tests && pnpm test
cd packages/evaluhealth && pnpm test
cd packages/evaluhealth-tests && pnpm test
```

**Lint a package**:

```bash
cd packages/evalite && pnpm lint
cd packages/evaluhealth && pnpm lint
```

## Working with pnpm Filters
Expand All @@ -67,64 +67,64 @@ When working on specific packages in this monorepo, **use pnpm's `--filter` flag
**Build a specific package**:

```bash
pnpm --filter evalite build
pnpm --filter evalite-ui build
pnpm --filter evaluhealth build
pnpm --filter evaluhealth-ui build
```

**Run tests for a specific package**:

```bash
pnpm --filter evalite-tests test
pnpm --filter evaluhealth-tests test
```

**Run dev mode for a specific package**:

```bash
pnpm --filter evalite dev
pnpm --filter evalite-ui dev
pnpm --filter evaluhealth dev
pnpm --filter evaluhealth-ui dev
```

**Lint a specific package**:

```bash
pnpm --filter evalite lint
pnpm --filter evalite-tests lint
pnpm --filter evaluhealth lint
pnpm --filter evaluhealth-tests lint
```

### Filter Patterns

pnpm supports several filter patterns:

- `--filter evalite` - Run task for the `evalite` package only
- `--filter evalite...` - Run task for `evalite` and all its dependencies
- `--filter ...evalite` - Run task for `evalite` and all packages that depend on it
- `--filter evaluhealth` - Run task for the `evaluhealth` package only
- `--filter evaluhealth...` - Run task for `evaluhealth` and all its dependencies
- `--filter ...evaluhealth` - Run task for `evaluhealth` and all packages that depend on it
- `--filter "./packages/*"` - Run task for all packages in the packages directory
- `--filter "!evalite"` - Run task for all packages except `evalite`
- `--filter "!evaluhealth"` - Run task for all packages except `evaluhealth`

### Examples for Common Workflows

**Working on the main evalite package**:
**Working on the main evaluhealth package**:

```bash
# Build evalite and watch for changes
pnpm --filter evalite dev
# Build evaluhealth and watch for changes
pnpm --filter evaluhealth dev

# Run tests after making changes
pnpm --filter evalite test
pnpm --filter evaluhealth test
```

**Working on the UI**:

```bash
# Build evalite first, then start UI dev server
pnpm run build:evalite && pnpm --filter evalite-ui dev
# Build evaluhealth first, then start UI dev server
pnpm run build:evaluhealth && pnpm --filter evaluhealth-ui dev
```

**Working on integration tests**:

```bash
# Ensure evalite is built before running tests
pnpm run build && pnpm --filter evalite-tests test
# Ensure evaluhealth is built before running tests
pnpm run build && pnpm --filter evaluhealth-tests test
```

### When to Use Filters
Expand All @@ -138,7 +138,7 @@ pnpm run build && pnpm --filter evalite-tests test
**Direct package commands are fine for**:

- Quick one-off commands (like `pnpm install`)
- Running the evalite CLI itself (e.g., `cd packages/example && pnpm evalite watch`)
- Running the evaluhealth CLI itself (e.g., `cd packages/example && pnpm evaluhealth watch`)
- When already in the package directory

## Architecture
Expand All @@ -147,21 +147,21 @@ pnpm run build && pnpm --filter evalite-tests test

This is a pnpm workspace:

- **`packages/evalite`**: Main package that users install. Exports the `evalite()` function, CLI binary (`evalite`), server, database layer, and utilities. Built with TypeScript.
- **`packages/evaluhealth`**: Main package that users install. Exports the `evaluhealth()` function, CLI binary (`evaluhealth`), server, database layer, and utilities. Built with TypeScript.

- **`packages/evalite-core`**: Shared core utilities (currently appears to be deprecated or minimal)
- **`packages/evaluhealth-core`**: Shared core utilities (currently appears to be deprecated or minimal)

- **`packages/evalite-tests`**: Integration tests for evalite functionality
- **`packages/evaluhealth-tests`**: Integration tests for evaluhealth functionality

- **`packages/example`**: Example eval files demonstrating usage patterns (e.g., `example.eval.ts`, `traces.eval.ts`)

- **`apps/evalite-ui`**: React-based web UI that displays eval results. Built with Vite, TanStack Router, and Tailwind. Gets copied to `packages/evalite/dist/ui` during build via the `after-build` script.
- **`apps/evaluhealth-ui`**: React-based web UI that displays eval results. Built with Vite, TanStack Router, and Tailwind. Gets copied to `packages/evaluhealth/dist/ui` during build via the `after-build` script.

- **`apps/evalite-docs`**: Documentation site
- **`apps/evaluhealth-docs`**: Documentation site

### Core Concepts

**Eval files**: Files matching `*.eval.ts` (or `.eval.mts`) that contain `evalite()` calls. These define:
**Eval files**: Files matching `*.eval.ts` (or `.eval.mts`) that contain `evaluhealth()` calls. These define:

- A dataset (via `data()` function returning input/expected pairs)
- A task (the LLM interaction to test)
Expand All @@ -170,49 +170,49 @@ This is a pnpm workspace:

**Execution flow**:

1. The `evalite` CLI uses Vitest under the hood to discover and run `*.eval.ts` files
1. The `evaluhealth` CLI uses Vitest under the hood to discover and run `*.eval.ts` files
2. Each eval creates a Vitest `describe` block with concurrent `it` tests for each data point
3. Results are stored in a SQLite database (`evalite.db`)
3. Results are stored in a SQLite database (`evaluhealth.db`)
4. A Fastify server serves the UI and provides WebSocket updates during runs
5. Files (images, audio, etc.) are saved to `.evalite` directory
5. Files (images, audio, etc.) are saved to `.evaluhealth` directory

**Key architecture points**:

- Uses Vitest's `inject("cwd")` to get the working directory
- Supports async iterables (streaming) from tasks via `executeTask()`
- Files in input/output/expected are automatically detected and saved using `createEvaliteFileIfNeeded()`
- Files in input/output/expected are automatically detected and saved using `createEvaluhealthFileIfNeeded()`
- Traces can be reported via `reportTraceLocalStorage` for nested LLM calls
- Integrates with AI SDK via `evalite/ai-sdk` export (provides `traceAISDKModel()`)
- Integrates with AI SDK via `evaluhealth/ai-sdk` export (provides `traceAISDKModel()`)

### Database Layer

SQLite database (`evalite.db`) stores:
SQLite database (`evaluhealth.db`) stores:

- Runs (full or partial)
- Evals (distinct eval names with metadata)
- Results (individual test case results with scores, traces, columns)
- Scores and traces are stored as JSON

Key queries in `packages/evalite/src/db.ts`:
Key queries in `packages/evaluhealth/src/db.ts`:

- `getEvals()`, `getResults()`, `getScores()`, `getTraces()`
- `getMostRecentRun()`, `getPreviousCompletedEval()`

### Server & UI

The Fastify server in `packages/evalite/src/server.ts`:
The Fastify server in `packages/evaluhealth/src/server.ts`:

- Serves the UI from `dist/ui/`
- Provides REST API at `/api/*` (menu-items, server-state, evals, results, etc.)
- WebSocket endpoint at `/api/socket` for live updates during eval runs

## Important Notes

**Linking for local development**: If you need to test the global `evalite` command locally:
**Linking for local development**: If you need to test the global `evaluhealth` command locally:

```bash
pnpm build
cd packages/evalite && npm link
cd packages/evaluhealth && npm link
```

**Node version**: Requires Node.js >= 22
Expand All @@ -235,7 +235,7 @@ The format of the file should be:

```md
---
"evalite": patch
"evaluhealth": patch
---

Description of the change.
Expand Down
16 changes: 8 additions & 8 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Contributing to Evalite
# Contributing to Evaluhealth

## Development Commands

Expand All @@ -7,25 +7,25 @@ This monorepo uses pnpm workspaces (no Turbo). All scripts use `pnpm --filter` t
**Available scripts:**

```bash
pnpm run dev # Build evalite + UI, then run tsc -w on evalite + vitest on evalite-tests
pnpm run example # Build, then run evalite watch + UI dev server (parallel)
pnpm run test # Build, then run tests on evalite + evalite-tests
pnpm run build # Build evalite, then evalite-ui
pnpm run dev # Build evaluhealth + UI, then run tsc -w on evaluhealth + vitest on evaluhealth-tests
pnpm run example # Build, then run evaluhealth watch + UI dev server (parallel)
pnpm run test # Build, then run tests on evaluhealth + evaluhealth-tests
pnpm run build # Build evaluhealth, then evaluhealth-ui
pnpm run ci # Full CI: build, test, lint, check-format
```

**Individual package scripts:**

```bash
pnpm build:evalite # Build evalite package only
pnpm build:evalite-ui # Build UI and copy to evalite/dist/ui
pnpm build:evaluhealth # Build evaluhealth package only
pnpm build:evaluhealth-ui # Build UI and copy to evaluhealth/dist/ui
```

**Setup:**

1. Create `.env` in `packages/example` with `OPENAI_API_KEY=your-key`
2. Run `pnpm install`
3. For global `evalite` command: `pnpm build && cd packages/evalite && npm link`
3. For global `evaluhealth` command: `pnpm build && cd packages/evaluhealth && npm link`

## Styling Guidelines

Expand Down
10 changes: 0 additions & 10 deletions apps/evalite-docs/src/assets/logo-dark.svg

This file was deleted.

10 changes: 0 additions & 10 deletions apps/evalite-docs/src/assets/logo-light.svg

This file was deleted.

15 changes: 0 additions & 15 deletions apps/evalite-ui/README.md

This file was deleted.

File renamed without changes.
File renamed without changes.
File renamed without changes.
Loading