Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
249 changes: 249 additions & 0 deletions .agents/skills/pytest/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,249 @@
# pytest Skill Reference

## Project Convention: Top-Level Test Functions

**RULE**: Always write pytest tests as top-level functions, NOT grouped in test classes.

### Correct Pattern
```python
def test_something() -> None:
assert True

def test_another_thing(monkeypatch: pytest.MonkeyPatch) -> None:
assert True
```

### Wrong Pattern (Do NOT use)
```python
class TestSomething:
def test_something(self) -> None:
assert True
```

---

## Type Hints in Tests

Always add type hints to test function parameters:

```python
def test_with_fixtures(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""Test using fixtures with proper type hints."""
pass
```

### Common Fixture Type Hints
- `tmp_path: Path` - Temporary directory for file operations
- `monkeypatch: pytest.MonkeyPatch` - Environment variable and attribute mocking
- `fixture_name: FixtureType` - Custom fixtures from conftest.py

---

## Fixtures

### Using tmp_path for Temporary Files

**PREFERRED**: Use pytest's built-in `tmp_path` fixture for automatic cleanup.

```python
def test_with_temp_files(tmp_path: Path) -> None:
"""Create temporary files that are automatically cleaned up."""
config_file = tmp_path / "config.yaml"
config_file.write_text("key: value")

# Read and test
assert config_file.read_text() == "key: value"
# Automatic cleanup after test
```

**NOT PREFERRED**: Manual tempfile operations (requires cleanup).

```python
# Avoid this pattern
import tempfile
import os

with tempfile.NamedTemporaryFile(mode="w", delete=False) as f:
temp_path = f.name

try:
# test code
pass
finally:
os.unlink(temp_path) # Manual cleanup needed
```

### Custom Fixtures with tmp_path

```python
@pytest.fixture
def config_file(tmp_path: Path) -> Path:
"""Create a config file for testing."""
config_data = {"key": "value"}

config_file = tmp_path / "config.yaml"
with open(config_file, "w") as f:
yaml.dump(config_data, f)

return config_file # No manual cleanup needed
```

---

## Environment Variables with monkeypatch

Use `monkeypatch` for setting/clearing environment variables with automatic rollback:

```python
def test_env_var_override(monkeypatch: pytest.MonkeyPatch) -> None:
"""Test with environment variable override."""
monkeypatch.setenv("MY_VAR", "test-value")

# MY_VAR is set in this test
assert os.getenv("MY_VAR") == "test-value"

# Automatically reset after test


def test_env_var_not_set(monkeypatch: pytest.MonkeyPatch) -> None:
"""Test that ensures environment variable is NOT set."""
monkeypatch.delenv("MY_VAR", raising=False)

# MY_VAR is not set
assert os.getenv("MY_VAR") is None
```

---

## Test Organization

Organize tests with comment sections:

```python
# Basic functionality tests


def test_basic_operation() -> None:
pass


def test_another_basic() -> None:
pass


# Edge case tests


def test_edge_case_empty() -> None:
pass


def test_edge_case_special_chars() -> None:
pass


# Integration tests


def test_full_workflow() -> None:
pass
```

---

## Test Style Guidelines

### Function Naming
- Start with `test_` prefix
- Use descriptive names: `test_override_list_with_multi_value` ✓
- Avoid generic names: `test_something` ✗

### Assertions
- Use simple `assert` statements
- Add comments explaining complex assertions

```python
def test_config_override() -> None:
config = load_config(config_file)

# Value should be overridden by environment variable
assert config["key"] == "env-value"
```

### Docstrings
- Always add docstrings explaining what the test validates

```python
def test_override_nested_key(temp_config_file: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""Test that nested config keys can be overridden via environment variables."""
pass
```

---

## Best Practices

1. **One concept per test**: Each test should verify one specific behavior
2. **Clear test names**: Names should explain what is being tested
3. **Use fixtures**: Reduce code duplication with fixtures
4. **Type hints everywhere**: All parameters and return types
5. **Automatic cleanup**: Use `tmp_path` and `monkeypatch` instead of manual cleanup
6. **Top-level functions only**: Never use test classes
7. **Group related tests**: Use comment sections to organize logically related tests

---

## Running Tests

```bash
# Run all tests
uv run pytest

# Run specific test file
uv run pytest tests/test_config.py -v

# Run specific test
uv run pytest tests/test_config.py::test_override_scalar_value -v

# Run with coverage
uv run pytest --cov=src

# Stop on first failure
uv run pytest -x
```

---

## Common Patterns in This Project

### Configuration Testing with YAML + Environment Variables

```python
@pytest.fixture
def config_file(tmp_path: Path) -> Path:
"""Create a test config file."""
config_data = {"api-key": "default", "cors": {"allow-origins": ["http://localhost"]}}
config_file = tmp_path / "config.yaml"
with open(config_file, "w") as f:
yaml.dump(config_data, f)
return config_file


def test_override_scalar_value(
config_file: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Test overriding scalar config values via env vars."""
monkeypatch.setenv("LLMOCK_API_KEY", "env-override")
config = load_config(config_file)
assert config["api-key"] == "env-override"


def test_override_list_value(
config_file: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Test overriding list values with semicolon-separated env vars."""
monkeypatch.setenv("LLMOCK_CORS_ALLOW_ORIGINS", "http://prod.com;http://api.com")
config = load_config(config_file)
assert config["cors"]["allow-origins"] == ["http://prod.com", "http://api.com"]
```

15 changes: 11 additions & 4 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,11 +33,18 @@ This project is designed for AI-first development. All agents MUST follow these
2. Update `docs/learnings/INSTRUCTION_UPDATES.md` with a new rule to prevent recurrence.
3. Append relevant rules to `AGENTS.md` if they are project-wide.

### 4. Test-Driven Completion
- **RULE**: A task is NOT done until tests pass.
### 4. Test-Driven Completion (MANDATORY)
- **RULE**: A task is NOT done until tests pass. ALWAYS write tests for new functionality.
- **PROCESS**:
1. Every work package MUST include tests.
2. Tests MUST pass before the task is marked `completed` in the todo list.
1. Every new feature/function MUST have corresponding tests.
2. Create tests BEFORE marking the task as complete.
3. Organize tests in appropriate test files (e.g., `tests/test_<module>.py`).
4. Tests MUST pass before the task is marked `completed` in the todo list.
5. Run full test suite to ensure no regressions: `uv run pytest -v`
- **COMMON MISTAKES TO AVOID**:
- Adding new functionality without tests
- Forgetting to test edge cases (empty values, special characters, nested structures)
- Not testing error conditions and validation

### 5. Code Quality Gate (MANDATORY)
- **RULE**: ALWAYS run linting and formatting before completing any code task.
Expand Down
19 changes: 19 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,25 @@ models:
- id: "gpt-4o-mini"
created: 1721172741
owned_by: "openai"

### Environment Variable Overrides

You can override values from `config.yaml` using environment variables with the `LLMOCK_` prefix.
Nested keys are joined with underscores, and dashes are converted to underscores.

Examples:

```bash
# Scalar override
export LLMOCK_API_KEY=your-secret-api-key

# Nested override: cors.allow-origins
export LLMOCK_CORS_ALLOW_ORIGINS="http://localhost:8000;http://localhost:5173"
```

Notes:
- Lists are parsed from semicolon-separated values.
- Only keys that exist in `config.yaml` are overridden.
```

## Development
Expand Down
48 changes: 46 additions & 2 deletions src/llmock/config.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
"""Configuration management - loads YAML as a raw dict."""
"""Configuration management - loads YAML with environment variable overrides."""

from functools import lru_cache
import os
from pathlib import Path
from typing import Any

Expand All @@ -9,6 +10,9 @@
# Type alias for the config dict
Config = dict[str, Any]

# Environment variable prefix for config overrides
ENV_PREFIX = "LLMOCK_"


def load_config(config_path: Path = Path("config.yaml")) -> Config:
"""Load configuration from YAML file.
Expand All @@ -26,7 +30,47 @@ def load_config(config_path: Path = Path("config.yaml")) -> Config:
raise FileNotFoundError(f"Config file not found: {config_path}")

with open(config_path) as f:
return yaml.safe_load(f) or {}
config = yaml.safe_load(f) or {}

# Apply environment variable overrides
_apply_env_overrides(config)
return config


def _apply_env_overrides(
config: Config, prefix: str = ENV_PREFIX, path: str = ""
) -> None:
"""Recursively traverse config dict and apply environment variable overrides.

Environment variables use the format: LLMOCK_SECTION_KEY=value
For lists, use semicolon-separated values: LLMOCK_CORS_ALLOW_ORIGINS=http://localhost:8000;http://localhost:5173

Args:
config: Configuration dict to modify in-place.
prefix: Current environment variable prefix (includes parent keys).
path: Current path in the config tree (for debugging).
"""
for key, value in list(config.items()):
# Build the environment variable name
env_key = f"{prefix}{key.upper().replace('-', '_')}"

# Check if env var exists
env_value = os.getenv(env_key)

if env_value is not None:
# Apply the override
if isinstance(value, list):
# For lists, split by semicolon
config[key] = env_value.split(";")
elif isinstance(value, dict):
# For dicts, can't override directly from env, but traverse it
_apply_env_overrides(value, f"{env_key}_", f"{path}{key}.")
else:
# For scalars, use the value directly
config[key] = env_value
elif isinstance(value, dict):
# Recursively process nested dicts
_apply_env_overrides(value, f"{env_key}_", f"{path}{key}.")


@lru_cache
Expand Down
Loading