From 5f3d3f35dde3f57a26158c0228cd44012b0271ec Mon Sep 17 00:00:00 2001 From: Nachiket Paranjape Date: Sun, 1 Feb 2026 23:33:00 -0800 Subject: [PATCH 1/9] tweak 2 --- CONTRIBUTING.md | 872 ++++++++++++++++++++++++++---------------------- 1 file changed, 475 insertions(+), 397 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index be940723..1d53a66a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,540 +1,618 @@ # Contributing to Agent Control -Thanks for contributing! This document covers conventions, setup, and workflows for all contributors. +First off, thanks for taking the time to contribute! ❤️ -## Project Architecture +Agent Control is an open source project, and we welcome contributions from the community. Whether you're fixing bugs, adding features, improving documentation, or sharing feedback, your involvement helps make Agent Control better for everyone. -Agent Control is a **uv workspace monorepo** with these components: +> And if you like the project, but just don't have time to contribute code, that's fine. There are other easy ways to support the project and show your appreciation: +> +> - Star the project on GitHub +> - Share it with colleagues and in your network +> - Reference it in your project's documentation +> - Mention it at meetups or conferences +> - Submit and discuss feature ideas -``` -agent-control/ -├── models/ # Shared Pydantic models (agent-control-models) -├── server/ # FastAPI server (agent-control-server) -├── sdks/python/ # Python SDK (agent-control) -├── engine/ # Control evaluation engine (agent-control-engine) -├── evaluators/ # Evaluator implementations (agent-control-evaluators) -└── examples/ # Usage examples -``` +## Ways to Contribute -**Dependency flow:** -``` -SDK ──────────────────────────────────────┐ - ▼ -Server ──► Engine ──► Models ◄── Evaluators -``` +There are many ways to help move the project forward: ---- +- **Report Bugs**: Found an issue? Help us fix it by reporting it. +- **Suggest Features**: Have an idea? We'd love to discuss it with you. +- **Improve Documentation**: Help make our docs clearer and more comprehensive. +- **Contribute Code**: Fix bugs, implement features, or add new evaluators. +- **Add Integrations**: Extend Agent Control with new agent framework integrations. -## Development Setup +## Reporting Bugs -### Prerequisites +Found a bug? Please help us fix it by following these steps: -- Python 3.12+ -- [uv](https://docs.astral.sh/uv/) (package manager) -- Docker (for server database) +### 1. Search Existing Issues -### Initial Setup +Check if the issue already exists in our [GitHub Issues](https://github.com/rungalileo/agent-control/issues). If you find a similar issue, add a comment with additional context rather than creating a duplicate. -```bash -# Clone the repository -git clone -cd agent-control +### 2. Create a New Issue + +If no issue exists, create a new one. When writing your bug report, please include: + +- **Clear title and description**: Summarize the problem concisely. +- **Steps to reproduce**: Provide a [minimal, reproducible example](https://stackoverflow.com/help/minimal-reproducible-example) that demonstrates the issue. +- **Expected vs. actual behavior**: Describe what you expected to happen and what actually happened. +- **Environment details**: + - OS and version (e.g., macOS 14.0, Ubuntu 22.04) + - Python version + - Agent Control version + - Relevant package versions +- **Error messages**: Include full stack traces if applicable. +- **Code snippets**: Share relevant code that triggers the issue. -# Install all dependencies (creates single .venv for workspace) -make sync +### 3. Wait for Triage + +A project maintainer will review your issue and may ask for additional information. Please be patient as we manage a high volume of issues. **Do not bump the issue unless you have new information to provide.** + +If you are adding an issue, please try to keep it focused on a single topic. If two issues are related or blocking, please [link them](https://docs.github.com/en/issues/tracking-your-work-with-issues/using-issues/linking-a-pull-request-to-an-issue) rather than combining them: -# Install git hooks (recommended) -make hooks-install +``` +This issue is blocked by #123 and related to #456. ``` ---- +## Suggesting Features -## Working with Components +Have an idea for a new feature or enhancement? -### Models (`models/`) +### 1. Search Existing Requests -Shared Pydantic models used by both server and SDK. +Search the [GitHub Issues](https://github.com/rungalileo/agent-control/issues?q=is%3Aissue+label%3Aenhancement) for existing feature requests. -```bash -# Location -models/src/agent_control_models/ - -# Key files -├── agent.py # Agent, Step models -├── controls.py # Control definitions, evaluators -├── evaluation.py # EvaluationRequest/Response -├── policy.py # Policy model -└── health.py # Health response -``` +### 2. Start a Discussion -**When to modify:** -- Adding new API request/response models -- Changing shared data structures -- Adding validation rules +If no similar request exists, open a new issue with the `enhancement` label. In your feature request: -**Testing:** -```bash -cd models -uv run pytest -``` +- **Describe the use case**: Explain the problem you're trying to solve. +- **Explain the value**: Why would this be valuable to other users? +- **Provide examples**: Include mockups, code examples, or references to similar features in other projects. +- **Consider alternatives**: Have you considered other approaches? +- **Outline test cases**: What should be tested to ensure the feature works correctly? ---- +### 3. Await Feedback -### Server (`server/`) +Project maintainers and the community will provide feedback. Be open to discussion and iteration on your idea. -FastAPI server providing the Agent Control API. +## Before You Start Coding -```bash -# Location -server/src/agent_control_server/ - -# Key files -├── main.py # FastAPI app entrypoint -├── endpoints/ # API route handlers -├── services/ # Business logic -└── db/ # Database models & queries -``` +**For significant changes, please open an issue first.** Discussing your proposed changes ahead of time will make the contribution process smooth for everyone. Changes that were not discussed in an issue may be rejected. -**Running the server:** -```bash -cd server +For small bug fixes or documentation improvements, you can proceed directly to opening a pull request. -# Start dependencies (PostgreSQL via Docker) -make start-dependencies +A good first step is to search for [issues labeled "good first issue"](https://github.com/rungalileo/agent-control/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) or "help wanted". These are specifically marked as suitable for new contributors. -# Run database migrations -make alembic-upgrade +**If you start working on an issue, please comment on it or assign it to yourself to avoid duplicate work.** -# Start server with hot-reload -make run -``` +## Development Process -**Database migrations:** -```bash -cd server +Follow these steps to set up your environment and contribute changes. -# Create new migration -make alembic-migrate MSG="add new column" +### 1. Fork and Clone the Repository -# Apply migrations -make alembic-upgrade +1. **Fork** the repository by clicking the "Fork" button on the [Agent Control GitHub page](https://github.com/rungalileo/agent-control). +2. **Clone** your fork locally: + ```bash + git clone https://github.com//agent-control.git + cd agent-control + ``` +3. **Add the upstream remote** (to keep your fork in sync): + ```bash + git remote add upstream https://github.com/rungalileo/agent-control.git + ``` -# Rollback one migration -make alembic-downgrade +### 2. Set Up Your Development Environment -# View migration history -make alembic-history -``` +This project is a Python monorepo managed as a `uv` workspace. We use `make` for common tasks. -**Testing:** -```bash -cd server -make test -``` +**Requirements:** +- Python 3.12+ +- `uv` package manager +- `make` ---- +**Setup steps:** -### SDK (`sdks/python/`) +1. **Install dependencies**: + ```bash + make sync + ``` + This installs all workspace dependencies and sets up the development environment. -Python client SDK for interacting with the Agent Control server. +2. **Install git hooks** (optional but recommended): + ```bash + make hooks-install + ``` + This sets up pre-commit hooks that automatically format and lint your code. -```bash -# Location -sdks/python/src/agent_control/ - -# Key files -├── __init__.py # Public API exports, init() function -├── client.py # AgentControlClient (HTTP client) -├── agents.py # Agent registration operations -├── policies.py # Policy management -├── controls.py # Control management -├── control_sets.py # Control set management -├── evaluation.py # Evaluation checks -├── control_decorators.py # @control decorator -└── evaluators/ # Evaluator system -``` +3. **Verify your setup**: + ```bash + make check + ``` + This runs tests, linting, and type checking to ensure everything is working. -**Key exports:** -```python -import agent_control +### 3. Create a Feature Branch -# Initialization -agent_control.init(agent_name="...", agent_id="...") +Create a new branch for your changes. Use descriptive names with prefixes: -# Decorator -@agent_control.control() -async def my_function(): ... +- `feature/add-regex-evaluator` - for new features +- `fix/handle-null-agent-name` - for bug fixes +- `docs/improve-evaluator-guide` - for documentation +- `refactor/simplify-core-logic` - for refactoring -# Client -async with agent_control.AgentControlClient() as client: - await agent_control.agents.get_agent(client, "id") +```bash +git checkout -b feature/my-new-feature ``` -**Testing:** +**Keep your branch up to date** with the main branch: ```bash -cd sdks/python -make test # Starts server automatically +git fetch upstream +git rebase upstream/main ``` -**Adding new SDK functionality:** -1. Add operation function in appropriate module (e.g., `policies.py`) -2. Export in `__init__.py` if needed -3. Add tests in `tests/` -4. Update docstrings with examples +### 4. Make Your Changes + +Choose the appropriate package for your changes (see [Project Structure](#project-structure) below): + +- **Writing code**: Follow the [code conventions in AGENTS.md](AGENTS.md#code-conventions). +- **Adding tests**: All behavior changes require tests (see [Testing](#testing-your-changes) below). +- **Updating documentation**: Update docstrings, README files, and the `docs/` directory as needed. +- **Adding dependencies**: Use `uv add ` in the appropriate workspace package directory. ---- +**Keep your changes focused**: Prefer the smallest diff that fixes the issue. Avoid mixing unrelated changes in a single PR. -### Engine (`engine/`) +### 5. Test Your Changes -Core control evaluation logic. The engine loads evaluators and executes evaluations. +Before submitting your PR, ensure all tests pass: ```bash -# Location -engine/src/agent_control_engine/ +# Run all checks (tests, lint, typecheck) +make check -# Key files -├── core.py # Main ControlEngine class -├── evaluators.py # Evaluator loader and caching -└── selectors.py # Data selection from payloads -``` +# Or run individual checks +make test # Run all tests +make lint # Check code style +make typecheck # Run mypy type checker -**How it works:** -- The engine uses the evaluator registry to find evaluators -- Evaluators are cached for performance (LRU cache) -- Selectors extract data from payloads before evaluation +# Run tests for a specific package +make engine-test # Test the engine package +make sdk-test # Test the SDK +make server-test # Test the server -**Testing:** -```bash -cd engine -make test +# Auto-fix linting issues +make lint-fix ``` -> **Note:** To add new evaluators, create an evaluator in `evaluators/` rather than modifying the engine directly. See the Evaluators section below. +**All tests must pass before your PR can be merged.** If you've introduced linter errors, fix them before submitting. ---- +### 6. Commit Your Changes -### Evaluators (`evaluators/`) - -Extensible evaluators for custom detection logic. +We use [Conventional Commits](https://www.conventionalcommits.org/) for commit messages: ```bash -# Location -evaluators/src/agent_control_evaluators/ - -# Key directories -├── builtin/ # Built-in evaluators -│ ├── regex.py # RegexEvaluator - pattern matching -│ └── list.py # ListEvaluator - value matching -└── luna2/ # Galileo Luna-2 integration - ├── evaluator.py # Luna2Evaluator implementation - ├── config.py # Luna2Config model - └── client.py # Direct HTTP client (no SDK dependency) +feat: add regex pattern evaluator +fix: handle missing agent_id in evaluation +docs: update evaluator implementation guide +refactor: simplify control selector logic +test: add coverage for SQL evaluator edge cases ``` -**Adding a new evaluator:** +**Commit message format:** +``` +: -1. **Create evaluator directory:** +[optional body] + +[optional footer] +``` + +Common types: `feat`, `fix`, `docs`, `refactor`, `test`, `chore`, `perf` + +### 7. Push and Open a Pull Request + +1. **Push** your branch to your fork: ```bash - mkdir evaluators/src/agent_control_evaluators/my_evaluator/ + git push origin feature/my-new-feature ``` -2. **Define configuration model (`config.py`):** - ```python - from pydantic import BaseModel, Field +2. **Open a Pull Request** against the `main` branch of the upstream repository. - class MyEvaluatorConfig(BaseModel): - """Configuration for MyEvaluator.""" - threshold: float = Field(0.5, ge=0.0, le=1.0) - api_endpoint: str = Field(default="https://api.example.com") - ``` +3. **Fill out the PR template** with: + - **Title**: Use conventional commit format (e.g., `feat: add regex evaluator`) + - **Description**: Explain what problem you're solving and how + - **Issue reference**: Link to related issues (e.g., "Fixes #123", "Closes #456") + - **Testing**: Describe how you tested the changes + - **Checklist**: Confirm you've run tests, updated docs, etc. -3. **Implement evaluator (`evaluator.py`):** - ```python - from typing import Any - from agent_control_models import ( - EvaluatorResult, - Evaluator, - EvaluatorMetadata, - register_evaluator, - ) - from .config import MyEvaluatorConfig - - @register_evaluator - class MyEvaluator(Evaluator[MyEvaluatorConfig]): - """My custom evaluator.""" - - metadata = EvaluatorMetadata( - name="my-evaluator", - version="1.0.0", - description="Custom detection logic", - requires_api_key=False, - timeout_ms=5000, - ) - config_model = MyEvaluatorConfig - - def __init__(self, config: MyEvaluatorConfig) -> None: - super().__init__(config) - # Initialize any clients or resources - - async def evaluate(self, data: Any) -> EvaluatorResult: - # Your detection logic here - score = await self._analyze(str(data)) - - return EvaluatorResult( - matched=score > self.config.threshold, - confidence=score, - message=f"Analysis score: {score:.2f}", - metadata={"score": score}, - ) - ``` +4. **Wait for review**: A maintainer will review your PR. Be responsive to feedback and questions. -4. **Export in `__init__.py`:** - ```python - from .config import MyEvaluatorConfig - from .evaluator import MyEvaluator +5. **Address review comments**: Make requested changes by pushing new commits to your branch. - __all__ = ["MyEvaluator", "MyEvaluatorConfig"] - ``` +6. **Celebrate** when your PR is merged! 🎉 -5. **Add optional dependencies in `evaluators/pyproject.toml`:** - ```toml - [project.optional-dependencies] - my-evaluator = ["httpx>=0.24.0"] # Add your dependencies - all = ["httpx>=0.24.0", ...] # Include in 'all' extra - ``` +## Common Contribution Scenarios -6. **Add tests in `evaluators/tests/`** +### Adding a New Evaluator -**Evaluator Best Practices:** -- Use Pydantic for config validation -- Make API calls async with httpx -- Return confidence scores (0.0-1.0) -- Include helpful metadata for debugging -- Handle errors gracefully (respect `on_error` config) -- Avoid storing request-scoped state (evaluators are cached) +Evaluators are a core part of Agent Control. Here's how to add a new one: ---- +**1. Create the evaluator class** in `evaluators/src/agent_control_evaluators/builtin/`: -## Code Quality +```python +from agent_control_models.evaluator import Evaluator, register_evaluator +from agent_control_models.evaluation import EvaluationResult +from pydantic import BaseModel, Field -### Linting (Ruff) +class MyEvaluatorConfig(BaseModel): + """Configuration for the evaluator.""" + pattern: str = Field(description="The pattern to match") -```bash -# Check all packages -make lint +@register_evaluator("my_evaluator") +class MyEvaluator(Evaluator[MyEvaluatorConfig]): + """ + A custom evaluator that does X. + + Configuration: + pattern: The pattern to match against input + """ + + def evaluate(self, **kwargs) -> EvaluationResult: + # Your evaluation logic here + return EvaluationResult( + passed=True, + reason="Explanation of the result" + ) +``` -# Auto-fix issues -make lint-fix +**2. Register the entry point** in `evaluators/pyproject.toml`: -# Single package -cd server && make lint +```toml +[project.entry-points."agent_control.evaluators"] +my_evaluator = "agent_control_evaluators.builtin.my_module:MyEvaluator" ``` -### Type Checking (mypy) +**3. Add tests** in `evaluators/tests/test_my_evaluator.py`: -```bash -# Check all packages -make typecheck +```python +from agent_control_evaluators.builtin.my_module import MyEvaluator, MyEvaluatorConfig -# Single package -cd sdks/python && make typecheck +def test_my_evaluator(): + config = MyEvaluatorConfig(pattern="test") + evaluator = MyEvaluator(config=config) + result = evaluator.evaluate(input_text="test") + assert result.passed is True ``` -### Pre-push Checks +**4. Add documentation** in `docs/evaluators/my_evaluator.md` explaining: +- What the evaluator does +- Configuration options +- Usage examples +- Edge cases and limitations +**5. Run the tests**: ```bash -# Run all checks (test + lint + typecheck) +make evaluators-test make check - -# Or manually run pre-push hook -make prepush ``` ---- +**6. Open a PR** with your changes. See [Making a Pull Request](#7-push-and-open-a-pull-request) above. -## Testing Conventions +### Adding a New API Endpoint -Write tests using **Given/When/Then** comments: +If you're adding a new server endpoint: +**1. Define or update models** in `models/src/agent_control_models/` if needed: ```python -def test_create_control(client: TestClient) -> None: - # Given: a valid control payload - payload = {"name": "pii-protection"} +from pydantic import BaseModel - # When: creating the control via API - response = client.put("/api/v1/controls", json=payload) +class MyRequest(BaseModel): + field: str - # Then: the control is created successfully - assert response.status_code == 200 - assert "control_id" in response.json() +class MyResponse(BaseModel): + result: str ``` -**Guidelines:** -- Keep tests small and focused -- Use explicit setup over hidden fixtures -- Test both success and error cases -- Mock external services (database, Galileo API) +**2. Add the endpoint** in `server/src/agent_control_server/endpoints/`: +```python +from fastapi import APIRouter +from agent_control_models.my_models import MyRequest, MyResponse ---- +router = APIRouter() -## Building & Publishing +@router.post("/my-endpoint", response_model=MyResponse) +async def my_endpoint(request: MyRequest) -> MyResponse: + # Your logic here + return MyResponse(result="success") +``` + +**3. Add business logic** in `server/src/agent_control_server/services/` (keep endpoints thin). -### Build Packages +**4. Add SDK wrapper** in `sdks/python/src/agent_control/`: +```python +class AgentControlClient: + def my_method(self, field: str) -> str: + response = self._request("POST", "/my-endpoint", json={"field": field}) + return response["result"] +``` +**5. Add tests** for both server and SDK: +- `server/tests/test_my_endpoint.py` +- `sdks/python/tests/test_my_method.py` + +**6. Update documentation** and examples if the endpoint is user-facing. + +### Adding an Integration Example + +To add an example for a new agent framework: + +**1. Create a new directory** in `examples/` (e.g., `examples/my_framework/`). + +**2. Add example files**: +- `README.md` - Setup and usage instructions +- `pyproject.toml` - Dependencies for the example +- `.env.example` - Environment variables needed +- Python files demonstrating the integration + +**3. Ensure the example is runnable**: ```bash -# Build all -make build - -# Build individual packages -make build-models -make build-server -make build-sdk -cd engine && make build +cd examples/my_framework +uv sync +python main.py ``` -### Publish Packages +**4. Update** `examples/README.md` to include your new example. + +### Improving Documentation + +Documentation improvements are always welcome: + +- **Typos and clarity**: Fix them directly and open a PR. +- **Missing examples**: Add code examples to `docs/` or `examples/`. +- **API documentation**: Update docstrings in the code (they're the source of truth). +- **Architecture guides**: Update `docs/OVERVIEW.md` or other guides. + +Run `make check` to ensure your changes don't break anything, then open a PR. + +## Pull Request Guidelines + +To ensure a smooth review process: + +- **Open an issue first** for significant changes to discuss the approach. +- **Keep it focused**: Smaller, focused PRs are easier to review and merge. +- **Follow conventions**: Use [Conventional Commits](https://www.conventionalcommits.org/) for commit messages. +- **Write tests**: All behavior changes require tests (see `docs/testing.md`). +- **Update documentation**: If you change user-facing behavior, update docs. +- **Pass all checks**: Ensure `make check` passes locally before opening the PR. +- **Be responsive**: Address review comments promptly and be open to feedback. + +## Testing Your Changes + +All tests run in CI and must pass before merging. We follow the testing conventions outlined in `docs/testing.md`. + +**Key testing principles:** + +- **Behavior changes require tests**: If you modify behavior, add tests that verify the new behavior. +- **Test at the right level**: Unit tests for logic, integration tests for workflows. +- **Use existing patterns**: Review existing tests to understand conventions. +- **Avoid flaky tests**: Tests should be deterministic and reliable. + +**Running tests:** ```bash -# Publish all (requires PyPI credentials) -make publish +# Run all tests +make test + +# Run tests for a specific package +make engine-test +make sdk-test +make server-test +make evaluators-test -# Publish individual packages -make publish-models -make publish-server -make publish-sdk +# Run a specific test file +cd engine +uv run pytest tests/test_evaluators.py + +# Run a specific test case +uv run pytest tests/test_evaluators.py::test_json_evaluator ``` -**Version bumping:** -Update `version` in respective `pyproject.toml` files: -- `models/pyproject.toml` -- `server/pyproject.toml` -- `sdks/python/pyproject.toml` -- `engine/pyproject.toml` -- `evaluators/pyproject.toml` +See `docs/testing.md` for comprehensive testing guidance. ---- +## Acceptable Use of AI Tools -## Git Workflow +Generative AI can be a useful tool for contributors, but like any tool should be used with critical thinking and good judgment. -### Branch Naming +We encourage contributors to use AI tools efficiently where they help. However, **AI assistance must be paired with meaningful human intervention, judgment, and contextual understanding.** -- `feature/description` - New features -- `fix/description` - Bug fixes -- `refactor/description` - Code refactoring +**Guidelines:** -### Commit Messages +- ✅ **Acceptable**: Using AI for boilerplate code, docstrings, test generation as a starting point that you review and refine. +- ✅ **Acceptable**: Using AI to help understand unfamiliar code or concepts. +- ✅ **Acceptable**: Using AI to suggest improvements that you critically evaluate. +- ❌ **Not acceptable**: Submitting entirely AI-generated code without meaningful human review. +- ❌ **Not acceptable**: Mass automated contributions that lack contextual relevance. +- ❌ **Not acceptable**: Low-effort, AI-generated spam PRs. -Use conventional commits: -``` -feat: add policy assignment endpoint -fix: handle missing agent gracefully -refactor: extract evaluator logic to engine -docs: update SDK usage examples -test: add control set integration tests +**If the human effort required to create a pull request is less than the effort required for maintainers to review it, that contribution should not be submitted.** + +We will close pull requests and issues that appear to be low-effort, AI-generated spam. With great tools comes great responsibility. + +## Project Structure + +Understanding the layout will help you know where to make changes. + +```mermaid +flowchart TB + SDK["Python SDK
(sdks/python)"] + Server["API Server
(server)"] + Engine["Evaluation Engine
(engine)"] + Models["Shared Models
(models)"] + Evaluators["Evaluators
(evaluators)"] + Examples["Examples & Docs
(examples, docs)"] + + SDK --> Server + Server --> Engine + Engine --> Models + Evaluators --> Models + Examples --- SDK + Examples --- Server ``` -### Pull Request Checklist +### Package Descriptions + +- **`models/`** (`agent_control_models`): Shared Pydantic v2 models and base classes. + - Defines API request/response models + - Base classes for evaluators + - Shared types used across packages + - **Change here if**: Adding new API models or evaluator base functionality + +- **`engine/`** (`agent_control_engine`): Core evaluation logic and orchestration. + - Control evaluation engine + - Evaluator discovery and registration + - Evaluation orchestration + - **Change here if**: Modifying evaluation logic or evaluator discovery + +- **`server/`** (`agent_control_server`): FastAPI server providing HTTP APIs. + - REST API endpoints + - Business logic services + - Database interactions (Alembic migrations) + - **Change here if**: Adding new endpoints or server functionality + +- **`sdks/python/`** (`agent_control`): Python SDK for users. + - Client library wrapping server APIs + - Control decorators and policies + - Local evaluation support (uses engine) + - **Change here if**: Adding user-facing SDK features + +- **`evaluators/`** (`agent_control_evaluators`): Built-in evaluator implementations. + - JSON, SQL, regex, list evaluators + - Luna2 integration + - All evaluators extend base classes from `models/` + - **Change here if**: Adding new evaluators or modifying existing ones + +- **`ui/`**: Next.js web application for managing agent controls. + - TypeScript/React frontend + - **Change here if**: Adding UI features (separate contribution process) + +- **`examples/`**: Runnable examples demonstrating integrations. + - LangChain, CrewAI, and other framework examples + - Demo agents and setup scripts + - **Change here if**: Adding new integration examples + +- **`docs/`**: Documentation and architectural guides. + - `OVERVIEW.md`: Architecture overview + - `REFERENCE.md`: API reference + - `testing.md`: Testing conventions + - Evaluator guides in `docs/evaluators/` + - **Change here if**: Improving documentation + +See `AGENTS.md` for detailed development conventions and the full change map. + +## Code Review Process + +Once you've opened a pull request: -- [ ] Tests pass (`make test`) -- [ ] Linting passes (`make lint`) -- [ ] Type checking passes (`make typecheck`) -- [ ] Documentation updated if needed -- [ ] Examples updated if API changed +1. **Automated checks**: CI will run tests, linting, and type checking. All checks must pass. ---- +2. **Maintainer review**: A project maintainer will review your code. This may take a few days depending on the size and complexity of your PR. -## Common Tasks +3. **Feedback and iteration**: The reviewer may request changes. Please: + - Address all feedback + - Push new commits to your branch (don't force-push unless asked) + - Respond to comments to acknowledge you've addressed them + - Ask questions if anything is unclear -### Add a new API endpoint +4. **Approval and merge**: Once approved, a maintainer will merge your PR. We use squash merge to keep history clean, so your commits will be combined into one. -1. Add Pydantic models in `models/` if needed -2. Add route handler in `server/src/agent_control_server/endpoints/` -3. Add service logic in `server/src/agent_control_server/services/` -4. Add SDK wrapper in `sdks/python/src/agent_control/` -5. Add tests for both server and SDK -6. Update examples if user-facing +5. **Post-merge**: Your contribution will be included in the next release. Thank you! 🎉 -### Add a new evaluator +**Review timeline expectations:** +- Simple PRs (docs, small fixes): Usually within 2-3 days +- Complex PRs (new features, evaluators): May take 5-7 days +- We're a small team, so please be patient -1. Create evaluator directory in `evaluators/src/agent_control_evaluators/` -2. Implement `Evaluator` interface (see Evaluators section above) -3. Add `@register_evaluator` decorator to your evaluator class -4. Add optional dependencies in `evaluators/pyproject.toml` -5. Export from `evaluators/src/agent_control_evaluators/__init__.py` -6. Add tests in `evaluators/tests/` -7. Update `docs/OVERVIEW.md` with usage examples +If your PR has been waiting more than a week without review, feel free to politely ping in the PR comments. -### Add a built-in evaluator (regex/list style) +## Need Help? + +If you have questions or need guidance: + +- **📚 Read the docs**: + - `docs/OVERVIEW.md` - Architecture overview + - `docs/REFERENCE.md` - API reference + - `docs/testing.md` - Testing conventions + - `AGENTS.md` - Developer guide for AI coding assistants + +- **💡 Check examples**: Review `examples/` for integration patterns. -1. Add evaluator class in `evaluators/src/agent_control_evaluators/builtin/` -2. Add config model in `models/src/agent_control_models/controls.py` -3. Register with `@register_evaluator` decorator -4. Add comprehensive tests in `evaluators/tests/` +- **🔍 Search issues**: Your question may have been answered in [existing issues](https://github.com/rungalileo/agent-control/issues). -### Update shared models +- **💬 Open a discussion**: For questions that don't fit issues, start a [GitHub Discussion](https://github.com/rungalileo/agent-control/discussions). -1. Modify models in `models/src/agent_control_models/` -2. Run tests across all packages: `make test` -3. Update any affected server endpoints -4. Update SDK if client-facing +- **🐛 Report a problem**: If you're stuck on a bug, [open an issue](https://github.com/rungalileo/agent-control/issues/new) with details. ---- +We appreciate your contribution to making Agent Control better! ❤️ ## Quick Reference -| Task | Command | -|------|---------| -| Install dependencies | `make sync` | -| Run server | `cd server && make run` | -| Run all tests | `make test` | -| Run linting | `make lint` | -| Run type checks | `make typecheck` | -| Run all checks | `make check` | -| Build packages | `make build` | -| Database migration | `cd server && make alembic-migrate MSG="..."` | - ---- - -## Evaluator Development Quick Reference - -| Task | Location | -|------|----------| -| Evaluator base class | `agent_control_models.Evaluator` | -| Evaluator metadata | `agent_control_models.EvaluatorMetadata` | -| Evaluator result | `agent_control_models.EvaluatorResult` | -| Register decorator | `@agent_control_models.register_evaluator` | -| Built-in evaluators | `evaluators/src/agent_control_evaluators/builtin/` | -| Evaluator tests | `evaluators/tests/` | - -**Evaluator config model fields:** -```python -from pydantic import BaseModel, Field +### Common Commands -class MyConfig(BaseModel): - # Required field - pattern: str = Field(..., description="Pattern to match") - - # Optional with default - threshold: float = Field(0.5, ge=0.0, le=1.0) - - # List field - values: list[str] = Field(default_factory=list) +```bash +# Setup and dependencies +make sync # Install/sync all dependencies +make hooks-install # Install git pre-commit hooks + +# Development +make dev # Run in development mode +make server-run # Run the server +make check # Run all checks (tests + lint + typecheck) + +# Testing +make test # Run all tests +make engine-test # Test engine package +make sdk-test # Test SDK package +make server-test # Test server package +make evaluators-test # Test evaluators package + +# Code quality +make lint # Check code style +make lint-fix # Auto-fix linting issues +make typecheck # Run mypy type checker + +# Database (server) +make server-alembic-upgrade # Apply database migrations +make server-alembic-downgrade # Rollback migrations +make server-db-seed # Seed database with test data ``` -**EvaluatorResult fields:** -```python -EvaluatorResult( - matched=True, # Did this trigger the control? - confidence=0.95, # How confident (0.0-1.0)? - message="Explanation", # Human-readable message - metadata={"key": "val"} # Additional context -) +### Package Structure + +``` +agent-control/ +├── models/ → agent_control_models +├── engine/ → agent_control_engine +├── server/ → agent_control_server +├── sdks/python/ → agent_control +├── evaluators/ → agent_control_evaluators +├── ui/ → Next.js web app +├── examples/ → Integration examples +└── docs/ → Documentation ``` ---- +### Conventional Commit Types -## Need Help? +- `feat`: New feature +- `fix`: Bug fix +- `docs`: Documentation changes +- `refactor`: Code refactoring +- `test`: Test additions or changes +- `chore`: Maintenance tasks +- `perf`: Performance improvements + +## License + +Agent Control is Apache 2.0 licensed. See [LICENSE](LICENSE) for more details. -- **Documentation:** See `docs/OVERVIEW.md` for architecture overview -- **Examples:** Check `examples/` for usage patterns -- **Tests:** Look at existing tests for patterns to follow +By contributing to Agent Control, you agree that your contributions will be licensed under the Apache 2.0 License. From 8d47695cb52bbeadde5fd555e80ca83c7a3557b2 Mon Sep 17 00:00:00 2001 From: Nachiket Paranjape Date: Mon, 2 Feb 2026 18:04:06 -0800 Subject: [PATCH 2/9] update --- CONTRIBUTING.md | 170 +++++++++++++++++++++++++++++++++++------------- 1 file changed, 125 insertions(+), 45 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1d53a66a..182f8250 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -22,6 +22,25 @@ There are many ways to help move the project forward: - **Contribute Code**: Fix bugs, implement features, or add new evaluators. - **Add Integrations**: Extend Agent Control with new agent framework integrations. +## Contributing Evaluators (Quickstart) + +If your goal is to add an evaluator, start here. We support two paths: + +| Path | Use when | Location | Install | +| --- | --- | --- | --- | +| Built-in evaluator | Lightweight deps, broadly useful | `evaluators/src/agent_control_evaluators/builtin/` | `pip install agent-control-evaluators` | +| Contrib evaluator | Heavy deps or vendor-specific | `evaluators/contrib/agent-control-evaluator-/` | `pip install agent-control-evaluators[org]` | + +**Fast path steps:** + +1. Create the evaluator class and config schema. +2. Register the evaluator entry point. +3. Add tests. +4. Add docs. +5. Run `make evaluators-test` (and ideally `make check`). + +Full details and examples are in [Adding a New Evaluator](#adding-a-new-evaluator) below. + ## Reporting Bugs Found a bug? Please help us fix it by following these steps: @@ -236,9 +255,16 @@ Common types: `feat`, `fix`, `docs`, `refactor`, `test`, `chore`, `perf` ### Adding a New Evaluator -Evaluators are a core part of Agent Control. Here's how to add a new one: +Evaluators are a core part of Agent Control. Start by choosing a path: + +| Path | Use when | Entry point name | Install | +| --- | --- | --- | --- | +| Built-in evaluator | Lightweight deps, broadly useful | `regex`, `sql`, `list` | `pip install agent-control-evaluators` | +| Contrib evaluator | Heavy deps or vendor-specific | `org.evaluator_name` | `pip install agent-control-evaluators[org]` | -**1. Create the evaluator class** in `evaluators/src/agent_control_evaluators/builtin/`: +**Option A: Built-in evaluator (core package)** + +1. Create the evaluator class in `evaluators/src/agent_control_evaluators/builtin/`: ```python from agent_control_models.evaluator import Evaluator, register_evaluator @@ -247,57 +273,93 @@ from pydantic import BaseModel, Field class MyEvaluatorConfig(BaseModel): """Configuration for the evaluator.""" + pattern: str = Field(description="The pattern to match") @register_evaluator("my_evaluator") class MyEvaluator(Evaluator[MyEvaluatorConfig]): - """ - A custom evaluator that does X. - - Configuration: - pattern: The pattern to match against input - """ - + """A custom evaluator that does X.""" + def evaluate(self, **kwargs) -> EvaluationResult: - # Your evaluation logic here return EvaluationResult( passed=True, - reason="Explanation of the result" + reason="Explanation of the result", ) ``` -**2. Register the entry point** in `evaluators/pyproject.toml`: +2. Register the entry point in `evaluators/pyproject.toml`: ```toml [project.entry-points."agent_control.evaluators"] my_evaluator = "agent_control_evaluators.builtin.my_module:MyEvaluator" ``` -**3. Add tests** in `evaluators/tests/test_my_evaluator.py`: +3. Add tests in `evaluators/tests/test_my_evaluator.py`. +4. Add documentation in `docs/evaluators/my_evaluator.md`. +5. Run tests with `make evaluators-test` and `make check`. +6. Open a PR with your changes. -```python -from agent_control_evaluators.builtin.my_module import MyEvaluator, MyEvaluatorConfig +**Option B: Contrib evaluator (third-party package)** + +This path is for evaluators with heavy dependencies (CUDA, large ML libs) or vendor-specific +requirements. Each publisher gets its own package and namespaced entry points. + +1. Create a new package under `evaluators/contrib/`: -def test_my_evaluator(): - config = MyEvaluatorConfig(pattern="test") - evaluator = MyEvaluator(config=config) - result = evaluator.evaluate(input_text="test") - assert result.passed is True +``` +evaluators/ + contrib/ + agent-control-evaluator-acme/ + pyproject.toml + src/agent_control_evaluator_acme/ + tests/ ``` -**4. Add documentation** in `docs/evaluators/my_evaluator.md` explaining: -- What the evaluator does -- Configuration options -- Usage examples -- Edge cases and limitations +2. Define the package and entry points in +`evaluators/contrib/agent-control-evaluator-acme/pyproject.toml`: -**5. Run the tests**: -```bash -make evaluators-test -make check +```toml +[project] +name = "agent-control-evaluator-acme" +version = "0.1.0" +dependencies = ["some-heavy-lib"] + +[project.entry-points."agent_control.evaluators"] +"acme.toxicity" = "agent_control_evaluator_acme.toxicity:ToxicityEvaluator" +"acme.hallucination" = "agent_control_evaluator_acme.hallucination:HallucinationEvaluator" +``` + +3. Implement evaluators in `src/agent_control_evaluator_acme/`. Use the same evaluator base +class pattern as Option A and set the decorator name to the namespaced entry point, for example +`@register_evaluator("acme.toxicity")`. +4. Add tests in `evaluators/contrib/agent-control-evaluator-acme/tests/`. +5. Add docs in `docs/evaluators/acme_toxicity.md` (and similar for other evaluators). +6. Add a convenience extra in `evaluators/pyproject.toml`: + +```toml +[project.optional-dependencies] +acme = ["agent-control-evaluator-acme>=0.1.0"] ``` -**6. Open a PR** with your changes. See [Making a Pull Request](#7-push-and-open-a-pull-request) above. +7. Ensure the workspace and release config include the new package: + +``` +pyproject.toml + [tool.uv.workspace] + members = ["models", "server", "sdks/python", "engine", "evaluators", "evaluators/contrib/*"] + + [tool.semantic_release] + version_toml = [ + "evaluators/contrib/agent-control-evaluator-acme/pyproject.toml:project.version", + ] +``` + +**Contrib evaluator conventions:** + +- Entry points must be namespaced as `org.evaluator_name`. +- A single package can export multiple evaluators. +- Keep heavy or optional dependencies inside the contrib package. +- If optional dependencies exist, override `is_available()` to skip cleanly. ### Adding a New API Endpoint @@ -441,25 +503,41 @@ We will close pull requests and issues that appear to be low-effort, AI-generate ## Project Structure -Understanding the layout will help you know where to make changes. +Understanding the layout will help you know where to make changes. This diagram shows how the different packages interact and where they reside in the repository. ```mermaid -flowchart TB - SDK["Python SDK
(sdks/python)"] - Server["API Server
(server)"] - Engine["Evaluation Engine
(engine)"] - Models["Shared Models
(models)"] - Evaluators["Evaluators
(evaluators)"] - Examples["Examples & Docs
(examples, docs)"] - - SDK --> Server - Server --> Engine - Engine --> Models - Evaluators --> Models - Examples --- SDK - Examples --- Server +graph TD + subgraph "Core System" + Models["models/
(Shared Types & Base Classes)"] + Engine["engine/
(Evaluation Logic)"] + Evaluators["evaluators/
(Built-in Checks)"] + end + + subgraph "Server-Side" + Server["server/
(FastAPI & DB)"] + end + + subgraph "Client-Side" + SDK["sdks/python/
(User Interface)"] + UI["ui/
(Web Dashboard)"] + end + + %% Dependencies + SDK -->|"Uses API"| Server + SDK -->|"Local Eval"| Engine + Server -->|"Remote Eval"| Engine + + Engine -->|"Uses"| Models + Evaluators -->|"Implements"| Models + Engine -->|"Loads"| Evaluators + + %% Examples + Examples["examples/
(Demos & Integrations)"] -.->|"Imports"| SDK ``` +Third-party evaluators live in `evaluators/contrib/` as separate packages and are installed via +extras in `agent-control-evaluators` (for example, `pip install agent-control-evaluators[acme]`). + ### Package Descriptions - **`models/`** (`agent_control_models`): Shared Pydantic v2 models and base classes. @@ -490,6 +568,7 @@ flowchart TB - JSON, SQL, regex, list evaluators - Luna2 integration - All evaluators extend base classes from `models/` + - Third-party packages live in `evaluators/contrib/` (see [Adding a New Evaluator](#adding-a-new-evaluator)) - **Change here if**: Adding new evaluators or modifying existing ones - **`ui/`**: Next.js web application for managing agent controls. @@ -596,6 +675,7 @@ agent-control/ ├── server/ → agent_control_server ├── sdks/python/ → agent_control ├── evaluators/ → agent_control_evaluators +├── evaluators/contrib/ → Third-party evaluator packages ├── ui/ → Next.js web app ├── examples/ → Integration examples └── docs/ → Documentation From d8101ccf244d45f5efb1f4b91a2c46cebbfd2a65 Mon Sep 17 00:00:00 2001 From: Lev Neiman Date: Wed, 25 Feb 2026 19:30:22 -0500 Subject: [PATCH 3/9] fix(docs): restructure CONTRIBUTING.md to address review feedback - Remove promotional "star the project" blockquote - Move dev setup instructions out (link to README/AGENTS.md instead) - Restructure around 5 contribution categories per reviewer request - Trim inline code examples; keep as routing doc to detailed docs --- CONTRIBUTING.md | 732 +++++++----------------------------------------- 1 file changed, 94 insertions(+), 638 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 182f8250..f11cd246 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,698 +1,154 @@ # Contributing to Agent Control -First off, thanks for taking the time to contribute! ❤️ +Thanks for taking the time to contribute! This guide covers the process for contributing to Agent Control. -Agent Control is an open source project, and we welcome contributions from the community. Whether you're fixing bugs, adding features, improving documentation, or sharing feedback, your involvement helps make Agent Control better for everyone. +For development setup, architecture details, and code conventions, see the [README](README.md) and [AGENTS.md](AGENTS.md). -> And if you like the project, but just don't have time to contribute code, that's fine. There are other easy ways to support the project and show your appreciation: -> -> - Star the project on GitHub -> - Share it with colleagues and in your network -> - Reference it in your project's documentation -> - Mention it at meetups or conferences -> - Submit and discuss feature ideas +## Table of Contents -## Ways to Contribute +1. [Integrate Agent Control into Your Agents](#1-integrate-agent-control-into-your-agents) +2. [Contribute New Evaluators](#2-contribute-new-evaluators) +3. [Extend Agent Control](#3-extend-agent-control) +4. [Improve Code and Documentation](#4-improve-code-and-documentation) +5. [Suggest Features and Report Bugs](#5-suggest-features-and-report-bugs) -There are many ways to help move the project forward: +--- -- **Report Bugs**: Found an issue? Help us fix it by reporting it. -- **Suggest Features**: Have an idea? We'd love to discuss it with you. -- **Improve Documentation**: Help make our docs clearer and more comprehensive. -- **Contribute Code**: Fix bugs, implement features, or add new evaluators. -- **Add Integrations**: Extend Agent Control with new agent framework integrations. +## Before You Start -## Contributing Evaluators (Quickstart) +- **For significant changes, open an issue first.** Discussing your approach ahead of time avoids wasted effort. Changes that were not discussed may be rejected. +- **For small fixes** (typos, minor bug fixes), you can go straight to a pull request. +- **Claim your work.** If you start on an issue, comment on it or assign yourself to avoid duplicate effort. +- Look for issues labeled [good first issue](https://github.com/agentcontrol/agent-control/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) or **help wanted** if you're new. -If your goal is to add an evaluator, start here. We support two paths: +--- -| Path | Use when | Location | Install | -| --- | --- | --- | --- | -| Built-in evaluator | Lightweight deps, broadly useful | `evaluators/src/agent_control_evaluators/builtin/` | `pip install agent-control-evaluators` | -| Contrib evaluator | Heavy deps or vendor-specific | `evaluators/contrib/agent-control-evaluator-/` | `pip install agent-control-evaluators[org]` | +## 1. Integrate Agent Control into Your Agents -**Fast path steps:** +We welcome examples showing how Agent Control works with different agent frameworks. -1. Create the evaluator class and config schema. -2. Register the evaluator entry point. -3. Add tests. -4. Add docs. -5. Run `make evaluators-test` (and ideally `make check`). +To add an integration example: -Full details and examples are in [Adding a New Evaluator](#adding-a-new-evaluator) below. +1. Create a directory in `examples/` (e.g., `examples/my_framework/`). +2. Include a `README.md` with setup instructions, a `pyproject.toml` for dependencies, and runnable Python files. +3. Update `examples/README.md` to list your new example. +4. Open a PR. -## Reporting Bugs +See existing examples in `examples/` for the expected structure. -Found a bug? Please help us fix it by following these steps: +--- -### 1. Search Existing Issues +## 2. Contribute New Evaluators -Check if the issue already exists in our [GitHub Issues](https://github.com/rungalileo/agent-control/issues). If you find a similar issue, add a comment with additional context rather than creating a duplicate. +Evaluators are a core extension point. We support two paths: -### 2. Create a New Issue +| Path | Use when | Location | +| --- | --- | --- | +| Built-in | Lightweight deps, broadly useful | `evaluators/src/agent_control_evaluators/builtin/` | +| Contrib | Heavy deps or vendor-specific | `evaluators/contrib/agent-control-evaluator-/` | -If no issue exists, create a new one. When writing your bug report, please include: +**Steps:** -- **Clear title and description**: Summarize the problem concisely. -- **Steps to reproduce**: Provide a [minimal, reproducible example](https://stackoverflow.com/help/minimal-reproducible-example) that demonstrates the issue. -- **Expected vs. actual behavior**: Describe what you expected to happen and what actually happened. -- **Environment details**: - - OS and version (e.g., macOS 14.0, Ubuntu 22.04) - - Python version - - Agent Control version - - Relevant package versions -- **Error messages**: Include full stack traces if applicable. -- **Code snippets**: Share relevant code that triggers the issue. +1. Create the evaluator class extending `Evaluator` from `agent_control_models`. +2. Register via `@register_evaluator` and add the entry point in `pyproject.toml`. +3. Add tests and documentation. +4. Run `make evaluators-test` (and ideally `make check`). +5. Open a PR. -### 3. Wait for Triage +For built-in evaluators, add your module to `evaluators/src/agent_control_evaluators/builtin/` and register the entry point in `evaluators/pyproject.toml`. -A project maintainer will review your issue and may ask for additional information. Please be patient as we manage a high volume of issues. **Do not bump the issue unless you have new information to provide.** +For contrib evaluators, create a new package under `evaluators/contrib/` with its own `pyproject.toml`. Entry points must be namespaced as `org.evaluator_name`. Add a convenience extra in `evaluators/pyproject.toml` and include the package in the workspace members in the root `pyproject.toml`. -If you are adding an issue, please try to keep it focused on a single topic. If two issues are related or blocking, please [link them](https://docs.github.com/en/issues/tracking-your-work-with-issues/using-issues/linking-a-pull-request-to-an-issue) rather than combining them: +See existing evaluators (e.g., `regex`, `list`) and `AGENTS.md` for detailed conventions and code patterns. -``` -This issue is blocked by #123 and related to #456. -``` +--- -## Suggesting Features +## 3. Extend Agent Control -Have an idea for a new feature or enhancement? +For changes to the core system (new API endpoints, SDK features, engine logic): -### 1. Search Existing Requests +1. **Models** (`models/`): Shared Pydantic models and base classes. Change here when adding new API request/response types or evaluator base functionality. +2. **Engine** (`engine/`): Evaluation logic and evaluator discovery. Change here when modifying how controls are evaluated. +3. **Server** (`server/`): FastAPI endpoints and business logic. Change here when adding API endpoints or server functionality. +4. **SDK** (`sdks/python/`): Python client library. Change here when adding user-facing SDK features. +5. **UI** (`ui/`): Next.js web dashboard. Change here for frontend features. -Search the [GitHub Issues](https://github.com/rungalileo/agent-control/issues?q=is%3Aissue+label%3Aenhancement) for existing feature requests. +See `AGENTS.md` for the full architecture diagram, dependency flow, and code conventions. -### 2. Start a Discussion +--- -If no similar request exists, open a new issue with the `enhancement` label. In your feature request: +## 4. Improve Code and Documentation -- **Describe the use case**: Explain the problem you're trying to solve. -- **Explain the value**: Why would this be valuable to other users? -- **Provide examples**: Include mockups, code examples, or references to similar features in other projects. -- **Consider alternatives**: Have you considered other approaches? -- **Outline test cases**: What should be tested to ensure the feature works correctly? +Documentation and code quality improvements are always welcome: -### 3. Await Feedback +- **Typos and clarity fixes**: Open a PR directly. +- **Missing examples**: Add to `docs/` or `examples/`. +- **API documentation**: Docstrings in the code are the source of truth. +- **Architecture guides**: Update files in `docs/`. -Project maintainers and the community will provide feedback. Be open to discussion and iteration on your idea. +Run `make check` before submitting to ensure nothing is broken. -## Before You Start Coding +--- -**For significant changes, please open an issue first.** Discussing your proposed changes ahead of time will make the contribution process smooth for everyone. Changes that were not discussed in an issue may be rejected. +## 5. Suggest Features and Report Bugs -For small bug fixes or documentation improvements, you can proceed directly to opening a pull request. +### Reporting Bugs -A good first step is to search for [issues labeled "good first issue"](https://github.com/rungalileo/agent-control/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) or "help wanted". These are specifically marked as suitable for new contributors. +1. **Search first**: Check [GitHub Issues](https://github.com/agentcontrol/agent-control/issues) for duplicates. If you find one, add context in a comment rather than opening a new issue. +2. **Create an issue** with: clear title, steps to reproduce ([minimal example](https://stackoverflow.com/help/minimal-reproducible-example)), expected vs. actual behavior, environment details (OS, Python version, Agent Control version), and full error messages/stack traces. +3. **Keep issues focused.** One topic per issue. Link related issues rather than combining them. -**If you start working on an issue, please comment on it or assign it to yourself to avoid duplicate work.** +### Suggesting Features -## Development Process +1. **Search first**: Check [existing feature requests](https://github.com/agentcontrol/agent-control/issues?q=is%3Aissue+label%3Aenhancement). +2. **Open an issue** with the `enhancement` label. Describe the use case, explain the value, provide examples, and consider alternatives. +3. Be open to discussion and iteration on your idea. -Follow these steps to set up your environment and contribute changes. +--- -### 1. Fork and Clone the Repository +## Contribution Workflow -1. **Fork** the repository by clicking the "Fork" button on the [Agent Control GitHub page](https://github.com/rungalileo/agent-control). -2. **Clone** your fork locally: - ```bash - git clone https://github.com//agent-control.git - cd agent-control - ``` -3. **Add the upstream remote** (to keep your fork in sync): - ```bash - git remote add upstream https://github.com/rungalileo/agent-control.git - ``` +### Fork, Branch, and PR -### 2. Set Up Your Development Environment +1. **Fork** the repository and clone your fork. +2. **Create a branch** with a descriptive name: `feature/...`, `fix/...`, `docs/...`, or `refactor/...`. +3. **Make your changes.** Keep diffs focused. Follow the conventions in [AGENTS.md](AGENTS.md). +4. **Write tests.** All behavior changes require tests. See `docs/testing.md`. +5. **Run checks locally**: `make check` (runs tests, lint, and typecheck). +6. **Commit** using [Conventional Commits](https://www.conventionalcommits.org/): `feat:`, `fix:`, `docs:`, `refactor:`, `test:`, `chore:`, `perf:`. +7. **Push** and open a PR against `main`. Link related issues (e.g., "Fixes #123"). -This project is a Python monorepo managed as a `uv` workspace. We use `make` for common tasks. +### Code Review -**Requirements:** -- Python 3.12+ -- `uv` package manager -- `make` +1. **CI must pass.** All automated checks (tests, lint, typecheck) must be green. +2. **A maintainer will review.** Simple PRs: 2-3 days. Complex PRs: up to a week. +3. **Address feedback** by pushing new commits (don't force-push unless asked). +4. **Merge.** We use squash merge to keep history clean. -**Setup steps:** +If your PR has waited more than a week, feel free to politely ping in the comments. -1. **Install dependencies**: - ```bash - make sync - ``` - This installs all workspace dependencies and sets up the development environment. - -2. **Install git hooks** (optional but recommended): - ```bash - make hooks-install - ``` - This sets up pre-commit hooks that automatically format and lint your code. - -3. **Verify your setup**: - ```bash - make check - ``` - This runs tests, linting, and type checking to ensure everything is working. - -### 3. Create a Feature Branch - -Create a new branch for your changes. Use descriptive names with prefixes: - -- `feature/add-regex-evaluator` - for new features -- `fix/handle-null-agent-name` - for bug fixes -- `docs/improve-evaluator-guide` - for documentation -- `refactor/simplify-core-logic` - for refactoring - -```bash -git checkout -b feature/my-new-feature -``` - -**Keep your branch up to date** with the main branch: -```bash -git fetch upstream -git rebase upstream/main -``` - -### 4. Make Your Changes - -Choose the appropriate package for your changes (see [Project Structure](#project-structure) below): - -- **Writing code**: Follow the [code conventions in AGENTS.md](AGENTS.md#code-conventions). -- **Adding tests**: All behavior changes require tests (see [Testing](#testing-your-changes) below). -- **Updating documentation**: Update docstrings, README files, and the `docs/` directory as needed. -- **Adding dependencies**: Use `uv add ` in the appropriate workspace package directory. - -**Keep your changes focused**: Prefer the smallest diff that fixes the issue. Avoid mixing unrelated changes in a single PR. - -### 5. Test Your Changes - -Before submitting your PR, ensure all tests pass: - -```bash -# Run all checks (tests, lint, typecheck) -make check - -# Or run individual checks -make test # Run all tests -make lint # Check code style -make typecheck # Run mypy type checker - -# Run tests for a specific package -make engine-test # Test the engine package -make sdk-test # Test the SDK -make server-test # Test the server - -# Auto-fix linting issues -make lint-fix -``` - -**All tests must pass before your PR can be merged.** If you've introduced linter errors, fix them before submitting. - -### 6. Commit Your Changes - -We use [Conventional Commits](https://www.conventionalcommits.org/) for commit messages: - -```bash -feat: add regex pattern evaluator -fix: handle missing agent_id in evaluation -docs: update evaluator implementation guide -refactor: simplify control selector logic -test: add coverage for SQL evaluator edge cases -``` - -**Commit message format:** -``` -: - -[optional body] - -[optional footer] -``` - -Common types: `feat`, `fix`, `docs`, `refactor`, `test`, `chore`, `perf` - -### 7. Push and Open a Pull Request - -1. **Push** your branch to your fork: - ```bash - git push origin feature/my-new-feature - ``` - -2. **Open a Pull Request** against the `main` branch of the upstream repository. - -3. **Fill out the PR template** with: - - **Title**: Use conventional commit format (e.g., `feat: add regex evaluator`) - - **Description**: Explain what problem you're solving and how - - **Issue reference**: Link to related issues (e.g., "Fixes #123", "Closes #456") - - **Testing**: Describe how you tested the changes - - **Checklist**: Confirm you've run tests, updated docs, etc. - -4. **Wait for review**: A maintainer will review your PR. Be responsive to feedback and questions. - -5. **Address review comments**: Make requested changes by pushing new commits to your branch. - -6. **Celebrate** when your PR is merged! 🎉 - -## Common Contribution Scenarios - -### Adding a New Evaluator - -Evaluators are a core part of Agent Control. Start by choosing a path: - -| Path | Use when | Entry point name | Install | -| --- | --- | --- | --- | -| Built-in evaluator | Lightweight deps, broadly useful | `regex`, `sql`, `list` | `pip install agent-control-evaluators` | -| Contrib evaluator | Heavy deps or vendor-specific | `org.evaluator_name` | `pip install agent-control-evaluators[org]` | - -**Option A: Built-in evaluator (core package)** - -1. Create the evaluator class in `evaluators/src/agent_control_evaluators/builtin/`: - -```python -from agent_control_models.evaluator import Evaluator, register_evaluator -from agent_control_models.evaluation import EvaluationResult -from pydantic import BaseModel, Field - -class MyEvaluatorConfig(BaseModel): - """Configuration for the evaluator.""" - - pattern: str = Field(description="The pattern to match") - -@register_evaluator("my_evaluator") -class MyEvaluator(Evaluator[MyEvaluatorConfig]): - """A custom evaluator that does X.""" - - def evaluate(self, **kwargs) -> EvaluationResult: - return EvaluationResult( - passed=True, - reason="Explanation of the result", - ) -``` - -2. Register the entry point in `evaluators/pyproject.toml`: - -```toml -[project.entry-points."agent_control.evaluators"] -my_evaluator = "agent_control_evaluators.builtin.my_module:MyEvaluator" -``` - -3. Add tests in `evaluators/tests/test_my_evaluator.py`. -4. Add documentation in `docs/evaluators/my_evaluator.md`. -5. Run tests with `make evaluators-test` and `make check`. -6. Open a PR with your changes. - -**Option B: Contrib evaluator (third-party package)** - -This path is for evaluators with heavy dependencies (CUDA, large ML libs) or vendor-specific -requirements. Each publisher gets its own package and namespaced entry points. - -1. Create a new package under `evaluators/contrib/`: - -``` -evaluators/ - contrib/ - agent-control-evaluator-acme/ - pyproject.toml - src/agent_control_evaluator_acme/ - tests/ -``` - -2. Define the package and entry points in -`evaluators/contrib/agent-control-evaluator-acme/pyproject.toml`: - -```toml -[project] -name = "agent-control-evaluator-acme" -version = "0.1.0" -dependencies = ["some-heavy-lib"] - -[project.entry-points."agent_control.evaluators"] -"acme.toxicity" = "agent_control_evaluator_acme.toxicity:ToxicityEvaluator" -"acme.hallucination" = "agent_control_evaluator_acme.hallucination:HallucinationEvaluator" -``` - -3. Implement evaluators in `src/agent_control_evaluator_acme/`. Use the same evaluator base -class pattern as Option A and set the decorator name to the namespaced entry point, for example -`@register_evaluator("acme.toxicity")`. -4. Add tests in `evaluators/contrib/agent-control-evaluator-acme/tests/`. -5. Add docs in `docs/evaluators/acme_toxicity.md` (and similar for other evaluators). -6. Add a convenience extra in `evaluators/pyproject.toml`: - -```toml -[project.optional-dependencies] -acme = ["agent-control-evaluator-acme>=0.1.0"] -``` - -7. Ensure the workspace and release config include the new package: - -``` -pyproject.toml - [tool.uv.workspace] - members = ["models", "server", "sdks/python", "engine", "evaluators", "evaluators/contrib/*"] - - [tool.semantic_release] - version_toml = [ - "evaluators/contrib/agent-control-evaluator-acme/pyproject.toml:project.version", - ] -``` - -**Contrib evaluator conventions:** - -- Entry points must be namespaced as `org.evaluator_name`. -- A single package can export multiple evaluators. -- Keep heavy or optional dependencies inside the contrib package. -- If optional dependencies exist, override `is_available()` to skip cleanly. - -### Adding a New API Endpoint - -If you're adding a new server endpoint: - -**1. Define or update models** in `models/src/agent_control_models/` if needed: -```python -from pydantic import BaseModel - -class MyRequest(BaseModel): - field: str - -class MyResponse(BaseModel): - result: str -``` - -**2. Add the endpoint** in `server/src/agent_control_server/endpoints/`: -```python -from fastapi import APIRouter -from agent_control_models.my_models import MyRequest, MyResponse - -router = APIRouter() - -@router.post("/my-endpoint", response_model=MyResponse) -async def my_endpoint(request: MyRequest) -> MyResponse: - # Your logic here - return MyResponse(result="success") -``` - -**3. Add business logic** in `server/src/agent_control_server/services/` (keep endpoints thin). - -**4. Add SDK wrapper** in `sdks/python/src/agent_control/`: -```python -class AgentControlClient: - def my_method(self, field: str) -> str: - response = self._request("POST", "/my-endpoint", json={"field": field}) - return response["result"] -``` - -**5. Add tests** for both server and SDK: -- `server/tests/test_my_endpoint.py` -- `sdks/python/tests/test_my_method.py` - -**6. Update documentation** and examples if the endpoint is user-facing. - -### Adding an Integration Example - -To add an example for a new agent framework: - -**1. Create a new directory** in `examples/` (e.g., `examples/my_framework/`). - -**2. Add example files**: -- `README.md` - Setup and usage instructions -- `pyproject.toml` - Dependencies for the example -- `.env.example` - Environment variables needed -- Python files demonstrating the integration - -**3. Ensure the example is runnable**: -```bash -cd examples/my_framework -uv sync -python main.py -``` - -**4. Update** `examples/README.md` to include your new example. - -### Improving Documentation - -Documentation improvements are always welcome: - -- **Typos and clarity**: Fix them directly and open a PR. -- **Missing examples**: Add code examples to `docs/` or `examples/`. -- **API documentation**: Update docstrings in the code (they're the source of truth). -- **Architecture guides**: Update `docs/OVERVIEW.md` or other guides. - -Run `make check` to ensure your changes don't break anything, then open a PR. - -## Pull Request Guidelines - -To ensure a smooth review process: - -- **Open an issue first** for significant changes to discuss the approach. -- **Keep it focused**: Smaller, focused PRs are easier to review and merge. -- **Follow conventions**: Use [Conventional Commits](https://www.conventionalcommits.org/) for commit messages. -- **Write tests**: All behavior changes require tests (see `docs/testing.md`). -- **Update documentation**: If you change user-facing behavior, update docs. -- **Pass all checks**: Ensure `make check` passes locally before opening the PR. -- **Be responsive**: Address review comments promptly and be open to feedback. - -## Testing Your Changes - -All tests run in CI and must pass before merging. We follow the testing conventions outlined in `docs/testing.md`. - -**Key testing principles:** - -- **Behavior changes require tests**: If you modify behavior, add tests that verify the new behavior. -- **Test at the right level**: Unit tests for logic, integration tests for workflows. -- **Use existing patterns**: Review existing tests to understand conventions. -- **Avoid flaky tests**: Tests should be deterministic and reliable. - -**Running tests:** - -```bash -# Run all tests -make test - -# Run tests for a specific package -make engine-test -make sdk-test -make server-test -make evaluators-test - -# Run a specific test file -cd engine -uv run pytest tests/test_evaluators.py - -# Run a specific test case -uv run pytest tests/test_evaluators.py::test_json_evaluator -``` - -See `docs/testing.md` for comprehensive testing guidance. +--- ## Acceptable Use of AI Tools -Generative AI can be a useful tool for contributors, but like any tool should be used with critical thinking and good judgment. - -We encourage contributors to use AI tools efficiently where they help. However, **AI assistance must be paired with meaningful human intervention, judgment, and contextual understanding.** - -**Guidelines:** - -- ✅ **Acceptable**: Using AI for boilerplate code, docstrings, test generation as a starting point that you review and refine. -- ✅ **Acceptable**: Using AI to help understand unfamiliar code or concepts. -- ✅ **Acceptable**: Using AI to suggest improvements that you critically evaluate. -- ❌ **Not acceptable**: Submitting entirely AI-generated code without meaningful human review. -- ❌ **Not acceptable**: Mass automated contributions that lack contextual relevance. -- ❌ **Not acceptable**: Low-effort, AI-generated spam PRs. - -**If the human effort required to create a pull request is less than the effort required for maintainers to review it, that contribution should not be submitted.** - -We will close pull requests and issues that appear to be low-effort, AI-generated spam. With great tools comes great responsibility. - -## Project Structure - -Understanding the layout will help you know where to make changes. This diagram shows how the different packages interact and where they reside in the repository. - -```mermaid -graph TD - subgraph "Core System" - Models["models/
(Shared Types & Base Classes)"] - Engine["engine/
(Evaluation Logic)"] - Evaluators["evaluators/
(Built-in Checks)"] - end - - subgraph "Server-Side" - Server["server/
(FastAPI & DB)"] - end - - subgraph "Client-Side" - SDK["sdks/python/
(User Interface)"] - UI["ui/
(Web Dashboard)"] - end - - %% Dependencies - SDK -->|"Uses API"| Server - SDK -->|"Local Eval"| Engine - Server -->|"Remote Eval"| Engine - - Engine -->|"Uses"| Models - Evaluators -->|"Implements"| Models - Engine -->|"Loads"| Evaluators - - %% Examples - Examples["examples/
(Demos & Integrations)"] -.->|"Imports"| SDK -``` - -Third-party evaluators live in `evaluators/contrib/` as separate packages and are installed via -extras in `agent-control-evaluators` (for example, `pip install agent-control-evaluators[acme]`). - -### Package Descriptions +AI tools can help with contributions, but **AI assistance must be paired with meaningful human review and judgment.** -- **`models/`** (`agent_control_models`): Shared Pydantic v2 models and base classes. - - Defines API request/response models - - Base classes for evaluators - - Shared types used across packages - - **Change here if**: Adding new API models or evaluator base functionality +- **Acceptable**: Using AI for boilerplate, understanding unfamiliar code, or generating a starting point that you refine. +- **Not acceptable**: Submitting entirely AI-generated code without review, mass automated contributions, or low-effort spam PRs. -- **`engine/`** (`agent_control_engine`): Core evaluation logic and orchestration. - - Control evaluation engine - - Evaluator discovery and registration - - Evaluation orchestration - - **Change here if**: Modifying evaluation logic or evaluator discovery +**If the effort to create a PR is less than the effort to review it, that contribution should not be submitted.** We will close PRs that appear to be low-effort AI-generated spam. -- **`server/`** (`agent_control_server`): FastAPI server providing HTTP APIs. - - REST API endpoints - - Business logic services - - Database interactions (Alembic migrations) - - **Change here if**: Adding new endpoints or server functionality - -- **`sdks/python/`** (`agent_control`): Python SDK for users. - - Client library wrapping server APIs - - Control decorators and policies - - Local evaluation support (uses engine) - - **Change here if**: Adding user-facing SDK features - -- **`evaluators/`** (`agent_control_evaluators`): Built-in evaluator implementations. - - JSON, SQL, regex, list evaluators - - Luna2 integration - - All evaluators extend base classes from `models/` - - Third-party packages live in `evaluators/contrib/` (see [Adding a New Evaluator](#adding-a-new-evaluator)) - - **Change here if**: Adding new evaluators or modifying existing ones - -- **`ui/`**: Next.js web application for managing agent controls. - - TypeScript/React frontend - - **Change here if**: Adding UI features (separate contribution process) - -- **`examples/`**: Runnable examples demonstrating integrations. - - LangChain, CrewAI, and other framework examples - - Demo agents and setup scripts - - **Change here if**: Adding new integration examples - -- **`docs/`**: Documentation and architectural guides. - - `OVERVIEW.md`: Architecture overview - - `REFERENCE.md`: API reference - - `testing.md`: Testing conventions - - Evaluator guides in `docs/evaluators/` - - **Change here if**: Improving documentation - -See `AGENTS.md` for detailed development conventions and the full change map. - -## Code Review Process - -Once you've opened a pull request: - -1. **Automated checks**: CI will run tests, linting, and type checking. All checks must pass. - -2. **Maintainer review**: A project maintainer will review your code. This may take a few days depending on the size and complexity of your PR. - -3. **Feedback and iteration**: The reviewer may request changes. Please: - - Address all feedback - - Push new commits to your branch (don't force-push unless asked) - - Respond to comments to acknowledge you've addressed them - - Ask questions if anything is unclear - -4. **Approval and merge**: Once approved, a maintainer will merge your PR. We use squash merge to keep history clean, so your commits will be combined into one. - -5. **Post-merge**: Your contribution will be included in the next release. Thank you! 🎉 - -**Review timeline expectations:** -- Simple PRs (docs, small fixes): Usually within 2-3 days -- Complex PRs (new features, evaluators): May take 5-7 days -- We're a small team, so please be patient - -If your PR has been waiting more than a week without review, feel free to politely ping in the PR comments. +--- ## Need Help? -If you have questions or need guidance: - -- **📚 Read the docs**: - - `docs/OVERVIEW.md` - Architecture overview - - `docs/REFERENCE.md` - API reference - - `docs/testing.md` - Testing conventions - - `AGENTS.md` - Developer guide for AI coding assistants - -- **💡 Check examples**: Review `examples/` for integration patterns. - -- **🔍 Search issues**: Your question may have been answered in [existing issues](https://github.com/rungalileo/agent-control/issues). - -- **💬 Open a discussion**: For questions that don't fit issues, start a [GitHub Discussion](https://github.com/rungalileo/agent-control/discussions). - -- **🐛 Report a problem**: If you're stuck on a bug, [open an issue](https://github.com/rungalileo/agent-control/issues/new) with details. - -We appreciate your contribution to making Agent Control better! ❤️ - -## Quick Reference - -### Common Commands - -```bash -# Setup and dependencies -make sync # Install/sync all dependencies -make hooks-install # Install git pre-commit hooks - -# Development -make dev # Run in development mode -make server-run # Run the server -make check # Run all checks (tests + lint + typecheck) - -# Testing -make test # Run all tests -make engine-test # Test engine package -make sdk-test # Test SDK package -make server-test # Test server package -make evaluators-test # Test evaluators package - -# Code quality -make lint # Check code style -make lint-fix # Auto-fix linting issues -make typecheck # Run mypy type checker - -# Database (server) -make server-alembic-upgrade # Apply database migrations -make server-alembic-downgrade # Rollback migrations -make server-db-seed # Seed database with test data -``` - -### Package Structure - -``` -agent-control/ -├── models/ → agent_control_models -├── engine/ → agent_control_engine -├── server/ → agent_control_server -├── sdks/python/ → agent_control -├── evaluators/ → agent_control_evaluators -├── evaluators/contrib/ → Third-party evaluator packages -├── ui/ → Next.js web app -├── examples/ → Integration examples -└── docs/ → Documentation -``` - -### Conventional Commit Types - -- `feat`: New feature -- `fix`: Bug fix -- `docs`: Documentation changes -- `refactor`: Code refactoring -- `test`: Test additions or changes -- `chore`: Maintenance tasks -- `perf`: Performance improvements +- Read `docs/OVERVIEW.md` (architecture), `docs/REFERENCE.md` (API), `docs/testing.md` (testing), and `AGENTS.md` (dev conventions). +- Check `examples/` for integration patterns. +- Search [existing issues](https://github.com/agentcontrol/agent-control/issues). +- Start a [GitHub Discussion](https://github.com/agentcontrol/agent-control/discussions) for open-ended questions. ## License -Agent Control is Apache 2.0 licensed. See [LICENSE](LICENSE) for more details. +Agent Control is Apache 2.0 licensed. See [LICENSE](LICENSE) for details. -By contributing to Agent Control, you agree that your contributions will be licensed under the Apache 2.0 License. +By contributing, you agree that your contributions will be licensed under the Apache 2.0 License. From 2a56924c785ba5cc3140e608c8d4c5be7cbc9c03 Mon Sep 17 00:00:00 2001 From: Lev Neiman Date: Mon, 2 Mar 2026 15:38:27 -0800 Subject: [PATCH 4/9] updates --- AGENTS.md | 4 +- CONTRIBUTING.md | 109 ++---------------- Makefile | 8 +- docs/OVERVIEW.md | 2 +- evaluators/builtin/pyproject.toml | 2 +- .../{extra => contrib}/galileo/Makefile | 0 .../{extra => contrib}/galileo/README.md | 0 .../{extra => contrib}/galileo/pyproject.toml | 0 .../__init__.py | 0 .../luna2/__init__.py | 0 .../luna2/client.py | 0 .../luna2/config.py | 0 .../luna2/evaluator.py | 0 .../galileo/tests/__init__.py | 0 .../galileo/tests/test_luna2_evaluator.py | 0 .../{extra => contrib}/template/README.md | 2 +- .../template/pyproject.toml.template | 0 examples/galileo/README.md | 3 +- pyproject.toml | 4 +- sdks/python/pyproject.toml | 2 +- server/pyproject.toml | 2 +- 21 files changed, 23 insertions(+), 115 deletions(-) rename evaluators/{extra => contrib}/galileo/Makefile (100%) rename evaluators/{extra => contrib}/galileo/README.md (100%) rename evaluators/{extra => contrib}/galileo/pyproject.toml (100%) rename evaluators/{extra => contrib}/galileo/src/agent_control_evaluator_galileo/__init__.py (100%) rename evaluators/{extra => contrib}/galileo/src/agent_control_evaluator_galileo/luna2/__init__.py (100%) rename evaluators/{extra => contrib}/galileo/src/agent_control_evaluator_galileo/luna2/client.py (100%) rename evaluators/{extra => contrib}/galileo/src/agent_control_evaluator_galileo/luna2/config.py (100%) rename evaluators/{extra => contrib}/galileo/src/agent_control_evaluator_galileo/luna2/evaluator.py (100%) rename evaluators/{extra => contrib}/galileo/tests/__init__.py (100%) rename evaluators/{extra => contrib}/galileo/tests/test_luna2_evaluator.py (100%) rename evaluators/{extra => contrib}/template/README.md (98%) rename evaluators/{extra => contrib}/template/pyproject.toml.template (100%) diff --git a/AGENTS.md b/AGENTS.md index e10f7f93..9ae9e74d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -27,7 +27,7 @@ Forwarded targets: - `server/`: FastAPI server (`server/src/agent_control_server/`) - `sdks/python/`: Python SDK — uses engine for evaluation (`sdks/python/src/agent_control/`) - `evaluators/builtin/`: builtin evaluator implementations (`evaluators/builtin/src/agent_control_evaluators/`) -- `evaluators/extra/`: optional evaluator packages (e.g., `evaluators/extra/galileo/`) +- `evaluators/contrib/`: optional evaluator packages (e.g., `evaluators/contrib/galileo/`) - `ui/`: Nextjs based web app to manage agent controls - `examples/`: runnable examples (ruff has relaxed import rules here) @@ -75,7 +75,7 @@ All testing guidance (including “behavior changes require tests”) lives in ` 5) evaluator is automatically available to server and SDK via `discover_evaluators()` - Add an external evaluator package: - 1) copy `evaluators/extra/template/` as a starting point + 1) copy `evaluators/contrib/template/` as a starting point 2) implement evaluator class extending `Evaluator` from `agent_control_evaluators` 3) add entry point using `org.name` format (e.g., `galileo.luna2`) 4) package is discovered automatically when installed alongside agent-control diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f11cd246..4b74cb36 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,14 +1,13 @@ # Contributing to Agent Control -Thanks for taking the time to contribute! This guide covers the process for contributing to Agent Control. +Thanks for taking the time to contribute! -For development setup, architecture details, and code conventions, see the [README](README.md) and [AGENTS.md](AGENTS.md). +For development setup, architecture details, and code conventions, see the [README](README.md). ## Table of Contents -1. [Integrate Agent Control into Your Agents](#1-integrate-agent-control-into-your-agents) +1. [Integrate Agent Control into Your Agent Framework](#1-integrate-agent-control-into-your-agents) 2. [Contribute New Evaluators](#2-contribute-new-evaluators) -3. [Extend Agent Control](#3-extend-agent-control) 4. [Improve Code and Documentation](#4-improve-code-and-documentation) 5. [Suggest Features and Report Bugs](#5-suggest-features-and-report-bugs) @@ -18,79 +17,30 @@ For development setup, architecture details, and code conventions, see the [READ - **For significant changes, open an issue first.** Discussing your approach ahead of time avoids wasted effort. Changes that were not discussed may be rejected. - **For small fixes** (typos, minor bug fixes), you can go straight to a pull request. -- **Claim your work.** If you start on an issue, comment on it or assign yourself to avoid duplicate effort. -- Look for issues labeled [good first issue](https://github.com/agentcontrol/agent-control/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) or **help wanted** if you're new. --- -## 1. Integrate Agent Control into Your Agents +## 1. Integrate Agent Control into Your Agent Framework We welcome examples showing how Agent Control works with different agent frameworks. -To add an integration example: - -1. Create a directory in `examples/` (e.g., `examples/my_framework/`). -2. Include a `README.md` with setup instructions, a `pyproject.toml` for dependencies, and runnable Python files. -3. Update `examples/README.md` to list your new example. -4. Open a PR. - -See existing examples in `examples/` for the expected structure. +See existing examples in [examples/](examples/) for the expected structure. --- ## 2. Contribute New Evaluators -Evaluators are a core extension point. We support two paths: - -| Path | Use when | Location | -| --- | --- | --- | -| Built-in | Lightweight deps, broadly useful | `evaluators/src/agent_control_evaluators/builtin/` | -| Contrib | Heavy deps or vendor-specific | `evaluators/contrib/agent-control-evaluator-/` | - -**Steps:** - -1. Create the evaluator class extending `Evaluator` from `agent_control_models`. -2. Register via `@register_evaluator` and add the entry point in `pyproject.toml`. -3. Add tests and documentation. -4. Run `make evaluators-test` (and ideally `make check`). -5. Open a PR. - -For built-in evaluators, add your module to `evaluators/src/agent_control_evaluators/builtin/` and register the entry point in `evaluators/pyproject.toml`. - -For contrib evaluators, create a new package under `evaluators/contrib/` with its own `pyproject.toml`. Entry points must be namespaced as `org.evaluator_name`. Add a convenience extra in `evaluators/pyproject.toml` and include the package in the workspace members in the root `pyproject.toml`. - -See existing evaluators (e.g., `regex`, `list`) and `AGENTS.md` for detailed conventions and code patterns. +See [evaluators/contrib/template/README.md](evaluators/contrib/template/README.md) for instructions on how to create a new evaluator package. --- -## 3. Extend Agent Control - -For changes to the core system (new API endpoints, SDK features, engine logic): - -1. **Models** (`models/`): Shared Pydantic models and base classes. Change here when adding new API request/response types or evaluator base functionality. -2. **Engine** (`engine/`): Evaluation logic and evaluator discovery. Change here when modifying how controls are evaluated. -3. **Server** (`server/`): FastAPI endpoints and business logic. Change here when adding API endpoints or server functionality. -4. **SDK** (`sdks/python/`): Python client library. Change here when adding user-facing SDK features. -5. **UI** (`ui/`): Next.js web dashboard. Change here for frontend features. - -See `AGENTS.md` for the full architecture diagram, dependency flow, and code conventions. - ---- - -## 4. Improve Code and Documentation - -Documentation and code quality improvements are always welcome: - -- **Typos and clarity fixes**: Open a PR directly. -- **Missing examples**: Add to `docs/` or `examples/`. -- **API documentation**: Docstrings in the code are the source of truth. -- **Architecture guides**: Update files in `docs/`. +## 3. Improve Code and Documentation -Run `make check` before submitting to ensure nothing is broken. +Agent control is under active development and documentation and code quality improvements are always welcome! --- -## 5. Suggest Features and Report Bugs +## 4. Suggest Features and Report Bugs ### Reporting Bugs @@ -106,47 +56,6 @@ Run `make check` before submitting to ensure nothing is broken. --- -## Contribution Workflow - -### Fork, Branch, and PR - -1. **Fork** the repository and clone your fork. -2. **Create a branch** with a descriptive name: `feature/...`, `fix/...`, `docs/...`, or `refactor/...`. -3. **Make your changes.** Keep diffs focused. Follow the conventions in [AGENTS.md](AGENTS.md). -4. **Write tests.** All behavior changes require tests. See `docs/testing.md`. -5. **Run checks locally**: `make check` (runs tests, lint, and typecheck). -6. **Commit** using [Conventional Commits](https://www.conventionalcommits.org/): `feat:`, `fix:`, `docs:`, `refactor:`, `test:`, `chore:`, `perf:`. -7. **Push** and open a PR against `main`. Link related issues (e.g., "Fixes #123"). - -### Code Review - -1. **CI must pass.** All automated checks (tests, lint, typecheck) must be green. -2. **A maintainer will review.** Simple PRs: 2-3 days. Complex PRs: up to a week. -3. **Address feedback** by pushing new commits (don't force-push unless asked). -4. **Merge.** We use squash merge to keep history clean. - -If your PR has waited more than a week, feel free to politely ping in the comments. - ---- - -## Acceptable Use of AI Tools - -AI tools can help with contributions, but **AI assistance must be paired with meaningful human review and judgment.** - -- **Acceptable**: Using AI for boilerplate, understanding unfamiliar code, or generating a starting point that you refine. -- **Not acceptable**: Submitting entirely AI-generated code without review, mass automated contributions, or low-effort spam PRs. - -**If the effort to create a PR is less than the effort to review it, that contribution should not be submitted.** We will close PRs that appear to be low-effort AI-generated spam. - ---- - -## Need Help? - -- Read `docs/OVERVIEW.md` (architecture), `docs/REFERENCE.md` (API), `docs/testing.md` (testing), and `AGENTS.md` (dev conventions). -- Check `examples/` for integration patterns. -- Search [existing issues](https://github.com/agentcontrol/agent-control/issues). -- Start a [GitHub Discussion](https://github.com/agentcontrol/agent-control/discussions) for open-ended questions. - ## License Agent Control is Apache 2.0 licensed. See [LICENSE](LICENSE) for details. diff --git a/Makefile b/Makefile index 7d54fa70..599ec081 100644 --- a/Makefile +++ b/Makefile @@ -15,7 +15,7 @@ SDK_DIR := sdks/python TS_SDK_DIR := sdks/typescript ENGINE_DIR := engine EVALUATORS_DIR := evaluators/builtin -GALILEO_DIR := evaluators/extra/galileo +GALILEO_DIR := evaluators/contrib/galileo help: @echo "Agent Control - Makefile commands" @@ -30,7 +30,7 @@ help: @echo "" @echo "Test:" @echo " make test - run tests for core packages (server, engine, sdk, evaluators)" - @echo " make test-extras - run tests for extra evaluators (galileo, etc.)" + @echo " make test-extras - run tests for contrib evaluators (galileo, etc.)" @echo " make test-all - run all tests (core + extras)" @echo " make sdk-ts-test - run TypeScript SDK tests" @echo "" @@ -81,7 +81,7 @@ openapi-spec-check: openapi-spec test: server-test engine-test sdk-test evaluators-test -# Run tests for extra evaluators (not included in default test target) +# Run tests for contrib evaluators (not included in default test target) test-extras: galileo-test # Run all tests (core + extras) @@ -215,7 +215,7 @@ server-%: $(MAKE) -C $(SERVER_DIR) $(patsubst server-%,%,$@) # --------------------------- -# Extra Evaluators (Galileo) +# Contrib Evaluators (Galileo) # --------------------------- galileo-test: diff --git a/docs/OVERVIEW.md b/docs/OVERVIEW.md index bc427af9..0a098137 100644 --- a/docs/OVERVIEW.md +++ b/docs/OVERVIEW.md @@ -664,7 +664,7 @@ We welcome contributions! See [CONTRIBUTING.md](../CONTRIBUTING.md) for guidelin ### Adding an Evaluator 1. Fork the repository -2. Create your evaluator in `evaluators/builtin/src/agent_control_evaluators/` (for builtins) or in `evaluators/extra/` (for external packages) +2. Create your evaluator in `evaluators/builtin/src/agent_control_evaluators/` (for builtins) or in `evaluators/contrib/` (for external packages) 3. Add tests in `evaluators/tests/` 4. Submit a pull request diff --git a/evaluators/builtin/pyproject.toml b/evaluators/builtin/pyproject.toml index 0609af6d..07729bc0 100644 --- a/evaluators/builtin/pyproject.toml +++ b/evaluators/builtin/pyproject.toml @@ -34,4 +34,4 @@ packages = ["src/agent_control_evaluators"] [tool.uv.sources] agent-control-models = { workspace = true } # For local dev: use local galileo package instead of PyPI -agent-control-evaluator-galileo = { path = "../extra/galileo", editable = true } +agent-control-evaluator-galileo = { path = "../contrib/galileo", editable = true } diff --git a/evaluators/extra/galileo/Makefile b/evaluators/contrib/galileo/Makefile similarity index 100% rename from evaluators/extra/galileo/Makefile rename to evaluators/contrib/galileo/Makefile diff --git a/evaluators/extra/galileo/README.md b/evaluators/contrib/galileo/README.md similarity index 100% rename from evaluators/extra/galileo/README.md rename to evaluators/contrib/galileo/README.md diff --git a/evaluators/extra/galileo/pyproject.toml b/evaluators/contrib/galileo/pyproject.toml similarity index 100% rename from evaluators/extra/galileo/pyproject.toml rename to evaluators/contrib/galileo/pyproject.toml diff --git a/evaluators/extra/galileo/src/agent_control_evaluator_galileo/__init__.py b/evaluators/contrib/galileo/src/agent_control_evaluator_galileo/__init__.py similarity index 100% rename from evaluators/extra/galileo/src/agent_control_evaluator_galileo/__init__.py rename to evaluators/contrib/galileo/src/agent_control_evaluator_galileo/__init__.py diff --git a/evaluators/extra/galileo/src/agent_control_evaluator_galileo/luna2/__init__.py b/evaluators/contrib/galileo/src/agent_control_evaluator_galileo/luna2/__init__.py similarity index 100% rename from evaluators/extra/galileo/src/agent_control_evaluator_galileo/luna2/__init__.py rename to evaluators/contrib/galileo/src/agent_control_evaluator_galileo/luna2/__init__.py diff --git a/evaluators/extra/galileo/src/agent_control_evaluator_galileo/luna2/client.py b/evaluators/contrib/galileo/src/agent_control_evaluator_galileo/luna2/client.py similarity index 100% rename from evaluators/extra/galileo/src/agent_control_evaluator_galileo/luna2/client.py rename to evaluators/contrib/galileo/src/agent_control_evaluator_galileo/luna2/client.py diff --git a/evaluators/extra/galileo/src/agent_control_evaluator_galileo/luna2/config.py b/evaluators/contrib/galileo/src/agent_control_evaluator_galileo/luna2/config.py similarity index 100% rename from evaluators/extra/galileo/src/agent_control_evaluator_galileo/luna2/config.py rename to evaluators/contrib/galileo/src/agent_control_evaluator_galileo/luna2/config.py diff --git a/evaluators/extra/galileo/src/agent_control_evaluator_galileo/luna2/evaluator.py b/evaluators/contrib/galileo/src/agent_control_evaluator_galileo/luna2/evaluator.py similarity index 100% rename from evaluators/extra/galileo/src/agent_control_evaluator_galileo/luna2/evaluator.py rename to evaluators/contrib/galileo/src/agent_control_evaluator_galileo/luna2/evaluator.py diff --git a/evaluators/extra/galileo/tests/__init__.py b/evaluators/contrib/galileo/tests/__init__.py similarity index 100% rename from evaluators/extra/galileo/tests/__init__.py rename to evaluators/contrib/galileo/tests/__init__.py diff --git a/evaluators/extra/galileo/tests/test_luna2_evaluator.py b/evaluators/contrib/galileo/tests/test_luna2_evaluator.py similarity index 100% rename from evaluators/extra/galileo/tests/test_luna2_evaluator.py rename to evaluators/contrib/galileo/tests/test_luna2_evaluator.py diff --git a/evaluators/extra/template/README.md b/evaluators/contrib/template/README.md similarity index 98% rename from evaluators/extra/template/README.md rename to evaluators/contrib/template/README.md index cd52e07b..21e8e18e 100644 --- a/evaluators/extra/template/README.md +++ b/evaluators/contrib/template/README.md @@ -80,7 +80,7 @@ class MyEvaluator(Evaluator[MyEvaluatorConfig]): Run tests with: ```bash -cd evaluators/extra/{{org}} +cd evaluators/contrib/{{org}} uv run pytest ``` diff --git a/evaluators/extra/template/pyproject.toml.template b/evaluators/contrib/template/pyproject.toml.template similarity index 100% rename from evaluators/extra/template/pyproject.toml.template rename to evaluators/contrib/template/pyproject.toml.template diff --git a/examples/galileo/README.md b/examples/galileo/README.md index 96e141d0..5ccb9625 100644 --- a/examples/galileo/README.md +++ b/examples/galileo/README.md @@ -84,5 +84,4 @@ Testing toxicity detection with Central Stage... - [Galileo Protect Overview](https://v2docs.galileo.ai/concepts/protect/overview) - [Luna-2 Python API Reference](https://v2docs.galileo.ai/sdk-api/python/reference/protect) -- [Agent Control Luna-2 Evaluator](../../evaluators/extra/galileo/) - +- [Agent Control Luna-2 Evaluator](../../evaluators/contrib/galileo/) diff --git a/pyproject.toml b/pyproject.toml index 3944b70c..c79a1dd3 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -20,7 +20,7 @@ members = [ "sdks/python", "engine", "evaluators/builtin", - # NOTE: evaluators/extra/* excluded - install separately when needed + # NOTE: evaluators/contrib/* excluded - install separately when needed ] [tool.uv] @@ -69,7 +69,7 @@ version_toml = [ "sdks/python/pyproject.toml:project.version", "server/pyproject.toml:project.version", "evaluators/builtin/pyproject.toml:project.version", - "evaluators/extra/galileo/pyproject.toml:project.version", + "evaluators/contrib/galileo/pyproject.toml:project.version", ] version_source = "tag" commit_message = "chore(release): v{version}" diff --git a/sdks/python/pyproject.toml b/sdks/python/pyproject.toml index bf9c6e2e..5e15ed1c 100644 --- a/sdks/python/pyproject.toml +++ b/sdks/python/pyproject.toml @@ -79,4 +79,4 @@ agent-control-models = { workspace = true } agent-control-engine = { workspace = true } agent-control-evaluators = { workspace = true } # For local dev: use local galileo package instead of PyPI -agent-control-evaluator-galileo = { path = "../../evaluators/extra/galileo", editable = true } +agent-control-evaluator-galileo = { path = "../../evaluators/contrib/galileo", editable = true } diff --git a/server/pyproject.toml b/server/pyproject.toml index 29b9c52d..fac2213d 100644 --- a/server/pyproject.toml +++ b/server/pyproject.toml @@ -88,4 +88,4 @@ agent-control-models = { workspace = true } agent-control-engine = { workspace = true } agent-control-evaluators = { workspace = true } # For local dev: use local galileo package instead of PyPI -agent-control-evaluator-galileo = { path = "../evaluators/extra/galileo", editable = true } +agent-control-evaluator-galileo = { path = "../evaluators/contrib/galileo", editable = true } From dbe26946b4c4222d821e407e063347151e977fc3 Mon Sep 17 00:00:00 2001 From: Lev Neiman Date: Mon, 2 Mar 2026 15:59:02 -0800 Subject: [PATCH 5/9] docs: add default GitHub PR and issue templates --- .github/ISSUE_TEMPLATE/general.md | 30 ++++++++++++++++++++++++++++++ .github/pull_request_template.md | 21 +++++++++++++++++++++ 2 files changed, 51 insertions(+) create mode 100644 .github/ISSUE_TEMPLATE/general.md create mode 100644 .github/pull_request_template.md diff --git a/.github/ISSUE_TEMPLATE/general.md b/.github/ISSUE_TEMPLATE/general.md new file mode 100644 index 00000000..00d63385 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/general.md @@ -0,0 +1,30 @@ +--- +name: General issue +about: Report a bug, request a feature, or propose a change +title: "[Issue] " +labels: [] +assignees: [] +--- + +## Summary +- What is the problem or request? + +## Motivation +- Why does this matter? + +## Current behavior +- What happens today? + +## Expected behavior +- What should happen instead? + +## Reproduction (if bug) +1. +2. +3. + +## Proposed solution (optional) +- Suggested approach, tradeoffs, or constraints. + +## Additional context +- Links, screenshots, logs, related issues/PRs. diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 00000000..127f9d4d --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,21 @@ +## Summary +- What changed and why. + +## Scope +- User-facing/API changes: +- Internal changes: +- Out of scope: + +## Risk and Rollout +- Risk level: low / medium / high +- Rollback plan: + +## Testing +- [ ] Added or updated automated tests +- [ ] Ran `make check` (or explained why not) +- [ ] Manually verified behavior + +## Checklist +- [ ] Linked issue/spec (if applicable) +- [ ] Updated docs/examples for user-facing changes +- [ ] Included any required follow-up tasks From 3bf32a3cbf0dba2f0144e63a5d868f07d0f98cfc Mon Sep 17 00:00:00 2001 From: Lev Neiman Date: Mon, 2 Mar 2026 16:07:25 -0800 Subject: [PATCH 6/9] fixups --- CONTRIBUTING.md | 50 +++++++++++++++---------------------------------- 1 file changed, 15 insertions(+), 35 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 4b74cb36..321aaacc 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,23 +2,22 @@ Thanks for taking the time to contribute! -For development setup, architecture details, and code conventions, see the [README](README.md). - -## Table of Contents +## Before You Start -1. [Integrate Agent Control into Your Agent Framework](#1-integrate-agent-control-into-your-agents) -2. [Contribute New Evaluators](#2-contribute-new-evaluators) -4. [Improve Code and Documentation](#4-improve-code-and-documentation) -5. [Suggest Features and Report Bugs](#5-suggest-features-and-report-bugs) +- **For significant changes, please open an issue first.** Discussing your approach ahead of time avoids wasted effort. Changes that were not discussed may be rejected. +- **For small fixes** (typos, minor bug fixes), you can go straight to a pull request. ---- +### Reporting Bugs -## Before You Start +1. **Search first**: Check [GitHub Issues](https://github.com/agentcontrol/agent-control/issues) for duplicates. If you find one, add context in a comment rather than opening a new issue. +2. **Create an issue**: Make sure to fill out the template and give us enough context to understand and reproduce the issue. +3. **Keep issues focused.** One topic per issue. Link related issues rather than combining them. -- **For significant changes, open an issue first.** Discussing your approach ahead of time avoids wasted effort. Changes that were not discussed may be rejected. -- **For small fixes** (typos, minor bug fixes), you can go straight to a pull request. +### Suggesting Features ---- +1. **Search first**: Check [existing feature requests](https://github.com/agentcontrol/agent-control/issues?q=is%3Aissue+label%3Aenhancement). +2. **Open an issue** with the `enhancement` label. Describe the use case, explain the value, provide examples, and consider alternatives. +3. Be open to discussion and iteration on your idea. ## 1. Integrate Agent Control into Your Agent Framework @@ -26,35 +25,16 @@ We welcome examples showing how Agent Control works with different agent framewo See existing examples in [examples/](examples/) for the expected structure. ---- - ## 2. Contribute New Evaluators See [evaluators/contrib/template/README.md](evaluators/contrib/template/README.md) for instructions on how to create a new evaluator package. ---- - ## 3. Improve Code and Documentation -Agent control is under active development and documentation and code quality improvements are always welcome! - ---- - -## 4. Suggest Features and Report Bugs - -### Reporting Bugs - -1. **Search first**: Check [GitHub Issues](https://github.com/agentcontrol/agent-control/issues) for duplicates. If you find one, add context in a comment rather than opening a new issue. -2. **Create an issue** with: clear title, steps to reproduce ([minimal example](https://stackoverflow.com/help/minimal-reproducible-example)), expected vs. actual behavior, environment details (OS, Python version, Agent Control version), and full error messages/stack traces. -3. **Keep issues focused.** One topic per issue. Link related issues rather than combining them. - -### Suggesting Features - -1. **Search first**: Check [existing feature requests](https://github.com/agentcontrol/agent-control/issues?q=is%3Aissue+label%3Aenhancement). -2. **Open an issue** with the `enhancement` label. Describe the use case, explain the value, provide examples, and consider alternatives. -3. Be open to discussion and iteration on your idea. - ---- +Your PR must fill out the template. + - All tests must pass. + - Code changes must be covered by behavioral tests. + - Related documentation must be updated. ## License From 95135c468394bd83dcb4674e1b98f968f50d50b7 Mon Sep 17 00:00:00 2001 From: Lev Neiman Date: Tue, 3 Mar 2026 15:13:46 -0800 Subject: [PATCH 7/9] fix: update evaluators/extra/ references to evaluators/contrib/ Update release.yaml and .githooks/pre-push to use the renamed evaluators/contrib/galileo path so releases publish correctly and local pre-push hooks run Galileo lint/typecheck. --- .githooks/pre-push | 6 +++--- .github/workflows/release.yaml | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.githooks/pre-push b/.githooks/pre-push index c4ac972d..d96954b5 100755 --- a/.githooks/pre-push +++ b/.githooks/pre-push @@ -16,9 +16,9 @@ echo "pre-push: running make typecheck" make typecheck # Check extras if they exist and have changes -if [ -d "evaluators/extra/galileo" ]; then - echo "pre-push: checking evaluators/extra/galileo" - cd evaluators/extra/galileo +if [ -d "evaluators/contrib/galileo" ]; then + echo "pre-push: checking evaluators/contrib/galileo" + cd evaluators/contrib/galileo uv run --extra dev ruff check --config ../../../pyproject.toml src/ uv run --extra dev mypy --config-file ../../../pyproject.toml src/ cd "$REPO_ROOT" diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml index f7f09dfb..8103788a 100644 --- a/.github/workflows/release.yaml +++ b/.github/workflows/release.yaml @@ -71,7 +71,7 @@ jobs: if: steps.release.outputs.released == 'true' uses: pypa/gh-action-pypi-publish@release/v1 with: - packages-dir: evaluators/extra/galileo/dist/ + packages-dir: evaluators/contrib/galileo/dist/ user: __token__ password: ${{ secrets.PYPI_API_TOKEN }} @@ -87,4 +87,4 @@ jobs: evaluators/builtin/dist/* sdks/python/dist/* server/dist/* - evaluators/extra/galileo/dist/* + evaluators/contrib/galileo/dist/* From 6ef446289d96183e7261ebdf4b723d94a679b93f Mon Sep 17 00:00:00 2001 From: Lev Neiman Date: Tue, 3 Mar 2026 15:17:16 -0800 Subject: [PATCH 8/9] fix: correct test path in OVERVIEW.md and remove redundant [Issue] prefix Update evaluator test path to match actual directory structure and drop the [Issue] title prefix from the general issue template. --- .github/ISSUE_TEMPLATE/general.md | 2 +- docs/OVERVIEW.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/general.md b/.github/ISSUE_TEMPLATE/general.md index 00d63385..0db637ae 100644 --- a/.github/ISSUE_TEMPLATE/general.md +++ b/.github/ISSUE_TEMPLATE/general.md @@ -1,7 +1,7 @@ --- name: General issue about: Report a bug, request a feature, or propose a change -title: "[Issue] " +title: "" labels: [] assignees: [] --- diff --git a/docs/OVERVIEW.md b/docs/OVERVIEW.md index cdfc324b..0614ddfa 100644 --- a/docs/OVERVIEW.md +++ b/docs/OVERVIEW.md @@ -662,7 +662,7 @@ We welcome contributions! See [CONTRIBUTING.md](../CONTRIBUTING.md) for guidelin 1. Fork the repository 2. Create your evaluator in `evaluators/builtin/src/agent_control_evaluators/` (for builtins) or in `evaluators/contrib/` (for external packages) -3. Add tests in `evaluators/tests/` +3. Add tests in `evaluators/builtin/tests/` (for builtins) or `evaluators/contrib//tests/` (for contrib) 4. Submit a pull request --- From 637a95ca7443d7d11b4fc92ea492e2fc8cec84ce Mon Sep 17 00:00:00 2001 From: Lev Neiman Date: Tue, 3 Mar 2026 15:32:02 -0800 Subject: [PATCH 9/9] docs: replace contrib template README with expanded guide Use Abhinav's human-friendly version with complete examples for config, evaluator, entry points, testing, and pre-submit checks. --- evaluators/contrib/template/README.md | 244 ++++++++++++++++++++------ 1 file changed, 187 insertions(+), 57 deletions(-) diff --git a/evaluators/contrib/template/README.md b/evaluators/contrib/template/README.md index 21e8e18e..b71e1dff 100644 --- a/evaluators/contrib/template/README.md +++ b/evaluators/contrib/template/README.md @@ -1,100 +1,230 @@ -# Evaluator Package Template +Creating a Contrib Evaluator +============================ -This template provides a starting point for creating new evaluator packages for agent-control. +This guide walks you through building and publishing a new evaluator package for agent-control. Start to finish, it takes about 15 minutes. -## Setup +For a working reference, see the [Galileo evaluator](../galileo/). -1. Copy this template: `cp -r template/ {{org}}/` -2. Replace placeholders in pyproject.toml.template: - - `{{ORG}}`: Your organization name (e.g., `acme`) - - `{{EVALUATOR}}`: Evaluator name (e.g., `toxicity`) - - `{{CLASS}}`: Python class name (e.g., `ToxicityEvaluator`) - - `{{AUTHOR}}`: Author name -3. Rename to `pyproject.toml` -4. Create your evaluator in `src/agent_control_evaluator_{{org}}/` -5. Register via entry point in pyproject.toml +Quick Start +----------- -## Directory Structure +Pick your org name and evaluator name. Everything else derives from these two: + +> **Example**: org = `acme`, evaluator = `toxicity` +> +> * PyPI package: `agent-control-evaluator-acme` +> * Python package: `agent_control_evaluator_acme` +> * Entry point: `acme.toxicity` +> * Evaluator class: `AcmeToxicityEvaluator` + +From the repo root: + +```bash +cp -r evaluators/contrib/template/ evaluators/contrib/acme/ +mv evaluators/contrib/acme/pyproject.toml.template evaluators/contrib/acme/pyproject.toml +``` + +Edit `pyproject.toml` and replace all placeholders (`{ORG}`, `{EVALUATOR}`, `{CLASS}`, `{AUTHOR}`). Then create the source layout: + +```bash +mkdir -p evaluators/contrib/acme/src/agent_control_evaluator_acme/toxicity +mkdir -p evaluators/contrib/acme/tests +touch evaluators/contrib/acme/src/agent_control_evaluator_acme/__init__.py +touch evaluators/contrib/acme/src/agent_control_evaluator_acme/toxicity/__init__.py +touch evaluators/contrib/acme/src/agent_control_evaluator_acme/toxicity/config.py +touch evaluators/contrib/acme/src/agent_control_evaluator_acme/toxicity/evaluator.py +touch evaluators/contrib/acme/tests/__init__.py +touch evaluators/contrib/acme/tests/test_toxicity.py +``` + +You'll end up with: ``` -{{org}}/ +acme/ ├── pyproject.toml -├── src/agent_control_evaluator_{{org}}/ +├── src/agent_control_evaluator_acme/ │ ├── __init__.py -│ └── {{evaluator}}/ +│ └── toxicity/ │ ├── __init__.py -│ ├── config.py # Extends EvaluatorConfig -│ └── evaluator.py # Extends Evaluator, uses @register_evaluator +│ ├── config.py +│ └── evaluator.py └── tests/ ├── __init__.py - └── test_{{evaluator}}.py + └── test_toxicity.py ``` -## Entry Point Naming Convention +Writing the Evaluator +--------------------- -Use `org.evaluator_name` format (e.g., `acme.toxicity`). - -This naming convention: -- Uses dots (`.`) as separators for external evaluators -- Distinguishes from built-in evaluators (no namespace) and agent-scoped evaluators (colon separator) - -## Implementation Pattern - -Your evaluator should: - -1. **Extend `EvaluatorConfig`** for configuration: +**Config** - extend `EvaluatorConfig` with your evaluator's settings: ```python +# toxicity/config.py +from pydantic import Field from agent_control_evaluators import EvaluatorConfig -class MyEvaluatorConfig(EvaluatorConfig): - threshold: float = 0.5 - # ... other config fields +class AcmeToxicityConfig(EvaluatorConfig): + threshold: float = Field(default=0.7, ge=0.0, le=1.0) + categories: list[str] = Field(default_factory=lambda: ["hate", "violence"]) ``` -2. **Extend `Evaluator` and use `@register_evaluator`**: +**Evaluator** - extend `Evaluator` and decorate with `@register_evaluator`: ```python +# toxicity/evaluator.py +from typing import Any + from agent_control_evaluators import Evaluator, EvaluatorMetadata, register_evaluator from agent_control_models import EvaluatorResult +from agent_control_evaluator_acme.toxicity.config import AcmeToxicityConfig + @register_evaluator -class MyEvaluator(Evaluator[MyEvaluatorConfig]): +class AcmeToxicityEvaluator(Evaluator[AcmeToxicityConfig]): metadata = EvaluatorMetadata( - name="myorg.myevaluator", # Must match entry point + name="acme.toxicity", # Must match entry point key exactly version="1.0.0", - description="My custom evaluator", + description="Acme toxicity detection", + requires_api_key=True, # Set if you need external credentials + timeout_ms=5000, # Timeout for external calls ) - config_model = MyEvaluatorConfig + config_model = AcmeToxicityConfig async def evaluate(self, data: Any) -> EvaluatorResult: - # Your evaluation logic here - return EvaluatorResult( - matched=..., - confidence=..., - message=..., - ) + if data is None: + return EvaluatorResult(matched=False, confidence=1.0, message="No data") + + try: + score = await self._score(str(data)) + return EvaluatorResult( + matched=score >= self.config.threshold, + confidence=score, + message=f"Toxicity: {score:.2f}", + ) + except Exception as e: + # Fail-open on infrastructure errors + return EvaluatorResult( + matched=False, confidence=0.0, + message=f"Failed: {e}", error=str(e), + ) + + async def _score(self, text: str) -> float: + # Your API call or local logic here + ... ``` -## Testing +**Entry point** in `pyproject.toml` - this is how discovery finds your evaluator: -Run tests with: -```bash -cd evaluators/contrib/{{org}} -uv run pytest +```toml +[project.entry-points."agent_control.evaluators"] +"acme.toxicity" = "agent_control_evaluator_acme.toxicity:AcmeToxicityEvaluator" ``` -## Publishing +The entry point key (`acme.toxicity`) must exactly match `metadata.name` in the evaluator class. If these don't match, `get_evaluator()` returns `None`. + +**Exports** in `toxicity/__init__.py`: + +```python +from agent_control_evaluator_acme.toxicity.config import AcmeToxicityConfig +from agent_control_evaluator_acme.toxicity.evaluator import AcmeToxicityEvaluator + +__all__ = ["AcmeToxicityEvaluator", "AcmeToxicityConfig"] +``` + +Testing +------- + +Write tests using Given/When/Then style. Cover at least three cases: + +1. **Null input** - returns `matched=False`, no error +2. **Normal evaluation** - returns correct `matched` based on threshold +3. **Infrastructure failure** - returns `matched=False` with `error` set (fail-open) + +```python +# tests/test_toxicity.py +import pytest +from agent_control_evaluator_acme.toxicity import AcmeToxicityEvaluator, AcmeToxicityConfig + +@pytest.fixture +def evaluator() -> AcmeToxicityEvaluator: + return AcmeToxicityEvaluator(AcmeToxicityConfig(threshold=0.5)) + +@pytest.mark.asyncio +async def test_none_input(evaluator): + result = await evaluator.evaluate(None) + assert result.matched is False + assert result.error is None + +@pytest.mark.asyncio +async def test_score_above_threshold_matches(evaluator, monkeypatch): + async def _high(self, text): + return 0.8 + + monkeypatch.setattr(AcmeToxicityEvaluator, "_score", _high) + result = await evaluator.evaluate("test") + assert result.matched is True + assert result.error is None + +@pytest.mark.asyncio +async def test_api_failure_fails_open(evaluator, monkeypatch): + async def _fail(self, text): + raise ConnectionError("timeout") + + monkeypatch.setattr(AcmeToxicityEvaluator, "_score", _fail) + result = await evaluator.evaluate("test") + assert result.matched is False + assert result.error is not None +``` + +Rules to Know +------------- + +**Error handling** - The `error` field is only for infrastructure failures (network errors, API 500s, missing credentials). If your evaluator ran and produced a judgment, that's `matched=True` or `matched=False` - not an error. When `error` is set, `matched` must be `False` (fail-open). + +**Thread safety** - Evaluator instances are cached and reused across concurrent requests. Never store request-scoped state on `self`. Use local variables in `evaluate()`. + +**Performance** - Pre-compile patterns in `__init__()`. Use `asyncio.to_thread()` for CPU-bound work. Respect `timeout_ms` for external calls. + +Before You Submit +----------------- + +From the repo root: -Build and publish your package: ```bash -uv build -uv publish +PKG=evaluators/contrib/acme + +# Check for leftover placeholders (should print nothing; non-zero exit is OK here) +grep -rn '{ORG}\|{EVALUATOR}\|{CLASS}\|{AUTHOR}' "$PKG"/ || true + +# Lint, typecheck, test +(cd "$PKG" && uv run --extra dev ruff check --config ../../../pyproject.toml src/) +(cd "$PKG" && uv run --extra dev mypy --config-file ../../../pyproject.toml src/) +(cd "$PKG" && uv run pytest) + +# Verify discovery works +(cd "$PKG" && uv run python -c " +from agent_control_evaluators import discover_evaluators, get_evaluator +discover_evaluators() +ev = get_evaluator('acme.toxicity') +assert ev is not None, 'Discovery failed - entry point key does not match metadata.name' +print(f'OK: {ev.metadata.name}') +") + +# Build +(cd "$PKG" && uv build) ``` -Once published, users can install via: +Publishing +---------- + ```bash -pip install agent-control-evaluator-{{org}} +(cd evaluators/contrib/acme && uv build && uv publish) ``` -The evaluator will be automatically discovered via entry points when used alongside agent-control. +Users install with `pip install agent-control-evaluator-acme` and the evaluator is discovered automatically. + +Reference +--------- + +* [Galileo evaluator](../galileo/) - complete working example +* [Built-in evaluators](../../builtin/src/agent_control_evaluators/) - regex, list, json, sql patterns