Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/replicantx.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ jobs:
env:
SUPABASE_URL: ${{ secrets.SUPABASE_URL }}
SUPABASE_ANON_KEY: ${{ secrets.SUPABASE_ANON_KEY }}
REPLICANTX_TARGET: pr-${{ github.event.pull_request.number }}-helix-api.onrender.com
REPLICANTX_TARGET: pr-${{ github.event.pull_request.number }}-your-api.onrender.com
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
Expand Down
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -209,5 +209,7 @@ helix_tests/
*.secret.yaml
*_private.yaml
*_secret.yaml
tests/*.md
tests/*.json

reports/
112 changes: 112 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
- **Technical Debugging**: Debug mode (`--debug`) with detailed HTTP, validation, and AI processing logs
- **Multiple Authentication**: Supabase email+password, custom JWT, or no-auth
- **CLI Interface**: Easy-to-use command-line interface with `replicantx run`
- **Parallel Execution**: Run multiple test scenarios concurrently for faster execution
- **Automatic .env Loading**: No manual environment variable sourcing required
- **GitHub Actions Ready**: Built-in workflow for PR testing with Render preview URLs
- **Rich Reporting**: Markdown and JSON reports with timing and assertion results
Expand Down Expand Up @@ -394,6 +395,12 @@ replicantx run tests/*.yaml --debug
# Combined monitoring and debugging
replicantx run tests/*.yaml --debug --watch

# Run tests in parallel for faster execution
replicantx run tests/*.yaml --parallel

# Run with limited concurrency to prevent API overload
replicantx run tests/*.yaml --parallel --max-concurrent 3

# Validate test files without running
replicantx validate tests/*.yaml --verbose
```
Expand Down Expand Up @@ -510,6 +517,111 @@ replicantx run tests/*.yaml --debug --ci
replicantx run tests/*.yaml --debug --verbose --report performance.json
```

### ⚡ Parallel Test Execution

ReplicantX supports parallel execution of test scenarios for significantly faster test runs, especially when testing multiple scenarios against the same API.

#### 🚀 **Basic Parallel Execution**

Run all scenarios in parallel (overrides individual scenario settings):

```bash
# Run all tests in parallel
replicantx run tests/*.yaml --parallel

# Run with limited concurrency to prevent API overload
replicantx run tests/*.yaml --parallel --max-concurrent 3
```

#### 📋 **Per-Scenario Configuration**

Control parallel execution at the scenario level:

```yaml
# tests/parallel_scenario.yaml
name: "Parallel Test Scenario"
base_url: "https://api.example.com/chat"
auth:
provider: noop
level: basic
parallel: true # Enable parallel execution for this scenario
steps:
- user: "Hello, test message"
expect_contains: ["response"]
```

```yaml
# tests/sequential_scenario.yaml
name: "Sequential Test Scenario"
base_url: "https://api.example.com/chat"
auth:
provider: noop
level: basic
parallel: false # Run sequentially (default)
steps:
- user: "Hello, test message"
expect_contains: ["response"]
```

#### 🔄 **Execution Modes**

**Automatic Detection:**
- If any scenario has `parallel: true`, all scenarios run in parallel
- If `--parallel` flag is used, all scenarios run in parallel (overrides individual settings)
- Otherwise, scenarios run sequentially

**Mixed Execution:**
```bash
# Some scenarios parallel, some sequential - all run in parallel
replicantx run tests/parallel_*.yaml tests/sequential_*.yaml
```

#### ⚙️ **Concurrency Control**

**Unlimited Concurrency (Default):**
```bash
replicantx run tests/*.yaml --parallel
```

**Limited Concurrency:**
```bash
# Limit to 3 concurrent scenarios
replicantx run tests/*.yaml --parallel --max-concurrent 3

# Limit to 1 (effectively sequential but with parallel infrastructure)
replicantx run tests/*.yaml --parallel --max-concurrent 1
```

#### 📊 **Performance Benefits**

**Example: 10 scenarios, each taking 5 seconds**

| Mode | Duration | Speed Improvement |
|------|----------|-------------------|
| Sequential | ~50 seconds | 1x |
| Parallel (unlimited) | ~5 seconds | 10x |
| Parallel (max 3) | ~17 seconds | 3x |

#### ⚠️ **Considerations**

**API Rate Limits:**
- Use `--max-concurrent` to avoid overwhelming your API
- Monitor API response times during parallel execution
- Consider your API's rate limiting policies

**Resource Usage:**
- Parallel execution uses more memory and network connections
- Monitor system resources during large parallel test runs

**Test Dependencies:**
- Tests that depend on execution order should use `parallel: false`
- Consider using sequential execution for tests that modify shared state

**Debugging:**
- Parallel execution may make debugging more complex
- Use `--verbose` to see detailed output from all scenarios
- Consider running problematic tests sequentially for debugging

### Authentication Providers

#### Supabase
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

[project]
name = "replicantx"
version = "0.1.5"
version = "0.1.6"
description = "End-to-end testing harness for AI agents via web service API"
readme = "README.md"
requires-python = ">=3.11"
Expand Down
Loading
Loading