GA4GH WES API Service

A production-ready implementation of the GA4GH Workflow Execution Service (WES) API v1.1.0 specification using FastAPI and Python 3.12.

Overview

GA4GH WES is designed with a clean separation: the API service logs workflow requests to a database, while a separate daemon monitors requests and handles execution on the target platform. This implementation provides a robust, scalable foundation for workflow execution services.

Features

✅ Complete GA4GH WES v1.1.0 API - All 8 endpoints implemented
✅ FastAPI - Modern, fast, async web framework
✅ SQLAlchemy - Async database ORM with MySQL support
✅ Flexible Storage - Local filesystem or AWS S3
✅ Authentication - HTTP Basic Auth with OAuth2 hooks
✅ Database Migrations - Alembic for version control
✅ Workflow Monitoring - Separate daemon for execution
✅ OpenAPI Documentation - Auto-generated API docs
✅ Production Ready - Error handling, logging, CORS

Architecture

┌─────────────┐      REST API      ┌──────────────┐
│   Client    │ ───────────────────>│  FastAPI     │
│             │                     │  Service     │
└─────────────┘                     └──────┬───────┘
                                           │
                                           ↓
                                    ┌──────────────┐
                                    │   MySQL      │
                                    │   Database   │
                                    └──────┬───────┘
                                           ↑
                                           │
                                    ┌──────────────┐
                                    │   Workflow   │
                                    │   Daemon     │
                                    └──────┬───────┘
                                           │
                                    ┌──────────────┐
                                    │   Storage    │
                                    │  (Local/S3)  │
                                    └──────────────┘

Quick Start

Prerequisites

Python 3.12+
MySQL 8.0+
uv package manager (recommended) or pip

Installation

Clone the repository

git clone <repository-url>
cd GA4GH-WES-API-Service

Install uv (if not already installed)

curl -LsSf https://astral.sh/uv/install.sh | sh

Install dependencies

uv sync

Set up environment variables

cp .env.example .env
# Edit .env with your configuration

Configure database

# Create MySQL database
mysql -u root -p -e "CREATE DATABASE wes_db CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;"
mysql -u root -p -e "CREATE USER 'wes_user'@'localhost' IDENTIFIED BY 'your_password';"
mysql -u root -p -e "GRANT ALL PRIVILEGES ON wes_db.* TO 'wes_user'@'localhost';"

# Update SQLALCHEMY_DATABASE_URI in .env
# SQLALCHEMY_DATABASE_URI=mysql+aiomysql://wes_user:your_password@localhost:3306/wes_db

Run database migrations

uv run alembic upgrade head

Start the API service

uv run python -m src.wes_service.main

Start the workflow daemon (in a separate terminal)

uv run python -m src.wes_service.daemon.workflow_monitor

The API will be available at http://localhost:8000

Configuration

All configuration is managed through environment variables. Copy .env.example to .env and customize:

Database Configuration

SQLALCHEMY_DATABASE_URI=mysql+aiomysql://wes_user:wes_password@localhost:3306/wes_db

Storage Configuration

# Local storage
STORAGE_BACKEND=local
LOCAL_STORAGE_PATH=/var/wes/storage

# OR S3 storage
STORAGE_BACKEND=s3
S3_BUCKET_NAME=wes-workflows
S3_REGION=us-east-1
S3_ACCESS_KEY_ID=your_access_key
S3_SECRET_ACCESS_KEY=your_secret_key

Authentication Configuration

AUTH_METHOD=basic  # or 'oauth2' or 'none'

# For basic auth, generate password hash:
# python -c "from passlib.context import CryptContext; print(CryptContext(schemes=['bcrypt']).hash('your_password'))"
BASIC_AUTH_USERS=admin:$2b$12$hashedpassword,user2:$2b$12$hashedpassword

Service Configuration

SERVICE_NAME=GA4GH WES Service
SERVICE_ORGANIZATION_NAME=Your Organization
SERVICE_ENVIRONMENT=production
API_PREFIX=/ga4gh/wes/v1
PORT=8000

API Endpoints

Service Information

GET /ga4gh/wes/v1/service-info - Get service metadata

Workflow Runs

GET /ga4gh/wes/v1/runs - List workflow runs
POST /ga4gh/wes/v1/runs - Submit new workflow
GET /ga4gh/wes/v1/runs/{run_id} - Get run details
GET /ga4gh/wes/v1/runs/{run_id}/status - Get run status
POST /ga4gh/wes/v1/runs/{run_id}/cancel - Cancel run

Tasks

GET /ga4gh/wes/v1/runs/{run_id}/tasks - List tasks
GET /ga4gh/wes/v1/runs/{run_id}/tasks/{task_id} - Get task details

Additional Endpoints

GET /healthcheck - Health check
GET /ga4gh/wes/v1/docs - Swagger UI documentation
GET /ga4gh/wes/v1/redoc - ReDoc documentation

Usage Examples

Submit a Workflow

curl -X POST "http://localhost:8000/ga4gh/wes/v1/runs" \
  -u admin:password \
  -F "workflow_type=CWL" \
  -F "workflow_type_version=v1.0" \
  -F "workflow_url=https://example.com/workflow.cwl" \
  -F "workflow_params={\"input_file\": \"s3://bucket/input.txt\"}" \
  -F "tags={\"project\": \"test\"}"

List Runs

curl -X GET "http://localhost:8000/ga4gh/wes/v1/runs?page_size=10" \
  -u admin:password

Get Run Status

curl -X GET "http://localhost:8000/ga4gh/wes/v1/runs/{run_id}/status" \
  -u admin:password

Cancel a Run

curl -X POST "http://localhost:8000/ga4gh/wes/v1/runs/{run_id}/cancel" \
  -u admin:password

Development

Running Tests

# Install dev dependencies
uv sync --dev

# Run tests
uv run pytest

# Run with coverage
uv run pytest --cov=src --cov-report=html

Code Quality

# Format code
uv run ruff format src tests

# Lint code
uv run ruff check src tests

# Type checking
uv run mypy src

Database Migrations

# Create a new migration
uv run alembic revision --autogenerate -m "description"

# Apply migrations
uv run alembic upgrade head

# Rollback one migration
uv run alembic downgrade -1

# View migration history
uv run alembic history

Production Deployment

Using Docker (Recommended)

# Build image
docker build -t wes-service .

# Run with docker-compose
docker-compose up -d

Using systemd

Create service file /etc/systemd/system/wes-api.service:

[Unit]
Description=GA4GH WES API Service
After=network.target mysql.service

[Service]
Type=simple
User=wes
WorkingDirectory=/opt/wes-service
Environment="PATH=/opt/wes-service/.venv/bin"
ExecStart=/opt/wes-service/.venv/bin/python -m src.wes_service.main
Restart=always

[Install]
WantedBy=multi-user.target

Create daemon service file /etc/systemd/system/wes-daemon.service:

[Unit]
Description=GA4GH WES Workflow Daemon
After=network.target mysql.service

[Service]
Type=simple
User=wes
WorkingDirectory=/opt/wes-service
Environment="PATH=/opt/wes-service/.venv/bin"
ExecStart=/opt/wes-service/.venv/bin/python -m src.wes_service.daemon.workflow_monitor
Restart=always

[Install]
WantedBy=multi-user.target

Enable and start services:

sudo systemctl enable wes-api wes-daemon
sudo systemctl start wes-api wes-daemon

Using Nginx as Reverse Proxy

server {
    listen 80;
    server_name wes.example.com;

    location / {
        proxy_pass http://localhost:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Project Structure

GA4GH-WES-API-Service/
├── src/
│   └── wes_service/
│       ├── __init__.py
│       ├── main.py                    # FastAPI app factory
│       ├── config.py                  # Configuration management
│       ├── api/
│       │   ├── deps.py               # Dependency injection
│       │   ├── routes/               # API endpoints
│       │   │   ├── service_info.py
│       │   │   ├── runs.py
│       │   │   └── tasks.py
│       │   └── middleware/           # Error handlers
│       ├── core/
│       │   ├── security.py           # Authentication
│       │   └── storage.py            # Storage backends
│       ├── db/
│       │   ├── base.py               # SQLAlchemy base
│       │   ├── models.py             # Database models
│       │   └── session.py            # DB session
│       ├── schemas/                  # Pydantic models
│       ├── services/                 # Business logic
│       └── daemon/                   # Workflow daemon
├── alembic/                          # Database migrations
├── tests/                            # Test suite
├── pyproject.toml                    # Project dependencies
├── .env.example                      # Environment template
└── README.md                         # This file

Workflow Engines

The current implementation includes:

A stub local executor for demonstration
AWS Omics integration for running workflows on AWS HealthOmics

To integrate additional workflow engines:

Create a new executor in src/wes_service/daemon/executors/
Implement the WorkflowExecutor interface
Configure the daemon to use your executor

Example executors:

LocalExecutor - Stub implementation for demonstration
OmicsExecutor - For AWS HealthOmics workflows
CWLToolExecutor - For Common Workflow Language (to be implemented)
CromwellExecutor - For WDL workflows (to be implemented)
NextflowExecutor - For Nextflow pipelines (to be implemented)

For AWS Omics usage instructions, see docs/aws_omics_usage.md.

Supported Workflow Types

✅ CWL (Common Workflow Language) - v1.0, v1.1, v1.2
✅ WDL (Workflow Description Language) - 1.0, draft-2
🔄 Custom types can be added via configuration

Troubleshooting

Database Connection Errors

# Check MySQL is running
systemctl status mysql

# Test connection
mysql -u wes_user -p -h localhost wes_db

Storage Permission Errors

# For local storage, ensure directory is writable
chmod 755 /var/wes/storage
chown -R wes:wes /var/wes/storage

Authentication Issues

# Generate new password hash
python -c "from passlib.context import CryptContext; print(CryptContext(schemes=['bcrypt']).hash('newpassword'))"

Contributing

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Authors

Ryan Golhar ryan.golhar@bms.com
Maggie Chen yu.chen2@bms.com
Claude 4.5 Sonnet (AI Assistant)

Acknowledgments

GA4GH for the WES specification
FastAPI framework
SQLAlchemy ORM

Support

For issues and questions:

GitHub Issues: [repository-url]/issues
Documentation: [repository-url]/wiki
GA4GH WES Spec: https://github.com/ga4gh/workflow-execution-service-schemas

Name		Name	Last commit message	Last commit date
Latest commit History 254 Commits
.github		.github
alembic		alembic
docs		docs
examples		examples
scripts		scripts
src/wes_service		src/wes_service
tests		tests
wes-frontend		wes-frontend
.env.example		.env.example
.flake8		.flake8
.gitignore		.gitignore
.pylintrc		.pylintrc
Makefile		Makefile
Procfile		Procfile
README.md		README.md
alembic.ini		alembic.ini
pyproject.toml		pyproject.toml
uv.lock		uv.lock
workflow_execution_service.openapi.yaml		workflow_execution_service.openapi.yaml

NGS360/GA4GH-WES-API-Service

Folders and files

Latest commit

History

Repository files navigation

GA4GH WES API Service

Overview

Features

Architecture

Quick Start

Prerequisites

Installation

Configuration

Database Configuration

Storage Configuration

Authentication Configuration

Service Configuration

API Endpoints

Service Information

Workflow Runs

Tasks

Additional Endpoints

Usage Examples

Submit a Workflow

List Runs

Get Run Status

Cancel a Run

Development

Running Tests

Code Quality

Database Migrations

Production Deployment

Using Docker (Recommended)

Using systemd

Using Nginx as Reverse Proxy

Project Structure

Workflow Engines

Supported Workflow Types

Troubleshooting

Database Connection Errors

Storage Permission Errors

Authentication Issues

Contributing

License

Authors

Acknowledgments

Support

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors 3

Uh oh!

Languages

Packages