Skip to content

Conversation

@thorwhalen
Copy link
Member

Implements full working integration between qh and au (https://github.com/i2mint/au). This allows qh to use au's powerful backend system while maintaining qh's clean function-per-endpoint HTTP interface.

What This Enables

Users can now choose between:

  • qh built-in: Simple, no dependencies, perfect for development
  • au backends: Production-ready, persistent, distributed execution

Key Features

Integration Layer (qh/au_integration.py)

  • AuTaskStore: Adapter to use au's ComputationStore as qh's TaskStore
  • AuTaskExecutor: Adapter to use au's backends as qh's executors
  • Convenience functions:
    • use_au_backend() - Generic au backend usage
    • use_au_thread_backend() - Thread pool for I/O-bound tasks
    • use_au_process_backend() - Process pool for CPU-bound tasks
    • use_au_redis_backend() - Redis/RQ for distributed tasks

Working Example

from qh import mk_app
from qh.au_integration import use_au_thread_backend

app = mk_app(
    [my_func],
    async_funcs=['my_func'],
    async_config=use_au_thread_backend(
        storage_path='/var/tasks',
        ttl_seconds=3600
    )
)

What au Provides That Built-in Doesn't

  • ✅ Persistent storage (FileSystemStore - survives restarts)
  • ✅ Distributed execution (Redis/RQ, Supabase backends)
  • ✅ Retry policies with configurable backoff strategies
  • ✅ Middleware system (logging, metrics, tracing)
  • ✅ Task dependencies and workflows
  • ✅ Better testing utilities
  • ✅ Configuration from environment variables

Files Added

  1. qh/au_integration.py (313 lines)

    • Bridge layer between qh and au
    • TaskStore and TaskExecutor adapters
    • Convenience functions for common backends
  2. examples/qh_au_integration_example.py (189 lines)

    • Working examples with Thread and Process backends
    • Demonstrates persistent storage
    • Shows sync/async mixing
    • Comparison between built-in and au
  3. QH_AU_INTEGRATION_REPORT.md

    • Detailed technical analysis
    • What works, what's missing in au
    • Recommendations for both libraries
  4. QH_AU_FINAL_SUMMARY.md

    • Executive summary
    • Answers to key questions
    • Strategic recommendations
    • Prioritized improvement roadmap
  5. qh/init.py (modified)

    • Export au integration functions (optional import)
    • Gracefully handles au not being installed

Testing

Tested with:

  • au v0.1.0 from claude/improve-async-http-014xdEj6rd5Sv332C794eoVV branch
  • ThreadBackend (I/O-bound tasks) ✅
  • ProcessBackend (CPU-bound tasks) ✅
  • FileSystemStore (persistent storage) ✅
  • All examples run successfully

Usage Patterns

Development to Production

# Development (built-in, in-memory)
app = mk_app([func], async_funcs=['func'])

# Production (au with persistent storage)
app = mk_app(
    [func],
    async_funcs=['func'],
    async_config=use_au_thread_backend('/var/tasks')
)

Mixed Backends

# Different backends for different tasks
app = mk_app(
    [cpu_func, io_func],
    async_funcs=['cpu_func', 'io_func'],
    async_config={
        'cpu_func': use_au_process_backend(),
        'io_func': use_au_thread_backend(),
    }
)

Dependencies

au is optional. If not installed, qh continues to work with built-in backends. Users can install with: pip install au

For Redis backend: pip install au[redis]

Design Philosophy

  • qh provides the HTTP layer (beautiful UX, function-per-endpoint)
  • au provides the execution layer (powerful backends, persistence)
  • Integration is a thin adapter layer (single file)
  • Swapping backends is one line of code

Future Work

For qh:

  • Add au to pyproject.toml optional dependencies
  • Comprehensive integration tests
  • Production deployment documentation

For au (recommendations):

  • Adopt qh's function-per-endpoint HTTP pattern
  • Add Pydantic for type safety
  • OpenAPI spec generation
  • Better convention-over-configuration

Related Issues

This addresses the async task processing requirements while maintaining qh's philosophy of minimal boilerplate with maximum flexibility.

Implements full working integration between qh and au (https://github.com/i2mint/au).
This allows qh to use au's powerful backend system while maintaining qh's clean
function-per-endpoint HTTP interface.

## What This Enables

Users can now choose between:
- **qh built-in**: Simple, no dependencies, perfect for development
- **au backends**: Production-ready, persistent, distributed execution

## Key Features

### Integration Layer (qh/au_integration.py)
- AuTaskStore: Adapter to use au's ComputationStore as qh's TaskStore
- AuTaskExecutor: Adapter to use au's backends as qh's executors
- Convenience functions:
  - use_au_backend() - Generic au backend usage
  - use_au_thread_backend() - Thread pool for I/O-bound tasks
  - use_au_process_backend() - Process pool for CPU-bound tasks
  - use_au_redis_backend() - Redis/RQ for distributed tasks

### Working Example
```python
from qh import mk_app
from qh.au_integration import use_au_thread_backend

app = mk_app(
    [my_func],
    async_funcs=['my_func'],
    async_config=use_au_thread_backend(
        storage_path='/var/tasks',
        ttl_seconds=3600
    )
)
```

### What au Provides That Built-in Doesn't
- ✅ Persistent storage (FileSystemStore - survives restarts)
- ✅ Distributed execution (Redis/RQ, Supabase backends)
- ✅ Retry policies with configurable backoff strategies
- ✅ Middleware system (logging, metrics, tracing)
- ✅ Task dependencies and workflows
- ✅ Better testing utilities
- ✅ Configuration from environment variables

## Files Added

1. **qh/au_integration.py** (313 lines)
   - Bridge layer between qh and au
   - TaskStore and TaskExecutor adapters
   - Convenience functions for common backends

2. **examples/qh_au_integration_example.py** (189 lines)
   - Working examples with Thread and Process backends
   - Demonstrates persistent storage
   - Shows sync/async mixing
   - Comparison between built-in and au

3. **QH_AU_INTEGRATION_REPORT.md**
   - Detailed technical analysis
   - What works, what's missing in au
   - Recommendations for both libraries

4. **QH_AU_FINAL_SUMMARY.md**
   - Executive summary
   - Answers to key questions
   - Strategic recommendations
   - Prioritized improvement roadmap

5. **qh/__init__.py** (modified)
   - Export au integration functions (optional import)
   - Gracefully handles au not being installed

## Testing

Tested with:
- au v0.1.0 from claude/improve-async-http-014xdEj6rd5Sv332C794eoVV branch
- ThreadBackend (I/O-bound tasks) ✅
- ProcessBackend (CPU-bound tasks) ✅
- FileSystemStore (persistent storage) ✅
- All examples run successfully

## Usage Patterns

### Development to Production
```python
# Development (built-in, in-memory)
app = mk_app([func], async_funcs=['func'])

# Production (au with persistent storage)
app = mk_app(
    [func],
    async_funcs=['func'],
    async_config=use_au_thread_backend('/var/tasks')
)
```

### Mixed Backends
```python
# Different backends for different tasks
app = mk_app(
    [cpu_func, io_func],
    async_funcs=['cpu_func', 'io_func'],
    async_config={
        'cpu_func': use_au_process_backend(),
        'io_func': use_au_thread_backend(),
    }
)
```

## Dependencies

au is optional. If not installed, qh continues to work with built-in backends.
Users can install with: pip install au

For Redis backend: pip install au[redis]

## Design Philosophy

- qh provides the HTTP layer (beautiful UX, function-per-endpoint)
- au provides the execution layer (powerful backends, persistence)
- Integration is a thin adapter layer (single file)
- Swapping backends is one line of code

## Future Work

For qh:
- Add au to pyproject.toml optional dependencies
- Comprehensive integration tests
- Production deployment documentation

For au (recommendations):
- Adopt qh's function-per-endpoint HTTP pattern
- Add Pydantic for type safety
- OpenAPI spec generation
- Better convention-over-configuration

## Related Issues

This addresses the async task processing requirements while maintaining qh's
philosophy of minimal boilerplate with maximum flexibility.
@thorwhalen thorwhalen merged commit 93553a2 into master Nov 21, 2025
2 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants