-
Notifications
You must be signed in to change notification settings - Fork 1
Add admission control hook for worker backpressure #57
Description
Description
Add a user-configurable admission control hook to workers that determines whether an incoming task should be accepted or rejected. The hook is a callable that receives context about the worker's current state (e.g., active task count, task metadata) and returns an accept/reject decision. When a task is rejected, the worker responds with gRPC RESOURCE_EXHAUSTED, which the load balancer already classifies as transient — causing it to skip to the next worker automatically.
The default behavior (no hook configured) accepts all tasks, preserving the current semantics. The existing WorkerConnection semaphore continues to cap per-worker concurrent dispatches on the client side; this hook adds server-side admission control.
Motivation
Workers currently have no way to signal that they are overloaded. The client-side semaphore limits concurrency per connection, but the worker itself accepts all incoming tasks unconditionally. In production, workers may need to reject tasks based on workload-specific criteria — active task count, memory pressure, GPU utilization, or custom application logic.
Backpressure is inherently workload-specific, so a hook-based approach lets users define the rejection predicate without Wool prescribing what "overloaded" means. The gRPC RESOURCE_EXHAUSTED → transient error → skip-to-next-worker path is already fully implemented in the load balancer, so the only missing piece is letting the worker trigger it.
Expected outcome
LocalWorkeraccepts an optionalbackpressureparameter — a callable that receives worker state and returns whether to reject the task.- A rejected task triggers gRPC
RESOURCE_EXHAUSTED, causing the load balancer to skip to the next available worker. - The hook is invoked after task deserialization but before acknowledgment, so rejected tasks never enter the worker's active task set.
- Both sync and async hook implementations are supported.
- When no hook is configured, all tasks are accepted (backward compatible).