Adapt hybrid data pipeline #72
Description
Currently, the pipeline is driven by async events. This is great for latency and CPU usage in idle/low workload cases. But when the pipeline is handling high throughput, the event synchronization overhead becomes significant, taking 10-15% of the CPU.
Luckily, Linux 6.1+ has done something similar in NAPI: Kernel Polling with Interrupt Suppression in Linux 6.1.
More details: https://www.linkedin.com/pulse/napi-new-api-linux-kernel-networking-david-zhu-wcz6c/
Instead of being pure event-driven, the loop can poll until the budget is hit.