-
Notifications
You must be signed in to change notification settings - Fork 29
Open
Description
As noted by @wence- in rapidsai/cudf#21382 (comment), cudf-polars' use of asyncio.to_thread ends up offloading blocking function calls to a thread from a new ThreadPoolExecutor created by asyncio.
We'd (maybe?) prefer that those functions run on the py_executor passed into run_streaming_pipeline. I haven't thought through all the consequences of this, but it seems reasonable.
Because rapidsmpf creates and controls the event loop, we should be able to get a running loop in the runner closure called by asynico.run around
| async def runner(): |
loop.set_default_executor. Then calls to asyncio.to_thread will run on py_executor.Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels