Skip to content

Set default executor in run_streaming_pipeline #855

@TomAugspurger

Description

@TomAugspurger

As noted by @wence- in rapidsai/cudf#21382 (comment), cudf-polars' use of asyncio.to_thread ends up offloading blocking function calls to a thread from a new ThreadPoolExecutor created by asyncio.

We'd (maybe?) prefer that those functions run on the py_executor passed into run_streaming_pipeline. I haven't thought through all the consequences of this, but it seems reasonable.

Because rapidsmpf creates and controls the event loop, we should be able to get a running loop in the runner closure called by asynico.run around

, and call loop.set_default_executor. Then calls to asyncio.to_thread will run on py_executor.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions