Any plans for Async support in sampling?

[guox18]

Hi there!

I’ve been doing quite a bit of data synthesis lately, and I just wanted to say the design pattern of data designer is super elegant! Honestly, it’s way better than the hard-coded scripts I usually write. I’m planning to use it to build out my future data pipelines.

I’ve been digging into the source code to learn more, and I noticed that the sampling is currently handled via multi-threading. It surprised me a little because I usually go the async route when I need to hit high concurrency (like 1,000+ requests).

To be fair, I haven't done a side-by-side benchmark for this specific use case, so I might be overthinking it! If there’s a specific reason why threading was chosen over async, I’d love to learn more about it.

Also, if it makes sense for the project, are there any plans to add async support in the future? I think it could be a great performance boost for an already awesome tool.

Thanks for the great work!

[johnnygreco]

Hi @guox18!

Thank you very much for your kind words about Data Designer!

Also, if it makes sense for the project, are there any plans to add async support in the future? I think it could be a great performance boost for an already awesome tool.

This is a really good question, and the short answer is most definitely YES! We are actively thinking about and will soon start building an async backend implementation.

There are historical reasons why the current implementation uses threads instead of async. It turns out you can push threads surprisingly far, but we definitely agree that there are significant performance gains to be found via an async approach.

cc @nabinchha and @eric-tramel who have worked the most in this area and will be driving the development of the future async engine. Please feel free to chime in guys!

[guox18]

Thanks! Happy to see DataDesigner improve!

[johnnygreco]

@guox18 – Thank YOU for your engagement 🙌

Please don't hesitate to reach out again if you have any questions / thoughts / ides!