Modin

[javihern98]

Overview on Modin

[mla2001]

Installing Modin with Dask:

Replacing "pandas as pd" for "modin.pandas as pd" so its no needn't to change anything else:

And after executing the test-suite, not only some of the fails, but it takes too long, 48 mins for only 470 tests:

[ghomem]

I have several notes on this. One is that this summary should contain KPIs.

We currently have:

• some fails
• takes too long

Would be useful to have:

• % failures compared to Pandas
• % of time increase compared to Pandas

And given the insignificant effort of replacing pandas with Modin, at least in regards to code, we should review the KPIs and check if there is effort that can be done to make Modin work.

Bottom line comments:

1. the decision needs to consider the TCO: I understand that the main proposal with DuckDB is spending an initial 3-4 months * 1.5 HR and changing considerably the code
2. but how much effort would be necessary to make Modin work in order to not change the code at all? can it be solved by opening some issues in the project? by sponsoring some fixes?

The promise of Modin is very specific:

so if it does not work - even with reasonable investment - it is up to us to make it clear that is does not work.

Ref:

https://en.wikipedia.org/wiki/Total_cost_of_ownership

[javihern98]

Regarding Modin, the main problem is also that it does not support the blocking operators within the execution out-of-core:

• Analytic with window
• Join operators
• Ordering and Grouping

These methods (except the join operators) are implemented using duckdb as of today, due to the impossibility of implementing them with pandas. Using modin, we will need to rely on the backend engine we use (probably Dask, as Ray is more focused on ML). We could not find any way to implement in Dask all VTL operators.