Distributed hive-style join testing with superset satisfaction (#256)

* Added join testdata and method to generate parquet from csv.

* Added basic single node join test for comparison.

* Use hive partitioning.

* Results are the same with hive partitioning and Gene's PR.

* Fixed configs and achieved optimal distributed plan.

* Refactoring, adding comments.

* Added check to ensure optimal plan is achieved.

* Update based on Nga's comments.

* Added second test.

* Added third test.

* Refactor.

* Add ORDER BY to queries and switch to snapshot testing.

* Nulls last instead of nulls first.

* Fix tests.

Author: JSOD11

Cargo.lock

@@ -0,0 +1,3 @@
+env,service,host
+dev,log,host-y
+

testdata/join/csv/dim/d_dkey=A/data0.csv

@@ -0,0 +1,3 @@
+env,service,host
+prod,log,host-x
+

testdata/join/csv/dim/d_dkey=B/data0.csv

@@ -0,0 +1,3 @@
+env,service,host
+dev,trace,host-z
+

testdata/join/csv/dim/d_dkey=C/data0.csv

@@ -0,0 +1,3 @@
+env,service,host
+prod,trace,host-x
+

testdata/join/csv/dim/d_dkey=D/data0.csv

@@ -0,0 +1,9 @@
+timestamp,value
+2023-01-01T09:00:00,95.5
+2023-01-01T09:00:10,102.3
+2023-01-01T09:00:20,98.7
+2023-01-01T09:12:20,105.1
+2023-01-01T09:12:30,100.0
+2023-01-01T09:12:40,150.0
+2023-01-01T09:12:50,120.8
+

testdata/join/csv/fact/f_dkey=A/data0.csv

@@ -0,0 +1,9 @@
+timestamp,value
+2023-01-01T09:00:00,75.2
+2023-01-01T09:00:10,82.4
+2023-01-01T09:00:20,78.9
+2023-01-01T09:00:30,85.6
+2023-01-01T09:12:30,80.0
+2023-01-01T09:12:40,120.0
+2023-01-01T09:12:50,92.3
+

testdata/join/csv/fact/f_dkey=B/data0.csv

@@ -0,0 +1,9 @@
+timestamp,value
+2023-01-01T10:00:00,310.5
+2023-01-01T10:00:10,225.7
+2023-01-01T10:00:20,380.2
+2023-01-01T10:00:30,205.8
+2023-01-01T10:00:40,350.0
+2023-01-01T10:12:40,200.0
+2023-01-01T10:12:50,205.4
+

testdata/join/csv/fact/f_dkey=C/data0.csv

@@ -0,0 +1,5 @@
+timestamp,value
+2023-01-01T10:00:00,24.8
+2023-01-01T10:00:10,72.1
+2023-01-01T10:00:20,42.5
+

testdata/join/csv/fact/f_dkey=D/data0.csv

@@ -0,0 +1,36 @@
+-- datafusion-cli -f testdata/join/generate_parquet_from_csv.sql
+
+-- Generate parquet dim files from csv files.
+COPY (SELECT * FROM "testdata/join/csv/dim/d_dkey=A/data0.csv")
+TO "testdata/join/parquet/dim/d_dkey=A/data0.parquet"
+STORED AS PARQUET;
+
+COPY (SELECT * FROM "testdata/join/csv/dim/d_dkey=B/data0.csv")
+TO "testdata/join/parquet/dim/d_dkey=B/data0.parquet"
+STORED AS PARQUET;
+
+COPY (SELECT * FROM "testdata/join/csv/dim/d_dkey=C/data0.csv")
+TO "testdata/join/parquet/dim/d_dkey=C/data0.parquet"
+STORED AS PARQUET;
+
+COPY (SELECT * FROM "testdata/join/csv/dim/d_dkey=D/data0.csv")
+TO "testdata/join/parquet/dim/d_dkey=D/data0.parquet"
+STORED AS PARQUET;
+
+-- Generate parquet fact files from csv files.
+COPY (SELECT * FROM "testdata/join/csv/fact/f_dkey=A/data0.csv")
+TO "testdata/join/parquet/fact/f_dkey=A/data0.parquet"
+STORED AS PARQUET;
+
+COPY (SELECT * FROM "testdata/join/csv/fact/f_dkey=B/data0.csv")
+TO "testdata/join/parquet/fact/f_dkey=B/data0.parquet"
+STORED AS PARQUET;
+
+COPY (SELECT * FROM "testdata/join/csv/fact/f_dkey=C/data0.csv")
+TO "testdata/join/parquet/fact/f_dkey=C/data0.parquet"
+STORED AS PARQUET;
+
+COPY (SELECT * FROM "testdata/join/csv/fact/f_dkey=D/data0.csv")
+TO "testdata/join/parquet/fact/f_dkey=D/data0.parquet"
+STORED AS PARQUET;
+