Skip to content

Update RayDP to be compatible with Ray 2.5.0 #354

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kira-lin wants to merge 7 commits into
base: master
Choose a base branch
from
+13 −5
Open

Update RayDP to be compatible with Ray 2.5.0 #354

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
326d359
fix bug
Jun 21, 2023
69e4781
use ray 2.5.0 in CI
Jun 21, 2023
d24cc9b
fix lint
Jun 21, 2023
491963f
fix lint
Jun 21, 2023
a3d3ba9
skip horovod test
Jun 25, 2023
fbe6497
test ray 2.4.0
Jun 26, 2023
703789a
revert horovod test
Jun 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 13 additions & 5 deletions python/raydp/spark/dataset.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -259,14 +259,22 @@ def get_locations(blocks):
]

def ray_dataset_to_spark_dataframe(spark: sql.SparkSession,
arrow_schema,
dataset_schema,
blocks: List[ObjectRef],
locations = None) -> DataFrame:
locations = get_locations(blocks)
if not isinstance(arrow_schema, pa.lib.Schema):
if hasattr(arrow_schema, "base_schema") and \
not isinstance(arrow_schema.base_schema, pa.lib.Schema):
raise RuntimeError(f"Schema is {type(arrow_schema)}, required pyarrow.lib.Schema. \n" \
arrow_schema = dataset_schema
if not isinstance(dataset_schema, pa.lib.Schema):
if hasattr(dataset_schema, "base_schema"):
if isinstance(dataset_schema.base_schema, pa.lib.Schema):
arrow_schema = dataset_schema.base_schema
else:
raise RuntimeError(f"Schema is {type(dataset_schema.base_schema)}, " \
f"required pyarrow.lib.Schema. \n" \
f"to_spark does not support converting non-arrow ray datasets.")
else:
raise RuntimeError(f"Schema is {type(dataset_schema)}, " \
f"required pyarrow.lib.Schema. \n" \
f"to_spark does not support converting non-arrow ray datasets.")
schema = StructType()
for field in arrow_schema:
Expand Down