Spark

[javihern98]

Overview on Spark

[mla2001]

Issues encountered while testing with Spark:

• Very significant increase in the execution cost (in AWS-Glue tests), the costs with the user's own infrastructure may be lower in the long term, but implicitly it is already a problem that the user must have the infrastructure to run it.
• Increased deployment complexity (need to create a Docker-Swarm-like installer, or any other similar technology) for launch and coordination across all nodes.
• The Pyspark pandas module (pyspark.pandas) does not have all the pandas functions and methods, and some of them are outdated (implying the addition of missing methods and updating of outdated ones).

[ghomem]

I am familiar with the first two points. I don't know anything about the second. I would like to add a third:

• we are not sure as of today if we can directly execute on a Spark cluster or if we have to prepare the cluster first, with the specific libraries we use

• if we need to update the cluster with the specific libraries we are asking for more operational trouble in VTL Suite scenarios: not only the VTL Suite needs to be updated but also the Spark cluster needs to be "refreshed" if the VTL Engine has changed. In the case where the cluster is owned by a different team this will be a synchronization challenge.