source pyspark-csv from a data frame or a parquet table

[Tagar]

We have a lot of use cases for pyspark-csv where source files are parquet files..
Would it be possible for pyspark-csv to add a parquet table or an arbitrary data frame?

[Tagar]

We have source parquet tables with hundreds of string columns (some up to 800) and it would be great to use something like pyspark-csv to convert to a dataframe (or a parquet table) with correct data types.

[Tagar]

http://stackoverflow.com/questions/32722132/spark-data-type-guesser-udaf

[Siddhant]

I'm not sure I understand your problem.
You should be able to read parquet data directly into Spark, no?
What's the point of CSV here ?

[Tagar]

Let's say we have a dataframe where all columns are strings (because of how we got that data).
We'd like to run this dataframe through the same datatype inference code, to create another
dataframe with actual data types.
So my point is, data inference logic should be separated/abstracted from spark-csv logic so
we can run this data type inference also for a dataframes of all-string columns to come up
with a realt data types that should be used, based on actual data.