2

I have csv file which I am converting to parquet files using databricks library in scala. I am using below code:

val spark = SparkSession.builder().master("local[*]").config("spark.sql.warehouse.dir", "local").getOrCreate()
var csvdf = spark.read.format("org.apache.spark.csv").option("header", true).csv(csvfile)
csvdf.write.parquet(csvfile + "parquet")

Now the above code works fine if I don't have space in my column headers. But if any csv file have spaces in the column headers, it doesn't work and errors out stating invalid column headers. My csv files are delimited by ,.

Also, I cannot change the spaces of column names of the csv. The column names has to be as they are even if they contain spaces as those are given by end user.

Any idea on how to fix this?

2
  • 2
    the parquet file format does not allow for spaces in column names; contains invalid character(s) among " ,;{}()\n\t=". ORC also does not allow for spaces in column names Aug 1, 2018 at 16:13
  • @JamesTobin can you provide this as a resolution to the OP? I think your comment clears it out. Thanks.
    – CodeHunter
    Aug 6, 2018 at 15:03

2 Answers 2

3

per @CodeHunter's request

sadly, the parquet file format does not allow for spaces in column names;
the error that it'll spit out when you try is: contains invalid character(s) among " ,;{}()\n\t=".

ORC also does not allow for spaces in column names :(

Most sql-engines don't support column names with spaces, so you'll probably be best off converting your columns to your preference of foo_bar or fooBar or something along those lines

2

I would rename the offending columns in the dataframe, to change space to underscore, before saving. Could be with select "foo bar" as "foo_bar" or .withColumnRenamed("foo bar", "foo_bar")

2
  • you mean before writing to parquet, right? I thought of that as a solution but I was thinking if there is something that can directly take care of this scenario.
    – CodeHunter
    Aug 1, 2018 at 16:03
  • Yes, I mean before writing to parquet. There may also be some options on quoting / escaping in the csv read that would work, but I'm a) less confident about them and b) tend to prefer names without spaces in my own code, so am projecting!
    – Dan
    Aug 1, 2018 at 16:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.