0

I am writing a Spark dataframe where one of the column is of Vector datatype as ORC. When I load back the dataframe the schema changes.

var df : DataFrame = spark.createDataFrame(Seq(
  (1.0, Vectors.dense(0.0, 1.1, 0.1)),
  (0.0, Vectors.dense(2.0, 1.0, -1.0)),
  (0.0, Vectors.dense(2.0, 1.3, 1.0)),
  (1.0, Vectors.dense(0.0, 1.2, -0.5))
)).toDF("label", "features")

df.printSchema

df.write.mode(SaveMode.Overwrite).orc("/some/path")
val newDF = spark.read.orc("/some/path")

newDF.printSchema

The output of df.printSchema is

|-- label: double (nullable = false)
|-- features: vector (nullable = true)

The output of newDF.printSchema is

|-- label: double (nullable = true)
|-- features: struct (nullable = true)
|    |-- type: byte (nullable = true)
|    |-- size: integer (nullable = true)
|    |-- indices: array (nullable = true)
|    |    |-- element: integer (containsNull = true)
|    |-- values: array (nullable = true)
|    |    |-- element: double (containsNull = true)

What is the issue here? I am using Spark 2.2.0 with Scala 2.11.8

2
  • The issue is that you wrote to "/some/path" and then you read from "/som/path" (you are missing an "e")... Just kidding! I believe that ORC does not support Vector types well. Avro just crashes with Vectors. Parquet will give you "good enough" results. Commented Mar 22, 2018 at 17:08
  • Yeah, We were initially using Parquet. But it had problems writing Dataframes with large number of columns. Thankfully this issue was fixed in Spark 2.3 Commented Aug 30, 2018 at 6:27

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.