2

I already have some older data stored in parquet with a schema represented by

case class A(name: String)

I'd like to add a new non-mandatory field in

case class B(name: String, age: Option[Int])

and read in both the old and new data to the same DataFrame. Each time I'm trying to read the data with spark.read.parquet("test_path").as[B].collect(), I'm getting the following exception:

Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve '`age`' given input columns: [name];

Is there a way to specify a backward compatible schema for all of my data?

1 Answer 1

3

In order to read older data with a backward compatible schema, it's not enough to specify the new Encoder, you have to manually specify a StructType for the DataSet, and do not let Spark infer it based on either the . This way there isn't going to have missing fields during the conversion into a DataFrame:

spark.read.schema(Encoders.product[B].schema).parquet("test").as[B].collect()

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.