0

I have a few dataframes that I want to take portions of and then combine all the portions together into a dataframe..

I have run limit on each portion after the manipulations I get in all cases an output of

org.apache.spark.sql.Dataset[org.apache.spark.sql.Row]

How can I take all of these sql.rows and combine them into a single dataframe?

4
  • If the columns are same in all of them, you can use union. If all the dataframes are different, try to select one unique key with all of them so that you can join them later. Apr 11, 2018 at 10:40
  • can I just reference the columns from the original dataframes using withcolumn? DF.withColumn("A",DF1("A")) . That seems like the best way.. but I can't seem to get that to work.
    – Brian
    Apr 11, 2018 at 10:57
  • Nope, it doesn't work like that. You'll probably need the usual SQL joins. add some sample data and expected output and you're likely to get help more quickly. Also, read how to create a minimal reproducible example Apr 11, 2018 at 12:02
  • I'm used to other dataframe type API's Spark API doesn't work the same way. Thanks for the help.
    – Brian
    Apr 11, 2018 at 17:04

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.