2

I have an external partitioned Hive table with underling file ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' Reading data via Hive directly is just fine, but when using Spark's Dataframe API the delimiter '|' is not taken into consideration.

Create external partitioned table:

hive> create external table external_delimited_table(value1 string, value2 string)
partitioned by (year string, month string, day string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'
location '/client/edb/poc_database/external_delimited_table';

Create data file containg just one row and place it to external partitioned table location:

shell>echo "one|two" >> table_data.csv
shell>hadoop fs -mkdir -p /client/edb/poc_database/external_delimited_table/year=2016/month=08/day=20
shell>hadoop fs -copyFromLocal table_data.csv /client/edb/poc_database/external_delimited_table/year=2016/month=08/day=20

Make partition active:

hive> alter table external_delimited_table add partition (year='2016',month='08',day='20');

Sanity check:

hive> select * from external_delimited_table;
select * from external_delimited_table;
+----------------------------------+----------------------------------+--------------------------------+---------------------------------+-------------------------------+--+
| external_delimited_table.value1  | external_delimited_table.value2  | external_delimited_table.year  | external_delimited_table.month  | external_delimited_table.day  |
+----------------------------------+----------------------------------+--------------------------------+---------------------------------+-------------------------------+--+
| one                              | two                              | 2016                           | 08                              | 20 

Spark code:

import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.{SparkContext, SparkConf}
object TestHiveContext {

  def main(args: Array[String]): Unit = {

    val conf = new SparkConf().setAppName("Test Hive Context")

    val spark = new SparkContext(conf)
    val hiveContext  = new HiveContext(spark)

    val dataFrame: DataFrame = hiveContext.sql("SELECT * FROM external_delimited_table")
    dataFrame.show()

    spark.stop()
  }

dataFrame.show() output:

+-------+------+----+-----+---+
| value1|value2|year|month|day|
+-------+------+----+-----+---+
|one|two|  null|2016|   08| 20|
+-------+------+----+-----+---+

1 Answer 1

2

This turned out to be a problem with Spark version 1.5.0. In version 1.6.0 issue doesn't take place:

scala> sqlContext.sql("select * from external_delimited_table")
res2: org.apache.spark.sql.DataFrame = [value1: string, value2: string, year: string, month: string, day: string]

scala> res2.show
+------+------+----+-----+---+
|value1|value2|year|month|day|
+------+------+----+-----+---+
|   one|   two|2016|   08| 20|
+------+------+----+-----+---+

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.