2

I've been playing around with Spark, Hive and Parquet, I have some data in my Hive table and here is how it looks like ( warning french language ahead ) :

Bloqu� � l'arriv�e      NULL
Probl�me de connexion   Bloqu� en hub

Obviously there's something wrong here.

What I do is : I read a teradata table as a dataframe with spark, I store it as a parquet file and then I use this file to store it to hive, here's my create table script :

CREATE TABLE `table`(
   `lib` VARCHAR(255),
   `libelle_sous_cause` VARCHAR(255),
   )
 STORED AS PARQUET
 LOCATION
   'hdfs://location';

I don't really know what cause this, it might be caused by some special encoding between Teradata > parquet or Parquet > Hive, I'm not sure.

Any help will be appreciated, thanks.

2 Answers 2

1

I figured that out, the solution was to simply use STRING instead of VARCHAR

CREATE TABLE `table`(
   `lib` STRING,
   `libelle_sous_cause` STRING,
   )
 STORED AS PARQUET
 LOCATION
   'hdfs://location';
0

I've run into the same problem when doing sqoop from Teradata to Hadoop. When extracting from Teradata, in the SELECT, please try wrapping your varchar columns, that may have this issue, with this line:

SELECT
NAME,
AGE,
TRIM(CAST(TRANSLATE(COLUMNOFINTEREST USING latin_to_unicode WITH ERROR) AS VARCHAR(50)))
FROM
TABLENAME;

COLUMNOFINTEREST is your column that would have special characters.

Let me know if that works.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.