1

I want to import data from SQL Server and query it from hive.

I created a VirtualBox using cloudera template and also started reading its tutorial.

I am successfully able to import data from SQL Server using sqoop as avro files and then create table in hive and import data from avro file. Then query it from hive.

But import-all-tables command of sqoop only imports table of schema "dbo". What if I want to import tables with a schema dw also? I tried to use import command to import specific table exist in dw schema. but that also doesn't work.

Any idea how to import data from SQL Sever using sqoop for non dbo. schema related tables as avro? Or import data from SQL Server for other than dbo. schema and load it directly into hive?

1 Answer 1

2

Download JDBC driver and copy it to sqoop directory

$ curl -L 'http://download.microsoft.com/download/0/2/A/02AAE597-3865-456C-AE7F-613F99F850A8/sqljdbc_4.0.2206.100_enu.tar.gz' | tar xz
$ sudo cp sqljdbc_4.0/enu/sqljdbc4.jar /var/lib/sqoop/

Import table from Sql Server using sqoop

sqoop import --driver="com.microsoft.sqlserver.jdbc.SQLServerDriver" --connect="jdbc:sqlserver://sqlserver;database=databasename;username=username;password=passwordofuserprovidedinusername" --username=username --password= passwordofuserprovidedinusername --table="schemaname.tablename" --split-by=primarykeyoftable --compression-codec=snappy --as-avrodatafile --warehouse-dir=/user/hive/warehouse/tablename

Verify if table imported properly

hadoop fs -ls /user/hive/warehouse
ls -l *.avsc

create new directory and provide appropriate permissions

sudo -u hdfs hadoop fs -mkdir /user/examples
sudo -u hdfs hadoop fs -chmod +rw /user/examples
hadoop fs -copyFromLocal ~/*.avsc /user/examples

start hive

hive

import table schema and data from sqoop to hive warehouse

CREATE EXTERNAL TABLE tablename
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
LOCATION 'hdfs:///user/hive/warehouse/tablename’
TBLPROPERTIES ('avro.schema.url'='hdfs://quickstart.cloudera/user/examples/sqoop_import_schemaname_tablename.avsc');

Note: make sure while typing command the single quote may change if you are coping the command. There should not be any space in path or filenames.

1
  • in avro.schema.url, why there's a quickstart.cloudera prefix? Is it a typo?
    – David S.
    Feb 23, 2016 at 10:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.