Questions tagged [sqoop]
Sqoop is an open source connectivity framework that facilitates transfer between multiple Relational Database Management Systems (RDBMS) and HDFS. Sqoop uses MapReduce programs to import and export data; the imports and exports are performed in parallel.
2,580
questions
0
votes
0
answers
7
views
Sqoop exec job - : java.lang.ClassNotFoundException: com.mysql..jdbc.Driver
sqoop executive the job:sqoop job --skip-dist-cache --exec sqoop_full_bi_fact_autoloan_contract
The following error:
enter image description here
and sqoop libs like :
enter image description here
So, ...
0
votes
0
answers
21
views
external db connection from EMR (Sqoop or Spark) using proxy
external db connection from EMR (Sqoop or Spark) using proxy:
I am trying to connect MS SQL Server DB using Sqoop eval command but i am getting connecion refused error as it is external DB from m ...
0
votes
0
answers
14
views
Error during Export: Hive Reserved Words (floor) as column name when using sqoop export
How should I export hive to mysql with column named "floor"
sqoop export --columns floor
0
votes
0
answers
11
views
In sqoop import : splitz size
In sqoop import if the primary key id of mysql table has all values from 1 to 10000. An outlier 12000 came in. You are trying to import with a where clause mentioning id < 20000. What is the split ...
0
votes
1
answer
18
views
Sqoop - NoClassDefFoundError: org/apache/commons/lang/StringUtils
Am trying to connect sqoop (1.4.7) to Server version: 8.0.29-0 on ubuntu0.22.04.2 using
mysql-connector-java-8.0.12.jar on a hadoop-3.3.3 system.Have placed the .jar file in sqoop's lib folder but ...
0
votes
0
answers
6
views
changning the num-mappers for mutiple tables in single sqoop command
I have some tables in MySQL and I need to import all the tables to HDFS at a time using the single command for this I will use import-all-tables. But for 2 tables I need to get 3 mappers and for 2 ...
0
votes
0
answers
13
views
java.lang.NoSuchMethodError: org.apache.hadoop.ipc.RPC.getProxy while doing sqoop import
Getting below error :
Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.ipc.RPC.getProxy(Ljava/lang/Class;JLjava/net/InetSocketAddress;Lorg/apache/hadoop/security/UserGroupInformation;Lorg/...
0
votes
0
answers
11
views
Fetching less data woth sqoop
I am running a sqoop import in a database with a time range of 2 days and 4m rows and at the end it shows me that it fetched only 500k rows. Is there any way to increase the fetching volume in sqoop ? ...
0
votes
0
answers
5
views
Problem with dolphinscheduler configuration "workflow definition"-SQOOP"
My dolphinscheduler version is 2.0.5.
I found that when configuring the workflow definition - SQOOP, the options I marked in the picture below cannot be modified or the mouse click does not respond.
0
votes
0
answers
31
views
Sqoop import Oracle Number datatype
In Oracle, a table contains column defined as Number(without precision and scale), and actual values in this column could be integer or decimal.
When using sqoop in informatica to import oracle table ...
0
votes
0
answers
35
views
NullPointerException in Apache Kylin when using JDBC
I use Kylin from Docker: https://kylin.apache.org/docs/install/kylin_docker.html
I downloaded sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz to SQOOP_HOME directory.
When I am trying to follow example here: ...
0
votes
0
answers
59
views
sqoop import error HCat exited with status
I am running the following sqoop import from Teradata:
sqoop import --driver com.teradata.jdbc.TeraDriver \
--connect jdbc:teradata://telearg7/DATABASE=AR_PROD_HUB_DIM_VW,CHARSET=UTF8,...
0
votes
0
answers
12
views
Sqoop import command is not working in crontab but working manually from shell script
I have a shell script which has the below statements.
#!/bin/sh
Print Hello world message
echo "Hello, Good morning"
pwd=$(pwd)
echo "pwd is ${pwd}"
HIVEDB="rsdb"
...
0
votes
1
answer
21
views
Is there a way to exclude the "partitioned by" column from the hive?
I created the table by entering the command below.
create external table test (A STRING, C STRING, D FLOAT) partitioned by (B STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '...
0
votes
0
answers
47
views
The datetime columns in all the tables on SQL server have shifted by a few hours
I've created a few tables all of which had a column of datatype datetime. Recently, I think after the daylight savings adjustment, for some reason all the dates have shifted by a few hours i.e. '2022-...
0
votes
0
answers
29
views
Sqoop export having troubles when using a csv file with lots of rows (Cloudera-Quickstart-vm)
I have a data in a csv file with 10,000 rows and 21 columns. and i want to export the data in a table i have in rdbms. When i try running the a sqoop export the job fails. when i tried reducing the ...
0
votes
0
answers
24
views
Is it possible to import the table by date through Sqoop?
I am trying to import a huge amount of data from sql server.
There is a date column in this table, and I'm going to import the table by dividing it by date.
Is there any way?
Or is there a way to ...
0
votes
0
answers
19
views
Sqoop command not found when running through Oozie
When I am running Sqoop script in CLI, it is running fine without any issue. But when run it using Oozie, it failed with Sqoop command not found. It seems sqoop is not installed in other data nodes. ...
0
votes
0
answers
32
views
How to send data stored in biguqery to downstream oracle databse?
I have a requirement where, I would like to send data from my BigQuery to oracle db?
do we have any similar functionality like hadoop sqoop export in GCP, to send data from BigQuery tables to oracle ...
0
votes
0
answers
20
views
error to load main class for sqoop after successful instalation
OS: Windows 10
Sqoop Version: 1.4.7
Hadoop Version: 2.9.2
Hi after installing Sqoop on win 10 following the instructions from
https://medium.com/analytics-vidhya/sqoop-how-to-install-in-5-steps-in-...
0
votes
0
answers
17
views
Why can't I find the file when importing from sqoop?
I try to import mysql table into hive through sqoop.
I used the command below.
sqoop import --username testuser --password test123 --num-mappers 1 --connect "jdbc:sqlserver://testdomain.com:2031;...
0
votes
0
answers
25
views
Unable to import tables into hdfs using sqoop because of : Found interface, class was expected
I need help with this please, when trying to import table sqoops to hdfs using the below command line
sqoop import --connect jdbc:mysql://localhost/testing --username aya --password root0711 --table ...
0
votes
0
answers
19
views
Sqoop hive importing incorrect data
I am trying to import data from PGSQL and load into Hive using sqoop
Query used:
sqoop import --connect jdbc:postgresql://${HOST}:${PORT}/${INST} --username=${USER} --password=$PASS} --delete-target-...
0
votes
0
answers
28
views
Sqoop export to Vertica issue
I am trying to export the data from HDFS to Vertica as stated below. fields are terminated by ^
xxx-0010091042^DL360G10 srv & Tply - eDirham Project^abcd^\N^Unb^Unb^\N^won^06 - asd, Depos & ...
0
votes
0
answers
39
views
Sqoop from SQL Server with no primary keys
I am trying to do a sqoop import for a table that has few billion records and doesn't have any primary key in the table, so I can't do --split-by any existing column.
Is it possible to split by row ...
0
votes
0
answers
25
views
Sqoop not returning the table Schema in sql server
I am trying to run sqoop command to get the SCHEMA of a table (ABC_XYZ) from SQL Server. But it's not returning the SCHEMA.
Command that is being Executed :
sqoop eval --connect $conStr --username $...
0
votes
0
answers
24
views
Unable to run Sqoop from Oozie
I am running the sqoop command from oozie but it got failed with file not found Exception for Sqoop-site.xml.
While the same is working fine from command line.
StdERR
Log Type: stderr
...
1
vote
1
answer
209
views
Hive managed table issue to create a hive table from a hdfs location in CDP
I have a CDP 7.3.1 where using sqoop , I have loaded data from Postgres database table into HDFS location /ts/gp/node. Now I am trying to create a hive table on this. I get the below error. Please ...
1
vote
1
answer
244
views
Apache Sqoop moved into the Attic in 2021-06
I have installed hadoop version 3.3.1 and sqoop 1.4.7 which doesn't seem compatible , I am getting depreciated API implemented error while importing rdbms table.
As I tried to google for compatible ...
0
votes
0
answers
20
views
Sqoop is loading 'image'-BLOB data from source with 'spacing' in hive table
i had ran the manual SQOOP script to ingest the 'image'-BLOB data from the SQL Server source(test DB) to Hive table. The source data appears like HEXA(example: 1000000000000000AAU8XXXYXXX1XAXX00) ...
0
votes
2
answers
51
views
SQOOP NOLOGGING export task to an Oracle DB
I'm totally stumped as I'm trying to include NOLOGGING into my SQOOP export task to an Oracle Database from HIVE.
The SQOOP user guide -https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#...
0
votes
0
answers
9
views
share last-value between two sqoop jobs
I have one primary sqoop job which fetches data from a table_a. Now for ha purposes I want to create in case of first's sqoop failure, a second sqoop job which will fetch data from the backup table_b. ...
0
votes
1
answer
36
views
Execute a job only when the previous has finished
I have a bash script which has a sqoop exec and after it three impala commands. I want to run it but only when the previous execution has finished. Is this possible to be done in cronjob or in oozie ?
0
votes
0
answers
88
views
Sqoop job failed with "illegal escape character" error
I used below command to sqoop data from a Oracle table to a Hive table with the same table structure.
sqoop import -Dorg.apache.sqoop.splitter.allow_text_splitter=true -query "select * from ...
0
votes
1
answer
25
views
How to add multiple filters in Sqoop --where clause?
My sample RDBMS data has below entries
+--------+-----------+----------+-----------+
| custid | fisrtname | lastname | city |
+--------+-----------+----------+-----------+
| 1 | Arun | ...
0
votes
1
answer
103
views
Sqoop export from HIVE to partitioned Oracle database
I have a non partitioned table in Hive which gets populated with 600 000 records using a batch. This batch clears the data from the table before populating it.
I am trying to sqoop export this data to ...
0
votes
0
answers
123
views
Connection to AWS RDS with Sqoop times out
I'm trying to use a connection from RDS on EMR through sqoop by the following command:
sqoop list-tables \
--connect jdbc:postgresql://sportsdb.cxri########.us-east-2.rds.amazonaws.com:5432/SportsDB \
...
0
votes
1
answer
92
views
No manager for connect string
As soon as my EMR-Cluster was ready to be run.
I started facing some issues when listing databases and importing sqoop
Apparently, sqoop has been installed normally and it is working normally when I ...
0
votes
1
answer
38
views
sqoop query is giving error for the Import command
I am trying to transfer 35 GB table from aws rds postgres to hive but when I try to full table it take much time and after long time execution get stop. so I decide to load incremental way.
schema:All ...
0
votes
0
answers
32
views
SQOOP to HIVE in Parquet
SQOOP is not able to create a hive table in PARQUET format.
sqoop import --connect jdbc:mysql://localhost:3306/sqoop_practice --username root --password hortonworks1 -m 1 --delete-target-dir --target-...
1
vote
1
answer
38
views
There are multiple '. 0' after sqoop exports data from Mysql to hive datetime type
mysql type is datetime,hive is string or timestamp
there are all happen in hive table
mysql
2016-10-13 15:18:10
2016-10-13 15:18:10
2016-10-13 15:18:10
2016-10-13 15:18:10
2016-10-13 15:18:10
2016-10-...
0
votes
0
answers
16
views
Fetch with sqoop a MySQL Table's status and store it in a variable
I am fetching data with sqoop from a MySQL table. Although some times the fetching process is happening when the table is in no readable mode(because it does its daily backup proccess). For this I ...
0
votes
0
answers
33
views
Sqoop hive-import doesnt output the HDFS folder
I am trying to import a table from MySql database to hive table using sqoop to understand how hive import works. From my reading, it should works:
Import data to HDFS (HOME_DIRECTORY/TABLE_NAME)
...
0
votes
0
answers
20
views
HBASE to HIVE one time data load
I want to sync full data one time from HBase to Hive. I am looking for the most efficient way to do that. I want to use sqoop but I am not sure if we can use sqoop to import from Hbase to hive.
-1
votes
1
answer
35
views
Difference between Temporary rootdir and bindir in sqoop job
I am not able to get the difference between Temporary rootdir and bindir in sqoop job.
can anyone explain this, it would be great
0
votes
0
answers
35
views
Error: java.io.IOException: mysqldump terminated with status 2
Sqoop Command:
sqoop import --connect jdbc:mysql://localhost/retail_db --username root -P --table orders -m 1 --direct --create-hive-table --hive-import --hive-database retail_db_staging
Error: java....
1
vote
1
answer
136
views
Sqoop Import command throwing error but it is working from terminal
Below command is working fine from terminal but when I extecute it from Oozie it is throwing error.
bash
sqoop import --connect jdbc:mysql://172.16.16.128:3306/employees -- username=hive --password=...
0
votes
2
answers
336
views
Transfer files from HDFS dir to sftp server
I am trying to transfer all the part* files from a directory directly from HDFS dir to sftp server. All the files in hdfs folder is pretty huge, so I do not want to copy them to local file system.
The ...
0
votes
0
answers
89
views
SQOOP export fails on java.lang.NumberFormatException: For input string: ""
I have created following table in MySQL under school database:
CREATE TABLE `student` (
`id` int NOT NULL,
`first_name` varchar(100) DEFAULT NULL,
`last_name` varchar(100) DEFAULT NULL,
`email`...
1
vote
0
answers
23
views
HiveQL is cutting zeros from fields
when I bring some data to the hive, it ends up cutting the 0 both at the beginning and at the end, normally this data comes from a document.
However, the CPF comes with 11 digits and the CNPJ comes ...