2,596
questions
0
votes
0
answers
34
views
Sqoop: error when migrating table in pgadmin4 database to HDFS
I'm a newbie to Sqoop and HDFS and I'm trying to migrate a table in pgadmin4 to HDFS by using Sqoop. I spent a day to fix errors but finally I encountered this error that I cant find a solution ...
0
votes
0
answers
16
views
Postgres/Sqoop : ERROR: invalid byte sequence for encoding "UTF8"
While Trying to import data with sqoop from a postgres table to hive a got the following error :
ERROR db.DBRecordReader: Top level exception:
org.postgresql.util.PSQLException: ERROR: invalid byte ...
0
votes
0
answers
94
views
Teradata jdbc driver error unexpected parcel kind received
I am facing some unknown issues in my data pipeline. There are few mappings which dumps data from hadoop to teradata one to one using informatica dei. For some of those mappings I am getting below ...
0
votes
0
answers
8
views
Use sqoop to import oracle data to hive. Although sqoop reports an error, hive can find the data
hadoop version is 3.3.3
hive version is 3.1.0
sqoop version is 1.4.7
Use sqoop to import oracle data to hive. Although sqoop reports an error, hive can find the data
I executed the following script as ...
0
votes
0
answers
17
views
Transfer files from a HDFS directory to a remote Linux server
We have a requirement to transfer files from an HDFS directory to a remote server.
I've noticed options to copy files from HDFS to the local filesystem first (using copyToLocal) and then transfer ...
0
votes
0
answers
9
views
Listing Sqoop jobs with details in terminal
Is there a way to list sqoop job's with for example their scheduled times from 9 am to 6 pm. Also, there's main jobs, in which there can be like 10 sqoop subjobs. Thanks in advance
0
votes
0
answers
32
views
Job failed with state FAILED due to: Job commit failed: org.apache.hive.hcatalog.common.HCatException : 2006 : Error adding partition to metastore
Using hive, sqoop, I can import data with instruction like this
sqoop import \
> --connect jdbc:mysql://192.168.88.101:3306/test?enabledTLSProtocols=TLSv1.2 \
> --username root \
> --password ...
1
vote
0
answers
167
views
Alternative to Apache Sqoop -- Bulk Transfer from RDBMS to HDFS
I am trying to migrate data from Postgres to HDFS (ideally stored in parquet). Apache Sqoop was built for this purpose, but it was moved into Attic. There are a large number of different tables I'm ...
0
votes
0
answers
43
views
Permission denied error while importing a table into HDFS using Scoop
I am new to Hadoop. I have connected to Cloudera quickstart VM. I am trying to import the categories table present in MySQL retail_db in HDFS but I am getting permission denied error. Following is the ...
0
votes
0
answers
10
views
Sqoop query at source DB is not terminating on sqoop job termination
I have a sqoop job that fetches data from SQL Server. For some reson if I am killing the sqoop-job, the --query trigerred at the source is not getting killed, it keeps running at the sql server end.
I ...
0
votes
0
answers
53
views
When using Sqoop to import data from MySQL into HBase, an error occurred
I tried to execute the following command
./bin/sqoop import --connect jdbc:mysql://localhost:3306/dblab --username root --password 123456 --table user_action --hbase-table user_action --column-...
0
votes
1
answer
22
views
import sqoop to hdfs error Application failed 2 times due to AM Container
enter image description here
I try to import data using sqoop
cmd :
sqoop import --connect "jdbc:sqlserver://192.168.26.8:1717;databasename=VMC" -- table contact -m 1 --username sa --...
0
votes
0
answers
14
views
Sqoop Export using multiple mappers issue
Facing issue in sqoop export to UDB database when we use more number of maps.
We receiving -911 abend (no other process is using the DB)
sqoop export -Dsqoop.export.records.per.statement=50000 -D
...
0
votes
0
answers
16
views
Airflow - copying Sqoop jars
Going through the sqoop documentation here and found below
Sqoop will copy the jars in $SQOOP_HOME/lib folder to job cache every
time when start a Sqoop job. When launched by Oozie this is
...
0
votes
1
answer
101
views
I got this error There are 0 datanode(s) running and no node(s) are excluded in this operation
Do i need to install hive in order to use sqoop on my machine?
while i try to import the data from mysql to hadoop it throws error like this:
bhandarisamir@Lomass-MacBook-Pro ~ % sqoop import \
--...
0
votes
0
answers
21
views
Error in Sqoop Import statement when running in Windows
Trying to import from SQL DB to D Drive using Sqoop. Hadoop and Sqoop are installed and configured correctly on Windows. Able to import data from UDB and Oracle database.
Hadoop 2.9.2
Sqoop 1.4.7
DB - ...
0
votes
0
answers
37
views
How to fine Tune Apache Sqoop on Python to run heavy loads?
I am looking for ways to finetune sqoop when used on python so it can handle heavy loads.
So far I have considered num_mappers, split_by, batch and increasing memory on map using something like ...
0
votes
0
answers
61
views
MapReduce is working normally and suddenly restarts the process without aborting the Job (Sqoop)
MapReduce is working normally and, suddenly, it restarts the process without aborting the Job (Sqoop), then at the end it duplicates the information in the table?
0
votes
0
answers
31
views
AWS S3 to Oracle ( Raw ) using sqoop
Am getting list of EmployeeID as output as part of my EMR result ( in S3) . One of the sample value is F576334E17894499B6334E178974990B . I want to load this to a Oracle table which is EMPLOYEE(...
0
votes
2
answers
343
views
Sqoop import CLOB column from oracle database to hive string
I am trying to run a sqoop process to get a table from an Oracle database into Hive. The table has a CLOB type column called VALUE, which is the one that is giving me the problems. To save this table ...
0
votes
0
answers
164
views
Sqoop import throws java.lang.ClassNotFoundException where class refers to table name
I'm using Ubuntu 22.04 and I built a Hadoop server with a sample database in MySql Server which has an orders table.
I'm using Sqoop to generate an import with the following command:
sqoop import --...
0
votes
1
answer
63
views
How to dynamic partitioning in Hive using tez engine?
I sqooped data from mysql to hdfs.
sqoop import --username aaa --password bbb --num-mappers 1 --connect "jdbc:sqlserver://xxx.xxx.xxx.xxx:1234;database=test_db" --query "select DATE, ...
0
votes
1
answer
96
views
Unable to add mysql-connector to sqoop lib folder in AWS EMR cluster
I am trying too run sqoop export in AWS EMR cluster. When I try to copy the MySQL connector jar to sqoop lib folder is gives me permission error:
`> cp mysql-connector-j-8.0.32/mysql-connector-j-8....
0
votes
0
answers
28
views
i can not import database from sqlserver to hadoop using sqoop
using cloudera quick start vm 5.12.0 with 8 gb ram and 64gb storage
**
I can not import database into hadoop when I write the command of sqoop list-database it work with me and show me the databases ...
0
votes
0
answers
66
views
Big table sqoop import to hive, fixed mapper less rows fetch
I wanna sqoop big table in mysql.
this table has about 300,000,000 rows and operating.
I wanna sqoop as soon as possible, not burden to system.
So, my sqoop command is
sqoop import \
--connect ...
0
votes
1
answer
213
views
Process real time data using kafka
I have a requirement to implement the solution for below usecase.
Currently Applications are storing data into Postgres database but Postgres database is facing storage issue. So the plan is to move ...
1
vote
1
answer
25
views
Can Sqoop update record on Oracle RDBMS table that have different column structure with Hive table
I'm a Hadoop newcomer trying to export data from Hive to Oracle. Can Sqoop update data to Oracle table let say,
Oracle Table have column A,B,C,D,E
I stored data on Hive table as B,C,E
Can Sqoop export ...
0
votes
1
answer
125
views
Cannot add JDBC driver in Sqoop command when running import command using Airflow 2.5.0
I am running a Sqoop import command which imports a table from MySQL db and loads it to HDFS using Sqoop. I have created a below DAG which performs this above activity.
from airflow.models import DAG
...
0
votes
0
answers
345
views
How to schedule Sqoop import from Mysql and load in HDFS using Airflow 2.5.0
I want to learn Airflow and thus am performing following task.
I want to read data from MySQL DB and load it to HDFS. I am doing this task using sqoop manually.
Now I want to schedule this activity ...
1
vote
0
answers
116
views
Sqoop import with Hcatalog failled:java.util.LinkedList cannot be cast to org.apache.hive.hcatalog.mapreduce.InputJobInfo
I am trying to import a table from mysql to hive and i getting this errors.
2023-01-09 07:05:51,198 INFO mapreduce.Job: Running job: job_1673201919048_0006
2023-01-09 07:06:30,327 INFO mapreduce.Job: ...
1
vote
0
answers
29
views
Sqoop MDSYS.SDO_GEOMETRY columns in GCP DataProc
I am trying to sqoop MDSYS.SDO_GEOMETRY column into GCS Bucket using dataproc but sqoop is ignoring MDSYS.SDO_GEOMETRY column from selection. I am not sure what is the issue. Also Do I need to convert ...
0
votes
1
answer
47
views
Parsing a column using shell scripting
I have a sqoop command in a shell script that returns a one column one row output like this:
Can you please help me with parsing this above output to get only the value 0? Thank you.
0
votes
1
answer
73
views
sqoop import communications link falilure
I am getting below error while importing a MySQL table using Sqoop to HDFS.
as MySQL is on my Name node and rest slaves are on different VMs, for this I also edited the MySQL confide file and disables ...
0
votes
0
answers
235
views
Error while importing table from mysql to hdfs using sqoop
i am running this import command to import E_INFO table from MySQL databases to HDFS using sqoop. the job is failing with below given error.
sqoop import --connect jdbc:mysql://localhost/...
1
vote
0
answers
151
views
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils
Given below is the error. I have added the apache.org.common.lang jar still am getting this . am having these .JARs in my lib folder am is missing on any?
ant-contrib-1.0b3.jar
ant-eclipse-1.0-jvm1.2....
0
votes
1
answer
71
views
Facing error while reading MySQL table in Sqoop. using Sqoop Eval
h-user@h-primary:~/sqoop-1.4.7/bin$ sqoop eval --connect jdbc:mysql://localhost/HADOOP_BANK_DATA_POC --username root --password xxxxxxxx --query "select * from E_INFO"
Warning: /home/h-user/...
0
votes
1
answer
156
views
Apache sqoop is not getting installed in MacOS by brew or jar
I have hadoop installed in my system. I want to copy data from postgresql table to hdfs using apache sqoop. I tried installing sqoop by brew on my mac, but I'm getting this error:
Error: sqoop has ...
0
votes
1
answer
95
views
Can we install Sqoop, Hive & MySQL in Name Node (Master) of a Hadoop Cluster?
I have a multi-node Hadoop cluster as 1 master and 2 Slaves.
i want to try import from MySQL and load in HDFS.
i want to have Hive to write hive queries on HDFS.
to perform above task, can I install ...
0
votes
0
answers
166
views
sqoop export hive->oracle error(string/varchar column to clob column)
[Hive side]
create table TEST.HIVE_TEST (
CONTENT string)
stored as textfile;
[oracle side]
create table TEST.ORACLE_TEST (
CONTENT clob);
[sqoop export]
/usr/...
0
votes
1
answer
100
views
I am getting an error while creating a sqoop job
Warning: /opt/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/sqoop/../accumulo does not exist! Accumulo ...
0
votes
1
answer
231
views
ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class
I am receiving an error when connecting mysql and I do not know the steps to connect the driver. I am using AWS EMR as a student. My intention is to run some pig commands after I connect.
[hadoop@ip-...
0
votes
0
answers
110
views
Informatica DEI 10.5.2 - Sqoop query importing data from Oracle 9i giving ORA-01882: timezone region not found
Informatica mapping is giving error while extracting data using JDBC connection string connecting to Oracle 9.2 version. Informatica mapping is using Sqoop to execute the command.
I am executing the ...
0
votes
0
answers
28
views
Why Sqoop uses 4 mappers by default
While using sqoop in my day to day activity, I don't get why the developer's of sqoop chosen default number of mappers as 4. As there is provision for increasing or decreasing the number of mappers. ...
0
votes
1
answer
51
views
How to incrementally update a table from SQL to HDFS
I have a table imported to HDFS from RDBMS using Sqoop. After I make updates and inserts on RDBMS, how do I update the HDFS version, only transferring new and updated records? There is no timestamp or ...
2
votes
2
answers
259
views
How to fix ORA-00933: SQL command not properly ended using sqoop
I am trying to import data from Oracle into hive using SQOOP. I have read the threads with the same error, it doesn't seem to solve my problem.
This is the command I am using
sqoop import --connect ...
1
vote
0
answers
320
views
'atlas.graph.index.search.max-result-set-size' doesn't map to a List object: 150
I tried to setup the sqoop-hook with Atlas Following these steps :
1- Set-up Atlas hook in sqoop-site.xml:
<property>
<name>sqoop.job.data.publish.class</name>
<value>...
1
vote
1
answer
231
views
How can I migrate thousands of Hive tables to a MySQL database?
Can this be possible with a sqoop export script? With the script below, I can only export one table at a time.
$sqoop export \
--connect jdbc:mysql://localhost/db \
--username root \
--table employee \...
0
votes
3
answers
201
views
Sqoop --where condition with multiple clause
I'm trying to get data from Oracle and import in Hadoop table. I'm making changes inn existing sqoop, i have to use --where to filter the record. For now we have in where date=somedate condition, now ...
1
vote
0
answers
98
views
sqoop import oracle got stock at progress 0.0% for hours
When sqoop import from oracle, we saw the following logs on yarn:
2022-05-10 04:52:13,351 INFO [IPC Server handler 12 on 44827] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of ...
0
votes
0
answers
249
views
sqoop import to s3 com.amazonaws.SdkClientException: Couldn't initialize a SAX driver to create an XMLReader
Using sqoop import mysql to s3, but it get ERROR tool.ImportTool: Import failed: org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on tmp/tb_room_type3/: com.amazonaws.SdkClientException: ...