Skip to main content
Filter by
Sorted by
Tagged with
0 votes
0 answers
34 views

Sqoop: error when migrating table in pgadmin4 database to HDFS

I'm a newbie to Sqoop and HDFS and I'm trying to migrate a table in pgadmin4 to HDFS by using Sqoop. I spent a day to fix errors but finally I encountered this error that I cant find a solution ...
Bùi Nam Quân's user avatar
0 votes
0 answers
16 views

Postgres/Sqoop : ERROR: invalid byte sequence for encoding "UTF8"

While Trying to import data with sqoop from a postgres table to hive a got the following error : ERROR db.DBRecordReader: Top level exception: org.postgresql.util.PSQLException: ERROR: invalid byte ...
Omar Tougui's user avatar
0 votes
0 answers
94 views

Teradata jdbc driver error unexpected parcel kind received

I am facing some unknown issues in my data pipeline. There are few mappings which dumps data from hadoop to teradata one to one using informatica dei. For some of those mappings I am getting below ...
KeenLearner's user avatar
0 votes
0 answers
8 views

Use sqoop to import oracle data to hive. Although sqoop reports an error, hive can find the data

hadoop version is 3.3.3 hive version is 3.1.0 sqoop version is 1.4.7 Use sqoop to import oracle data to hive. Although sqoop reports an error, hive can find the data I executed the following script as ...
M Master's user avatar
0 votes
0 answers
17 views

Transfer files from a HDFS directory to a remote Linux server

We have a requirement to transfer files from an HDFS directory to a remote server. I've noticed options to copy files from HDFS to the local filesystem first (using copyToLocal) and then transfer ...
Sbala's user avatar
  • 11
0 votes
0 answers
9 views

Listing Sqoop jobs with details in terminal

Is there a way to list sqoop job's with for example their scheduled times from 9 am to 6 pm. Also, there's main jobs, in which there can be like 10 sqoop subjobs. Thanks in advance
DariusB's user avatar
  • 13
0 votes
0 answers
32 views

Job failed with state FAILED due to: Job commit failed: org.apache.hive.hcatalog.common.HCatException : 2006 : Error adding partition to metastore

Using hive, sqoop, I can import data with instruction like this sqoop import \ > --connect jdbc:mysql://192.168.88.101:3306/test?enabledTLSProtocols=TLSv1.2 \ > --username root \ > --password ...
May's user avatar
  • 1
1 vote
0 answers
167 views

Alternative to Apache Sqoop -- Bulk Transfer from RDBMS to HDFS

I am trying to migrate data from Postgres to HDFS (ideally stored in parquet). Apache Sqoop was built for this purpose, but it was moved into Attic. There are a large number of different tables I'm ...
willshen's user avatar
0 votes
0 answers
43 views

Permission denied error while importing a table into HDFS using Scoop

I am new to Hadoop. I have connected to Cloudera quickstart VM. I am trying to import the categories table present in MySQL retail_db in HDFS but I am getting permission denied error. Following is the ...
Ora Aff's user avatar
  • 662
0 votes
0 answers
10 views

Sqoop query at source DB is not terminating on sqoop job termination

I have a sqoop job that fetches data from SQL Server. For some reson if I am killing the sqoop-job, the --query trigerred at the source is not getting killed, it keeps running at the sql server end. I ...
aiman's user avatar
  • 1,103
0 votes
0 answers
53 views

When using Sqoop to import data from MySQL into HBase, an error occurred

I tried to execute the following command ./bin/sqoop import --connect jdbc:mysql://localhost:3306/dblab --username root --password 123456 --table user_action --hbase-table user_action --column-...
阿ori's user avatar
  • 1
0 votes
1 answer
22 views

import sqoop to hdfs error Application failed 2 times due to AM Container

enter image description here I try to import data using sqoop cmd : sqoop import --connect "jdbc:sqlserver://192.168.26.8:1717;databasename=VMC" -- table contact -m 1 --username sa --...
Tien Le's user avatar
0 votes
0 answers
14 views

Sqoop Export using multiple mappers issue

Facing issue in sqoop export to UDB database when we use more number of maps. We receiving -911 abend (no other process is using the DB) sqoop export -Dsqoop.export.records.per.statement=50000 -D ...
vicky bangre's user avatar
0 votes
0 answers
16 views

Airflow - copying Sqoop jars

Going through the sqoop documentation here and found below Sqoop will copy the jars in $SQOOP_HOME/lib folder to job cache every time when start a Sqoop job. When launched by Oozie this is ...
luckyluke's user avatar
  • 654
0 votes
1 answer
101 views

I got this error There are 0 datanode(s) running and no node(s) are excluded in this operation

Do i need to install hive in order to use sqoop on my machine? while i try to import the data from mysql to hadoop it throws error like this: bhandarisamir@Lomass-MacBook-Pro ~ % sqoop import \ --...
Shamir Bhandari's user avatar
0 votes
0 answers
21 views

Error in Sqoop Import statement when running in Windows

Trying to import from SQL DB to D Drive using Sqoop. Hadoop and Sqoop are installed and configured correctly on Windows. Able to import data from UDB and Oracle database. Hadoop 2.9.2 Sqoop 1.4.7 DB - ...
vicky bangre's user avatar
0 votes
0 answers
37 views

How to fine Tune Apache Sqoop on Python to run heavy loads?

I am looking for ways to finetune sqoop when used on python so it can handle heavy loads. So far I have considered num_mappers, split_by, batch and increasing memory on map using something like ...
Stamatis Tiniakos's user avatar
0 votes
0 answers
61 views

MapReduce is working normally and suddenly restarts the process without aborting the Job (Sqoop)

MapReduce is working normally and, suddenly, it restarts the process without aborting the Job (Sqoop), then at the end it duplicates the information in the table?
Edvaldo Lucena's user avatar
0 votes
0 answers
31 views

AWS S3 to Oracle ( Raw ) using sqoop

Am getting list of EmployeeID as output as part of my EMR result ( in S3) . One of the sample value is F576334E17894499B6334E178974990B . I want to load this to a Oracle table which is EMPLOYEE(...
AccumuloUser's user avatar
0 votes
2 answers
343 views

Sqoop import CLOB column from oracle database to hive string

I am trying to run a sqoop process to get a table from an Oracle database into Hive. The table has a CLOB type column called VALUE, which is the one that is giving me the problems. To save this table ...
J. Mendoza's user avatar
0 votes
0 answers
164 views

Sqoop import throws java.lang.ClassNotFoundException where class refers to table name

I'm using Ubuntu 22.04 and I built a Hadoop server with a sample database in MySql Server which has an orders table. I'm using Sqoop to generate an import with the following command: sqoop import --...
alexdevs's user avatar
0 votes
1 answer
63 views

How to dynamic partitioning in Hive using tez engine?

I sqooped data from mysql to hdfs. sqoop import --username aaa --password bbb --num-mappers 1 --connect "jdbc:sqlserver://xxx.xxx.xxx.xxx:1234;database=test_db" --query "select DATE, ...
SecY's user avatar
  • 347
0 votes
1 answer
96 views

Unable to add mysql-connector to sqoop lib folder in AWS EMR cluster

I am trying too run sqoop export in AWS EMR cluster. When I try to copy the MySQL connector jar to sqoop lib folder is gives me permission error: `> cp mysql-connector-j-8.0.32/mysql-connector-j-8....
hns_03's user avatar
  • 1
0 votes
0 answers
28 views

i can not import database from sqlserver to hadoop using sqoop

using cloudera quick start vm 5.12.0 with 8 gb ram and 64gb storage ** I can not import database into hadoop when I write the command of sqoop list-database it work with me and show me the databases ...
Mohamed Abdelwares1000's user avatar
0 votes
0 answers
66 views

Big table sqoop import to hive, fixed mapper less rows fetch

I wanna sqoop big table in mysql. this table has about 300,000,000 rows and operating. I wanna sqoop as soon as possible, not burden to system. So, my sqoop command is sqoop import \ --connect ...
fleamon's user avatar
0 votes
1 answer
213 views

Process real time data using kafka

I have a requirement to implement the solution for below usecase. Currently Applications are storing data into Postgres database but Postgres database is facing storage issue. So the plan is to move ...
Lavanya varma's user avatar
1 vote
1 answer
25 views

Can Sqoop update record on Oracle RDBMS table that have different column structure with Hive table

I'm a Hadoop newcomer trying to export data from Hive to Oracle. Can Sqoop update data to Oracle table let say, Oracle Table have column A,B,C,D,E I stored data on Hive table as B,C,E Can Sqoop export ...
3ORZ's user avatar
  • 79
0 votes
1 answer
125 views

Cannot add JDBC driver in Sqoop command when running import command using Airflow 2.5.0

I am running a Sqoop import command which imports a table from MySQL db and loads it to HDFS using Sqoop. I have created a below DAG which performs this above activity. from airflow.models import DAG ...
Salva's user avatar
  • 91
0 votes
0 answers
345 views

How to schedule Sqoop import from Mysql and load in HDFS using Airflow 2.5.0

I want to learn Airflow and thus am performing following task. I want to read data from MySQL DB and load it to HDFS. I am doing this task using sqoop manually. Now I want to schedule this activity ...
Salva's user avatar
  • 91
1 vote
0 answers
116 views

Sqoop import with Hcatalog failled:java.util.LinkedList cannot be cast to org.apache.hive.hcatalog.mapreduce.InputJobInfo

I am trying to import a table from mysql to hive and i getting this errors. 2023-01-09 07:05:51,198 INFO mapreduce.Job: Running job: job_1673201919048_0006 2023-01-09 07:06:30,327 INFO mapreduce.Job: ...
DylanGong's user avatar
1 vote
0 answers
29 views

Sqoop MDSYS.SDO_GEOMETRY columns in GCP DataProc

I am trying to sqoop MDSYS.SDO_GEOMETRY column into GCS Bucket using dataproc but sqoop is ignoring MDSYS.SDO_GEOMETRY column from selection. I am not sure what is the issue. Also Do I need to convert ...
Technical Guy's user avatar
0 votes
1 answer
47 views

Parsing a column using shell scripting

I have a sqoop command in a shell script that returns a one column one row output like this: Can you please help me with parsing this above output to get only the value 0? Thank you.
vicmartin's user avatar
0 votes
1 answer
73 views

sqoop import communications link falilure

I am getting below error while importing a MySQL table using Sqoop to HDFS. as MySQL is on my Name node and rest slaves are on different VMs, for this I also edited the MySQL confide file and disables ...
Salva's user avatar
  • 91
0 votes
0 answers
235 views

Error while importing table from mysql to hdfs using sqoop

i am running this import command to import E_INFO table from MySQL databases to HDFS using sqoop. the job is failing with below given error. sqoop import --connect jdbc:mysql://localhost/...
Salva's user avatar
  • 91
1 vote
0 answers
151 views

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils

Given below is the error. I have added the apache.org.common.lang jar still am getting this . am having these .JARs in my lib folder am is missing on any? ant-contrib-1.0b3.jar ant-eclipse-1.0-jvm1.2....
Salva's user avatar
  • 91
0 votes
1 answer
71 views

Facing error while reading MySQL table in Sqoop. using Sqoop Eval

h-user@h-primary:~/sqoop-1.4.7/bin$ sqoop eval --connect jdbc:mysql://localhost/HADOOP_BANK_DATA_POC --username root --password xxxxxxxx --query "select * from E_INFO" Warning: /home/h-user/...
Salva's user avatar
  • 91
0 votes
1 answer
156 views

Apache sqoop is not getting installed in MacOS by brew or jar

I have hadoop installed in my system. I want to copy data from postgresql table to hdfs using apache sqoop. I tried installing sqoop by brew on my mac, but I'm getting this error: Error: sqoop has ...
Suraj221b's user avatar
0 votes
1 answer
95 views

Can we install Sqoop, Hive & MySQL in Name Node (Master) of a Hadoop Cluster?

I have a multi-node Hadoop cluster as 1 master and 2 Slaves. i want to try import from MySQL and load in HDFS. i want to have Hive to write hive queries on HDFS. to perform above task, can I install ...
Salva's user avatar
  • 91
0 votes
0 answers
166 views

sqoop export hive->oracle error(string/varchar column to clob column)

[Hive side] create table TEST.HIVE_TEST ( CONTENT string) stored as textfile; [oracle side] create table TEST.ORACLE_TEST ( CONTENT clob); [sqoop export] /usr/...
are123's user avatar
  • 1
0 votes
1 answer
100 views

I am getting an error while creating a sqoop job

Warning: /opt/sqoop/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /opt/sqoop/../accumulo does not exist! Accumulo ...
Raja's user avatar
  • 1
0 votes
1 answer
231 views

ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class

I am receiving an error when connecting mysql and I do not know the steps to connect the driver. I am using AWS EMR as a student. My intention is to run some pig commands after I connect. [hadoop@ip-...
Wing's user avatar
  • 1
0 votes
0 answers
110 views

Informatica DEI 10.5.2 - Sqoop query importing data from Oracle 9i giving ORA-01882: timezone region not found

Informatica mapping is giving error while extracting data using JDBC connection string connecting to Oracle 9.2 version. Informatica mapping is using Sqoop to execute the command. I am executing the ...
Arun's user avatar
  • 1
0 votes
0 answers
28 views

Why Sqoop uses 4 mappers by default

While using sqoop in my day to day activity, I don't get why the developer's of sqoop chosen default number of mappers as 4. As there is provision for increasing or decreasing the number of mappers. ...
Shubham Shrikhande's user avatar
0 votes
1 answer
51 views

How to incrementally update a table from SQL to HDFS

I have a table imported to HDFS from RDBMS using Sqoop. After I make updates and inserts on RDBMS, how do I update the HDFS version, only transferring new and updated records? There is no timestamp or ...
Prabhakar Yadav's user avatar
2 votes
2 answers
259 views

How to fix ORA-00933: SQL command not properly ended using sqoop

I am trying to import data from Oracle into hive using SQOOP. I have read the threads with the same error, it doesn't seem to solve my problem. This is the command I am using sqoop import --connect ...
CuriousCat's user avatar
1 vote
0 answers
320 views

'atlas.graph.index.search.max-result-set-size' doesn't map to a List object: 150

I tried to setup the sqoop-hook with Atlas Following these steps : 1- Set-up Atlas hook in sqoop-site.xml: <property> <name>sqoop.job.data.publish.class</name> <value>...
Omar Tougui's user avatar
1 vote
1 answer
231 views

How can I migrate thousands of Hive tables to a MySQL database?

Can this be possible with a sqoop export script? With the script below, I can only export one table at a time. $sqoop export \ --connect jdbc:mysql://localhost/db \ --username root \ --table employee \...
Anurag Baranwal's user avatar
0 votes
3 answers
201 views

Sqoop --where condition with multiple clause

I'm trying to get data from Oracle and import in Hadoop table. I'm making changes inn existing sqoop, i have to use --where to filter the record. For now we have in where date=somedate condition, now ...
aruntheimperfect's user avatar
1 vote
0 answers
98 views

sqoop import oracle got stock at progress 0.0% for hours

When sqoop import from oracle, we saw the following logs on yarn: 2022-05-10 04:52:13,351 INFO [IPC Server handler 12 on 44827] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of ...
user19564902's user avatar
0 votes
0 answers
249 views

sqoop import to s3 com.amazonaws.SdkClientException: Couldn't initialize a SAX driver to create an XMLReader

Using sqoop import mysql to s3, but it get ERROR tool.ImportTool: Import failed: org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on tmp/tb_room_type3/: com.amazonaws.SdkClientException: ...
littlely's user avatar
  • 1,408

1
2 3 4 5
52