1

I am using hadoop version 1.2.1 and sqoop 1.4.4

I am new to hadoop/sqoop and am having a problem. I have data in hdfs that I want to export into MySQL but the export keeps failing. The statement I used is:

sqoop export --connect jdbc:mysql://{ip address}/{database} --username username -P --table {tablename} --export-dir {export-dir} --input-fields-terminated-by ',' --lines-terminated-by '\n' --verbose

The error I get is:

14/02/28 10:12:40 INFO mapred.JobClient: Running job: job_201402040959_0234
14/02/28 10:12:41 INFO mapred.JobClient:  map 0% reduce 0%
14/02/28 10:12:51 INFO mapred.JobClient:  map 50% reduce 0%
14/02/28 10:22:51 INFO mapred.JobClient:  map 0% reduce 0%
14/02/28 10:22:52 INFO mapred.JobClient: Task Id : attempt_201402040959_0234_m_000000_0, Status : FAILED
Task attempt_201402040959_0234_m_000000_0 failed to report status for 600 seconds. Killing!
14/02/28 10:22:52 INFO mapred.JobClient: Task Id : attempt_201402040959_0234_m_000001_0, Status : FAILED
Task attempt_201402040959_0234_m_000001_0 failed to report status for 600 seconds. Killing!
14/02/28 10:23:00 INFO mapred.JobClient:  map 50% reduce 0%
14/02/28 10:33:00 INFO mapred.JobClient:  map 0% reduce 0%
14/02/28 10:33:00 INFO mapred.JobClient: Task Id : attempt_201402040959_0234_m_000000_1, Status : FAILED
Task attempt_201402040959_0234_m_000000_1 failed to report status for 600 seconds. Killing!
14/02/28 10:33:00 INFO mapred.JobClient: Task Id : attempt_201402040959_0234_m_000001_1, Status : FAILED
Task attempt_201402040959_0234_m_000001_1 failed to report status for 600 seconds. Killing!
14/02/28 10:33:09 INFO mapred.JobClient:  map 50% reduce 0%
14/02/28 10:43:09 INFO mapred.JobClient:  map 0% reduce 0%
14/02/28 10:43:09 INFO mapred.JobClient: Task Id : attempt_201402040959_0234_m_000000_2, Status : FAILED
Task attempt_201402040959_0234_m_000000_2 failed to report status for 600 seconds. Killing!
14/02/28 10:43:10 INFO mapred.JobClient: Task Id : attempt_201402040959_0234_m_000001_2, Status : FAILED
Task attempt_201402040959_0234_m_000001_2 failed to report status for 600 seconds. Killing!
14/02/28 10:43:18 INFO mapred.JobClient:  map 50% reduce 0%
14/02/28 10:53:18 INFO mapred.JobClient:  map 25% reduce 0%
14/02/28 10:53:19 INFO mapred.JobClient:  map 0% reduce 0%
14/02/28 10:53:20 INFO mapred.JobClient: Job complete: job_201402040959_0234
14/02/28 10:53:20 INFO mapred.JobClient: Counters: 7
14/02/28 10:53:20 INFO mapred.JobClient:   Job Counters 
14/02/28 10:53:20 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=11987
14/02/28 10:53:20 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
14/02/28 10:53:20 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
14/02/28 10:53:20 INFO mapred.JobClient:     Launched map tasks=8
14/02/28 10:53:20 INFO mapred.JobClient:     Data-local map tasks=8
14/02/28 10:53:20 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
14/02/28 10:53:20 INFO mapred.JobClient:     Failed map tasks=1
14/02/28 10:53:20 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 2,441.242 seconds (0 bytes/sec)
14/02/28 10:53:20 INFO mapreduce.ExportJobBase: Exported 0 records.
14/02/28 10:53:20 ERROR tool.ExportTool: Error during export: Export job failed!

An example of the data is:

201110,1.8181818181818181
201111,1.4597701149425288
201112,1.766990291262136
20119,1.6153846153846154
20121,1.5857142857142856
201210,1.55
201211,1.5294117647058822
201212,1.6528925619834711
20122,1.5789473684210527
20123,1.4848484848484849
20124,1.654320987654321
20125,1.5942028985507246
20126,1.5333333333333334
20127,1.4736842105263157
20128,1.4666666666666666
20129,1.4794520547945205
20131,1.6875
201310,8.233183856502242
201311,8.524886877828054
201312,9.333333333333334
20132,1.7272727272727273
20133,3.42
20134,6.380597014925373
20135,9.504716981132075
20136,8.538812785388128
20137,8.609649122807017
20138,8.777272727272727
20139,8.506787330316742
20141,4.741784037558685

I have tried exporting a similar data set with an identical export statement with just integers and not doubles, and that succeeds. I have also tried a similar data set with floats instead of doubles, but that also fails. Would someone please give me a hint as to why this isn't working? Am I doing something wrong with the datatypes that doesn't fit into MySQL?

I have also tried running the same query with the following addition:

-m 1

This gives the same error as above except the map step completes to 100% rather than just 50%.

-Thank you, please let me know if I should have provided some additional information.

2 Answers 2

0

Please update the question stating the Hadoop, Sqoop and MySQL versions so that the issue can be replicated.

I am going to assume that you're using Hadoop 0.21.0. If this is the case then it could be caused by the org.apache.sqoop.mapreduce.ProgressThread class which uses a TaskInputOutputContext, which is not reporting properly to the underlying reporter as described in issue [MAPREDUCE-1905].

If you are using 0.21.0 then you will need to use 0.21.1 or another Hadoop version.

Otherwise I would assume it's some issue in ProgressThread or how Sqoop is reporting. If that doesn't work then there might be something else in your YARN or MR1 logs.

YARN logs default folder (set in etc/hadoop/yarn-env.sh) :

cd $HADOOP_YARN_HOME/logs

MR1 logs default folder (set in etc/hadoop/mapred-env.sh):

cd $HADOOP_MAPRED_HOME/logs
1
  • I looked through the logs in $HADOOP_MAPRED_HOME/logs for quite some time, but I still don't see where my error is coming from. There were a bunch of different log files and somehow I did not find anything that explained why the error was being thrown. I tried comparing the successful integer export to the failed float export and I don't notice a difference in the logs. Do you have any suggestions? Thanks again, sorry if my question is a bit vague. I'm a bit confused, but I can try to clarify if you tell me.
    – Murium
    Mar 3, 2014 at 20:15
0

The error was due to underscores in column names. Apparently you cannot put underscores in column names.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.