0

When I run my MRJob script and use the CLI to spin up EMR clusters for the work, I am trying to figure out how to load the data from S3 onto HDFS in the clusters. I want to do this as part of the setup process.

I've searched a number of places to get insight into the right syntax to use with MRJob in order to pre-load my S3 data into HDFS. I'm still getting up to speed on the terminology and process.

MRJob mentions JarStep and Hadoop Streaming in its docs: https://pythonhosted.org/mrjob/guides/writing-mrjobs.html

Yelp has a discussion thread on this at: https://github.com/Yelp/mrjob/pull/665

AWS has a tool S3DistCp to provide syntax for loading and the examples appear related to Java. (I can't post more than 2 links but a google search will give more info on this)

My approach has been to include JarStep as a step before MRStep in my steps method of the MRJob script. I was trying to pass in the S3DistCp commands as arguments.

 def steps(self):
     return [JarStep(name='jar-step', jar='s3://<bucket to jar file>', 
                     args=['--src,s3://<bucket to data>, --dest,hdfs:///<bucket>']),
             MRStep(...)
             ]

I've tried a few variations on this with and without quotes or quotes splitting up input and output or spaces between commas. Primarily the error I get is "Exception in thread 'main' java.lang.RuntimeException: Argument --src,, --desc,, doesn't match.

The examples I've found are similar to the following:

args=['naive-bayes', JarStep.INPUT, JarStep.OUTPUT]

Its not clear if JarStep.INPUT should be replaced or defined elsewhere. I'm starting to think that I should pass the java commands on the CLI under --jobconf so it aligns to the S3DistCp documentation.

Any ideas or just plain examples, please send my way.

1 Answer 1

0

if you wish to copy single file to your hdfs use command like below

hadoop fs -cp s3://< s3bucket >/< path > hdfs:///output

Or use S3DistCp as below

./elastic-mapreduce --jobflow j-3GY8JC4179IOJ --jar \ /home/hadoop/lib/emr-s3distcp-1.0.jar \ --arg --s3Endpoint --arg 's3-eu-west-1.amazonaws.com' \ --arg --src --arg 's3://myawsbucket/logs/j-3GY8JC4179IOJ/node/' \ --arg --dest --arg 'hdfs:///output' \ --arg --srcPattern --arg '.*[a-zA-Z,]+'

2
  • Thanks for the fast response Sandesh. I'm trying to include the command with either the command I use to spin up the EMR instances or in the MRJob script. As far as I understand, the two options you listed require EMR to already be running. Do you know of an example that would go in the MRJob script, as part of MRJob bash command to start the job or in the conf file?
    – nyghtowl
    Jul 11, 2014 at 16:10
  • I wish someone had given you the answer to this, I'm having the same issue
    – Tom Busby
    Aug 28, 2014 at 22:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.