When I run my MRJob script and use the CLI to spin up EMR clusters for the work, I am trying to figure out how to load the data from S3 onto HDFS in the clusters. I want to do this as part of the setup process.
I've searched a number of places to get insight into the right syntax to use with MRJob in order to pre-load my S3 data into HDFS. I'm still getting up to speed on the terminology and process.
MRJob mentions JarStep and Hadoop Streaming in its docs: https://pythonhosted.org/mrjob/guides/writing-mrjobs.html
Yelp has a discussion thread on this at: https://github.com/Yelp/mrjob/pull/665
AWS has a tool S3DistCp to provide syntax for loading and the examples appear related to Java. (I can't post more than 2 links but a google search will give more info on this)
My approach has been to include JarStep as a step before MRStep in my steps method of the MRJob script. I was trying to pass in the S3DistCp commands as arguments.
def steps(self):
return [JarStep(name='jar-step', jar='s3://<bucket to jar file>',
args=['--src,s3://<bucket to data>, --dest,hdfs:///<bucket>']),
MRStep(...)
]
I've tried a few variations on this with and without quotes or quotes splitting up input and output or spaces between commas. Primarily the error I get is "Exception in thread 'main' java.lang.RuntimeException: Argument --src,, --desc,, doesn't match.
The examples I've found are similar to the following:
args=['naive-bayes', JarStep.INPUT, JarStep.OUTPUT]
Its not clear if JarStep.INPUT should be replaced or defined elsewhere. I'm starting to think that I should pass the java commands on the CLI under --jobconf so it aligns to the S3DistCp documentation.
Any ideas or just plain examples, please send my way.