When creating a new cluster using boto3, I want to use configuration from existing clusters (which is terminated) and thus clone it.
As far as I know, emr_client.run_job_flow
requires all the configuration(Instances, InstanceFleets etc
) to be provided as parameters.
Is there any way I can clone from existing cluster like I can do from aws console for EMR.
-
Possible Duplicate : stackoverflow.com/questions/43579746/…– Kush VyasOct 12, 2017 at 8:05
-
1Please read both the questions.They are different.– Amey DahaleOct 12, 2017 at 8:47
-
@AmeyDahale Have you got the answer? Seems that should be a popular function for boto3 to clone an existing EMR.– BlueDolphinSep 5, 2018 at 4:19
-
I have kept a static config file with all EMR details and using boto3's run_job_flow to reuse the existing EMR's properties. /i guess it would be good to have boto3 function to clone existing EMR.– Amey DahaleSep 6, 2018 at 5:45
-
I created this issue (albeit on the java sdk), since i see this is such a common request: github.com/aws/aws-sdk-java/issues/2002– Leonel AtencioMay 23, 2019 at 16:59
2 Answers
What i can recommend you, is using the AWS CLI to fire your Cluster.
It permit to versioning your cluster configuration and you can easily load steps configuration with a json file.
aws create-cluster --name "Cluster's name" --ec2-attributes KeyName=SSH_KEY --instance-type m3.xlarge --release-label emr-5.2.1 --log-uri s3://mybucket/logs/ --enable-debugging --instance-count 1 --use-default-roles --applications Name=Spark --steps file://step.json
Where step.json looks like :
[
{
"Name": "Step #1",
"Type":"SPARK",
"Jar":"command-runner.jar",
"Args":
[
"--deploy-mode", "cluster",
"--class", "com.your.data.set.class",
"s3://path/to/your/spark-job.jar",
"-c", "s3://path/to/your/config/or/not",
"--aws-access-key", "ACCESS_KEY",
"--aws-secret-key", "SECRET_KEY"
],
"ActionOnFailure": "CANCEL_AND_WAIT"
}
]
(Multiple steps is okey too)
After that you can always startUp the same configured Cluster.
And for example Schedule the whole Cluster and steps from one AirFlow job.
But if you really want to use Boto3, i suppose that the describe_cluster() method can help you to get the whole informations and use the returned object to Fire Up a new one.
There is no way to get "emr export cli" through command line. You should parse the parameter what you want to clone, through "describe-cluster".
See below sample, https://github.com/awslabs/aws-support-tools/tree/master/EMR/Get_EMR_CLI_Export
import boto3
import json
import sys
cluster_id = sys.argv[1]
client = boto3.client('emr')
clst = client.describe_cluster(ClusterId=cluster_id)
...
awscli += ' --steps ' + '\'' + json.dumps(cli_steps) + '\''
...
awscli += ' --instance-groups ' + '\'' + json.dumps(cli_igroups) + '\''
print(awscli)
It works parsing the parameters from “describe-cluster” at first, and make strings to fit “create-cluster” of aws-cli.