6

When creating a new cluster using boto3, I want to use configuration from existing clusters (which is terminated) and thus clone it. As far as I know, emr_client.run_job_flow requires all the configuration(Instances, InstanceFleets etc) to be provided as parameters. Is there any way I can clone from existing cluster like I can do from aws console for EMR.

enter image description here

5
  • Possible Duplicate : stackoverflow.com/questions/43579746/…
    – Kush Vyas
    Oct 12, 2017 at 8:05
  • 1
    Please read both the questions.They are different. Oct 12, 2017 at 8:47
  • @AmeyDahale Have you got the answer? Seems that should be a popular function for boto3 to clone an existing EMR. Sep 5, 2018 at 4:19
  • I have kept a static config file with all EMR details and using boto3's run_job_flow to reuse the existing EMR's properties. /i guess it would be good to have boto3 function to clone existing EMR. Sep 6, 2018 at 5:45
  • I created this issue (albeit on the java sdk), since i see this is such a common request: github.com/aws/aws-sdk-java/issues/2002 May 23, 2019 at 16:59

2 Answers 2

1

What i can recommend you, is using the AWS CLI to fire your Cluster.
It permit to versioning your cluster configuration and you can easily load steps configuration with a json file.

aws create-cluster --name "Cluster's name" --ec2-attributes KeyName=SSH_KEY --instance-type m3.xlarge --release-label emr-5.2.1 --log-uri s3://mybucket/logs/ --enable-debugging --instance-count 1 --use-default-roles --applications Name=Spark --steps file://step.json

Where step.json looks like :

     [
      {
        "Name": "Step #1",
        "Type":"SPARK",
        "Jar":"command-runner.jar",
        "Args":
        [
          "--deploy-mode", "cluster",
          "--class", "com.your.data.set.class",
          "s3://path/to/your/spark-job.jar",
          "-c", "s3://path/to/your/config/or/not",
          "--aws-access-key", "ACCESS_KEY",
          "--aws-secret-key", "SECRET_KEY"
        ],
        "ActionOnFailure": "CANCEL_AND_WAIT"
      }
     ]

(Multiple steps is okey too)

After that you can always startUp the same configured Cluster.
And for example Schedule the whole Cluster and steps from one AirFlow job.

But if you really want to use Boto3, i suppose that the describe_cluster() method can help you to get the whole informations and use the returned object to Fire Up a new one.

0

There is no way to get "emr export cli" through command line. You should parse the parameter what you want to clone, through "describe-cluster".

See below sample, https://github.com/awslabs/aws-support-tools/tree/master/EMR/Get_EMR_CLI_Export

import boto3
import json
import sys

cluster_id = sys.argv[1]
client = boto3.client('emr')
clst = client.describe_cluster(ClusterId=cluster_id)

...

awscli += ' --steps ' + '\'' + json.dumps(cli_steps) + '\''

...

awscli += ' --instance-groups ' + '\'' + json.dumps(cli_igroups) + '\''
print(awscli)

It works parsing the parameters from “describe-cluster” at first, and make strings to fit “create-cluster” of aws-cli.

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.