3

When I try to create an EC2 Resource with a AWS Data Pipeline, I don't see and option for defining EBS volume that will be associated with that compute engine. Is it possible to set the volume size? If yes, can someone give me an example.

2 Answers 2

5

No, currently you cannot specify EBS volumes in Data-Pipelines Ec2Resource object.

Often times ROOT volume is used as staging directory for most of the Data-Pipeline activities which is currently limited to 8GB for Data-Pipeline provided default AMI's.

So, you can make your own AMI out of an Ec2 instance with increased EBS root volume and include that AMI in the Resource object(Image-id field) of Data-Pipelines.

Tip: You can check the AMI-id of Data-pipelines launched ec2 instances in Ec2 console.Use that AMI to create an ec2 instance with increased EBS ROOT volume and Make Image(AMI) from this instance with even more volume size. This way you dont need to choose from a list of AMI's and you will preserve visualization type that is required for launching particular instance types.

1

Currently, datapipeline service does not support an option to specify ebs volumes to attach to a ec2 resource. But, you can write a shell command activity (http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-shellcommandactivity.html) which can create ebs volume and attach it to the instance.

You can then run the shell command activity on the ec2 resource.

You can use aws cli for creating and attaching ebs volumes assuming that you have appropriate role on the launched ec2 instance.

you can also make use of ec2 metadata service: http://169.254.169.254/2009-04-04/meta-data/ to get instance info like availability zone etc... so that you can create ebs volume in the same availability zone

related post: https://serverfault.com/questions/126408/creating-ebs-volumn-from-and-attaching-to-the-same-instance

1
  • 1
    Adding another EBS volume with shellCommandActivity may increase the overall volume size but this volume will not be used for staging directory for activities. Only the Root volume will be used for Staging(which cannot be altered).
    – jc mannem
    Jan 13, 2016 at 18:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.