1

Trying to save money on EBS snapshots, so the idea is to take manual copies of the file systems (using dd) and storing manually in S3 to lifecycle to IA and Glacier.

The following works fine for smaller files (tested with 1GB), but on larger (~800GB), after around 40GB, everything slows to a crawl and never finishes

sudo dd if=/dev/sdb bs=64M status=progress | aws s3 cp - s3://my-bucket/sdb_backup.img --sse AES256 --storage-class STANDARD_IA

Running this from an m4.4xlarge instance (16 vcpu, 64GB RAM)

Not exactly sure why it's crawling to a halt, or whether this is the best way to solve this problem (manually storing file systems on s3 Infrequent Access storage class)

Any thoughts?

Thanks!!

3
  • Have you verified that it's not creating a big temp file somewhere? How about CPU usage of the aws process? Large uploads, in my experience, are not handled well by aws-cli. I wrote my own utility to do this, several years back, though I no longer do streaming uploads, because sc1 EBS volumes and ephemeral disks provide cost-effective temporary space for staging large uploads. Aug 12, 2017 at 0:51
  • 1
    Have you taken a look at your AWS bill, at the charges for snapshot storage... and compared how much they are billing you against the total size of all your snapshots? You may be chasing imaginary savings. In one region, I have almost 92,000 GB of snapshots (92 TB) but EBS only bills me for 10,300 GB (10.3 TB). Deduplication and compression provided by EBS shapshots means I am paying effectively $0.0056/GB... just over 1/10th the list price. In another region, the ratio is not as high, so it's about $0.0126/GB. You are not better off unless you're also deduping and compressing, yourself. Aug 12, 2017 at 1:20
  • Will double check this @Michael-sqlbot. Thanks!
    – maafk
    Aug 14, 2017 at 11:47

4 Answers 4

0

It is not a good idea because snapshots are incremental, so you'll spend more starting from the next few hand-made snapshots.

If you still want this way then consider multi-part upload (chunks up to 5GB).

0

You can use something like goofys to redirect output to S3. I've personally tested with files up to 1TB.

0

First consider the multipart upload for large sizes.

Second use the compressed version,

dd if=/dev/sdX | gzip -c | aws s3 cp - s3://bucket-name/desired_image_name.img
-1

If you wish to copy files to Amazon S3, the easiest method is to use the AWS Command-Line Interface (CLI):

aws s3 sync dir s3://my-bucket/dir

As an alternative to Standard-Infrequent Access, you could create a lifecycle policy on the S3 bucket to move files to Glacier. (This is worthwhile for long-term storage, but not for the short-term due to higher request charges.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.