A new set of Cloud Storage commands in Cloud SDK – gcloud storage
– includes a new approach to parallelization that can accelerate both small and large data migrations. While gsutil cp
remains effective, gcloud storage
avoids the tuning gsutil
requires for certain large transfer scenarios.
How gcloud storage works
Like gsutil
before it, gcloud storage
takes large files and breaks them down into pieces, so that transfers can best take advantage of the available bandwidth. What’s new in gcloud storage
is its parallelization strategy, which treats task management as a graph problem, where each unit of work is treated as a node, and each dependency as an edge. This strategy allows more work to be done in parallel with far less overhead.
Under the hood, gcloud storage
also benefits from a new hashing library that enables faster integrity checking. It can also adjust its own settings based on the workload and local machine size to optimize for performance.
Just how fast is it?
We measured the performance of gcloud storage
on the following environment:
- us-east4 for both VM and bucket
- n2d-standard-16 (8 vCPUs, 32 GB memory)
- 1x375GB Local SSD
When transferring 100 files that were 100MB in size, gcloud storage
was 79% faster than gsutil
on download and 33% faster on upload, with both using a composite upload strategy, where a file was broken up and uploaded as separate files, and then recombined into a single file.
When transferring a single large file, the difference is even more pronounced. With a 10GB file, gcloud storage
was 94% faster than gsutil
on download and 57% faster on upload. This performance improvement comes without the need for extensive testing and tweaking, making it easy to see much faster transfer times.
Try it for yourself
Once you update to the latest version of Cloud SDK, you can try out gcloud storage
by running the following commands:
gcloud alpha storage ls gs://pub
gcloud alpha storage cp -r gs://pub/shakespeare/ .
gcloud storage
supports the following features that you use today in gsutil
:
- Recursive uploads/downloads
- S3 support
- Composite uploads
- Sliced downloads
- Resumable uploads/downloads
- Listing objects/buckets
- Removing objects/buckets
It’s worth noting that gcloud storage
is in Preview. We will be adding more features over time to best reflect the capabilities of Cloud Storage.
gsutil
at some point or is it justgcloud storage
specific. I'd love to use it in our projects once it goes GA, but changing the commands everywhere is a pain.gcloud
utility? It's been a while. Regards, kmx@