5

Running sysbench against volumes I'm finding the gp3 volumes are much slower. Even when I provision a volume with 16000IOPS after waiting for optimization I'm getting a cap at 1000 IOPS when monitoring through Percona PMM, New Relic CloudWatch.

Instance:

  • C5.4xlarge Ubuntu 18.04

Volumes:

  • gp2 3000GB (gives 9000 IOPS)
  • gp3 3000GB 9000 IOPS 250/s

Sysbench results below:

sysbench --file-total-size=15G --file-num=16 fileio prepare
sysbench --file-total-size=15G --file-num=16 --file-test-mode=rndrw --time=600 fileio run
sysbench --file-total-size=15G --file-num=16 fileio cleanup

gp3 | 9000 | 3000 | 250/s

File operations:
reads/s: 576.37
writes/s: 384.24
fsyncs/s: 153.70

Throughput:
read, MiB/s: 9.01
written, MiB/s: 6.00

General statistics:
total time: 600.0333s
total number of events: 668612

Latency (ms):
min: 0.00
avg: 0.90
max: 337.40
95th percentile: 3.89
sum: 599693.33

Threads fairness:
events (avg/stddev): 668612.0000/0.00
execution time (avg/stddev): 599.6933/0.00

gp2 | 9000 | 3000 | gp2

File operations:
reads/s: 1523.68
writes/s: 1015.79
fsyncs/s: 406.33

Throughput:
read, MiB/s: 23.81
written, MiB/s: 15.87

General statistics:
total time: 600.0064s
total number of events: 1767487

Latency (ms):
min: 0.00
avg: 0.34
max: 70.10
95th percentile: 1.06
sum: 599390.12

Threads fairness:
events (avg/stddev): 1767487.0000/0.00
execution time (avg/stddev): 599.3901/0.00

Percona PMM gp3 > gp2 comparison:

Percona PMM gp3 > gp2 comparison

3
  • You might get a better response at: reddit.com/r/aws Commented Jan 10, 2021 at 9:21
  • I'm doing benchmarks for gp3 volumes as well. However, I'm observing the opposite: gp3 with less latency than gp2. In which region are you operating?
    – Andreas
    Commented Jan 11, 2021 at 12:49
  • Amazon EBS volumes take slightly longer when using a block for the first time, due to the allocation of storage space. Before performing performance tests, it is recommended to zero-out the contents of the disk (eg with dd). This will then provide consistent performance. See: Initializing Amazon EBS volumes - Amazon Elastic Compute Cloud Commented Jan 11, 2021 at 23:04

1 Answer 1

2

My initial enthusiasm for gp3 was dampened by inferior performance to gp2, however, when I set the size, IOPS, and bandwidth the same on C5, M5, and M5A instance type I got similar performance measurements from fio: https://github.com/axboe/fio

I'm testing on CentOS 7.8, fio-3.7-2.el7.x86_64. Region us-east-1.

Are you sure sysbench is the right benchmarking tool? The fileio options don't seem greatly documented, and I can get wildly different results when changing the sync mode, flags, etc.

Your example command equivalent to this because of the default values:

sysbench --file-total-size=15G --file-num=16 --file-test-mode=rndrw --file-io-mode=sync --file-rw-ratio=1.5 --file-fsync-freq=100 --max-requests=0 --time=600 fileio run

When I run a different sysbench command, notably using direct access and fsync=off:

sysbench fileio --file-total-size=15G --file-test-mode=rndrw --time=600 --max-requests=0 --file-io-mode=async --file-extra-flags=direct  --file-num=16 --file-rw-ratio=1 --file-fsync-freq=0 run

I get equivalent performance from gp2 and gp3. (Actually, gp3 is slightly better).

Here's a summary of my fio data, using the command

fio --rw=TEST --direct=1 --ioengine=libaio --bs=16k --numjobs=8 --size=1G --group_reporting

with TEST={read,write,randread,randwrite,randrw}

gp2:

gp2-randread-16k.out:   read:   IOPS=3081,  BW=48.2MiB/s  (50.5MB/s)(8192MiB/170117msec)
gp2-randrw-16k.out:     read:   IOPS=1540,  BW=24.1MiB/s  (25.2MB/s)(4095MiB/170121msec)
gp2-randrw-16k.out:     write:  IOPS=1541,  BW=24.1MiB/s  (25.2MB/s)(4097MiB/170121msec)
gp2-randwrite-16k.out:  write:  IOPS=3081,  BW=48.2MiB/s  (50.5MB/s)(8192MiB/170116msec)
gp2-read-16k.out:       read:   IOPS=3090,  BW=48.3MiB/s  (50.6MB/s)(8192MiB/169642msec)
gp2-write-16k.out:      write:  IOPS=3081,  BW=48.2MiB/s  (50.5MB/s)(8192MiB/170115msec)

gp3:

gp3-randread-16k.out:   read:   IOPS=3080,  BW=48.1MiB/s  (50.5MB/s)(8192MiB/170188msec)
gp3-randrw-16k.out:     read:   IOPS=1539,  BW=24.1MiB/s  (25.2MB/s)(4095MiB/170238msec)
gp3-randrw-16k.out:     write:  IOPS=1540,  BW=24.1MiB/s  (25.2MB/s)(4097MiB/170238msec)
gp3-randwrite-16k.out:  write:  IOPS=3082,  BW=48.2MiB/s  (50.5MB/s)(8192MiB/170073msec)
gp3-read-16k.out:       read:   IOPS=3089,  BW=48.3MiB/s  (50.6MB/s)(8192MiB/169719msec)
gp3-write-16k.out:      write:  IOPS=3083,  BW=48.2MiB/s  (50.5MB/s)(8192MiB/170053msec)
2
  • Adding to my confusion, tests with ioping consistently show gp3 to have twice the average latency as gp2. I don't see any difference in latency when testing with fio, nor any other metric (IOPS, throughput).
    – Matt L
    Commented Jan 12, 2021 at 21:32
  • Hi Matt, Yep, I did equivalent tests with fio and got more consistent results, though the GP3s with 16000IOPS didn't perform that much better than the GP2s with 9000IOPS. I originally used sysbench since these are MySQL servers and I like to run checks with sysbench to get ideas of MySQL. In the end I think they are a suitable replacement but do not offer the performance boost stated. NOTE: AWS support said they are looking into inconsistencies with latency on GP3s, so hopefully that get's fixed. Commented Jan 15, 2021 at 23:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.