Memory copies seem to be surprisingly slow on my EC2 machine. I'd like to track down why.
On most computers I observe bulk memory copy bandwidths of 3000 MB/s. See the following Python snippet:
In [1]: data = b'0' * int(1e8) # 100 MB
In [2]: %time len(data[1:]) # memcopy
CPU times: user 18.4 ms, sys: 8.62 ms, total: 27.1 ms
Wall time: 27.1 ms
Out[2]: 99999999
In [3]: 100000000 / 0.027 / 1e6 # MB/s
Out[3]: 3703.703703703704
However, on EC2, this changes significantly
In [1]: data = b'0' * int(1e8) # 100 MB
In [2]: %time len(data[1:]) # memcopy
CPU times: user 44 ms, sys: 148 ms, total: 192 ms
Wall time: 192 ms
Out[2]: 99999999
In [3]: 100000000 / 0.192 / 1e6 # MB/s
Out[3]: 520.8333333333333
I have never seen memcpy run that slowly before. What is the likely cause?
Setup
Software environments are mostly the same. I'm running Ubuntu 14.04 and Anaconda with Python 2.7 on each machine. Hardware is different
- Laptop (Thinkpad w540): Intel(R) Core(TM) i7-4700MQ CPU @ 2.40GHz
- EC2 Machine (m4.xlarge): Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz