The Analysis of the Tests Results
I will further analyze performance of built-in PHP hashes, in addition to the analysis made earlier here by Michael (see the above post), because this topic is quite interesting and has unexpected outcomes.
The results are not that obvious, or even surprising. A simple algorithm - CRC32 or CRC16, is slower than the complex one - MD5. It seems that modern CPUs do not like particular old algorithms and execute them very slowly, at least when the algorithms are not implemented the new way, taking advantage of modern CPU architecture. CRC16 CCITT algorithm was relatively fast and efficient in good old days when there were 300 BPS dial-up modems. Now there are modern algorithm specially designed for new hardware which may work much faster on the same hardware than the old algorithms, which are inherently unfit to the new hardware, and even if you try to optimize them, they will anyway be relatively slow. For example, for algorithms where each byte depends on previous byte and you cannot take benefits of Out-of-order execution or 64-bit registers and process many bits in parallel.
You may see from other cryptographic libraries that confirm what we see in PHP -
that CRC32 IEEE has almost the same throughout speed as MD5. Here is the link with the results of another library: https://www.cryptopp.com/benchmarks.html
OpenSSL shows similar results. At first glance, it may seem irrational, because the algorithm for CRC32 is much simpler and than for MD5, but the reality shows the opposite.
I just want to show how simple the CRC32 function is.
Here is a code that updates CRCR32 counter with the next incoming byte (Delphi):
// Returns an updated CRC32
function UpdateCrc32(CurByte: Byte; CurCrc: Cardinal): Cardinal; inline;
begin
UpdateCrc32 := Crc32Table[Byte(CurCrc xor CurByte)] xor (CurCrc shr 8);
end;
Here is this code on Assembly:
@calc_crc32:
xor dl,[esi]
mov al,dl
shr edx,8
xor edx,dword ptr [edi+eax*4]
inc esi
loop @calc_crc32
You may also unroll this code, so you will get just 5 CPU instructions per byte:
xor dl,bl
shr rbx,8
mov al,dl
shr edx,8
xor edx,dword ptr [r8+rax*4]
You just need to load the rbx
register with next 8 bytes of data and then repeat this code 8 times until you need to load next 8 bytes to the rbx
64-bit register.
Here is the routine that calculates CRC32 of the whole string:
function CalcCRC32(const B; Size: NativeUINT;
const
InitialValue: Cardinal = CRC32_INIT): Cardinal;
var
C: Cardinal;
P: PAnsiChar;
i: NativeUINT;
begin
C := InitialValue;
if Size > 0 then
begin
P := @B;
for i := 0 to Size - 1 do
C := UpdateCrc32(Byte(P[i]), C);
end;
Result := C;
end;
And here is how it is compiled into machine code by Delphi – not very optimal, but quite simple – just 11 assembly instructions for each byte, which, surprisingly, on Intel Core i5-6600 work little bit faster than the above assembler code even after loop unrollment. As you see, and all these instructions to implement the CRC32 IEEE are straightforward, with no loops or comparisons, there is just one comparison at the end of each byte. This is just a debugger output of a compiled Delphi code, not an assembly code written by a human.
CRC32.pas.78: begin
push esi
push edi
CRC32.pas.80: if Size > 0 then
test edx,edx
jbe $00500601
CRC32.pas.82: P := @B;
mov edi,eax
CRC32.pas.83: for i := 0 to Size - 1 do
mov eax,edx
dec eax
test eax,eax
jb $00500601
inc eax
xor esi,esi
CRC32.pas.84: C := UpdateCrc32(Byte(P[i]), C);
movzx edx,[edi+esi]
xor dl,cl
movzx edx,dl
mov edx,[edx*4+$517dec]
shr ecx,$08
xor edx,ecx
mov ecx,edx
inc esi
CRC32.pas.83: for i := 0 to Size - 1 do
dec eax
jnz $005005e6
CRC32.pas.86: Result := C;
mov eax,ecx
CRC32.pas.87: end;
pop edi
pop esi
ret
Here is another variant of assembler code for CRC32, just 5 processor commands for each bytes, not 11, but it is essentially the same as the above assembler code, just uses different registers and avoids the "loop" command which is again on i5-6600 faster than the two different instructions. You can find the whole code at CRC32 assembler function called from C console app
586
.model flat, stdcall
.xmm
.data
.code
CRC32 proc sizeOfFile:DWORD, file:DWORD
push esi
push ecx
push edx
mov esi, file
xor edx, edx
or eax, -1
mov ecx, sizeOfFile
CRC32_loop:
mov dl, byte ptr [esi]
xor dl, al
shr eax, 8
xor eax, dword ptr [crc32_table + 4*edx]
inc esi
dec ecx
jnz CRC32_loop
not eax
pop edx
pop ecx
pop esi
ret
Now compare it with MD5, with this highly-optimized assembler code by Peter Sawatzki:
; MD5_386.Asm - 386 optimized helper routine for calculating
; MD Message-Digest values
; written 2/2/94 by
;
; Peter Sawatzki
; Buchenhof 3
; D58091 Hagen, Germany Fed Rep
;
; EMail: [email protected]
; EMail: [email protected]
; WWW: http://www.sawatzki.de
;
;
; original C Source was found in Dr. Dobbs Journal Sep 91
; MD5 algorithm from RSA Data Security, Inc.
.386
.MODEL FLAT
.CODE
R1 = ESi
R2 = EDi
FF Macro a,b,c,d,x,s,ac
; a:= ROL (a+x+ac + (b And c Or Not b And d), s) + b
Add a, [EBp+(4*x)]
Add a, ac
Mov R1, b
Not R1
And R1, d
Mov R2, c
And R2, b
Or R1, R2
Add a, R1
Rol a, s
Add a, b
EndM
GG Macro a,b,c,d,x,s,ac
; a:= ROL (a+x+ac + (b And d Or c And Not d), s) + b
Add a, [EBp+(4*x)]
Add a, ac
Mov R1, d
Not R1
And R1, c
Mov R2, d
And R2, b
Or R1, R2
Add a, R1
Rol a, s
Add a, b
EndM
HH Macro a,b,c,d,x,s,ac
; a:= ROL (a+x+ac + (b Xor c Xor d), s) + b
Add a, [EBp+(4*x)]
Add a, ac
Mov R1, d
Xor R1, c
Xor R1, b
Add a, R1
Rol a, s
Add a, b
EndM
II Macro a,b,c,d,x,s,ac
; a:= ROL (a+x+ac + (c Xor (b Or Not d)), s) + b
Add a, [EBp+(4*x)]
Add a, ac
Mov R1, d
Not R1
Or R1, b
Xor R1, c
Add a, R1
Rol a, s
Add a, b
EndM
Transform Proc
Public Transform
;Procedure Transform (Var Accu; Const Buf); Register;
; save registers that Delphi requires to be restored
Push EBx
Push ESi
Push EDi
Push EBp
Mov EBp, EDx ; Buf -> EBp
Push EAx ; Accu -> Stack
Mov EDx, [EAx+12]
Mov ECx, [EAx+8]
Mov EBx, [EAx+4]
Mov EAx, [EAx]
FF EAx,EBx,ECx,EDx, 0, 7, 0d76aa478h ; 1
FF EDx,EAx,EBx,ECx, 1, 12, 0e8c7b756h ; 2
FF ECx,EDx,EAx,EBx, 2, 17, 0242070dbh ; 3
FF EBx,ECx,EDx,EAx, 3, 22, 0c1bdceeeh ; 4
FF EAx,EBx,ECx,EDx, 4, 7, 0f57c0fafh ; 5
FF EDx,EAx,EBx,ECx, 5, 12, 04787c62ah ; 6
FF ECx,EDx,EAx,EBx, 6, 17, 0a8304613h ; 7
FF EBx,ECx,EDx,EAx, 7, 22, 0fd469501h ; 8
FF EAx,EBx,ECx,EDx, 8, 7, 0698098d8h ; 9
FF EDx,EAx,EBx,ECx, 9, 12, 08b44f7afh ; 10
FF ECx,EDx,EAx,EBx, 10, 17, 0ffff5bb1h ; 11
FF EBx,ECx,EDx,EAx, 11, 22, 0895cd7beh ; 12
FF EAx,EBx,ECx,EDx, 12, 7, 06b901122h ; 13
FF EDx,EAx,EBx,ECx, 13, 12, 0fd987193h ; 14
FF ECx,EDx,EAx,EBx, 14, 17, 0a679438eh ; 15
FF EBx,ECx,EDx,EAx, 15, 22, 049b40821h ; 16
GG EAx,EBx,ECx,EDx, 1, 5, 0f61e2562h ; 17
GG EDx,EAx,EBx,ECx, 6, 9, 0c040b340h ; 18
GG ECx,EDx,EAx,EBx, 11, 14, 0265e5a51h ; 19
GG EBx,ECx,EDx,EAx, 0, 20, 0e9b6c7aah ; 20
GG EAx,EBx,ECx,EDx, 5, 5, 0d62f105dh ; 21
GG EDx,EAx,EBx,ECx, 10, 9, 002441453h ; 22
GG ECx,EDx,EAx,EBx, 15, 14, 0d8a1e681h ; 23
GG EBx,ECx,EDx,EAx, 4, 20, 0e7d3fbc8h ; 24
GG EAx,EBx,ECx,EDx, 9, 5, 021e1cde6h ; 25
GG EDx,EAx,EBx,ECx, 14, 9, 0c33707d6h ; 26
GG ECx,EDx,EAx,EBx, 3, 14, 0f4d50d87h ; 27
GG EBx,ECx,EDx,EAx, 8, 20, 0455a14edh ; 28
GG EAx,EBx,ECx,EDx, 13, 5, 0a9e3e905h ; 29
GG EDx,EAx,EBx,ECx, 2, 9, 0fcefa3f8h ; 30
GG ECx,EDx,EAx,EBx, 7, 14, 0676f02d9h ; 31
GG EBx,ECx,EDx,EAx, 12, 20, 08d2a4c8ah ; 32
HH EAx,EBx,ECx,EDx, 5, 4, 0fffa3942h ; 33
HH EDx,EAx,EBx,ECx, 8, 11, 08771f681h ; 34
HH ECx,EDx,EAx,EBx, 11, 16, 06d9d6122h ; 35
HH EBx,ECx,EDx,EAx, 14, 23, 0fde5380ch ; 36
HH EAx,EBx,ECx,EDx, 1, 4, 0a4beea44h ; 37
HH EDx,EAx,EBx,ECx, 4, 11, 04bdecfa9h ; 38
HH ECx,EDx,EAx,EBx, 7, 16, 0f6bb4b60h ; 39
HH EBx,ECx,EDx,EAx, 10, 23, 0bebfbc70h ; 40
HH EAx,EBx,ECx,EDx, 13, 4, 0289b7ec6h ; 41
HH EDx,EAx,EBx,ECx, 0, 11, 0eaa127fah ; 42
HH ECx,EDx,EAx,EBx, 3, 16, 0d4ef3085h ; 43
HH EBx,ECx,EDx,EAx, 6, 23, 004881d05h ; 44
HH EAx,EBx,ECx,EDx, 9, 4, 0d9d4d039h ; 45
HH EDx,EAx,EBx,ECx, 12, 11, 0e6db99e5h ; 46
HH ECx,EDx,EAx,EBx, 15, 16, 01fa27cf8h ; 47
HH EBx,ECx,EDx,EAx, 2, 23, 0c4ac5665h ; 48
II EAx,EBx,ECx,EDx, 0, 6, 0f4292244h ; 49
II EDx,EAx,EBx,ECx, 7, 10, 0432aff97h ; 50
II ECx,EDx,EAx,EBx, 14, 15, 0ab9423a7h ; 51
II EBx,ECx,EDx,EAx, 5, 21, 0fc93a039h ; 52
II EAx,EBx,ECx,EDx, 12, 6, 0655b59c3h ; 53
II EDx,EAx,EBx,ECx, 3, 10, 08f0ccc92h ; 54
II ECx,EDx,EAx,EBx, 10, 15, 0ffeff47dh ; 55
II EBx,ECx,EDx,EAx, 1, 21, 085845dd1h ; 56
II EAx,EBx,ECx,EDx, 8, 6, 06fa87e4fh ; 57
II EDx,EAx,EBx,ECx, 15, 10, 0fe2ce6e0h ; 58
II ECx,EDx,EAx,EBx, 6, 15, 0a3014314h ; 59
II EBx,ECx,EDx,EAx, 13, 21, 04e0811a1h ; 60
II EAx,EBx,ECx,EDx, 4, 6, 0f7537e82h ; 61
II EDx,EAx,EBx,ECx, 11, 10, 0bd3af235h ; 62
II ECx,EDx,EAx,EBx, 2, 15, 02ad7d2bbh ; 63
II EBx,ECx,EDx,EAx, 9, 21, 0eb86d391h ; 64
Pop ESi ; get Accu from stack
Add [ESi], EAx
Add [ESi+4], EBx
Add [ESi+8], ECx
Add [ESi+12], EDx
; restore registers for Delphi
Pop EBp
Pop EDi
Pop ESi
Pop EBx
Ret
Transform EndP
End
You can find 32-bit and 64-bit versions of this code at https://github.com/maximmasiutin/MD5_Transform-x64
The performance of this code under IA-32 or x86-64 is 4.94 CPU cycles per byte (on Skylake) of data to calculate MD5.
The above code processes at one call 64 bytes of incoming data. It is called from the main routine that does preparation steps:
procedure CiphersMD5Update(var Context: TMD5Ctx; const ChkBuf; len: UInt32);
var
BufPtr: ^Byte;
Left: UInt32;
begin
If Context.Count[0] + UInt32(len) shl 3 < Context.Count[0] then
Inc(Context.Count[1]);
Inc(Context.Count[0], UInt32(len) shl 3);
Inc(Context.Count[1], UInt32(len) shr 29);
BufPtr := @ChkBuf;
if Context.BLen > 0 then
begin
Left := 64 - Context.BLen;
if Left > len then
Left := len;
Move(BufPtr^, Context.Buffer[Context.BLen], Left);
Inc(Context.BLen, Left);
Inc(BufPtr, Left);
If Context.BLen < 64 then
Exit;
Transform(Context.State, @Context.Buffer);
Context.BLen := 0;
Dec(len, Left)
end;
while len >= 64 do
begin
Transform(Context.State, BufPtr);
Inc(BufPtr, 64);
Dec(len, 64)
end;
if len > 0 then
begin
Context.BLen := len;
Move(BufPtr^, Context.Buffer[0], Context.BLen)
end
end;
And if your processor supports CRC32 opcodes (SSE 4.2), you can calculate checksums 10 times faster with this code:
function crc32csse42(crc: cardinal; buf: Pointer; len: NativeUInt): cardinal;
asm // ecx=crc, rdx=buf, r8=len
.NOFRAME
mov eax,ecx
not eax
test r8,r8; jz @0
test rdx,rdx; jz @0
@7: test rdx,7; jz @8 // align to 8 bytes boundary
crc32 eax,byte ptr [rdx]
inc rdx
dec r8; jz @0
test rdx,7; jnz @7
@8: mov rcx,r8
shr r8,3
jz @2
@1: crc32 eax,qword ptr [rdx] // calculate CRC of 8 bytes, aligned
dec r8
lea rdx,rdx+8
jnz @1
@2: // less than 8 bytes remaining
and rcx,7; jz @0
cmp rcx,4; jb @4
crc32 eax,dword ptr [rdx] // calculate CRC of 4 bytes
sub rcx,4
lea rdx,rdx+4
jz @0
@4: // less than 4 bytes remaining
crc32 eax,byte ptr [rdx]
dec rcx; jz @0
crc32 eax,byte ptr [rdx+1]
dec rcx; jz @0
crc32 eax,byte ptr [rdx+2]
@0: not eax
end;
Please note that in my example I'm using a buffer of just 5KB, to fit in the processor's cache and to exclude the effect of slower RAM on the speed of the digest calculation.
There are fast implementations of CRC32 algorighm for modern processors that does not use CRC32 opcodes, but take benefit of Out-of-order execution, including speculative execution via register renaming. An example of such implementation is CRC32 Slicing-By-8. IA-32 or x86-64 assembler code has 1,20 CPU clock cycles (on Skylake) per byte of data. You can find such implementation at https://github.com/maximmasiutin/CRC32-Slicing-x64-Asm-Pas
In PHP, even in version 7, there seem to be no support for hardware acceleration of CRC32, albeit these instructions are supported on Intel and AMD processors since ages. Intel support CRC32 since November 2008 (Nehalem (microarchitecture)) and AMD seems to support it since 2013.
My Own Tests that Confirm Michael's Results
I have tested various PHP hash functions on different configurations: (1) AMD FX-8320 (released in 2012) under Ubuntu with PHP 5, and (2) Intel Core i5-6600 released in 2015 under Windows with PHP 7. I have also run OpenSSL test on this Intel Core i5-6600. Besides that, I run tests of the cryptographic routines that we use in our software “The Bat!” written in Delphi. Although the main software is written in Delphi, the cryptographic routines that we use are written on Assembler for Intel processor (32-bit or 64-bit) or in C.
I have found out that the our Delphi code shows very big speed differences between various hash functions and data sizes. This is in contrast to PHP, where to a certain degree and with rare exceptions all hash functions from simplest CRC32 to once-cryptographically-strong MD5 have almost the same throughoutput speed.
So, here are the measurements that I have done on AMD FX-8320, PHP5, Ubuntu. I made two test cases. First, I ran 5000 iterations to hash a message consisting of just 5 bytes. By this small message size I intended to test the duration of initialization/finalization steps of various algorithms, and how it affects overall performance. For some algorithms, like CRC32, three are virtually no finalization steps - the digest is always ready, after each byte. Cryptographically strong functions like SHA1 or MD5 or other have a finalization step that compresses larger context to a smaller final digest. Second, I run 5000 iterations to hash a message 5000 bytes long. Both of the messages were filled in advance with pseudo-random bytes (they were not re-filled after each iteration; they were only filled once, when the program started).
Results of my PHP Hash Speed Test
I have modified your PHP code and made it now work for both PHP5 and PHP7, where are different functions to generate random data in different versions of PHP. I have just measured the time that was needed to hash 5000 iterations of 5-bytes messages and then 5000 iterations of 5000-bytes message. Here are the results:
Legend:
(1) 5b x 5000, AMD FX-8320, PHP5
(2) 5000b x 5000, AMD FX-8320, PHP5
PHP hash (1) (2)
-------- ------------ ------------
md2 0.021267 sec 2.602651 sec
md4 0.002684 sec 0.035243 sec
md5 0.002570 sec 0.055548 sec
sha1 0.003346 sec 0.106432 sec
sha224 0.004945 sec 0.210954 sec
sha256 0.004735 sec 0.238030 sec
sha384 0.005848 sec 0.144015 sec
sha512 0.006085 sec 0.142884 sec
ripemd128 0.003385 sec 0.120959 sec
ripemd160 0.004164 sec 0.174045 sec
ripemd256 0.003487 sec 0.121477 sec
ripemd320 0.004206 sec 0.177473 sec
whirlpool 0.009713 sec 0.509682 sec
tiger128,3 0.003414 sec 0.059028 sec
tiger160,3 0.004354 sec 0.059335 sec
tiger192,3 0.003379 sec 0.058891 sec
tiger128,4 0.003514 sec 0.073468 sec
tiger160,4 0.003602 sec 0.072329 sec
tiger192,4 0.003507 sec 0.071856 sec
snefru 0.022101 sec 1.190888 sec
snefru256 0.021972 sec 1.217704 sec
gost 0.013961 sec 0.653600 sec
adler32 0.001459 sec 0.038849 sec
crc32 0.001429 sec 0.068742 sec
crc32b 0.001553 sec 0.063308 sec
fnv132 0.001431 sec 0.038256 sec
fnv164 0.001586 sec 0.060622 sec
joaat 0.001569 sec 0.062947 sec
haval128,3 0.006747 sec 0.174759 sec
haval160,3 0.005810 sec 0.166154 sec
haval192,3 0.006129 sec 0.168382 sec
haval224,3 0.005918 sec 0.166792 sec
haval256,3 0.006119 sec 0.173360 sec
haval128,4 0.007364 sec 0.233829 sec
haval160,4 0.007917 sec 0.240273 sec
haval192,4 0.007676 sec 0.245864 sec
haval224,4 0.007580 sec 0.245249 sec
haval256,4 0.007442 sec 0.241091 sec
haval128,5 0.008651 sec 0.281248 sec
haval160,5 0.009304 sec 0.278619 sec
haval192,5 0.008972 sec 0.281235 sec
haval224,5 0.008917 sec 0.274923 sec
haval256,5 0.008853 sec 0.282171 sec
I then run the same PHP script under Intel Core i5-6600, with 64-bit version of PHP7 under Windows 10. Here are the results:
Legend:
(1) 5b x 5000, Intel Core i5-6600, PHP7
(2) 5000b x 5000, Intel Core i5-6600, PHP7
PHP hash (1) (2)
--------- ------------ ------------
md2 0.016131 sec 2.308100 sec
md4 0.001218 sec 0.040803 sec
md5 0.001284 sec 0.046208 sec
sha1 0.001499 sec 0.050259 sec
sha224 0.002683 sec 0.120510 sec
sha256 0.002297 sec 0.119602 sec
sha384 0.002792 sec 0.080670 sec
ripemd128 0.001984 sec 0.094280 sec
ripemd160 0.002514 sec 0.128295 sec
ripemd256 0.002015 sec 0.093887 sec
ripemd320 0.002748 sec 0.128955 sec
whirlpool 0.003402 sec 0.271102 sec
tiger128,3 0.001282 sec 0.038638 sec
tiger160,3 0.001305 sec 0.037155 sec
tiger192,3 0.001309 sec 0.037684 sec
tiger128,4 0.001618 sec 0.050690 sec
tiger160,4 0.001571 sec 0.049656 sec
tiger192,4 0.001711 sec 0.050682 sec
snefru 0.010949 sec 0.865108 sec
snefru256 0.011587 sec 0.867685 sec
gost 0.008968 sec 0.449647 sec
adler32 0.000588 sec 0.014345 sec
crc32 0.000609 sec 0.079202 sec
crc32b 0.000636 sec 0.074408 sec
fnv132 0.000570 sec 0.028157 sec
fnv164 0.000566 sec 0.028776 sec
joaat 0.000623 sec 0.042127 sec
haval128,3 0.002972 sec 0.084010 sec
haval160,3 0.002968 sec 0.083213 sec
haval192,3 0.002943 sec 0.082217 sec
haval224,3 0.002798 sec 0.084726 sec
haval256,3 0.002995 sec 0.082568 sec
haval128,4 0.003659 sec 0.112680 sec
haval160,4 0.003858 sec 0.111462 sec
haval192,4 0.003526 sec 0.112510 sec
haval224,4 0.003671 sec 0.111656 sec
haval256,4 0.003636 sec 0.111236 sec
haval128,5 0.004488 sec 0.140130 sec
haval160,5 0.005095 sec 0.137777 sec
haval192,5 0.004117 sec 0.140711 sec
haval224,5 0.004311 sec 0.139564 sec
haval256,5 0.004382 sec 0.138345 sec
As you see, to calculate CRC32 of messages in PHP, in almost all of my tests, it takes just about half of the time it takes to calculate MD5 of the same messages. The only exception was that on the test of 5000 messages of 5000 bytes on Intel Core i5-6600 with PHP7, CRC32 even took longer than MD5(!). This strange result was always repeatable on my case. I could not find a plausible explanation for it.
Also, on PHP, there was almost no noticeable speed difference between MD5 and SHA1, except on Ubuntu with PHP5, where on the test of 5000 messages of 5000 bytes, MD5 was twice faster.
Results of my OpenSSL's Tests of Hash Performance
Here are the tests of OpenSSL on the Intel i5-660. They show data differently. They don’t show how much time it took do digest certain set of data, but vice versa: they show the volume of data that OpenSSL have managed to hash in 3 seconds. So, higher value is better:
Legend:
(1) OpenSSL 1.1.0 on Intel Core i5-6600, number of 16-bytes messages processed in 3 seconds
(2) OpenSSL 1.1.0 on Intel Core i5-6600, number of 8192-bytes messages processed in 3 seconds
Algorighm (1) (2)
--------- --------- ----------
md4 50390.16k 817875.48k
md5 115875.35k 680700.59k
sha1 118158.30k 995986.09k
ripemd160 30308.79k 213224.11k
whirlpool 39605.02k 182072.66k
Again, there is almost no differences between md5 and sha1 which is strange
and requires further investigation whether the MD5 and SHA-1 algorithm are inherently the same in terms of time consumption.
Results of our Delphi Hash Functions Performance
Here are the results of our Delphi library on Intel Core i5-6600 under Windows 10 64-bit, the code tested was a 32-bit Win32 application.
Legend:
(1) Delphi, 5b x 5000 iterations
(2) Delphi, 5000b x 5000 iterations
Algorighm (1) (2)
--------------- -------------- --------------
md2 0.0381010 secs 5.8495807 secs
md5 0.0005015 secs 0.0376252 secs
sha1 0.0050118 secs 0.1830871 secs
crc32 >0.0000001 secs 0.0581535 secs
crc32c (intel hw) >0.0000001 secs 0.0055349 secs
As you see, MD2 is also much much shower than the other hashes – the same outcome as with the PHP code, but MD5 is much faster than SHA-1, and overall it took less time in Delphi to do the same on the same machine as PHP
For example, PHP7 took 0.001284 sec to digest 5000 5-byte messages with MD5, 0.001499 sec with SHA1. As about 5000 bytes message – it took PHP7 0.046208 sec with MD5 and 0.050259 sec with SHA-1.
As about Delphi, it took 0.0005015 sec to digest 5000 5-byte messages with MD5 and 0.0050118 secs with SHA1. As about 5000 bytes message – it took Delphi 0.0376252 secs sec with MD5 and 0.1830871 secs with SHA-1. As you see, MD5 works much faster in Delphi, but SHA-1 is about the same. Also, Delphi is about 10 times faster on 5-bytes message, but as about 5000-bytes messages, it is about the same or even slower with SHA-1.
But when it comes to CRC32 and CRC32C, Delphi is unbeatable, from 10 to 1000 time faster than PHP.
Conclusion
PHP is inherently very slow on loops and small operations. So, it doesn’t matter on PHP which hash function you call if you need to calculate a hash of a tiny message. But if you need to digest a large message, the differences in algorithm speed began to show them: for example, MD2 performance is about ten times worse than for MD5. Anyway, there is absolutely no reason today to use MD2. There are no advantages for a PHP user of MD2 over MD5. As about MD5, being initially designed as a cryptographic hash function and then used by PGP in RFC-1991, now can no longer be used in Cryptography, but can be used as a checksum in trusted environments, for example for ETags or other means. This function is very fast when implemented correctly (and on PHP, it is not slow at least), and MD5 produces a very compact digest comparing to other functions. Here is my PHP code that made these benchmarks. I have made it based on the original code sample by Michael (see above).
<?
define (TRAILING_ZEROS, 6);
$strlens = array(5, 30, 90, 1000, 5000);
$hashes = hash_algos();
function generate_bytes($len)
{
if (function_exists('random_bytes')) {$fn='random_bytes';$str = random_bytes($len);} else // for php 5
if (function_exists('openssl_random_pseudo_bytes')) {$fn='openssl_random_pseudo_bytes';$str = openssl_random_pseudo_bytes($strlen);} else // for php 7
{
flush();
ob_start () ;
phpinfo () ;
$str = str_pad(substr(ob_get_contents (), 0, $len), $len) ;
ob_end_clean () ;
$fn = 'phpinfo';
}
return array(0=>$str, 1=>$fn);
}
foreach ($strlens as $strlen)
{
$loops = 5000;
echo "<h1>$loops iterations on $strlen bytes message</h1>".PHP_EOL;
echo '<p>';
$r = generate_bytes($strlen);
$str = $r[0];
$gotlen = strlen($str);
while ($gotlen < $strlen)
{
// for some uncodumented reason, the openssl_random_pseudo_bytes returned less bytes than needed
$left = $strlen-$gotlen;
echo "The ".$r[1]."() function returned $left byes less, trying again to get these remaining bytes only<br>";
$r = generate_bytes($left);
$str.= $r[0];
$gotlen = strlen($str);
};
echo "Got the whole string of ".strlen($str)." bytes!";
echo '</p>';
echo PHP_EOL;
echo "<pre>";
foreach ($hashes as $hash)
{
$tss = microtime(true);
for($i=0; $i<$loops; $i++)
{
$x = hash($hash, $str, true);
}
$tse = microtime(true);
echo "\n".str_pad($hash, 15, ' ')."\t" . str_pad(round($tse-$tss, TRAILING_ZEROS), TRAILING_ZEROS+2, '0') . " sec \t" . bin2hex($x);
}
echo PHP_EOL."</pre>".PHP_EOL;
flush();
}
?>
Continuation of the post...
c/2^N
, where c is your number of hashes (files) and N is your number of bits (e.g. 128 for MD5). I'd just use MD5 as it is one of the fastest widely available algorithms and is good as long as you do not need security. Note that you will always need to handle collisions (even if you leave them to the user) as they can happen with every hash function. CRC32 is also widely available and fast but uses only 32 bits so a collision is a more realistic possibility.2^128
is indeed very large. Even if you have2^32+1
(~4,3 billion) files the chance is still marginal (1/2^100
) while a CRC32 will always have at least one collision.