250

I'm essentially preparing phrases to be put into the database, they may be malformed so I want to store a short hash of them instead (I will be simply comparing if they exist or not, so hash is ideal).

I assume MD5 is fairly slow on 100,000+ requests so I wanted to know what would be the best method to hash the phrases, maybe rolling out my own hash function or using hash('md4', '...' would be faster in the end?

I know MySQL has MD5(), so that would complement a bit of speed on the query end, but maybe there's further a faster hashing function in MySQL I don't know about that would work with PHP..

11
  • 6
    What's stopping you from benchmarking the hashes? Sep 8, 2010 at 6:50
  • 4
    NullUserException: You're right, I'll try them with random length phrases. Just wanted insight on what would be the norm if any to handle this sort of thing.
    – John
    Sep 8, 2010 at 6:55
  • 8
    MD5 isn't really that slow...
    – Amber
    Sep 8, 2010 at 6:55
  • 2
  • 54
    This is a very good question to ask, and the comments implying it isn't, or is unimportant, and/or should be obvious and/or intuitive -- are disappointing & frustrating. (And also not at all unexpected.)
    – michael
    Nov 12, 2017 at 8:10

13 Answers 13

190
fcn     time  generated hash
crc32:  0.03163  798740135
md5:    0.0731   0dbab6d0c841278d33be207f14eeab8b
sha1:   0.07331  417a9e5c9ac7c52e32727cfd25da99eca9339a80
xor:    0.65218  119
xor2:   0.29301  134217728
add:    0.57841  1105

And the code used to generate this is:

 $loops = 100000;
 $str = "ana are mere";

 echo "<pre>";

 $tss = microtime(true);
 for($i=0; $i<$loops; $i++){
  $x = crc32($str);
 }
 $tse = microtime(true);
 echo "\ncrc32: \t" . round($tse-$tss, 5) . " \t" . $x;

 $tss = microtime(true);
 for($i=0; $i<$loops; $i++){
  $x = md5($str);
 }
 $tse = microtime(true);
 echo "\nmd5: \t".round($tse-$tss, 5) . " \t" . $x;

 $tss = microtime(true);
 for($i=0; $i<$loops; $i++){
  $x = sha1($str);
 }
 $tse = microtime(true);
 echo "\nsha1: \t".round($tse-$tss, 5) . " \t" . $x;

 $tss = microtime(true);
 for($i=0; $i<$loops; $i++){
  $l = strlen($str);
  $x = 0x77;
  for($j=0;$j<$l;$j++){
   $x = $x xor ord($str[$j]);
  }
 }
 $tse = microtime(true);
 echo "\nxor: \t".round($tse-$tss, 5) . " \t" . $x;

 $tss = microtime(true);
 for($i=0; $i<$loops; $i++){
  $l = strlen($str);
  $x = 0x08;
  for($j=0;$j<$l;$j++){
   $x = ($x<<2) xor $str[$j];
  }
 }
 $tse = microtime(true);
 echo "\nxor2: \t".round($tse-$tss, 5) . " \t" . $x;

 $tss = microtime(true);
 for($i=0; $i<$loops; $i++){
  $l = strlen($str);
  $x = 0;
  for($j=0;$j<$l;$j++){
   $x = $x + ord($str[$j]);
  }
 }
 $tse = microtime(true);
 echo "\nadd: \t".round($tse-$tss, 5) . " \t" . $x;
10
  • 4
    Ah, Thank you for this insight actually, just fortifies my use of CRC32 being fastest.
    – John
    Sep 8, 2010 at 8:04
  • 1
    @John - You can retrieve the hashing algorithms using: hash_algos(). The following hash benchmarking code was in the PHP comments ==> codepad.viper-7.com/5Wdhw6 Aug 29, 2011 at 0:48
  • 1
    Thank you for your code. I have improved it a little bit. I don't think that we should compare functions like md5() that process the whole string and loops that do byte by byte like you made with xor. In PHP, these loops are very slow and are even slower than the md5 itself. We should compare one hases with another, all implemented as functions. Apr 18, 2017 at 2:48
  • 3
    Just a quick note - I tried this with a much longer string (~5000 chars) and CRC32 was slower than MD5 and SHA1 on my machine (i7-6650U, 16GB). CRC32 - 1.7s , MD5 - 1.4s, SHA1 - 1.5s. Always test for yourself.
    – Sam Tolton
    Oct 24, 2017 at 11:03
  • 6
    @Quamis the test is nice but may be misleading - as @samTolton noted the results are different and md5 is faster. A better test will be to randomize the strings content and length too. this way we get a better idea about the actual real world performance. This will also avoid caching. Take a look: php hashing checksum performance Apr 4, 2018 at 16:32
104

There's a speed comparison on the xxHash repository. This is what it shows, on January 12, 2021.

Hash Name Width Bandwidth (GB/s) Small Data Velocity Quality Comment
XXH3 (SSE2) 64 31.5 GB/s 133.1 10
XXH128 (SSE2) 128 29.6 GB/s 118.1 10
RAM sequential read N/A 28.0 GB/s N/A N/A for reference
City64 64 22.0 GB/s 76.6 10
T1ha2 64 22.0 GB/s 99.0 9 Slightly worse [collisions]
City128 128 21.7 GB/s 57.7 10
XXH64 64 19.4 GB/s 71.0 10
SpookyHash 64 19.3 GB/s 53.2 10
Mum 64 18.0 GB/s 67.0 9 Slightly worse [collisions]
XXH32 32 9.7 GB/s 71.9 10
City32 32 9.1 GB/s 66.0 10
Murmur3 32 3.9 GB/s 56.1 10
SipHash 64 3.0 GB/s 43.2 10
FNV64 64 1.2 GB/s 62.7 5 Poor avalanche properties
Blake2 256 1.1 GB/s 5.1 10 Cryptographic
SHA1 160 0.8 GB/s 5.6 10 Cryptographic but broken
MD5 128 0.6 GB/s 7.8 10 Cryptographic but broken

It seems xxHash is by far the fastest one, while many others beat older hashes, like CRC32, MD5 and SHA.

1
  • 1
    Note that the xxHash algorithms are available in PHP starting with version 8.1. The above link is to the C-language library implementing xxHash.
    – CXJ
    Mar 23, 2023 at 13:29
69

CRC32 is pretty fast and there's a function for it: http://www.php.net/manual/en/function.crc32.php

But you should be aware that CRC32 will have more collisions than MD5 or even SHA-1 hashes, simply because of the reduced length (32 bits compared to 128 bits respectively 160 bits). But if you just want to check whether a stored string is corrupted, you'll be fine with CRC32.

3
  • 1
    Wow, only required datatype is an unsigned integer, this will be SIGNIFICANLY faster than other hashing.
    – John
    Sep 8, 2010 at 7:31
  • 5
    @John: or not. CRC32 turns out to be slower than MD4, and not much faster than MD5, on ARM processors. Besides, CRC32 uses an unsigned 32-bit integer type, which is exactly all that MD5 needs... Sep 10, 2010 at 14:06
  • 5
    if you have the benefit/luxury of a newer Intel cpu, there is a crc32c assembly command that is...probably really fast (though isn't the traditional crc32 value). See also xxhash code.google.com/p/xxhash
    – rogerdpack
    Dec 2, 2013 at 20:34
56

Ranked list where each loop shares the same thing to crypt as all the others.

<?php

set_time_limit(720);

$begin = startTime();
$scores = array();


foreach(hash_algos() as $algo) {
    $scores[$algo] = 0;
}

for($i=0;$i<10000;$i++) {
    $number = rand()*100000000000000;
    $string = randomString(500);

    foreach(hash_algos() as $algo) {
        $start = startTime();

        hash($algo, $number); //Number
        hash($algo, $string); //String

        $end = endTime($start);

        $scores[$algo] += $end;
    }   
}


asort($scores);

$i=1;
foreach($scores as $alg => $time) {
    print $i.' - '.$alg.' '.$time.'<br />';
    $i++;
}

echo "Entire page took ".endTime($begin).' seconds<br />';

echo "<br /><br /><h2>Hashes Compared</h2>";

foreach($scores as $alg => $time) {
    print $i.' - '.$alg.' '.hash($alg,$string).'<br />';
    $i++;
}

function startTime() {
   $mtime = microtime(); 
   $mtime = explode(" ",$mtime); 
   $mtime = $mtime[1] + $mtime[0]; 
   return $mtime;   
}

function endTime($starttime) {
   $mtime = microtime(); 
   $mtime = explode(" ",$mtime); 
   $mtime = $mtime[1] + $mtime[0]; 
   $endtime = $mtime; 
   return $totaltime = ($endtime - $starttime); 
}

function randomString($length) {
    $characters = '0123456789abcdefghijklmnopqrstuvwxyz';
    $string = '';    
    for ($p = 0; $p < $length; $p++) {
        $string .= $characters[mt_rand(0, strlen($characters) - 1)];
    }
    return $string;
}

?>

And the output

1 - crc32b 0.111036300659
2 - crc32 0.112048864365
3 - md4 0.120795726776
4 - md5 0.138875722885
5 - sha1 0.146368741989
6 - adler32 0.15501332283
7 - tiger192,3 0.177447080612
8 - tiger160,3 0.179498195648
9 - tiger128,3 0.184012889862
10 - ripemd128 0.184052705765
11 - ripemd256 0.185411214828
12 - salsa20 0.198500156403
13 - salsa10 0.204956293106
14 - haval160,3 0.206098556519
15 - haval256,3 0.206891775131
16 - haval224,3 0.206954240799
17 - ripemd160 0.207638263702
18 - tiger192,4 0.208125829697
19 - tiger160,4 0.208438634872
20 - tiger128,4 0.209359407425
21 - haval128,3 0.210256814957
22 - sha256 0.212738037109
23 - ripemd320 0.215386390686
24 - haval192,3 0.215610980988
25 - sha224 0.218329429626
26 - haval192,4 0.256464719772
27 - haval160,4 0.256565093994
28 - haval128,4 0.257113456726
29 - haval224,4 0.258928537369
30 - haval256,4 0.259262084961
31 - haval192,5 0.288433790207
32 - haval160,5 0.290239810944
33 - haval256,5 0.291721343994
34 - haval224,5 0.294484138489
35 - haval128,5 0.300224781036
36 - sha384 0.352449893951
37 - sha512 0.354603528976
38 - gost 0.392376661301
39 - whirlpool 0.629067659378
40 - snefru256 0.829529047012
41 - snefru 0.833986997604
42 - md2 1.80192279816
Entire page took 22.755341053 seconds


Hashes Compared

1 - crc32b 761331d7
2 - crc32 7e8c6d34
3 - md4 1bc8785de173e77ef28a24bd525beb68
4 - md5 9f9cfa3b5b339773b8d6dd77bbe931dd
5 - sha1 ca2bd798e47eab85655f0ce03fa46b2e6e20a31f
6 - adler32 f5f2aefc
7 - tiger192,3 d11b7615af06779259b29446948389c31d896dee25edfc50
8 - tiger160,3 d11b7615af06779259b29446948389c31d896dee
9 - tiger128,3 d11b7615af06779259b29446948389c3
10 - ripemd128 5f221a4574a072bc71518d150ae907c8
11 - ripemd256 bc89cd79f4e70b73fbb4faaf47a3caf263baa07e72dd435a0f62afe840f5c71c
12 - salsa20 91d9b963e172988a8fc2c5ff1a8d67073b2c5a09573cb03e901615dc1ea5162640f607e0d7134c981eedb761934cd8200fe90642a4608eacb82143e6e7b822c4
13 - salsa10 320b8cb8498d590ca2ec552008f1e55486116257a1e933d10d35c85a967f4a89c52158f755f775cd0b147ec64cde8934bae1e13bea81b8a4a55ac2c08efff4ce
14 - haval160,3 27ad6dd290161b883e614015b574b109233c7c0e
15 - haval256,3 03706dd2be7b1888bf9f3b151145b009859a720e3fe921a575e11be801c54c9a
16 - haval224,3 16706dd2c77b1888c29f3b151745b009879a720e4fe921a576e11be8
17 - ripemd160 f419c7c997a10aaf2d83a5fa03c58350d9f9d2e4
18 - tiger192,4 112f486d3a9000f822c050a204d284d52473f267b1247dbd
19 - tiger160,4 112f486d3a9000f822c050a204d284d52473f267
20 - tiger128,4 112f486d3a9000f822c050a204d284d5
21 - haval128,3 9d9155d430218e4dcdde1c62962ecca3
22 - sha256 6027f87b4dd4c732758aa52049257f9e9db7244f78c132d36d47f9033b5c3b09
23 - ripemd320 9ac00db553b51662826267daced37abfccca6433844f67d8f8cfd243cf78bbbf86839daf0961b61d
24 - haval192,3 7d706dd2d37c1888eaa53b154948b009e09c720effed21a5
25 - sha224 b6395266d8c7e40edde77969359e6a5d725f322e2ea4bd73d3d25768
26 - haval192,4 d87cd76e4c8006d401d7068dce5dec3d02dfa037d196ea14
27 - haval160,4 f2ddd76e156d0cd40eec0b8d09c8f23d0f47a437
28 - haval128,4 f066e6312b91e7ef69f26b2adbeba875
29 - haval224,4 1b7cd76ea97c06d439d6068d7d56ec3d73dba0373895ea14e465bc0e
30 - haval256,4 157cd76e8b7c06d432d6068d7556ec3d66dba0371c95ea14e165bc0ec31b9d37
31 - haval192,5 05f9ea219ae1b98ba33bac6b37ccfe2f248511046c80c2f0
32 - haval160,5 e054ec218637bc8b4bf1b26b2fb40230e0161904
33 - haval256,5 48f6ea210ee1b98be835ac6b7dc4fe2f39841104a37cc2f06ceb2bf58ab4fe78
34 - haval224,5 57f6ea2111e1b98bf735ac6b92c4fe2f43841104ab7cc2f076eb2bf5
35 - haval128,5 ccb8e0ac1fd12640ecd8976ab6402aa8
36 - sha384 bcf0eeaa1479bf6bef7ece0f5d7111c3aeee177aa7990926c633891464534cd8a6c69d905c36e882b3350ef40816ed02
37 - sha512 8def9a1e6e31423ef73c94251d7553f6fe3ed262c44e852bdb43e3e2a2b76254b4da5ef25aefb32aae260bb386cd133045adfa2024b067c2990b60d6f014e039
38 - gost ef6cb990b754b1d6a428f6bb5c113ee22cc9533558d203161441933d86e3b6f8
39 - whirlpool 54eb1d0667b6fdf97c01e005ac1febfacf8704da55c70f10f812b34cd9d45528b60d20f08765ced0ab3086d2bde312259aebf15d105318ae76995c4cf9a1e981
40 - snefru256 20849cbeda5ddec5043c09d36b2de4ba0ea9296b6c9efaa7c7257f30f351aea4
41 - snefru 20849cbeda5ddec5043c09d36b2de4ba0ea9296b6c9efaa7c7257f30f351aea4
42 - md2 d4864c8c95786480d1cf821f690753dc
1
  • 5
    There's a minimal off-by-one error at the end. strlen($characters) should be strlen($characters) - 1 :)
    – MM.
    May 7, 2013 at 8:58
33

2019 update: This answer is the most up to date. Libraries to support murmur are largely available for all languages.

The current recommendation is to use the Murmur Hash Family (see specifically the murmur2 or murmur3 variants).

Murmur hashes were designed for fast hashing with minimal collisions (much faster than CRC, MDx and SHAx). It's perfect to look for duplicates and very appropriate for HashTable indexes.

In fact it's used by many of the modern databases (Redis, ElastisSearch, Cassandra) to compute all sort of hashes for various purposes. This specific algorithm was the root source of many performance improvements in the current decade.

It's also used in implementations of Bloom Filters. You should be aware that if you're searching for "fast hashes", you're probably facing a typical problem that is solved by Bloom filters. ;-)

Note: murmur is a general purpose hash, meaning NON cryptographic. It doesn't prevent to find the source "text" that generated a hash. It's NOT appropriate to hash passwords.

Some more details: MurmurHash - what is it?

2
  • 3
    There is an open request here to add murmurhash to php, which you can vote on.
    – keune
    Jun 14, 2017 at 17:06
  • 1
    As of PHP 8.1, MurmurHash3 is supported natively (hash('murmur3f', $string)). But whilst it's very fast, it's still not as quick as crc32 :)
    – Jack
    Nov 30, 2022 at 18:07
19
+-------------------+---------+------+--------------+
|       NAME        |  LOOPS  | TIME |     OP/S     |
+-------------------+---------+------+--------------+
| sha1ShortString   | 1638400 | 2.85 | 574,877.19   |
| md5ShortString    | 2777680 | 4.11 | 675,834.55   |
| crc32ShortString  | 3847980 | 3.61 | 1,065,922.44 |
| sha1MediumString  | 602620  | 4.75 | 126,867.37   |
| md5MediumString   | 884860  | 4.69 | 188,669.51   |
| crc32MediumString | 819200  | 4.85 | 168,907.22   |
| sha1LongString    | 181800  | 4.95 | 36,727.27    |
| md5LongString     | 281680  | 4.93 | 57,135.90    |
| crc32LongString   | 226220  | 4.95 | 45,701.01    |
+-------------------+---------+------+--------------+

It seems that crc32 is faster for small messages(in this case 26 characters) while md5 for longer messages(in this case >852 characters).

14

Instead of assuming that MD5 is "fairly slow", try it. A simple C-based implementation of MD5 on a simple PC (mine, a 2.4 GHz Core2, using a single core) can hash 6 millions of small messages per second. A small message is here anything up to 55 bytes. For longer messages, MD5 hashing speed is linear with the message size, i.e. it crunches data at about 400 megabytes per second. You may note that this is four times the maximum speed of a good harddisk or a gigabit ethernet network card.

Since my PC has four cores, this means that hashing data as fast as my harddisk can provide or receive uses at most 6% of the available computing power. It takes a very special situation for hashing speed to become a bottleneck or even to induce a noticeable cost on a PC.

On much smaller architectures where hashing speed may become somewhat relevant, you may want to use MD4. MD4 is fine for non-cryptographic purposes (and for cryptographic purposes, you should not be using MD5 anyway). It has been reported that MD4 is even faster than CRC32 on ARM-based platforms.

4
  • 1
    There's a point to consider. MD5 takes 128 bits instead of 32. This means that database storage takes 4 times more space and hence 4 times slower to look up for comparing hashes (I think). What I'm concerned with (for my uses) is how fast it will be to query the database later when it's full of hashes. Mar 27, 2011 at 16:33
  • 3
    If you do not use a wide enough output then you will get random collisions, which will be bad since the goal is to query a database to know whether a given "phrase" is already known; collisions here turn into false positives. With 32 bits, you will begin to see collisions as soon as you have 60000 or so phrases. This is true for all hash functions, cryptographic or not. That being said, you can always take the output of a hash function and truncate it to any length you see fit, within the limitations explained above. Mar 28, 2011 at 11:42
  • @ThomasPornin If we go by the truncating way, wouldn't it again face the collision problem, I mean the only reason the md5 is supposed to not get easy collision is the extra no of characters its having when compared to CRC32, right? Nov 18, 2014 at 21:38
  • If you are concerned about how much space the hash requires in the database, it's perfectly valid to only use the first X bits of a hash. Not necessarily recommending it, but you could use MD5 and only use the first four bytes.
    – Aurast
    Dec 10, 2020 at 1:42
5

If you're looking for fast and unique, I recommend xxHash or something that uses newer cpu's crc32c built-in command, see https://stackoverflow.com/a/11422479/32453. It also links there to possibly even faster hashes if you don't care about the possibility of collision as much.

5

Caveat

The answer below does not answer the question as asked, since it does not recommend hash functions. Remember, "A hash function is any function that can be used to map data of arbitrary size to fixed-size values." (Wikipedia) The answer below recommends transformations that do not guarantee fixed-size results.

If you are willing to relax the requirement of using a hash function, read on...

Original Answer

I suggest urlencode() or base64_encode() for these reasons:

  • You don't need cryptography
  • You want speed
  • You want a way to identify unique strings while cleaning up 'malformed' strings

Adapting the benchmark code elsewhere in these replies, I've demonstrated that either of these are way faster than any hash algorithm. Depending on your application, you might be able to use urlencode() or base64_encode() to clean up any 'malformed' strings you want to store.

4
  • Re: "You want a way to identify unique strings while cleaning up 'malformed' strings": would you elaborate please?
    – David J.
    Jun 5, 2020 at 23:54
  • It's hard to remember what I was thinking over six years ago... I might have been alluding to the fact that you don't get collisions with urlencode or base64_encode, so the results would be as unique as the original strings. Jun 7, 2020 at 0:36
  • Base64 makes the string longer, so you'd be better off just comparing the original strings.
    – Andrew
    Apr 26, 2023 at 17:37
  • True. Well... actually, if you're encoding just text, then using compress with base64 usually makes the string shorter. Possibly that's what I meant all those years ago. Apr 27, 2023 at 18:35
3

Step One: Install libsodium (or make sure you're using PHP 7.2+)

Step Two: Use one of the following:

  1. sodium_crypto_generichash(), which is BLAKE2b, a hash function more secure than MD5 but faster than SHA256. (Link has benchmarks, etc.)
  2. sodium_crypto_shorthash(), which is SipHash-2-4, which is appropriate for hash tables but should not be relied on for collision resistance.

_shorthash is about 3x as fast as _generichash, but you need a key and you have a small-but-realistic risk of collisions. With _generichash, you probably don't need to worry about collisions, and don't need to use a key (but may want to anyway).

3
  • 1
    question is "how quick is this thing"?
    – My1
    Mar 7, 2016 at 11:43
  • 2
    sodium_crypto_generichash(), which is BLAKE2b, a hash function more secure than MD5 but faster than SHA256. (Link has benchmarks, etc.) - blake2b sure is, but a USERLAND PHP implementation of blake2b is going to be a lot slower than the C-implemented sha256 for PHP ... i wish PHP could adobt blake2b in the hash_algos() suite..
    – hanshenrik
    Feb 6, 2019 at 13:27
  • The pure PHP implementation wasn't suggested here. Feb 6, 2019 at 19:14
2

CRC32 is faster, but less secure than MD5 and SHA1. There is not that much speed difference between MD5 and SHA1.

1
  • MD5 is now considered insecure. It's way more insecure than SHA1. Read MD5 wiki page.
    – Ahmed
    May 13, 2015 at 19:33
2

Adler32 performs best on my machine. And md5() turned out faster than crc32().

1
  • 3
    If MD5 is faster than a generic CRC32 function then something is very wrong.
    – nxasdf
    Mar 14, 2015 at 16:59
0

The implementation for md5 inside hash is a little bit faster than md5(). So this can be an option or some else, please try:

echo '<pre>';

$run = array();

function test($algo)
{
  #static $c = 0;
  #if($c>10) return;
  #$c++;

 $tss = microtime(true);
 for($i=0; $i<100000; $i++){
  $x = hash($algo, "ana are mere");
 }
 $tse = microtime(true);

 $GLOBALS['run'][(string)round($tse-$tss, 5)] = "\nhash({$algo}): \t".round($tse-$tss, 5) . " \t" . $x;
 #echo "\n$i nhash({$algo}): \t".round($tse-$tss, 5) . " \t" . $x;
}
array_map('test', hash_algos());
ksort($run);
print_r($run);
echo '</pre>';

You can see at http://www.dozent.net/Tipps-Tricks/PHP/hash-performance

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.