15

Memcached has length limitations for keys (250?) and values (roughtly 1MB), as well as some (to my knowledge) not very well defined character restrictions for keys. What is the best way to work around those in your opinion? I use the Perl API Cache::Memcached.

What I do currently is store a special string for the main key's value if the original value was too big ("parts:<number>") and in that case, I store <number> parts with keys named 1+<main key>, 2+<main key> etc.. This seems "OK" (but messy) for some cases, not so good for others and it has the intrinsic problem that some of the parts might be missing at any time (so space is wasted for keeping the others and time is wasted reading them).

As for the key limitations, one could probably implement hashing and store the full key (to work around collisions) in the value, but I haven't needed to do this yet.

Has anyone come up with a more elegant way, or even a Perl API that handles arbitrary data sizes (and key values) transparently? Has anyone hacked the memcached server to support arbitrary keys/values?

5 Answers 5

20

The server does already allow you to specify whatever size you want:

-I            Override the size of each slab page. Adjusts max item size
              (default: 1mb, min: 1k, max: 128m)

However, most of the time, when people are wanting to cache larger objects, they're doing something wrong. Do you really need that much data in one cache key? Uncompressed?

If you have sufficiently large tasks, the benefit of low-latency access is dwarfed by the time it takes you to actually transfer the data. Or you find that tossing everything in the same key means your frontend ends up having to do a lot of work to deserialize a bit of data they want.

It depends on your needs, and I can't tell you what's best for you without knowing more of what you're doing. If you truly do need something bigger than 1MB, that's why we added -I, though.

9

$key=abs(crc32($long_key))

This way you get unique key for queries and other long keys which may have changes beyond the 250 chars memcache sees.

Whoa...careful. Good advice, but without an important caveat. That can cause collisions. Sure it's highly improbable, but it only ever has to happen once to cause an earth shattering bug. You will still probably want to store the long key with memcached and always double-check for collisions at the key. Best way to deal with them would be to store a simple list of long_key/value pairs.

2
  • In (most) scenarios where you are using memcached to cache data that is persisted elsewhere, the probability of a collision negates the concern for program around this. Simply monitoring memcached for high collision rates will tell you if there is a problem, and only then is it worth it to spend the time looking for keys that are overwriting each other. If the caching strategy is correctly implemented and you are caching data that has been persisted, there will be no earth shattering bugs, only a performance degradation. Jul 22, 2013 at 19:04
  • 3
    Depends what you're caching. If the information cached is private to a user, a hash collision could mean leaking private data (which I would call an earth shattering bug). Granted there are hashes that make hash collision probability effectively nil, but crc32 isn't one of them. It only takes slightly over 77000 values for the odds of a hash collision to be 1 in 2 (preshing.com/20110504/hash-collision-probabilities). Using a better hash function for the key helps, but every hash table implementation I know of accounts for collisions. It's not hard; just store and check the key.
    – peabody
    Jul 23, 2013 at 18:38
0

For values that were too large, instead of storing a standard value (which, when decoded, was always a dictionary) we stored a list of keys. We then read the data in each key, and restored the main value. I think we also hashed the keys when they were too long (which in our dataset could happen, but extremely rarely).

We did write all this code directly on top of the memcached client (we were using Python), so at a higher level it was all transparent.

-1

Not actual related to the initial question but in case you think about switching to APC: its max. len for keys is 9727 chars. (tested on PHP 5.3.2)

-3
$key=abs(crc32($long_key))

This way you get unique key for queries and other long keys which may have changes beyond the 250 chars memcache sees.

2
  • 21
    May the collision be with you. Jul 1, 2012 at 2:08
  • 2
    Agreed. A CRC32 will give you a lot of collision, at least use an MD5 hash. Jan 11, 2016 at 18:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.