Here is what we have currently:
- we're trying to get cached django model instance, cache key includes name of model and instance id. Django's standard memcached backend is used. This procedure is a part of common procedure used very widely, not only in celery.
- sometimes(randomly and/or very rarely) cache.get(key) returns wrong object: either int or different model instance, even same-model-different-id case appeared. We catch this by checking correspondence of model name & id and cache key.
- bug appears only in context of three of our celery tasks, never reproduces in python shell or other celery tasks. UPD: appears under long-running CPU-RAM intensive tasks only
- cache stores correct value (we checked that manually at the moment the bug just appeared)
- calling same task again with same arguments might don't reproduce the issue, although probability is much higher, so bug appearances tend to "group" in same period of time
- restarting celery solves the issue for the random period of time (minutes - weeks)
- *NEW* this isn't connected with memory overflow. We always have at least 2Gb free RAM when this happens.
- *NEW* we have
cache_instance = cache.get_cache("cache_entry")
in static code. During investigation, I found that at the moment the bug happenscache_instance.get(key)
returns wrong value, althoughget_cache("cache_entry").get(key)
on the next line returns correct one. This means either bug disappears too quickly or for some reason cache_instance object got corrupted. Isn't cache instance object returned by django's cache thread safe? - *NEW* we logged very strange case: as another wrong object from cache, we got model instance w/o id set. This means, the instance was never saved to DB therefore couldn't be cached. (I hope)
- *NEW* At least one
MemoryError
was logged these days
I know, all of this sounds like some sort of magic.. And really, any ideas how that's possible or how to debug this would be very appreciated.
PS: My current assumption is that this is connected with multiprocessing: as soon as cache instance is created in static code and before Worker process fork this would lead to all workers sharing same socket (Does it sound plausibly?)