27

I can't decide whether the following deque is thread-safe.
In short, I've created a class with a deque that displays its contents every 1 sec in a new thread (so it won't pause the main program while printing).
The deque is filled from the main thread, so basically there SHOULD be a chance of collision.
HOWEVER, the deque is filled using a class method, so essentially it is accessed from within the instance itself, therefore from the same thread.
Here's the simplified code:

import threading
import time
from collections import deque

class MyQueue(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self)
        self.q = deque()
        self.start()

    def run(self):
        # pop out queue items every 1 sec
        # (please ignore empty deque for now)
        while True:
            print self.q.popleft()
            time.sleep(1)

    def add_to_q(self, val):
        # this function is called from outside
        self.q.append(val)

# main
# fill the queue with values
qu = MyQueue()
for i in range(1:100):
    qu.add_to_q(i)

So, although adding and removing items from queue take place inside the instance, is there a risk due to the adding function being called from outside the instance?

EDIT:
Since I need to modify items in my deque, I had to use Deque. What I do is: roatate() to the given item, pop it out, modify, push it back in and rotate() it back to its original position.
Unless I find a way of implementing modifying items in a Queue, I'll have to stick to Deque

4
  • 2
    If you just need a thread-safe queue (I don't see you using any deque-specific features), you should probably use the built-in thread-safe queue.
    – user395760
    Dec 18, 2011 at 19:23
  • David, thanks, I added the loop. delnan, I excluded some features. The deque is being rotated every now and then so items can be popped out, modified, pushed back in and rotate back to their original position. I didn't find a way of doing that with Queue Dec 18, 2011 at 19:29
  • 8
    It's impossible to write code to test if something is threadsafe and instead you have to read the documentation or study the source code. Often, if something's not threadsafe it'll still work nearly all the time in a threaded context and then explode weeks later.
    – user97370
    Dec 18, 2011 at 20:25
  • Rather than having to rotate(), deque allows insert() at an arbitrary position, at least in python 2.7 / 3.7. Oct 3, 2019 at 23:44

3 Answers 3

31

Deque is thread-safe (http://docs.python.org/library/collections.html#deque-objects) for appends and pops from opposite sides. Beneath here, the docs only mention that append() and popleft() are thread-safe.

There is a thread-safe implementation of the Queue itself. So you should be using it unless you have some strange requirements.

4
  • 13
    Not only is deque threadsafe, but it's performance is much higher than the Queue module. The valuable thing that the Queue module adds is that get can block.
    – amcnabb
    Apr 5, 2012 at 18:33
  • Only thread safe appends and pops as I read it (for example, you can't necessarily use len() across threads can you?) Dec 13, 2013 at 14:48
  • 1
    deque definitely throws if you iterate and push at the same time
    – Brannon
    Jan 11, 2018 at 7:01
  • 1
    "... Deques support thread-safe, memory efficient appends and pops from either side of the deque with approximately the same O(1) performance in either direction..." (docs.python.org/3/library/collections.html#deque-objects), based on this, it's thread-safe appending or popping from either direction... Aug 2, 2019 at 5:14
28

Deque author here.

The MyQueue() class looks correct to me (at least with respect to the question of thread-safety).

The append() and popleft() methods are both atomic.

The code does need EAFP logic to handle the case where the input is empty:

def run(self):
    while True:
        try:
            print self.q.popleft()
        except IndexError:
            pass
        time.sleep(1)
1
  • 3
    And btw, EAFP is not the European Association of Fish Pathologists, but "Easier to Ask for Forgiveness than Permission’". 🙂 Oct 27, 2022 at 13:38
10

For information there is a Python ticket referenced for deque thread-safety (https://bugs.python.org/issue15329).

Title "clarify which deque methods are thread-safe", bottom line is:

The deque's append(), appendleft(), pop(), popleft(), and len(d) operations are thread-safe in CPython. The append methods have a DECREF at the end (for cases where maxlen has been set), but this happens after all of the structure updates have been made and the invariants have been restored, so it is okay to treat these operations as atomic.

Anyway, if you are not 100% sure and you prefer reliability over performance, just put a like Lock for print self.q.popleft() and self.q.append(val) ;)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.