3

I am considering creating a numpy table as key/value database. The inputs/update would be multi-theaded.

Exploring the idea, Problem: would GIL stop theads and only allow one update at time. Problem: can numpy table (tablespace) be mutlitheaded.

2 Answers 2

5

Some numpy functions are not atomic, so if two threads were to operate on the same array by calling some non-atomic numpy functions, then the array will become mangled because the order of operations will be mixed up in some non-anticipated way.

There are many examples, but just to be concrete, numpy.apply_along_axis is a long sequence of Python statements, clearly not atomic.

The GIL will not help you since it could stop one thread while it is only partly through some non-atomic numpy function, then start another thread which is operating on the same array...

So to be thread-safe, you would need to use a threading.Lock and only operate on the array after the Lock has been acquired:

with lock:
    arr = ...

Having to use a lock everywhere calls into question whether there is any benefit to having multiple threads operating on same array. Note that sometimes multithreading on a CPU-bound problem may result in slower performance than a comparable single-threaded version.

See also the ParallelProgramming with numpy and scipy wiki page for more alternatives and discussion.

1

I ve just needed that so I wrote it.. Going to try it now so not sure if it works as expected yet..

class LockedNumpyArray(object):
    """
    Thread safe numpy array
    """
    def __init__(self):
        self.lock = threading.Lock()
        self.val = None

    def __get__(self, obj, objtype):
        self.lock.acquire()
        if self.val != None:
            ret_val = self.val.copy()
        else:
            ret_val = None
        self.lock.release()
        # print('getting', ret_val)
        return ret_val

    def __set__(self, obj, val):
        self.lock.acquire()
        # print('setting', val)
        self.val = val.copy()
        self.lock.release()

That is the class for numpy array. Then I have a class for controlling because later I want to have more thread safe numpy arrays operating.

class CaptureControl():
    """
    Shared class to control source capture execution
    """
    frame = LockedNumpyArray()

    def __init__(self):
        print(self.frame)
        self.frame = np.array([2])
        print(self.frame)

In the end I put an instance of this CaptureControl class into thread(s) as follows.

class CaptureThread(threading.Thread):
    """
    Thread running source capturing
    """
    def __init__(self, capture_control):
        threading.Thread.__init__(self)
        self.capture_control = capture_control
        self.sleepTime = 0.01
    def run(self):
        while True:
            self.capture_control.capture_frame()
            time.sleep(self.sleepTime)

if __name__ == '__main__':
    capture_control = CaptureControl()
    capture_thread = CaptureThread(capture_control)
    capture_thread.start()

I will be glad if anyone shared any thoughts about this solution, not including the time.sleep in thread run function as this is just an example ;).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.