3

I am running a Python 3.7 Flask application which uses flask_socketio to setup a socketio server for browser clients, another python process to connect to a separate remote socketio server & exchange messages, and another python process to read input from a PIR sensor.

Both python processes communicate over multiprocessing.Queue - but, the socketio process always gets either [WinError6] - Invalid Handle or [WinError5] - Permission Denied. I have absolutely no idea what I'm doing wrong.

Here's the top-level (server) code; it does not appear to have issues:

from shotsocket import init as shotsocket_init
from shotsocket import util as matchmaking_util
import multiprocessing, os, config, uuid

match_queue = multiprocessing.Queue()
shot_queue = multiprocessing.Queue()

app = Flask(__name__, static_url_path='', static_folder='templates')
socketio = SocketIO(app)

_rooms = [] # I don't plan to keep this in memory, just doing it for debug / dev 

...

The above works fine and dandy. The 2nd to last line in the following block is the issue.

# THIS IS THE FUNC WHERE WE ARE TRYING TO USE 
# THE BROKEN QUEUE

@socketio.on('connect')
def listen():
    room_key = str(uuid.uuid4())
    join_room(room_key)
    _rooms.append((room_key, request.sid))
    possible_match = matchmaking_util.match_pending_clients(_rooms)
    if possible_match:
        shot_queue.put_nowait(possible_match)
        print('put it in there')

Here's how I start these processes:


if __name__ == '__main__':
    debug = os.environ.get('MOONSHOT_DEBUG', False)
    try:
        proc = multiprocessing.Process(target=start, args=(debug,match_queue))
        proc.start()
        shot_proc = multiprocessing.Process(target=shotsocket_init, args=(shot_queue,))
        shot_proc.start()
        socketio.run(app, host='0.0.0.0')
    except KeyboardInterrupt:
        socketio.stop()
        proc.join()
        shot_proc.join()

And here's the entirety of shotsocket (the code that cannot read the queue)

import socketio, multiprocessing # mp for the type

sio = socketio.Client(engineio_logger=True)
sio.connect('redacted woot', transports=['websocket'])


@sio.on('connect')
def connect():
    print("connected to shot server")

def init(queue: multiprocessing.Queue):
    while True:
        try:
            # WE NEVER GET PAST THIS LINE
            print(queue.get())
        except Exception as e:
            continue
        if not queue.empty():
            print('queue empty')
            shot = queue.get()
            print(shot)         
            match_id, opponents = shot
            sio.emit('start', {'id': match_id, 'opponents': [opponents[0], opponents[1]]})

I'm pulling my hair out. What the heck am I doing wrong?

5
  • Move all instantiations from your first code snippet below if __name__ == '__main__':, else they will be re-executed for every child process when the target-functions get imported by them.
    – Darkonaut
    Feb 11, 2019 at 22:23
  • @Darkonaut can you make this an answer and submit it? Include code if possible. I made the change and see important differences in execution but am still getting the same errors. Feb 12, 2019 at 20:50
  • I cannot replicate sorry. Obviously for some reason the inheritence of the (FIFO) handles fails. I'm not on Windows nor am I using Flask so what I wrote already is pretty much everything I can say. Are you on Python 3.7.2?
    – Darkonaut
    Feb 12, 2019 at 21:01
  • Yeah I am - found a solution though. See my answer; any explanation of why this fixed the issue would be appreciated. (But obviously, if you can't repro, it's understandable you wouldn't know) Feb 12, 2019 at 22:21
  • Staying with multithreading instead of going multiprocess is riding a different pony. The queues are differently implemented under the hood. Threads in the same process also naturally share memory. Using multithreading would be the first choice for I/O-bound tasks anyway. AFAIK Python 3.7.2 introduced a regression bug with multiprocessing and virtual environments, see.
    – Darkonaut
    Feb 12, 2019 at 22:32

1 Answer 1

0

Solution

I have no idea why this fixes the problem, but switching from multiprocessing.Queue to queue.Queue and multiprocessing.Process to threading.Thread did it.

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.