0

Is there a way to limit the number of threads spawned by Qt?

I have a GUI with 2 parts that change dynamically with time. For these 2 parts I had to create and use threads to update dynamically the content of theses dynamic parts of the GUI. So in the end, my GUI is made of :

  • a QApplication

  • a QWidget (GUI with lots of checkbox edits and so on - does PyQt5 needs so many dedicated threads in backend to handle them ?!?!)

  • 2 threads (executing tasks)

  • pyqtSignals to spawn events and data between GUI (where commands are) and threads (where commanded tasks are executed: results returned to GUI).

OK, so far so good. For a 'theoretical' point of view I need 'only' 3 threads (main + one thread by task).

For the threads parts, I initially used QThreads and get something to work. OK, but performance are bad. Then I noticed with htop that Qt spawned something like 17 threads ?!?!... Instead of just the 3 I had in mind ?!?! After googling, I found this interesting post https://www.pythonguis.com/tutorials/multithreading-pyqt-applications-qthreadpool/ and decided to use QThreadPool hoping to get better thread management. I changed the code to use QThreadPool (setting setMaxThreadCount(2)) and QRunnable: Qt still uses 17 threads ?!...

I use PyQt on a regular basis for a long time generally on x86 without power limitations so I never had bad surprise and never need to spy over Qt with htop. But today, I am running on ARM... And performance are very bad (even regularly freezing).... So I was wondering if it may be in relation with the 17-instead-of-3 threads (looks like ARM overflows) ?!

Questions are :

  • Is there a way to limit the number of threads spawned by Qt? Didnt't find any.

  • Is it normal that Qt spawns so many threads ? I would understand yes if there is a reason for it.

  • If there is no way to limit threads, is there a workaround or a pattern for this kind of situations?

UPDATE

Finally succeed to get a minimal example that:

  • Does no more suffer from performance problem (like one can expect - thread task are super simple). In real case, performance are decent or drop dramatically (lag) depending on options chosen in the GUI (the GUI has options to enable user control: according to these options threads need to do more or less complex tasks).

  • Does use more threads (8) than the only 3 threads that one could expected to be needed (1 main thread + 2 threads in a pool). In real case, the GUI has a lot of checkbox, edits and so on and I get 30+ threads instead of 4 (initially I had 'only' 2 threads in the pool and htop reported 17+ threads, but now, the GUI had to become more complex and I had to add a 3d thread in the pool and htop shows now 30+ threads ?!). In real case, performance are correct but drop if I activate some options in the GUI that trigger heavy tasks for the threads to handle: I expected 4 threads (main + 3 threads in the pool) instead of 30+!

>> more dummyGUI.py
#!/usr/bin/env python3
# -*- coding:utf-8 -*-

import sys
import time
from PyQt5.QtWidgets import QWidget, QApplication, QLabel, QGridLayout
from PyQt5.QtCore import Qt, pyqtSignal, QThreadPool, QObject, QRunnable

class DummyThreadSignals(QObject):
    # Signals enabling to update application from threads.
    updateGUI = pyqtSignal(int, str)

class DummyThread(QRunnable):
    def __init__(self, side, parent):
        # Initialise.
        super().__init__()
        self.signals = DummyThreadSignals()
        self._data = 0
        self._side = side
        self._run = True
        parent.signals.stop.connect(self.stop)

    def run(self):
        # Run thread.
        while self._run:
            time.sleep(1)
            self._data += 1
            self.signals.updateGUI.emit(self._data, self._side)

    def stop(self):
        # Stop thread.
        self._run = False

class DummyGUISignals(QObject):
    # Signals enabling to update threads from application.
    stop = pyqtSignal()

class DummyGUI(QWidget):
    def __init__(self):
        # Initialise.
        super().__init__()
        self.setWindowTitle('DummyGUI')
        self.signals = DummyGUISignals()

        # Create widgets.
        self._txtLeft = QLabel('')
        self._txtRight = QLabel('')

        # Handle layout.
        grdLay = QGridLayout()
        grdLay.addWidget(self._txtLeft, 0, 0)
        grdLay.addWidget(self._txtRight, 0, 1)
        self.setLayout(grdLay)

        # Handle alignment.
        self._txtLeft.setAlignment(Qt.AlignCenter)
        self._txtRight.setAlignment(Qt.AlignCenter)

        # Start threads.
        self.signals = DummyGUISignals()
        self._threadPool = QThreadPool()
        self._threadPool.setMaxThreadCount(3)
        self._threadLeft = DummyThread('left', self)
        self._threadLeft.signals.updateGUI.connect(self.updateGUI)
        self._threadPool.start(self._threadLeft)
        self._threadRight = DummyThread('right', self)
        self._threadRight.signals.updateGUI.connect(self.updateGUI)
        self._threadPool.start(self._threadRight)

    def updateGUI(self, data, side):
        # Update thread label.
        txtLbl = self._txtLeft if side == 'left' else self._txtRight
        txtLbl.setText('%s'%data)

    def closeEvent(self, event):
        # Close application.
        self.signals.stop.emit() # Warn threads to stop.
        self._threadPool.waitForDone() # Wait for threads to stop.
        event.accept()

# Main program.
if __name__=="__main__":
    # Create application.
    app = QApplication(sys.argv)
    gui = DummyGUI()
    gui.show()
    sys.exit(app.exec_())

htop running on x86 (laptop with regular CPU and no power limitations) gives :

dummyGUI-x86

htop running on ARM (embedded device running on ARM with power limitations) gives:

dummyGUI-ARM

In real case, I can only run the GUI on ARM (not on x86) as GUI need device only supported/pluggable by/on ARM platform. So I can unfortunately not compare real behaviors on ARM and x86. ARM has 6 cores so one could think running 6 threads max on it.

A few answers to comments:

  • I use apt-get install python3-pyqt5 that trigger Qt install: I do not build from source and config Qt by hand (on ARM, compiling is slow: compiling Qt would be endless)

  • created versus concurrently active threads: my intuition is that the 30+ threads may indeed be concurrently active and properly handled by Qt. But why so many?

  • race conditions: yes, obviously! I was very careful on this. GUI send signals to threads to warn them options have changed (checkbox click, ...): I change params in thread callbacks using locks (thread data member changed between lock acquire/release) such that callbacks do the minimum to let the thread run method do the real work (possible more or less heavy tasks) as soon as it's possible (thread is disturbed as less as possible - callbacks only change thread parameters as quickly as possible, and, the thread run adapts to new parameter when possible to do simple or complex tasks when all is set up for it). Obviously bugs are always possible (asynchronous stuffs are the most difficult ones), but, I was careful on that kind of stuffs. I added timers in threads: for complex tasks timers report high values and the whole thing lag.

At this point, my feeling is that ARM is overflowed and can't take too heavy tasks, and, that Qt is not the problem (even if it creates many more threads that I expected - they may be handled properly). The root problem is: how to know for sure? That's why at first I spied threads over htop.

So the question is: is it normal/expected that Qt use 'more' threads than 'needed' (in the sense that one could expect)? I'd totally understand the answer could be yes (even if I was surprised at first) if there's a reason for it.

Corollary question: is there a way to say to Qt 'just for this script, use only 3/4 threads (as it's likely to be enough)'? I'd totally understand the answer could be no, but, if there's a way to do so I'd like to know. If so, I could compare performance in both cases.

6
  • First of all, the main difference between QThread and QRunnable is that the former supports the event loop and is a QObject subclass, which makes using signal/slot mechanism easier, while the latter is a higher level class that allows reusability of a thread and does not have an event loop. Due to the way threading works in Python (which is the major bottleneck in PyQt/PySide) there's little (or probably no) actual difference in performance between the two. Also, QThreadPool already considers the maxThreadCount(), so if that amount is reached, any new thread is queued. Jan 25, 2022 at 23:11
  • Then, it's not clear how your case creates so many sub-threads, it may be related to the way ARM works (or htop reads it), but I am almost certain that those are not used for the UI, since UI elements can only exist (and be accessed to) in the main thread. While waiting for more insight from other experienced users, I'd suggest you to do some testing with a more simple program to see the results, and eventually edit your post with them, integrating a proper minimal reproducible example, which is required as right now your question is too vague to understand where (and if) the problem could be. Jan 25, 2022 at 23:14
  • OK, thanks for the feedback. I wanted to provide a minimal reproducible example but as the code is very complex/tricky it's a difficult/long task to get one... So I started with a SO question in the first place. I'll try to reduce down such an example if I can.
    – fghoussen
    Jan 26, 2022 at 7:27
  • Consider that 99% of times questions require a minimal reproducible example, and trying to hurry up by asking without providing any almost always results in people asking for them. Take your time, don't rush it. It could take a lot of time to provide a valid MRE, even more than a day. But it's almost always worth it. Jan 26, 2022 at 9:21
  • 1
    It's normal for Qt to create additional threads - see, for example, this question. But note that it's very important to distinguish between the number of threads created and the number of threads concurrently active. Without a proper test-case, it's impossible to know whether the number of threads per se has a significant impact on performance. Rather than making assumptions, I would suggest you do some profiling, so you can see where the real performance bottlenecks are.
    – ekhumoro
    Jan 26, 2022 at 19:21

1 Answer 1

0

Is there a way to limit the number of threads spawned by Qt?

Yes and Yes*.

There is a way to limit the user threads and you have it via QThreadPool; first Yes.

The htop will show all threads. For instance, pyQt5 might use libusb, Python, linux event handling, etc which spawns threads. These are the infra-structure threads which can varying depending on the options used to configure Qt. If you have just grabbed an OS version apt install or pip, then you have no control over this. In order to get this to a bare minimum, you would have to configure and build Qt yourself; second Yes*.

But today, I am running on ARM... And performance are very bad (even regularly freezing)....

It is most likely that you have some race in your application which is only apparent when running on a lower performance CPU. The infrastructure threads are most likely not responsible for this. Although that is possible. You have to get statistics on which thread is consuming the most CPU, as other alluded to in the comment to your question.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.