Overcoming GUI Freezes in PyQt: From Threading & Multiprocessing to ZeroMQ & QProcess
When working on a PyQt application, I encountered a graphical user interface (GUI) freezing issue while running a piece of computationally intensive code. PyQt applications use a main event loop to handle user interactions and GUI updates, since the code takes too long, it blocks the loop, causing the app to freeze.
I tried offloading the computation-heavy tasks to a separate thread. The computation is still freezing the GUI despite being in a thread. This might be due to insufficient resource contention. Also, the Global Interpreter Lock (GIL) in Python can bottleneck threads when performing CPU-intensive tasks. Then, I offloaded the computation to a separate process via multiprocessing instead of a thread, but I got an error stating that cannot pickle ‘XXX’ object , which contains a C# object that cannot be serialized directly in Python, so it cannot be used as an input argument. I found a way to overcome this by saving the program object as a temporary file and passing the temp file path as an argument. After resolving that small pickling error, I ran into another error: QWidget: Must construct a QApplication before a QWidget.
I stopped solving it for a week and have been thinking about how Jupyter Notebook enables users to run code interactively, and even when running an intensive operation such as plotting a graph with millions of data points or retrieving large amounts of data from the database, users can still create new and work on other Jupyter notebooks while waiting for those long computations to complete. We want to achieve something similar to Jupyter Notebook’s asynchronous communication between the front and backend, allowing the frontend to remain responsive while the backend processes long-running tasks. To achieve that, Jupyter uses ZeroMQ.
Then, I wondered if ZeroMQ could solve our GUI freezing issue with ZeroMQ handling communication and memory-intensive processes while PyQt provides the GUI. I reattempted to solve it via pyzmq and … it works! By using ZeroMQ to communicate between the PyQt GUI and a separate plotting process, it isolates the memory consumption of the plotting operation. This prevents excessive memory usage in the GUI thread from impacting the responsiveness of the application.
The implementation is first by creating a separate worker process to listen on a ZeroMQ socket for incoming plotting requests from the GUI. Upon receiving the request, the worker processes, serializes plot data, and sends the serialized plot data back to the GUI/ main process. In the PyQt application, there will be a ZeroMQ socket to connect to the plotting process, send requests to the worker process, and receive the results such as serialized plot data. Finally, you can deserialize the plot data and update the plot in the PyQt GUI.
Although this approach works, I believe introducing ZeroMQ to our PyQt app might be overkill as it adds potential overhead from message passing, and complexity to the application. Port management also needs to be taken care of when using ZeroMQ, especially in the production environment in which hardcoded port numbers can lead to the risk of conflicts, nevertheless, we can mitigate this in several ways, for example by doing dynamic port allocation.
Example: PyQt Plotting App via ZeroMQ
Given the potential complexity of ZeroMQ for this specific use case, QProcess might be a more suitable approach for offloading plotting tasks in our PyQt application. QProcess is part of Qt and is specifically designed for managing and communicating with external processes, making it well-suited for tasks like offloading CPU-bound operations. However, I got an error QProcess: Destroyed while process is still running when switching to QProcess. This will be my next experiment...
