Finding the root cause of hung Python test

How many times did you face a test that hanging for some reason? And how many times did you wonder what was the root cause of this? 🤷‍♂

Let’s say we can connect to an instance where tests run by SSH and have enough permissions to do it, but what should we do next?

In this article, we will try to figure out what we can do in this case and how

Let’s begin. I have an easy function that hangs.

At first, need to define on which PID our process started, usually, I use htop but in this case, it doesn’t matter let's find it via ps -x | grep python

Now we know that our program is running and has PID 7183

Let's try to use gdb for connecting to this process, it’s a low-level debugger that can be useful If you experience low-level problems such as crashes or deadlocks

By default, gdb doesn’t know about high-level information specific to the CPython interpreter, such as which Python function is currently executing, or what type or value has a given Python object represented by a standard pointer. We hereafter present two ways to overcome this limitation.

I used several commands to install gdb and extensions of Python for it

sudo gdb -p 7183

Now when we are inside this process we should check how many threads we have

info threads

And we have only a single thread, just let’s jump into thread 1

Then when we are inside the current thread we can check the stack trace bt

It seems useful but if you know the C programing language. But we are Python developers and already installed python-gdb. So we can use py-bt means pretty the same but in Python interpretation 😍

I guess it’s much better!

Also can be useful other commands:
, , , ,

https://devguide.python.org/gdb/

But what about the test which from time to time hangs. Let’s try the same things but with pytest

As a result of running python -m pytest -lvv test_run.py our test expectedly is hanging 😞

We can use the same steps as above, except need to find the PID of the pytest process

sudo gdb -p 7240

info threads

thread 1

In this case, let’s try another command py-list

It shows to us where our test stuck, to me, it is just amazing 🎉

In case if you have more threads than in the example thread apply all py-list could be useful

BTW: exiting from gdb pretty the same as in VIM 😁

So seems we figure out how to find the root cause of the hung test. Thanks for reading and good luck with tests 👍