C++(Qt) software debugging—thread deadlock debugging (15)

C++ (Qt) Software Debugging-Thread Deadlock Debugging (15)

Article directory

C++ (Qt) software debugging — thread deadlock debugging (15)
- 1 Introduction
- 2. Common deadlocks
- 3. Debugging C++ deadlock with gdb under Linux
- - 1.1 Using code
  - 1.2 gdb debugging
- 3. Debugging Qt deadlock with gdb under Linux
- - 1.1 Using code
  - 1.2 gdb debugging
- 4. Gdb debugging C++ deadlock under Windows
- 5. Gdb debugging Qt deadlock under Windows
- 6. Windbg debugging C++ deadlock under Windows
- - 1.1 Using code
  - 1.2 Windbg debugging
- 7. Windbg debugging Qt deadlock under Windows

1. Foreword

A deadlock is a situation in which two or more threads (or processes) wait for each other to release resources, resulting in neither of them being able to continue execution. This is a very annoying problem because it can cause the program to hang and be unable to continue running.

This article will describe in detail how to debug C++ thread deadlock and Qt thread deadlock under Linux and Windows.

System environment: ubuntu20.04, Windows10;
Compiler: g++10, MinGW, MSVC2017-64;
Debugging tools: gdb, WinDbg.
It is best to add debugging information when compiling all programs. If you are using Qt, use Debug or Profile mode.
The method used in this article is also suitable for debugging infinite loops, but the details are slightly different.

2. Common deadlocks

Single-threaded deadlock:
Sometimes, a thread applies for a lock resource but has not yet waited for the release. It applies for the lock again, and the result is that it hangs and waits for the lock to be released. However, the lock is held by itself, so it will hang and wait forever. This creates a deadlock. The reasons for repeated locking may be as follows:

Usually, it is because of locking in multiple branches, but one branch forgets to lock or the lock release is skipped due to return, break and other statements;
Because the program itself uses throw to throw an exception or the underlying library throws an exception, the execution flow of the program is disrupted and the lock is not released.

For example, consider the following pseudocode:

void threadFun1()
{<!-- -->
    g_mutex1.lock(); // Lock
    
    g_mutex1.lock(); // Repeat locking

    g_mutex1.unlock();
}

void threadFun1()
{<!-- -->
    g_mutex1.lock(); // Lock
    
    if(value > 10)
    {<!-- -->
        return; // Return early, skip release
    }

    g_mutex1.unlock();
}

void threadFun1()
{<!-- -->
    g_mutex1.lock(); // Lock
    
    if(value > 10)
    {<!-- -->
        throw; // Throw an exception, disrupt the execution flow, and skip the release
    }

    g_mutex1.unlock();
}

Multi-thread deadlock:
Multi-thread deadlock is a more common situation, usually occurs when resources are shared between multiple threads, and is more difficult to troubleshoot than single-thread deadlock.

Multi-thread deadlock means that two or more threads are blocked while waiting for each other to release resources and cannot continue to execute.

For example: Thread 1 locked lock1 and tried to acquire lock2, while thread 2 locked lock2 and tried to acquire lock1 , they wait for each other to release resources, resulting in deadlock.

/********************************************** **********************************
* File name: main1.cpp
* Creation time: 2023-10-25 10:57:54
*Developer: MHF
* Email: [email protected]
* Function: Multi-thread deadlock example
*************************************************** *******************************/
#include <iostream>
#include <thread>
#include <mutex>
#include <unistd.h>

using namespace std;
mutex mutex1;
mutex mutex2;

void threadA()
{<!-- -->
    cout << "Start thread A" << endl;

    mutex1.lock();
    cout << "Thread A locks mutex1" << endl;

    // In order to simulate deadlock, let thread A sleep for a period of time
    sleep(1);

    mutex2.lock(); // Since thread B has locked mutex2, we will wait for thread B to unlock
    cout << "Thread A locks mutex2" << endl;

    //Perform some operations...

    mutex2.unlock();
    mutex1.unlock();
}

void threadB()
{<!-- -->
    cout << "Start thread B" << endl;

    mutex2.lock();
    cout << "Thread B locks mutex2" << endl;

    // In order to simulate deadlock, let thread B sleep for a period of time
    sleep(1);

    mutex1.lock(); // Since thread A has locked mutex1, we will wait for thread A to unlock
    cout << "Thread B locks mutex1" << endl;

    //Perform some operations...

    mutex1.unlock();
    mutex2.unlock();
}

int main()
{<!-- -->
    thread t1(threadA);
    thread t2(threadB);

    t1.join();
    t2.join();

    return 0;
}

3. Gdb debugging C++ deadlock under Linux

1.1 Using code

 /************************************************ **********************************
* File name: main.cpp
* Creation time: 2023-10-24 21:40:05
*Developer: MHF
* Email: [email protected]
* Function: Single-threaded deadlock example
*************************************************** *******************************/
#include<iostream>
#include <thread>
#include <mutex>

using namespace std;

mutex g_mutex1;

void threadFun1()
{<!-- -->
    cout << 1 << endl;
    g_mutex1.lock(); // Lock
    cout << 2 << endl;
    g_mutex1.lock(); // Repeat locking
    cout << 3 << endl;
}


int main()
{<!-- -->
    thread t1(threadFun1);

    t1.join();
    return 0;
}

1.2 gdb debugging

Use the g++ -g main.cpp -lpthread command to compile the code;
If you use ./a.out to run the program, you will find that the program deadlocks and will not continue to execute;
Reopen a terminal window;
Use the ps -aux | grep "a.out\|USER" command to view the process information of the a.out program (note: there must be no spaces before and after \| );
- grep “a.out \| USER”: Indicates that only lines containing the a.out string or USER string are displayed;
Use sudo gdb -q -p 14742 to attach gdb to the process PID of a.out (note that sudo is required to attach to the process);
After entering gdb, use the info threads command to view information about all threads;
It can be seen from the figure that the stack of thread 2 stops at the **__lll_lock_wait** frame. The g_mutex1 lock is used at this location. The __lll_lock_wait function is a function used to implement thread mutex waiting in the Linux system. It allows the thread to enter Wait state until mutex lock is available.
Use the thread 2 command to enter thread 2;
Use the bt command to view the current stack information of thread 2 (you can also use the thread apply all bt command to view the stacks of all threads);
The stack can be stopped at line 21 of the main.cpp file, in the threadFun1() function;
Use the f 4 command to switch to frame 4 of the thread 2 stack. You can see that it stops at the line of locked code g_mutex1.lock();
Use the list command to view the context code, and you can see that it is locked twice;
Use the p g_mutex1 command to print the lock information. You can see that __lock = 2 is also locked twice.

3. Debugging Qt deadlock with gdb under Linux

1.1 Using code

#include "widget.h"
#include "ui_widget.h"
#include <QtConcurrent>
#include <QMutex>

QMutex g_mutex;

Widget::Widget(QWidget *parent)
    : QWidget(parent)
    , ui(new Ui::Widget)
{<!-- -->
    ui->setupUi(this);
}

Widget::~Widget()
{<!-- -->
    delete ui;
}


void Widget::on_pushButton_clicked()
{<!-- -->
    //Create a QtConcurrent thread
    QtConcurrent::run(QThreadPool::globalInstance(), [ & amp;]()
    {<!-- -->
        qDebug() << "Enter QtConcurrent thread";
        g_mutex.lock();
        qDebug() << "Locked once";
        g_mutex.lock();
        qDebug() << "Locked twice, repeated locking";

        g_mutex.unlock();
    });
}

1.2 gdb debugging

After compiling and running the Qt program, click the pushButton button to enter the QtConcurrent thread and trigger a deadlock;
Use the ps -aux | grep 'testMutex\|USER' command to view the deadlock process pid;
Use the sudo gdb -q -p 21714 command to attach gdb to the process;
Use the info threads command to view information about all threads;
As shown in the figure below, it can be seen that the type of thread 7 is Thread (pooled) (if it is a thread created using QThread, the type is QThread). This is a QtConcurren thread created using the thread pool and stopped. The status of the stack frame is syscall(); the program stopping at the syscall() function usually means that it is making a system call, and if a deadlock occurs, the thread will remain in This state;
Use the thread 7 command to switch to thread 7;
Use the bt command to view the stack information of thread 7;
As shown in the figure below, using QBasicMutex::lockInternal() or QMutex::lock() indicates that the thread 7 stack stops at the lock() function of the mutex The location, how to find the stack frame containing your own source code, is on line 29 of the widget.cpp file.
Use the f 3 command to switch to the third frame of the stack. You can see that this frame stops at the g_mutex.lock() position and is locking the position;
Use the list command to view the context code, and you can see that it is locked twice;
Use the p g_mutex command to print g_mutex lock information. Unlike the mutex lock in C++, QMutex lock printing cannot obtain helpful information.

4. Gdb debugging C++ deadlock under Windows

The usage code is the same as under linux.

Open the cmd window of MinGW-64 (open from here with complete environment variables to facilitate finding dependent libraries);

Enter the path where the source code is located;
Use the g++ .exe main.cpp -g -lpthread command to compile the code (if the upgrade cannot find g++, use the absolute path where MinGw is located);
Executing the a.exe program triggers a deadlock;

Open the task manager, find the a.exe program, right-click and select [Go to details] to view the pid number of the process.

Open another cmd window;
Use gdb -q -p 8740 to attach gdb to the process for debugging;
Use the info threads command to view all thread information (different from Linux, deadlock threads cannot be directly seen);

Use thread apply all bt to view the stack information of all threads;
As shown in the figure below, we can see that pthread_mutex_lock() appears in thread 2, which means that the stack of this thread stops at the locked position, so a deadlock occurs. Looking further down, we find that the deadlock position appears at No. 21 of the main.cpp file. line, the location of the threadFun1() function.
The subsequent operations are dispensable and are no different from those under Linux;

5. Gdb debugging Qt deadlock under Windows

The usage code is the same as that under Linux;

Note: When MinGW is used to compile programs under Windows, the gdb version selected when debugging should be the same as the compiled g++ version. You cannot use 32-bit gdb to debug 64-bit programs, or vice versa.

After Qt compiles and runs the program, a deadlock is triggered;
Open the cmd terminal of the corresponding version of MinGW;
Use the pid process number of the deadlock program in the Task Manager window;
Use gdb -q -p pid to attach gdb to the deadlock process;
Directly use thread apply all bt to display the stack information of all threads;
It can be seen that a deadlock occurs in thread 3, and subsequent operations are the same.
However, the following situations sometimes occur during gdb debugging in MinGW and debugging cannot be performed. No problem has been found so far;

6. Windbg debugging C++ deadlock under Windows

1.1 Using code

Directly using the mutex lock in C++ to repeatedly lock will throw an exception when triggered in the msvc compiler, so there is no need to debug.
Here, a multi-threaded deadlock is used for demonstration.

/********************************************** **********************************
* File name: main.cpp
* Creation time: 2023-10-25 10:57:54
*Developer: MHF
* Email: [email protected]
* Function: Multi-thread deadlock example
*************************************************** *******************************/
#include <iostream>
#include <thread>
#include <mutex>
#include <Windows.h>

using namespace std;
mutex mutex1;
mutex mutex2;

void threadA()
{<!-- -->
    cout << "start A" << endl;

    mutex1.lock();
    cout << "threadA mutex1 lock" << endl;

    // In order to simulate deadlock, let thread A sleep for a period of time
    Sleep(1000);

    mutex2.lock(); // Since thread B has locked mutex2, we will wait for thread B to unlock
    cout << "threadA mutex2 lock" << endl;

    //Perform some operations...

    mutex2.unlock();
    mutex1.unlock();
}

void threadB()
{<!-- -->
    cout << "start B" << endl;

    mutex2.lock();
    cout << "threadB mutex2 lock" << endl;

    // In order to simulate deadlock, let thread B sleep for a period of time
    Sleep(1000);

    mutex1.lock(); // Since thread A has locked mutex1, here we will wait for thread A to unlock
    cout << "threadB mutex1 lock" << endl;

    //Perform some operations...

    mutex1.unlock();
    mutex2.unlock();
}

int main()
{<!-- -->
    thread t1(threadA);
    thread t2(threadB);

    t1.join();
    t2.join();

    return 0;
}

1.2 Windbg Debugging

Use the MSVC compiler to compile the code, run it and trigger a deadlock;
Open the WinDbg program (under the path C:\Program Files\Windows Kits\10\Debuggers\x64);
Select [File]->[Attach to Process] or directly press the shortcut key F6;
Then select By ID, find the deadlock process, and click [OK];
Then enter the ~*k command to view the stack information of all threads. The words std::_Mutex_base::lock appear as shown below. It can be seen that threads 1 and 2 died. Lock;

Then select [View] to open the [Processes and Threads] window and the [Calls Stack] window;
Click thread 1 in the [Processes and Threads] window, and then click the stack frame in the [Calls Stack] window to jump to the source code location where the deadlock occurs;

Or you can directly click the stack frame in the Command window to jump to the deadlock source code location (however, the source code location in WinDbg is the next line to the actual location).

7. Windbg debugging Qt deadlock under Windows

The usage code is the same as that under Linux;

The previous steps are the same;
When using the stack information of all threads in the ~*k command window, you will find that you cannot see much helpful information. In this case, you can find the stack frame containing the source code file;

As shown in the figure, click on this frame to jump to the source code to check whether a deadlock occurs;

If you want to view more detailed debugging information, you need to download the debugging symbols of the Qt library from the Qt official website.

{__/}
(? ?′? ?^? ?`?)~?
| ? |