Reprinted from There is a Scenery on the Mountain https://www.cnblogs.com/ssyfj/p/16278883.html#ubuntu-20.04-kdump-±crash-first experience
1: Introduction to program debugging tools
Programs in Linux are mainly divided into two types: user space programs and kernel space programs. They use different tools to generate different dump files and use different software to analyze the dump files.
For programs in user space, the dump files generated, such as: core.3296 and core-hellotest.2816.xm.1593867625, can be analyzed with the gdb tool.
For programs in the kernel space, the dump files generated, such as /var/crash/202007092214/dump.202007092214, are commonly used for crash analysis.
This article will introduce how to install the crash tool and simple use of the crash tool! !
Linux kernel source code analysis tutorial: https://mp.weixin.qq.com/s?__biz=Mzk0MzE5MTkzOA== & amp;mid=2247488308 & amp;idx=1 & amp;sn=ed084f20c52a0a9f33c16aae07535190 & amp;chksm =c336f1d9f44178cffa89217e57d977c3e404b2001434d83af866ef4b6dca64c2f589818c2363 & amp;token=2002831924 & amp;lang=zh_CN#rd
(1) Introduction to the working principle of Kdump
coredump is a crash on-site dump mechanism triggered by user-mode process segmentation faults, etc. You can use gdb to debug corefile. The Kdump dump mechanism triggered by kernel crash/panic, etc. is generally debugged using the crash tool.
Kernel crash dump refers to dumping part of the contents of RAM to disk or other storage when the kernel is abnormal. When a kernel panic occurs, the kernel relies on the kexec mechanism to quickly restart a new kernel instance in a pre-reserved memory area. The size of the reserved memory area can be specified through the kernel startup parameter crashkernel.
In order to achieve a “dual-core” layout, Kdump uses kexec to boot to the dump capture kernel (capture kernel) immediately after a kernel crash, and uses kexec to boot to “overwrite” the currently running kernel. The dump capture kernel can be a separate, purpose-built Linux kernel image, or the main kernel image can be reused on system architectures that support relocatable kernels.
kexec (kernel execution, similar to the Unix or Linux system call exec) is a mechanism in the Linux kernel that allows a new kernel to be launched from the currently running kernel. kexec skips the bootloader stage and hardware initialization stage performed by the system firmware (BIOS or UEFI) and loads the new kernel directly into main memory and begins execution immediately. This avoids lengthy complete reboots and can meet system high availability requirements by minimizing downtime.
In short, the principle of Kdump is to reserve part of the memory (such as 192MB) for the dump kernel. When panic/crash occurs, the kexec system call is used to directly pull up the dump kernel. Use this kernel to store dumps at the crash site.
(2)Kdump installation
The Ubuntu16 system used in this article is as shown in the figure below:
Although Kdump comes with Ubuntu18, it does not exist in Ubuntu16. It is found through the kdump-config show command that it needs to be installed.
Therefore, just use sudo apt install kdump-tools to install it, and finally restart it.
Use the kdump-config show command again, as shown below, and it works normally.
(3) Crash installation
Install with the following command and restart after completion
sudo apt install linux-crashdump sudo apt install crash
By viewing related files, we can know that crashkernel in the kernel startup parameters has been set during the installation process.
sudo cat /etc/default/grub.d/kdump-tools.cfg
sudo cat /boot/grub/grub.cfg
Actually, I feel that the previous crash tool has been installed after installing Kdump in the front… To be on the safe side, let’s restart it again (configure the crashkernel size later and then restart it).
After the service restarts successfully, we can view relevant information in the kernel dmesg. The machine reserves a 128M RAM memory area for dump capture kernel use.
dmesg -T | grep -i crash
At the same time, we can see that the status of Kdump is Ready through the command kdump-config show
service kdump-tools status shows that the status of kdump-tools is Active.
View the startup command line sudo cat /proc/cmdline
Check the address space allocated by crashkernel memory (in root state), cat /proc/iomem | grep -i crash
Check the size of crashkernel memory allocation cat /proc/iomem | grep -i crash
At this point, the kdump service has taken effect. When the system encounters a crash, the corresponding dump file can be generated. The save directory is /var/crash.
The Crash tool is a tool developed by Red Hat to analyze dump files, which is equivalent to a gdb debugging experience for kernel snapshots.
Supplement: 128M space is not enough to support the generation of corresponding dump files when the system crashes, so it is necessary to adjust the crashkernel memory size (key point)
1. Modify crashkernel=512M-:768M in /etc/defatul/grub.d/kdump-tools.cfg
2. Use sudo grub-mkconfig -o /boot/grub/grub.cfg to update the configuration
3.reboot to perform restart operation
(4) Test verification
The Linux sysrq tool can manually trigger kernel panic, which we can use for temporary testing:
sudo echo 1 > /proc/sys/kernel/sysrq sudo echo c > /proc/sysrq-trigger
Note: It will crash after running. Restart. After restarting, /var/crash will save the vmcore generated by this manual trigger.
In addition, the above commands need to be used under the root user. If you cannot enter using su -, su: Authentication failure will occur. Generally, there is no password set for the root user, just set it!
sudo passwd root
After the command runs successfully, a directory named after the current date will be generated in the /var/carsh directory, including dmesg.x and dump.x Two files, of which demsg.x is the system kernel log at the time of the crash, and the dump.x file is the dumped kernel snapshot file.
Two: Install the vmlinux file with debugging information
In order to use the Crash tool, we also need to install the vmlinux file with debugging information.
(1) Introduction to vmlinux
Find the vmlinux file corresponding to vmlinuz. The vmlinux file is a necessary input parameter for the crash tool.
vmlinux is an ELF file, which is the most original file compiled by the Linux kernel.
vmlinuz is a compressed file obtained by OBJCOPYing the ELF file vmlinux.
(2)vmlinux installation
Find the vmlinux corresponding to the kernel through the following method.
1. Set up repo warehouse
echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/ddebs.list sudo apt-get update
2. Install dbgsym
sudo apt -y install linux-image-$(uname -r)-dbgsym --allow-unauthenticated
After entering the command, the dbgsym file under the corresponding kernel version will be automatically found, as shown below:
linux-image-unsigned-4.15.0-142-generic-dbgsym
Of course, it is also best to install the following versions together
linux-image-4.15.0-142-generic-dbgsym
If the installation using apt is too slow, you can also install it through website search (the first time I used the website https://launchpad.net/ubuntu/xenial/amd64/linux-image-unsigned-4.15.0-142-generic- dbgsym/4.15.0-142.146~16.04.1, the command used this time)
After installation is complete:
After the ubuntu-dbgsym-keyring package is installed successfully, we can see that vmlinux-5.4 has been installed in the directory file./usr/lib/debug/boot/ .0-80-generic
At this point, we have everything ready to happily use the Crash tool for debugging:
sudo crash /usr/lib/debug/boot/vmlinux-4.15.0-142-generic /var/crash/202205171635/dump.202205171635
Use the bt command to view the running stack at the time of the crash
Supplement: When we install different versions of dbgsym debugging files, it will cause the crash tool to be unable to be entered, and a problem seems to be prompted. Therefore, we need to follow the previous method to enter the website to find the appropriate file, and install it through the dpkg -i command.
The installation of the crash tool is now complete. If an error occurs when writing the kernel program in the future, the system will save the kernel state and automatically restart when it crashes. By using the crash tool and viewing the running stack through simple commands such as bt, we can roughly understand the problem. Function location and reasons, including the use of null pointers, etc.
Three: Crash subcommand usage
The operation of subcommands is similar to the operation of bash. You can use commands such as file redirection and grep/awk, which are very convenient for analysis.
For the specific usage format, you can learn the detailed usage of subcommands through man subcommand.
(1)bt: Used to view the stack and register status of the process
I often use this to know what error occurred in that function, and then modify it.
(2) ps command to view all processes in the system
The ps command views all processes in the system, where the ST field indicates the status, RU = “Running”, IN = “Interruptable” UN = “UnInterruptable”, ID = “Idle”. The TASK field represents the address of task_struct.
(3) files:files pid View details of files opened by the specified process
(4)task is used to display the task_struct structure
If you only want to view individual subfields, you can use -R to specify it. Commas are supported to separate multiple subfields:
(5) struct The struct command can view the detailed fields of the corresponding structure. If you need to view the offset of the field, add the -o parameter
If you clearly know the data structure corresponding to an address, you can also print it through struct: (ffff9e7e90b31700 corresponds to the previous 1781 process address)
(6) vm View the virtual memory of the specified process
(7) irq command view interrupt
(8)kmem is used to view system memory information
I have compiled some learning books and video materials (Linux kernel, Linux C++ backend/audio and video/games/embedded/high-performance network/storage/infrastructure/security learning materials, teaching videos and learning roadmaps) if needed. You can add your own learning and communication group: 739729163 to get it! ! !
