Foreword
Understanding the operating system helps locate and troubleshoot online problems. Mastering the key mechanisms of the kernel can deepen your understanding of the program’s operating mechanism, providing a strong foundation for writing high-performance code and solving performance problems.
The kernel module is a way to get started with the Linux kernel. Through the kernel module, the inscrutable veil of the Linux kernel is revealed. Taking the kernel module as the starting point, we will sort out the relevant contents of memory management and process management in sequence;
1. Background
1. Goal
a. Sort out the file calling process through llaolao driver
b. Spy the file system through the driver
c. Through the ixgbe driver registration process, have a perceptual understanding of the network card driver
2. Debugging Laolao Liu driver
1. Experimental environment
1. 1 Implementation environment introduction
Use GDK8 to debug VFS videos
GDK8 related information
Experimental code
The above debugging environments are all from Shengge School
1. 2 Trigger conditions
cat /proc/llaolao
Process analysis: The process corresponding to the cat command uses /proc/llaolao as the full path to trigger a system call.
The following function call stack is the entire process of the kernel
2. Function call stack
3. Call logic analysis
4. Assembly code analysis
4.1 Frame 0 proc_llll_read() function
4.2 1st frame proc_reg_read() function
4.3 Frame 2__vfs_read() function
4.4 Frame 3 vfs_read() function
4.5 Frame 4 ksys_read() function
4.6 el0_svc_handler() function in frame 7
5. Content summary
5.1 ARMv8 assembly basics
5.2 Summary of methodology
The Linux kernel code is huge and difficult to learn and debug. This chapter introduces it from both macro and micro perspectives;
a. Macroscopic Perspective -> Business Process
b. Microscopic perspective -> Specific implementation + assembly code; understand the logical structure of assembly code and lay the foundation for kernel debugging.
Assembly code summary
a. Determine the range of the function according to the stp and ldp instructions;
b. Find the sub-function calling logic according to instructions such as bl;
c. Before calling the sub-function, prepare parameters for the function;
3 Preliminary exploration of the file system
1. Thoughts before class
1.1 Problem Domain
a. What problems does the file system solve?
b. Why abstract the virtual file system?
c. Understand the status and value of file systems from the perspective of von Neumann architecture
1.2 Personal understanding
a. The file system is responsible for file organization, storage, search and data loading; in Unix/Linux systems, all devices are files, and device management is an important part of the file system;
b. The virtual file system is exactly the same as the abstract factory pattern in the design pattern, providing users with a unified interface, shielding the underlying business logic details, and being compatible with different file systems;
c. The von Neumann architecture is a memory-centered stored program idea. Although memory is the core, the memory capacity is limited and cannot be stored permanently, so IO solves this problem very well;
2. Macro view
2.1 Logical View
2.2 File reading process
a. The process initiates a file read operation internally through a system call;
b. The system selects the corresponding system call in the system call table based on the system interrupt number;
c. Find the file descriptor through the process task_struct, check whether the file is cached, and if it is cached, read it directly;
d. If there is no cache, further find the dentry and inode information, cache the corresponding information in the memory, and record the relevant information in AddressSpace;
e. If a cache page fault occurs in AddressSpace, a page fault interrupt will be generated, and the corresponding data information in the disk will be loaded according to the inode information;
3. Understand the four major objects
In Unix/Linux systems, peripherals are abstracted into files, and the file system is responsible for the organization and management of files. VFS shields the underlying specific devices/file systems and provides users with a unified access interface. The Linux kernel abstracts four core objects for the organization and management of the file system. SuperBlock and inode are used to organize the file system. The file object records the context of the process accessing files. Dentry establishes a mapping relationship between file names and inodes to improve system access efficiency;
3.1 Super Block
There are N file systems (such as ext2) in the Linux system. The kernel uses super_block to organize all file systems and organize them in the form of a linked list;
Each file system has a unique device number, file system type, file system operation method, inode number and inode related information;
The number of inodes is determined when the file system is initialized;
Part of the code is intercepted below for reference
struct super_block {<!-- --> //
struct list_head s_list; //Super block linked list
dev_t s_dev; //The device number corresponding to the super block
unsigned char s_blocksize_bits;
unsigned long s_blocksize;
loff_t s_maxbytes; /* Max file size */
struct file_system_type *s_type; //Specific file system type
const struct super_operations *s_op; //Operation methods of specific file types
struct dentry *s_root;
/***/
\t
const struct xattr_handler **s_xattr;
spinlock_t s_inode_list_lock ____cacheline_aligned_in_smp;
struct list_head s_inodes; /* All inode nodes of a specific file system */
spinlock_t s_inode_wblist_lock;
struct list_head s_inodes_wb; /* Write back node content */
} __randomize_layout;
3.2 inode
The logical concept of files in the file system is described by inodes, and each file has a unique inode number, that is, the files listed using the ls command.
The inode node records the user, user group, access and modification time, device ID, file_operations and other related information of the inode. For field information, please refer to the intercepted code;
struct inode {<!-- -->
kuid_t i_uid; //User ID
kgid_t i_gid; //User group permissions
const struct inode_operations *i_op; //inode operation object
struct super_block *i_sb; //The super block it belongs to
struct address_space *i_mapping; //Cache data
unsigned long i_ino; //Each file has a unique ino;
dev_t i_rdev;
loff_t i_size;
struct timespec64 i_atime; //File access and modification time
struct timespec64 i_mtime;
struct timespec64 i_ctime;
\t
const struct file_operations *i_fop; //file operations corresponding to inode
void *i_private; /* fs or device private pointer */
} __randomize_layout;
3.3 file
Problem scenario: When multiple processes access the same file on the disk at the same time, and each process accesses the file from a different location, how to record the context of the access?
Super_block and inode solve the problem of file organization and storage. When multiple processes access a file, it can be roughly divided into two parts of information: ① The original information of the file, that is, the inode data; ② The context of the process accessing the file; if each process When accessing a file, copying the original copy of the file will be a waste of space; inode and file can be considered as two-dimensional information, one is the dynamic information of the process accessing the file, and the other is the static information stored in the file; but the two There will be interaction between users, a process will write files and modify static original data;
The file data structure records the context in which the process accesses the file [Note: In the 2.x kernel, the file structure is coupled with super_block]
struct file {<!-- -->
struct path f_path; // file path
struct inode *f_inode;//points to the original information of the file
const struct file_operations *f_op; //Operation method corresponding to each file
\t
atomic_long_t f_count; //File reference count
loff_t f_pos; // Access file location
struct fown_struct f_owner;
\t
struct address_space *f_mapping;
} __randomize_layout __attribute__((aligned(4)));;
3.4 dentry
The access time of memory and disk is at least an order of magnitude different. Retrieving a file may access the disk multiple times, resulting in a large number of Cache misses and seriously affecting system performance. Caching inode content through dentry can reduce memory and disk access and improve system performance;
struct dentry {<!-- -->
struct dentry *d_parent; /* parent directory */
struct qstr d_name;
struct inode *d_inode; /* Where the name belongs to - NULL is
* negative */
unsigned char d_iname[DNAME_INLINE_LEN]; /* small names */
\t
const struct dentry_operations *d_op;
struct super_block *d_sb; /* The root of the dentry tree */
unsigned long d_time; /* used by d_revalidate */
} __randomize_layout __attribute__((aligned(4)));;
4 Understand the network card driver
1.Thinking before class
1.1 Problem Domain
a. How to organize and manage hardware devices and drivers?
b. How are external devices connected to the operating system? What are the relevant system interfaces?
c. How does the computer system access device storage? How does the device communicate with the computer system?
d. What are the interactions between external devices and computer systems? How to access the physical memory of a hardware device? How do peripherals interact with the CPU?
e. What data structures does the driver model have? What is the logical relationship between them?
f. What are the core concepts in the driver model?
g. What is the logic of device discovery? What is the logic for matching devices and drivers?
1.2 Personal understanding
The kernel manages peripherals and provides an interface to the system. The computer system needs to interact with peripherals and access the physical storage space of the peripherals; the peripherals work in an asynchronous manner and interact with the CPU in an interrupt manner. Each peripheral needs to configure physical memory space, respond to interrupts, and match driver and device identification. This configuration information needs to be decoupled from business logic. In the process of computer development, a PnP solution was proposed, in the form of DTS in ARM To achieve the relevant data required to flexibly configure the hardware.
The device driver mainly includes the following aspects of logic: ① Framework access code, including driver template (ixgbe_driver) and operation method (ixgbe_netdev_ops); ② Business logic, involving business details related to specific devices, such as ixgbe, with data Link layer, physical layer, storage information
2. Network card registration process
2.1 Macro view
2.2 Registration process
2.2.1 Trigger driver registration
The kernel module is a way to access the kernel. insmod triggers the driver registration process, and module_init() is the entry function to access the kernel
2.2.2 Registering the driver in the bus
pci_register_driver() is an interface that connects the driver to the driver management framework. It triggers user business logic (ixgbe_probe) during execution and creates the /sys/devxxx directory
2.2.3 Trigger driver mounting logic
The driver mounting logic mainly includes: configuring the physical address space for the device, setting interrupts, creating the device and registering the device into the system;
2.2.4 Register the ixgbe_poll method into the napi_hash structure
The kernel uses a vector + hashtable design, which takes into account both performance and scalability. Hashtable provides convenience for dynamic registration of devices
2.2.4 DTS
DTS contains information such as interrupts, physical memory address space, device number, etc. When the operating system is loading, it parses the DTS-related configuration files. When the device driver is registered, it configures the interrupt number, physical memory and other related information according to the device number
2.3 Network card packet collection process
a. The network card receives the network packet;
b. The network card DMAs the data into the memory;
c. The network card sends a hard interrupt to the CPU, and the CPU responds to the hard interrupt and triggers a soft interrupt after simple processing;
d. The ksoftirq kernel thread finds the ixgbe_poll method registered in napi_hash according to the interrupt vector table and starts collecting packets;
e. The data frame is removed from the RingBuffer, generates skb and enters the protocol stack, entering the protocol stack processing logic;
5 Xiankan’s ideas for learning the core
5.1 Difficulties in learning
a. The kernel code is huge, the data structure is highly relevant, and the function call path is long;
b. Involves a wide range of knowledge, including hardware, compilers, assembly, etc.;
5.2 Personal understanding
a. Have macro and micro perspectives, that is, the ability to see the big picture and start small. Start from the macro perspective of business needs and technology development, establish an overall view, and sort out the code from a top-to-bottom perspective.
What is the problem scenario → What concepts are abstracted to solve the problem → What data structures are abstracted → Go deep into a certain technical detail. Take memory management as an example: why is there memory (determined by Turing machine and von Neumann architecture)? → What problems have been encountered in the development of memory (memory is expensive and not enough, it is a rare resource) → Virtual memory technology → How are physical memory and virtual memory organized, and how do they interact? → From the perspective of a user process, how are its physical memory and virtual memory organized? → Go deep into a technical detail
b. The operating system is the steward of the hardware and understands the kernel from a management perspective. For example, device management, the kernel provides a framework for device management. This part provides the role of organization and management. The framework generally includes: access module, trigger user logic module;
c. Think about the kernel from multiple perspectives; the content of the kernel is complex, and you can try to look at the computer working process, the use of resources by applications, the efficient organization of resources, the use of resources by the kernel, and the provision of quality services;
d. Have the ability to thread the needle, using problems and macro views as a skeleton to connect data structures and key processes;
e. The kernel code is just the tip of the iceberg. It is necessary to explore the problem scenarios behind the code, its ideas for solving problems, and the cleverness of the code design;
f. A philosophy that is not combined with reality is empty, and a method that is not combined with details is embroidered pillows. The method is just to see the context of things clearly. Only by having methods and details can we better grasp the core.
Six Postscript
The above is an arrangement of the ideas and content of kernel learning, which is mainly presented in the form of diagrams and does not involve too many details
Many ideas such as having an overall view of the core when learning the core come from Teacher Zhang Yinkui. Teacher Zhang is a good teacher on the road to learning the core. I would like to thank Teacher Zhang here
References:
“Understanding Computer Systems with a Debugger”
“Understanding Armv8 under the debugger”
“In-depth Understanding of Linux Networks”