Continuing from the previous article: What does the Linux kernel have and what does the memory management subsystem have? Chapter 5 – Small memory allocation (3)
References to this article:
linux process virtual address space
“Interesting Talk about the Core Principles of Linux Operating System: Part 4 Memory Management–Liu Chao”
Thanks!
2. Small memory allocation – brk and sbrk
When we talked about the sys_brk function code last time, we talked about struct vm_area_struct. This time we will analyze this structure in detail.
1. brk source code analysis
In order to facilitate understanding, the code related to the vm_area_struct structure is posted again. The definition of struct vm_area_struct is also in include/linux/mm_types.h. The code is as follows:
/*
* This struct describes a virtual memory area. There is one of these
* per VM-area/task. A VM area is any part of the process virtual memory
* space that has a special rule for the page-fault handlers (ie a shared
* library, the executable area etc).
*/
struct vm_area_struct {
/* The first cache line has the info for VMA tree walking. */
unsigned long vm_start; /* Our start address within vm_mm. */
unsigned long vm_end; /* The first byte after our end address
within vm_mm. */
struct mm_struct *vm_mm; /* The address space we belong to. */
/*
* Access permissions of this VMA.
* See vmf_insert_mixed_prot() for discussion.
*/
pgprot_t vm_page_prot;
unsigned long vm_flags; /* Flags, see mm.h. */
/*
* For areas with an address space and backing store,
* linkage into the address_space->i_mmap interval tree.
*
* For private anonymous mappings, a pointer to a null terminated string
* containing the name given to the vma, or NULL if unnamed.
*/
union {
struct {
struct rb_node rb;
unsigned long rb_subtree_last;
} shared;
/*
* Serialized by mmap_sem. Never use directly because it is
* valid only when vm_file is NULL. Use anon_vma_name instead.
*/
struct anon_vma_name *anon_name;
};
/*
* A file's MAP_PRIVATE vma can be in both i_mmap tree and anon_vma
* list, after a COW of one of the file pages. A MAP_SHARED vma
* can only be in the i_mmap tree. An anonymous MAP_PRIVATE, stack
* or brk vma (with NULL file) can only be in an anon_vma list.
*/
struct list_head anon_vma_chain; /* Serialized by mmap_lock & amp;
* page_table_lock */
struct anon_vma *anon_vma; /* Serialized by page_table_lock */
/* Function pointers to deal with this struct. */
const struct vm_operations_struct *vm_ops;
/* Information about our backing store: */
unsigned long vm_pgoff; /* Offset (within vm_file) in PAGE_SIZE
units */
struct file * vm_file; /* File we map to (can be NULL). */
void * vm_private_data; /* was vm_pte (shared mem) */
#ifdef CONFIG_SWAP
atomic_long_t swap_readahead_info;
#endif
#ifndef CONFIG_MMU
struct vm_region *vm_region; /* NOMMU mapping region */
#endif
#ifdef CONFIG_NUMA
struct mempolicy *vm_policy; /* NUMA policy for the VMA */
#endif
struct vm_userfaultfd_ctx vm_userfaultfd_ctx;
} __randomize_layout;
According to the function description, the vm_area_struct structure describes a virtual memory area, and each VM area/task has one structure. A VM area is any part of a process’s virtual memory space that has special rules for page-fault exception handling (i.e. shared libraries, executable areas, etc.).
If you want to fully understand this structure, it is not enough to just rely on a few words in function comments. You need to complete relevant knowledge, which requires “starting from the beginning” and “it’s a long story.” The so-called “starting from the beginning”, where should we start? Let’s start with opening the Linux process virtual address space.
In a multitasking operating system, each process runs in its own memory sandbox, which is the virtual address space (Virtual Address Space). Taking a 32-bit system as an example, in 32-bit mode it is a 4GB memory address block. In Linux systems, the ratio of virtual memory occupied by kernel processes and user processes is 1:3 (the ratio can be adjusted), while in Windows systems it is 2:2 (it can also be 1:3 by setting the Large-Address-Aware Executables flag) ). However, this does not mean that the kernel uses that much physical memory, it just means that it has this portion of the address space at its disposal, mapping it into physical memory as needed.
The standard memory segment layout of a Linux process in virtual memory is as shown below:
Note:
(1) The blue strips in the user address space correspond to different memory segments mapped to physical memory, and the light yellow-green areas represent the unmapped parts;
(2) Random values such as Random stack offset and Random mmap offset are intended to prevent malicious programs. Linux disrupts the layout by adding random offsets to the starting addresses of the stack, memory mapping segment, and heap to prevent malicious programs from accessing stack, library function and other addresses through calculation.
As can be seen from the above figure, the virtual address space as a whole is divided into two parts: user space (User Space) and kernel space (Kernel Space). Currently we focus on the user space part.
Part of the memory area of the user process (segmented storage content) can be mainly divided into the following parts (in increasing order from low to high addresses):
- Reserved
Located in the lowest part of the virtual address space and not assigned a physical address. Any reference to it is illegal and is used to catch exceptions when using null pointers and pointers to small integer values to reference memory. It is not a single memory area, but a general name for the address area in the address space that is protected by the operating system and prohibited from being accessed by user processes. In most operating systems, very small addresses are usually not allowed to be accessed, such as NULL. C language assigns invalid pointers to 0 for this reason, because no valid accessible data is normally stored at address 0.
In a 32-bit x86 architecture Linux system, the user process executable program is generally loaded starting from the virtual address space 0x08048000. The load address is determined by the ELF file header, and the linker default configuration can be overridden by a custom linker script, thereby modifying the load address. The address space below 0x08048000 is usually occupied by the C dynamic link library, dynamic loader ld.so and kernel VDSO (virtual shared library provided by the kernel). By using the mmap system call, the address space below 0x08048000 is accessible.
- Code Segment / Text Segment
The code segment, also called the text segment or text segment, is usually used to store program execution code (that is, machine instructions executed by the CPU). The size of this area is determined before the program is run, and the memory area is usually read-only (some architectures also allow the code segment to be writable, which allows the program to be modified).
The code segment may also contain some read-only constant variables, such as string constants, etc. In other words, the content stored in the code segment includes: executable code, string literals, and read-only variables.
- Data Segment
The data segment is usually used to store global variables and static local variables that have been initialized in the program and have an initial value other than 0. The data segment belongs to static memory allocation (static storage area) and can be read and written.
The data segment is saved in the target file (generally solidified in the image file in embedded systems), and its content is initialized by the program.
- BSS segment (Block Started by Symbol Segment)
The BSS segment is usually used to store the following content:
- Uninitialized global variables and static local variables;
- Global variables and static local variables with an initial value of 0 (depends on compiler implementation);
- A symbol that is undefined and whose initial value is not 0 (the initial value is the size of the common block).
In C language, statically allocated variables that are not explicitly initialized are initialized to 0 (arithmetic types) or the null pointer NULL (pointer types). Since the BSS will be cleared by the operating system when the program is loaded, global variables that are not assigned an initial value or have an initial value of 0 are all in the BSS.
Note:
Although both are placed in the BSS segment, global variables with an initial value of 0 are strong symbols, while uninitialized global variables are weak symbols. If a strong symbol with the same name has been defined elsewhere (the initial value may be non-zero), the weak symbol will not cause a redefinition error when linked with it, but the initial value at runtime may not be the expected value (will be overwritten by the strong symbol);
The differences between the data segment and the BSS segment are as follows:
1) The BSS segment does not occupy the physical file size, but it occupies memory space; the data segment occupies the physical file and also occupies memory space;
2) When the program reads the data in the data segment, the system will initiate a page fault and allocate the corresponding physical memory; when the program reads the data in the BSS segment, the kernel will transfer it to an all-zero page, which will not occur. Page fault, the corresponding physical memory will not be allocated for it.
- Heap
The heap is used to store memory segments dynamically allocated when the process is running. Its size is not fixed and can be dynamically expanded or reduced. The contents of the heap are anonymous and cannot be accessed directly by name, but can only be accessed indirectly through pointers.
When a process calls functions such as malloc (C)/new (C++) to allocate memory, the newly allocated memory is dynamically added to the heap (expansion); when a function such as free (C)/delete (C++) is called to release the memory , the freed memory is removed from the heap (shrunk).
The end of the heap is marked by the break pointer. When the heap manager needs more memory, the break pointer can be moved to expand the heap through system calls brk() and sbrk(), which are generally called automatically by the system.
- Stack
The stack, also known as the stack, is automatically allocated and released by the compiler, and behaves like a stack in a data structure (first in, last out, last in, first out). The stack has three main purposes:
- Provide storage space for non-static local variables declared inside the function (called “automatic variables” in C language);
- Recording maintenance information related to the function calling process is called a stack frame or a process activity record (Procedure Activation Record). It includes: function return address, function parameters that are not suitable for loading into registers, and some register values. Except for recursive calls, the stack is not required. Because the space required for local variables, parameters and return addresses can be known at compile time and allocated to the BSS segment;
- Temporary storage area, used to temporarily store partial calculation results of long arithmetic expressions or stack memory allocated by the alloca() function.
Due to the last-in-first-out (LIFO) nature, the stack is particularly convenient for saving/restoring the call scene. In this sense, we can think of the stack as a memory area that stores and exchanges temporary data.
################################################ ##########################
Supplementary knowledge: The difference between heap and stack
1) Management method
The stack is automatically managed by the compiler;
The heap is controlled by the programmer and is easy to use, but it is prone to memory leaks.
2) Growth direction
The stack expands toward lower addresses (that is, “grows downwards“) and is a continuous memory area;
The heap expands toward higher addresses (that is, “grows upward“) and is a discontinuous memory area.
3) Space size
The top address of the stack and the maximum capacity of the stack are predetermined by the system (usually the default is 2MB or 10MB);
The size of the heap is limited by the effective virtual memory in the computer system. The heap memory in a 32-bit Linux system can reach 2.9G.
4) Store content
When a function is called, the stack first pushes the address of the next instruction in the calling function (the executable statement next to the function call statement), then the function arguments, and then the local variables of the called function. After this call is completed, the local variables are popped off the stack first, then the parameters, and finally the top pointer of the stack points to the address of the first stored instruction. From this point, the program continues to run the next executable statement;
The heap usually uses one byte in the header to store its size. The heap is used to store data whose lifetime has nothing to do with function calls. The specific content is arranged by the programmer.
5) Distribution method
The stack can be allocated statically or dynamically. Static allocation is completed by the compiler, such as the allocation of local variables; dynamic allocation is performed by the alloca function to apply for space on the stack and is automatically released after use;
The heap can only be allocated dynamically and freed manually.
6) Allocation efficiency
The stack is supported by the underlying computer. A special register is allocated to store the stack address, and pushing and popping the stack are executed by special instructions, so the efficiency is high;
The heap is provided by a function library, has a complex mechanism, and is much less efficient than the stack.
7) System response after allocation
As long as the remaining space of the stack is greater than the requested space, the system will provide memory for the program, otherwise an exception will be reported and a stack overflow will be reported;
The operating system maintains a linked list of free memory addresses for the heap. When the system receives a memory allocation application from a program, it will traverse the linked list to find the first heap node with a space larger than the requested space, then delete the node from the free node linked list and allocate the node space to program. If there is not enough space (perhaps due to too many memory fragments), it is possible to call the system function to increase the memory space of the program data segment so that there is a chance to allocate enough memory and then return.
8) Fragmentation problem
The stack will not have fragmentation problems because the stack is a first-in, last-out queue. Before the memory block is popped off the stack, the last stack content above it has been popped;
The heap has a fragmentation problem, because frequent application for release operations will cause discontinuity in the heap memory space, resulting in a large amount of fragmentation and reducing program efficiency.
################################################ ##########################
When the application is loaded into the memory space for execution, the operating system is responsible for loading the code segment, data segment and BSS segment, and allocating space for these segments in the memory. The stack is also allocated and managed by the operating system; the heap is managed by the programmer himself, that is, he explicitly applies for and releases space.
After such a long period of “reverse order”, the so-called “long story”, I finally got back to the topic. Now when you look at the structure annotations, do you feel a sense of enlightenment?
The vm_area_struct structure describes a virtual memory area, and each VM area/task has one structure. A VM area is any part of a process’s virtual memory space that has special rules for page-fault exception handling (i.e. shared libraries, executable areas, etc.).
To put it more clearly, each vm_area_struct structure corresponds to a unique virtual memory area VMA in the virtual memory space. The virtual memory area is the above code segment (Text area), data segment (Data area), BSS segment (BSS area), heap, stack, etc. Each of them corresponds to a unique vm_area_struct structure (instance). As shown below:
This clarifies the overall meaning of the vm_area_struct structure. For a detailed analysis of its members, please see the next chapter.
The knowledge points of the article match the official knowledge files, and you can further learn relevant knowledge. CS entry skill treeLinux introductionFirst introduction to Linux 38078 people are learning the system