From this lecture, we entered into the study of “Principles of C Program Operation”.
Compared with the previous content, in this module, you will move from “front stage” to “behind the scenes”: from the program logic intuitively represented by C code to the specific interaction between the program and the operating system during the running process. principle. I believe that after studying this module, you will have a deeper understanding of the complete process of a C program from code writing, to compilation, and finally to being run by the operating system. Among them, the running details of the program are closely related only to the operating system in which it is located. Therefore, the principle knowledge introduced in this module is also applicable to programs written in other system-level programming languages such as Rust, C++, and Go.
Today, let’s take a look at what exactly the often-mentioned “binary executable file” is.
Executable file format
We all know that after a C code is processed by a compiler, a binary executable program can be obtained that can be run directly. On different operating systems, the executable files generated by these compilations have different characteristics, and the most obvious difference is the file suffix name. For example, on Microsoft Windows operating systems, executable files are usually labeled with “.exe” as the suffix; while on Unix-like operating systems, executable files usually do not have any suffix.
In addition, the more important difference is reflected in the organization and structure of internal data in various executable files. Generally speaking, the most common executable file formats are PE (Portable Executable) format for Microsoft Windows platforms, ELF (Executable and Linkable Format) format for Unix-like platforms, and Mach-O for MacOS and IOS platforms. Format.
In addition, it is worth mentioning that in the early days of the Unix system, executable programs at that time still used an executable file format called “a.out”. The full name of “a.out” is “Assembler Output”, which literally translates to “assembler output”. The name comes from the default output file name of the assembler originally written for the PDP-7 microcomputer by Unix system author Ken Thompson. To this day, this name is still the default file name used by some compilers (such as GCC) when creating executable files. Not only that, as the first generation executable program format, it also has important reference significance for subsequent formats such as ELF and PE.
Next, let’s take the ELF format, which is most commonly used on Unix-like platforms, as an example to see how these executable file formats store application data.
ELF file format
Different executable file formats use different ways to organize the metadata required by the application to run. But overall, their basic way of organizing data conforms to the following characteristics:Use a unified “header” to save the basic information of the executable file. Other data is divided into a series of units organized in the form of Section or Segment according to function. Of course, the ELF format is no exception.
It should be noted that in some Chinese books and articles, the words Section and Segment may be uniformly translated as “section” or “section”. But for some formats, such as ELF, they actually correspond to different concepts. Therefore, in order to ensure the accuracy of students’ understanding, I directly retain the English here.
Next, start with a real C program, and by observing the contents of the binary file corresponding to this program, you can get a preliminary impression of the basic structure of the ELF format. The C source code for this program looks like this:
// elf.c
#include <stdio.h>
int main (void) {
const char* str = "Hello, world!";
printf("%s", str);
return 0;
}
After compilation, the binary executable file corresponding to the above code can be obtained. Next, use the file command to confirm the format information of the file. The execution result of this command is shown in the figure below:
According to the information at the beginning of the command execution result, it can be confirmed that this is an executable file in ELF format. The 64-bit indicates that the file uses a 64-bit address space. In addition, the command also returns the ELF format version, whether dynamic linking is used, and the dynamic linker address used.
Next, use the readelf command to view the internal structure of the executable file. As the name suggests, this command is specifically used to read information about a specific ELF format file.
ELF header
By specifying the “-h” parameter to readelf, you can observe the ELF header contents of the file. The command execution result is shown in the figure below:
The ELF header contains relevant information describing important attributes of the entire executable file. When an application is executed, the operating system can use the relevant fields in its header to quickly find the data needed to support the running of the program.
Among them, the operating system uses the Magic field to determine whether the file is a standard ELF format file. This field is 16 bytes long in total, and each byte represents a different meaning. The first four bytes constitute the “magic number” of the ELF file format. The first byte is the number 0x7f, and the last three bytes correspond to the ASCII encoding of the three uppercase letters “ELF”. The remaining bytes also mark the bitness (such as 32/64), byte order, version number, and ABI of the current ELF file.
In addition to this field, the ELF header also contains the ELF file type, the program’s entry load address (0x4004b0), that is, the location of the first instruction that will be executed when the program is running, and the target hardware platform for which the executable file is applicable. Target operating system type and other information. As a file format, ELF is not only used in executable files, but also in static link libraries, dynamic link libraries, and core dump files. We’ll discuss this further in the “ELF File Types” section below.
ELF Section header
In the ELF format, Sections are used to store data classified by function in executable files. In order to facilitate the operating system to find and use these data, ELF organizes the relevant information of each Section in its corresponding Section header. , many consecutive Section headers form the Section header table.
The Section header table records some basic information of each Section structure, such as the name of the Section, its length, its offset position in the executable file, and its read and write permissions, etc. When the operating system is actually used, it can directly obtain the offset position of the Section header table in the entire binary file and the size of the table from the ELF header.
By observing the ELF header information in the above figure, we can know that the ELF file contains 30 Section headers, which correspond to 30 Section structures, and the first Section header is located at the 15512th byte of the file starting offset. . By specifying the “-S” parameter for the readelf command, you can view the detailed information of all these Section headers.
The execution result of this command is shown in the figure below (limited by space limitations, I only list the more important contents of the Section header):
As you can see, here I have mainly filtered out the detailed content of the headers corresponding to the four Sections: .text, .rodata, .data, and .bss. If you still remember the knowledge about data storage locations that I introduced in Lecture 02 and Lecture 10, you will definitely be familiar with these four Sections. Among them, .text is mainly used to store the machine code corresponding to the program; .rodata is used to store read-only constant values used in the program; .data contains the values of global variables or static variables that have been initialized in the program; and .bss Global or static variable values with an initial value of 0 are stored in them.
The location of the actual data of each Section is also marked in the Section header. For .rodata, its actual contents can be seen at byte offset 0x658 in the file, or at offset 0x400658 in the process VAS when the program is running. Here we can use the objdump command to verify it.
The objdump command is a tool that can be used to view the contents of binary files. By specifying the “-s” parameter for it, you can view the complete contents of a Section. The execution result of this command is as follows:
It can be seen that the string data “Hello, world!” used in the C code is placed at an offset of 0x10 bytes from the beginning of the Section.
In the ELF format, numerous Sections form a static view describing the contents of the ELF file. A major role of static views is to complete the “linking” process throughout the entire life cycle of the application. Linking means that different types of ELF format files are integrated with each other, and finally generate an executable file, and the file can run normally. According to the period when the integration occurs, links can be divided into “static links” and “dynamic links”. This part of the content will be introduced in depth in Lectures 27 and 29.
ELF Program header
In addition to the static view composed of Sections, numerous Segments form a dynamic view that describes the executable file. Segment specifies how the data should be organized within the VAS of the process when the application is actually running. Similarly, we can also observe the Segment situation of the executable file corresponding to the program at the beginning of this lecture by specifying the “-l” parameter for the readelf command. The execution result of this command is shown in the figure below:
Similar to Section, each Segment also has its corresponding header to describe some basic information of the Segment, which is generally called the Program header.
The Program header contains information such as the type, offset address, size, alignment, and permissions of each Segment. Among them, Segments marked as “LOAD” type will be actually loaded into the VAS of the process when the program is running, while the remaining Segments are mainly used to assist the normal operation of the program (such as dynamic linking). Not only that, the specific offset position and size of the Program header table are also placed in the ELF header, so the operating system can quickly get this information at any time when needed.
Generally speaking, there is a certain correspondence between each Segment and Section. For example, in the picture above, the first LOAD type Segment contains multiple Sections including .text and .rodata, while .data is included in the second LOAD type Segment. If you observe further, you will find that the first LOAD Segment has permissions of “RE”, which means it can read and execute; while the second LOAD Segment has permissions of “RW”, which means it can read and write. So why is it distributed like this? I believe you must have the answer at this time.
In addition, we observed that the content contained in the first LOAD Segment corresponds to an offset of 0 within the executable file. This means that when the operating system executes the program, in addition to the Sections corresponding to each Segment, it will also load the ELF header of the file into memory together with its Program header table.
At this point, we have a general understanding of the internals of the ELF format binary executable file.
Although the above content does not cover all the design details inside ELF, for daily learning, it is enough to master the basic structure of the ELF format (the static view and dynamic view corresponding to the ELF header, Section and Segment respectively). You can refer to the figure below to review these contents more intuitively:
Next, let’s take a look at how to use C language for ELF programming.
ELF Programming
Currently, in Linux systems, you can directly use the header file elf.h provided by the kernel to perform application programming in the ELF format. In this header file, various structural types and macros for different ELF conceptual entities are predefined.
For example, for the ELF header, we can directly use the ElfN_Ehdr (N may be 32 or 64 depending on the system) type defined in the header file to represent it in the code. The following C code shows how to use this type:
#include <stdio.h>
#include <elf.h>
void print_elf_type(uint16_t type_enum) {
switch (type_enum) {
case ET_REL: printf("A relocatable file."); break;
case ET_DYN: printf("A shared object file."); break;
case ET_NONE: printf("An unknown type."); break;
case ET_EXEC: printf("An executable file."); break;
case ET_CORE: printf("A core file."); break;
}
}
int main (void) {
Elf64_Ehdr elf_header;
FILE* fp = fopen("./elf", "r");
fread( & amp;elf_header, sizeof(Elf64_Ehdr), 1, fp);
print_elf_type(elf_header.e_type); // "An executable file."
fclose(fp);
return 0;
}
In this C code, we simply open the binary executable file named “elf” in the current directory, and directly read the data corresponding to the Elf64_Ehdr type size from its beginning, and store it in a variable named elf_header middle. Finally, by accessing the e_type field of the structure object, we can get the type of the ELF file. In the elf.h header file, many macro constants representing ELF specific indicators are defined. For example, if the value of the e_type field is equal to the macro constant ET_EXEC, it means that the file is an executable file.
It can be said that the elf.h header file contains various custom types that can be used to describe all legal ELF format files. Therefore, it is a good way to gain a deeper understanding of the design details of ELF through selective reading and practice. If you want to know more about this header file, you can view the Linux help documentation about this header file through the command “man 5 elf”.
ELF file type
Before the end of this lecture, let’s look at another question: As a file format, what types of files is ELF used by?
Through the programming practice in the previous section, we can know that: in the definition of the elf.h header file, the ELF format can be applied to four different file types, and their corresponding macro constants are ET_REL, ET_DYN, ET_EXEC, and ET_CORE. . Here, the above four macro constants and their corresponding ELF file types are organized in the following table for your reference:
Although these four ELF file types have different names, the overall organization of their internal data follows the same ELF file format standard. The difference is that since the functional positioning of each file type is different, its internal ELF format structure is also different.
Take relocatable files, for example. This type of file can be used to support incremental development of large projects, that is, the functions in the program that can be modularized and distributed independently are compiled separately and formed into relocatable files. Application code that relies on the implementation of these functions can be compiled together with these relocatable files. Finally, after static linking processing by the linker, the executable file corresponding to the program can be obtained. The advantage of this approach is that every time the program functionality changes, the code that needs to be recompiled can be constrained to the minimum scope.
Relocatable files only contain Section-related information, but do not have Program headers and other ELF structures to support their operation, so files of this type cannot be run directly. One of the main functions of static linking is to collect the function implementations that need to be used in each relocatable file based on the program’s calling status in the main function, and finally generate the corresponding executable file containing the Program header. So, how exactly does this process work? The answer will be revealed in the next lecture.
Summary
The content of this lecture mainly uses executable binary files as the entry point. First, it introduces several common executable file formats on different operating systems, and then takes the most common ELF format as an example to conduct a more in-depth exploration of its composition details.
The basic structure of the ELF file format can be divided into three main parts: ELF header, Section and Segment. Among them, each Section contains various types of data divided according to functional categories and used to support ELF functions. Together, these data form a static view of the ELF file that is used to support the linking process of the ELF file. Numerous Segments form a dynamic view of the ELF file. This view describes the distribution of the relevant data that the ELF file depends on within the process VAS when it is loaded and executed by the operating system.
In addition to observing the internal conditions of ELF files through tools such as readelf that come with the operating system, you can also use the elf.h header file provided by the Linux kernel. Many ELF element types are predefined in this header file, which can help us write ELF analysis and processing tools that meet our own needs.
Finally, we explored the differences between several different ELF file types. Relocatable files, shared object files, executable files, and core dump files, although they have different application scenarios and internal data compositions, are all ELF file types and follow the basic rules of the ELF format. .