Directory
3.2 Program instrumentation method
3.2.1 Object code instrumentation
3.2.2 Source code instrumentation
Tip: HeisenBugs
Similarities and differences between black box testing and white box testing
3.2 Program instrumentation method
Program instrumentation is a widely used software testing technique, proposed by Professor J.C.Huang. Simply speaking, program instrumentation is a method of inserting test code into the program under test to achieve the purpose of testing. The inserted test code is called probe. According to the time of test code insertion, program instrumentation can be divided into object code instrumentation and source code instrumentation.
3.2.1 Object code instrumentation
Object code instrumentation refers to a test method that inserts test code into the object code (binary code) to obtain program running information, also known as dynamic program analysis method.
Object code instrumentation is of great significance for memory monitoring, instruction tracking, error detection, etc. when the program is running. Compared with the logic coverage method, object code instrumentation does not require code recompilation or linking during the test process, and the format of the object code has nothing to do with the specific programming language, but is mainly related to the operating system. Therefore, object code instrumentation has a wide range of applications. use.
1. The principle of object code instrumentation
The principle of the object code instrumentation method is to establish an intermediate layer between the program running platform and the underlying operating system, through which checks the execution program and modifies instructions, developers, software Analysis engineers observe the running program to determine whether the program has been maliciously attacked or has abnormal behavior, thereby improving the overall quality of the program.
2. Execution mode of object code instrumentation
Since the target code is an executable binary program, the instrumentation of the target code can be divided into two cases: one is to instrument the unrunning target code, insert the test code from the beginning to the end, and then execute program. This approach is suitable for code coverage testing when a complete system or simulation needs to be implemented. Another situation is to insert test code into the running program to detect the running status information of the program at a specific time.
Object code instrumentation has the following three execution modes:
(1) Just-In-Time mode (Just-In-Time): The original binary or executable file has not been modified or executed, stores a copy of the modified part of the binary code generated file in the new memory region, executes only the modified portion of the object code when testing.
(2) Interpretation Mode: In Interpretation Mode, the object code is regarded as data, and the test code inserted by the tester is used as the interpretation language of the object code instruction. Whenever an object code instruction is executed, the program will be Find and execute the corresponding replacement instruction in the test code, and the test can obtain the running information of the program through the execution information of the replacement instruction.
(3) Probe Mode: Probe Mode use new instructions to overwrite old instructions for testing, this mode is more useful in some architectures (such as x86).
Two cases: instrumentation of non-running target code, and insertion of test code into a running program;
Three modes: immediate mode, interpretation mode, detection mode.
3. Object code instrumentation tool
Since the target program is an executable binary file, it is impossible to manually insert the code, so the target code instrumentation is generally implemented through the corresponding instrumentation tool, and the API provided by the instrumentation tool can provide users with access instruction. Common object code instrumentation tools mainly include:
(1) Pin-Dynamic Binary Instrumentation Tools (Pin for short)
Pin is a free framework developed by Intel Corporation, which can be used for binary code detection and source code detection. Pin supports IA-32, x86-64, and MIC systems, and can run on Linux, Windows, and Android platforms. Pin has modules such as basic block analyzer, cache simulator, and instruction trace generator. Using this tool, you can create program analysis tools, monitor program running status information, etc. Pin is very stable and reliable, often used for large-scale program testing, such as Office software, virtual reality engine, etc.
(2) Dynamo RIO
DynamoRIO is a licensed dynamic binary code inspection framework, as an intermediate platform between applications and operating systems, it can realize code transformation of any part of the program during program execution. DynamoRIO supports IA-32, AMD64, Arch64 systems and can run on Linux, Windows and Android platforms. DynamoRIO includes memory debugging tools, memory tracking tools, instruction tracking tools, etc.
3.2.2 Source code instrumentation
Source code instrumentation refers to after a complete lexical and grammatical analysis of the source file, confirm the location of the instrumentation, and implant the probe code. Compared with object code instrumentation, source code instrumentation is more targeted and precise. Source code instrumentation model:
It can be seen from the figure that source code instrumentation is completed before program execution, so source code instrumentation will generate probe code overhead during program running. Compared with object code instrumentation, source code instrumentation is less complicated to implement. Source code instrumentation is a testing technology at the source code level. The probe code program has good versatility. Programs written in the same programming language can use a probe code program to complete the test.
The following uses a small case to explain source code instrumentation. The case is a division operation:
1 #include <stdio.h>
2 #define ASSERT(y) if(y){ printf("error file: %s\
",__FILE__);\
3 printf("Line %d:\
",__LINE__\);
4 printf("Prompt: the divisor cannot be 0!\
");\
5 } // define ASSERT(y)
6 int main()
7 {
8 int x,y;
9 printf("Please enter the dividend: ");
10 scanf("%d", &x);
11 printf("Please enter the divisor:");
12 scanf("%d", &y);
13 ASSERT(y==0); // Inserted stub (ie probe code)
14 printf("%d",x/y);
15 return 0;
16 }
In order to monitor whether the input of the divisor of the division operation is correct, insert the macro function ASSERT(y) in the 13th line of the code, and when the divisor is 0, print information prompts such as the cause of the error, the error file, and the number of error lines. The macro function ASSERT(y) uses the macro definition “__FILE__” of the C language standard library to prompt the error file, and “__LINE__” to prompt the location of the file error.
After the program runs, prompts to input the dividend and divisor. After inputting the divisor, the program macro function ASSERT(y) judges whether the divisor is 0. If the divisor is 0, an error message will be printed and the program will end; if the divisor If it is not 0, perform a division operation and print the calculation result. Design test cases according to the division operation rules, as shown in the table:
Compile and link the instrumented C source program, generate an executable file and run it, and then input the test case data in the table to observe whether the actual execution result of the test case is consistent with the expected result.
The program instrumentation test method effectively improves the code test coverage, but the instrumentation test method will bring code expansion, low execution efficiency and HeisenBugs. Generally, the code expansion rate after instrumentation is between 20% to 40%, or even 100%, will cause the instrumentation test to fail.
Tip: HeisenBugs
HeisenBugs, or Heisenberg Bugs, are software defects that have a low recurrence rate and that disappear or change behavior when people try to investigate them. In the actual development of software testing, this kind of defect is also relatively common. For example, after the tester tests a defect and submits it to the developer, the developer performs the defect reproduction steps but cannot get the reported defect because the defect has disappeared or other defects have occurred. defect.
Similarities and differences between black box testing and white box testing
1. Comparison between black box testing and white box testing
Black box testing don’t need to consider the internal logic structure, only need to verify whether the external functions of the software meet the actual needs of users. Black box testing can find the following defects. (1) Defects in external logic functions, such as incorrect information displayed on the interface. (2) Compatibility errors, such as system version support, operating environment, etc. (3) Performance issues, such as running speed, response time, etc.
White box testing can design test cases to cover as much as possible the branch statements in the program, and analyze the internal structure of the program. White box testing is often used in the following situations. (1) The source program contains multiple branches. When designing test cases, it is necessary to cover all branches as much as possible to improve the test coverage. (2) The memory leak check is quick. Black-box testing can only find memory leaks when the program is running for a long time, while white-box testing can find memory leaks immediately.
2. Testing phase
Black-box testing and white-box testing are used differently in different testing stages. The usage of the two in different stages is as follows:
In the testing process, the combination of black box testing and white box testing will greatly improve the quality of software testing.