1. Preface
With the rapid iteration of project versions, APP testing has the following characteristics:
- First of all, there are many and detailed function points, the testing workload is large, and it is easy to miss;
- Secondly, code modules are often changed, regression testing is frequent, and testing is repeated and inefficient;
- Finally, the data environment is diverse, the user scenarios are complex, and it is difficult to fully cover functional regression.
In order to save costs and ensure efficient and high-quality iterations, we need to adopt a more efficient testing method. App automated testing is a more efficient method.
In order to solve the many problems encountered in the previous automatic testing practice (low code reuse rate, cumbersome Case development and data construction, difficult problem location, high learning costs, etc.), we re-implemented an APP automatic testing framework to solve the related pain points. . This article will focus on technology selection, design ideas and the specific practice of Baidu Takeaway App.
2. Automated testing framework technology selection
Whether automated testing can be effectively carried out in a project, the automated testing framework is the key. Therefore, how to build a stable and easily scalable automated testing project is of great significance to agile testing. When designing the framework, the functions already provided by automated testing tools should be used as much as possible to avoid duplication of development and reduce development costs.
Through in-depth analysis of the principles of existing automated testing tools and comparison of their advantages and disadvantages, the above-mentioned problems encountered in automated testing are solved based on two automated testing frameworks: Appium and TestNG.
- First, by using TestNG combined with the use of csv, the test case data is converted into data in the test code, which reduces the number of tools for testers to enter data and prepare data;
- Thirdly, through the encapsulation of appium, the page elements used in the test are encapsulated into objects according to the object-oriented idea, which enhances the reuse rate of the test code, reduces the tester’s burden on the implementation of the underlying code, and improves the efficiency of writing test code;
- Finally, we introduced failed re-runs, failed screenshots, and generated test reports through reportng to gradually improve the testing process and improve the speed of locating problems;
TestNG
Testng is an open source automated testing framework that introduces many new innovative features such as dependency testing, grouping concepts to make testing more powerful and easier to do. Designed to cover all categories of testing: unit, functional, end-to-end, integration, etc. The TestNG framework can very well help us complete the page automation work of WebDriver + java. Through the flexible operation of various annotations, it can make your test cases more perfect and customize test cases that meet the requirements.
- TestNG is a testing framework designed to simplify a wide range of testing needs, from unit testing to integration testing. This is the starting point of TestNG design. It is not only unit testing, but also can be used for integration testing. The design goals are different. Compared with junit, which is only suitable for unit testing, TestNG undoubtedly goes further. It can be used for integration testing. This feature is the most important reason why I choose TestNG.
- There are three typical steps in the testing process. Compared with junit (4.0), there is one more step: adding test information to the testng.xml file. Test information, especially test data, is no longer hard-coded in the test code. The advantage is that there is no need to modify the code/compile when modifying the test data, which helps to introduce testers to unit testing/integration testing.
- The basic concept is that compared to Junit’s TestCase/TestSuite, TestNG has three levels: suite/test/test method, which clearly distinguishes test/test method.
Appium
Appium is an open source, cross-platform testing framework that can be used to test native and hybrid mobile applications. Appium supports iOS, Android and FirefoxOS platform testing. Appium uses WebDriver’s json wire protocol to drive the UIAutomation library of Apple systems and the UIAutomator framework of Android systems. Compared with other mobile automated testing tools, Appium testing can use any language, including Python, Ruby, Node.js, Objective-C, etc., because it calls Selenium’s client library.
Now I have also found a lot of test friends and created a communication group to share technology, sharing a lot of technical documents and video tutorials we collected. If you don’t want to experience the feeling of not being able to find resources when studying on your own, having no one to answer your questions, and persisting for a few days before giving up. You can join us to communicate. And there are many technical experts who have made certain achievements in automation, performance, security, test development, etc. Share their experience, and also share many live lectures and technical salons You can learn for free! Focus on it! Open source! ! ! QQ group number: 110685036 [password: csdn999]
3. Design ideas of automated testing framework
The test design process and the test automation framework must be developed as two separate entities.
The testing framework should be independent of the application;
The testing framework should be easy to extend, maintain and enhance;
Testing strategy/design should hide the complexity of the testing framework from testers.
4. Introduction to automation framework
The framework is developed based on Selenium WebDriver open source technology. This framework uses the Maven tool for project management, uses the TestNG tool to organize tests, and uses CSV files to store test data to achieve the separation of test data and test cases, facilitate test data management, reduce the maintenance cost of automated scripts, and achieve data drive. In addition, the framework also encapsulates a wealth of Selenium method keywords, draws on the QTP syntax structure, and implements intuitive and clear structured code syntax, such as: Page.Item.Operate, reducing redundancy and duplication of automated code. Use Jenkins to perform CI testing, implement the Schedule and Report functions of test tasks, manage virtual machine nodes through Jenkins Master/Slave mode, and realize distributed and concurrent execution management of multi-tasks and multiple machines, thereby improving testing efficiency.
- The benefits of this framework are: 1. Build a reusable and stable code set. By encapsulating appium, use case execution and data invocation are separated, parameterized configuration of common information is provided, and a unified interface is provided; 2. Modular management of automated test cases. It is mainly implemented based on the characteristics of the TestNG tool that supports parameter testing and dependency testing; 3. Test result analysis and statistics. Use the Jenkins tool to establish continuous integration, run automated test projects regularly, and display test results in a customized form.
Test framework layering
Based on UI testing, we hope that in addition to supporting web testing, we can also support app testing, and may also need interface testing. We need to consider layering issues and divide the testing framework into three layers. The upper layer manages the development, execution and maintenance of the entire automated test. In relatively large projects, it plays an important role. It can manage the entire automated test, including the order of execution of automated test cases, maintenance of test scripts, and centralized management of tests. Use cases, test reports and test tasks, etc. The lower layer is mainly responsible for the development of test scripts, making full use of relevant testing tools, building test drivers, and completing test business logic.
The first layer: data layer
That is, the test data required to execute the use case, such as merchant name, space name, URL, etc., these data are used to support the execution of the entire script. For the data layer, a data-driven approach is adopted.
Second layer: driver layer
This layer mainly encapsulates various drivers. For example, for web page testing, we use the selenium-webdriver development kit, and for app testing, we use the appium development kit. We encapsulate at this layer, by calling selenium-webdriver, the native method provided by appium, encapsulates it into a highly readable method and adds a fault-tolerant mechanism. Even if we want to switch to other third-party packages in the future, our test case layer and support layer methods do not need to be modified in any way. Just modify the way the driver layer is implemented. On the first layer, we mainly implement two aspects of encapsulation, one is driver encapsulation, and the other is encapsulation based on base class natural language functions.
driver package
We need to encapsulate it, depending on whether the parameters are based on web testing or app testing. for example:
Base class encapsulation
It mainly encapsulates various highly readable methods and also encapsulates element positioning identifiers and drivers. In order to support web page testing and app testing, we need two base classes, one for web page operation base class and one for app operation base class. At the same time, for the sake of consistency in web and app operations, we require that the methods provided externally must have the same name and the same parameter type and number of parameters for commonly used methods.
An example of the APP base class is as follows:
By encapsulating the driver and base classes, the driver layer implements support for web page testing and app testing, and provides a unified method for both tests, making it convenient for users to use the same method to test apps and web.
The third layer: test case layer
This layer is the specific implementation of the test case. Just like the case written above, the test case is implemented in a way close to natural language.
The fourth layer: support layer
This layer mainly provides support for workflow, general tools, and element libraries, making it easy for the test case layer to call directly.
- Workflow: Mainly encapsulates project-specific public methods that need to be frequently used in test projects for direct call by the test case layer. For example, frequently used actions such as user login, registering a user, searching for users, etc.;
- General tools: Provide some general methods, such as generating specified Page classes, file reading operations, DB operations, http operation support, etc.;
- Element library: positioning expression of each page element (xpath, id, name, css, link_text, etc. expressions). Our test cases all operate on each element. Think of every element on every page as a specific class that inherits the base class. Therefore, our first step is to find this element and locate this element. All elements of the test project are placed here.
Level 5: Result Saving Layer
The logs and results of the test script are displayed in a customized way. ReportNG is used here, which can enrich the display form of the test results and help the team locate and solve problems faster.
5. Analysis of framework technical points
5.1, PO mode
5.1.1. Problems encountered
After using webdriver for a period of testing, you will find that when locating elements on a certain page, the program lines are filled with id(), name(), xpath() and other methods, which will make the test program unreadable. It has poor performance and is inconvenient for later maintenance and modification.
Although we can make the program easier to understand by adding comments, we still cannot fundamentally solve this problem. We can avoid direct calls to these methods by secondary encapsulation of these methods. Although reuse can be achieved through method encapsulation. However, we found that the logical processing of page elements and the independence of test data cannot be achieved through encapsulation.
5.1.2. Solution to the problem: introduce PO
Page Object pattern is a test design pattern in Selenium, which refers to the objects on the UI interface used to interact with users. The main purpose is to design each page as a Class, which contains the elements that need to be tested in the page (buttons, input boxes, titles, etc.), so that the page elements can be obtained by calling the page class in the Selenium test page, which cleverly avoids When the page element ID or position changes, the test page code needs to be changed. When the page element id changes, you only need to change the attributes of the page in the test page Class. Reduce redundant code and improve the maintainability of test cases by encapsulating interface elements.
Generally speaking, for a Page Objects object, it has two characteristics:
- Self element (WebElement)
- Implement functions (Services)
Carefully analyze the test scenario and extract the core behavior of UI testing, which is nothing more than:
1. Checkpoint:
Whether the page element exists;
Whether the displayed content of page elements is correct;
Whether the page element is available;
…
2. Auxiliary functions:
Wait for the element to appear;
Click on a page element;
Enter content for the element;
…
After analyzing the core behaviors extracted, we found that these behaviors are basically operations performed on each page element. Then we can do the following actions:
Treat page elements as an object and encapsulate them into a class;
Encapsulate the core behaviors analyzed above into base class methods. Then make sure that any page element inherits this base class;
This base class provides method names similar to natural language. By calling these methods, you can clearly know what checks and behaviors the test case is doing, which can greatly improve the readability of the test case.
The main purpose of this base class is to extract common checkpoints and auxiliary methods of elements in UI testing, and encapsulate them into very easy-to-understand methods with exception handling capabilities.
After sorting out the above ideas, the test case can be rewritten as follows:
In the actual use process, QA who is not familiar with the code can be responsible for the implementation of test cases, and the underlying method packaging can be done by students with more experience.
5.2, Data-driven
The data-driven automated testing framework is a framework that reads input and output test data from a certain data file (such as ODBC source file, Excel file, Csv file, ADO object file, etc.), and then passes it in in advance through variables. in recorded or hand-written test scripts. Among them, these variables are used to pass (input/output) test data used to verify the application. In this process, the reading of data files, test status and all test information are written into the test script; the test data is only included in the data file, not in the script. The test script is just a “driver”, or in other words A mechanism for transferring data.
- Fill in the test data in the data file:
- Generate Page class:
- Initialize page elements in the Page class:
The benefits of being data-driven are:
Test scripts can be created simultaneously while the application is being developed, and when the application functions change, only the scripts for the business functions need to be modified;
Utilize model-based design to avoid duplicate scripts and reduce the cost of establishing or maintaining scripts.
5.3. Failed rerun and failed screenshot mechanism
During the automated testing process, automated testing often fails due to slow network and server response, long JS effects, and long page rendering times. For such scenarios, this framework has designed a set of NRetry mechanisms, that is, after a certain case fails to run, it will be rerun N times, and N can be customized. If it succeeds once in N times, it will continue to run. If it fails all N times, it will take a screenshot, throw an error, and stop running. The NRetry mechanism can, to a certain extent, reduce the instability of automated execution caused by slow network and server responses.
5.3.1. Automatic screenshot of failure
- Create a new Java class that inherits TestListenerAdapter:
- Rewrite onTestFailure, onTestSkipped and other methods, and add screenshot operations to these methods:
- Configure the listener class you wrote in the testng.xml file:
5.3.2. Automatic rerun after failure
When running automated test cases, some abnormal situations often occur that cause the test case to fail. So we may want to rerun the failed test case again. In the framework, we combine TestNG to implement this function.
- Create a new TestNGRetry class to implement automatic rerun logic if the use case fails:
- Add the use case rerun listener RetryListener to automatically rerun the use case if it fails:
- Configure the listener you wrote in the testng.xml file:
5.4, ReportNG
TestNG’s default HTML report, although its default report is comprehensive, is not easy to understand. Therefore, we use ReportNG to replace TestNG’s default report.
ReportNG provides an easy way to view test results, with the ability to colorize the resulting codes. You can also replace the default output style by modifying the CSS file. In addition, ReportNG can also generate XML output in Junit format.
Since we are using maven, we mainly look at the situation of pom.xml:
maven-surefire-plugin This plug-in is mainly used for testng. We use this plug-in to generate our test report directory in the corresponding directory ./target/${timestamp}. We can see the structure of this directory.
This is actually the generation path of reportng’s test report. But it will be difficult if we want to send it by email, because the html content needs to be added to additional css and js files. In fact, mail does not support external css and js files.
HTML generation
1. Define HTML template
View indexMain.html.vm:
When using ReportNG to replace the built-in report of TestNG, if the report contains Chinese, since the Log in ReportNG does not support Chinese, the ReportNG.jar source code is modified to support Chinese display in the report.
5.5. Log collection
Logging is an important part of software development. The log information of the automated execution process is very important for the analysis and location of failed use cases and the tracking and recording of the entire process.
Compared with simple output printing, this framework integrates the mainstream log collection tool log4j. Log4j is highly configurable and can be configured via external files at runtime. Collect log information (such as database, file, console, UNIX system log, etc.) by configuring the log4j.properties file, defining log level content and log output path, providing fast debugging, convenient maintenance, and the runtime information structure of the application. storage.
Configuration file
Log4j can be set dynamically through a java program. The obvious disadvantage of this method is: if you need to modify the log output level and other information, you must modify the java file and then recompile, which is very troublesome; log4j can also be set through the configuration file, which is currently supported Configuration files in two formats:
xml file;
properties file (recommended).
5.6. Email sending
The sending of test reports can be implemented in combination with Jenkins, which can be achieved through simple configuration. But if the team has not set up Jenkins or sometimes Jenkins is not available, how should we deal with this part?
Since email cannot rely on Jenkins, you must implement this part yourself. So we still have to rely on some third-party jar packages. We configure it in pom.xml.
6. Follow-up TODO
In the subsequent version evolution, automated testing, code security scanning, multi-machine parallel testing, and continuous release will be added to the overall process to truly achieve continuous delivery throughout the entire process.
6.1. Add automated testing to nightly builds
Nightly builds trigger an automated build process at scheduled intervals, but these builds are simply code compilations without reliable or repeatable functional testing. Later, we will consider Appium combined with Jenkins to implement post-build automated testing.
Whenever code updates are committed to git, the build server triggers a build, which runs a script to compile the application and run a series of automated unit tests and/or integration tests. The automated test results can clearly show which functional features passed and which failed. Whether a change is submitted or a build is triggered regularly at night, the application is automatically deployed to the test environment for the QA team to test.
6.2. Combining Jenkins with STF to achieve multi-machine parallel testing
After the Jenkins build script is completed, add the Android device connected to the computer without the stf component installed and map it to the machine with the stf platform service. Push the integration test case to the STF platform, and then distribute it to the runnable device by STF. Multi-machine parallel testing.
Advantages brought by STF performing APPIUM testing
First, parallel Appium tests can be executed on real machines. Since the initial use of Appium was to execute tests on the simulator or just one device at a time, STF has expanded Appium on the original basis to have the ability to execute tests on up to hundreds of real machines at the same time.
Second, there is no need to configure the Desired Capabilities of any device. This method is simple and reduces the different types of errors that occur when editing scripts.
Third, executing tests on STF allows users to instantly view the test status. In other words, you can view the progress of test execution, instant error feedback, and retain and review all test projects, test scripts and test results (test screenshots, test logs, performance data, etc.)
6.3. Code quality measurement
6.3.1. Why code analysis is necessary
When paying attention to code quality, it is necessary to arrange manual code review, but 100% code review requires the investment of personnel and consumes a lot of work, while tool automatic inspection only requires a small amount of manual configuration.
The main reason is to improve the code quality and understand the impact that RD’s mistakes made during the coding process may have on the functional logic. At the same time, it also promotes RD to make its code more readable and maintainable, so we draw on the principles of continuous improvement. process, hoping to gain something in the process.
6.3.2, Jenkins introduces Sonarqube for continuous code review
Sonar is an open source platform for code quality management, used to manage the quality of Java source code. Through the plug-in mechanism, Sonar can integrate different testing tools, code analysis tools, and continuous integration tools, such as pmd-cpd, checkstyle, findbugs, and Jenkins. These results are reprocessed through different plug-ins, and changes in code quality are measured in a quantitative way, so that code quality management can be easily carried out for projects of different sizes and types.
6.4, email-ext implements Jenkins email notification function
Configure email notifications in Jenkins. Jenkins provides two ways of configuration.
One is the built-in default email notification in Jenkins, but it has many limitations. For example, its email notification cannot provide detailed email content, cannot define the format of sending emails, cannot define flexible email receiving configurations, etc.
In this case, subsequent considerations can be used to customize all aspects of email notifications through the Email Extension Plugin. For example, when sending an email, you can customize who to send it to, what specific content to send, etc.
Finally, I would like to thank everyone who has read my article carefully. Looking at the increase in fans and attention, there is always some courtesy. Although it is not a very valuable thing, if you can use it, you can take it directly!
Software testing interview documents
We must study to find a high-paying job. The following interview questions are from the latest interview materials from first-tier Internet companies such as Alibaba, Tencent, Byte, etc., and some Byte bosses have given authoritative answers. After finishing this set I believe everyone can find a satisfactory job based on the interview information.
The knowledge points of the article match the official knowledge files, and you can further learn relevant knowledge. Python entry skill treeDesktop application developmentTkinter388920 people are learning the system