1. Foreword
Automated testing is to convert manually driven test behavior into automatic machine execution. Usually the operation is to write code under a certain framework to realize automatic discovery and execution of use cases. It is hosted on the CI/CD platform and triggered by conditions or manually. Carry out regression testing & online monitoring to replace part of manual testing;
Different automation frameworks are suitable for different projects. This series of automation articles will introduce the automation frameworks used in actual work one by one.
Frameworks used for automated interface testing:
● Postman + Newman + Allure + Jenkins
● Httprunner + Request + Pytest + Alluret + Jenkins
● JMeter + Ant + Allure + Jenkins
● Pytest + Request + Allure + Jenkins
Frameworks used for automated UI testing:
● Selenium + Pytest
● Appium + WDA + TestNG
● Minimum+pytest
This chapter first provides a comparative introduction to the four frameworks of automated interface testing. The subsequent series of articles will introduce the detailed construction process and packaging process.
2. Introduction to automated interface testing
1. The necessity of interface testing
Interface testing has an extremely efficient cost-benefit ratio and is an important part of shifting testing to the left.
Interface testing brings efficient defect detection and quality supervision capabilities to high-complexity platforms. The platform is complex and the larger the system, the more obvious the effect of interface testing.
In general, interface testing is the best solution driven by the intrinsic requirements of high-complexity system quality and low-cost economic benefits, which is mainly reflected in the following three aspects:
● Saves testing costs
● Interface testing is different from unit testing
● Higher returns
2. The necessity of automated interface testing
● Automated testing is a process that converts human-driven testing behavior into machine execution. It mainly involves writing codes and scripts to allow the software to run automatically, discover defects, and replace part of manual testing;
● When the server changes functions, adds new functions, switches the test environment, or releases a new program, to avoid new code causing existing functions to become unavailable, a full return of the interface is required. Manual triggering is not timely and is easy to miss.
● Interface automation is connected to continuous integration, and server-side release triggers the running of interface test code to detect problems as early as possible.
● Extract some interface test cases, run the program regularly, monitor common online business operations, and find and fix them in time.
3. Applicable scenarios for automated interface testing
● Smoke test before submission
● Pre-production/pre-launch regression testing
● Regression testing after changes to the underlying logic of the server
● Online server availability monitoring
4. Results
Through automated testing, online availability monitoring and regression testing can save a lot of manual testing time, reduce costs and increase efficiency. Many problems have been discovered both online and offline.
Now I have also found a lot of test friends and created a communication group to share technology, sharing a lot of technical documents and video tutorials we collected. If you don’t want to experience the feeling of not being able to find resources when studying on your own, having no one to answer your questions, and persisting for a few days before giving up. You can join us to communicate. And there are many technical experts who have made certain achievements in automation, performance, security, test development, etc. Share their experience, and also share many live lectures and technical salons You can learn for free! Focus on it! Open source! ! ! QQ group number: 110685036
3. Overview of automated interface testing framework
1. Comparison of several frameworks
There are many other frameworks. This time we only compare several frameworks that are used in actual work.
2. The necessity of automated interface testing framework
● Provide unified use case templates to simplify business access costs
● Handle some exceptions and error handling work in scripts
● Implement some common functions and simplify the script development process
● Test scene recovery
● Test results/report output
3. Framework for practical application in the project
● Project 1 – Framework Postman + Newman + Allure + Jenkins/Jiku Cloud, case number: 37
● Project 2 — Framework JMeter + Ant + Allure + Jenkins, case quantity: 2
● Project 3 — Framework Httprunner + Request + Pytest + Alluret + Jenkins, number of cases: 27
● Project 4 – Framework Pytest + Request + Allure + Jenkins, case number: 19
● Project 5 — Framework pytest + request + Allure + Jenkins, case quantity: 12
4. Comparison of automated interface testing frameworks
1. Postman + Newman framework
①.Introduction
Postman is an extension tool for Google Chrome and also has a separate client product.
postman is very convenient for daily debugging and easy to use.
The main functions include initiating interface requests, test assertions, setting preconditions, supporting post-operations, supporting the use of global variables/environment variables/collection variables/internal variables, parameterized applications, interface association, postman, and simple implementation by combining newman + jenkins. Interface automation and continuous integration and other operations.
Postman’s advanced features can be customized for a fee.
②.Advantages and disadvantages
advantage:
The free version is already very powerful, with low threshold, quick to get started, and cross-platform
The threshold for use is low and requires only simple programming skills. Some complex assertions may require understanding of js syntax;
Supports different authentication mechanisms, including Basic Auth, Digest Auth, OAuth1.0, OAuth2.0
Can send most types of HTTP/HTTPS requests, such as GET, POST, PUT, PATCH, DELETE, TRACE, etc.;
Support data drive, read data files, json, csv
Support setting environment variables: It is convenient to switch different environments for interface testing without modifying variables or code; (the same set of test cases can be run in different environments by switching environment variables, such as production environment/pre-release/test environment)
There is no duplication of workload, and the cases for interface testing can be directly exported to form test cases;
It is very convenient to convert interface tests into automated cases and relatively simple to run;
Postman is maintained by Google and is reliable;
shortcoming:
Only Node.js language is supported, and js is not universal;
File-related operations cannot be performed, databases cannot be read and written, and non-HTTP protocols cannot be used.
Configuring test case priority and test case grouping functions is not supported and can only be executed in order;
Dynamic data/test preparation functions/test assertions in test cases must be defined in advance and cannot meet the requirements of all scenarios (such as database interaction);
Test cases are in json format and are very unchanged for viewing, maintenance and changes;
The maintenance cost of test cases is high. When there are changes, they need to be imported into the newman project again;
The framework has poor scalability. Postman’s CI integration and extended packaging require separate development of new projects to be compatible with Postman’s native framework language.
Dynamic passing of environment variables at runtime is not supported
Failure retry is not supported
Only suitable for small projects with uncomplicated business logic
③.Usage requirements
Have a basic understanding of the http protocol
Understand interface testing concepts
Basic use of tools
Request header, request physical ability to distinguish
④.Applicable business
A large number of interfaces in the business
The business scenario is independent and the correlation is weak
Automation of small API projects
API regression testing for short-term projects
Testing team or beginners with weak coding skills
⑤.Environment dependence
Requires Node.js execution environment
Newman needs to be installed, npm install -g newman
Install the used plug-ins, such as htmlextra, allure, etc.
⑥.Application steps
1. Use case requirements: The interface use case has relatively complete interface management, global variable/environment variable definition, dynamic parameter application, request parameterization, interface association, and all interfaces have status/performance assertions + business assertions.
2. Generate projects: export interface test cases, environment variables, global variables, data-driven files, etc.
3. Configuration project: The specific configuration process will be introduced in a separate chapter of the framework.
4. Run the project
● Perform automation locally using the CLI
● Configure into jenkins job
5. View the report (other frameworks use Allure to output reports, so I won’t go into details)
6. Code upload to gitlab/github
7. Connect to continuous integration and configure scheduled tasks (other frameworks support task scheduling through Jikku Cloud or Jenkins, so I won’t go into details)
● Connect to Jiku Cloud Pipeline
● Connect to Jenkins
2. JMeter + Ant framework
①.Introduction
JMeter can be used for performance testing, it is very powerful in performance testing, and can also be used for automated interface testing.
②.Advantages and disadvantages
advantage:
Support parameterization
No need to write code
Supports many protocols, such as HTTP, FTP, soap, websocket, jdbc, thrift, dubbo, etc.
Support database operations
shortcoming:
Creating interface use cases is inefficient
High learning cost
Need to know the basic use of jmeter tool
Familiar with basic database operations and writing SQL statements
Understand the non-http protocols used, such as the websocket protocol used in the project [self-developed long connection]
③.Usage requirements
The learning cost of jmeter is mainly based on the use of jmeter tool
For those who have already mastered the use of tools, all you need to use jmeter for automated testing is to know how to configure Ant.
④.Applicable business
Involves database operations
Businesses involving non-https interfaces
The business needs to conduct performance testing and the main interface scripts have been written.
⑤.Environment dependence
Different versions of JMeter have different Java version requirements. for example:
JMeter3.3 only supports Java 8
JMeter4.0 requires Java 8+ (meaning greater than or equal to Java 8 version)
JMeter5.0 or above requires Java 8+
Install ant plugin
⑥.Application steps
1. Use case requirements: The interface use case has relatively complete interface management, appropriate component usage, dynamic parameter application, request parameterization, interface association, and all interfaces have status assertions + business assertions.
2. Generate projects: write test cases, export test cases, data-driven files, etc.
3. Configure Ant’s build.xml file
4. Run the project
● Use ant locally to perform automation
● Configure into jenkins job
3. HttpRunner + request
①.Introduction
HttpRunner is a general testing framework for the HTTP(S) protocol. You only need to write and maintain a YAML/JSON script to achieve various testing needs such as automated testing, performance testing, online monitoring, and continuous integration.
Supports network protocols 213 such as HTTP(S) / HTTP2 / WebSocket /thrift /dubbo, covering test types such as interface testing, performance testing, online monitoring, continuous integration, and digital experience monitoring.
It is easy to use, powerful, has rich plug-in mechanisms and a high degree of scalability.
Just write and maintain a YAML/JSON script
Formerly known as ApiTestEngine (2016), it was officially renamed HTTPRunner in 2017 and hosted on PyPI
②.Advantages and disadvantages
The design concept and main features are listed in detail on the official website and will not be repeated. Here are the main advantages and disadvantages.
advantage:
Based on YAML/JSON format, focusing on the writing of the interface itself
The interface recording function is easy to operate and can complete the test in just 3 steps, which is especially convenient for simpler scenarios.
The interface is simple to write, easy to use, and requires low coding ability.
Generate test reports, you can automatically generate test reports. The test report templates that come with the framework basically meet the needs, and support custom test report templates.
The layered mechanism is suitable for smoke process testing. There is no need to repeatedly write interfaces, just call them flexibly according to needs.
shortcoming:
HttpRunner does not have an editor plug-in, it is a configuration file itself, so as long as it is in legal YAML/JSON format, even if it is written incorrectly, it will not be displayed without verification. You will only know when it is run.
The framework has been launched for a relatively short time, the official documents do not provide particularly detailed instructions, and there are fewer online materials than other mainstream testing frameworks.
It is inconvenient to expand. The data driver needs to rely on other interfaces to return, and there is a sequence. This is more troublesome. The temporary framework does not support an elegant solution to this situation. This problem can be solved step by step.
Since the data export of the use case can only be done in one test cycle, we also need to solve the problem of test data transmission.
Solved by writing to a file. The test data returned by the interface is written to the file, and then the data is read back by reading the file where needed.
③.Usage requirements
Have a certain python foundation
Will use charles/fiddler/postman
Have a basic understanding of the request framework and at least know how to make get and post requests
④.Applicable business
Interface association is weak
There are many business logic branches and many scenarios
High interface/scenario reusability
⑤.Environment dependence
python V3.6+
pycharm
httprunner 3.1.11 or below (! Important, httprunner V4 has removed the startproject scaffolding and is not suitable for quick access by novices)
Anacanda
⑥.Framework implementation function
Ⅰ. Overview
The secondary packaging adopts a combination of V2.0 and V3.0:
1. Use the layered concept of V2.0 to reduce the duplication of scenario case writing and subsequent maintenance costs.
2. Run on V3.0. Used the pytest functions integrated in V3.0, including parameters, fixtures, hooks, allure and many other plug-ins in the pytest ecosystem.
3. Use case management still uses yaml, and the har2case method of V2.0 is used to generate yaml use cases, which is relatively friendly to students with weak coding foundation.
4. Retains the chain call and syntax check of V3.0, and provides intelligent syntax prompts for students with a coding background to provide syntax checks when writing use cases.
5. The report is generated using allure introduced in V3.0, and the report is more beautiful and detailed.
6. Portability, supports conda virtual environment + requirements.txt copy environment
Ⅱ. The main functions have been implemented, and subsequent functions will be continuously updated according to business needs.
to parameterize
business assertion
data driven
Test layering
util package
environment variables
Configuration management
Scenario test cases
setup/teardown, different scopes
Hot loading
dynamic parameters
Report
continuous integration
Ⅲ. Overall operation process
⑦.Application steps
1. Use case requirements:
All basic interfaces must be recorded in full
Design the scenarios to be covered in advance, and do not do unnecessary operations when recording use cases.
Put it into the specified use case storage directory
2. Convert har files into yaml use cases or pytest use cases
3. Configure the ini file
4. Run the project
● Local execution
● Configure and run in jenkins job
4. pytest + request
①.Introduction
Pytest is a third-party unit testing framework for Python. It is full-featured and very mature. It is similar to the built-in Unittest testing framework. Compared with the Unittest framework, it is simpler to use and more efficient. It is designed to make unit testing easier and can also be extended to support complex functional testing at the application level.
②.Advantages and disadvantages
advantage:
Able to support simple unit testing and complex functional testing
pytest has a rich plug-in ecosystem and can be customized and extended
Can be integrated well with jenkins
The report framework—-allure also supports pytest
Automatically identify test modules and test functions
Support parameterization
Modular fixtures to manage various testing resources
Fully compatible with unittest
The community is prosperous and the documentation is rich. There are many examples in the documentation for reference.
shortcoming:
High learning cost
③.Usage requirements
Have a basic understanding of the pytest framework
Have a basic understanding of request
Requires python basics
Learn about packaging
Learn about virtual environments
④.Applicable business
There are many project interfaces, but the input parameters are mostly the same.
Most interfaces are authenticated
⑤.Environment dependence
python V3.9 or above
pycharm
Anacanda
⑥.Framework implementation function
1. Test class base class encapsulation: The base class implements common methods for all test cases such as printing request/response, log management, assertions, etc. All test case classes inherit the base class
2. Unified request encapsulation:
● Globally maintain a session object and automatically implement cookie/session management functions without the need for separate authentication for each interface.
● Extract common parameters of interfaces, eliminating the need to process each interface individually. Such as base_url, referer, headers processing, ssl processing, etc.
● Provide a globally unified request method and shield the underlying implementation details of different request methods.
3. Configuration implementation: realize public parameter management through global configuration files
4. Parameterized implementation: test case scripts and test data classification, use case management through yaml
5. Public util: implements assertions, log management, database drivers, email encapsulation, file reading, etc.
for example
Taking the Rich Alliance applet as an example, when performing interface testing in postman, all interfaces need to encapsulate cookies and referers in headers. Even if parameterization is implemented in environment variables and can be managed uniformly, adding this interface is still very cumbersome. .
Convert it into an automated interface test script. Taking a certain interface as an example, the code before encapsulation is as follows:
After encapsulation, it is as follows. All interface cases are a unified set of codes.
You only need to implement the yaml use case separately for each interface
⑦.Application steps
1. Use case requirements: Use yaml to write test cases. The test cases need to contain basic fields, such as name, des, request, and validdata.
2. Prepare the running configuration ini file: mainly includes use case path, use case automatic recognition custom prefix, use case grouping, running parameter settings, etc.
3. Prepare project configuration ini file: mainly includes global entry of public parameters, log path and format, database connection information, etc.
4. Run the project
5. Summary
There is no best framework, only the most suitable framework. According to the project characteristics, technology stack, business needs, iteration frequency, team member technical level, etc., comprehensive selection of the automated interface testing framework can be carried out to reduce access costs and maintenance costs, so as to ensure the continuity of automated testing.
Finally, I would like to thank everyone who has read my article carefully. Looking at the increase in fans and attention, there is always some courtesy. Although it is not a very valuable thing, if you can use it, you can take it directly!
Software testing interview document
We must study to find a high-paying job. The following interview questions are from the latest interview materials from first-tier Internet companies such as Alibaba, Tencent, Byte, etc., and some Byte bosses have given authoritative answers. After finishing this set I believe everyone can find a satisfactory job based on the interview information.
