Why did you choose this topic?
First, we find that many “part-time workers” will ask questions related to automated testing during the confused period of transformation. It can be said that automated testing is the only way for “part-time workers” to upgrade; second, if you Google interface automation testing, you will find a lot of automation There are articles related to testing frameworks, but most articles have a common problem, that is, they only tell readers how (how to do it), and rarely introduce why, as well as what happens after the framework is developed (such as how to promote, maintain, etc.). Then let’s talk about some of my experiences and lessons learned in the construction of the interface automation testing framework. I hope it can give you some reference.
I have been working for many years, and the first time I heard about interface testing was in the book “The Art of Software Testing”. At that time, the school recruited students to review software testing theory. At that time, I had no sense of it, and I didn’t even know what an interface was. The first contact interface test is not derived from project needs, but from team KPI needs. At that time, I had just joined the company not long after the school recruitment, and the team had internal testing-related knowledge and skills training (although it was only the use of testing tools). The first interface testing tool I came into contact with at that time was JMeter. I still remember that my colleague introduced me to some of its common components through a word document, and then assigned me a task – to increase the code coverage of the team xx platform interface code to more than 40%. At that time, I didn’t know what code coverage was, but I just wanted to use JMeter to develop more use cases of single interface/business scenario combinations. So I started to do various things through the input parameter type, boundary, and whether it must be passed for a single interface. Combination, and scenario combination use cases do Cartesian products between interfaces to generate different business flows. At my second job, my boss wanted to build an interface testing platform for internal use within the team. At that time, he gave me the opportunity to be responsible for the construction of the interface testing framework. This is also the first time topractice the interface automation testing framework. The story we’re talking about today begins here. . .
1. Why build your own testing framework
1.1 Let’s talk about interface testing first
What is an interface? In layman’s terms, the common light switch in life is an interface, which receives on and off (default) commands (values) and outputs the control results of the light bulb (on, off). Interface testing is a type of testing that tests the interfaces between system components. The focus of testing is to check the exchange and transmission of data and the mutual logical dependencies between systems.
1. Interface testing “routine”
According to the example I gave, you can know that for interface testing, you can functionally treat the interface as a black box for input to observe its output, and test its internal logic based on different inputs. We can use boundary value analysis, equivalence classes, etc. Method design use cases. Non-functionally, interface performance, security, and idempotency should be tested. In addition, if the interface under test has dependencies on upstream and downstream interface calls, full-link joint debugging testing is also required (some interfaces do not exist independently and call each other with other interfaces). Joint debugging testing is to ensure that the upstream and downstream interfaces are connected. The accuracy of the contract between road interfaces.
2. Why is it suitable for automation
According to the idea of layered testing, system testing can be divided into unit testing, interface testing and interface testing. Generally speaking, unit testing is like the foundation of a house. If the foundation is well laid, the house will be more stable; interface testing is like the frame of the house, determining the shape of the house; and interface testing is like the decoration of the house, reflecting the appearance of the house. . It is true that the foundation and the frame of the house are relatively stable and do not change much, while the decoration of the house can be constantly changed according to mood. Everyone knows that change is directly proportional to cost. The greater the change, the higher the cost. According to ROI = output/input, the numerator remains unchanged. The larger the denominator, the smaller the ROI. This is also the main reason why interface testing is not suitable for automation. Reasons. Of course, it does not mean that all projects are not suitable for UI automation. If the UI is relatively fixed, it can also be automated.
Of course, large companies are already at the forefront in this regard. Take Google, the industry benchmark, as mentioned in the book “How Google Tests Software”. 70% of automation is in single testing, 20% in interface testing, and 10% in UI testing.
3. Commonly used interface testing tools
Let me introduce the two tools JMeter and Postman that I have used.
Generally speaking, JMeter is more suitable as an interface automation tool, although the official website defines it as a stress testing tool (The Apache JMeter? application is open source software, a 100% pure Java application designed to load test functional behavior and measure performance .) Because JMeter can meet almost all the requirements of interface automation testing, it has rich functions (similar to utils) (such as generating random numbers, uuid, etc.). These are capabilities often used in the testing process, and it also supports its own Extensions), rich assertions, continuous integration with Jenkins, test reports, a rich plug-in library for download, etc. It is very suitable for testing teams with weak technical skills. Postman can also be used for automation, but I haven’t practiced it yet. I usually use it more as a tool for interface debugging, so I won’t go into details.
Now I have also found a lot of test friends and created a communication group to share technology, sharing a lot of technical documents and video tutorials we collected. If you don’t want to experience the feeling of not being able to find resources when studying on your own, having no one to answer your questions, and persisting for a few days before giving up. You can join us to communicate. And there are many technical experts who have made certain achievements in automation, performance, security, test development, etc. Share their experience, and also share many live lectures and technical salons You can learn for free! Focus on it! Open source! ! ! QQ group number: 110685036
1.2 Purpose of automated testing
Originally, this subtitle was “Purpose of Interface Automation Testing”, but when I started writing it, I found that all automation purposes seemed to be public. Let me first throw out two arguments: Found flaws? Improve testing efficiency?
You may think that if automation is implemented, these two purposes are natural. Of course I don’t deny it but I don’t entirely agree either. So why should I discuss these two arguments? Because automated testing is a planned process, different purposes may have different implementation processes and different implementation focuses.
If the purpose is to find defects, then the automated test cases must be comprehensive and detailed (business level, assertion level). This will undoubtedly increase the development cost of automated test cases, so the investment process needs to focus on business analysis and more scheduling. Development time.
And what about improving testing efficiency? Of course, the efficiency improvement here is not only reflected in the testing process, but also in the daily work of testers, such as data creation and other tasks. Therefore, if the goal is to improve efficiency, then automated test case development should focus on analyzing which business flows are called more frequently, and there is no need to be too detailed at the assertion level. Take making numbers as an example. After all, the purpose of improving efficiency is to obtain results. , as long as the result is as expected, it is OK. Of course, these are the author’s personal opinions, and everyone can express their own opinions.
1.3 Why is a self-developed framework needed?
As mentioned above, for automated interface testing, there are readily available open source tools such as JMeter and Postman to choose from, as well as some open source automated testing platforms. However, these out-of-the-box tools also have drawbacks:
- Test cases developed by testing tools are often not easy to maintain. Just imagine the scenario when you maintain various JMX files generated by JMeter.
- A large number of test cases have poor reusability, such as the login module. Cookies must be obtained before each interface call, and login is a pre-module. Using JMeter to develop test cases requires that each JMX file must contain login, which is too repetitive.
- Unable to meet the demands of the automation platform, automation can indeed be quickly achieved in the short term. However, these tools have a higher cost for expanding the non-automation capabilities of the platform. After all, the cost of modifying open source tools is much higher than self-research.
- Using open source tools is not conducive to improving the team’s growth in automation technology. The improvement of automation capabilities is inseparable from the improvement of programming capabilities. Using open source tools can improve the ability to learn and use tools. In the end, your growth is nothing more than mastering the use of a testing tool.
- Unable to maximize one’s problem-solving abilities.
1.4 Interface Test Framework Selection
- About language
The language suggestions are consistent with the system language responsible for the project. If the language of the project under test is Java, the test framework will be developed in Java language and built in conjunction with the Spring boot framework.
- About scalability
Scalability is very important, because the framework is not built overnight. It needs to be continuously “updated” based on business iterations. Good scalability allows you to develop the required features more efficiently. You can refer to JMeter’s architectural design for this point.
- About the implementation plan
After determining the technology to be used, be sure to sort out the implementation ideas and write an implementation plan, just like the project documentation, which will be of guiding significance to you. If you think about it carefully, the development of the test framework requires you to take into account the role of production, research and testing (if possible, you can let the product/UI help you design a platform sketch).
2. Experience accumulation
1. The importance of planning
Everything will be done if it is forewarned, otherwise it will be ruined. Planning is especially important for testing. On the one hand, the task of building a test framework does not allow you to do it 100% of your working time, because you still have business testing work to be responsible for and follow up on. On the other hand, the boss wants to see the results, and it is very important to set milestones for each stage. Of course, these plans will be reflected in OKR.
In my experience, you can schedule yourself according to the number of features, for example, implement x features every week. This way you can roughly estimate the number of weeks required to complete the development of the test framework. Try to reflect the progress of the implementation in the weekly report, as well as next week. todo.
2. Documentation is essential
Why is documentation so important? Documentation is actually another implementation method of “automated testing framework”. Code implements the framework itself, text “dresses” the framework. Furthermore, documentation can save you a lot of “customer service” time in the future. There is no limit to the scope of document content. Generally speaking, there are three categories: technical implementation documents, usage instructions documents, and problem record documents.
- Technical implementation documents are used to describe the technical means and coding specifications for framework implementation, and can be supplemented by system documents.
- Instructions for use are used to introduce how to use it. It is best to take the quick start as an example to teach you how to use the testing framework to develop a test case. Let’s introduce in detail how to use (call) some public components (tools). Of course it is recommended to record the operation video.
- The problem record document is used for reference by other partners when it comes to collaborative development. After all, the pitfalls you have stepped on may have been stepped on by others. This can reduce the time for troubleshooting and locating similar problems. (The author once spent a lot of time helping a group of students troubleshoot problems.)
3. Development model under multi-person collaboration
The automation framework is ultimately promoted to everyone to use it to develop interface test cases to replace interface testing tools such as JMeter. Before the platform is implemented, it is inevitable for multiple people to submit the developed use case code at the same time. For this, I recommend using the master\dev branch. Daily automated use cases run using the master branch code, and the dev branch is used for everyone to submit the code of their respective branches. This requires frequently merging the code from the dev branch to the master branch.
4. Do a good job of code review
Just like Experience 3, after determining the development model, it is also inevitable that some students will “misbehave” when submitting code (not standardizing development and submission, such as modifying other people’s code at will, etc.), resulting in code conflicts. As the person in charge of the project, like a door god, you must do a good job in code review. This can reduce the probability of code conflicts to a certain extent. In short, try not to waste precious time on these code management issues.
5. Promotion group trial
After the framework is developed, it is not recommended to rush to promote it in large quantities. It is recommended to trial it in grayscale with “small traffic” first. Our large team is divided into many small teams. Basically, the business owner brings some outsourcing to ensure the quality of the project. Managers, on the other hand, focus on the results of the framework. Framework developers are full-time employees, and users of the framework are mainly outsourced and full-time students who focus on business testing. After completing the framework development at that time (it was considered version 1.0 and already available for use), I started using it in my team. In this way, we can first collect the usage effects and feedback issues, make a small iteration first, and solve the problems that they think affect the use first.
6. Sharing and training
First, I will share with you, demonstrating the operating principles of the framework and how to develop interface test cases. In order to make it easier for everyone to understand, I will make an analogy between the steps of using postman and testing an interface in the framework. Even so, everyone still encountered a lot of problems when they first started using the testing framework (which is also expected, most outsourcers have almost no programming experience), including the high cost of developing interface test cases, the poor coding skills of outsourced students, and the inability to solve problems when they encounter them. I know how to debug code, and when uploading developed use cases to GitLab, conflicts often arise that need to be resolved. Each student’s use cases are written in various ways, and so on. I still remember the content of that period. In addition to doing the business at hand, I also helped everyone solve various problems encountered when writing use cases. In order to solve these problems, I also squeezed in time to write the test framework instruction manual, provide everyone with git usage training, debug training, etc., and encourage everyone to learn more Java and read more “Java Programming Thoughts”.
7. Pay attention to everyone’s feedback
Using feedback is very important. Because it is the driving force behind your version iterations. In fact, just like product projects, each version must analyze user feedback and carry out version optimization and iteration based on the feedback. You can communicate with users and discover some pain points during their use. Of course this is just the first step, more importantly there is a solution.
8. Consider code generation
In response to the problem of high use case development costs, I concluded during the test case development process that the interface test writing methods corresponding to different HTTP request methods are different, and the test interface writing methods for the same methods are almost the same. It suddenly occurred to me that templates (freemarker) can be used to automatically generate interface tests. test cases, and then I will organize several interface test case generation templates (post, get, etc.), which greatly reduces the cost of test case development. We focus more on the preparation and assertion of test data.
Personally, I feel that code generation is a must-consider point in the design of every test framework, because tools like postman, jmeter, etc. are essentially the same as when you develop a test framework. If use cases can be converted into each other, the reliability of the use cases can be improved. Portability and reusability.
3. Lesson Summary
1. Lack of statistics & measurement of automation effects
As mentioned above, the boss values the results more than the process. How to reflect the results requires data statistics and measurement. I remember being challenged by developers on how many defects you found through automation and how much time you saved. This was not done well at the time.
1. The framework itself has not yet become a platform. The function implemented at that time was the implementation of the test case development module, and the statistical data was not recorded in the DB. This required manual statistics, which was quite time-consuming (at that time, in order to collect statistics on the results of automated use cases, the data was collected on the Jenkins server) Fishing for logs is simply a pain).
2. There was no clear measurement goal at the time. It was really difficult to measure things like how much time was saved.
Now I feel that this issue was missed at the beginning of the framework design at that time, and I suggest that latecomers can store all test data in DB.
2. No testing
This is mainly due to the fact that the testing framework itself is not tested, especially in the multi-person collaborative development model. Single testing is particularly important. Single testing is a means of ensuring code quality to a certain extent, and can effectively prevent some students from “chaotically changing” other people’s code. Of course, if you have enough time, it is recommended to write a single test for the test framework.
3. The automation use case development plan does not highlight the focus
This is also one of the reasons for Lesson 1. After the framework was completed, we planned xx in OKR to complete 100% automatic coverage of the interface. At the beginning, we did not analyze the importance of the interface under test, which resulted in no time being spent on the development of core interface use cases, which also indirectly led to poor output. high.
Thinking about it now, it turns out that the plan is not clear and focused, and the granularity is not enough. The current suggestion is that everyone should prioritize the implementation of core interfaces. These interfaces have large traffic and are easier to produce output. Don’t put the cart before the horse in pursuit of coverage.
4. Imperfections of assertions
The framework at that time did not develop DB checker, so the assertion relied entirely on http response. In fact, such use cases were of low quality, that is, the test was not expected to be comprehensive, and there might be fish that slipped through the net (missed defects).
It is recommended that everyone must take the assertion module into consideration. Including http level (response time, response code, key field assertion, status code, etc.), DB level (key field value comparison, etc.).
Finally, I would like to thank everyone who has read my article carefully. Looking at the increase in fans and attention, there is always some courtesy. Although it is not a very valuable thing, if you can use it, you can take it directly!
Software testing interview document
We must study to find a high-paying job. The following interview questions are the latest interview materials from first-tier Internet companies such as Alibaba, Tencent, Byte, etc., and some Byte bosses have given authoritative answers. After finishing this set I believe everyone can find a satisfactory job based on the interview information.
