I often hear people say how high the amount of concurrency is. How is this value measured?

Foreword

An online exam system will be launched in the near future. Since the client of Party A pays more attention to this exam, various preparations must be done. Therefore, I did a relatively comprehensive stress test on the online system. Through this test, many previous ideas were verified, and I feel that I have gained a lot, so I leave this article.

Prepare test cases

Normally, the use case to be tested may be an interface, or a page, or a series of operations, such as logging in first, then browsing a certain page, and then submitting a certain form, etc. The specific situation is different, and the complexity of preparing use cases is also different. I just prepared a few interfaces here, and then implemented some operations that simulate real situations in the interfaces, so that the prepared use cases will be very simple.

In a nutshell, there are 4 operations: database read, cache read, database write, and database update

I won’t post the specific code, it’s all business operations, just paste the interface name and some interface configuration

#region Performance stress test related interface, simulate various actions of the test

        /// <summary>
        /// Simulate getting the test list and cache it
        /// </summary>
        /// <param name="associationId"></param>
        /// <returns></returns>
        [Allow Anonymous]
        [ResponseCache(Duration = 100)]
        public async Task<IActionResult> TestGetExaminations(string token)
        {
            //business action
}

        /// <summary>
        /// Simulate authentication interface
        /// Only verify the interface access status here, so set the return value of all cases as code=1
        /// </summary>
        /// <returns></returns>
        [Allow Anonymous]
        public async Task<IActionResult> TestInfoVerification(string token,string testExamId="",string testIdNumber="")
        {
            //business action
        }

        /// <summary>
        /// Simulate drawing,
        /// There is no actual question extraction here, a relatively time-consuming insertion operation is performed, and it is marked as deleted
        /// </summary>
        /// <returns></returns>
        [AllowAnonymous,HttpPost]
        public async Task<IActionResult> TestConfirmMyPaper(string token)
        {
            //business action
        }

        /// <summary>
        /// Simulate submitting answers
        /// </summary>
        /// <returns></returns>
        [AllowAnonymous, HttpPost]
        public async Task<IActionResult> TestSubmitPaper(string token)
        {
            //business action
        }
        #endregion

Note that I have set the access level of these interfaces to be accessible without authentication. One is for convenience, and the other is because our user center is isolated, and it is also to avoid the impact of multiple systems. Direct test performance of this system.

Build JMeter distributed pressure testing environment

Why?

JMeter needs no introduction, the official document is here: https://jmeter.apache.org/, the latest version is 5.5. The reason why we want to build a distributed pressure testing environment is to simulate the real situation to the maximum extent, because JMeter is written in the Java language, and the consumption of resources is not low. The maximum UVs that can be simulated by a single machine has an upper limit. If the value is set too high, the software itself will crash. In order to solve this problem, JMeter officially provides a solution for distributed deployment.

Prepare the dependent environment

JMeter’s necessary dependent environment is JDK, but since the commercial use of JDK1.8 and later versions will have certain risks, the version I use here is OpenJDK 17 built by Microsoft. Here, Microsoft also provides users with detailed installation manuals. Including almost all mainstream systems on the market, the address is here: https://learn.microsoft.com/zh-cn/java/openjdk/install My here is the CentOS environment and Windows environment, the screenshot after installation is as follows

Attention, the jdk version of all terminal machines should be consistent!

Download JMeter

Go to the official website to download the latest version of JMeter, which is currently 5.5, address: https://jmeter.apache.org/download_jmeter.cgi The specific download process will not be detailed, centos can use wget, or it can be downloaded through windows GUI tool upload to Linux environment

Configure the press

Here I configured two CentOS machines as slaves. The configuration process is as follows: enter the bin directory, open the jmeter.properties file to find and modify the following parameters

  • language: Release the comment and set it to zh_CN. This is not necessary. After this operation, Chinese will be displayed when using the GUI interface.

  • remote_hosts: modify to the current press ip address + port, the default port is 1099, you can change it to any unoccupied port

  • server_port: follow the above port setting, comment or not set, then it is not 1099

  • server.rmi.localport: Follow the port settings above, this needs to be commented

  • server.rmi.ssl.disable: set to true for convenience,

If you consider the safety factor, you can find the ‘create-rmi-keystore.bat’ or ‘create-rmi-keystore.sh’ file in the bin directory, take the bat file as an example, copy the file to the bin directory of jdk, and execute this file, according to the console prompt, after the input is completed, you will get a rmi_keystore.jks file, copy this file to all presses and control machines.

For convenience, I directly set server.rmi.ssl.disable to true. Finally, the main configuration of my press here is as follows, the two configurations are the same except for ip

#...other configuration

remote_hosts=10.185.1.178:1029
server_port=1029
server.rmi.localport=1029
server.rmi.ssl.disable=true

#...other configuration

After the configuration is complete, the press here also needs to modify the execution permission of the jmeter-server file

chmod 777 jmeter-server

Then modify the jmeter-server file to release the comment of the RMI_HOST_DEF parameter, and modify the hostname to the local ip (all slave presses must be modified)

DIRNAME=`dirname $0`

# If the client fails with:
# ERROR - jmeter.engine.ClientJMeterEngine: java.rmi.ConnectException: Connection refused to host: 127.0.0.1
# then it may be due to the server host returning 127.0.0.1 as its address

# One way to fix this is to define RMI_HOST_DEF below
RMI_HOST_DEF=-Djava.rmi.server.hostname=10.185.1.176

${DIRNAME}/jmeter ${RMI_HOST_DEF} -Dserver_port=${SERVER_PORT:-1099} -s -j jmeter-server.log "$@"

At this point, you can execute the service node on the press

Note: Due to data transmission problems, the firewall needs to be turned off for the transmission between the slave machine and the master machine. Just turn off the entire firewall directly, and then the problem is solved.

I don’t know if this is a problem with JMeter, or there are other solutions. But in any case, closing the firewall is a very dangerous operation. In a non-intranet security environment, operate with caution!

The shutdown command is as follows

systemctl stop firewalld.service

After solving the problem of data transmission, enter the bin directory to start the service

./jmeter-server

If the following interface appears, it means success.

Here, it should also be noted that in order to increase the upper limit of the pressure measurement performance of a single press as much as possible, the default jvm memory limit can be increased. Here, according to the suggestion of the official document, add the setenv.sh file (Windows is sentenv.bat) to write Just enter the following code

export JVM_ARGS="-Xms1G -Xmx1G -XX:MaxMetaspaceSize=192m"
# Default is -Xms256m -Xmx256m -XX:MaxMetaspaceSize=256m

The specific explanation here can refer to the official website introduction, enter this address: https://jmeter.apache.org/usermanual/get-started.html, search for jvm keywords

Configure the controller

It is recommended to configure the control machine on a familiar and convenient system environment. The Windows I use here is slightly different from the configuration content of the press. Enter the bin directory and open the jmeter.properties file

  • remote_hosts: configured as ip+port for all presses

  • server.rmi.ssl.disable: set to true, the precautions are the same as those described in the press configuration section

Because the controller does not undertake pressure testing tasks, only configure these parameters, and keep the others as default

The control machine can also work together as a press machine and slave node, just release the port in the configuration, and configure the current ip to remote_hosts

Prepare the pressure test configuration file

There are many ways to configure here, you can start the GUI on the control machine, set up the interface, monitor, etc., and then save it as a jmx file; you can also rely on some third-party tools, such as ApiFox, after configuring the test scene, you can use it in the performance Test window, export configuration file directly.

The JMeter GUI configuration interface is as follows

The configuration interface of ApiFox is as follows

This part is not introduced much, and the final generated file is an xml-style jmx file~Long like this

Start stress test

After the above process configuration is completed, it is very simple to execute the stress test. I wrote a bat file here for reference. I created a new result folder under the jmeter root directory, and then put the jmx test file and the following executable file put together

@echo off
cd /d %~dp0
call ..\bin\jmeter.bat -n -t exam.jmx -r -l testResult?te:~0,4%?te:~5,2%?te:~8,2%%time:~0 ,2%%time:~3,2%.jtl -e -o ResultReport_?te:~0,4%?te:~5,2%?te:~8,2%%time:~0,2% %time:~3,2%
echo ?te:~0,4%?te:~5,2%?te:~8,2%%time:~0,2%%time:~3,2%%time:~6,2% jmeter ok! >> log.txt

After executing the bat file, the following information can be seen on the control machine under normal circumstances

on the corresponding press

According to the pressure test parameters we set, including duration, number of threads (UVs), ramp-up, etc., after the program is executed, a report will be generated at the address we set (configured in the above bat file) directory, and then click index.html to view the data report of the pressure test

At this point, the environment construction and execution of distributed pressure testing is over.

In the future, we can dynamically adjust the parameters and conduct stress tests under different parameter conditions.

For example, under the load of a single machine, what is the maximum concurrency, dual machines, and multiple machines?

For another example, compare different deployment environments, such as apache, iis, kestrel, nginx

Let me say one more thing here, I actually measured the two situations of iis and kestrel here, and iis is really scum!

For another example, compare the situation of increasing the cache and not increasing the cache

For another example, the database stand-alone, the situation of the database cluster, etc.

After getting the chart data of each test, analyze various items, such as response time (RT), throughput (QPS), received traffic per second, sent traffic, sample size and other data indicators, just like watching a physical examination Report card.

Through rounds of tests, we can slowly find the performance bottleneck of our system. In addition to helping us optimize the system and hardware configuration, it can also give users a reliable and objective concurrent load scale to avoid wrangling.

All in all, stress testing is a very necessary link in some direct ToC, or scenarios where the user scale is relatively high but difficult to estimate!

Additional recommendations

Finally, I would like to introduce several SaaS stress testing tools.

When we do not have press resources on hand, or it is not very convenient to prepare resources, we can consider some third-party SaaS tools. For example, big companies such as Alibaba Cloud and Tencent Cloud all provide performance testing tools for stress testing and have specific operating documents (I often read their documents when I do stress testing myself. First, look at their How to use the tools, the more important thing is to learn the knowledge of stress testing, after all, we are not professional testing ~ haha).

I have personally used Tencent Cloud’s cloud stress testing product, and overall it is very easy to use. The key point is that it supports direct import of jmx configuration files, which is more convenient. The charts are also richer than those generated by jmeter itself, and new users are given a 20,000 UVs quota. I recommend everyone to try it. Portal?: Tencent Cloud Operations – Tencent Cloud

In addition to Tencent Cloud’s cloud stress testing, APIFox mentioned above also provides stress testing capabilities with a very cool interface.

There is also a very good pressure testing tool made in China, RunnerGo, whose bottom layer is implemented in go language. I tried it and it is very good. You can customize various test scenarios and add some control mechanisms. It is closer to reality, and I recommend everyone to try it.

Synchronous forwarding in infoq: high concurrency, high concurrency, how high is high? Where did this value come from? _Jmeter_Take yourself with salt_InfoQ Writing Community