HttpClient optimization solution in high concurrency scenarios, QPS is greatly improved!

Taro Source Code 2023-10-14 10:14 Published in Shanghai

This is a community that may be useful to you

One-on-one communication/interview brochure/resume optimization/job search questions, welcome to join the “Yudao Rapid Development Platform” Knowledge Planet. The following is some information provided by Planet:

  • “Project Practice (Video)”: Learn from books, “practice” from past events

  • “Internet High Frequency Interview Questions”: Studying with your resume, spring blossoms

  • “Architecture x System Design”: Overcoming difficulties and mastering high-frequency interview scenario questions

  • “Advancing Java Learning Guide”: systematic learning, the mainstream technology stack of the Internet

  • “Must-read Java Source Code Column”: Know what it is and why it is so

Picture

This is an open source project that may be useful to you

Domestic Star is a 100,000+ open source project. The front-end includes management backend + WeChat applet, and the back-end supports monomer and microservice architecture.

Functions cover RBAC permissions, SaaS multi-tenancy, data permissions, mall, payment, workflow, large-screen reports, WeChat public account, etc.:

  • Boot address: https://gitee.com/zhijiantianya/ruoyi-vue-pro

  • Cloud address: https://gitee.com/zhijiantianya/yudao-cloud

  • Video tutorial: https://doc.iocoder.cn

Source: blog.csdn.net/u010285974/article/details/85696239

  • 1.Background

  • 2.Analysis

    • 2.1 Overhead of repeated creation of httpclient

    • 2.2 The cost of repeatedly creating tcp connections

    • 2.3 The cost of repeatedly caching entities

  • 3. Realize

    • 3.1 Define a keep alive strategy

    • 3.2 Configure a PoolingHttpClientConnectionManager

    • 3.3 Generate httpclient

    • 3.4 Reduce overhead when using httpclient to execute methods

  • 4.Others

    • 4.1 Some timeout configurations of httpclient

    • 4.2 If nginx is configured, nginx must also set up keep-alive for both ends.

Picture

HttpClient optimization ideas:

  1. Pooling

  2. Long connection

  3. Reuse of httpclient and httpget

  4. Reasonable configuration parameters (maximum number of concurrent requests, various timeouts, number of retries)

  5. asynchronous

  6. Read more source code

1.Background

We have a business that calls an http-based service provided by other departments, with daily calls in the tens of millions. Use httpclient to complete the business. Because qps could not be uploaded before, I took a look at the business code and made some optimizations, which are recorded here.

Compare before and after: before optimization, the average execution time was 250ms;

After optimization, the average execution time is 80ms, which reduces the consumption by two-thirds. The container no longer constantly alerts that the thread is exhausted, which is refreshing~

Backend management system + user applet implemented based on Spring Boot + MyBatis Plus + Vue & Element, supporting RBAC dynamic permissions, multi-tenancy, data permissions, workflow, three-party login, payment, SMS, mall and other functions

  • Project address: https://github.com/YunaiV/ruoyi-vue-pro

  • Video tutorial: https://doc.iocoder.cn/video/

2.Analysis

The original implementation of the project is relatively rough, which is to initialize an httpclient every time a request is made, generate an httpPost object, execute it, then take out the entity from the return result, save it as a string, and finally close the response and client explicitly.

Let’s analyze and optimize bit by bit:

2.1 Overhead of repeated creation of httpclient

httpclient is a thread-safe class. It does not need to be created by each thread every time it is used. Just keep one globally.

2.2 The cost of repeatedly creating tcp connections

TCP’s three-way handshake and four-way waving process are too expensive for high-frequency requests. Just imagine if we need to spend 5ms for the negotiation process for each request, then for a single system with qps of 100, we will spend 500ms for handshakes and waving in 1 second. We are not senior leaders, so we programmers should stop being so aggressive and change to keep alive mode to achieve connection reuse!

2.3 The cost of repeatedly caching entities

In the original logic, the following code was used:

HttpEntity entity = httpResponse.getEntity();

String response = EntityUtils.toString(entity);

Here we are equivalent to copying an additional copy of content into a string, while the original httpResponse still retains a copy of content and needs to be consumed. In the case of high concurrency and very large content, a large amount of memory will be consumed. And, we need to explicitly close the connection, ugly.

Backend management system + user applet implemented based on Spring Cloud Alibaba + Gateway + Nacos + RocketMQ + Vue & Element, supporting RBAC dynamic permissions, multi-tenancy, data permissions, workflow, three-party login, payment, SMS, mall and other functions

  • Project address: https://github.com/YunaiV/yudao-cloud

  • Video tutorial: https://doc.iocoder.cn/video/

3. Implementation

According to the above analysis, we mainly need to do three things: first, the singleton client, second, the cached keep-alive connection, and third, better processing of the returned results. I won’t talk about the first one, but let’s talk about the second one.

When it comes to connection caching, it is easy to think of database connection pools. httpclient4 provides a PoolingHttpClientConnectionManager as a connection pool. Next we optimize through the following steps:

3.1 Define a keep alive strategy

Regarding keep-alive, this article will not elaborate on it. I will only mention one point. Whether to use keep-alive depends on the business situation. It is not a panacea. Another point, there are many stories between keep-alive and time_wait/close_wait.

In this business scenario, we have a small number of fixed clients that access the server very frequently for a long time. It is very suitable to enable keep-alive.

One more thing, http’s keep-alive and tcp’s KEEPALIVE are not the same thing. Back to the text, define a strategy as follows:

ConnectionKeepAliveStrategy myStrategy = new ConnectionKeepAliveStrategy() {
    @Override
    public long getKeepAliveDuration(HttpResponse response, HttpContext context) {
        HeaderElementIterator it = new BasicHeaderElementIterator
            (response.headerIterator(HTTP.CONN_KEEP_ALIVE));
        while (it.hasNext()) {
            HeaderElement he = it.nextElement();
            String param = he.getName();
            String value = he.getValue();
            if (value != null & amp; & amp; param.equalsIgnoreCase
               ("timeout")) {
                return Long.parseLong(value) * 1000;
            }
        }
        return 60 * 1000; //If there is no agreement, the default definition duration is 60s
    }
};

3.2 Configure a PoolingHttpClientConnectionManager

PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
connectionManager.setMaxTotal(500);
connectionManager.setDefaultMaxPerRoute(50);//For example, the default maximum concurrency per route is 50, which depends on the business.

You can also set the concurrency number for each route.

3.3 Generate httpclient

httpClient = HttpClients.custom()
     .setConnectionManager(connectionManager)
     .setKeepAliveStrategy(kaStrategy)
     .setDefaultRequestConfig(RequestConfig.custom().setStaleConnectionCheckEnabled(true).build())
     .build();

?

Note: Using the setStaleConnectionCheckEnabled method to evict a closed link is not recommended. A better way is to manually enable a thread and run the closeExpiredConnections and closeIdleConnections methods regularly, as shown below.

?

public static class IdleConnectionMonitorThread extends Thread {
    
    private final HttpClientConnectionManager connMgr;
    private volatile boolean shutdown;
    
    public IdleConnectionMonitorThread(HttpClientConnectionManager connMgr) {
        super();
        this.connMgr = connMgr;
    }
 
    @Override
    public void run() {
        try {
            while (!shutdown) {
                synchronized (this) {
                    wait(5000);
                    // Close expired connections
                    connMgr.closeExpiredConnections();
                    // Optionally, close connections
                    // that have been idle longer than 30 sec
                    connMgr.closeIdleConnections(30, TimeUnit.SECONDS);
                }
            }
        } catch (InterruptedException ex) {
            // terminate
        }
    }
    
    public void shutdown() {
        shutdown = true;
        synchronized (this) {
            notifyAll();
        }
    }
    
}

3.4 Reduce overhead when using httpclient to execute methods

?

The thing to note here is not to close the connection.

?

A feasible way to obtain content is similar to copying the content in the entity:

res = EntityUtils.toString(response.getEntity(),"UTF-8");
EntityUtils.consume(response1.getEntity());

However, the more recommended way is to define a ResponseHandler, which is convenient for you and me, without having to catch exceptions and close the stream yourself. Here we can take a look at the relevant source code:

public <T> T execute(final HttpHost target, final HttpRequest request,
        final ResponseHandler<? extends T> responseHandler, final HttpContext context)
        throws IOException, ClientProtocolException {
    Args.notNull(responseHandler, "Response handler");

    final HttpResponse response = execute(target, request, context);

    final T result;
    try {
        result = responseHandler.handleResponse(response);
    } catch (final Exception t) {
        final HttpEntity entity = response.getEntity();
        try {
            EntityUtils.consume(entity);
        } catch (final Exception t2) {
            // Log this exception. The original exception is more
            // important and will be thrown to the caller.
            this.log.warn("Error consuming content after an exception.", t2);
        }
        if (t instanceof RuntimeException) {
            throw (RuntimeException) t;
        }
        if (t instanceof IOException) {
            throw (IOException) t;
        }
        throw new UndeclaredThrowableException(t);
    }

    // Handling the response was successful. Ensure that the content has
    // been fully consumed.
    final HttpEntity entity = response.getEntity();
    EntityUtils.consume(entity);//Look hereLook here
    return result;
}

As you can see, if we use resultHandler to execute the execute method, the consume method will eventually be automatically called, and this consume method is as follows:

public static void consume(final HttpEntity entity) throws IOException {
    if (entity == null) {
        return;
    }
    if (entity.isStreaming()) {
        final InputStream instream = entity.getContent();
        if (instream != null) {
            instream.close();
        }
    }
}

You can see that eventually it closes the input stream.

4. Others

Through the above steps, we have basically completed the writing of an httpclient that supports high concurrency. Here are some additional configurations and reminders:

4.1 Some timeout configurations of httpclient

CONNECTION_TIMEOUT is the connection timeout time, SO_TIMEOUT is the socket timeout time, the two are different. The connection timeout is the waiting time before initiating a request; the socket timeout is the timeout for waiting for data.

HttpParams params = new BasicHttpParams();
//Set the connection timeout
Integer CONNECTION_TIMEOUT = 2 * 1000; //Set the request timeout to 2 seconds and adjust it according to the business
Integer SO_TIMEOUT = 2 * 1000; //Set the timeout for waiting for data to 2 seconds and adjust it according to the business
 
//Define the millisecond timeout used when retrieving ManagedClientConnection instances from ClientConnectionManager
//This parameter expects a value of type java.lang.Long. If this parameter is not set, it defaults to CONNECTION_TIMEOUT, so it must be set.
Long CONN_MANAGER_TIMEOUT = 500L; //In httpclient4.2.3, I remember that it was changed into an object, causing an error to be reported when using long directly, but it was changed back later.
 
params.setIntParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, CONNECTION_TIMEOUT);
params.setIntParameter(CoreConnectionPNames.SO_TIMEOUT, SO_TIMEOUT);
params.setLongParameter(ClientPNames.CONN_MANAGER_TIMEOUT, CONN_MANAGER_TIMEOUT);
//Test whether the connection is available before submitting the request
params.setBooleanParameter(CoreConnectionPNames.STALE_CONNECTION_CHECK, true);
 
//In addition, set the number of retries of the http client. The default is 3 times; currently it is disabled (if the number of projects is not enough, this default is enough)
httpClient.setHttpRequestRetryHandler(new DefaultHttpRequestRetryHandler(0, false));

4.2 If nginx is configured, nginx must also set up keep-alive for both ends

In today’s business, situations without nginx are relatively rare. By default, nginx opens long connections with the client and uses short links with the server.

?

Pay attention to the keepalive_timeout and keepalive_requests parameters on the client side, as well as the keepalive parameter settings on the upstream side. The meaning of these three parameters will not be repeated here.

?

That’s all my settings. Through these settings, the original time consumption of each request was reduced from 250ms to about 80ms, and the effect was remarkable.

The JAR package is as follows:

<!-- httpclient -->
<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.5.6</version>
</dependency>

code show as below:

//Basic certification
private static final CredentialsProvider credsProvider = new BasicCredentialsProvider();
//httpClient
private static final CloseableHttpClient httpclient;
//httpGet method
private static final HttpGet httpget;
//
private static final RequestConfig reqestConfig;
//response handler
private static final ResponseHandler<String> responseHandler;
//jackson parsing tool
private static final ObjectMapper mapper = new ObjectMapper();
static {
    System.setProperty("http.maxConnections","50");
    System.setProperty("http.keepAlive", "true");
    //Set basic verification
    credsProvider.setCredentials(
            new AuthScope(AuthScope.ANY_HOST, AuthScope.ANY_PORT, AuthScope.ANY_REALM),
            new UsernamePasswordCredentials("", ""));
    //Create http client
    httpclient = HttpClients.custom()
            .useSystemProperties()
            .setRetryHandler(new DefaultHttpRequestRetryHandler(3,true))
            .setDefaultCredentialsProvider(credsProvider)
            .build();
    //Initialize httpGet
    httpget = new HttpGet();
    //Initialize HTTP request configuration
    reqestConfig = RequestConfig.custom()
            .setContentCompressionEnabled(true)
            .setSocketTimeout(100)
            .setAuthenticationEnabled(true)
            .setConnectionRequestTimeout(100)
            .setConnectTimeout(100).build();
    httpget.setConfig(reqestConfig);
    //Initialize the response parser
    responseHandler = new BasicResponseHandler();
}
/*
 * Function: Return response
 * @author zhangdaquan
 * @param [url]
 * @return org.apache.http.client.methods.CloseableHttpResponse
 * @exception
 */
public static String getResponse(String url) throws IOException {
    HttpGet get = new HttpGet(url);
    String response = httpclient.execute(get,responseHandler);
    return response;
}
 
/*
 * Function: Send http request and parse it with net.sf.json tool
 * @author zhangdaquan
 * @param [url]
 * @return org.json.JSONObject
 * @exception
 */
public static JSONObject getUrl(String url) throws Exception{
    try {
        httpget.setURI(URI.create(url));
        String response = httpclient.execute(httpget,responseHandler);
        JSONObject json = JSONObject.fromObject(response);
        return json;
    } catch (IOException e) {
        e.printStackTrace();
    }
    return null;
}
/*
 * Function: Send http request and parse it with jackson tool
 * @author zhangdaquan
 * @param [url]
 * @return com.fasterxml.jackson.databind.JsonNode
 * @exception
 */
public static JsonNode getUrl2(String url){
    try {
        httpget.setURI(URI.create(url));
        String response = httpclient.execute(httpget,responseHandler);
        JsonNode node = mapper.readTree(response);
        return node;
    } catch (IOException e) {
        e.printStackTrace();
    }
    return null;
}
/*
 * Function: Send http request and parse it with fastjson tool
 * @author zhangdaquan
 * @param [url]
 * @return com.fasterxml.jackson.databind.JsonNode
 * @exception
 */
public static com.alibaba.fastjson.JSONObject getUrl3(String url){
    try {
        httpget.setURI(URI.create(url));
        String response = httpclient.execute(httpget,responseHandler);
        com.alibaba.fastjson.JSONObject jsonObject = com.alibaba.fastjson.JSONObject.parseObject(response);
        return jsonObject;
    } catch (IOException e) {
        e.printStackTrace();
    }
    return null;
}

Welcome to join my knowledge planet and comprehensively improve your technical capabilities.