We talked about many problems in the basics and case studies, such as RT jitter problems, packet loss problems, inability to establish connections, etc. These problems are usually accompanied by TCP retransmissions, so we often capture TCP retransmission information to help us analyze these problems.
And TCP retransmission is also a signal. We usually use this signal to judge whether the system is stable. For example, if the TCP retransmission rate of a server is very high, then there must be a problem with this server, and we need to take timely measures, otherwise more serious failures may occur.
However, TCP retransmission rate analysis is not an easy task. For example, if the TCP retransmission rate of a certain server is very high, then what business is performing TCP retransmission? Many people don’t know how to analyze this. So, in this lesson, I will take you to understand what TCP retransmission is and how to analyze it efficiently.
What is TCP retransmission?
In the “Opening Words”, an example of TCP retransmission rate is given, as shown in the following figure:
This is a common TCP retransmission rate monitoring for Internet companies. It is an indicator of server stability. If it is too high, like the glitches in the picture above, it often means that the server is unstable. So what does the TCP retransmission rate mean?
In fact, the TCP retransmission rate is calculated by parsing the indicators in the /proc/net/snmp file. The key indicators related to TCP in this file are as follows:
The calculation formula of TCP retransmission rate is as follows:
retrans = (RetransSegs－last RetransSegs) / (OutSegs－last OutSegs) * 100
In other words, the number of TCP retransmission packets per unit time divided by the total number of TCP packets sent is the TCP retransmission rate. Then let’s continue to look at what’s going on with RetransSegs and OutSegs in this formula. I drew two example graphs to demonstrate the changes in these two indicators:
Through these two example diagrams, you can find that after the sender sends a TCP packet, it will put the packet in the sender’s send queue, also called the retransmission queue. At this time, OutSegs will be increased by 1 accordingly, and the queue length is also 1. If the receiving end’s ACK for this data packet can be received, the data packet will be deleted from the sending queue, and then the queue length will become 0; if the ACK for this data packet cannot be received, the retransmission mechanism will be triggered. What we demonstrate here is the situation of timeout retransmission, which means that when the sender sends a data packet, it will start a timeout retransmission timer (RTO). If this time is exceeded, the sender has not received the ACK. , the data packet will be retransmitted, then OutSegs will be incremented by 1, and RetransSegs will also be incremented by 1.
This is the meaning of OutSegs and RetransSegs: for every TCP packet sent out (including retransmission packets), OutSegs will increase by 1 accordingly; for every retransmission packet sent out, RetransSegs will increase by 1 accordingly. At the same time, the changes in the retransmission queue are also shown in the figure, you can take a closer look.
In addition to the timeout retransmission shown in the figure above, there is also a fast retransmission mechanism. Regarding fast retransmission, you can refer to “Lecture 13”, which will not be described in detail here.
After understanding how TCP retransmission is defined, let’s continue to look at what situations will cause TCP retransmission.
The situations that cause TCP retransmission can be divided into the following two categories as a whole.
Packet loss
TCP data packets may be discarded during network transmission; the receiving end may also discard the data packet; the ACK returned by the receiving end may also be discarded during network transmission; data packets may be discarded due to errors during transmission. The receiving end discards… These situations will cause the sending end to retransmit the TCP packet.
Congestion
TCP data packets may be queued on a switch/router during network transmission, such as the notorious Bufferbloat; TCP data packets may be out of order due to routing changes during network transmission; ACK queued at some switch/router… These situations will cause the sender to retransmit the TCP packet again.
In summary, TCP retransmissions can be a good signal of communication quality, and we need to pay attention to it.
So, when we find that the TCP retransmission rate on a certain host is very high, how should we analyze it?
Conventional means of analyzing TCP retransmissions
The most common analysis method is tcpdump, which we can use to save the data packets entering and leaving a certain network card:
$ tcpdump -s 0 -i eth0 -w tcpdumpfile
Then on Linux we can use the tshark tool (the Linux version of wireshark) to filter out TCP retransmission packets:
$ tshark -r tcpdumpfile -R tcp.analysis.retransmission
If there are retransmission packets, they can be displayed. The following is an example of TCP retransmission:
3481 20.277303 10.17.130.20 -> 124.74.250.144 TCP 70 [TCP Retransmission] 35993 > https [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=3231504691 TSecr=0 3659 22.277070 10.17.130.20 -> 124.74.250.144 TCP 70 [TCP Retransmission] 35993 > https [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=3231506691 TSecr=0 8649 46.539393 58.216.21.165 -> 10.17.130.20 TLSv1 113 [TCP Retransmission] Change Cipher Spec, Encrypted Handshake Message
With the help of tcpdump, we can see the details of TCP retransmissions. From the above TCP retransmission information, we can see that this is a retransmission that occurred on the TCP connection 10.17.130.20:35993 – 124.74.250.144: 443; through the [SYN] TCP connection status, we can see This is a retransmission that occurs during the three-way handshake phase. Based on this information, we can continue to go to the host 124.74.250.144 to analyze why the https service cannot establish a new connection.
However, we all know that tcpdump is very heavy. If collected directly in the production environment, it will inevitably have a performance impact on the business. Are there any more lightweight analysis methods?
How to analyze TCP retransmissions efficiently?
In fact, just as an application needs to call corresponding functions to implement some functions, TCP retransmission also needs to call specific kernel functions. This kernel function is tcp_retransmit_skb(). You can understand the skb in the name of this function as a network packet that needs to be sent. So, if we want to track TCP retransmissions efficiently, we can just track this function directly.
The most common way to track kernel functions is to use Kprobe. The general principle of Kprobe is as follows:
A kernel module can be implemented. In the kernel module, Kprobe is used to insert a probe into the tcp_retransmit_skb function entry, and then register a break_handler, so that when tcp_retransmit_skb is executed, an exception will be jumped to the registered break_handler, and then the TCP report is parsed in break_handler. Text (skb) is enough to determine what is being retransmitted.
If you find it troublesome to implement the kernel module, you can use Kprobe with the help of the ftrace framework. tcpretrans implemented by Brendan Gregg uses this method, and you can also use it directly as a tool to track TCP retransmissions. However, this tool also has some flaws, because it reads the /proc/net/tcp file to parse what is being retransmitted, so the information it can parse is relatively limited, and if the TCP connection duration is short (such as short connection), then the tool cannot parse it. In addition, when using it, you need to ensure that the kernel has turned on the tracing function of ftrace, that is
The content in /sys/kernel/debug/tracing/tracing_on needs to be 1; on CentOS-6, it also needs
/sys/kernel/debug/tracing/tracing_enabled is also 1.
$ cat /sys/kernel/debug/tracing/tracing_on 1
If 0, you need to turn them on, for example:
$ echo 1 > /sys/kernel/debug/tracing/tracing_on
Then after the traces are finished, you need to close them:
$ echo 0 > /sys/kernel/debug/tracing/tracing_on
Since Kprobe works through exceptions, it still has some performance overhead. You should avoid using Kprobe on the fast path of TCP packet sending. However, since the retransmission path is a slow path, there is no need to worry about performance overhead by adding Kprobe on the retransmission path.
Kprobe is still a little inconvenient to use. In order to allow Linux users to observe TCP retransmission events more conveniently, the 4.16 kernel version specifically adds TCP tracepoint to parse TCP retransmission events. If the operating system you are using is CentOS-7 and older versions, you cannot use this Tracepoint to observe; if your version is CentOS-8 and subsequent newer versions, then you can directly use this Tracepoint to track TCP redirects. To transfer, you can use the following command:
$ cd /sys/kernel/debug/tracing/events/ $ echo 1 > tcp/tcp_retransmit_skb/enable
Then you can track TCP retransmission events:
$ cat trace_pipe <idle>-0 [007] ... 0.18.225 state=TCP_ESTABLISHED
As you can see, when a TCP retransmission occurs, the basic information of the event is printed. One more thing, there is no “state=TCP_ESTABLISHED” in the initial version. Without this item, we cannot identify whether the retransmission event occurred in the three-way handshake stage, so we contribute a PATCH to the kernel to display the TCP connection status to facilitate problem analysis. For details, see tcp: expose sk_state in tcp_retransmit_skb tracepoint This commit.
After tracing is completed, this Tracepoint needs to be closed:
$ echo 0 > tcp/tcp_retransmit_skb/enable
Tracepoint is not only more convenient to use, but its performance overhead is smaller than Kprobe, so we can also use it on the fast path.
Because Tracepoint supports TCP retransmission events, the tcpretrans tool has also been upgraded. It implements tracking of TCP retransmission events by parsing the Tracepoint, instead of using the previous Kprobe method. For details, please refer to bcc tcpretrans.
Our analysis of TCP retransmission events will start here. We hope to bring some inspiration to develop some more efficient tools to analyze TCP problems or other problems encountered.
Summary
In this class, we mainly talked about some knowledge of TCP retransmission. You need to remember the following points about TCP retransmission:
The TCP retransmission rate can be used as a signal of TCP communication quality. If it is high, it means that the TCP connection is unstable;
The main problems that cause TCP retransmission are packet loss and network congestion;
A specific kernel function will be called during TCP retransmission. We can track the call of this function to track TCP retransmission events;
Kprobe is a very general tracking tool. On lower version kernels, you can use this method to track TCP retransmission events;
Tracepoint is a more lightweight and convenient tool for tracking TCP retransmissions, but it requires a kernel version of 4.16+; if you want something simpler, you can use the tcpretrans tool directly.
The knowledge points of the article match the official knowledge files, and you can further learn related knowledge. Network skill treeCommunication learning in cross-regional networksThe role of the network layer 42,397 people are learning the system