The etcd cluster space exceeded the quota in two k8s clusters on the same day. When kubectl get cs, it was found that all etcd returned 503 errors. Check etcd alarms and found that NO SPACE information and etcdctl --endpoints=${ETCD_ENDPOINT} --cert=${ETCD_CERTFILE} --key=${ETCD_KEYFILE} --cacert=${ETCD_CAFILE} endpoint status DB SIZE in is greater than 2GiB.
Version Information
-
kubernetes version: v1.17.0
-
etcd version: 3.3.10, running mode is docker container
Solution steps
Note: The repair will cause all nodes to be Not Ready, and the
kubeletof each node needs to be restarted.
1. Adjust the API version to 3
$ export ETCDCTL_API=3
2. Declare variable ETCD_ENDPOINT
export ETCD_ENDPOINT=(https://127.0.0.1:2379) export ETCD_CAFILE=(/etc/kubernetes/pki/etcd/ca.crt) export ETCD_CERTFILE=(/etc/kubernetes/pki/etcd/server.crt) export ETCD_KEYFILE=(/etc/kubernetes/pki/etcd/server.key)
If you don’t know where the etcd certificate is stored, you can use the ps command to view it. Of course, the endpoint can also use this method.
$ ps -ef | grep -v grep | grep apiserver |sed 's/ /\\ /g' | grep etcd --etcd-cafile=/etc/ssl/etcd/ssl/ca.pem --etcd-certfile=/etc/ssl/etcd/ssl/node-master-1.pem --etcd-keyfile=/etc/ssl/etcd/ssl/node-master-1-key.pem --etcd-servers=https://172.16.200.101:2379,https://172.16.200.102:2379,https://172.16.200.103:2379 --storage-backend=etcd3
3. Back up etcd
$ etcdctl --endpoints=${ETCD_ENDPOINT} --cert=${ETCD_CERTFILE} --key=${ETCD_KEYFILE} --cacert=${ETCD_CAFILE} \
snapshot save /var/lib/etcd/my-snapshot.db
4. Get the current revision number and select the history that needs to be truncated
$ rev=$(etcdctl --cert=${ETCD_CERTFILE} --key=${ETCD_KEYFILE} --cacert=${ETCD_CAFILE} endpoint status --write-out="json" | \
egrep -o '"revision":[0-9]*' | egrep -o '[0-9]*'| awk 'NR==1{print $1}')
5. Compress according to the number obtained in step 4
$ etcdctl --endpoints=${ETCD_ENDPOINT} --cert=${ETCD_CERTFILE} --key=${ETCD_KEYFILE} --cacert=${ETCD_CAFILE} compact $rev
6. Free up physical space
$ etcdctl --endpoints=${ETCD_ENDPOINT} --cert=${ETCD_CERTFILE} --key=${ETCD_KEYFILE} --cacert=${ETCD_CAFILE} defrag
It is best to separate the nodes in the endpoints list and execute this step in sequence, otherwise it will easily cause cluster shock.
7. Clear the alarm
$ etcdctl --endpoints=${ETCD_ENDPOINT} --cert=${ETCD_CERTFILE} --key=${ETCD_KEYFILE} --cacert=${ETCD_CAFILE} alarm disarm
This command may need to be executed twice, make sure
etcdctl --endpoints=${ETCD_ENDPOINT} --cert=${ETCD_CERTFILE} --key=${ETCD_KEYFILE} --cacert=${ETCD_CAFILE} alarm listNo more output
8. Restart kubelet
After a period of time, if it is found that all nodes are in the Not Ready state, you only need to restart the kubelet of each node.
$ for node in `kubectl get node --no-headers | awk '{print $1}' | xargs`;do ssh ${node} 'systemctl restart kubelet';done
9. Observe the DB SIZE for a period of time to ensure that there is no significant increase.
$ etcdctl --endpoints=${ETCD_ENDPOINT} --cert=${ETCD_CERTFILE} --key=${ETCD_KEYFILE} --cacert=${ETCD_CAFILE} endpoints status
Reference: After compression, the DB SIZE is 30MiB. It increases to 132MiB after a period of time and then remains unchanged.
Why is the limit exceeded?
What surprised me was that the two clusters with the problem had 30 and 33 nodes respectively, and the etcd of other clusters with more nodes (48 nodes) was not only normal, but also the DB SIZE of etcd was only It’s 143MiB, not even half of it. At the same time, after the problem occurred and the two clusters were repaired, the etcd space was full again the next afternoon. This forced me to explore the cause of the problem.
Automatic Compression Mechanism
Regarding the automatic compression mechanism, the official said that in order to prevent a large amount of data from being stored in etcd, which will lead to performance degradation and eventually space exhaustion, etcd will periodically perform a compression operation on historical data. After the compression is completed, the data before the specified revision version will become inaccessible, and the compressed space will leave room for the storage of new keys. So what is the period of this automatic compression?
Since this is an example of version 3.3.x, I will only explain the automatic compression of version 3.3.x. The automatic compression parameter --auto-compaction-mode=periodic --auto-compaction-retention=30m means to compress every 30 minutes and retain 30 minutes of data each time. Assume that the current revision is 7617 and the revision number will be 14221 in 30 minutes. Then after compression, the keys stored before the revision number is 7617 will not be accessible. . So what is the default etcd configuration that causes problems?
$ cat /etc/etcd.env # Environment file for etcd v3.3.10 ... ETCD_AUTO_COMPACTION_RETENTION=8 ...
This is the default configuration, which retains 8 hours of data. At the same time, the compression mode is periodic if not specified, that is, periodic compression.
--auto-compaction-mode - Interpret 'auto-compaction-retention' one of: 'periodic', 'revision'. 'periodic' for duration based retention, defaulting to hours if no time unit is provided (e.g. '5m '). 'revision' for revision number based retention.- - default: periodic - env variable: ETCD_AUTO_COMPACTION_MODE
However: unlike the data retention amount less than 1 hour, it is compressed according to the configuration period of
auto-compaction-retention. When the amount of retained data is greater than 1 hour, compression is performed at a frequency of 1h/time. That is, the relationship between the retention amount and the compression frequency is as follows:if ( “–auto-compaction-retention” < 1h ):
? Compression frequency == 1 time/”–auto-compaction-retention”
else:
? Compression frequency == 1 time/h
That is to say, the default configuration of etcd where the problem occurred is: Compression is performed every 1 hour, and 8 hours of data is retained. At the same time, the maximum space is the default 2GiB. In other words, if the amount of data in 8 hours exceeds 2GiB, an etcd alarm will be triggered. So why are only these two clusters having problems? Later I discovered that these two clusters are test environments. In the test environment, a large amount of data will always be written to the etcd cluster, and if different business groups test applications at the same time, etcd will frequently perform creation operations. etcd only performs compression once every hour. If too much data is stored in etcd in one hour, etcd will not have time to compress it. Over time, the limit will be reached and an alarm will be triggered. In contrast, in a production environment with more nodes, since few creation operations are performed, the DB SIZE will naturally be small. After understanding this, you will understand how to avoid etcd space alarms. Just modify the following parameters in etcd.env and restart etcd:
#Update retention time shortened to 4 hours ETCD_AUTO_COMPACTION_RETENTION=4 #Add the configuration that the etcd maximum space is 8GiB ETCD_QUOTA_BACKEND_BYTES=8589934592 The etcd cluster parameters started without docker are as follows: --quota-backend-bytes=8589934592 --auto-compaction-retention=4
After observing for a few more days, the two etcd clusters no longer had space excess problems.
KeySpace and storage space
In step 6 of the solution step, the defrag command will be executed, which is used to release fragmented files. The fragmented file is the historical data after etcd performed the compaction operation. This part of the space can be used to store new keys. Therefore, there is no need to perform defrag after performing the compaction operation. The defrag operation is only required when the etcd space is full and the space is released.
How to check the actual usage of etcd space?
The DB SIZE viewed through the etcdctl command is the largest data size in etcd history, not the current data size. etcd exposes mertics to provide monitoring information, in which the value named etcd_mvcc_db_total_size_in_use_in_bytes is the current size of the data. The query method can use the following command:
curl -s --cacert ${ETCD_CA_FILE} --cert ${ETCD_CERT_FILE} --key ${ETCD_KEY_FILE} `echo ${ETCD_ENDPOINTS} \
| awk -F, '{print $1}'`/metrics -o-| egrep ^etcd_mvcc_db_total_size_in_use_in_bytes | awk '{print $2}' | awk '{printf("%.2fMiB",$0/ 1024/1024)}'
The knowledge points of the article match the official knowledge files, and you can further learn relevant knowledge. Cloud native entry-level skills treeHomepageOverview 16742 people are learning the system