Author: lqbyz Original source: https://tidb.net/blog/7ecfecd2
TiDB Operator is used to deploy and manage TiDB clusters on Kubernetes, and the version of TiDB clusters can be upgraded through rolling updates to reduce the impact on business. This article describes how to use rolling update to upgrade a TiDB cluster on Kubernetes.
Rolling update feature introduction
Kubernetes provides a rolling update feature to perform updates without affecting application availability.
When using rolling update, TiDB Operator will serially delete the old version of Pods and create new version of Pods in the order of PD, TiFlash, TiKV, and TiDB. When the new version of the Pod is running normally, the next Pod is processed.
During the rolling update, TiDB Operator will automatically handle the leader migration of PD and TiKV. Therefore, under the multi-node deployment topology (minimum environment: PD * 3, TiKV * 3, TiDB * 2), the rolling update of TiKV and PD will not affect the normal operation of the business. For clients with connection retry function, the rolling update of TiDB will also not affect the business.
Notes
-
For clients that cannot perform connection retries, the rolling update of TiDB will cause the connection to the database of the closed node to fail, causing some business requests to fail. For this type of business, it is recommended to add a retry function to the client, or perform TiDB rolling update operations during off-peak periods. -
Before upgrading, please refer to thedocumentationto make sure there are no DDL operations in progress.
Upgrade steps
In TidbCluster, by modifying the mirror configuration of cluster components, generally modify spec.version . If you want to set different versions for different components in the cluster, modify spec, .version .
[root@k8s-master tidb]# kubectl apply -f tidb.yaml tidbcluster.pingcap.com/lqb configured
View upgrade progress
[root@k8s-master ~]# kubectl get pod -ntidb -w NAME READY STATUS RESTARTS AGE yz-discovery-68674b48b8-49vrl 1/1 Running 0 2d1h yz-pd-0 1/1 Running 0 2d22h yz-pd-1 1/1 Running 0 2d22h yz-pd-2 1/1 Running 0 2m55s yz-tidb-0 2/2 Running 0 2d4h yz-tidb-1 2/2 Running 0 2d22h yz-tidb-initializer-b8l8f 0/1 Completed 0 35d yz-tiflash-0 4/4 Running 0 6d4h yz-tikv-0 1/1 Running 0 2d4h yz-tikv-1 1/1 Running 0 6d4h yz-tikv-2 1/1 Running 2 3d yz-tikv-3 1/1 Running 0 3d yz-pd-1 1/1 Terminating 0 2d22h yz-pd-2 1/1 Running 0 3m24s yz-pd-1 0/1 Terminating 0 2d22h yz-pd-1 0/1 Terminating 0 2d22h yz-pd-1 0/1 Terminating 0 2d22h yz-pd-1 0/1 Pending 0 0s yz-pd-1 0/1 Pending 0 0s yz-pd-1 0/1 ContainerCreating 0 0s yz-pd-1 1/1 Running 0 1s yz-pd-0 1/1 Terminating 0 2d22h yz-pd-0 0/1 Terminating 0 2d22h yz-pd-1 1/1 Running 0 21s yz-pd-0 0/1 Terminating 0 2d22h yz-pd-0 0/1 Terminating 0 2d22h yz-pd-0 0/1 Pending 0 0s yz-pd-0 0/1 Pending 0 0s yz-pd-0 0/1 ContainerCreating 0 0s yz-pd-0 0/1 ErrImagePull 0 20s yz-pd-0 0/1 ImagePullBackOff 0 31s
Check whether the cluster upgrade is complete
When all Pods are rebuilt, enter Running and TC is True to indicate that the upgrade is complete.
[root@k8s-master tidb]# kubectl get tc -ntidb NAME READY PD STORAGE READY DESIRE TIKV STORAGE READY DESIRE TIDB READY DESIRE AGE lqb True pingcap/pd:v6.1.0 12Gi 3 3 pingcap/tikv:v6.1.0 12Gi 3 3 pingcap/tidb:v6.1.0 1 1 6d21h yz True pingcap/pd:v6.5.0 12Gi 3 3 pingcap/tikv:v6.5.0 12Gi 4 4 pingcap/tidb:v6.5.0 2 2 6d5h [root@k8s-master tidb]# kubectl get pod -ntidb NAME READY STATUS RESTARTS AGE tidbngmonitoring-yz-ng-monitoring-0 1/1 Running 1 4d13h yz-discovery-68674b48b8-49vrl 1/1 Running 0 2d2h yz-pd-0 1/1 Running 0 55m yz-pd-1 1/1 Running 0 55m yz-pd-2 1/1 Running 0 59m yz-tidb-0 2/2 Running 0 35m yz-tidb-1 2/2 Running 0 38m yz-tidb-initializer-b8l8f 0/1 Completed 0 35d yz-tikv-0 1/1 Running 0 39m yz-tikv-1 1/1 Running 0 43m yz-tikv-2 1/1 Running 0 44m yz-tikv-3 1/1 Running 0 49m
Troubleshooting when upgrading
If the PD cluster is unavailable due to related reasons such as PD configuration errors, PD image tag errors, and NodeAffinity, the TiDB cluster version cannot be successfully upgraded at this time. In this case, force-upgrade can be used to force upgrade the cluster to restore cluster functionality.
The mandatory upgrade steps are as follows:
Set annotation for the cluster and view it
[root@k8s-master tidb]# kubectl annotate --overwrite tc yz -n tidb tidb.pingcap.com/force-upgrade=true tidbcluster.pingcap.com/yz annotated
Modify the PD related configuration to ensure that the PD enters a normal state.
After repairing the PD configuration, delete the forced upgrade function just now, otherwise there may be exceptions in the next upgrade process:
[root@k8s-master tidb]# kubectl annotate tc yz -n tidb tidb.pingcap.com/force-upgrade- tidbcluster.pingcap.com/yz annotated
After completing the above steps, the TiDB cluster function will return to normal and can be upgraded normally.
The knowledge points of the article match the official knowledge files, and you can further learn relevant knowledge. Java skill treeHomepageOverview 108638 people are studying systematically