Level 6 K8s Overcoming Strategy (3) – Service Health Check
——> Course videos are shared simultaneously on Toutiao and Bilibili
Health Check
Hello everyone, I am Boge AiOperation and Maintenance. Observe the 2023 Kubernetes Practical Strategy column. Here we go one step further and talk about the health detection features of services on K8s. On K8s, the powerful self-healing capability is a very important feature of this container orchestration engine. The default implementation of self-healing is to automatically restart failed containers to restore them to normal. In addition, we can also use the Liveness and Readiness detection mechanisms to set more detailed health detection indicators to achieve the following requirements:
Zero downtime deployment
Avoid deploying invalid service images
More secure rolling upgrades
Let’s first practice and learn the Healthz Check function of K8s. Let’s first learn the default health detection mechanism of K8s:
Each container will execute a process when it starts. This process is specified by CMD or ENTRYPOINT of Dockerfile. It will be returned when the process in the container exits. If the status code is non-zero, it will be considered that the container has failed, and K8s will restart the container according to the restartPolicy to achieve self-healing effects.
Let’s try it out and simulate the scenario when a container fails:
# First generate a pod yaml configuration file and modify it accordingly
# kubectl run busybox --image=busybox --dry-run=client -o yaml > testHealthz.yaml
# vim testHealthz.yaml
apiVersion: v1
Kind: Pod
metadata:
creationTimestamp: null
labels:
run: busybox
name: busybox
spec:
containers:
- image: registry.cn-hangzhou.aliyuncs.com/acs/busybox:v1.29.2
name: busybox
resources: {<!-- -->}
args:
-/bin/sh
- -c
- sleep 10; exit 1 # And add pod to run the specified script command to simulate a failure 10 seconds after the container starts, and the exit status code is 1
dnsPolicy:ClusterFirst
restartPolicy: OnFailure # Change the default Always to OnFailure
status: {<!-- -->}
Restart Strategy
Description
Always
When a container fails, kubelet automatically restarts the container
OnFailure
When the container terminates and the exit code is not 0, kubelet automatically restarts it The container
Never
No matter the running status of the container, kubelet will not restart the container
< /table>
Execute configuration to create pod
# kubectl apply -f testHealthz.yaml
pod/busybox created
# Observe for a few minutes and use the -w parameter to continuously monitor the status changes of the pod.
# kubectl get pod -w
NAME READY STATUS RESTARTS AGE
busybox 0/1 ContainerCreating 0 4s
busybox 1/1 Running 0 6s
busybox 0/1 Error 0 16s
busybox 1/1 Running 1 22s
busybox 0/1 Error 1 34s
busybox 0/1 CrashLoopBackOff 1 47s
busybox 1/1 Running 2 63s
busybox 0/1 Error 2 73s
busybox 0/1 CrashLoopBackOff 2 86s
busybox 1/1 Running 3 109s
busybox 0/1 Error 3 2m
busybox 0/1 CrashLoopBackOff 3 2m15s
busybox 1/1 running 4 3m2s
busybox 0/1 Error 4 3m12s
busybox 0/1 CrashLoopBackOff 4 3m23s
busybox 1/1 Running 5 4m52s
busybox 0/1 Error 5 5m2s
busybox 0/1 CrashLoopBackOff 5 5m14s
As you can see above, this test pod has been restarted 5 times. However, the service is still not normal and will remain in CrashLoopBackOff, waiting for the operation and maintenance personnel to conduct the next error troubleshooting.
Note: kubelet will restart them with exponential backoff delays (10s, 20s, 40s, etc.), capped at 5 minutes
Here we are testing by artificially simulating service failures. In actual production work, for business services, how can we use this mechanism to restart containers to recover to configure business services? The answer is `liveness` detection
Liveness
Liveness detection allows us to customize conditions to determine whether the container is healthy. If the detection fails, K8s will restart the container. Let’s take an example to practice. Prepare the following yaml configuration and save it as liveness.yaml:
apiVersion: v1
Kind: Pod
metadata:
labels:
test:liveness
name: liveness
spec:
restartPolicy: OnFailure
containers:
- name: liveliness
image: registry.cn-hangzhou.aliyuncs.com/acs/busybox:v1.29.2
args:
-/bin/sh
- -c
- touch /tmp/healthy; sleep 30; rm -f /tmp/healthy; sleep 600
livenessProbe:
exec:
command:
-cat
- /tmp/healthy
initialDelaySeconds: 10 # Start detection 10 seconds after the container is started
periodSeconds: 5 # Check again every 5 seconds
The startup process first creates the file /tmp/healthy and deletes it after 30 seconds. In our settings, if the /tmp/healthy file exists, the container is considered to be in a normal state. Otherwise, a failure occurs.
The livenessProbe section defines how to perform Liveness detection:
The detection method is: use the cat command to check whether the /tmp/healthy file exists. If the command is executed successfully and the return value is zero, K8s considers the Liveness detection successful; if the command return value is non-zero, the Liveness detection fails.
initialDelaySeconds: 10 specifies that the Liveness detection will start after 10 seconds when the container is started. We generally set it according to the preparation time for application startup. For example, if an application takes 30 seconds to start normally, the value of initialDelaySeconds should be greater than 30.
periodSeconds: 5 specifies that Liveness checks should be performed every 5 seconds. If K8s fails to perform Liveness detection three times in a row, it will kill and restart the container.
Next, create this Pod:
# kubectl apply -f liveness.yaml
pod/liveness created
It can be seen from the configuration file that in the first 30 seconds, /tmp/healthy exists, the cat command returns 0, and Liveness detection is successful. During this period, the Events section of kubectl describe pod liveness will display normal logs.
# kubectl describe pod liveness
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 53s default-scheduler Successfully assigned default/liveness to 10.0.1.203
Normal Pulling 52s kubelet Pulling image "busybox"
Normal Pulled 43s kubelet Successfully pulled image "busybox"
Normal Created 43s kubelet Created container liveness
Normal Started 42s kubelet Started container liveness
After 35 seconds, the log will show that /tmp/healthy no longer exists and the Liveness detection failed. After a few tens of seconds and several tests fail, the container will be restarted.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m53s default-scheduler Successfully assigned default/liveness to 10.0.1.203
Normal Pulling 73s (x3 over 3m52s) kubelet Pulling image "busybox"
Normal Pulled 62s (x3 over 3m43s) kubelet Successfully pulled image "busybox"
Normal Created 62s (x3 over 3m43s) kubelet Created container liveness
Normal Started 62s (x3 over 3m42s) kubelet Started container liveness
Warning Unhealthy 18s (x9 over 3m8s) kubelet Liveness probe failed: cat: can't open '/tmp/healthy': No such file or directory
Normal Killing 18s (x3 over 2m58s) kubelet Container liveness failed liveness probe, will be restarted
In addition to Liveness detection, the Kubernetes Health Check mechanism also includes Readiness detection.
Readiness
We can use readiness detection to tell K8s when the pod can be added to the load balancing pool of the service to provide external services. This is very important when the production scenario service releases a new version. When a program error occurs in the new version we launch, Readiness will be released through detection, so that traffic is not imported into the pod and service failures are controlled internally. In production scenarios, it is recommended that this be added. Liveness can be omitted, because sometimes we need to retain the scene of service errors. To query the logs, locate the problem, and notify development to fix the program.
The configuration syntax of the Readiness test is exactly the same as the Liveness test. Here is an example:
apiVersion: v1
Kind: Pod
metadata:
labels:
test: readiness
name: readiness
spec:
restartPolicy: OnFailure
containers:
- name: readiness
image: registry.cn-hangzhou.aliyuncs.com/acs/busybox:v1.29.2
args:
-/bin/sh
- -c
- touch /tmp/healthy; sleep 30; rm -f /tmp/healthy; sleep 600
readinessProbe: # Just replace livenessProbe with readinessProbe. Other configurations are the same.
exec:
command:
- cat
- /tmp/healthy
#initialDelaySeconds: 10 # Start detection 10 seconds after the container is started
periodSeconds: 5 # Check again every 5 seconds
startupProbe: # Startup probe, more flexible, perfect replacement for initialDelaySeconds mandatory waiting time configuration, detected every 3 seconds at startup, 100 times in total
exec:
command:
-cat
- /tmp/healthy
failureThreshold: 100
periodSeconds: 3
timeoutSeconds: 1
Save the above configuration as readiness.yaml and execute it to generate pods:
# kubectl apply -f readiness.yaml
pod/liveness created
# Observe that the file was not deleted when it was first created, so everything is normal.
# kubectl get pod
NAME READY STATUS RESTARTS AGE
Liveness 1/1 Running 0 50s
# Then after 35 seconds, the file is deleted. At this time, the READY status will change, and K8s will disconnect the traffic from the Service to the pod.
# kubectl describe pod liveness
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 56s default-scheduler Successfully assigned default/liveness to 10.0.1.203
Normal Pulling 56s kubelet Pulling image "busybox"
Normal Pulled 40s kubelet Successfully pulled image "busybox"
Normal Created 40s kubelet Created container liveness
Normal Started 40s kubelet Started container liveness
Warning Unhealthy 5s (x2 over 10s) kubelet Readiness probe failed: cat: can't open '/tmp/healthy': No such file or directory
# You can see that the pod's traffic is disconnected. Even if the service fails at this time, it will not be perceived by the outside world. At this time, our operation and maintenance personnel can perform troubleshooting.
# kubectl get pod
NAME READY STATUS RESTARTS AGE
liveness 0/1 Running 0 61s
The following is a comparison between the Liveness test and the Readiness test:
Liveness detection and readiness detection are two health check mechanisms. If not configured specifically, Kubernetes will adopt the same default behavior for both detections, that is, by determining whether the return value of the container startup process is zero to determine whether the detection is successful.
The configuration methods of the two detections are exactly the same, and the supported configuration parameters are also the same. The difference lies in the behavior after the detection fails: Liveness detection is to restart the container; Readiness detection is to set the container to be unavailable and not accept requests forwarded by the Service.
Liveness detection and readiness detection are executed independently, and there is no dependency between them, so they can be used separately or at the same time. Use the Liveness test to determine whether the container needs to be restarted to achieve self-healing; use the Readiness test to determine whether the container is ready to provide external services.
Three ways to use Liveness and Readiness
readinessProbe: # Define only the http detection container 6222 port request return is 200-400, then receive the following Service web-svc request
httpGet:
scheme: HTTP
path: /check
port: 6222
initialDelaySeconds: 10 # The detection starts 10 seconds after the container starts. Pay attention to the startup success time of g1.
periodSeconds: 5 # Detect again every 5 seconds
timeoutSeconds: 5 # http detection request timeout
successThreshold: 1 # If one success is detected, the service is considered `ready`
failureThreshold: 3 # If 3 failures are detected, the service is considered `not ready`
livenessProbe: # Define only http detection container 6222 port request return is 200-400, otherwise restart the pod
httpGet:
scheme: HTTP
path: /check
port: 6222
initialDelaySeconds: 10
periodSeconds: 5
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 3
#---------------------------------------------
readinessProbe:
exec:
command:
- sh
- -c
- "redis-cli ping"
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 3
livenessProbe:
exec:
command:
- sh
- -c
- "redis-cli ping"
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 3
#---------------------------------------------
readinessProbe:
tcpSocket:
port: 9092
initialDelaySeconds: 15
periodSeconds: 10
livenessProbe:
tcpSocket:
port: 9092
initialDelaySeconds: 15
periodSeconds: 10
Application scenarios of Health Check in rolling update in business production
For operation and maintenance personnel, updating the new project code of the service and ensuring its stable operation is a critical and highly repetitive task. In the traditional mode, we usually use batch management tools such as saltsatck or ansible. To push the code to each server for update, then on K8s, this update process is simplified. In the later high-level chapters, I will talk about the CI/CD automation process, which is roughly that the developer develops the code and uploads it to the code warehouse. The CI/CD process will be triggered, and there is basically no need for operation and maintenance personnel to participate. So in such a highly automated process, how do our operation and maintenance personnel ensure that the service can be stably online? Readiness in Health Check can play a key role. This has actually been mentioned above. Here we will explain it again with an example to deepen our impression:
# kubectl apply -f myapp-v1.yaml --record
deployment.apps/mytest created
# Wait for a while, you can see that all pods are running normally
# kubectl get pod
NAME READY STATUS RESTARTS AGE
mytest-d9f48585b-2lmh2 1/1 Running 0 3m22s
mytest-d9f48585b-5lh9l 1/1 Running 0 3m22s
mytest-d9f48585b-cwb8l 1/1 Running 0 3m22s
mytest-d9f48585b-f6tzc 1/1 Running 0 3m22s
mytest-d9f48585b-hb665 1/1 Running 0 3m22s
mytest-d9f48585b-hmqrw 1/1 Running 0 3m22s
mytest-d9f48585b-jm8bm 1/1 Running 0 3m22s
mytest-d9f48585b-kxm2m 1/1 Running 0 3m22s
mytest-d9f48585b-lqpr9 1/1 Running 0 3m22s
mytest-d9f48585b-pk75z 1/1 Running 0 3m22s
Then we are going to prepare to update this service, and artificially simulate version failures to observe, and prepare a new configuration myapp-v2.yaml
# cat myapp-v2.yaml
apiVersion: apps/v1
Kind: Deployment
metadata:
name: mytest
spec:
strategy:
rollingUpdate:
maxSurge: 35% # Maximum total number of copies for rolling update (taking the base of 10 as an example): 10 + 10 * 35% = 13.5 --> 14
maxUnavailable: 35% # Maximum number of available replicas (both default values are 25%): 10 - 10 * 35% = 6.5 --> 7
replicas: 10
selector:
matchLabels:
app:mytest
template:
metadata:
labels:
app:mytest
spec:
containers:
- name: mytest
image: registry.cn-hangzhou.aliyuncs.com/acs/busybox:v1.29.2
args:
-/bin/sh
- -c
- sleep 30000 # It can be seen that the file /tmp/healthy is not generated here, so the following detection must fail.
readinessProbe:
exec:
command:
-cat
- /tmp/healthy
initialDelaySeconds: 10
periodSeconds: 5
Obviously, because the /tmp/healthy file will not be generated in the v2 version we updated, it will automatically fail the Readiness detection. The details are as follows:
# kubectl apply -f myapp-v2.yaml --record
deployment.apps/mytest configured
# kubectl get deployment mytest
NAME READY UP-TO-DATE AVAILABLE AGE
mytest 7/10 7 7 4m58s
# READY There are only 7 pods running now
# UP-TO-DATE indicates the number of copies that have been updated: 7 new copies
#AVAILABLE represents the number of replicas currently in READY state
# kubectl get pod
NAME READY STATUS RESTARTS AGE
mytest-7657789bc7-5hfkc 0/1 Running 0 3m2s
mytest-7657789bc7-6c5lg 0/1 Running 0 3m2s
mytest-7657789bc7-c96t6 0/1 Running 0 3m2s
mytest-7657789bc7-nbz2q 0/1 Running 0 3m2s
mytest-7657789bc7-pt86c 0/1 Running 0 3m2s
mytest-7657789bc7-q57gb 0/1 Running 0 3m2s
mytest-7657789bc7-x77cg 0/1 Running 0 3m2s
mytest-d9f48585b-2bnph 1/1 Running 0 5m4s
mytest-d9f48585b-965t4 1/1 Running 0 5m4s
mytest-d9f48585b-cvq7l 1/1 Running 0 5m4s
mytest-d9f48585b-hvpnq 1/1 Running 0 5m4s
mytest-d9f48585b-k89zs 1/1 Running 0 5m4s
mytest-d9f48585b-wkb4b 1/1 Running 0 5m4s
mytest-d9f48585b-wrkzf 1/1 Running 0 5m4s
# As you can see above, since the Readiness test has never passed, the new version of the pod is in the Not ready state. This ensures that the wrong business code will not be requested by the outside world.
# kubectl describe deployment mytest
# Below are some key information needed here.
...
Replicas: 10 desired | 7 updated | 14 total | 7 available | 7 unavailable
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 5m55s deployment-controller Scaled up replica set mytest-d9f48585b to 10
Normal ScalingReplicaSet 3m53s deployment-controller Scaled up replica set mytest-7657789bc7 to 4 # Start 4 new versions of pods
Normal ScalingReplicaSet 3m53s deployment-controller Scaled down replica set mytest-d9f48585b to 7 # Reduce the number of old version pods to 7
Normal ScalingReplicaSet 3m53s deployment-controller Scaled up replica set mytest-7657789bc7 to 7 # Add 3 new startups to 7 new versions
Based on the above analysis, we realistically simulated the K8s last erroneous code online process. Fortunately, the Readiness detection of Health Check helped us block the erroneous copy so that it would not be requested by external traffic, while retaining Most of the old version pods have been removed, so the business of the entire service has not been affected by this update failure.
Next, let’s analyze the principle of rolling update in detail. Why is the number of pods created by the new version of the service above 7, while only 3 pods of the old version are destroyed?
The reason lies in this configuration:
If we do not configure this section explicitly, the default value is 25%.
Rolling update uses the parameters maxSurge and maxUnavailable to control the update and replacement of the number of pod copies.
maxSurge
This parameter controls the total number of pod copies to exceed the upper limit of the set total number of copies during the rolling update process. maxSurge can be a specific integer (such as 3) or a percentage, rounded up. maxSurge default value is 25%
In the example tested above, the total number of copies of the pod is 10. Then during the update process, the maximum maximum planning formula for the upper limit of the total number of copies is:
10 + 10 * 35% = 13.5 --> 14
Let’s check the description information of the updated deployment:
Replicas: 10 desired | 7 updated | 14 total | 7 available | 7 unavailable
The quantity of the old version available is 7 + the quantity of the new version unavailable is 7 = total quantity 14 total
maxUnavailable-00:38:51
This parameter controls the value of the total number of unavailable pod copies during the rolling update process. Similarly, maxUnavailable can be a specific integer (such as 3), or it can be a hundred percent, rounded down. maxUnavailable default value is 25%.
In the example tested above, the total number of pod copies is 10, so the number of pod copies that must be normally available is:
10 - 10 * 35% = 6.5 --> 7
So in the description information we looked at above, 7 available. The number of normally available pods is 7.
The larger the maxSurge value, the more new copies are initially created; the larger the maxUnavailable value is, the more old copies are initially destroyed.
Normal update Ideally, our rolling update process for this version release case is:
First create 4 new versions of pods, bringing the total number of replicas to 14
Then destroy 3 old version pods, reducing the number of available replicas to 7
When these 3 old version pods are successfully destroyed, 3 new version pods can be created, keeping the total number of copies at 14
When a new version of a pod passes the readiness test, the number of available pod copies will increase by more than 7.
You can then continue to destroy more old version pods, bringing the overall number of available pods back to 7
As old versions of pods are destroyed, keep the total number of pod replicas below 14 so that more new versions of pods can continue to be created.
This new destruction process will continue, and eventually all old version pods will be gradually replaced by new version pods, and the entire rolling update will be completed.
The actual situation here is that we are stuck in step 4, and the number of pods in the new version cannot pass the readiness test. The log at the end of the event section of the description above also details all this:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 5m55s deployment-controller Scaled up replica set mytest-d9f48585b to 10
Normal ScalingReplicaSet 3m53s deployment-controller Scaled up replica set mytest-7657789bc7 to 4 # Start 4 new versions of pods
Normal ScalingReplicaSet 3m53s deployment-controller Scaled down replica set mytest-d9f48585b to 7 # Reduce the number of old version pods to 7
Normal ScalingReplicaSet 3m53s deployment-controller Scaled up replica set mytest-7657789bc7 to 7 # Add 3 new startups to 7 new versions
According to the normal production process here, after obtaining enough new version error information and submitting it to development analysis, we can roll back to the previous normal service version through kubectl rollout undo :
# First check the number in front of the version number to be rolled back, here it is 1
# kubectl rollout history deployment mytest
deployment.apps/mytest
REVISION CHANGE-CAUSE
1 kubectl apply --filename=myapp-v1.yaml --record=true
2 kubectl apply --filename=myapp-v2.yaml --record=true
# kubectl rollout undo deployment mytest --to-revision=1
deployment.apps/mytest rolled back
# kubectl get deployment mytest
NAME READY UP-TO-DATE AVAILABLE AGE
mytest 10/10 10 10 96m
# kubectl get pod
NAME READY STATUS RESTARTS AGE
mytest-d9f48585b-2bnph 1/1 Running 0 96m
mytest-d9f48585b-8nvhd 1/1 Running 0 2m13s
mytest-d9f48585b-965t4 1/1 Running 0 96m
mytest-d9f48585b-cvq7l 1/1 Running 0 96m
mytest-d9f48585b-hvpnq 1/1 Running 0 96m
mytest-d9f48585b-k89zs 1/1 Running 0 96m
mytest-d9f48585b-qs5c6 1/1 Running 0 2m13s
mytest-d9f48585b-wkb4b 1/1 Running 0 96m
mytest-d9f48585b-wprlz 1/1 Running 0 2m13s
mytest-d9f48585b-wrkzf 1/1 Running 0 96m
OK, so far, we have actually simulated a problematic version release and rollback, and we can see that during this entire process, although problems have occurred, our business has not been affected in any way. This is the charm of K8s.
pod monster battle (homework)
# Test the entire update and rollback case above yourself, pay attention to the pod changes, and deepen your understanding.