Table of Contents
Resource limits
Resource limits overview
resource unit
Example
Example 1
Example 2
probe
Three rules for probes
Probe supports three inspection methods
Example
Example 1: exec mode
Example 2: httpGet method
Example 3: tcpSocket method
Example 4: Readiness detection
Example 5: Readiness Detection 2
Expand
pod status
Container life cycle
Resource Limitations
Resource Limitation Overview
When defining a Pod, you can optionally set the number of resources required for each container. The most common configurable resources are CPU and memory size, among other types of resources.
When a request resource is specified for a container in a Pod, the scheduler uses this information to decide which node to schedule the Pod on. When limit resources are also specified for the container, kubelet will ensure that the running container does not use more than the set limit resources. The kubelet will also reserve the set amount of request resources for the container for use by the container.
A container can use more than the requested amount of resources if the node the Pod is running on has enough available resources. However, the container cannot use more resources than the set limit.
If the memory limit value is set for a container, but the memory request value is not set, Kubernetes will automatically set a request value that matches the memory limit. Similarly, if the CPU limit value is set for a container but the CPU request value is not set, Kubernetes automatically sets the CPU request value for it and matches it with the CPU limit value.
Official website example:
https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/
Resource requests and limitations for Pods and Containers:
spec.containers[].resources.requests.cpu //Define pre-allocated CPU resources when creating a container
spec.containers[].resources.requests.memory //Define the memory resources pre-allocated when creating a container
spec.containers[].resources.limits.cpu //Define the resource limit of cpu
spec.containers[].resources.limits.memory //Define the upper limit of memory resources
Resource Unit
CPU resource unit
The request and limit of CPU resources are in cpu. One CPU in Kubernetes is equivalent to 1 vCPU (1 hyperthread).
Kubernetes also supports requests with fractional CPUs. A container whose spec.containers[].resources.requests.cpu is 0.5 can obtain half of the CPU resources of a cpu (similar to Cgroup’s time slicing of CPU resources). The expression 0.1 is equivalent to the expression 100m (millicore), which means that the total amount of CPU time that the container can use every 1000 milliseconds is 0.1*1000 milliseconds.
Kubernetes does not allow setting CPU resources with less than 1m precision.
memory resource unit
Memory request and limit are in bytes. It can be expressed as an integer, or in units of base 10 exponent (E, P, T, G, M, K), or in units of base 2 exponent (Ei, Pi, Ti, Gi, Mi, Ki) to represent.
For example: 1KB=10^3=1000, 1MB=10^6=1000000=1000KB, 1GB=10^9=1000000000=1000MB
1KiB=2^10=1024, 1MiB=2^20=1048576=1024KiB
PS: When buying a hard drive, the quantity quoted by the operating system is smaller than what is marked on the product or claimed by the merchant. The main reason is that the quantity marked is in MB and GB. 1GB is 1,000,000,000Byte, and the operating system is in units of MB and GB. Binary is the processing unit, so when checking the hard disk capacity, MiB and GiB are used as the unit, 1GiB=2^30=1,073,741,824. In comparison, 1GiB is 1,073,741,824-1,000,000,000=73,741,824Byte more than 1GB, so the actual detection The result is less than indicated.
https://kubernetes.io/zh-cn/docs/concepts/configuration/manage-resources-containers/
Example
Example 1
vimpod1.yaml
apiVersion: v1
Kind: Pod
metadata:
name: frontend
spec:
containers:
- name: app
image: images.my-company.example/app:v4
env:
- name: MYSQL_ROOT_PASSWORD
value: "password"
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
- name: log-aggregator
image: images.my-company.example/log-aggregator:v6
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
The Pod in this example has two containers. The request value of each container is 0.25 cpu and 64MiB memory, and the limit value of each container is 0.5 cpu and 128MiB memory. Then it can be considered that the total resource request of the Pod is 0.5 cpu and 128 MiB memory, and the total resource limit is 1 cpu and 256 MiB memory.
Example 2
vimpod2.yaml
apiVersion: v1
Kind: Pod
metadata:
name: frontend
spec:
containers:
- name: web
image: nginx
env:
- name: WEB_ROOT_PASSWORD
value: "password"
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
- name: db
image: mysql
env:
- name: MYSQL_ROOT_PASSWORD
value: "abc123"
resources:
requests:
memory: "512Mi" 128
cpu: "0.5"
limits:
memory: "1Gi" 256
cpu: "1"
kubectl apply -f pod2.yaml kubectl describe pod frontend
kubectl get pods -o wide
kubectl describe nodes node02 #Since the current virtual machine has 2 CPUs, the Pod’s CPU Limits occupy a total of 50%
Probe
Health check: also called probe (Probe)
Probes are periodic diagnostics performed on the container by the kubelet.
Three rules of probes
●livenessProbe: Determine whether the container is running. If the probe fails, the kubelet kills the container and the container sets the Pod state according to the restartPolicy. If the container does not provide a liveness probe, the default state is Success.
●readinessProbe: Determine whether the container is ready to accept requests. If the detection fails, the endpoint controller will delete the IP address of the Pod from all service address endpoints matching the Pod. The ready state before the initial delay defaults to Failure. If the container does not provide a readiness probe, the default status is Success.
●startupProbe (added in version 1.17): Determines whether the application in the container has been started, mainly for applications that cannot determine the specific startup time. If the startupProbe probe is configured, all other probes are inactive until the startupProbe status is Success, and no other probes take effect until it succeeds. If startupProbe fails, kubelet will kill the container and the container will be restarted according to restartPolicy. If the container does not have a startupProbe configured, the default status is Success.
#Note: The above rules can be defined at the same time. Before the readinessProbe test is successful, the running state of the Pod will not change to the ready state.
Probe supports three inspection methods
●exec: Execute the specified command within the container. If the return code is 0 when the command exits, the diagnosis is considered successful.
●tcpSocket: Perform TCP inspection (three-way handshake) on the IP address of the container on the specified port. If the port is open, the diagnosis is considered successful.
●httpGet: Perform an HTTPGet request to the IP address of the container on the specified port and path. If the response’s status code is greater than or equal to 200 and less than 400, the diagnosis is considered successful.
Each probe will have one of three outcomes:
●Success: The container passed the diagnosis.
●Failed: The container failed diagnostics.
●Unknown: Diagnosis failed, so no action will be taken
Official website example:
https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
Example
Example 1: exec mode
apiVersion: v1
Kind: Pod
metadata:
labels:
test:liveness
name:liveness-exec
spec:
containers:
- name: liveliness
image: k8s.gcr.io/busybox
args:
-/bin/sh
- -c
- touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 60
livenessProbe:
exec:
command:
- cat
- /tmp/healthy
failureThreshold: 1
initialDelaySeconds: 5
periodSeconds: 5
#initialDelaySeconds: Specifies that kubelet should wait 5 seconds before executing the first detection, that is, the first detection will not start until 6 seconds after the container is started. Default is 0 seconds, minimum value is 0.
#periodSeconds: Specifies that the kubelet should perform a liveness probe every 5 seconds. The default is 10 seconds. The minimum value is 1.
#failureThreshold: When a probe fails, the number of times Kubernetes will retry before giving up. Giving up in the case of a liveness probe means restarting the container. Pods that are abandoned during readiness detection will be labeled as not ready. The default value is 3. The minimum value is 1.
#timeoutSeconds: How many seconds to wait after the probe times out. The default value is 1 second. The minimum value is 1. (Prior to Kubernetes 1.20, the exec probe ignored timeoutSeconds and the probe continued running indefinitely, possibly even beyond the configured deadline, until results were returned.)
You can see that there is only one container in the Pod. The kubelet needs to wait 5 seconds before performing the first probe. The kubelet will perform a survival probe every 5 seconds. kubelet executes the command cat /tmp/healthy in the container to detect. If the command is executed successfully and the return value is 0, kubelet will consider the container to be healthy and alive. When the 31st second is reached, this command returns a non-zero value and the kubelet kills the container and restarts it.
vim exec.yaml
apiVersion: v1
Kind: Pod
metadata:
name:liveness-exec
namespace:default
spec:
containers:
- name: liveness-exec-container
image: busybox
imagePullPolicy: IfNotPresent
command: ["/bin/sh","-c","touch /tmp/live ; sleep 30; rm -rf /tmp/live; sleep 3600"]
livenessProbe:
exec:
command: ["test","-e","/tmp/live"]
initialDelaySeconds: 1
periodSeconds: 3
kubectl create -f exec.yaml
kubectl describe pods liveness-exec
kubectl get pods -w
Example 2: httpGet method
apiVersion: v1
Kind: Pod
metadata:
labels:
test:liveness
name:liveness-http
spec:
containers:
- name: liveness
image: k8s.gcr.io/liveness
args:
-/server
livenessProbe:
httpGet:
path: /healthz
port: 8080
httpHeaders:
- name: Custom-Header
value: Awesome
initialDelaySeconds: 3
periodSeconds: 3
In this configuration file, you can see that Pod also has only one container. The initialDelaySeconds field tells the kubelet that it should wait 3 seconds before performing the first probe. The periodSeconds field specifies that the kubelet should perform a liveness probe every 3 seconds. The kubelet will send an HTTP GET request to the service running in the container (the service will listen on port 8080) to perform the detection. If the handler under the /healthz path on the server returns a success code, the kubelet considers the container to be healthy. If the handler returns a failure code, the kubelet kills the container and restarts it.
Any return code greater than or equal to 200 and less than 400 indicates success, other return codes indicate failure.
vim httpget.yaml
apiVersion: v1
Kind: Pod
metadata:
name:liveness-httpget
namespace:default
spec:
containers:
- name: liveness-httpget-container
image: soscscs/myapp:v1
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 80
livenessProbe:
httpGet:
port: http
path: /index.html
initialDelaySeconds: 1
periodSeconds: 3
timeoutSeconds: 10
kubectl create -f httpget.yaml
kubectl exec -it liveness-httpget — rm -rf /usr/share/nginx/html/index.html
kubectl getpods
Example 3: tcpSocket method
apiVersion: v1
Kind: Pod
metadata:
name: goproxy
labels:
app: goproxy
spec:
containers:
- name: goproxy
image: k8s.gcr.io/goproxy:0.1
ports:
- containerPort: 8080
readinessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 15
periodSeconds: 20
This example uses both the readinessProbe and livenessProbe probes. The kubelet sends the first readinessProbe probe 5 seconds after the container starts. This will attempt to connect to port 8080 of the goproxy container. If the probe is successful, the kubelet will continue to run the probe every 10 seconds. In addition to the readinessProbe probe, this configuration includes a livenessProbe probe. The kubelet will perform the first livenessProbe probe 15 seconds after the container starts. Just like the readinessProbe probe, it will try to connect to port 8080 of the goproxy container. If the livenessProbe probe fails, the container will be restarted.
vim tcpsocket.yaml
apiVersion: v1
Kind: Pod
metadata:
name: probe-tcp
spec:
containers:
- name: nginx
image: soscscs/myapp:v1
livenessProbe:
initialDelaySeconds: 5
timeoutSeconds: 1
tcpSocket:
port: 8080
periodSeconds: 10
failureThreshold: 2
kubectl create -f tcpsocket.yaml
kubectl exec -it probe-tcp — netstat -natp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1/nginx: master pro
kubectl get pods -w
NAME READY STATUS RESTARTS AGE
probe-tcp 1/1 Running 0 1s
probe-tcp 1/1 Running 1 25s #The first time is init(5 seconds) + period(10 seconds) * 2
probe-tcp 1/1 Running 2 45s #The second time is period(10 seconds) + period(10 seconds) and retried twice.
probe-tcp 1/1 Running 3 65s
Example 4: Readiness detection
vim readiness-httpget.yaml
apiVersion: v1
Kind: Pod
metadata:
name: readiness-httpget
namespace:default
spec:
containers:
- name: readiness-httpget-container
image: soscscs/myapp:v1
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 80
readinessProbe:
httpGet:
port: 80
path: /index1.html
initialDelaySeconds: 1
periodSeconds: 3
livenessProbe:
httpGet:
port: http
path: /index.html
initialDelaySeconds: 1
periodSeconds: 3
timeoutSeconds: 10
kubectl create -f readiness-httpget.yaml
//Readiness detection failed and cannot enter READY state
kubectl getpods
NAME READY STATUS RESTARTS AGE
readiness-httpget 0/1 Running 0 18s
kubectl exec -it readiness-httpget sh
# cd /usr/share/nginx/html/
#ls
50x.html index.html
# echo 123 > index1.html
#exit
kubectl getpods
NAME READY STATUS RESTARTS AGE
readiness-httpget 1/1 Running 0 2m31s
kubectl exec -it readiness-httpget — rm -rf /usr/share/nginx/html/index.html
kubectl get pods -w
NAME READY STATUS RESTARTS AGE
readiness-httpget 1/1 Running 0 4m10s
readiness-httpget 0/1 Running 1 4m15s
Example 5: Readiness detection 2
vim readiness-myapp.yaml
apiVersion: v1
Kind: Pod
metadata:
name: myapp1
labels:
app:myapp
spec:
containers:
- name: myapp
image: soscscs/myapp:v1
ports:
- name: http
containerPort: 80
readinessProbe:
httpGet:
port: 80
path: /index.html
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 10
---
apiVersion: v1
Kind: Pod
metadata:
name: myapp2
labels:
app:myapp
spec:
containers:
- name: myapp
image: soscscs/myapp:v1
ports:
- name: http
containerPort: 80
readinessProbe:
httpGet:
port: 80
path: /index.html
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 10
---
apiVersion: v1
Kind: Pod
metadata:
name: myapp3
labels:
app:myapp
spec:
containers:
- name: myapp
image: soscscs/myapp:v1
ports:
- name: http
containerPort: 80
readinessProbe:
httpGet:
port: 80
path: /index.html
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 10
---
apiVersion: v1
Kind: Service
metadata:
name: myapp
spec:
selector:
app:myapp
type:ClusterIP
ports:
- name: http
port: 80
targetPort: 80
kubectl create -f readiness-myapp.yaml
kubectl get pods,svc,endpoints -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/myapp1 1/1 Running 0 3m42s 10.244.2.13 node02
pod/myapp2 1/1 Running 0 3m42s 10.244.1.15 node01
pod/myapp3 1/1 Running 0 3m42s 10.244.2.14 node02
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
…
service/myapp ClusterIP 10.96.138.13
NAME ENDPOINTS AGE
…
endpoints/myapp 10.244.1.15:80,10.244.2.13:80,10.244.2.14:80 3m42s
kubectl exec -it pod/myapp1 -- rm -rf /usr/share/nginx/html/index.html
//Readiness detection fails, the Pod cannot enter the READY state, and the endpoint controller will delete the IP address of the Pod from endpoints
kubectl get pods,svc,endpoints -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/myapp1 0/1 Running 0 5m17s 10.244.2.13 node02
pod/myapp2 1/1 Running 0 5m17s 10.244.1.15 node01
pod/myapp3 1/1 Running 0 5m17s 10.244.2.14 node02
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
…
service/myapp ClusterIP 10.96.138.13
NAME ENDPOINTS AGE
…
endpoints/myapp 10.244.1.15:80,10.244.2.14:80 5m17s
//Start and exit actions
vim post.yaml
apiVersion: v1
Kind: Pod
metadata:
name: lifecycle-demo
spec:
containers:
- name: lifecycle-demo-container
image: soscscs/myapp:v1
lifecycle: #This is the key field
postStart:
exec:
command: ["/bin/sh", "-c", "echo Hello from the postStart handler >> /var/log/nginx/message"]
preStop:
exec:
command: ["/bin/sh", "-c", "echo Hello from the poststop handler >> /var/log/nginx/message"]
volumeMounts:
- name: message-log
mountPath: /var/log/nginx/
readOnly: false
initContainers:
- name: init-myservice
image: soscscs/myapp:v1
command: ["/bin/sh", "-c", "echo 'Hello initContainers' >> /var/log/nginx/message"]
volumeMounts:
- name: message-log
mountPath: /var/log/nginx/
readOnly: false
volumes:
- name: message-log
hostPath:
path: /data/volumes/nginx/log/
type: DirectoryOrCreate
kubectl create -f post.yaml
kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
lifecycle-demo 1/1 Running 0 2m8s 10.244.2.28 node02
kubectl exec -it lifecycle-demo — cat /var/log/nginx/message
Hello initContainers
Hello from the postStart handler
//View on node02 node
[root@node02 ~]# cd /data/volumes/nginx/log/
[root@node02 log]# ls
access.log error.log message
[root@node02 log]# cat message
Hello initContainers
Hello from the postStart handler
#As can be seen from the above, init Container is executed first, and then when a main container is started, Kubernetes will immediately send the postStart event.
//After deleting the pod, view it on the node02 node
kubectl delete pod lifecycle-demo
[root@node02 log]# cat message
Hello initContainers
Hello from the postStart handler
Hello from the poststop handler
#As you can see from the above, Kubernetes will send a preStop event before the container is terminated.
Extension
pod status
1. Pending: The pod has been recognized by the system, but the internal container has not yet been created. This includes the time scheduled to the node and the time to download the image, which will last for a short period of time.
2. Running: The pod has been bound to the node (scheduling is successful), and all containers in the pod have been created, at least one container is running, or the container process is starting or restarting. –It should be noted here that although the pod is already running, the internal container may not be fully available. Therefore, it is necessary to further detect the status of the container.
3. Succeeded: This status rarely appears, indicating that all containers in the pod have been successfully terminated and will no longer be pulled up.
4. Failed: All containers in the pod have been terminated, and at least one container has been terminated abnormally. (A non-zero value was returned when exiting or it was directly terminated by the system)
5. Unknown: For some reason, the status of the pod cannot be obtained. It may be due to communication problems. Generally speaking, the most common pod states are the first two states. And when running, you need to further pay attention to the status of the container.
Container life cycle
1. Waiting: a waiting state between startup and operation.
2. Running: running status.
3. Terminated: terminated state. If there are no exceptions, the container should change from the Waiting state to the Running state, and the container is available.
But if it is in the Waiting state for a long time, the container will have a field reason to indicate its state and reason. If the reason can easily indicate that the container can no longer be started, such as ContainerCannotRun, the entire service startup will return quickly. (This is a feature of failure status return, so I won’t elaborate on it)