Scene
When the pod is in the crash state, the container keeps restarting. At this time, using kubelet logs may never capture the logs.
Solution:
kubectl previous Parameter function: If true, print the logs for the previous instance of the container in a pod if it exists.
- Single container pod:
kubectl logs pod-name --previous
- Multi-container pod:
kubectl logs pod-name --previous -c container-name
for example:
NAME READY STATUS RESTARTS AGE nginx-7d8b49557c-c2lx9 2/2 Running 5 kubectl logs nginx-7d8b49557c-c2lx9 --previous Error: xxxxxxxxxxx
Principle
Kubelet will keep the first few failed containers of the pod, which is a prerequisite for viewing.
Core Principles
The principle of kubelet implementing previous: store the pod’s log in /var/log/pods/podname, and it is a link file, which is linked to the log file of the docker container. At the same time, kubelet will also retain the previous container. , and there is a link file linked to the log file of a crashed container on the pod. Use previous to view this file.
Practice
For example, to view a pod:
ubuntu@~$ kubelet get pod NAME READY STATUS RESTARTS AGE busybox 1/1 Running 2394 99d nginx-deployment-6wlhd 1/1 Running 0 79d redis 1/1 Running 0 49d
Go to the node where the pod is located to view the two log files placed by kubelet:
ls /var/log/pods/default_busybox_f72ab71a-5b3b-4ecf-940d-28a5c3b30683/busybox 2393.log 2394.log
The meaning of the numbers: 2393 proves to be the log after the 2393rd restart, and 2394 means the log after the 2394th restart.
In fact, these two log files are link files, pointing to the docker log file:
/busybox#stat 2393.log File: 2393.log -> /data/kubernetes/docker/containers/68a5b32c9fdb1ad011b32e6252f9cdb759f69d7850e6b7b8591cb4c2bf00bcca/68a5b32c9fdb1ad011b32e6252f9cdb759f6 9d7850e6b7b8591cb4c2bf00bcca-json.log Size: 173 Blocks: 8 IO Block: 4096 symbolic link Device: fc02h/64514d Inode: 529958 Links: 1 Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2023-01-31 13:32:03.751514283 + 0800 Modify: 2023-01-31 13:32:03.039526838 + 0800 Change: 2023-01-31 13:32:03.039526838 + 0800 Birth: - /busybox#stat 2394.log File: 2394.log -> /data/kubernetes/docker/containers/2ed9ebf0585215602874b076783e12191dbb010116038b8eb4646273ebfe195c/2ed9ebf0585215602874b076783e12191dbb010116 038b8eb4646273ebfe195c-json.log Size: 173 Blocks: 8 IO Block: 4096 symbolic link Device: fc02h/64514d Inode: 529955 Links: 1 Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2023-01-31 14:32:03.991106950 + 0800 Modify: 2023-01-31 14:32:03.183119308 + 0800 Change: 2023-01-31 14:32:03.183119308 + 0800 Birth: -
I saw log files pointing to these two containers respectively. One is the container running in the current pod, and the other is the container the pod ran last time. It has now exited.
docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2ed9ebf05852 ff4a8eb070e1 "sleep 3600" 24 minutes ago Up 24 minutes k8s_busybox_busybox_default_f72ab71a-5b3b-4ecf-940d-28a5c3b30683_2394 68a5b32c9fdb ff4a8eb070e1 "sleep 3600" About an hour ago Exited (0) 24 minutes ago k8s_busybox_busybox_default_f72ab71a-5b3b-4ecf-940d-28a5c3b30683_2393
When using logs, the file of the current container is read. When using –previous, the log file of the last exited container is read, because kubelet retains the last exited container for the pod.
We manually edit the contents of these two files to see if kubelet reads these two files.
/busybox# cat 2393.log
{"log":"last crash logs\
","stream":"stderr","time":"2022-11-05T08:11:27.31523845Z"}
/busybox# cat 2394.log
{"log":"now pod log\
","stream":"stderr","time":"2022-11-05T08:11:27.31523845Z"}
ubuntu@10-234-32-51:~$ k logs busybox --previous
last crash logs
ubuntu@10-234-32-51:~$ k logs busybox
now pod log
Since it is a link file, it may actually be read from other places, or directly read from the container directory. Since we change the link file, the log file in the container directory will also be changed. We directly create two files. Do verification:
ubuntu@10-234-32-51:~$ k get pod
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 2395 99d
nginx-deployment-6wlhd 1/1 Running 0 79d
redis 1/1 Running 0 49d
/busybox#ls
2394.log 2395.log
/busybox# rm 2394.log 2395.log
We delete it and create it ourselves. This time it is a regular file, not a link file:
/busybox#ls
2394.log 2395.log
/busybox#stat 2394.log
File: 2394.log
Size: 100 Blocks: 8 IO Block: 4096 regular file
Device: fc02h/64514d Inode: 529965 Links: 1
Access: (0640/-rw-r-----) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2023-01-31 15:42:11.307170422 + 0800
Modify: 2023-01-31 15:42:07.711225229 + 0800
Change: 2023-01-31 15:42:07.711225229 + 0800
Birth: -
/busybox#stat 2395.log
File: 2395.log
Size: 86 Blocks: 8 IO Block: 4096 regular file
Device: fc02h/64514d Inode: 529967 Links: 1
Access: (0640/-rw-r-----) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2023-01-31 15:41:17.539989934 + 0800
Modify: 2023-01-31 15:41:14.348038586 + 0800
Change: 2023-01-31 15:41:14.352038525 + 0800
Birth: -
/busybox# cat 2394.log
{"log":"previous logs create by myself\
","stream":"stderr","time":"2022-11-05T08:11:27.31523845Z"}
/busybox# cat 2395.log
{"log":"create by myself\
","stream":"stderr","time":"2022-11-05T08:11:27.31523845Z"}
ubuntu@10-234-32-51:~$ k logs busybox
create by myself
ubuntu@10-234-32-51:~$ k logs busybox --previous
previous logs create by myself
Draw conclusions
The kubelet reads the log files under /var/log/pods/, and the log files under –previous are also read under /var/log/pods/ log file, and there is a link file specifically to point to the log file of the previous exit container, so as to obtain the log before the container crashes.
The knowledge points of the article match the official knowledge archives, and you can further learn relevant knowledge. Cloud native entry-level skill treeContainer orchestration (production environment k8s)kubelet, kubectl, kubeadm three-piece set 16742 people Currently studying the system