Kubernetes cluster troubleshooting-use crictl to debug Kubernetes nodes

crictl is a CRI-compliant container runtime command line interface. You can use it to inspect and debug container runtimes and applications on Kubernetes nodes. crictl and its source code are in the cri-tools repository.

1. Get ready

crictl requires a Linux operating system with the CRI runtime.

2. Install crictl

You can download a compressed crictl archive for several different architectures from the cri-tools releases page. Download the version that corresponds to your kubernetes version. Extract it and move it to a location on your system path, such as /usr/local/bin/.

3. General usage

The crictl command has several subcommands and runtime parameters. For more information, use crictl help or crictl help for help.

You can set the endpoint for crictl using one of the following methods:

  • Set parameters –runtime-endpoint and –image-endpoint.
  • Set the environment variables CONTAINER_RUNTIME_ENDPOINT and IMAGE_SERVICE_ENDPOINT.
  • Set the endpoint in the configuration file –config=/etc/crictl.yaml. To set a different file, you can use the –config=PATH_TO_FILE flag when running crictl.

Description:

If you don’t set an endpoint, crictl will try to connect to a list of known endpoints, which may affect performance.

You can also specify a timeout value when connecting to the server and enabling or disabling debugging by specifying a timeout or debug value in the configuration file, or by using the –timeout and –debug command line parameters.

To view or edit the current configuration, view or edit the contents of /etc/crictl.yaml. For example, a configuration using the containerd container runtime would look like this:

runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10
debug: true

To learn more about crictl, see the crictl documentation.

4. crictl command example

Warning:

If you use crictl to create pod sandboxes or containers on a running Kubernetes cluster, the kubelet will eventually delete them. crictl is not a general workflow tool, but a useful tool for debugging.

1. Print Pod list

Print a list of all Pods:

crictl pods

The output is similar to:

POD ID CREATED STATE NAME NAMESPACE ATTEMPT
926f1b5a1d33a About a minute ago Ready sh-84d7dcf559-4r2gq default 0
4dccb216c4adb About a minute ago Ready nginx-65899c769f-wv2gp default 0
a86316e96fa89 17 hours ago Ready kube-proxy-gblk4 kube-system 0
919630b8f81f1 17 hours ago Ready nvidia-device-plugin-zgbbv kube-system 0

Print a list of Pods by name:

crictl pods --name nginx-65899c769f-wv2gp

The output is similar to this:

POD ID CREATED STATE NAME NAMESPACE ATTEMPT
4dccb216c4adb 2 minutes ago Ready nginx-65899c769f-wv2gp default 0

Print the Pod list based on labels:

crictl pods --label run=nginx

The output is similar to this:

POD ID CREATED STATE NAME NAMESPACE ATTEMPT
4dccb216c4adb 2 minutes ago Ready nginx-65899c769f-wv2gp default 0

2. Print the image list

Print a list of all images:

crictl images

The output is similar to this:

IMAGE TAG IMAGE ID SIZE
busybox latest 8c811b4aec35f 1.15MB
k8s-gcrio.azureedge.net/hyperkube-amd64 v1.10.3 e179bbfe5d238 665MB
k8s-gcrio.azureedge.net/pause-amd64 3.1 da86e6ba6ca19 742kB
nginx latest cd5239a0906a6 109MB

Print the image list according to the warehouse:

crictl images nginx

The output is similar to this:

IMAGE TAG IMAGE ID SIZE
nginx latest cd5239a0906a6 109MB

Print only the image ID:

crictl images -q

The output is similar to this:

sha256:8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a
sha256:e179bbfe5d238de6069f3b03fccbecc3fb4f2019af741bfff1233c4d7b2970c5
sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e
sha256:cd5239a0906a6ccf0562354852fae04bc5b52d72a2aff9a871ddb6bd57553569

3. Print container list

Print a list of all containers:

crictl ps -a

The output is similar to this:

CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT
1f73f2d81bf98 busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47 7 minutes ago Running sh 1
9c5951df22c78 busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47 8 minutes ago Exited sh 0
87d3992f84f74 nginx@sha256:d0a8828cccb73397acb0073bf34f4d7d8aa315263f1e7806bf8c55d8ac139d5f 8 minutes ago Running nginx 0
Running kube-proxy 18 hours ago

Print a list of running containers:

crictl ps

The output is similar to this:

CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT
1f73f2d81bf98 busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47 6 minutes ago Running sh 1
87d3992f84f74 nginx@sha256:d0a8828ccb73397acb0073bf34f4d7d8aa315263f1e7806bf8c55d8ac139d5f 7 minutes ago Running nginx 0
Running kube-proxy 0

4. Execute commands on the running container

crictl exec -i -t 1f73f2d81bf98 ls

The output is similar to this:

bin dev etc home proc root sys tmp usr var

5. Obtain container logs

Get all logs of the container:

crictl logs 87d3992f84f74

The output is similar to this:

10.240.0.96 - - [06/Jun/2018:02:45:49 + 0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" " -"
10.240.0.96 - - [06/Jun/2018:02:45:50 + 0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" "-"
10.240.0.96 - - [06/Jun/2018:02:45:51 + 0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" "-"

Get the most recent N lines of logs:

crictl logs --tail=1 87d3992f84f74

The output is similar to this:

10.240.0.96 - - [06/Jun/2018:02:45:51 + 0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" " -"

6. Run Pod Sandbox

Running Pod Sandbox with crictl is very helpful for container runtime troubleshooting. In a running Kubernetes cluster, the sandbox is randomly stopped and deleted by the kubelet.

  • Write the following JSON file:
{ "metadata": { "name": "nginx-sandbox", "namespace": "default", "attempt": 1, "uid" : "hdishd83djaidwnduwk28bcsb" }, "log_directory": "/tmp", "linux": { } }
  • Use the crictl runp command to apply the JSON file and run the sandbox.
crictl runp pod-config.json
  • Returned the ID of the sandbox.

7. Create a container

Creating a container with crictl is very helpful for troubleshooting the container runtime. In a running Kubernetes cluster, the sandbox is randomly stopped and deleted by the kubelet.

  • Pull the busybox image
crictl pull busybox
Image is up to date for busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47
  • Create pod and container configurations:

Pod configuration:

{ "metadata": { "name": "busybox-sandbox", "namespace": "default", "attempt": 1, "uid" : "aewi4aeThua7ooShohbo1phoj" }, "log_directory": "/tmp", "linux": { } }

Container configuration:

{ "metadata": { "name": "busybox" }, "image":{ "image": "busybox" }, "command": [ "top" ], "log_path":"busybox.log", "linux": { } }
  • Create a container, passing the ID of the previously created Pod, the container configuration file, and the Pod configuration file. Returns the ID of the container.
crictl create f84dd361f8dc51518ed291fbadd6db537b0496536c1d2d6c05ff943ce8c9a54f container-config.json pod-config.json
  • Query all containers and confirm that newly created containers have a status of Created.
crictl ps -a
  • The output is similar to this:
CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT 3e025dd50a72d busybox 32 seconds ago Created busybox 0

8. Start the container

To start a container, pass the container ID to crictl start:

crictl start 3e025dd50a72d956c4f14881fbb5b1080c9275674e95fb67f965f6478a957d60

The output is similar to this:

3e025dd50a72d956c4f14881fbb5b1080c9275674e95fb67f965f6478a957d60

Confirm that the container’s status is Running.

crictl ps

The output is similar to this:

CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT
3e025dd50a72d busybox About a minute ago Running busybox 0

The knowledge points of the article match the official knowledge files, and you can further learn relevant knowledge. Cloud native entry-level skills treeHomepageOverview 16,761 people are learning the system