Recently, the server disk on the project was full. In order to free up some space, the project manager deleted part of the docker overlay2 file, causing some containers to report an error when starting:
docker: Error response from daemon: open /var/lib/docker/overlay2/ffe5563b8c6a834b21dadb22106209d9fa1ab64ebe063e3ec040a05f1c: no such file or directory
The general meaning is that the file under overlay2 is gone.
Processing method:
1. First ensure that there is a backup of the previous business database 2. Shut down and delete the currently running container 3. Back up the relevant images and then clean them up. 4. Re-import the new images in order, then restart the container and import the backed-up data
The root of the problem is that docker’s Root Dir is full. The default path is /var/lib/docker
We can modify this default path when deploying docker and change it to a relatively larger path.
1. systemctl stop docker<br>Close docker
2. Modify the /etc/docker/daemon.json configuration file
vi /etc/docker/daemon.json
{
"data-root": "/data/lib/docker"
}
3. rsnyc -avz source dest
-a includes subdirectories and synchronizes meta information (such as modification time, permissions, etc.)
-v console output information
-z Compress first and then transmit
It can synchronize files between a local computer and a remote computer, or between two local directories (but does not support synchronization between two remote computers)
If it is not installed, you can use yum install rsync to install it or use cp mv and other commands to migrate.
4. systemctl enable docker<br>Set up to start docker on boot
5. Verify whether the path has changed
docker info|grep 'Root Dir'
6. Verify whether the image exists
docker images
However, migrating to a large disk treats the symptoms but not the root cause. We need to find out the root cause. Let’s go to docker’s Root Dir and enter the command.
du -h --max-depth=1|sort -n
Looking at the large file path, you can find that the two directories are easily larger. One is the container directory and the other is the overlay2 directory.
For the overlay2 directory, you can use
docker system prune -a Clean the disk, delete closed containers, useless data volumes and networks, and dangling images (ie, untagged images), The -a command cleans more thoroughly and can delete all containers that use Docker images
For the container directory, it is basically caused by the container running log. Enter the container directory and enter the command. You can see large files, which are all log files.
du -h --max-depth=1|sort -n
For log
cat /dev/null > *-json.log clears the log
At the same time, add startup restrictions to the container
docker run ...... --log-opt max-size=10m --log-opt max-file=1
Or modify the /etc/docker/daemon.json configuration file
{
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "3",
"labels": "production_status",
"env": "os,customer"
}
}
Or add restrictions in the started docker-compose.yml
logging:
options:
max-size: '12m'
max-file: '5'
driver: json-file
Several directories under /var/lib/docker and several docker IDs
1. The directory structure under /var/lib/docker is:
container image network swarm volumes overlay2 ...
You can see that some concepts involved in docker, such as containers, mirrors, networks, mounts, etc., all have corresponding directories here. This is the storage directory of docker.
2. docker storage driver:
Docker now supports many file driver types, such as AUFS and OverlayFS. The implementation of OverlayFS is similar to AUFS, but it is faster and simple to implement.
For Linux kernel 4.0 and above or Red Hat and CentOS 3.10.0-514 versions, you can use the overlay2 storage driver. Other lower versions use overlay (more advanced overlay2, better inode management and higher efficiency)
We can adjust the driver type by modifying the configuration file /etc/docker/daemon.json
{
"storage-driver": "overlay2"
}
3. container directory:
Inside the container directory is some configuration information for starting the container.
Hosts, hostname domain name and other information;
config.v2.json configuration file, the container configuration information obtained when we use docker inspect container;
*-json.log container product log file
4. image directory:
distribution directory
diffid-by-digest saves the mapping relationship of digest(layerID)->diffID v2metadata-by-diffid saves the mapping relationship of diffid -> (digest, repository) digest(layerID) is the hash ID when pulling the image. The image layer file is a compressed file in the compressed state. diffid is the image layer hash ID viewed by docker inspect. At this time, the image layer file is decompressed and in the decompressed state
Therefore, although both IDs represent the image layer hash ID, one is compressed and the other is decompressed, so the hash operations are inconsistent.
imagedb and layerdb are both metadata files Imagedb can see the metadata information of the image. Under cat, you can see dffffid written inside. repositories.json records the local image list, image name-version: imageId rootfs: diffid: sha256 records the values calculated by the sha256 algorithm after decompression of each layer. Two advantages: one is convenient for verification and the other is convenient for sharing
5. layerdb directory
sha directory, internally records ChainID
If the layer is the bottom layer and does not have any parent layer, then diffID = chainID;
Otherwise, chainID(n)=sha256sum(chainID(n-1)) diffID(n))
For example: two-layer diffid diffid2 performs shasum256 operation to obtain chainID
echo -n "sha256:5dacd731af1b0386ead06c8b1feff9f65d9e0bdfec032d2cd0bc03690698feda sha256:dd0338cdfab32cdddd6c30efe8c89d0229d9f939e2bb736fbb0a5 2f27c2b0ee9" | shasum -a 256
After finding the chainID, go into a specific directory and you will see the cache-id, which points to the specific and real overlay2 file.
mounts: Every time a container is started, there will be one more mounts file under layerdb (that is, the file echoed after starting the container)
6. Overlay2:
lower: underlying file system. For Docker, it is the read-only image layer; upper: upper file system. For Docker, it is the container layer that is readable and writable; merged: serves as a joint mount point for a unified view. For Docker, it is the file system from the user’s perspective; work: provides auxiliary functions.
As a final note, back to the original question, if you accidentally delete a file and the docker container cannot be started, you can rename the /var/lib/docker directory on the disk, then restart the docker service and let docker pull the image again, but If the previous file system is not mapped, files will be lost, so it is recommended that the stored content must be directory mapped.
Thanks for reading, welcome to follow