Exploring the black technology of Docker containerization technology “storage drive Overlay2”

The public account pays attention to “Wonderful Linux World”

Set it as a “star” and take you to play with Linux every day!

In this section, I will introduce you to Overlay2, the storage driver currently recommended by Docker. Before starting, you can execute the following command to view the storage driver that Docker is using:

(MoeLove) ? ~ docker info --format '{<!-- -->{.Driver}}'
overlay2

If the result you see is also overlay2, it means that your Docker is already using the overlay2 storage driver. I use btrfs on my personal workstation because btrfs has been the default file system for Fedora since Fedora 33. But the server is all overlay2.

You may also see other different results, which can be specified by the --storage-driver parameter when starting the docker daemon, or in /etc/docker/daemon.json file is configured through the storage-driver field.

Currently with the latest version of Docker, you have the following storage drivers to choose from:

overlay2
fuse-overlayfs
btrfs
zfs
aufs
overlay
devicemapper
vfs

But they all have different requirements for things like the filesystem you use, and they’re implemented in different ways. I’ll take the key overlay2 storage driver in this section as an example. It requires you to use a kernel version above Linux 4.x, or a kernel above 3.10.0-514 for RHEL/CentOS ( There were some compatibility issues in the old version, which I mentioned in a previous article).

At the same time, it supports you to use the ext4 file system, or the xfs file system with ftype=1 added. Information related to the file system can be obtained through docker info.

# omit part of the output
(MoeLove) ? ~ docker info
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true

The role of the storage driver

Although we have talked about how to set and check the storage driver currently in use, we have not yet introduced why the storage driver must be used and its function.

Do you still remember how Docker stores image-related content that I introduced to you in my previous article “Thousand-Character Long Article: Thoroughly Understanding Container Image Construction”. If you forget it, you can go back and review it.

Docker stores container images in layers, and each layer contains a Dockerfile instruction. The storage method of these layers on the disk, and how to organize these layers and provide a writable layer when starting the container are the main functions of the storage driver.

In addition, it should be noted that different storage drivers have different implementations and different performances, and the use of different storage drivers will also result in different disk space occupation.

At the same time: due to their different implementations, when you modify the storage driver, you may not be able to see the original image, container, etc. This is normal, don’t worry, you can see it after switching back to the original driver.

OverlayFS

After understanding the previous background knowledge, you also saw that there are two overlay and overlay2 in the available storage drivers I just listed. In fact, overlay2 It is an upgraded version of overlay, and both storage drivers use OverlayFS.

The overlay driver first entered Docker in August 2014, while overlay2 was merged in June 2016 and first appeared in Docker 1.12. It was created to solve the problem of overlay storage drivers that could cause inode exhaustion.

After briefly introducing overlay and overlay2 , we will focus on OverlayFS .

Let’s start a container and use it as an entry point to understand OverlayFS. Note: the following content uses the Linux 5.4 kernel and Docker 20.10.21, and the results may be slightly different in different environments.

# Check if there are no running containers and overlay mounts
(MoeLove) ? ~ mount |grep overlay
(MoeLove) ? ~ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

# start a container
(MoeLove) ? ~ docker run --rm -d alpine sleep 99999
caa9517ce0d799602735a30aaaaf123c07e07ff6e44c5a4b07e776af85780abe
(MoeLove) ? ~ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
caa9517ce0d7 alpine "sleep 99999" 23 seconds ago Up 22 seconds hopeful_dubinsky

# Check overlay mounts
(MoeLove) ? ~ mount |grep overlay
overlay on /var/lib/docker/overlay2/f4356a8f14342008fc298bf3d313b863d10f30ef447a3b2f51ea9ece0dec09db/merged type overlay (rw,relatime,seclabel,lowerdir=/var/lib/docker/overlay2/l/5OO3RLR XHJPEH3IFEXNCTO4PY5:/var/lib/docker/overlay2/l/UVA7IR67ZZTN2BNTKCZ7T6HUWU ,upperdir=/var/lib/docker/overlay2/f4356a8f14342008fc298bf3d313b863d10f30ef447a3b2f51ea9ece0dec09db/diff,workdir=/var/lib/docker/overlay2/f4356a8f14342008fc298bf3 d313b863d10f30ef447a3b2f51ea9ece0dec09db/work)

It can be seen that after the container is started, an OverlayFS (overlay) mount is added to the system. Pay attention to a few of them:

The mount point is: /var/lib/docker/overlay2/f4356a8f14342008fc298bf3d313b863d10f30ef447a3b2f51ea9ece0dec09db/merged

(MoeLove) ? ~ sudo ls /var/lib/docker/overlay2/f4356a8f14342008fc298bf3d313b863d10f30ef447a3b2f51ea9ece0dec09db/merged
bin dev etc home lib media mnt opt proc root run sbin srv sys tmp usr var

The content, which looks familiar, is the content in the root directory of the container we started. In order to verify this statement, I wrote a new file in the container:

(MoeLove) ? ~ docker exec -it $(docker ps -ql) sh
/ # echo 'Hello Docker' > moelove-info

Look again at what’s in this mount point:

(MoeLove) ? ~ sudo ls /var/lib/docker/overlay2/22be5e4dc4541a60aa4f6de628c3938e7fdc9c4b117277274cd911c46166986b/merged
bin dev moelove-info etc home lib media mnt opt proc root run sbin srv sys tmp usr var
(MoeLove) ? ~ sudo cat /var/lib/docker/overlay2/f4356a8f14342008fc298bf3d313b863d10f30ef447a3b2f51ea9ece0dec09db/merged/moelove-info
Hello Docker

You can see that the content you just wrote is already in the directory of this mount point.

lowerdir: This is our mount directory.

This lowerdir contains two directories, which use the kernel’s support for the OverlayFS multi layer feature. Let’s check the contents separately:

(MoeLove) ? ~ sudo ls -a /var/lib/docker/overlay2/l/5OO3RLRXHJPEH3IFEXNCTO4PY5
. .. dev .dockerenv etc
(MoeLove) ? ~ sudo ls -a /var/lib/docker/overlay2/l/UVA7IR67ZZTN2BNTKCZ7T6HUWU
. .. bin dev etc home lib media mnt opt proc root run sbin srv sys tmp usr var

Do these two directories look familiar?

Yep, that’s most of what’s in the root directory of the container we started. Why do you say most of the content? When we look at the content, you will also find that their content is not complete. For example, the moelove-info file we just wrote, or when we look at the files in the etc directory, you will also find that it is just a regular system /etc directory.

(MoeLove) ? ~ sudo ls /var/lib/docker/overlay2/l/5OO3RLRXHJPEH3IFEXNCTO4PY5/etc
hostname hosts mtab resolv.conf
(MoeLove) ? ~ sudo ls /var/lib/docker/overlay2/l/UVA7IR67ZZTN2BNTKCZ7T6HUWU/etc
alpine-release fstab init.d modprobe.d mtab passwd protocols shells udhcpd.conf
apk group inittab modules network periodic securetty ssl
conf.d hostname issue modules-load.d opt profile services sysctl.conf
crontabs hosts logrotate.d motd os-release profile.d shadow sysctl.d

upperdir is another important directory, let’s take a look at its contents
```
(MoeLove) ? ~ sudo ls -a /var/lib/docker/overlay2/f4356a8f14342008fc298bf3d313b863d10f30ef447a3b2f51ea9ece0dec09db/diff
. .. moelove-info root
```
We find that this directory contains the moelove-info file we just created. At the same time, it also contains a root directory, which is the home directory of the root user we use by default.

If you go to check the content, you will also find the history of the commands we executed just now.

workdir This directory and upperdir are in the same parent directory, check its content and find that there is only one work directory

(MoeLove) ? ~ sudo ls -a /var/lib/docker/overlay2/f4356a8f14342008fc298bf3d313b863d10f30ef447a3b2f51ea9ece0dec09db/work
. .. work

After reading the above introduction, you must have discovered some of the connections between them. Before that, we are looking at an additional directory, which is upperdir and workdir and hanging Load point common parent directory:

(MoeLove) ? ~ sudo ls /var/lib/docker/overlay2/f4356a8f14342008fc298bf3d313b863d10f30ef447a3b2f51ea9ece0dec09db
diff link lower merged work

You will find that the contents of this directory are more intuitive. We have just seen the contents of the diff, merged and work directories, now look at the lower content bar:

(MoeLove) ? ~ sudo cat /var/lib/docker/overlay2/f4356a8f14342008fc298bf3d313b863d10f30ef447a3b2f51ea9ece0dec09db/lower
l/5OO3RLRXHJPEH3IFEXNCTO4PY5:l/UVA7IR67ZZTN2BNTKCZ7T6HUWU

We found that the contents of the lower file are two lowerdir directory names separated by :.

So far, we can draw the following conclusions:

lower is the base layer and can contain multiple lowerdir;
diff is a writable layer, that is, the upperdir when mounted, and the files changed in the container are stored in this layer;
merged is the final merged result, that is, the result presented to us by the container;

Overlay2

After the previous introduction to OverlayFS mounted after Docker starts the container, the workflow of Overlay2 must be clear to you.

Use each layer of the image as the lower base layer, and add the writable layer diff at the same time. Through the working mechanism of OverlayFS, finally use merged as the container The file directory of the file is displayed to the user.

You may have doubts, if it is just such a simple organization, will there be any restrictions? The answer is yes, of course there are limitations, we can see it through the code of Overlay2

// daemon/graphdriver/overlay2/overlay.go#L442
func (d *Driver) getLower(parent string) (string, error) {
// omit some content
 if len(lowers) > maxDepth {
  return "", errors. New("max depth exceeded")
 }
}

You can see that it has a hard-coded limit on the depth of lower, and the current hard-coded limit is 128. If you encounter this error during use, it means that you have exceeded the maximum depth limit, and you need to find some way to reduce the level.

Summary

In this section, I introduce the content related to OverlayFS and Overlay2 storage drivers. Introduce the workflow of overlay2 by actually starting the relevant directories generated by the container. It must be easier to understand in this way.

This article is reproduced from: “MoeLove”, the original text: https://url.hi-linux.com/oKqJu, the copyright belongs to the original author. Contributions are welcome, submission email: [email protected].

Recently, we established a WeChat group for technology exchange. At present, many great masters in the industry have joined the group. Interested students can join us to exchange technology with us, and reply directly to the “Join Group” in the “Wonderful Linux World” official account invites you to join the group.

You may also like

Click on the image below to read

Next-generation data plane Cilium: supporting 100Gbit/s Kubernetes cluster

Click on the picture above, “Meituan | Are you hungry” takeaway red envelopes for free every day

For more interesting Internet news, follow the “Wonderful Internet” video account to learn all about it!