How to implement large-scale image management and distribution based on Harbor and Registry

Follow the “Wonderful World of Linux” on the public account

Set it as a “star” and let you play Linux every day!

0d9b5cab3f81c8e812ec3823ca1a7801.jpeg

1. Challenges brought by Harbor’s cross-region

If you simply store image data, Registry will be a good choice as an image warehouse. Registry not only supports multiple storage backends, but can also configure HTTPS certificates and access credentials. It is worth noting that Harbor also uses Registry to store mirror data.

If the team needs functions such as role management, storage control, and docking LDAP authentication, Harbor can be used. Only a 4C8GB machine with external high-availability PGSQL and object storage is enough to support dozens of Kubernetes clusters and hundreds of VM nodes, and open up the CI and CD image delivery processes.

By stacking memory, CPU and other resources, a single instance of Harbor is enough to support hundreds of clusters and thousands of nodes. Another important optimization point is that when the backend uses object storage such as Obs, uploading and downloading mirror layer data will directly request the object storage bucket without causing excessive traffic burden on Harbor. Benefiting from object storage’s global acceleration, upstream and downstream speeds can easily reach 200+ Mbps or even 1Gbps of bandwidth.

But that’s the limit. Harbor is only suitable for a single zone and cannot meet the needs of image management in cross-zone scenarios. As shown in the figure below, when services need to be deployed in multiple regions, and the regions are interconnected through the public network, once the regional network is controlled by the country where it is located and becomes unstable, Harbor will be helpless in this scenario.

68ac736065fd03b6fd4d7b56459f4a73.png

At the same time, the inefficient Harbor task queue makes the multi-Harbor synchronization solution lose its effectiveness. After one area is pushed, the other area needs to wait for more than ten hours before the synchronization is completed and the application is allowed to be updated. This is intolerable, refer to: Some problems with harbor-[1].

In a multi-region scenario, a single instance of Harbor cannot support the mirroring traffic requirements of the entire region. On the one hand, the cross-region network is not stable enough and traffic costs are high. On the other hand, it is also about scalability. The IT infrastructure does not support the continuous addition of regions, and it is unreliable for all loads to pull the same Harbor.

2. Why not use distribution solutions such as Dragonfly

2.1 Introduction to Dragonfly

Dragonfly is an image distribution tool recommended by many people, but I have not found any cases that fit the scenario.

The document structure of Dragonfly is not very clear. As I understand it, Dragonfly can be divided into two parts:

  • Dfdaemon

Dfdaemon is similar to the Mirror function of the official Docker Registry and is used to proxy mirror layer traffic. This part can be used alone and does not depend on other components.

  • distribution network

The distribution network is the core function of Dragonfly. The components of the V1 version are Supernode, and the components of the V2 version are Scheduler, Manager, etc. Similar to how we use P2P tools to download large files, Dragonfly has built an internal network specifically for mirror acceleration. The download client of this distribution network is Dfclient, also called Dfdaemon.

2.2 The principle of using Mirror to accelerate mirroring

Because Dragonfly V1 and most articles on the Internet speed up mirror downloads by configuring Mirror. Before talking about the Dragonfly problem, let’s first talk about the principle of Docker Mirror acceleration.

By consulting the source code, you can get the following snippet:

func (s *DefaultService) lookupV2Endpoints(hostname string) (endpoints []APIEndpoint, err error) {
 tlsConfig := tlsconfig.ServerDefault()
 if hostname == DefaultNamespace || hostname == IndexHostname {
  for _, mirror := range s.config.Mirrors {
   endpoints = append(endpoints, APIEndpoint{
    URL: mirrorURL,
    Version: APIVersion2,
    Mirror: true,
    TrimHostname: true,
    TLSConfig: mirrorTLSConfig,
   })
  }
  endpoints = append(endpoints, APIEndpoint{
   URL: DefaultV2Registry,
   Version: APIVersion2,
   Official: true,
   TrimHostname: true,
   TLSConfig: tlsConfig,
  })
 }

Address: https://github.com/docker/docker-ce/blob/8bb27fc680463da975f386e3a325fe4d52b05f8e/components/engine/registry/service_v2.go

The DefaultNamespace = "docker.io" and IndexHostname = "index.docker.io" here means that Mirror actually only takes effect on the images officially provided by Docker.

This feature is also clearly stated in Docker’s official documentation: It's currently not possible to mirror another private registry. Only the central Hub can be mirrored., reference: https://docs. docker.com/registry/recipes/mirror/ .

In https://github.com/distribution/distribution/issues/1483, the developer provided additional explanations on this feature. The reason why private warehouses cannot use Mirror is: Private warehouse sources will cause local namespace conflicts and mirror management. Therefore, it is limited to a single official docker.io warehouse image and cannot accelerate private warehouses.

2.3 Configure the proxy to use Dragonfly V2 to accelerate private images

  • Acceleration Dilemma

Since Mirror can only accelerate Docker official images, this is inconsistent with the scenario of private image warehouses. What is confusing is why so many articles on the Internet are accelerated by Mirror, and they can also be used with Harbor private warehouses.

If you only accelerate the Docker official image layer data, the acceleration effect will be greatly reduced. Due to the phased construction, the business’s runtime uses the alpine base image on docker.io, which is usually only a few to dozens of MB. Business-related dependency packages, binaries, images, JS, CSS, Jar, and WASM packages account for a large proportion of the entire image data.

  • The correct way to accelerate is to configure the proxy

The official documentation of Dragonfly V2 provides acceleration solutions for different runtimes. However, for private warehouses, it is not through configuring Mirror, but configuring Docker with HTTP_PROXY and HTTPS_PROXY.

Unless HTTPS_PROXY has been configured when installing Containerd, I don’t think anyone is willing to take the risk and modify it directly online. What’s more, the Live Restore feature does not necessarily have to be turned on, and this feature also requires the Containerd version.

3. In fact, private mirroring can also be accelerated through Mirror

As mentioned earlier, private mirroring cannot be accelerated by configuring Mirror. But after thinking about it, I still found a feasible solution. Although it has not been actually adopted, it is still written down for everyone’s reference.

3.1 Idea

Whether it is docker.io or the private image warehouse built by Harbor, they all follow the same set of interface specifications. This gives us the possibility to switch the traffic of the private image warehouse to the docker.io domain name by modifying the DNS record.

3.2 Specific steps

Assume that the intranet mirror warehouse is named private.chenshaowen.com and the IP address is 1.2.3.4.

The first step is to add DNS records to point docker.io to 1.2.3.4 on the intranet where the business is located.

The second step is to switch the image service pointing to docker.io.

The complete image name is private.chenshaowen.com/project/biz:v1. In fact, if the equivalent domain name [2] is added, we can also use private-peer-a.chenshaowen.com/project/biz:v1, private-peer-b.chenshaowen.com/project/biz:v1 pull. The domain name in the image format only indicates the service. The domain name of the service can be changed at will. It only requires re-authentication and authentication, and will not affect the push and pull of the image.

Since private.chenshaowen.com and docker.io both point to 1.2.3.4, we can use docker.io/project/biz:v1 Replace private.chenshaowen.com/project/biz:v1.

2918d658fb544eabaead89acad6e5a93.png

As shown in the figure above, when deploying business applications, although the docker.io image is used, the image data is requested from the private warehouse. But this has no acceleration effect, so you need to use services such as Dragonfly for accelerated distribution to work.

a4ea41be7883ba3b30ca11b8b847a9ee.png

3.3 Insecure Registry issues

Since there is no HTTPS certificate for docker.io, we cannot simply modify the DNS record of docker.io to achieve seamless switching on the intranet. Two ideas are provided here, which will not be elaborated here:

  • Add docker.io to the Insecure Registry list

  • Add the root certificate to the host, and then use the root certificate to self-sign the HTTPS certificate for the docker.io domain name.

3.4 Should you hijack docker.io to speed up image pulling

It depends on whether the conditions for implementation are met.

If you have strong control over the infrastructure, including root certificates and container configurations, you can still consider it. The reason is that the image service is also a basic service. Hijacking docker.io can also enhance the level of management and control of the infrastructure and help provide value-added services around the image.

Otherwise, the gain outweighs the gain by destroying the understanding of docker.io among development and operation and maintenance personnel for the sake of image acceleration.

This also leads to the following exploration of other solutions.

4. Use Registry for image distribution

4.1 Unknown functions of Registry

Ordinary programmers may only know that Registry can be used as a mirror warehouse. In fact, Registry has three uses:

  • Mirror warehouse

  • Mirror accelerates mirroring on docker.io

  • Proxy forwards pull mirror requests, similar to Mirror, but used in completely different ways.

Here is a brief introduction to the Proxy function:

The construction method is the same as Mirror, refer to: How to build a private mirror warehouse mirror[3]

But when using it, it is not configured as "registry-mirrors": ["http://registry_ip:5000"], but as "insecure-registries": ["http:// registry_ip:5000"].

At this time, we can pull the docker.io/shaowenchen/docker-robotframework:latest image through docker pull registry_ip:5000/shaowenchen/docker-robotframework:latest.

After my testing, here are a few key points:

  • If the image on docker.io is updated, you will also get the latest image when pulling the Registry

  • Failure to connect to docker.io does not affect the Registry’s ability to provide cached images.

  • Support pulling private images on docker.io

  • Support pulling registry_ip:5000 authentication and authentication

  • The image will be cached locally, and repeated pulling will speed up the process.

4.2 Mirror acceleration architecture

Business services are distributed in many regions such as China, Singapore, Japan, and India, but R&D is in China. Our image management plane is on domestic Harbor, and we mainly distribute images to other regions.

18ef807f0205a5646fea161d2de04319.png

As shown above, the core component consists of two parts:

  1. Harbor, deployed domestically. Developers use the CICD platform to push images to Harbor for unified management

  2. Registry needs to be deployed in the area where each business service is located. When Kubernetes runs the first Pod copy, the Registry will pull the domestic Harbor image through the public network and cache it to the Registry in the current region. When running the second copy, the image will be pulled directly from the Registry through the regional VPC intranet to achieve acceleration.

4.3 About DNS configuration

It would be best if you could directly modify the DNS records of each zone. Otherwise, you need to add /etc/hosts records.

  • In the R&D environment, the image warehouse points to Harbor

  • In the deployment environment, the image warehouse points to the Registry

4.4 About certificates

Since the Registry cannot proxy authentication, there will be two sets of credential systems.

  • In the R&D environment, Harbor credentials are used, which can be customized according to roles.

  • In the deployment environment, use Registry credentials, pull globally, and cannot push.

4.5 Quickly add Registry Proxy

Here are some simple configurations for everyone to test.

  • Generate credentials

mkdir auth
docker run\
  --entrypoint htpasswd \
  httpd:2 -Bbn global-read xxxxxx > auth/htpasswd

  • Create configuration file

Local disk storage is used as an example here, but the backend using object storage can provide faster network access, especially in cross-region scenarios.

vim config.yml
version: 0.1
log:
  fields:
    service: registry
storage:
  cache:
    blobdescriptor: memory
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
proxy:
  remoteurl: https://private.chenshaowen.com
  username: [username]
  password: [password]

  • Run agent

mkdir /proxy
docker run -d -p 8001:5000 --restart=always --name proxy \
             --add-host=private.chenshaowen.com:1.2.3.4 \
             -v `pwd`/auth:/auth \
             -e "REGISTRY_AUTH=htpasswd" \
             -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
             -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
             -v `pwd`/config.yml:/etc/docker/registry/config.yml \
             -v /proxy:/var/lib/registry \
             registry:2

5. Summary

This article mainly considers how to manage and distribute images for multi-region services in weak interconnection environments.

The multi-Harbor synchronization solution was rejected, and the Dragonfly distribution solution was rejected. Finally, the image distribution problem was solved with the help of the unknown Registry proxy mode and DNS resolution.

If distribution is not an issue, then a single Harbor management plane will suffice.

Although the proxy implemented by Registry meets the current functions, it cannot forward authentication and does not support multi-backend multi-point transmission, so there is still a lot of room for optimization.

Another benefit is that it can be expanded horizontally and can quickly provide mirroring services in new areas to support rapid business development.

Reference materials

[1]

Some problems with harbor: https://www.chenshaowen.com/blog/an-enterprise-class-registry-of-harbor.html#6-Some problems with harbor-

[2]

Equivalent domain name: https://www.chenshaowen.com/blog/peer-domain.html

[3]

How to build a private mirror warehouse mirror: https://www.chenshaowen.com/blog/how-to-run-a-private-registry-mirror.html

This article is reprinted from: “Chen Shaowen”, original text: https://url.hi-linux.com/cwCRX, the copyright belongs to the original author. Welcome to submit articles. Submission email: [email protected].

fa14765f51f0ba1316a3a2a083d77f09.gif

Recently, we established a Technical Exchange WeChat Group. At present, many great masters in the industry have joined the group. Interested students can join and exchange technology with us. In the “Wonderful World of Linux”, directly reply to “Add the Group” on the public account Invite you to join the group.

b6b8f8d63cbba1b0f98b680dec973c7c.png

You may also like

Click on the image below to read

cfbfea007f5e56a1592ed5f809ab2108.png

How to elegantly connect the private image warehouse Harbor to Containerd management

f53387f1b725b631586ee3c4fae68446.png
Click on the picture above to receive takeaway red envelopes for free every day from “Meituan | Ele.me”

59503354468a6b021376febfafdf522f.png

For more interesting Internet news, follow the “Wonderful Internet” video account to learn all about it!