Copa: No need to rebuild the image, one-click hot repair of container security vulnerabilities

Follow the “Wonderful World of Linux” on the public account

Set it as a “star” and let you play Linux every day!

a711c6792757dbe094ed8aae04faf3b7.png

copa is a CLI tool written in Go and based on buildkit that can directly patch container images based on vulnerability scan results from popular tools like Trivy.

Why?

We need to be able to quickly patch containers without requiring a complete rebuild upstream. As the window between vulnerability disclosure and active exploitation continues to shrink, there is an urgent need to patch critical security vulnerabilities in container images so they can be quickly redeployed into production environments. Especially when these vulnerabilities are:

  • ? Inherited from the base image, has multiple tiers, and cannot be resolved by propagating updated versions through the supply chain

  • ? Exists in third-party application images that you do not maintain, and their update frequency cannot meet your security service level agreement.

ce4c2d0ec81f2765274ff657fe082abe.png

COPA process

In addition to filling an operational gap unmet by shift-left security practices and tools, copa’s ability to patch containers without the need to rebuild container images providesother benefits:

  • ? Allow users other than the image publisher to patch container images, such as DevSecOps engineers.

  • ? Reduce the storage and transport costs of redistributing patched images by only creating additional patched layers rather than rebuilding the entire image, which often results in different layer hashes, corrupting the layer cache.

  • ? Reduce the time it takes to patch container images by not waiting for the base image to be updated and faster than a full image rebuild operation.

  • ? Reduce the complexity of patching images by moving from running a rebuild pipeline to running a single tool on the image.

How to implement?

The copa tool is an extensible engine with the following capabilities:

  1. 1. Parse required update packages from container image vulnerability reports generated by scanners like Trivy. New adapters can be written to support additional report formats.

  2. 2. Use appropriate package management tools (such as apt, apk, etc.) to obtain and process the required update packages. New adapters can be written to support more package managers.

  3. 3. Use buildkit to apply the generated update binary to the container image.

81d26f1df77a87d8b52360d54f80b526. png

copa implementation

This approach is motivated by the following core principles to make direct container patching broadly applicable and accessible:

  • ? Copa supports patching existing container images.

    • ? Developers do not need to use specific tools to build images or modify them in some way to support container patching.

  • ? Copa works with the existing vulnerability scanning and mitigation ecosystem.

    • ? Image publishers do not need to create new workflows for container patching because Copa supports patching container images with already released security updates.

    • ? Consumers do not need to migrate to a new, potentially more restricted support ecosystem to handle custom distributions, nor do they need to change their container vulnerability scanning processes to include remediation, as Copa can be seamlessly integrated as an additional step, Containers are patched based on these scan reports.

  • ? Copa reduces the technical expertise and waiting dependencies required to patch images.

    • ? For operating system package vulnerabilities, no specialized knowledge of a specific image is required to patch it, as Copa relies on vulnerability remediation knowledge already embedded in reports generated by today’s popular container scanning tools.

Installation

Homebrew

On macOS and Linux, copa can be installed via the Homebrew tool [2]:

brew install copa

GitHub

You can download the latest version and earlier versions of copa from the Copa GitHub repository [3].

The following instructions are forUbuntu 22.04,whose dependency versions are part of the development container environment we use for building and testing. For other distributions and operating systems, see the appropriate installation instructions for each component.

git clone https://github.com/project-copacetic/copacetic
cd copacetic
make
# OPTIONAL: install copa to a pathed folder
sudo mv dist/linux_amd64/release/copa /usr/local/bin/

Quick start

This example shows how to use vulnerability reporting and copa to patch a container.

Prerequisites

  • ? The copa tool is installed.

  • ? The buildkit[4] daemon is installed.

  • ? The docker[5] daemon is running and the CLI is installed.

  • ? trivy CLI[6] is installed and installed.

Example

  1. 1. Download the target container for scanning and patching:

docker pull mcr.microsoft.com/oss/nginx/nginx:1.21.6

  1. 1. Scan the container image for patchable operating system vulnerabilities and output the results to a JSON file:

trivy image --vuln-type os --ignore-unfixed -f json -o nginx.1.21.6.json mcr.microsoft.com/oss/nginx/nginx:1.21.6

You can also view existing patchable vulnerabilities in a table on the shell:

trivy image --vuln-type os --ignore-unfixed mcr.microsoft.com/oss/nginx/nginx:1.21.6

  1. 1. To patch the image, use Trivy reporting and specify the buildkit instance to connect to: By default, copa will try to automatically connect to the instance in order: if the instance does not exist or the instance does not support all the features required by copa, then Will try next instance. You may need to specify a custom address using the --addr flag. The following are the supported formats:

    Buildkit connection example

    Example: Connect using default values:

  • ? unix:///path/to/buildkit.sock – Connect to buildkit via unix socket.

  • ? tcp://$BUILDKIT_ADDR:$PORT – Connect to the buildkit via TCP. (not recommended for security reasons)

  • ? docker://– Connect to docker, currently only supports unix sockets, such as docker://unix:///var/run/docker.sock (or only supports docker://).

  • ? docker-container://my-buildkit-container – Connect to buildkitd running in a docker container.

  • ? buildx://my-builder – Connect to the buildx builder (or buildx:// the currently selected builder). NOTE: Currently only container-backed buildx instances are supported

  • ? nerdctl-container://my-container-name – Like docker-container but uses nerdctl.

  • ? podman-container://my-container-name – Similar to docker-container but uses podman.

  • ? ssh://myhost – Connect to the buildkit instance via SSH. The host specification should be formatted to mimic SSH commands.

  • ? kubepod://mypod – Connect to the buildkit running in a Kubernetes Pod. You can also specify the kubectl context and pod namespace ( kubepod://mypod?context=foo & amp;namespace=notdefault).

  1. 1. Default docker buildkit endpoint (requires at least docker v24.0 and containerd snapshotter[7] support enabled)

  2. 2. The currently selected buildx builder (see docker buildx --help🙂

  3. 3. Default address of the buildkit daemon /run/buildkit/buildkitd.sock

copa patch -i mcr.microsoft.com/oss/nginx/nginx:1.21.6 -r nginx.1.21.6.json -t 1.21.6-patched

Example: Connect to buildx

docker buildx create --name demo
copa patch -i mcr.microsoft.com/oss/nginx/nginx:1.21.6 -r nginx.1.21.6.json -t 1.21.6-patched --addr buildx://demo

Example: Buildkit in a container

export BUILDKIT_VERSION=v0.12.0
docker run \
    --detach \
    --rm \
    --privileged \
    --name buildkitd \
    --entrypoint buildkitd \
    "moby/buildkit:$BUILDKIT_VERSION"

copa patch -i mcr.microsoft.com/oss/nginx/nginx:1.21.6 -r nginx.1.21.6.json -t 1.21.6-patched --addr docker-container://buildkitd

Example: TCP-based Buildkit

export BUILDKIT_VERSION=v0.12.0
export BUILDKIT_PORT=8888
 docker run \
     --detach \
     --rm \
     --privileged \
     -p 127.0.0.1:$BUILDKIT_PORT:$BUILDKIT_PORT/tcp \
     --name buildkitd \
     --entrypoint buildkitd \
     "moby/buildkit:$BUILDKIT_VERSION" \
     --addr tcp://0.0.0.0:$BUILDKIT_PORT
 copa patch \
     -i mcr.microsoft.com/oss/nginx/nginx:1.21.6 \
     -r nginx.1.21.6.json \
     -t 1.21.6-patched \
     -a tcp://0.0.0.0:$BUILDKIT_PORT

In either case, copa is non-destructive and exports a new image tagged 1.21.6-patched locally.

Note: If you are running this example against an image in a private registry, please make sure that the credentials are configured in the default Docker’s config.json before running, for example via sudo docker login -u -p .

  1. 1. Scan the patched image and verify that the vulnerability has been patched:

trivy image --vuln-type os --ignore-unfixed mcr.microsoft.com/oss/nginx/nginx:1.21.6-patched

You can also inspect the structure of the patched image to see the new patched layers attached to the image with docker history:

docker history mcr.microsoft.com/oss/nginx/nginx:1.21.6-patched
   IMAGE CREATED CREATED BY SIZE COMMENT
   a372df41e06d 1 minute ago mount / from exec sh -c apt install --no-ins… 26.1MB buildkit.exporter.image.v0
   <missing> 3 months ago CMD ["nginx" "-g" "daemon off;"] 0B buildkit.dockerfile.v0
   <missing> 3 months ago STOPSIGNAL SIGQUIT 0B buildkit.dockerfile.v0
   <missing> 3 months ago EXPOSE map[80/tcp:{}] 0B buildkit.dockerfile.v0
   <missing> 3 months ago ENTRYPOINT ["/docker-entrypoint.sh"] 0B buildkit.dockerfile.v0
   <missing> 3 months ago COPY 30-tune-worker-processes.sh /docker-ent… 4.61kB buildkit.dockerfile.v0
   <missing> 3 months ago COPY 20-envsubst-on-templates.sh /docker-ent… 1.04kB buildkit.dockerfile.v0
   <missing> 3 months ago COPY 10-listen-on-ipv6-by-default.sh /docker… 1.96kB buildkit.dockerfile.v0
   <missing> 3 months ago COPY docker-entrypoint.sh / # buildkit 1.2kB buildkit.dockerfile.v0
   <missing> 3 months ago RUN /bin/sh -c set -x & amp; & amp; addgroup --syst… 61.1MB buildkit.dockerfile.v0
   <missing> 3 months ago ENV PKG_RELEASE=1~bullseye 0B buildkit.dockerfile.v0
   <missing> 3 months ago ENV NJS_VERSION=0.7.0 0B buildkit.dockerfile.v0
   <missing> 3 months ago ENV NGINX_VERSION=1.20.2 0B buildkit.dockerfile.v0
   <missing> 3 months ago LABEL maintainer=NGINX Docker Maintainers <d… 0B buildkit.dockerfile.v0
   <missing> 4 months ago /bin/sh -c #(nop) CMD ["bash"] 0B
   <missing> 4 months ago /bin/sh -c #(nop) ADD file:09675d11695f65c55… 80.4MB

  1. 1. Run the container to verify that the image has not fallen back:

docker run -it --rm --name nginx-test
   
   mcr.microsoft.com/oss/nginx/nginx:1.21.6-patched
   /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
   /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
   /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
   10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
   10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
   /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
   /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
   /docker-entrypoint.sh: Configuration complete; ready for start up
   2022/05/16 18:00:17 [notice] 1#1: using the "epoll" event method
   2022/05/16 18:00:17 [notice] 1#1: nginx/1.20.2
   2022/05/16 18:00:17 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
   2022/05/16 18:00:17 [notice] 1#1: OS: Linux 5.10.102.1-microsoft-standard-WSL2
   2022/05/16 18:00:17 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
   2022/05/16 18:00:17 [notice] 1#1: start worker processes
   2022/05/16 18:00:17 [notice] 1#1: start worker process 31
   2022/05/16 18:00:17 [notice] 1#1: start worker process 32
   2022/05/16 18:00:17 [notice] 1#1: start worker process 33
   2022/05/16 18:00:17 [notice] 1#1: start worker process 34
   2022/05/16 18:00:17 [notice] 1#1: start worker process 35
   2022/05/16 18:00:17 [notice] 1#1: start worker process 36
   2022/05/16 18:00:17 [notice] 1#1: start worker process 37
   2022/05/16 18:00:17 [notice] 1#1: start worker process 38
   2022/05/16 18:00:17 [notice] 38#38: signal 28 (SIGWINCH) received
   2022/05/16 18:00:17 [notice] 36#36: signal 28 (SIGWINCH) received
   2022/05/16 18:00:17 [notice] 33#33: signal 28 (SIGWINCH) received
   2022/05/16 18:00:17 [notice] 32#32: signal 28 (SIGWINCH) received
   2022/05/16 18:00:17 [notice] 34#34: signal 28 (SIGWINCH) received
   2022/05/16 18:00:17 [notice] 35#35: signal 28 (SIGWINCH) received
   2022/05/16 18:00:17 [notice] 37#37: signal 28 (SIGWINCH) received
   2022/05/16 18:00:17 [notice] 1#1: signal 28 (SIGWINCH) received
   2022/05/16 18:00:17 [notice] 31#31: signal 28 (SIGWINCH) received

You can stop the container by opening a new shell instance and running the following command: docker stop nginx-test

Reference materials

Citation link

[1] Copa design: https://project-copacetic.github.io/copacetic/website/design
[2] Homebrew tool: https://brew.sh/
[3] Copa GitHub repository: https://github.com/project-copacetic/copacetic/releases
[4] buildkit: https://github.com/moby/buildkit/#quick-start
[5] docker: https://docs.docker.com/desktop/linux/install/#generic-installation-steps
[6] trivy CLI: https://aquasecurity.github.io/trivy/latest/getting-started/installation/
[7] containerd snapshotter: https://docs.docker.com/storage/containerd/#enable-containerd-image-store-on-docker-engine

This article is reprinted from: “Cloud Native Treasure Box”, original text: https://url.hi-linux.com/mGUwC, the copyright belongs to the original author. Welcome to submit articles. Submission email: [email protected].

167cbc39e550c2c17c1be0cfad55ec0d.gif

Recently, we established a Technical Exchange WeChat Group. At present, many great masters in the industry have joined the group. Interested students can join and exchange technology with us. In the “Wonderful World of Linux”, directly reply “Join the Group” on the public account Invite you to join the group.

b77056d993530925854372bf799f8c08.png

You may also like

Click on the image below to read

e7c1e16754515385f424c8cf09119e88.png

How to use GitLab CI/CD to quickly automate build and release

b29886b248d546d5e2a96d74908adeaf.png
Click on the picture above to receive takeaway red envelopes for free every day from “Meituan | Ele.me”

6396651af7bec995f51284ec738b7de0.png

For more interesting Internet news, follow the “Wonderful Internet” video account to learn all about it!

The knowledge points of the article match the official knowledge files, and you can further learn relevant knowledge. Cloud native entry-level skills treeHomepageOverview 16525 people are learning the system