Cutout is an essential capability of image processing tools, which can be used in scenes such as redrawing, reorganization, and background replacement. Recently, I have been exploring various capabilities of Stable Diffusion WebUI, so how does SD WebUI’s cutout capability perform? This article will share with you.
Install plugin
As a generative AI, SD itself does not have fine-grained matting control capabilities, it needs to be implemented with the help of plug-ins.
Here we use the plugin stable-diffusion-webui-rembg. After the plugin is installed successfully, it will appear at the bottom of the “HD” page.
Installation method one
It is suitable for students who have relatively smooth access to various resources on the Internet.
Plugin address: GitHub – AUTOMATIC1111/stable-diffusion-webui-rembg: Removes backgrounds from pictures. Extension for webui.
Install directly through the URL in the Stable Diffusion WebUI, the operation steps are shown in the figure below:
Point 5 indicates that the plug-in has been installed successfully, you only need to restart the SD WebUI on the “Installed” page.
During actual matting, the plug-in will automatically download related models according to the selected matting algorithm, and the waiting time will be longer when using a certain algorithm for matting for the first time.
Installation method two
It is suitable for students who are not very convenient to access the Internet.
First download it locally through other tools, such as Thunder, and then deploy it to the corresponding directory.
Plugin address: GitHub – AUTOMATIC1111/stable-diffusion-webui-rembg: Removes backgrounds from pictures. Extension for webui.
Algorithm model address: GitHub – danielgatis/rembg: Rembg is a tool to remove images background
Or download the files I have packaged, follow the official account: Yinghuo AI, send a message: cutout, and you can get the relevant files.
stable-diffusion-webui-rembg.zip is a plug-in program, after decompression, put it into the stable-diffusion-webui/extensions directory.
u2net.zip is the algorithm model file, after decompression, put it in the temporary directory of the current user:
- Windows is: C:\Users\{Your login Windows user name}\.u2net
- Linux is: /root/.u2net/
If deployed on a cloud server, you need to upload the zip compressed file first, and then decompress it to the specified directory. Refer to the decompression command:
unzip -o /root/autodl-tmp/stable-diffusion-webui-rembg.zip -d /root/stable-diffusion-webui/extensions/stable-diffusion-webui-rembg/ unzip -o /root/autodl-tmp/u2net.zip -d /root/.u2net/
After the installation is complete, remember to restart SD.
use rembg
Basic use
Click the “High Definition” tab, upload the photos to be cutout, select the cutout algorithm for “Remove background”, and finally click “Generate”.
Look at the default cutout effect: there is still some background color in the hair, which is somewhat interlaced with the background, and the algorithm cannot recognize it well.
Note that the black parts around the portrait are actually transparent. If we put this picture on a white background, the surrounding of the portrait will be white.
We tick the “Alpha matting”, and then several options will appear, change the value of “Erode size” to “15”, which can optimize the segmentation of the edge of the foreground image.
Regenerate, look at the result this time, the hair is basically invisible from the previous background color, but the collar is a little blurred. I’ve tried different parameters and haven’t been able to achieve perfection. Perhaps it is the right way to put the picture in PS and then process it. It is result-oriented, and you can’t cling to AI. If you are interested, you can try it.
Parameter introduction
A basic demonstration has been made above, but how can we configure these parameters to achieve the best results? This section will take a look at the definition of these parameters.
Let’s take a look at these algorithm models:
- u2net: A general pre-training model, usually this will do.
- u2netp: A lightweight version of u2net.
- u2net_human_seg: A pre-trained model specifically for portrait segmentation, it is only recommended to use when segmenting portraits.
- u2net_cloth_seg: A pre-trained model that specializes in picking clothes from portraits. It divides clothes into three parts: upper body, lower body and whole body.
- silueta: Same as u2net, but the size is reduced to 43Mb, which is convenient for use on small memory machines.
- isnet-general-use: A new general-purpose pre-training model.
- isnet-anime: High-precision segmentation specifically for animated characters.
Look at its two options again:
- Alpha matting: Alpha matting, this is a more professional image processing term. In image processing, there is something used to represent the transparency of each pixel in the image, which is called the Alpha channel; then the Alpha mask uses the Alpha channel to control the transparency of the image, so as to achieve the purpose of hiding or showing certain parts ; In the cutout, it is trying to show the foreground part and make the background part transparent. This option has three parameters, which are used to control the effect of cutout, let’s see:
-
- Erode size: Alpha matting erosion size, erode by constructing a rectangle whose length and width are this value in the image. If it is too small, the foreground and background will not be separated completely, and the edges will cross; if it is too large, the foreground and background will be corroded too much, and the edge defect will be obvious.
- Foreground threshold: The threshold of the foreground image. If the value is too small, the background may be recognized as the foreground, and if the value is too large, the foreground may be recognized as the background.
- Background threshold: The threshold of the background image. If the value is small, the foreground may be recognized as the background, and if the value is large, the background may be recognized as the foreground.
When using the Alpha mask, you can initially use these empirical values: (15, 220, 100), and then adjust the specific parameter values according to the actual situation.
- Return mask: Returns the mask image of the cutout image, and its usage and purpose will be introduced shortly below.
Using masks
This section takes changing the image background as an example to demonstrate the usage of masks. The specific idea is to generate a photo of Musk landing on Mars and encountering aliens.
Generate mask
As long as you check “Return mask” when generating, the final output image will become a mask image.
As shown in the figure below, you can see that the character has become a white mask. We save this mask image to the local machine first, and we will use it later.
Change background
Open “Partial drawing (upload mask)” in “Graphic Image”. Here you need to upload two photos, one original image and one mask image of the character.
The specific parameters of the graph are as follows:
Prompt words: (the desert), ((night)), dim sun, (stargate), a man in a suit and white shirt smiling for a picture, a alien standing in the distance, digital painting, stargatejackal, surrealistic, hdri, smooth, sharp focus, illustration, fantasy, intricate, elegant, highly detailed, 8k
Reverse cue words: EasyNegative, moon
Scale mode: fill, because I want to adjust the aspect ratio of the picture to 16:9, the original picture is not so wide, so the newly expanded space needs to be filled.
Mask mode: Draw non-masked content. In “Partial drawing (upload mask)”, the white area is a mask, because we want to redraw the background, so here we choose “Draw non-masked content”.
Sampler: DPM + + 2M SDE Karras, this is a newly added sampler, it is recommended to experience it. Of course other samplers can also be used.
Sampling steps: match the sampler settings.
Width, Height: Set according to the ratio of 16:9.
Look at the effect of the plot:
The above is the main content of this article, and interested students should try it quickly.
If you are just starting to learn AI painting, it is recommended to read these two Stable Diffusion WebUI installation guides first:
Teach you how to install the Stable Diffusion Autumn Leaf integration package locally
Teach you how to make alchemy in cloud environment (deploy Stable Diffusion WebUI)
The knowledge points of the article match the official knowledge files, and you can further learn relevant knowledgePython entry skill treeHomepageOverview 330725 people are studying systematically