How to run a stable diffusion model locally

You can see the complete list of possible parameters with default values in the file. Now, let’s see a more complex build prompt using these optional parameters. txt2img.py

In the directory, create a file called . Create multiple prompts, one for each line of the file. For example: stable-diffusionprompts.txt

Now, in the terminal’s directory, run stable-diffusion

python scripts/txt2img.py \
--from-file prompts.txt \
--ckpt sd-v1-4.ckpt \
--outdir generated-images \
--skip_grid \
--ddim_steps 100 \
--n_iter 3 \
--H 256 \
--W 512 \
--n_samples 3 \
--scale 8.0 \
--seed 119

Two resulting images for each title can be seen below. The above command is intended to be used as an example of using more command line parameters, rather than an example of optimal parameters. In general, experience has found that larger images are of higher quality and image/title similarity, and that lower bootstrap ratios may produce better results. Continue to the next section to learn more about improving stable diffusion results.

How to install Stable Diffusion (CPU)

Step 1: Install Python

First, check if Python is installed on your system by typing in the terminal. If the Python version returns, continue to the next step. Otherwise, use python --version

sudo apt-get update
yes | sudo apt-get install python3.8

Step 2: Download the repository

Now we need to clone the Stable Diffusion repository. We will use a fork that can accommodate CPU inference. In the terminal, execute the following command:

git clone https://github.com/bes-dev/stable_diffusion.openvino.git
cd stable_diffusion.openvino

If you don’t have git, you’ll need to use .Be sure to read and accept the Stable Diffusion license before cloning the repository. sudo apt install git

Step 3: Installation Requirements

Install all necessary requirements

pip install -r requirements.txt

Note that Scipy version 1.9.0 is a listed requirement, but it is not compatible with older versions of python. You may need to change the Scipy version by editing, for example before running the above command. requirements.txtscipy==1.7.3

Step 4: Download Stable Diffusion Weights

Now that we are using Stable Diffusion in a proper environment, we need to download the weights required to run it. If you have not already read and accepted the Stable Diffusion License, be sure to do so now. Several Stable Diffusion checkpoint versions have been released. Higher version numbers have been trained on more data and generally perform better than lower version numbers. We will be using Checkpoint v1.4. Use the following command to download the weights:

curl https://f004.backblazeb2.com/file/aai-blog-files/sd-v1-4.ckpt > sd-v1-4.ckpt

That’s all the setup we need to get started with Stable Diffusion! Read on to learn how to use the model to generate images.

How to generate images with stable diffusion (CPU)

Now that everything is installed, we are ready to generate images using Stable Diffusion. To generate an image, just run the following command, change the prompt to whatever you want.

python demo.py --prompt "bright beautiful solarpunk landscape, photorealism"

Inference time is approximately8-12 minutes, so feel free to grab a cup of coffee while Stable Diffusion is running. Below we can see the output of running the above command:

Tips and Tricks

As you get started with Stable Diffusion, keep these tips and tricks in mind as you explore.

Quick Project

The results of text-to-image models can be sensitive to the wording used to describe the desired scene. Prompt engineering is the practice of tailoringpromptsto obtain desired results. For example, if the resulting image is of lower quality, try prefixing the title with “an image of.” You can also specify different styles and media to achieve different effects. Check out each of the drop-down menus below to get ideas:

Image type:

Try preceding the title with one of the following for a different effect:

"A photograph of"
"A headshot of"
"A painting of"
"A vision of"
"A depiction of"
"A cartoon of"
"A drawing of"
"A figure of"
"An illustration of"
"A sketch of"
"A portrayal of"

style

You can specify different styles to achieve different results. Try adding one or more of the following adjectives to the prompt and see the effect.

"Modernist(ic)"
"Abstract"
"Impressionist(ic)"
"Expressionist(ic)"
"Surrealist(ic)"

Aesthetics

You can also try specifying a different aesthetic. Try adding one or more of the following adjectives to the prompt and see the effect.

"Vaporwave"
"Synthwave"
"Cyberpunk"
"Solarpunk"
"Steampunk"
"Cottagecore"
"Angelcore"
"Aliencore"

artist

You can even try assigning different artists to achieve different visual effects. Try appending one of the following options to the prompt:

Painter:

 "in the style of Vincent van Gogh"
"in the style of Pablo Picasso"
"in the style of Andrew Warhol"
"in the style of Frida Kahlo"
"in the style of Jackson Pollock"
"in the style of Salvador Dali"

Sculptor:

 "in the style of Michelangelo"
"in the style of Donatello"
"in the style of Auguste Rodin"
"in the style of Richard Serra"
"in the style of Henry Moore"

Architect:

 "in the style of Frank Lloyd Wright"
"in the style of Mies van der Rohe"
"in the style of Eero Saarinen"
"in the style of Antoni Gaudi"
"in the style of Frank Gehry"

Adjust sampling parameters

When adjusting sampling parameters, you can use the following empirical observations to guide your exploration.

Image size

Generally speaking, as a rule of thumb, larger images perform much better than smaller images in both image quality and title alignment. For tips on 256×256 and 512×512 size images“Guy Fieri gives a tour of a haunted house”, see the following example:

Comparison of 256×256 and 512×512 samples for the prompt “Guy Fieri Visits the Haunted Mansion”

Number of diffusion steps

The number of steps in the diffusion process does not seem to have much of an impact on the results beyond a certain thresholdof about 50 time steps. The image below was generated using the same random seed and prompt“A red sports car”. It can be seen that more time steps consistently improve the quality of the generated images, but the improvement over the past 50 time steps is only reflected in slight changes to the incidental environment of the object of interest. In fact, starting at time step 25, the details of the car are almost identical, and the environment is improving to become more suitable for the car at larger time steps.

Image aspect ratio

Image quality and caption similarity as a function of aspect ratio appear to depend on the input caption. The images below have the same area but different aspect ratios, both generated with the title“Steel and Glass Modern Architecture”. The results are relatively even, although vertical images look best, followed by squares and then horizontal images. This should come as no surprise given that modern buildings of this type are tall and skinny. Therefore, performance as a function of aspect ratio appears to be subject-dependent.

Unfortunately, stable diffusion is limited to resolvable aspect ratios, which makes finer-grained experiments impossible, but regardless, square images should be sufficient for most purposes.

Checkpoint symbolic link

To avoid having to provide a checkpoint every time you build an image, you can create a symbolic link between the checkpoint and the default value. In the terminal, navigate to the directory and execute the following command: --ckpt sd-v1-4.ckpt--ckptstable-diffusion

mkdir -p models/ldm/stable-diffusion-v1/ ln -s sd-v1-4.ckpt models/ldm/stable-diffusion-v1/ model.ckpt

Alternatively, simply move the checkpoint to the default location: --ckpt

mv sd-v1-4.ckpt models/ldm/stable-diffusion-v1/model.ckpt

Final words

That’s all you need to use the new Stable Diffusion model to generate images, if you want to learn more about how Stable Diffusion works, feel free to check out the 3D Modeling Learning Studio.

Reprint: How to run a stable diffusion model locally (mvrlink.com)