Run Django better with Gunicorn, Supervisor and Nginx

Article directory

  • Running Django via runserver
  • Running Django via Gunicorn
  • Reverse proxy through Nginx
  • Host gunicorn and nginx through Supervisor

Also published on personal website: https://panzhixiang.cn

Run Django through runserver

I believe that those who have used Django for development must be familiar with the python manage.py runserver command. This command uses a web server that comes with Django to help us run Django locally easily. It is enough for local testing, but it cannot be used in a production environment, or even a test environment. The main problems are as follows:

  1. Poor performance
    It is single-process, single-threaded, so it can only handle one request at a time. As the number of requests increases, the server’s CPU and memory usage will continue to rise, eventually leading to performance degradation.

  2. Limited functionality
    It only supports basic HTTP requests and does not support HTTPS, load balancing, static file serving and other functions. In a production environment, these features are essential.

Run Django through Gunicorn

Before officially using Gunicorn, let’s talk about its advantages:
(The following content is from GPT4)

Gunicorn (Green Unicorn) is a Python WSGI HTTP server. Many large websites and high-performance applications use it to host Python web applications. Here are some of the main benefits of Gunicorn:

  1. Easy to use: Gunicorn is very easy to install and configure. You just need to install it via pip and start a WSGI application with a few lines of commands. This allows developers to focus more on their application code and less on deployment and operational issues.

  2. Strong compatibility: Gunicorn fully supports the WSGI specification and is therefore theoretically compatible with all Python web frameworks that follow the specification, including but not limited to Django, Flask, and Pyramid.

  3. Pre-fork model: Gunicorn uses the Unix fork system call to create child processes (workers), and each worker can handle requests independently. Using the pre-fork model, the parent process can kill and rebuild the child process when necessary, providing a protection mechanism against memory leaks.

  4. Load Management: Gunicorn has multiple types of workers to choose from, including synchronous workers and asynchronous workers. Synchronous workers are suitable for CPU-intensive tasks, while asynchronous workers are suitable for IO-intensive or network-intensive tasks. This means you can choose the most appropriate worker type based on the characteristics of your application.

  5. Error isolation: Since each request is processed by its own worker process, if a request causes a crash or uncaught exception, it will only affect the worker processing the request, and other workers or requests will Not affected.

  6. Robustness and reliability: Gunicorn can automatically manage worker processes. If a worker exits abnormally, the Gunicorn main process will automatically restart a new worker process to replace it, thus ensuring the continuous availability of the service.

  7. Flexible configuration: Gunicorn provides a large number of configuration options. You can control the log level, output location, number of workers, request timeout, SSL settings and many other aspects. At the same time, Gunicorn supports reading configurations from Python files, environment variables, and command line parameters, meeting the needs of different scenarios.

  8. Hot reload: Gunicorn supports seamless restart, which means you can upgrade code or configuration without interrupting service.

  9. Community support and documentation: Last but not least, Gunicorn has an active developer community and detailed official documentation, which can provide a lot of help and guidance to developers using Gunicorn.

Before running Django with Gunicorn, you need to make sure that Django and Gunicorn are installed correctly. Let’s say your Django project is called myproject and it’s located under /path/to/myproject/.

  1. Install Gunicorn
    If you haven’t installed Gunicorn yet, you can use pip to install it:

    pip install gunicorn

  2. Run Gunicorn
    Now you should be able to launch your Django application with Gunicorn. The basic command format of Gunicorn is gunicorn [OPTIONS] APP_MODULE, where APP_MODULE is a Python import path pointing to the module containing the WSGI application object.

    In a standard Django project, this object (often called the application) is defined in the wsgi.py file. So if your project is named myproject, then APP_MODULE is myproject.wsgi.

    To tell Gunicorn to start a Django project, you can execute the following command:

    cd /path/to/myproject/
gunicorn myproject.wsgi

    This will start your Django application on the Gunicorn server listening on localhost:8000.

    Note: This is just to demonstrate the simple operation method of gunicorn. It is not recommended to be used in a formal environment

  3. Configuring Gunicorn

    Gunicorn provides many configurable options, and you can adjust its behavior according to your own needs. The more common way is to create a Gunicorn configuration file to make your configuration more structured and easier to manage. Gunicorn’s configuration file is usually a Python script that defines some global variables.

    Suppose we create the following configuration file in /path/to/myproject/gunicorn_config.py:

     # gunicorn_config.py
 import multiprocessing
 
 # Bind ip and port number
 bind = "0.0.0.0:8080"
 
 # Use gevent mode, you can also use sync mode, the default is sync mode
 worker_class = 'gevent'
 
 # Number of processes started
 workers = multiprocessing.cpu_count() * 2 + 1
 
 # Number of requests processed concurrently
 threads = 2
 
 #Maximum number of pending connections
 backlog=2048
 
 #Work mode coroutine
 worker_connections = 1000
 
 # Automatically reload the program after reloading or modifying the configuration
 reload=True
 
 #Access log file
 accesslog = "/var/log/gunicorn/access.log"
 
 # Error log file
 errorlog = "/var/log/gunicorn/error.log"

    In the above configuration, we set multiple parameters, such as binding address, working mode, log location, etc. This is just a basic configuration example, you can modify or expand it according to actual needs.

    You can then run the Django application by specifying the path to the configuration file via the -c or --config command line option, as follows:

    cd /path/to/myproject/
gunicorn myproject.wsgi -c gunicorn_config.py

    This command tells Gunicorn to load the gunicorn_config.py file and apply the configuration defined there.

Reverse proxy through Nginx

Gunicorn is much better than Django’s runserver, but in practice, gunicorn is generally not exposed directly to the outside world. Instead, a layer of reverse proxy is added. The most commonly used one is Nginx.

Using Nginx as a reverse proxy has the following main advantages:

  1. Static file processing: Nginx is very good at processing static content (such as CSS, JavaScript files, or images), while Python WSGI servers are generally not suitable for serving static files directly, which may cause performance issues. By offloading static file serving tasks to Nginx, you free up Gunicorn’s resources to handle dynamic content.

  2. Load balancing: If you have multiple backend servers or multiple worker processes, Nginx can effectively distribute incoming requests to each backend server to achieve load balancing. It also supports multiple load balancing strategies and health checks.

  3. Buffered requests: Nginx provides a layer of protection to the backend because it intercepts and handles all client connections. This means that the backend server only needs to handle complete requests and does not need to worry about network issues or slow connections. In addition, if the backend application hangs or restarts, Nginx can still continue to provide services to users during this period (such as returning a friendly error page).

  4. SSL Termination: If your website requires SSL encryption, Nginx can handle all HTTPS handshakes and communicate unencrypted with the backend server, taking the load off the backend server.

  5. HTTP/2 support: Nginx supports the HTTP/2 protocol, and most WSGI servers including Gunicorn currently have no plans to directly support HTTP/2. By enabling HTTP/2 in Nginx, your users can enjoy faster loading times and lower latency.

  6. Access control and security protection: Nginx provides a series of security-related functions, such as IP whitelist/blacklist, rate limiting, preventing DDOS attacks, etc.

  7. gzip compression: Nginx can perform gzip compression on response data, thereby reducing network bandwidth consumption and improving page loading speed.

Here are the detailed steps to use Nginx as a reverse proxy for Gunicorn

  1. Install Nginx

    On Ubuntu/Debian, you can install Nginx via apt-get:

    sudo apt-get install nginx

  2. Configuring Nginx

    Configure Nginx so that it correctly forwards requests to Gunicorn. Nginx’s settings files are usually located at /etc/nginx/sites-available/default.

    Here is a basic configuration example:

    server {<!-- -->
    listen 80;
    server_name yourdomain.com;

    location /static/ {<!-- -->
        alias /path/to/myproject/static/; # This points to the static file directory in Django
    }

    location / {<!-- -->
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        
        proxy_pass http://localhost:8000; # Replace with the address and port number gunicorn is listening on
    }
}

  3. Start Nginx

    After completing the configuration, you can start Nginx:

    sudo service nginx start

The above is the basic process. Of course, this is just the simplest configuration, if you need more advanced features (such as HTTPS, load balancing or caching), more configuration will be required. It is recommended to consult the Nginx official documentation for more detailed information.

Host gunicorn and nginx through Supervisor

I only came into contact with supervisor in my second job. After getting to know it, I liked this tool very much. The biggest advantage is that it can host a certain process. Especially if the process has a problem and dies, the supervisor will automatically try to restart the process. This is very useful for Very important in the online environment.

Supervisor is a process management tool written in Python, which can be easily used to start, restart (automatically) and shut down processes under UNIX-like systems (Windows is not supported).

Here are the steps on how to host gunicorn and nginx using Supervisor:

  1. Install Supervisor

    On Ubuntu/Debian, you can install Supervisor via apt-get:

     sudo apt-get install supervisor

  2. Create a Supervisor profile

    You need to create a configuration file for each program that is to be managed by Supervisor. These files are usually located in the /etc/supervisor/conf.d/ directory and end with .conf.

    Assuming above, the Django project path is /path/to/myproject/, and Gunicorn’s configuration file is named gunicorn_config.py, then we need to create a file named for Gunicorn >myproject_gunicorn.conf file:

    # /etc/supervisor/conf.d/myproject_gunicorn.conf

[program:myproject_gunicorn]
command=/usr/local/bin/gunicorn myproject.wsgi:application -c /path/to/myproject/gunicorn_config.py
directory=/path/to/myproject/
user=yourusername
autostart=true
autorestart=true
redirect_stderr=true

    Similarly, we also need to create a file named nginx.conf for Nginx:

    # /etc/supervisor/conf.d/nginx.conf

[program:nginx]
command=/usr/sbin/nginx -g "daemon off;"
autostart=true
autorestart=true
redirect_stderr=true

  3. Run Supervisor

    After installing Supervisor and creating the relevant configuration files, you can let Supervisor start working. First, you need to read any new or modified configuration files:

    sudo supervisorctl reread

    Next, you can update the status of the Supervisor service so that it starts running the newly added program:

    sudo supervisorctl update

    Or, if you want to start a separate program, such as myproject_gunicorn (that is, Django above), you can do this:

    sudo supervisorctl start myproject_gunicorn