Django+Nginx+uWSGI+Supervisor actual combat

Hello everyone, I haven’t updated the article for a long time. I miss it very much. I have a lot of things going on recently, including family matters and work matters. I really don’t have much time to write articles. In fact, I have shelved some articles and haven’t organized them yet. Taking advantage of the calm situation during this period, I still remember my original intention and share it with you.

Let’s talk about network automation (NetDevOps):

Why are we discussing this? In fact, during this time I was also searching for a job and thinking about my future career. Most of my fans are network engineers, mainly network workers. There are very few who specialize in network automation. What? At most, use it as a tool.

Housework:

I am a father of two children. I send my children to kindergarten and buy groceries in the morning. I have to make soup/cook a dish after get off work at noon. I pick up my children in the afternoon. I also have to cook and take them to bed at night. Unknowingly it has been more than 3 years and this is a part of my daily routine.
I had very little time to study. I didn’t start coding until early in the morning. In the past few years, I have basically devoted all my energy to learning and development. Maybe it was my interest that made me persevere, but I neglected my own skills.

Leave some pitfalls:
I am also a network engineer, with dual IEs. The Huawei IE number is 4 digits (a lie), and python is my auxiliary tool.
On the way to job hunting, I found that there are very few network automation positions (NetDevOps) outside (it can be said that there are none). If you want to take a look at the outside market, you can search on the recruitment platform.
Focus on network engineering technology/management. Don’t get too deep into network development. Don’t go in-depth to learn the front-end framework or the back-end Django. Don’t go astray (except for full-time network automation) ;
Because learning the front-end is too expensive, time-consuming and energy-consuming, it is easy to ignore the technical direction of one’s job. The same is true for back-end development. If you are a top student, please ignore it.
Understand that companies that specialize in network development: Byte, Tencent, XX Bank, Vipshop, iFlytek, etc. are all full-time developers and rarely merge network and development (professionals do professional things).
Currently, I am focusing on technical learning directions such as cloud computing, SDN network, SegmentRouting(SR), SRv6, IB network and TCP/IP detailed explanation, supplemented by development.
Make good use of chatGPT and use AI tools in multiple ways;

You can learn development if you are interested, but don’t take it seriously. Think about what the market needs in the future and what do you want to do?

Time is precious, spend it wisely.

Okay, let’s stop chatting here, and then share the following article with friends in need.

Introduction

What is Django?

Django is a Web framework developed based on Python language. It provides an efficient way to design, develop and deploy web applications, allowing developers to focus more on the implementation of business logic rather than the implementation of the underlying technology. The Django framework provides many powerful features, such as ORM (Object Relational Mapping), template system, form processing, etc., which can help developers develop web applications more efficiently. There are other open source web frameworks such as Flask.

What is Nginx?

Nginx is a web server that can also be used as a reverse proxy, load balancer and HTTP cache. It is known for its high performance, stability and low resource consumption. Nginx is commonly used to improve the performance and reliability of web applications by serving static content directly and passing dynamic content to application servers such as uWSGI.

Someone may ask here, isn’t Django’s runserver accessible after it is started? What else does nginx need to do? This is not recommended in a production environment, and the concurrency efficiency is low. It should only be used in a development environment.

What is uWSGI?

Web Server Gateway Interface, Web Server Gateway Interface

Client(user)<=>Nginx(Web server)<=>uWSGI<=>Django-APP(application)

WSGI is a web server gateway interface used for communication between web servers and web applications. uWSGI is a protocol used for communication between web servers and web applications. When using uWSGI, the web server sends a request to the uWSGI server through the uwsgi protocol, and then the uWSGI server forwards the request to the web application. Finally, the web application sends the response back to the uWSGI server, and then the Which is sent back to the web server. Therefore, it can be said that uWSGI is a software (bridge in the middle) used to connect web servers and web applications.

Nginx: It is user-oriented
uWSGI: Web server
WSGI: specification for communication between web servers and web applications
uwsgi: It is a self-owned protocol of WSGI communication specification
uWSGI is an intermediate role, bridging Nginx and APP

nginx-uWsgi

What is Supervisord?

Supervisor is a process control system written in Python. It can run and monitor multiple processes (like ngnix/uwsgi/celery, etc.) in the background and restart them if they fail. Supervisor can also provide web interface access (closed by default) to facilitate viewing and management of processes.

Warm reminder: supervisord does not support the windows environment, supervisor-win can support the windows environment.

Supervisord official introduction

How to deploy?

Django deployment

For detailed installation and deployment, please refer to the previous article. Here is a brief list below.

Install

# Take version 3.2 as an installation example
$ python -m pip install django==3.2

New Project

$ python manage.py startproject <your-project-name>

Create a new application APP

$ python manage.py startapp <your-app-name>

Test starting the server
```
$ python manage.py runserver 0.0.0.0:8080
```
If you just start deploying, you can see that Little Rocket is started normally. You can handle the subsequent routing, pages, styles, etc. by yourself.

Key uWSGI configuration

First create the file uwsgi.ini and place it in the same directory as settings. The configuration is as follows.

First, let’s give the basic directory of the project for easy viewing:

(py389) [netdevops@NetDevOps]$ pwd
/home/netdevops/django/NetDevOps
# Mainly lists the information of the same level directory of settings
(py389) [netdevops@NetDevOps]$tree NetDevOps
NetDevOps/
├── asgi.py
├── celery.py
├── __init__.py
├── NetDevOps.sock # Automatically created
├── routing.py
├── settings.py
├── urls.py
├── uwsgi.ini # I put it in the same directory as settings
└── wsgi.py

? uwsgi.ini configuration

# Create file
$ touch uwsgi.ini
$ cat uwsgi.ini
[uwsgi]
# Use unix socket to communicate with nginx, only when uwsgi and nginx are on the same host
# uwsgi_pass in Nginx configuration should point to the same socket file
# socket indicates that the received uwsgi protocol
socket=/home/netdevops/django/NetDevOps/NetDevOps/NetDevOps.sock

# Set up the project virtual environment, not required for Docker deployment
home=/home/netdevops/django/NetDevOps
# Absolute path to the project
chdir=/home/netdevops/django/NetDevOps
# wsgi file location
module=NetDevOps.wsgi:application

#wsgi file, relative path
wsgi-file=NetDevOps/wsgi.py

# python virtual environment
virtualenv=/home/netdevops/.pyenv/versions/py389

# socket permission settings
chmod-socket=664
#Configure the number of processes to start. There is no need to configure this too much. Configure it according to your own computer core.
processes=4
#Configure the number of threads for each process
threads=2
#Enable main process management mode
master=True

#Configure the process ID file that stores the main process. After starting uwsgi, the **uwsgi.pid file will be automatically generated**
pidfile=uwsgi.pid
#Start in the background and specify the log file location. If you use supervisor to manage uwsgi, it must be disabled.
#daemonize=uwsgi.log
#Automatically remove unix Socket and pid files when the service stops
vacuum=True

Nginx deployment

Install nginx using yum method

# Installation
[netdevops@NetDevOps ~]$ yum install nginx

# View version
[netdevops@NetDevOps ~]$ nginx -v
nginx version: nginx/1.20.1

Enter the nginx program directory and test nginx

[netdevops@NetDevOps ~]$ cd /usr/sbin/
[root@NetDevOps sbin]# ./nginx -t
# Displaying the following two lines means it can run normally.
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Configure nginx

For the configuration of nginx, refer to Dajianggou’s article and adjust it as needed.

$ cat /etc/nginx/nginx.conf
########################## Global Start ###################### ####
# Set user
user netdevops;
# Number of processes, usually set to 1-2 times the number of CPU cores
worker_processes auto;
# Error log file
error_log /var/log/nginx/error.log;
# Store process ID file
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

######################### Global End ###################### ####

######################### Event Start ###################### ####

events {<!-- -->
    #Use epoll's I/O model to handle polling events
    # use epoll
    #Maximum number of connections for worker processes, default 1024
    worker_connections 1024;
    # Keep-alive timeout at http level
    # keepalive_timeout 60;
    #Buffer size of client request header
    # client_header_buffer_size 2k;
}

######################### Event End ###################### ####


######################### HTTP Start ###################### ####
http {<!-- -->
    # Log format
    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    # access log path
    access_log /var/log/nginx/access.log main;
    # Allow sendfile mode to transfer files, the default is off.
    sendfile on;
    #Open only when sendfile is enabled.
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 4096;

    #Import file extension and file type mapping table
    include /etc/nginx/mime.types;
    #Default file type
    default_type application/octet-stream;

    include /etc/nginx/conf.d/*.conf;

    # Enable gzip compression function
    gzip on;
    # Set the minimum number of bytes of pages allowed for compression; this means that if the file is less than 10k, compression is meaningless.
    gzip_min_length 10k;
    # Set the compression ratio, the minimum is 1, the processing speed is fast, the transmission speed is slow
    #9 is the maximum compression ratio, with slow processing speed and fast transmission speed; 6 is recommended
    gzip_comp_level 6;
    #Set the compression buffer size, here it is set to 16 8K memories as the compression result buffer
    gzip_buffers 16 8k;
    # Set which files need to be compressed. Compression is recommended for general text, css and js. Pictures can be compressed as needed
    gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml + rss text/javascript;

       # Reverse proxy
    server {<!-- -->
        listen 80;
        listen [::]:80;
        server_name localhost;
        root /usr/share/nginx/html;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

    # The directory where the static files of the website are located
    location /static{<!-- -->
        alias /home/netdevops/django/NetDevOps/static;
# Static file cache, valid for 10 days
expires 10d;
    }

    # The directory where the uploaded file is located
    location /media{<!-- -->
        alias /home/netdevops/django/NetDevOps/media;
    }

    # Handle dynamic files to uwsgi for processing
    location / {<!-- -->
        uwsgi_pass unix:/home/netdevops/django/NetDevOps/NetDevOps/NetDevOps.sock;
        include /etc/nginx/uwsgi_params;

# Allow all users on the intranet to access, fill in as needed
allow 10.0.0.0/8;
# reject all
deny all;
        
    }

        location = /favicon.ico {<!-- -->
log_not_found off;
# Do not record this access in access_log
access_log off;
              }

        error_page 404 /404.html;
        location = /404.html {<!-- -->
        }

        error_page 500 502 503 504 /50x.html;
        location = /50x.html {<!-- -->
        }
    }
}
########################## HTTP End ###################### ####

? Note: My environment here only has HTTP, not HTTPS.

Supervisor deployment:

First, let’s briefly introduce the four major components of Supervisor:

supervisord: The main process of Supervisor. It is responsible for starting and stopping child processes and monitoring their status. If a child process exits abnormally, supervisord will automatically restart it and record the child process stdout and stderr output.
supervisorctl: Supervisor’s command line interface, start, stop, restart child processes, view process status and logs, etc.
Web interface: Supervisor provides a Web interface to view and manage processes through a Web browser. The web interface provides the same functionality as supervisorctl, but is more intuitive and easier to use in a web browser (not enabled by default).
xml-rpc interface: Provides an xml-rpc interface for interrogating and controlling management programs and other running programs.

Installation

# pip install supervisord
$ python -m pip install supervisor
# Check the installation path
$ which supervisord
/usr/local/bin/supervisord

Create directories and files

 #Create a directory to store the subprogram file xxxx.ini
 # Example: nginx.ini, uwsgi.ini, celery.ini
$ mkdir -p /etc/supervisord.d/
$ mkdir -p /etc/supervisord.d/logs/uwsgi
$ mkdir -p /etc/supervisord.d/logs/nginx
$ cd /etc/supervisord.d/
$ touch nginx.ini
$ touch uwsgi.ini
$ ll /etc/supervisord.d/
-rw-r--r-- 1 root root 0 Aug 20 15:52 nginx.ini
-rw-r--r-- 1 root root 0 Aug 20 15:53 uwsgi.ini

Generate configuration file

 # Execute directly, you can view the configuration
 $ echo_supervisord_conf
  
 # Generate supervisord.conf file and start with this file
 $ echo_supervisord_conf > /etc/supervisord.conf

Modify configuration file

Each subprocess is stored in the /etc/supervisord.d/ directory as a subprocess.ini file to facilitate differentiated management.

$ vim /etc/supervisord.conf
# Modify to the following content
[unix_http_server]
chmod=0777 # Allow non-root running

[include]
files = /etc/supervisord.d/*.ini

Add uwsgi process

$ vim /etc/supervisord.d/uwsgi.ini
# Added program
[program:uwsgi]
# directory
directory=/home/netdevops/django/NetDevOps/NetDevOps
# Start uwsgi execution command
command=/home/netdevops/.pyenv/versions/py389/bin/uwsgi --ini /home/netdevops/django/NetDevOps/NetDevOps/uwsgi.ini
# priority
priority=10
# Process name
process_name=%(program_name)s
# Log level, default info
loglevel=info
# Input stderr to the stdout file
redirect_stderr=true
# The file path of stdout output
stdout_logfile=/etc/supervisord.d/logs/uwsgi/uwsgi.log
# Maximum size of each file is 100MB, default is 50MB
stdout_logfile_maxbytes=100MB
# Save 10 copies of the file
stdout_logfile_backups=10
# The maximum size of each file is 100MB, any excess will be saved to a new file
stdout_capture_maxbytes=100MB
# The file path output by stderr
stderr_logfile=/etc/supervisord.d/logs/uwsgi/uwsgi_error.log
stderr_logfile_maxbytes=100MB
stderr_logfile_backups=10
stderr_capture_maxbytes=100MB

Other parameters:

autostart: By default true, the supervisor starts and the child process starts.
startsecs: Default is 1s, the time the child process needs to keep running after it is successfully started.
autorestart: Default is unexpected, and the subroutine will automatically restart when it exits.
startretries: Default is 3 times, the number of failed restarts allowed.
priority: Default is 999, smaller values will be started first.
redirect_stderr: The default is false. If set to true, it means that stderr is input to the stdout file.
logfile_backups: number of backup copies, default 10 copies;
loglevel: Log levels include critical, error, warn, info, debug, trace, or blather, the default is info;
pidfile: process file, $CWD/supervisord.pid

Add nginx process

$ vim /etc/supervisord.d/nginx.ini
[program:nginx]
# directory
directory=/usr/sbin
# Run in foreground mode
command=/usr/sbin/nginx -g 'daemon off;'
# priority
priority=20
# Process name
process_name=%(program_name)s
#user
user=root
# Log level, default info
loglevel=info
# Input stderr to the stdout file
redirect_stderr=true
# The file path of stdout output
stdout_logfile=/etc/supervisord.d/logs/nginx/nginx.log
# Maximum size of each file is 100MB, default is 50MB
stdout_logfile_maxbytes=100MB
# Save 10 copies of the file
stdout_logfile_backups=10
# The maximum size of each file is 100MB, any excess will be saved to a new file
stdout_capture_maxbytes=100MB
# The file path output by stderr
stderr_logfile=/etc/supervisord.d/logs/nginx/nginx_error.log
stderr_logfile_maxbytes=100MB
stderr_logfile_backups=10
stderr_capture_maxbytes=100MB

Temporarily start supervisor

First temporarily start it to see if it can manage the nginx and uwsgi processes, and then close it after normal operation.

# Start
$ supervisord -c /etc/supervisord.conf

# closure
 supervisorctl stop all

View and operate processes

Usually, you mainly use the following operation commands.

View status

$ supervisorctl status
uwsgi RUNNING pid 19835, uptime 1 day, 0:09:14

Restart all processes

# Restart all child processes
$ supervisorctl reload
$ supervisorctl status
celery RUNNING pid 122784, uptime 0:00:37
celery_beat RUNNING pid 122785, uptime 0:00:37
uwsgi RUNNING pid 122786, uptime 0:00:37
# I have 3 sub-processes here. Looking at the uptime time, they all started not long ago.

Pause and start

# Pause a process
$ supervisorctl stop uwsgi
# Start a process
$ supervisorctl start uwsgi
# Check the status of a process
$ supervisorctl status uwsgi
# Pause all processes
$ supervisorctl stop all
# Start all processes
$ supervisorctl start all
# Restart all processes
$ supervisorctl reload

Reload configuration

# Reload configuration
$ supervisorctl update

Start up

Here, supervisord is started through systemctl. Supervisord will be automatically activated when the host is shut down and restarted.

Create supervisord.service file

$ cat /etc/systemd/system/supervisord.service
[Unit]
Description=Supervisor process control system for UNIX

[Service]
Type=forking
ExecStart=/usr/local/bin/supervisord -c /etc/supervisord.conf
ExecStop=/usr/local/bin/supervisorctl shutdown
ExecReload=/usr/bin/supervisorctl reload
KillMode=process

[Install]
WantedBy=multi-user.target

Turn on startup

# Start at boot
$ systemctl enable supervisord

# Start supervisord
$ sudo systemctl start supervisord
$ sudo systemctl status supervisord
● supervisord.service - Supervisor process control system for UNIX
   Loaded: loaded (/etc/systemd/system/supervisord.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2023-08-20 23:38:41 CST; 23s ago
...omitted...

# Stop supervisord
$ sudo systemctl stop supervisord

Finally verify the status of supervisorctl

$ supervisorctl status
celery RUNNING pid 2298, uptime 0:01:01
celery_beat RUNNING pid 2299, uptime 0:01:01
nginx RUNNING pid 2300, uptime 0:01:01
uwsgi RUNNING pid 2301, uptime 0:01:01

Warm reminder:: Mainly observe whether the status remains RUNNING, otherwise you need to check Log abnormality problem.

Finally, let’s post a rendering!!! Start normally.