Linux – supervisor

What is a supervisor?

Supervisor is a process manager on linux and a management tool. When the process stops, Supervisor can automatically start it, and can run on various unix-like machines, and does not support windows system. Supervisor runs on python3.4 and above and python2.7.

Supervisor is a set of process management programs developed based on the Python language. It can turn ordinary scripts and command line processes into daemons in the background, monitor the status, realize automatic restart, and provide a visual management interface.

How it works

The supervisor starts these managed processes as child processes of the supervisor through fork/exec. When the child process hangs up, the parent process can accurately obtain the information that the child process is down, and can choose whether to start and give an early warning by itself.

Supervisor can set a non-root user for supervisord or each child process, and this user can manage its corresponding process.

Composition

supervisord: The main manager of the supervisor service, responsible for managing the sub-processes we configure, including restarting sub-processes that crash or exit abnormally, and responding to requests from clients.

supervisorctl: The client command line of the supervisord service. Get the status of child processes controlled by the main process, stop and start child processes, and get the run list of the main process.

Web Server: View and control the process status through the web interface.

XML-RPC Interface: The same HTTP server that serves the web UI provides an XML-RPC interface that can be used to interrogate and control the management program and the programs it runs.

Configuration file details

[unix_http_server]
file=/tmp/supervisor.sock ;UNIX socket file, supervisorctl will use
;chmod=0700 ;The mode of the socket file, the default is 0700
;chown=nobody:nogroup ;The owner of the socket file, format: uid:gid

;[inet_http_server] ;HTTP server, providing web management interface
;port=127.0.0.1:9001 ;IP and port of web management background running
;username=user ;Username for logging in to the management background
;password=123 ;Password for logging in to the management background

[supervisord]
logfile=/tmp/supervisord.log ;log file
logfile_maxbytes=50MB ; log file size
logfile_backups=10 ; the number of log files to keep backups
loglevel=info ;log level, default info
pidfile=/tmp/supervisord.pid ;pid file
nodaemon=false ;Whether to start in the foreground, the default is false
minfds=1024 ;The minimum value of the file descriptor that can be opened
minprocs=200 ; the minimum number of processes that can be opened

[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; connect supervisord through UNIX socket, the path is consistent with the file in the unix_http_server part
;serverurl=http://127.0.0.1:9001 ; Connect supervisord via HTTP

; [program:xx] is the managed process configuration parameter, xx is the name of the process
[program:xx]
command=/opt/apache-tomcat-8.0.35/bin/catalina.sh run ; program start command
autostart=true ; Automatically start when supervisord starts
startssecs=10 ; No abnormal exit after 10 seconds of startup
autorestart=true ; The program restarts automatically after exiting, optional values: [unexpected, true, false], the default is unexpected, which means that the process restarts after it is accidentally killed
startretries=3 ; The number of automatic retries for startup failures
user=tomcat ; which user to start the process with
priority=999 ; Process startup priority
redirect_stderr=true ; redirect stderr to stdout, default false
stdout_logfile_maxbytes=20MB ; stdout log file size, default 50MB
stdout_logfile_backups = 20 ; stdout log file backup number, the default is 10
; stdout log file, please note that when the specified directory does not exist, it cannot be started normally, supervisord will automatically create a log file
stdout_logfile=/opt/apache-tomcat-8.0.35/logs/catalina.out
stopasgroup=false ; The default is false, when the process is killed, whether to send a stop signal to this process group, including child processes
killasgroup=false ; the default is false, send a kill signal to the process group, including child processes

;Include other configuration files
[include]
files =/etc/supervisord.d/*.ini ; You can specify one or more configuration files ending with .ini

Installation method

1. Source installation

# Download the source package
wget https://pypi.python.org/packages/7b/17/88adf8cb25f80e2bc0d18e094fcd7ab300632ea00b601cbbbb84c2419eae/supervisor-3.3.4.tar.gz

# unzip
tar -zxvf supervisor-3.3.4.tar.gz

# Enter supervisor-3.3.4
cd supervisor-3.3.4

# download
python setup.py install

2.yum install

yum install supervisor -y

Start the service

[root@master ~]# service supervisord start
Redirecting to /bin/systemctl start supervisord.service

# Watch the process
[root@master ~]# ps aux|grep supervisord
root 17082 0.0 0.6 221476 11880 ? Ss 00:02 0:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf
root 17084 0.0 0.0 112824 988 pts/0 S + 00:02 0:00 grep --color=auto supervisord

Edit configuration file, configure web page access

# remove the comment (;)
[inet_http_server] ; inet (TCP) server disabled by default
port=127.0.0.1:9001 ; (ip_address:port specifier, *:port for all iface)
username=user ; (default is no username (open server))
password=123 ; (default is no password (open server))

[root@master ~]# vim /etc/supervisord.conf
[inet_http_server] ; inet (TCP) server disabled by default
port=192.168.1.120:9001 ; (ip_address:port specifier, *:port for all iface)
username=user ; (default is no username (open server))
password=123 ; (default is no password (open server))

# restart service
[root@master ~]# service supervisord restart
Redirecting to /bin/systemctl restart supervisord.service

[root@master ~]# ps -aux|grep supervisord
root 17112 0.0 0.6 221480 11904 ? Ss 00:05 0:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf
root 17120 0.0 0.0 112824 988 pts/0 S + 00:06 0:00 grep --color=auto supervisord


Browser access: http://192.168.1.120:9001/

Create a guardian nginx process configuration file nginx.ini

[root@master ~]# cd /etc/supervisord.d/
[root@master supervisord.d]# vim nginx.ini
[root@master supervisord.d]# cat nginx.ini
[program:nginx]
command=/usr/./sbin/nginx -g 'daemon off;' user=root
autostart=false
autorestart=true
stopasgroup=true
killasgroup=true
stdout_logfile=/etc/supervisord.d/nginx-out.log
stderr_logfile=/etc/supervisord.d/nginx-err.log


# restart service
[root@master supervisord.d]# service supervisord restart
Redirecting to /bin/systemctl restart supervisord.service

[root@master supervisord.d]# ps aux|grep supervisord
root 17148 0.0 0.6 221764 11916 ?Ss 00:13 0:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf
root 17150 0.0 0.0 112824 988 pts/0 S + 00:13 0:00 grep --color=auto supervisord

Download nginx

yum install nginx -y

Administrative with Supervisor

supervisorctl status # status
supervisorctl stop nginx #close nginx
supervisorctl start nginx #Start nginx
supervisorctl restart nginx #restart nginx
supervisorctl reload #restart all
supervisorctl update #update configuration
[root@master supervisord.d]# supervisorctl status
nginx STOPPED Not started

[root@master supervisord.d]# supervisorctl start nginx
nginx: ERROR (spawn error)

Error: ERROR (spawn error)

View log

[root@master supervisord.d]# tail -f /var/log/supervisor/supervisord.log
2023-08-08 00:13:44,662 INFO supervisord started with pid 17148
2023-08-08 00:17:04,233 INFO spawned: 'nginx' with pid 17191
2023-08-08 00:17:04,249 INFO exited: nginx (exit status 1; not expected)
2023-08-08 00:17:05,258 INFO spawned: 'nginx' with pid 17192
2023-08-08 00:17:05,278 INFO exited: nginx (exit status 1; not expected)
2023-08-08 00:17:07,292 INFO spawned: 'nginx' with pid 17193
2023-08-08 00:17:07,314 INFO exited: nginx (exit status 1; not expected)
2023-08-08 00:17:10,334 INFO spawned: 'nginx' with pid 17194
2023-08-08 00:17:10,340 INFO exited: nginx (exit status 1; not expected)
2023-08-08 00:17:10,340 INFO gave up: nginx entered FATAL state, too many start retries too quickly


[root@master supervisord.d]# cat nginx-err.log
nginx: invalid option: "user=root"
nginx: invalid option: "user=root"
nginx: invalid option: "user=root"
nginx: invalid option: "user=root"

Resolve

[root@master supervisord.d]# cat nginx.ini
[program:nginx]
command=/usr/./sbin/nginx -g 'daemon off;'
user=root
autostart=false
autorestart=true
stopasgroup=true
killasgroup=true
stdout_logfile=/etc/supervisord.d/nginx-out.log
stderr_logfile=/etc/supervisord.d/nginx-err.log

[root@master supervisord.d]# service supervisord restart
Redirecting to /bin/systemctl restart supervisord.service

[root@master supervisord.d]# supervisorctl status
nginx STOPPED Not started

[root@master supervisord.d]# supervisorctl start nginx
nginx: started

[root@master supervisord.d]# supervisorctl status
nginx STARTING