1. Common cluster preparation environment
Note that these three servers are all connected, and the RabbitMQ cluster nodes must be in the same segment. If they are connected across domains, the effect will be worse.
10.36.192.150 rabbitmq-1
10.36.192.151 rabbitmq-2
10.36.192.152 rabbitmq-3
All three servers have their firewalls and selinux turned off
[root@rabbitmq-1 ~]#systemctl stop firewalld & amp; & amp; systemctl disable firewalld [root@rabbitmq-1 ~]#setenforce 0 & amp; & amp; sed -ri 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux
Configure and install rabbitmq software on three nodes
#Install dependencies [root@rabbitmq-1 ~]# yum install -y epel-release gcc-c + + unixODBC unixODBC-devel openssl-devel ncurses-devel # Download and yum install erlang [root@rabbitmq-1 ~]# wget https://packagecloud.io/rabbitmq/erlang/packages/el/7/erlang-21.3.8.21-1.el7.x86_64.rpm?distro_version_id=140 [root@rabbitmq-1 ~]# yum install erlang-21.3.8.21-1.el7.x86_64 # Download and yum install rabbitmq [root@rabbitmq-1 ~]# wget https://github.com/rabbitmq/rabbitmq-server/releases/tag/v3.7.10 [root@rabbitmq-1 ~]# yum install rabbitmq-server-3.7.10-1.el7.noarch.rpm # Check rabbitmq and erlang compatible versions https://www.rabbitmq.com/which-erlang.html #erlang version selection https://packagecloud.io/rabbitmq/erlang # rabbitmq version selection https://www.rabbitmq.com/news.html
Start the RabbitMQ service
method one:
[root@rabbitmq-1 ~]# systemctl daemon-reload [root@rabbitmq-1 ~]# systemctl start rabbitmq-server [root@rabbitmq-1 ~]# systemctl enable rabbitmq-server
Method two:
[root@rabbitmq-1 ~]# /sbin/service rabbitmq-server start
Open rabbitmq’s web access interface
[root@rabbitmq-1 ~]# rabbitmq-plugins enable rabbitmq_management
Create a user (note: you can operate the experiment on one machine)
#Add user and password [root@rabbitmq-1 ~]# rabbitmqctl add_user suqin 123456 #Set user role [root@rabbitmq-1 ~]# rabbitmqctl set_user_tags suqin administrator #View users [root@rabbitmq-1 ~]# rabbitmqctl list_users
Set permissions
When setting permissions here, please note that there needs to be a space between ‘.*’. Three ‘.*’ represent conf permissions, read permissions and write permissions respectively. For example: when these three permissions are not set for newrain, there are none before. The permission query queue is not visible in the UI interface.
[root@rabbitmq-1 ~]# rabbitmqctl set_permissions -p "/" suqin ".*" ".*" ".*"
Enable remote user login
[root@rabbitmq-1 rabbitmq]# cp /usr/share/doc/rabbitmq-server-3.7.10/rabbitmq.config.example /etc/rabbitmq/rabbitmq.config
[root@rabbitmq-1 rabbitmq]# ls
enabled_plugins rabbitmq.config
[root@rabbitmq-1 rabbitmq]# vim /etc/rabbitmq/rabbitmq.config + 61
{loopback_users, []} #Open the comment and delete the following ,
[root@rabbitmq-1 ~]# systemctl restart rabbitmq-server #Restart service
View port
[root@rabbitmq-1 ~]# ss -nplt
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:4369 *:* users:(("epmd",pid=8723,fd=3))
LISTEN 0 128 *:15672 *:* users:(("beam.smp",pid=8525,fd=74))
LISTEN 0 128 *:25672 *:* users:(("beam.smp",pid=8525,fd=62))
LISTEN 0 128 [::]:4369 [::]:* users:(("epmd",pid=8723,fd=4))
LISTEN 0 128 [::]:5672 [::]:* users:(("beam.smp",pid=8525,fd=73))
4369 — erlang discovery port
5672 –Program connection port
15672 — Management interface ui port
25672 — Internal communication port between servers
Visit:
10.36.192.150:15672
rabbitmq default administrator user: guest password: guest
The newly added user is: newrain password: 123456
2. Deploy cluster (three machines)
Repeat the above steps first, then operate the remaining two machines as above and the test is completed!
Then perform the following operations on the three servers:
1. Create and configure the data storage directory and log storage directory
#Create data and log directories: [root@rabbitmq-1 ~]# mkdir -p /data/rabbitmq/data [root@rabbitmq-1 ~]# mkdir -p /data/rabbitmq/logs #Choose one of the following two options: [root@rabbitmq-1 ~]# chmod 777 -R /data/rabbitmq [root@rabbitmq-1 ~]# chown rabbitmq.rabbitmq /data/ -R #Create configuration file: [root@rabbitmq-1 ~]# vim /etc/rabbitmq/rabbitmq-env.conf #Add the following information: RABBITMQ_MNESIA_BASE=/data/rabbitmq/data RABBITMQ_LOG_BASE=/data/rabbitmq/logs Restart service [root@rabbitmq-1 ~]# systemctl restart rabbitmq-server
2.Copy?erlang.cookie
erlang.cookie generally has these two addresses:
The first one is home/.erlang.cookie (the location where installation packages such as rpm are installed)
The second place is /var/lib/rabbitmq/.erlang.cookie (the location where the decompression method is installed and deployed)
[root@rabbitmq-1 ~]# cat /var/lib/rabbitmq/.erlang.cookie HOUCUGJDZYTFZDSWXTHJ #?Scp's formula copies the value of .erlang.cookie of rabbitmq-1 node to the other two nodes [root@rabbitmq-1 ~]# scp /var/lib/rabbitmq/.erlang.cookie [email protected]:/var/lib/rabbitmq/ [root@rabbitmq-1 ~]# scp /var/lib/rabbitmq/.erlang.cookie [email protected]:/var/lib/rabbitmq/
3. Add mq-2 and mq-3 as memory nodes to the mq-1 node cluster
#Execute the following commands in mq-2 and mq-3: [root@rabbitmq-2 ~]# systemctl restart rabbitmq-server #Stop node [root@rabbitmq-2 ~]# rabbitmqctl stop_app #If there is data that needs to be reset, if not, don’t use it. [root@rabbitmq-2 ~]# rabbitmqctl reset #Add to disk node [root@rabbitmq-2 ~]# rabbitmqctl join_cluster --ram rabbit@rabbitmq-1 Clustering node 'rabbit@rabbitmq-2' with 'rabbit@rabbitmq-1' ... #Start node [root@rabbitmq-2 ~]# rabbitmqctl start_app Starting node 'rabbit@rabbitmq-2' ... [root@rabbitmq-3 ~]# systemctl restart rabbitmq-server [root@rabbitmq-3 ~]# rabbitmqctl stop_app [root@rabbitmq-3 ~]# rabbitmqctl reset [root@rabbitmq-3 ~]# rabbitmqctl join_cluster --ram rabbit@rabbitmq-1 [root@rabbitmq-3 ~]# rabbitmqctl start_app #If you need to use disk nodes to join the cluster [root@rabbitmq-2 ~]# rabbitmqctl join_cluster rabbit@rabbitmq-1 [root@rabbitmq-3 ~]# rabbitmqctl join_cluster rabbit@rabbitmq-1
Note:
(1) By default, rabbitmq is a disk node after startup. Under this cluster command, mq-2 and mq-3 are memory nodes.
mq-1 is the disk node.
(2) If you want to make mq-2 and mq-3 both disk nodes, remove the –ram parameter.
(3) If you want to change the node type, you can use the command rabbitmqctl change_cluster_node_type
disc(ram), the premise is that the rabbit application must be stopped?
4. Check cluster status
Execute rabbitmqctl cluster_status on any node of the RabbitMQ cluster to check whether the cluster configuration is successful.
[root@rabbitmq-1 ~]# rabbitmqctl cluster_status
Each machine displays three nodes, indicating that it has been added successfully!
(1) By default, rabbitmq is a disk node after startup. Under this cluster command, mq-2 and mq-3 are memory nodes, and mq-1 is a disk node.
(2) If you want to make mq-2 and mq-3 both disk nodes, remove the –ram parameter.
(3) If you want to change the node type, you can use the command rabbitmqctl change_cluster_node_type disc(ram), provided that it must be stopped.
5. Log in to the rabbitmq web management console and create a new queue
Open the browser and enter?http://10.36.192.150:15672
Enter the default Username: guest
Enter the default Password: guest
Create a queue according to the interface prompts
After successful creation, you can see it in Queues
3. RabbitMQ mirror cluster configuration
What has been completed above is the default cluster mode of RabbitMQ, which does not guarantee the high availability of the queue, and the downtime of the queue node will directly cause the queue to be unavailable and can only wait for restart. To solve this problem, it is necessary to copy the queue content to the cluster. For each node, a mirror queue must be created. The mirror queue is based on the ordinary cluster mode, and some policies are added, so you still have to configure the ordinary cluster first, and then you can set up the mirror queue. We will continue with the above cluster. Ensure data synchronization between various nodes.
1.Create a mirror cluster
Granted permission:
rabbitmqctl set_permissions “.*” “.*” “.*” # The last three “*” represent that the user has full permissions to configure, write, and read
[root@rabbitmq-1 ~]# rabbitmqctl set_policy ha-all "^" '{"ha-mode":"all"}’
[root@rabbitmq-2 ~]# rabbitmqctl set_policy ha-all "^" '{"ha-mode":"all"}'
[root@rabbitmq-3 ~]# rabbitmqctl set_policy ha-all "^" '{"ha-mode":"all"}'
#After successful creation, the following information will be displayed
Setting policy "ha-all" for pattern "^" to "{"ha-mode":"all"}" with priority "0" for vhost "/" ...
Check that the queue has been synchronized to the other two nodes:
“-p Vhost”: vhost name, “^” matches all queues, ha-all policy name is ha-all, ‘{“ha-mode”:”all”}’ If the policy mode is all, that is, it is copied to all nodes, including new nodes, then the mirror queue is set successfully.
Mirror queue policy setting instructions:
rabbitmqctl set_policy [-p Vhost] Name Pattern Definition [Priority]
-p Vhost: Optional parameter, set for the queue under the specified vhost
Name: name of the policy
Pattern: Queue matching pattern (regular expression)
Definition: Image definition, including three parts ha-mode, ha-params, ha-sync-mode
ha-mode: Specifies the mode of the mirror queue. The valid value is all/exactly/nodes
all: means mirroring on all nodes in the cluster
exactly: means mirroring on the specified number of nodes. The number of nodes is specified by ha-params.
nodes: means mirroring on the specified node, the node name is specified through ha-params
ha-params: parameters needed for ha-mode mode
ha-sync-mode: synchronization mode of messages in the queue, valid values are automatic and manual
priority: optional parameter, priority of policy
2. Web page experiment
Create a vhost and add queues to user permissions:
First, in the web interface, after logging in, click “Admin–Virtual Hosts (right side of the page)”, add a virtual host coresystem at the bottom of the opened page at “Add a new virtual host”, and create the user “admin” ” and “guest” are both added with permissions (just set it directly on the page and click)
Create user:
The user creation is completed.
3. Add a queue to the server
Add a new queue and execute the following command on the first server;
[root@rabbitmq-1 ~]# rabbitmqctl set_policy -p coresystem ha-all "^" '{"ha-mode":"all"}'
Set all queues as mirror queues, that is, the queues will be copied to each node, and the status of each node will remain consistent. After completing these 6 steps, the RabbitMQ high-availability cluster is completed, and the last step is to build the balancer.
4. Install and configure load balancer HA
Note: If you use Alibaba Cloud, you can use Alibaba Cloud’s intranet slb to achieve load balancing without setting up HA yourself.
Install and configure load balancer HA
Install HAProxy on 10.36.192.150
[root@rabbitmq-1 ~]# yum -y install haproxy
Modify /etc/haproxy/haproxy.cfg
[root@rabbitmq-1 ~]# cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak
[root@rabbitmq-1 ~]# vim /etc/haproxy/haproxy.cfg
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
nbproc 4
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#------------------------------------------------ -------------------
defaults
mode http
log global
retries 3
timeout connect 10s
timeout client 1m
timeout server 1m
timeout check 10s
maxconn 2048
#------------------------------------------------ -------------------
##Monitor and view local status#####
listenadmin_stats
bind *:80
mode http
option httplog
option httpclose
log 127.0.0.1 local0 err
stats uri /haproxy
stats auth newrain:123456
stats refresh 30s
####################################
###Anti-generation monitoring
frontend server
bind *:5670
log global
mode tcp
#option forwardfor
default_backend rabbitmq
maxconn 3
backend rabbitmq
mode tcp
log global
balance roundrobin
server rabbitmq1 10.36.192.150:5672 check inter 2000s rise 2 fall 3
server rabbitmq2 10.36.192.151:5672 check inter 2000s rise 2 fall 3
server rabbitmq3 10.36.192.152:5672 check inter 2000s rise 2 fall 3
[root@rabbitmq-1 ~]# systemctl start haproxy #Start haproxy
Enter 10.36.192.150/haproxy in the browser to check the status of rabbitmq.
5. Install MySQL database
For installation, please refer to: http://t.csdnimg.cn/1zslv (yum installation) http://t.csdnimg.cn/spizv (source code installation)
Log in to the database to perform authorized remote login operations:
mysql> update mysql.user set host = '%' where user = 'root'; Query OK, 1 row affected (0.00 sec) Rows matched: 1 Changed: 1 Warnings: 0 mysql> flush privileges; Query OK, 0 rows affected (0.00 sec) mysql> ^DBye
6. Install python and test
git test package and install python software:
[root@rabbitmq-1 ~]# yum -y install git [root@rabbitmq-1 ~]# git clone https://gitea.beyourself.org.cn/newrain001/rabbitmq-test.git & amp; & amp; \ cd rabbitmq-test & amp; & amp; yum install -y python3 python3-devel & amp; & amp; \ pip3 install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple
Modify rabbitmq-test related files, namely mysql database password and port:
[root@rabbitmq-1 ~]# vim rabbitmq-test/settings.py # rabbitmq configuration MQName="message_queue" MQUser="guest" MQPassword="guest" MQHost="localhost" MQPort=5670 # mysql database configuration DBHost="localhost" DBUser="root" DBPassword="Mawenbao@123"
Set up the environment and start the flask service:
[root@rabbitmq-1 ~]# export FLASK_ENV=development ; flask run --reload -p 80 -h 0.0.0.0
Enter 10.36.192.150 in the browser to view the order page.
Enter 10.36.192.150/read in the browser to view the shipping page.
Changes when placing an order:
Data changes at the time of shipment:
7. Common mistakes
1. When using rabbitmq-server -detached command to start rabbitmq, the following prompt appears: Warning: PID file not written; -detached was passed. At this time, use rabbitmqctl status to prompt that the service has been started. It can be seen that this problem does not need to be solved.
2. Due to the change of the hostname file, every time rabbitmqctl stop or rabbitmqctl cluster_status, etc., an error is reported as long as it is a rabbitmq command. The prompt is as follows
Cluster status of node rabbit@web2 ... Error: unable to connect to node rabbit@web2: nodedown DIAGNOSTICS =========== attempted to contact: [rabbit@web2] rabbit@web2: * connected to epmd (port 4369) on web2 * epmd reports node 'rabbit' running on port 25672 * TCP connection succeeded but Erlang distribution failed * Hostname mismatch: node "rabbit@mq2" believes its host is different. Please ensure that hostnames resolve the same way locally and on "rabbit@mq2" current node details: - node name: 'rabbitmq-cli-11@web2' - home dir: /root - cookie hash: SGwxMdJ3PjEXG1asIEFpBg==
At this time, first ps aux | grep mq, then kill -9 the process, and then rabbitmq-server -detached to solve the problem. (That is, forcefully kill the process first and then restart it)
3. After restarting using rabbitmqctl stop and rabbitmq-server -detached, the originally added user admin, virtual host coresystem, etc. will be lost and need to be added again.
4. Use a script to start, and write in the script all the configuration items that need to be loaded during startup (create an admin user and authorize it, create a virtual host and authorize it, and configure the mirror queue).
5. RabbitMQ error handling
Error: returned an error: shutdown: failed to start child: Logger.ErrorHandler
Reason: The erlang version does not correspond to the RabbitMQ version
Solution: Check the version according to RabbitMQ Erlang Version Requirements https://www.rabbitmq.com/which-erlang.html
6. Port conflict problem
The corresponding conflicting ports need to be modified, usually in the configuration file.