1. Message middle key
1 Introduction
Message middleware can also be called message queue, which refers to the use of efficient and reliable message delivery mechanism for platform-independent data exchange, and the integration of distributed systems based on data communication. By providing message passing and message queuing models, process communication can be extended in a distributed environment.
The current mainstream message middleware includes RabbitMQ, Kafka, ActiveMQ, RocketMQ, etc.
2. Function
1. Main functions of message middleware
Redundancy (storage)
Scalability
recoverability
Order guaranteed
buffer
Asynchronous communication
2. Two modes of message middleware
1. P2P mode
The P2P model contains three roles: message queue (Queue), sender (Sender), and receiver (Receiver). Each message is sent to a specific queue and the receiver gets the message from the queue. The queue holds messages until they are consumed or time out.
Features of P2P:
Each message has only one consumer (Consumer), that is, once consumed, the message is no longer in the message queue.
There is no time dependency between the sender and the receiver. That is to say, after the sender sends the message, whether the receiver is running or not, it will not affect the message being sent to the queue.
After successfully receiving the message, the receiver needs to respond successfully to the queue.
If you want every message sent to be processed successfully, you need P2P mode
2. Pub/Sub mode
The Pub/Sub model includes three roles: Topic, Publisher, and Subscriber. Multiple publishers send messages to a Topic, and the system delivers these messages to multiple subscribers.
Pub/Sub features:
Each message can have multiple consumers
There is a time dependency between publishers and subscribers. For a subscriber of a certain topic (Topic), it must create a subscriber before it can consume the publisher’s messages.
In order to consume messages, the subscriber must remain running
If you want to send a message without any processing, or only be processed by one messager, or can be processed by multiple consumers, then you can use the Pub/Sub model
3. Introduction and comparison of commonly used middleware
1. Kafka
Kafka is LinkedIn’s open source distributed publish-subscribe messaging system and is currently a top-level Apache project. The main feature of Kafka is to handle message consumption based on the Pull mode and pursue high throughput. Its initial purpose is to use log collection and transmission. Version 0.8 starts to support replication, but does not support transactions. There are no strict requirements for message duplication, loss, and errors. It is suitable for data collection business of Internet services that generate large amounts of data.
2. RabbitMQ
RabbitMQ is an open source message queue system developed using the Erlang language and implemented based on the AMQP protocol. The main features of AMQP are message-oriented, queue-oriented, routing (including point-to-point and publish/subscribe), reliability, and security. The AMQP protocol is more commonly used in enterprise systems that require high data consistency, stability, and reliability. The requirements for performance and throughput are secondary.
3. RocketMQ
RocketMQ is Alibaba’s open source messaging middleware. It is developed purely in Java and has the characteristics of high throughput, high availability, and is suitable for large-scale distributed system applications. The RocketMQ idea originated from Kafka, but it is not a copy of Kafka. It optimizes the reliable transmission and transaction of messages. It is currently widely used in Alibaba Group for transactions, recharges, stream computing, message push, log streaming, Binlog distribution and other scenarios.
RabbitMQ is more reliable than Kafka, and Kafka is more suitable for high-throughput IO processing. It is generally used in big data log processing or scenarios with slightly lower requirements for real-time (a small amount of delay) and reliability (a small amount of data loss), such as ELK log collection.
RabbitMQ cluster
RabbiMQ was developed by Erlang, and clustering is very convenient because Erlang is designed to be a distributed language, but it does not support load balancing. Supports high concurrency and scalability. It supports AJAX and persistence, and is used to store and forward messages in distributed systems. It performs well in terms of ease of use, scalability, and high availability.
2. RabbitMQ features
reliability
Scalability
High availability
Multiple protocols
Multilingual client
Management interface
Plug-in mechanism
3. What is a message queue
MQ stands for Message Queue. Is an application-to-application communication method. Applications communicate by reading and writing messages (application-specific data) to and from queues without requiring a dedicated connection to link them.
Message passing means that programs communicate with each other by sending data in messages, rather than by calling each other directly.
RabbiMQ mode
Note: RabbitMQ mode can be roughly divided into the following three types:
(1) Single mode.
(2) Normal mode (default cluster mode).
(3) Mirror mode (make the required queue into a mirror queue and exist on multiple nodes. It is a RabbiMQ HA solution and is more suitable in situations where business reliability is required). To implement mirroring mode, you need to first build a common cluster mode, and then configure the mirroring mode based on this mode to achieve reliability.
Understand basic concepts in clustering:
RabbitMQ’s cluster nodes include memory nodes and disk nodes. As the name suggests, memory nodes store all data in memory, and disk nodes store data on disk. If the persistence of the message is turned on when delivering the message, the data is still safely placed on the disk even if it is a memory node.
A rabbitmq cluster can share user, vhost, queue, exchange, etc., and all data and status must be replicated on all nodes.
ConnectionFactory (Connection Manager): The manager that establishes a connection between the application and Rabbit, used in the program code; Channel: the channel used for message push; Exchange: used to accept and distribute messages; Queue: used to store messages from producers; RoutingKey: used to distribute the generator's data to the exchanger; BindingKey (binding key): used to bind messages from the exchanger to the queue; Broker: Simply put, it is the message queue server entity vhost: virtual host. Multiple vhosts can be opened in a broker to separate permissions for different users. producer: Message producer is the program that delivers messages. consumer: A message consumer is a program that accepts messages. user:user
Cluster environment preparation
Prepare three connected servers. The Rabbitmq cluster nodes must be in the same network segment. If it is across a wide area network, the effect will be worse. Turn off the firewall and selinux.
1. Preparation
1. Prepare three servers 192.168.146.133 rabbitmq-1 192.168.146.134 rabbitmq-2 192.168.146.135 rabbitmq-3 2. Local domain name resolution (operated on all three machines) [root@rabbitmq-1 ~]# vim /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.50.133 rabbitmq-1 192.168.50.134 rabbitmq-2 192.168.50.135 rabbitmq-3 3. Modify the host names of the three servers (must be consistent with the domain name resolution, otherwise an error will be reported!!!) hostnamectl set-hostname rabbitmq-1 hostnamectl set-hostname rabbitmq-2 hostnamectl set-hostname rabbitmq-3 The preparation work has been completed. Next, install the Rabbitmq software
2. Install rabbitmq software on three servers
Install dependencies [root@rabbitmq-1 ~]# yum install -y epel-release gcc-c + + unixODBC unixODBC-devel openssl-devel ncurses-devel yum install erlang [root@rabbitmq-1 ~]# curl -s https://packagecloud.io/install/repositories/rabbitmq/erlang/script.rpm.sh | sudo bash [root@rabbitmq-1 ~]# yum install erlang-21.3.8.21-1.el7.x86_64 test; [root@rabbitmq-1 ~]# erl Erlang/OTP 20 [erts-9.3] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:10] [hipe] [kernel-poll:false] Eshell V9.3 (abort with ^G) 1> Install rabbitmq https://github.com/rabbitmq/rabbitmq-server/releases/tag/v3.7.10 [root@rabbitmq-1 ~]# yum install rabbitmq-server-3.7.10-1.el7.noarch.rpm # rabbitmq and erlang compatible version https://www.rabbitmq.com/which-erlang.html #erlang version selection https://packagecloud.io/rabbitmq/erlang # rabbitmq version selection https://www.rabbitmq.com/news.html
3. Start
[root@rabbitmq-1 ~]# systemctl daemon-reload [root@rabbitmq-1 ~]# systemctl start rabbitmq-server [root@rabbitmq-1 ~]# systemctl enable rabbitmq-server
4. Create user
Note: Operate on one machine Add user and password [root@rabbitmq-1 ~]# rabbitmqctl add_user newrain 123456 Creating user "newrain" ... ...done. Set up administrator [root@rabbitmq-1 ~]# rabbitmqctl set_user_tags newrain administrator Setting tags for user "newrain" to [administrator] ... ...done. View users [root@rabbitmq-1 ~]# rabbitmqctl list_users Listing users... guest [administrator] newrain [administrator] ...done. When setting permissions here, pay attention to the need for spaces between '.*'. The three '.*' represent conf permissions, read permissions and write permissions respectively. For example: when no Before newrain set these three permissions, it did not have permission to query the queue, and it could not be seen in the UI interface. [root@rabbitmq-1 ~]# rabbitmqctl set_permissions -p "/" newrain ".*" ".*" ".*" Setting permissions for user "newrain" in vhost "/" ... ...done.
5. All machines operate: enable remote login
[root@rabbitmq-1 ~]# cd /etc/rabbitmq/ [root@rabbitmq-1 rabbitmq]# cp /usr/share/doc/rabbitmq-server-3.7.10/rabbitmq.config.example /etc/rabbitmq/rabbitmq.config [root@rabbitmq-1 rabbitmq]# ls enabled_plugins rabbitmq.config [root@rabbitmq-1 rabbitmq]# vim rabbitmq.config Modify as follows:
All three machines operate to restart the service: [root@rabbitmq-1 ~]# systemctl restart rabbitmq-server View port [root@rabbitmq-1 ~]# ss -lntp Omit 4369 -- erlang discovery port 5672 --Program connection port 15672 -- Management interface ui port 25672 -- Internal communication port between servers
6. Access the server
192.168.146.133:15672 192.168.146.134:15672 192.168.146.135:15672
Note here:** rabbitmq default administrator user: guest password: guest The newly added user is: newrain password: 123456