MQTT over QUIC – EMQX5&NanoMQ installation configuration

MQTT over QUIC – EMQX5 & amp;NanoMQ installation configuration

  • Bridge topology diagram (personal understanding)
  • Install and configure EMQX5 server
    • Install server
    • Configure server
  • Install and configure client NanoMQ
    • Install NanoMQ
    • Configure NanoMQ bridge
    • Start NanoMQ
    • Test bridging
      • Test message forwarding
      • Test message reception
  • Wireshark packet capture analysis

Bridge topology diagram (personal understanding)

Bridge topology diagram

Install and configure EMQX5 server

Install the server

Since it is an experimental feature, QUIC compilation is not included under CentOS 6, macOS and Windows systems. We follow the official website’s recommendation to compile from source code and specify the environment variable BUILD_WITH_QUIC=1 before compilation. We also use the recommended Docker environment.
Docker Desktop on Windows download page
docker desktop for windows
My docker is installed in Windows 11. After the installation is completed, as shown below
docker under win11Open cmd with the administrator and enter the following command to obtain the docker image

docker pull emqx/emqx:5.2.1

Run the following command to start the Docker container.

docker run -d --name emqx -p 1883:1883 -p 8083:8083 -p 8084:8084 -p 8883:8883 -p 18083:18083 -p 14567:14567/udp -e EMQX_LISTENERS__QUIC__DEFAULT__keyfile="etc/ certs/key.pem" -e EMQX_LISTENERS__QUIC__DEFAULT__certfile="etc/certs/cert.pem" -e EMQX_LISTENERS__QUIC__DEFAULT__ENABLED=true -e BUILD_WITH_QUIC=1 emqx/emqx:5.2.1

The following figure is obtained when running, but the Listener quic:default on 0.0.0.0:14567 started. is Listener quic:default on :14567 started. and needs to be modified as follows
docker startup

Configuration server

Find /opt/emqx/etx/emqx.conf in Files, right-click and select Edit file
Edit file emqx.conf
Paste the following into emqx.conf

# etc/emqx.conf
listeners.quic.default {<!-- -->
  enabled = true
  bind = "0.0.0.0:14567"
  max_connections = 1024000
  keyfile = "etc/certs/key.pem"
  certfile = "etc/certs/cert.pem"
}

Restart docker, QUIC will be started normally, and Listener quic:default on 0.0.0.0:14567 started. will appear normally.
Restart dockerAt this point, the server configuration is complete.

Install and configure client NanoMQ

NanoMQ also needs to manually turn on QUIC when compiling, and only supports bridge mode.

Install VMware 16
Install Ubuntu 22.04.3 desktop

Install NanoMQ

Open the terminal and enter the following command.

cd
git clone https://github.com/emqx/nanomq.git
cd nanomq
## Using the domestic network to pull MSQUIC submodule may take a long time
## If there is a network problem, you can consider ladders, agents, etc.
git submodule update --init --recursive
mkdir build
cd build
## By default, `msquic` is compiled as a dynamic library. If you need to set the compilation target as a static library, add the cmake compilation option `-DQUIC_BUILD_SHARED=OFF`
cmake -G Ninja -DNNG_ENABLE_QUIC=ON ..
sudo ninja install

What I downloaded is v0.20.0. If it is updated, please refer to the official location of github to download this version, as shown below.


Just unzip it after downloading.
In addition, if there is no .git folder downloaded here, an error will be reported when executing git submodule update --init --recursive. At this time, my operation is from git clone https://github.com/emqx/nanomq.gitCopy the higher version of the .git folder obtained by this command to the decompressed folder, and then execute it git submodule update --init --recursive.

Configuring NanoMQ bridge

After starting the QUIC module, you need to configure the MQTT over QUIC bridging function and corresponding topics in the nanomq/etc/nanomq.conf file. For example, in the following configuration file, the MQTT over QUIC bridging is defined server address, connection credentials, connection parameters, message forwarding rules, subscription topics, queue length, etc. Just add the following code to the end of the nanomq/etc/nanomq.conf file.

bridges.mqtt.name {<!-- -->
## TCP URL format: mqtt-tcp://host:port
## TLS URL format: tls + mqtt-tcp://host:port
## QUIC URL format: mqtt-quic://host:port
## My EMQX IP is 192.168.43.54, and the quic port number is 14567
server = "mqtt-quic://192.168.43.54:14567"
proto_ver = 4
username = dyx #username
password = dyx #Password
clean_start = true
keepalive=60s
## forwards -- forward the remote Topic array (supports MQTT wildcards)
forwards = [
{<!-- -->
remote_topic = "fwd/topic1"
local_topic = "topic1"
qos=0
},
{<!-- -->
remote_topic = "fwd/topic2"
local_topic = "topic2"
qos=0
}
]
quic_keepalive = 120s
quic_idle_timeout = 120s
quic_discon_timeout = 20s
quic_handshake_timeout = 60s
hybrid_bridging = false
## subscription -- Subscribe to the remote Topic array (supports MQTT wildcards)
subscription = [
{<!-- -->
remote_topic = "cmd/topic1"
local_topic = "topic3"
qos=0
},
{<!-- -->
remote_topic = "cmd/topic2"
local_topic = "topic4"
qos=0
}
]
max_parallel_processes = 2
max_send_queue_len = 1024
max_recv_queue_len = 1024
}

Start NanoMQ

Open a new terminal – Terminal A, go to /nanomq/build/nanomq and enter the following code

./nanomq start

If the following information appears, it means successful startup

NanoMQ Broker is started successfully!

Test bridging

NanoMQ (i.e. terminal A) does not close during testing

Test message forwarding

Subscribe to the forwarding topic “fwd/#” for EMQX to receive data forwarded by NanoMQ:
Open a new terminal – Terminal B, go to /nanomq/build/nanomq_cli and enter the following code

./nanomq_cli sub --quic -h "192.168.43.54" -t "fwd/#" -q 0

Note: Parameter meaning of ./nanomq_cli

## -h {host}
## -p {Port number, if not specified, the default port number 1883 (MQTT) or 14567 (QUIC) will be used}
## -t {topic name}
## --quic {turn on quic}
## -q {Message QoS, optional values 0, 1, 2}
## --m {message payload}
## -u {username}
## -P {password}

If you get the following output, it means that the quic connection is successful.

quic_connect_cb: mqtt-quic://192.168.43.54:14567 connect

Create a new terminal – Terminal C, publish messages to NanoMQ, the topic is “fwd/topic1”:

./nanomq_cli pub -h "192.168.233.132" -t "fwd/topic1" -m "123456" -q 0

192.168.233.132 is the IP of my NanoMQ host

Then terminal C outputs

connect_cb: mqtt-tcp://192.168.233.132:1883 connect result: 0
disconnected reason : 0

At the same time, terminal A outputs

2023-10-25 14:46:07 [7195] WARN /home/zxx/nanomq/nng/src/sp/transport/mqtt/broker_tcp.c:612 tcptran_pipe_recv_cb: nni aio recv error!! Connection shutdown

2023-10-25 14:46:07 [7195] WARN /home/zxx/nanomq/nng/src/sp/transport/mqtt/broker_tcp.c:880 tcptran_pipe_recv_cb: tcptran_pipe_recv_cb: parse error rv: 139

Apparently NanoMQ rejected the tcp connection, I don’t know the specific reason.

Test message reception

Subscribe to the topic “cmd/topic1” for NanoMQ to receive data published by EMQX:

In Terminal C, execute the following command to subscribe:

./nanomq_cli sub -h "192.168.233.132" -t "cmd/topic1" -q 0

The following output indicates successful subscription

connect_cb: mqtt-tcp://192.168.233.132:1883 connect result: 0

In terminal B, publish the message to the remote EMQX Broker with the topic “cmd/topic1”:

./nanomq_cli pub --quic -h "192.168.43.54" -t "cmd/topic1" -m "cmd_msg" -q 0 -u dyx -P dyx

But get the following output

quic_connect_cb: mqtt-quic://192.168.43.54:14567 connect
bridge client disconnected!

Don’t know why.

Wireshark packet capture analysis

In the test message forwarding, when entering ./nanomq_cli sub --quic -h "192.168.43.54" -t "fwd/#" -q 0, you will get a series of QUIC packets when capturing the packet. , TCP, HTTP and other messages, filter them, and get the data packets in the figure below, which can be used to analyze the QUIC connection process. However, if you want to analyze the data forwarding and subscription process, you need further configuration, but I have not yet succeeded in the specific operation. .
Packet capture analysis