A complete collection of Dubbo network communication protocols from shallow to deep

1 Network communication protocol
- 1.1 dubbo protocol
- 1.2 rmi agreement
- 1.3 hessian protocol
- 1.4 http protocol
- 1.5 webservice protocol
- 1.6 thrift protocol
- 1.7 rest protocol
- 1.8 grpc protocol
- 1.9 memcached protocol
- 1.10 redis protocol
2 Serialization implementation analysis

1 Network communication protocol

In the previous content, we explained the relevant content of consumer-side service discovery and provider-side service exposure. At the same time, we also know that the consumer side obtains the appropriate invoker for remote calls through the built-in load balancing algorithm. Then, this chapter focuses on the remote call process, that is, network communication.

Serialization is to convert an object into a byte stream

Network communication is located in the Remoting module:

The Remoting implementation is the implementation of the Dubbo protocol. If you choose the RMI protocol, the entire Remoting will not be used;
Remoting is further divided into Transport transport layer and Exchange information exchange layer;
The Transport layer is only responsible for one-way message transmission, which is an abstraction of Mina, Netty, Grizzly, and it can also extend UDP transmission;
The Exchange layer encapsulates the Request-Response semantics above the transport layer;

Internet communication problems:

Client and server connectivity issues
Sticky package unpacking problem
Asynchronous multi-thread data consistency problem

Communication protocol

Dubbo built-in, dubbo protocol, rmi protocol, hessian protocol, http protocol, webservice protocol, thrift protocol, rest protocol, grpc protocol, memcached protocol, redis protocol and other 10 communication protocols. The characteristics of each agreement are as follows

1.1 dubbo protocol

Dubbo’s default protocol uses a single long connection and NIO asynchronous communication, which is suitable for small data volume and large concurrent service calls, and the situation where the number of service consumer machines is much larger than the number of service provider machines.

Default protocol, using tbremoting interaction based on mina 1.1.7 and hessian 3.2.1.

Number of connections: single connection
Connection mode: long connection
Transport protocol: TCP
Transmission method: NIO asynchronous transmission
Serialization: Hessian binary serialization
Scope of application: The incoming and outgoing parameter data packets are small (recommended less than 100K), the number of consumers is more than that of the provider, and a single consumer cannot fill the provider. Try not to use the dubbo protocol to transfer large files or very large strings.
Applicable scenario: regular remote service method call

1.2 rmi protocol

The RMI protocol is implemented using JDK standard java.rmi.*, using blocking short connections and JDK standard serialization.

Number of connections: multiple connections
Connection method: short connection
Transport protocol: TCP
Transmission method: synchronous transmission
Serialization: Java Standard Binary Serialization
Scope of application: The size of incoming and outgoing parameter data packets is mixed, the number of consumers and providers is similar, and files can be transferred.
Applicable scenarios: regular remote service method calls, interoperability with native RMI services

1.3 Hessian protocol

The Hessian protocol is used to integrate Hessian’s services. The bottom layer of Hessian uses Http communication and Servlet to expose services. Dubbo’s default embedded Jetty is implemented as a server.

Dubbo’s Hessian protocol can interoperate with native Hessian services, namely:

The provider uses Dubbo’s Hessian protocol to expose the service, and the consumer directly uses the standard Hessian interface to call
Or the provider uses the standard Hessian to expose the service, and the consumer uses Dubbo’s Hessian protocol to call.
Number of connections: multiple connections
Connection method: short connection
Transport protocol: HTTP
Transmission method: synchronous transmission
Serialization: Hessian binary serialization
Scope of application: The incoming and outgoing parameter data packets are large, the number of providers is larger than that of consumers, the pressure on providers is high, and files can be transferred.
Applicable scenarios: page transfer, file transfer, or interoperability with native Hessian services

1.4 http protocol

Remote invocation protocol based on HTTP form, implemented by Spring’s HttpInvoker

Number of connections: multiple connections
Connection method: short connection
Transport protocol: HTTP
Transmission method: synchronous transmission
Serialization: form serialization
Scope of application: Incoming and outgoing parameters are mixed in packet size, there are more providers than consumers, you can view them with a browser, you can use forms or URLs to pass in parameters, and file transfers are not currently supported.
Applicable scenarios: services that need to be used by both application and browser JS.

1.5 webservice protocol

WebService-based remote call protocol, implemented based on Apache CXF](http://dubbo.apache.org/zh-cn/docs/user/references/protocol/webservice.html#fn2).

It can interoperate with native WebService services, namely:

The provider uses Dubbo’s WebService protocol to expose the service, and the consumer directly uses the standard WebService interface to call,
Or the provider uses the standard WebService to expose the service, and the consumer uses Dubbo’s WebService protocol to call.
Number of connections: multiple connections
Connection method: short connection
Transport protocol: HTTP
Transmission method: synchronous transmission
Serialization: SOAP text serialization (http + xml)
Applicable scenarios: system integration, cross-language calls

1.6 thrift protocol

The thrift protocol currently supported by dubbo [1] is an extension of the native thrift protocol [2], adding some additional header information on the basis of the native protocol, such as service name, magic number, etc.

1.7 rest protocol

REST call support implemented based on the standard Java REST API – JAX-RS 2.0 (short for Java API for RESTful Web Services)

1.8 grpc protocol

Dubbo has supported the gRPC protocol since version 2.7.5. For developers who plan to use HTTP/2 communication, or want to take advantage of the Stream, backpressure, and reactive programming capabilities brought by gRPC, they can consider enabling the gRPC protocol.

Bring service governance capabilities to users who expect to use the gRPC protocol, and facilitate access to the Dubbo system
Users can use Dubbo-style, interface-based programming style to define and use remote services

1.9 memcached protocol

RPC protocol implemented based on memcached

1.10 redis protocol

RPC protocol implemented based on Redis

2 Serialization Implementation Analysis

Serialization is to convert objects into byte streams for network transmission, and convert byte streams into objects for restoration into objects after receiving byte stream data. There are many advantages of serialization, such as better security, cross-platform, etc. We know that dubbo conducts network communication based on netty, and you can see related classes of Netty in the NettyClient.doOpen() method

bootstrap.setPipelineFactory(new ChannelPipelineFactory() {<!-- -->
    public ChannelPipeline getPipeline() {<!-- -->
        NettyCodecAdapter adapter = new NettyCodecAdapter(getCodec(), getUrl(), NettyClient. this);
        ChannelPipeline pipeline = Channels. pipeline();
        pipeline.addLast("decoder", adapter.getDecoder());
        pipeline.addLast("encoder", adapter.getEncoder());
        pipeline. addLast("handler", nettyHandler);
        return pipeline;
    }
});

Then go to the NettyCodecAdapter class and finally enter the encodeRequest method of the ExchangeCodec class, as follows:

 protected void encodeRequest(Channel channel, ChannelBuffer buffer, Request req) throws IOException {<!-- -->
        Serialization serialization = getSerialization(channel);
        // header.
        byte[] header = new byte[HEADER_LENGTH];

Yes, it is the Serialization interface, and the default is the Hessian2Serialization serialization interface.

Dubbo serialization supports java, compactedjava, nativejava, fastjson, dubbo, fst, hessian2, kryo, protostuff, among which hessian2 is the default. Among them, java, compactedjava, and nativejava belong to the serialization of native java.

Dubbo serialization: Ali has not yet developed a mature and efficient java serialization implementation, and Ali does not recommend using it in a production environment.
hessian2 serialization: Hessian is a cross-language efficient binary serialization method. But here is actually not the original hessian2 serialization, but modified by Ali, which is the serialization method enabled by dubbo RPC by default.
JSON serialization: There are currently two implementations, one is to use Ali’s fastjson library, and the other is to use the simple json library implemented by dubbo, but the implementation is not particularly mature, and the text serialization performance of json Generally not as good as the above two binary serialization.
Java serialization: It is mainly implemented by the Java serialization that comes with the JDK, and the performance is not ideal.

In recent years, various new efficient serialization methods have emerged one after another, constantly refreshing the upper limit of serialization performance, the most typical ones include:

Specifically for the Java language: Kryo, FST, etc.
Cross-language: Protostuff, ProtoBuf, Thrift, Avro, MsgPack, etc.

The performance of most of these serialization methods is significantly better than hessian2 (even including the immature dubbo serialization). So we can introduce Kryo and FST, two high-efficiency Java, for dubbo to optimize dubbo’s serialization.

Using Kryo and FST is very simple, just add an attribute in the XML configuration of dubbo RPC:

<dubbo:protocol name="dubbo" serialization="kryo"/>

Table of Contents