Directory
1. The concept of serialization and deserialization
2. Design a network calculator with a custom protocol
2.1 TCP protocol, how to ensure that the receiver has received the complete message?
2.2 Implementation of custom protocol
2.3 Implementation of custom protocol in client and server
3. Use Json for serialization and deserialization
3.1 Installation of jsoncpp library
3.2 Transform custom protocol
3.3 Naming distinction of custom protocols
The online version of the calculator code can refer to the blogger gitee.
1. The concept of serialization and deserialization
Serialization refers to converting an object into a byte stream or other storable or transportable format, so that it can be stored in a file or sent over a network to another system. Deserialization refers to converting serialized data back into objects. During the serialization and deserialization process, the state information of the object is saved and restored to ensure the integrity and correctness of the data. In distributed systems, serialization and deserialization are important technologies for remote method invocation and message delivery.
For example, when WeChat sends a message, it will serialize “structured” data such as avatar, nickname, message content, and sending time to form a byte stream message, and send the message to the receiver through the network, and the receiver will respond. Serialization re-disassembles the message into “structured” data such as avatar, nickname, message content, and sending time.
2. Design a network calculator with a custom protocol
2.1TCP protocol, how to ensure that the receiver receives the complete message?
1. All the send/receive functions we call do not directly send/receive data from the network. The send/receive function called by the application layer is essentially a copy function.
For example, when the client sends data, the application layer calls the send function to copy the send buffer data of the application layer to the send buffer of the transport layer. The transport layer independently decides when to send the data in the sending buffer to the network, and then sends it to the receiving buffer of the server through the network, so the TCP protocol is a transmission control protocol.2. The sending buffer and receiving buffer of both sides of the communication of the TCP protocol do not interfere with each other, and can communicate in both directions at the same time. TCP is a full-duplex communication protocol.
3. If the reading speed of the TCP server cannot keep up with the sending speed of the client, it will cause the server to accumulate a large number of messages in the receiving buffer. These message data are glued together in a series. What about text extraction? Use Agreement! Protocol design method:
- Fixed length (for example, the fixed length of the message is specified as 1024 bytes)
- Special symbols (add special symbols between messages)
- Self-describing method (design protocol yourself)
The code protocol design of this article is shown in the figure below: use this protocol to design a network calculator.
If it is a UDP protocol, the UDP client only needs to create a request when sending a message, serialize the request and then send it; when receiving a message, it only needs to deserialize the received data. There is no need to add and analyze the protocol content. This is because UDP sends and receives every time in the form of datagrams, and the data is complete. Unlike TCP, which is byte-oriented, it is necessary to use related protocols to define message boundaries.
2.2 Implementation of custom protocol
#pragma once
#include <iostream>
#include <string>
#include <vector>
#include <cstring>
enum
{
OK=0,
DIV_ZERO_ERR,
MOD_ZERO_ERR,
OP_ZERO_ERR,
};
#define SEP " "
#define SEP_LEN strlen(SEP)//Cannot use sizeof, use sizeof will count '\0'
#define LINE_SEP "\r\
"
#define LINE_SEP_LINE strlen(LINE_SEP)
//"_exitcode result" -> "content_len"\r\
"_exitcode result"\r\
//"_x _op _y" -> "content_len"\r\
"_x _op _y"\r\
std::string enLength(const std::string & text)//text:_x _op _y. Add protocol rules to build a complete message (similar to "packaging")
{
std::string send_string=std::to_string(text.size());//Calculate the length of the payload "_x _op _y"
send_string + = LINE_SEP;
send_string + =text;
send_string + = LINE_SEP;
return send_string;
}
//_exitcode result
bool deLength(const std::string & amp; package,std::string* text)//Get the payload in the message (similar to "unpacking")
{
auto pos=package.find(LINE_SEP);
if(pos==std::string::npos){return false;}
int textLen=std::stoi(package.substr(0,pos));//calculate the length of the payload
*text=package.substr(pos + LINE_SEP_LINE,textLen);
return true;
}
class Request//request class
{
public:
Request(int x, int y, char op)
:_x(x)
,_y(y)
,_op(op)
{}
Request()
:_x(0)
,_y(0)
,_op(0)
{}
bool serialize(std::string* out)//Serialize, convert member variable to string
{
// structured -> "_x _op _y"
*out="";//clear the string object
std::string x_tostring = std::to_string(_x);
std::string y_tostring = std::to_string(_y);
*out=x_tostring + SEP + _op + SEP + y_tostring;//_x _op _y
return true;
}
bool deserialize(const std::string & amp; in)//deserialize
{
//"_x _op _y" -> structured
auto leftSpace=in.find(SEP);//Space on the left
auto rightSpace=in.rfind(SEP);//Space on the right
if(leftSpace==std::string::npos||rightSpace==std::string::npos){return false;}
if(leftSpace==rightSpace){return false;}
//substring extraction
std::string x_tostring=in.substr(0,leftSpace);
if(rightSpace-(leftSpace + SEP_LEN)!=1){return false;}//Indicates that the operator must only occupy 1 position
_op=in.substr(leftSpace + SEP_LEN,rightSpace-(leftSpace + SEP_LEN))[0];
std::string y_tostring=in.substr(rightSpace + SEP_LEN);
// convert x, y
_x=std::stoi(x_tostring);
_y=std::stoi(y_tostring);
return true;
}
public:
//_x _op _y
int _x;//left operand
int _y;//right operand
char _op;//operator
};
class Response//response class
{
public:
Response()
:_exitCode(0)
,_result(0)
{}
Response(int exitCode,int result)
:_exitCode(exitCode)
,_result(result)
{}
bool serialize(std::string* out)//Serialize, convert member variable to string object
{
*out="";//clear the string object
std::string outString=std::to_string(_exitCode) + SEP + std::to_string(_result);
*out=outString;
return true;
}
bool deserialize(const std::string & amp; in)//deserialize
{
auto space=in.find(SEP);//Find spaces
if(space==std::string::npos){return false;}
std::string exitString=in.substr(0,space);
std::string resString = in. substr(space + SEP_LEN);
if(exitString.empty()||resString.empty()){return false;}//If a string is empty, it will be false
_exitCode=std::stoi(exitString);
_result=std::stoi(resString);
return true;
}
public:
int _exitCode;//0 means the calculation is successful, non-zero means division by zero and other errors
int _result;//operation result
};
bool recvPackage(int sock,std::string & amp; inbuffer,std::string* text)//server/client read message
{
//Split the buffer data into individual messages "content_len"\r\
"_x _op _y"\r\
char buffer[1024];
while(1)
{
ssize_t n=recv(sock,buffer,sizeof(buffer)-1,0);//blocking read, equivalent to read interface
if(n>0)
{
buffer[n]=0;//Add '\0' at the end of the string
inbuffer + =buffer;
//Split into a message
auto pos=inbuffer.find(LINE_SEP);//Find the starting position of \r\
if(pos==std::string::npos)//Not found means that the \r\
delimiter has not been found yet, skip this cycle and wait for the next reading
{
continue;
}
std::string textLenString=inbuffer.substr(0,pos);
int textLen=std::stoi(textLenString);//take out the length of the payload
int totalLen=textLenString.size() + 2*LINE_SEP_LINE + textLen;//total length of a single message
if(inbuffer.size()<totalLen)//Indicates that the length of the buffer is less than the size of a message, and it is necessary to skip this cycle and continue reading
{
continue;
}
std::cout<<"Intercept the contents of the inbuffer before the message:\
"<<inbuffer<<std::endl;
//Come here, there must be a complete message
*text=inbuffer.substr(0,totalLen);//take out a message
inbuffer.erase(0,totalLen);//Delete the message data just extracted from the buffer
std::cout<<"The content in the inbuffer after intercepting the message:\
"<<inbuffer<<std::endl;
break;
}
else
{
return false;
}
}
return true;
}
Code explanation:
This protocol designs a Request request class and a Response response class. The request class stores the operands and operators of the calculator, and the response class stores calculation results and exit codes. Each of these two classes implements the interface of serialization and deserialization of payload.
enLength is used to add a header to the payload, that is, the rules of the custom protocol. deLength is used to parse the received message, remove the protocol header in the message, and extract the payload. The protocol is a special format that defines the boundary between packets in the buffer, and enLength and deLength are used to add headers and remove headers.
2.3 Implementation of custom protocol in client and server
3. Use Json for serialization and deserialization
3.1 Installation of jsoncpp library
As you can see from the above code, it is very troublesome to use string objects to serialize and deserialize manually. Json can be used for serialization and deserialization operations.
Json (JavaScript Object Notation) is a lightweight data exchange format that is often used for data transmission in web applications. It is a text-based format that is easy to read, write and parse. Data in Json format can be supported by a variety of programming languages, including JavaScript, Python, Java, C#, C++, etc. Json data consists of key-value pairs, curly brackets are used to represent objects, and square brackets are used to represent arrays.
C ++ use Json needs to include the Jsoncpp library, yum installs the Jsoncpp library command: execute the second sentence first, and execute the first sentence if an error is reported!
sudo mv /var/lib/rpm/__db.00* /tmp/ & amp; & amp; yum clean all sudo yum install -y jsoncpp-devel
3.2 Retrofit custom protocol
makefile: Use the jsoncpp library and remember to add -ljsoncpp when compiling
cc=g ++ #Set cc variable to g ++ compiler
LD=#-DMYSELF
.PHONY: all
all:calClient calServer
calClient: calClient.cc
$(cc) -o $@ $^ -std=c++11 -ljsoncpp ${LD}
calServer:calServer.cc
$(cc) -o $@ $^ -std=c++11 -ljsoncpp ${LD}
\t
.PHONY:clean
clean:
rm -f calClient calServer
Modified protocol:
#pragma once
#include <iostream>
#include <string>
#include <vector>
#include <cstring>
#include <jsoncpp/json/json.h>
#include <sys/types.h>
#include <sys/socket.h>
enum
{
OK=0,
DIV_ZERO_ERR,
MOD_ZERO_ERR,
OP_ZERO_ERR,
};
#define SEP " "
#define SEP_LEN strlen(SEP)//Cannot use sizeof, use sizeof will count '\0'
#define LINE_SEP "\r\
"
#define LINE_SEP_LINE strlen(LINE_SEP)
//"_exitcode result" -> "content_len"\r\
"_exitcode result"\r\
//"_x _op _y" -> "content_len"\r\
"_x _op _y"\r\
std::string enLength(const std::string & text)//text:_x _op _y. Add protocol rules to build a complete message (similar to "packaging")
{
std::string send_string=std::to_string(text.size());//Calculate the length of the payload "_x _op _y"
send_string + = LINE_SEP;
send_string + =text;
send_string + = LINE_SEP;
return send_string;
}
//_exitcode result
bool deLength(const std::string & amp; package,std::string* text)//Get the payload in the message (similar to "unpacking")
{
auto pos=package.find(LINE_SEP);
if(pos==std::string::npos){return false;}
int textLen=std::stoi(package.substr(0,pos));//calculate the length of the payload
*text=package.substr(pos + LINE_SEP_LINE,textLen);
return true;
}
class Request//request class
{
public:
Request(int x, int y, char op)
:_x(x)
,_y(y)
,_op(op)
{}
Request()
:_x(0)
,_y(0)
,_op(0)
{}
bool serialize(std::string* out)//Serialize, convert member variable to string
{
#ifdef MYSELF
// structured -> "_x _op _y"
*out="";//clear the string object
std::string x_tostring = std::to_string(_x);
std::string y_tostring = std::to_string(_y);
*out=x_tostring + SEP + _op + SEP + y_tostring;//_x _op _y
#else
//Json serialization
Json::Value root;//Json::Value universal object, can receive any object
root["first"]=_x;//Automatically convert _x to a string
root["second"]=_y;
root["oper"]=_op;
//Serialization
Json::FastWriter writer;//Json::StyledWriter write; equivalent
*out=writer.write(root);//Serialize root, the return value is a string object, just receive it
#endif
return true;
}
bool deserialize(const std::string & amp; in)//deserialize
{
#ifdef MYSELF
//"_x _op _y" -> structured
auto leftSpace=in.find(SEP);//Space on the left
auto rightSpace=in.rfind(SEP);//Space on the right
if(leftSpace==std::string::npos||rightSpace==std::string::npos){return false;}
if(leftSpace==rightSpace){return false;}
//substring extraction
std::string x_tostring=in.substr(0,leftSpace);
if(rightSpace-(leftSpace + SEP_LEN)!=1){return false;}//Indicates that the operator must only occupy 1 position
_op=in.substr(leftSpace + SEP_LEN,rightSpace-(leftSpace + SEP_LEN))[0];
std::string y_tostring=in.substr(rightSpace + SEP_LEN);
// convert x, y
_x=std::stoi(x_tostring);
_y=std::stoi(y_tostring);
#else
//Json deserialization
Json::Value root;//Json::Value universal object, can receive any object
Json::Reader reader;
reader.parse(in,root);//The first parameter: which stream to parse; the second parameter: store the parsed data into the object
// deserialize
_x=root["first"].asInt();//The default is a string, converted to an integer
_y=root["second"].asInt();
_op=root["oper"].asInt();//Convert to integer type, integer type can be given to char type.
#endif
return true;
}
public:
//_x _op _y
int _x;//left operand
int _y;//right operand
char _op;//operator
};
class Response//response class
{
public:
Response()
:_exitCode(0)
,_result(0)
{}
Response(int exitCode,int result)
:_exitCode(exitCode)
,_result(result)
{}
bool serialize(std::string* out)//Serialize, convert member variable to string object
{
#ifdef MYSELF
*out="";//clear the string object
std::string outString=std::to_string(_exitCode) + SEP + std::to_string(_result);
*out=outString;
#else
//Json serialization
Json::Value root;//Json::Value universal object, can receive any object
root["exitCode"]=_exitCode;//Automatically convert _exitCode to a string
root["result"]=_result;
//Serialization
Json::FastWriter writer;//Json::StyledWriter write; equivalent
*out=writer.write(root);//Serialize root, the return value is a string object, just receive it
#endif
return true;
}
bool deserialize(const std::string & amp; in)//deserialize
{
#ifdef MYSELF
auto space=in.find(SEP);//Find spaces
if(space==std::string::npos){return false;}
std::string exitString=in.substr(0,space);
std::string resString = in. substr(space + SEP_LEN);
if(exitString.empty()||resString.empty()){return false;}//If a string is empty, it will be false
_exitCode=std::stoi(exitString);
_result=std::stoi(resString);
#else
//Json deserialization
Json::Value root;//Json::Value universal object, can receive any object
Json::Reader reader;
reader.parse(in,root);//The first parameter: which stream to parse; the second parameter: store the parsed data into the object
// deserialize
_exitCode=root["exitCode"].asInt();//The default is a string, converted to an integer
_result=root["result"].asInt();
#endif
return true;
}
public:
int _exitCode;//0 means the calculation is successful, non-zero means division by zero and other errors
int _result;//operation result
};
bool recvPackage(int sock,std::string & amp; inbuffer,std::string* text)//server/client read message
{
//Split the buffer data into individual messages "content_len"\r\
"_x _op _y"\r\
char buffer[1024];
while(1)
{
ssize_t n=recv(sock,buffer,sizeof(buffer)-1,0);//blocking read, equivalent to read interface
if(n>0)
{
buffer[n]=0;//Add '\0' at the end of the string
inbuffer + =buffer;
//Split into a message
auto pos=inbuffer.find(LINE_SEP);//Find the starting position of \r\
if(pos==std::string::npos)//Not found means that the \r\
delimiter has not been found yet, skip this cycle and wait for the next reading
{
continue;
}
std::string textLenString=inbuffer.substr(0,pos);
int textLen=std::stoi(textLenString);//take out the length of the payload
int totalLen=textLenString.size() + 2*LINE_SEP_LINE + textLen;//total length of a single message
if(inbuffer.size()<totalLen)//Indicates that the length of the buffer is less than the size of a message, and it is necessary to skip this cycle and continue reading
{
continue;
}
std::cout<<"Intercept the contents of the inbuffer before the message:\
"<<inbuffer<<std::endl;
//Come here, there must be a complete message
*text=inbuffer.substr(0,totalLen);//take out a message
inbuffer.erase(0,totalLen);//Delete the message data just extracted from the buffer
std::cout<<"The content in the inbuffer after intercepting the message:\
"<<inbuffer<<std::endl;
break;
}
else
{
return false;
}
}
return true;
}
bool recvPackageAll(int sock, std::string & inbuffer, std::vector<std::string>* out)
{
std::string line;
while(recvPackage(sock,inbuffer, &line))
{
out->push_back(line);
}
}
3.3 Naming distinction of custom protocols
In the future, multiple protocols can be customized in one system. In order to distinguish different custom protocols, the format of the protocol can be designed with reference to the following format:
Of course, the predecessors have already designed common network protocols, such as http/https.
The knowledge points of the article match the official knowledge files, and you can further learn relevant knowledge Network skill tree Protocols supporting applicationsThe role of the application layer 33637 people are studying systematically