Network programming–overlapping IO model

Write in front

The asynchronous notification IO model introduced in the article Asynchronous notification IO model belongs to the notification IO model, and the IO multiplexing described in the similar select implementation of IO multiplexing also belongs to the communication IO model.

What is the notification IO model, that is, the IO is completed to notify the developer that a certain IO operation has been completed. According to the timing of the notification, it is divided into a synchronous notification IO model (select realizes IO multiplexing) and an asynchronous notification IO model (WSAEventSelect realizes asynchronous notification IO model). Here, looking back at the implementation of the first two models, it is found that the synchronous IO functions send and recv are used, and they are only called at the timing of the call (IO completion).

This chapter will introduce the asynchronous way to handle IO, pay attention to the difference from the previous two notification models and synchronous IO functions.

Overlapped IO

The IO overlap phenomenon caused by transmitting (or receiving from) data to multiple targets in the same thread is called overlapped IO.

In order to achieve this, the called IO function should return immediately instead of waiting for the data to be completely transferred to the output buffer (or return from the input buffer after reading data of a specified size). Only in this way can subsequent data be sent without waiting.

It can be seen that in order to complete asynchronous IO, the called IO function should work in non-blocking mode.

Create overlapping IO socket

For IO functions to work in non-blocking mode, overlapping IO sockets that provide non-blocking mode work are required. This can be done with the following functions:

#include <winsock2.h>

SOCKET WSASocket(int af, int type, int protocol, LPWSAPROTOCOL_INFO lpProtocolInfo, GROUP g, DWORD dwFlags);

af: protocol family information
type: the data transmission method of the socket
protocol: the protocol information used by the two socket quality checks
lpProtocolInfo: WSAPROTOCOL_INFO structure variable address value containing the created socket information, pass NULL if not needed
g: Parameter reserved for expansion, not used yet, pass 0
dwFlags: socket attribute information, through which to specify the creation of overlapping IO sockets

It can be seen that the first three parameters are the same as the socket function. The fourth and fifth parameters are not used yet and pass NULL and 0. By passing WSA_FLAG_OVERLAPPED to the last parameter, the created socket is given the overlapping IO feature.

Commonly used creation statements are as follows:

SOCKET hSock = WSASOCKET(PF_INET, SOCK_STREAM, 0, NULL, 0, WSA_FLAG_OVERLAPPED);
if (INVALID_SOCKET == hSock)
{<!-- -->
//WSASOCKET error!
}

The subsequent connection process (bind, listen, accept, connect), etc. is the same as the general socket connection process.

Let’s start to introduce the WSASend and WSARecv functions for asynchronous IO.

The WSASend function that performs overlapping IO

The prototype is as follows:

#include <winsock2.h>

int WSASend(SOCKET s, LPWSABUF lpBuffers, DWORD dwBufferCount, LPDWORD lpNumberOfBytesSent, DWORD dwFlags, LPWSAOVERLAPPED lpOverLapped, LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine);

s: socket handle with overlapping IO properties
lpBuffers: the address value of the WSABUF structure variable array, and the data to be transmitted is stored in WSABUF
dwBufferCount: the length of the structure variable array in the second parameter
lpNumberOfBytesSent: Used to save the variable address value of the actual number of bytes sent (because it returns immediately asynchronously, it is meaningless when the data is not transmitted)
dwFlags: Used to change data transmission characteristics, such as sending OOB mode data when passing MSG_OOB
lpOverLapped: The address value of the WSAOVERLAPPED structure variable, using the time object to confirm the completion of data transmission
lpCompletionRoutine: Pass in the entry address value of the Completion Routine function, and you can use this function to confirm whether the data transmission is completed.

Return value: return 0 on success, return SOCKET_ERROR on failure

The structures involved in the function parameters are introduced below.

WSABUF

It is defined as follows:

typedef struct _WSABUF {<!-- -->
    ULONG len; /* The size of the data to be transmitted */
    __field_bcount(len) CHAR FAR *buf; /* buffer address value */
} WSABUF, FAR * LPWSABUF;

WSAOVERLAPPED

It is defined as follows:

#define WSAOVERLAPPED OVERLAPPED

typedef struct _OVERLAPPED
{<!-- -->
    ULONG_PTR Internal;
    ULONG_PTR InternalHigh;
    union
    {<!-- -->
        structure
        {<!-- -->
            DWORD Offset;
            DWORD OffsetHigh;
        } DUMMYSTRUCTNAME;
        PVOID Pointer;
    } DUMMYUNIONNAME;

    HANDLE hEvent;
} OVERLAPPED, *LPOVERLAPPED;

Internal and InternalHigh members are members used internally by the operating system when performing overlapping IO, while Offset and OffsetHigh are used for members with special purposes, so there is no need to pay attention here.

Therefore, only the hEvent member needs to be paid attention to in actual development, which is a handle used to save the event object corresponding to the socket in the mode of confirming IO completion through events.

It should be noted that:In order to perform overlapping IO, the lpOverlapped parameter of the WSASend function should pass a valid structure variable address value instead of NULL.

If you pass NULL to lpOverlapped, the socket pointed to by the handle in the first parameter of the WSASend function (even if it has the overlapping attribute) will work in blocking mode.

In addition, when using the WSASend function to transmit data to multiple targets at the same time, it is necessary to construct the WSAOVERLAPPED structure variable passed in the sixth parameter instead of using the same variable. This is because, during the process of overlapping IO, the operating system will use the WSAOVERLAPPED structure variable.

Here is an example call to WSASend:

int nRecvLen = 0;

WSAEVENT event = WSACreateEvent(); //Create an event object in manual-reset mode

WSAOVERLAPPED overlapped;
memset( &overlapped, 0, sizeof(overlapped));
overlapped.hEvent = event;

WSABUF dataBuf;
char buf[] = {<!-- -->"Data to be transmitted"};
dataBuf.buf = buf;
dataBuf.len = sizeof(buf);

//Because the number of buffers to be transmitted in the second parameter dataBuf is 1, so just pass 1 here.
WSASend(hSock, &dataBuf, 1, &recvBytes, 0, &overlapped, NULL);

Supplement

The fourth parameter of WSASend, lpNumberOfBytesSent, is used to save the actual transmitted data size.

But here WSASend returns immediately after calling, so the value saved here is meaningless.

In fact, during the WSASend function call, the function return time point and the data transmission completion time point are not always inconsistent. If the output buffer is empty and the transmitted data is not large, the data transmission can be completed immediately after the function is called. At this time, WSASend returns 0, and the actual transmitted data size information is stored in lpNumberOfBytesSent.

Conversely, when the data transmission is not completed after the WSASend function returns, SOCKET_ERROR will be returned, and WSA_IO_PENDING will be registered as an error code, which can be obtained through the WSAGetLastError function. At this time, the actual transmitted data size should be obtained through the following function.

#include <winsock2.h>

BOOL WSAGetOverlappedResult(SOCKET s, LPWSAOVERLAPPED lpOverlapped, LPDWORD lpcbTransfer, BOOL fWait, LPDWORD lpdwFlags);

s: socket handle for overlapping IO
lpOverlapped: The address value of the WSAOVERLAPPED structure variable passed when performing overlapping IO
lpcbTransfet: variable address value used to save the actually transferred bytes
fWait: If IO is still in progress when calling this function, if fWait is TRUE, wait for IO to complete, and when fWait is FALSE, it will return FALSE and jump out of the function
lpdwFlags: Used to obtain additional information (such as OOB messages) when calling the WSARecv function. Pass NULL if not needed.

Through this function, not only the actual transmission result can be obtained, but also the status of the received data can be verified, see the subsequent examples.

WSARecv function for overlapping IO

The prototype is as follows:

#include <winsock2.h>

int WSARecv(SCOKET s, LPWSABUF lpBuffers, DWORD dwBufferCount, LPDWORD lpNumberOfBytesRecvd, LPDWORD lpFlags, LPWSAOVERLAPPED lpOverlapped, LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine);

s: socket handle to assign overlapped IO attributes
lpBuffers: save received data
dwBufferCount: the length of the received data array
lpNumberOfBytesRecvd: save the received data size
lpFlags: Used to set or read transmission characteristic information (for example, receive OOB out-of-band messages)
lpOverlapped: WSAOVERLAPPED structure variable address value
lpCompletionRoutine: CompletionRoutine function address value

IO completion confirmation of overlapping IO

The above WSASend and WSARecv functions return immediately after the call is completed. At this time, other tasks can be performed. How to determine that the IO of the previous WSASend or WSARecv has been completed after the other tasks are executed?

There are 2 methods in overlapping IO to confirm the completion of IO and get the IO result:
① Use the hEvent member of the sixth parameter lpOverlapped of the WSASend and WSARecv functions to confirm IO completion based on the event object
② Use the seventh parameter lpCompletionRoutine of the WSASend and WSARecv functions to confirm IO completion based on the Completion Routine callback function

Use the event object to confirm IO completion

When the IO is completed, the hEvent member of the lpOverlapped parameter will change to the signaled state. Later, the WSAWaitForMultipleEvents function can be used to wait for the event to be triggered, and then the WSAGetOverlappedResult function can be used to verify the completion of the IO and obtain the result.

The example is as follows, where the server and client are no longer distinguished, but the sender and receiver.

Sender

// OverlappedSend.cpp : Defines the entry point for the console application.
//

#include "stdafx.h"
#include <WinSock2.h>
#pragma comment(lib, "ws2_32.lib")


#define BUF_SIZE 1024


int _tmain(int argc, _TCHAR* argv[])
{<!-- -->

if (argc != 3)
{<!-- -->
puts("argc error!");
return -1;
}

WSADATA wsaData;
if (0 != WSAStartup(MAKEWORD(2, 2), &wsaData))
{<!-- -->
puts("WSAStartup error!");
return -1;
}

//Use the WSASocket function to create a socket for overlapping IO
SOCKET cltSock = WSASocket(PF_INET, SOCK_STREAM, 0, NULL, 0, WSA_FLAG_OVERLAPPED);
if (INVALID_SOCKET == cltSock)
{<!-- -->
puts("WSASocket error!");
WSACleanup();
return -1;
}

SOCKADDR_IN srvAddr;
memset( & amp; srvAddr, 0, sizeof(srvAddr));
srvAddr.sin_family = PF_INET;
srvAddr. sin_addr. s_addr = inet_addr(argv[1]);
srvAddr. sin_port = htons(_ttoi(argv[2]));

if (SOCKET_ERROR == connect(cltSock, (sockaddr*) & srvAddr, sizeof(srvAddr)))
{<!-- -->
puts("connect error!");
closesocket(cltSock);
WSACleanup();
return -1;
}

char buf[] = "Network is Computer";

// Overlapped IO model preparation
WSAEVENT overlappedEvent = WSACreateEvent();
WSAOVERLAPPED overlapped;
memset( &overlapped, 0, sizeof(WSAOVERLAPPED));
overlapped.hEvent = overlappedEvent;

WSABUF dataBuf;
dataBuf.buf = buf;
dataBuf.len = strlen(buf) + 1;

int nTmp = sizeof(buf);

int nSendLen = 0, flags = 0;
if (SOCKET_ERROR == WSASend(cltSock, &dataBuf, 1, (LPDWORD) &nSendLen, 0, &overlapped, NULL))
{<!-- -->
if (GetLastError() == WSA_IO_PENDING)
{<!-- -->
puts("Background data send.");

//Wait for the data to be sent
WSAWaitForMultipleEvents(1, &overlappedEvent, TRUE, WSA_INFINITE, FALSE);

// Get sent data information
WSAGetOverlappedResult(cltSock, & amp;overlapped, (LPDWORD) & amp;nSendLen, FALSE, NULL);
}
else
{<!-- -->
puts("WSASend error");
}
}

printf("send data size: %d \
", nSendLen);
WSACloseEvent(overlappedEvent);
closesocket(cltSock);
WSACleanup();

puts("any key to continue...");
getchar();

return 0;
}

Receiver

// OverlappedRecv.cpp : Defines the entry point of the console application.
//

#include "stdafx.h"
#include <WinSock2.h>
#pragma comment(lib, "ws2_32.lib")


#define BUF_SIZE 1024

int _tmain(int argc, _TCHAR* argv[])
{<!-- -->
if (argc != 2)
{<!-- -->
puts("argc error!");
return -1;
}

WSADATA wsaData;
if (0 != WSAStartup(MAKEWORD(2, 2), &wsaData))
{<!-- -->
puts("WSAStartup error!");
return -1;
}

//Use the WSASocket function to create a socket for overlapping IO
SOCKET srvSock = WSASocket(PF_INET, SOCK_STREAM, 0, NULL, 0, WSA_FLAG_OVERLAPPED);
if (INVALID_SOCKET == srvSock)
{<!-- -->
puts("WSASocket error!");
WSACleanup();
return -1;
}

SOCKADDR_IN srvAddr;
memset( & amp; srvAddr, 0, sizeof(srvAddr));
srvAddr.sin_family = PF_INET;
srvAddr.sin_addr.s_addr = htonl(ADDR_ANY);
srvAddr. sin_port = htons(_ttoi(argv[1]));

if (SOCKET_ERROR == bind(srvSock, (sockaddr*) & srvAddr, sizeof(srvAddr)))
{<!-- -->
puts("bind error!");
closesocket(srvSock);
WSACleanup();
return -1;
}

if (SOCKET_ERROR == listen(srvSock, 5))
{<!-- -->
puts("listen error!");
closesocket(srvSock);
WSACleanup();
return -1;
}

SOCKADDR_IN cltAddr;
int nCltAddrSize = sizeof(cltAddr);
memset( &cltAddr, 0, nCltAddrSize);
\t
SOCKET cltSock = accept(srvSock, (sockaddr*) &cltAddr, &nCltAddrSize);


WSAEVENT overlappedEvent = WSACreateEvent();
WSAOVERLAPPED overlapped;
memset( &overlapped, 0, sizeof(overlapped));
overlapped.hEvent = overlappedEvent;

char buf[BUF_SIZE] = {<!-- -->};
WSABUF dataBuf;
dataBuf.buf = buf;
dataBuf.len = BUF_SIZE;

int nRecvLen = 0, flags = 0;
if (SOCKET_ERROR == WSARecv(cltSock, &dataBuf, 1, (LPDWORD) &nRecvLen, (LPDWORD) &flags, &overlapped, NULL))
{<!-- -->
if (WSAGetLastError() == WSA_IO_PENDING)
{<!-- -->
//Background is receiving data
puts("Background data receive");

//Block here and wait for the data transmission to complete
WSAWaitForMultipleEvents(1, &overlappedEvent, TRUE, WSA_INFINITE, FALSE);

//Get received data information
WSAGetOverlappedResult(cltSock, & amp;overlapped, (LPDWORD) & amp;nRecvLen, FALSE, NULL);
}
else
{<!-- -->
puts("WSARecv error!");
}
}

//If WSARecv does not return SOCKET_ERROR, it means that the data is received immediately
//dataBuf.buf[nRecvLen] = 0;
printf("Received len: %d, message: %s \
", nRecvLen, dataBuf.buf);

closesocket(cltSock);
closesocket(srvSock);
WSACloseEvent(overlappedEvent);
WSACleanup();

puts("any key to continue...");
getchar();

return 0;
}

Use Completion Routine to confirm IO completion

In addition to verifying whether the IO is completed through the event object, you can also specify the Completion Routine (CR for short) to verify the completion of the IO through the seventh parameter of the WSASend and WSARecv functions.

“Register CR” has the following meaning: This function is called when the Pending IO is completed.

When the IO is completed, the registered function is called for post-processing. This is how the Completion Routine works.

It should be noted that if the Compleiton Routine is called suddenly when performing an important task, it may disrupt the normal execution flow of the program. Therefore, the operating system usually predefines the rules: the Completion Routine function can only be called when the thread requesting IO is in the alertable wait state!

alertable wait state is the thread state waiting to receive operating system messages. Enter the alertable wait state when calling the following functions:

• WaitForSingleObjectEx
• WaitForMultipleObjectsEx
• WSAWaitForMultipleEvents
• SleepEx

After calling, the above functions will all return WAIT_IO_COMPLETION and start to execute subsequent programs.

Examples are as follows:

Receiver based on Completion Routine

// CmplRoutinesRecv.cpp : Defines the entry point of the console application.
//

#include "stdafx.h"

#include <WinSock2.h>
#pragma comment(lib, "ws2_32.lib")


#define BUF_SIZE 1024

void CALLBACK CpmpRoutine(DWORD , DWORD , LPWSAOVERLAPPED , DWORD);

WSABUF dataBuf;
char buf[BUF_SIZE];
int nRecvLen = 0;

int _tmain(int argc, _TCHAR* argv[])
{<!-- -->
if (argc != 2)
{<!-- -->
puts("argc error!");
return -1;
}

WSADATA wsaData;
if (0 != WSAStartup(MAKEWORD(2, 2), &wsaData))
{<!-- -->
puts("WSAStartup error!");
return -1;
}

//Use the WSASocket function to create a socket for overlapping IO
SOCKET srvSock = WSASocket(PF_INET, SOCK_STREAM, 0, NULL, 0, WSA_FLAG_OVERLAPPED);
if (INVALID_SOCKET == srvSock)
{<!-- -->
puts("WSASocket error!");
WSACleanup();
return -1;
}

SOCKADDR_IN srvAddr;
memset( & amp; srvAddr, 0, sizeof(srvAddr));
srvAddr.sin_family = PF_INET;
srvAddr.sin_addr.s_addr = htonl(ADDR_ANY);
srvAddr. sin_port = htons(_ttoi(argv[1]));

if (SOCKET_ERROR == bind(srvSock, (sockaddr*) & srvAddr, sizeof(srvAddr)))
{<!-- -->
puts("bind error!");
closesocket(srvSock);
WSACleanup();
return -1;
}

if (SOCKET_ERROR == listen(srvSock, 5))
{<!-- -->
puts("listen error!");
closesocket(srvSock);
WSACleanup();
return -1;
}

SOCKADDR_IN cltAddr;
int nCltAddrSize = sizeof(cltAddr);
memset( &cltAddr, 0, nCltAddrSize);

SOCKET cltSock = accept(srvSock, (sockaddr*) &cltAddr, &nCltAddrSize);

//Use CompletionRoutine function instead of event
WSAOVERLAPPED overlapped;
memset( &overlapped, 0, sizeof(overlapped));

dataBuf.buf = buf;
dataBuf.len = BUF_SIZE;

WSAEVENT waitEvent = WSACreateEvent();

int flags = 0;

//Receive data, when the reception is complete, judge whether the thread is in the alertable wait state, and then call CpmpRoutine
if (SOCKET_ERROR == WSARecv(cltSock, &dataBuf, 1, (LPDWORD) &nRecvLen, (LPDWORD) &flags, &overlapped, CpmpRoutine))
{<!-- -->
if (GetLastError() == WSA_IO_PENDING)
{<!-- -->
//Background data is being received
puts("Background data receive");

\t\t\t
}
else
{<!-- -->
puts("WSARecv error!");
closesocket(cltSock);
closesocket(srvSock);
WSACloseEvent(waitEvent);
WSACleanup();
return -1;
}
}

//nRecvLen = szRecvBytes;
//printf("Receive len: %d, message: %s\
", nRecvLen, dataBuf.buf);

//Enter the alertable wait state, waiting for the data reception to complete
int idx = WSAWaitForMultipleEvents(1, &waitEvent, FALSE, WSA_INFINITE, TRUE);

if (idx == WSA_WAIT_IO_COMPLETION)
{<!-- -->
puts("Overlapped IO Completed!");
}
else
{<!-- -->
puts("WSAWaitForMultipleEvents error!");
}

closesocket(cltSock);
closesocket(srvSock);
WSACloseEvent(waitEvent);
WSACleanup();

puts("any key to continue...");
getchar();

return 0;
}

void CALLBACK CpmpRoutine(DWORD dwError, DWORD szRecvBytes, LPWSAOVERLAPPED lpOverlapped, DWORD flags)
{<!-- -->
if (dwError != 0)
{<!-- -->
puts("CompRoutine error!");
}
else
{<!-- -->
nRecvLen = szRecvBytes;
printf("CpmpRoutine--Receive len: %d, message: %s\
", nRecvLen, dataBuf.buf);
}
}