epoll+reactor combination

Background

1. Use background and overall structure
2. How to use epoll
3. Understand the reactor (reactor) mode
4. epoll + reactor joint

1.1 Why IO multiplexing occurs

Compared with the traditional simple tcp server, the user connection is based on threads, that is, when a new client (new fd) is connected, the server will allocate a thread to serve the connection, but the system resources are limited (that is, thread resources is limited) leading to a limited number of client connections, which is the so-called C10k problem, then such a server is a defective server. Therefore, with the development of technology, IO multiplexing (EPOLL/SELECT/POLL) is introduced.

1.2 IO multiplexing

IO refers to network IO, multi-channel refers to multiple TCP connections, and multiplexing refers to multiplexing one or more threads. IO multiplexing is a synchronous IO model (image_1), which means that one thread can monitor multiple file handles. It can be seen from the figure below that once the data is ready (that is, after a file handle is ready), the application can be notified to perform corresponding read and write operations; when no file is ready, the application will be blocked and handed over at this time. CPU.

image_1

2.1 EPOLL related functions

int epoll_create(int size);

Function: Create an epoll file descriptor (a listening red-black tree)
Parameters: size – the number of listener nodes to create a red-black tree (for kernel reference only)
Return value: successfully returns the root node fd of the newly created red-black tree; fails and returns -1, and errno is set at the same time.

int epoll_ctl(int epfd, int op, int fd, struct epoll_event *event);

Function: Add, modify, and delete operations on the red-black tree. That is, the control operation is performed on the customer.
Parameters:

  • int epfd
    The return value of the epoll_create function (monitoring the file descriptor of the root node of the red-black tree)
  • int op
    Operations performed on the listener red-black tree:
    EPOLL_CTL_ADD
    Add fd to monitor red-black tree
    EPOLL_CTL_MOD
    Modify fd to listen to the listening events on the red-black tree
    EPOLL_CTL_DEL
    Remove an fd from the monitoring red-black tree [cancel monitoring/delete a monitoring time]
  • int fd
    The file descriptor to monitor
  • struct epoll_event *event
    The essence of event is a struct epoll_event structure pointer
struct epoll_event {<!-- -->
uint32_t events; /* Epoll events /
epoll_data_t data; / user data */
};

Variables in this structure:

  • uint32_t events: EPOLLIN/EPOLLOUT/EPOLLERR
  • epoll_data_t data: Union epoll_data [This means that if ptr is used, data cannot be used! If data is used, ptr cannot be used]
typedef union epoll_data {<!-- -->
void *ptr;
int fd;
uint32_t u32;
uint64_t u64;
} epoll_data_t;
  • void *ptr: Use this pointer to call back functions in the epoll reactor model [When the ptr pointer is used, the fd parameter will not work! Because it is a union! ! 】
  • int fd corresponds to the fd that listens to the event
  • uint32_t u32; Generally not used
  • uint64_t u64; Generally not used
    Return value
    Return 0 on success, -1 on failure, and set errno

int epoll_wait(int epfd, struct epoll_event *events, int maxevents, int timeout);

Function: Wait for the generation of all fd corresponding events provided by the interval.
Parameters:

  • int epfd: fd of the epoll instance returned by the epoll_create() function.
  • struct epoll_event * events: The return parameter of the interface, epoll copies the collection of events that occurred from the kernel to the events array. The events array is an array with a size allocated by the user, and the length of the array is greater than or equal to maxevents. (events cannot be a null pointer, the kernel is only responsible for copying data into the events array, and will not help us allocate memory in user mode).
  • int maxevents: Indicates the maximum number of events that can be returned this time, usually the maxevents parameter is equal to the size of the pre-allocated events array.
  • int timeout: Indicates the maximum waiting time when no event is detected, the timeout period (>=0), the unit is ms, -1 means blocking, 0 means not blocking.
    Return value:
  • Returning >0 means the total number of monitors is satisfied, which can be used as the upper limit of the loop.
  • Return = 0 means there is no satisfied listening event
  • return <0 on failure, and set errno

2.2 The general flow of the EPOLL model

int lfd = socket();
bind();
listen();
// Create a listener red-black tree, OPEN_MAX is the maximum number of connections
int epfd = epoll_create(OPEN_MAX);
// Add lfd to the monitoring red-black tree
// Set the corresponding monitoring properties in tep
epoll_ctl(epfd, EPOLL_CTL_ADD, lfd, &tep);
while(1){<!-- -->
    // Monitor whether there is a request
    epoll_wait(epfd, ep, OPEN_MAX, -1);
    // Traverse and handle listening events
    for() {<!-- -->
        // Handle lfd foreground events
        if(ep[i].data.fd == lfd)
            // Add the newly obtained cfd to the monitoring red-black tree
            epoll_ctl(epfd, EPOLL_CTL_ADD, cfd, &tep);
        else{<!-- -->
            Read/Write handles requests from cfd
        }
    }
}

3. Reactor

The reactor mode is a typical event-driven programming model, which converts the processing of IO into the processing of events.

4. EPOLL + single-mode REACTOR instance

Background: This example is a small demo of the main thread receiving the data without using the thread pool to solve the business.
Single reactor mode

  • Instantiate each connection (that is, the context of each connection)
struct ntyevent {<!-- -->
int fd;
int events;
void *arg;
int (*callback)(int fd, int events, void *arg);
\t
int status;
char buffer[BUFFER_LENGTH];
int length;
long last_active;
};
  • All connections are managed by ntyreactor
struct ntyreactor {<!-- -->
int epfd;
struct ntyevent *events;
};
  • As shown in the figure, the reactor reactor manages the entire connection in a unified manner
  • code show as below
 reactor->events = (struct ntyevent*)malloc((MAX_EPOLL_EVENTS) * sizeof(struct ntyevent));
  • When an event comes the code is as follows
int reactor_run(struct ntyreactor *reactor) {<!-- -->
if (reactor == NULL) return -1;
if (reactor->epfd < 0) return -1;
if (reactor->events == NULL) return -1;
\t
struct epoll_event events[MAX_EPOLL_EVENTS + 1];
\t
int checkpos = 0, i;

while (1) {<!-- -->

long now = time(NULL);
for (i = 0;i < 100;i ++ , checkpos ++ ) {<!-- -->
if (checkpos == MAX_EPOLL_EVENTS) {<!-- -->
checkpos = 0;
}

if (reactor->events[checkpos].status != 1) {<!-- -->
continue;
}

long duration = now - reactor->events[checkpos].last_active;

if (duration >= 60) {<!-- -->
close(reactor->events[checkpos].fd);
printf("[fd=%d] timeout\\
", reactor->events[checkpos].fd);
nty_event_del(reactor->epfd, &reactor->events[checkpos]);
}
}


int nready = epoll_wait(reactor->epfd, events, MAX_EPOLL_EVENTS, 1000);
if (nready < 0) {<!-- -->
printf("epoll_wait error, exit\\
");
continue;
}

for (i = 0;i < nready;i ++ ) {<!-- -->

struct ntyevent *ev = (struct ntyevent*)events[i].data.ptr;

if ((events[i].events & amp; EPOLLIN) & amp; & amp; (ev->events & amp; EPOLLIN)) {<!-- -->
ev->callback(ev->fd, events[i].events, ev->arg);
}
if ((events[i].events & amp; EPOLLOUT) & amp; & amp; (ev->events & amp; EPOLLOUT)) {<!-- -->
ev->callback(ev->fd, events[i].events, ev->arg);
}
\t\t\t
}

}
}