[Source code explanation] sylar server framework—-hook module

hook introduction and implementation

The hook module is used to 100% simulate the original system call functions related to network IO, and encapsulate the original system call functions. Its purpose is to create a layer of encapsulation on top of the original system call function. When using it, users will feel that the behavior is exactly the same as the original function, but in fact some processing will be done inside the function to adapt to the coroutine. It’s just that the user doesn’t know.

The hook of this module is implemented using dynamic linking. The implementation method of hook is to use the global symbols of the dynamic library to replace the original system call function with the same function signature with a custom interface. Because the system call interface is basically provided by the C standard function library libc. So what this module has to do is to use a custom dynamic library to overwrite the function of the same name in libc.

There are two ways to implement hooks, one is intrusive hook and the other is non-intrusive hook.

Intrusive hook:

After using the gcc command to compile the program, you can use the ldd command to view the shared libraries that the executable program depends on, such as

# ldd a.out
        linux-vdso.so.1 (0x00007ffc96519000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fda40a61000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fda40c62000)

You can see the libc shared library it depends on

Next, without recompiling the code, replace the original system call implementation in the executable program a.out with a custom dynamic library and create a new hook.c:

#include <unistd.h>
#include <sys/syscall.h>
#include <string.h>
 
ssize_t write(int fd, const void *buf, size_t count) {
    syscall(SYS_write, STDOUT_FILENO, "12345\
", strlen("12345\
"));
}

A write function is implemented here. The signature of this function is exactly the same as the write function provided by libc. The content of the function is to directly call the system call numbered SYS_write using syscall. The effect of the implementation is to write content to the standard output, but here we Replace the output content with other values. Compile hook.c into a dynamic library:

gcc -fPIC -shared hook.c -o libhook.so

By setting the LD_PRELOAD environment variable, set libhoook.so to be loaded first, thus overriding the write function in libc.

The above does not recompile the specifyable program, but the write function has been replaced with its own implementation.

Non-intrusive hook:

This hook implementation requires modifying the code or recompiling it to specify the dynamic library to be used. Write code with the same function signature directly inside the program, and link the custom dynamic library before libc through compilation parameters.

So now a question arises, how to find the system call function that has been overridden by the user-defined function? Linux has a dslym system call function:

#include <dlfcn.h>
 
void *dlsym(void *handle, const char *symbol);

The function of the dslym system call function is to operate handles and symbols based on the dynamic link library and return the address corresponding to the symbol. The first parameter is passed in the pointer returned after opening the dynamic link library, and the second parameter is passed in the address corresponding to the function required to be obtained (global variables are also acceptable). When using it, the first parameter can be passed directly to RTLD_NEXT, which means the function pointer of the function name that appears for the first time. In this way, the original function address can be found.

HOOK module introduction:

The hook function implemented by this module is based on threads. You can freely set whether the current thread uses hooks. By default, the thread where the scheduler is located will be enabled, and other threads will not be enabled.

extern “C”: Specifies the compiler to compile the code in the code segment in the form of C language.

The hooks implemented by this module can be roughly divided into three categories: ① socket IO series interfaces ② sleep delay series interfaces ③ closs/fcntl and other interfaces

1. Socket IO series interface

Because the behavior of this part of the interface is quite consistent, we decided to implement it with the variable parameter template in C++. The interfaces in this part include read, readv, recv, recvfrom, recvmsg, write, writev, send, sendto, and sendmsg. The series of functions are introduced as follows:

read：

Function prototype: ssize_t read(int fd, void *buf, size_t count);

Description: read is used to read data from the file descriptor fd and store the data in buf. Up to count bytes can be read. .

Function: Mainly used for file I/O, usually used to read file contents.

Difference: It is blocking, waiting for data to be available, and then reading the specified number of bytes.

readv：

Function prototype: ssize_t readv(int fd, const struct iovec *iov, int iovcnt);

Description: readv allows reading data from multiple buffers at the same time, using the iovec structure array to describe the location and length of different buffers.

Function: Reduce the need for data copying and improve performance, especially suitable for combining multiple buffers into a single continuous block.

Difference: It is used for file I/O and is blocking.

recv：

Function prototype: ssize_t recv(int sockfd, void *buf, size_t len, int flags);

Description: recv is used to receive data from the socket sockfd, store the data into buf, and receive up to len bytes.

Function: Mainly used for network communication, such as TCP or UDP sockets, for receiving data.

The difference: It is blocking, waiting for data to be available, and then reading the specified number of bytes.

recvfrom：

Function prototype: ssize_t recvfrom(int sockfd, void *buf, size_t len, int flags, struct sockaddr *src_addr, socklen_t *addrlen);

Description: recvfrom is used to receive data from the socket sockfd and obtain the source address information of the data. The address information is stored in src_addr.

Function: Mainly used for UDP sockets, because UDP is connectionless and each packet may come from a different address.

Difference: Used for network communication to obtain the source address information of data.

recvmsg：

Function prototype: ssize_t recvmsg(int sockfd, struct msghdr *msg, int flags);

Description: recvmsg is used to receive data from the socket sockfd and provide more information, such as auxiliary data and control information, stored in the msg structure.

Purpose: Suitable for more complex communication needs, such as Unix domain sockets, providing more control and information.

Difference: Provides more control and information.

write：

Function prototype: ssize_t write(int fd, const void *buf, size_t count);

Description: write is used to write data from buf to file descriptor fd, up to count bytes.

Function: Mainly used for file I/O, usually used for writing file contents.

Difference: It is blocking and will wait for data to be written successfully.

writev：

Function prototype: ssize_t writev(int fd, const struct iovec *iov, int iovcnt);

Description: writev allows writing data from multiple buffers to file descriptors at the same time, using an array of iovec structures to describe the location and length of different buffers.

Function: Reduce the need for data copying and improve performance, especially suitable for writing data to file descriptors.

send：

Function prototype: ssize_t send(int sockfd, const void *buf, size_t len, int flags);

Description: send is used to send data from buf to socket sockfd.

Function: Mainly used for network communication, such as TCP or UDP sockets, for sending data.

The difference: it is blocking, waiting for data to be sent successfully.

sendto：

Function prototype: ssize_t sendto(int sockfd, const void *buf, size_t len, int flags, const struct sockaddr *dest_addr, socklen_t addrlen);

Description: sendto is used to send data from buf to socket sockfd, while specifying the target address information, which is stored in dest_addr.

Function: Mainly used for UDP sockets to send data to a specific target address.

Difference: used for network communication, the target address can be specified.

sendmsg：

Function prototype: ssize_t sendmsg(int sockfd, const struct msghdr *msg, int flags);

Description: sendmsg is used to send data from the msg structure to the socket sockfd, while providing more information, such as auxiliary data and control information.

Function: Suitable for more complex communication needs, providing more control and information.

Difference: Provides more control and information.

Specific code implementation:

Args & amp; & amp;... args is the Template Parameter Pack syntax in C++11, usually used in function templates to support a variable number of parameters. This syntax allows you to accept a variable number of arguments in a function template and handle them in a more flexible way in the function body.

Specifically, Args & amp; & amp;... args is typically used in template function parameter lists, where:

Args is a placeholder for the template parameter pack, which represents any type of parameters.

& amp; & amp; represents an rvalue reference, which can be bound to an rvalue (temporary object).

... represents a parameter pack expansion, allowing any number of parameters to be accepted.

Forward parameter packet:

Under the C++11 standard, we can use a combination of variadic templates and the forwad mechanism to write functions to pass their parameters to other functions unchanged.

std::forward(args)…

std::forward is a standard library function template used to perform perfect forwarding.

is a template parameter package, indicating the type of parameters.

(args) is a function parameter pack, indicating the accepted parameters.

... represents parameter pack expansion, allowing any number of parameters to be passed.

The main goal of perfect forwarding is to avoid losing the lvalue or rvalue properties of parameters when passing them to ensure correct parameter passing. In function templates, you may encounter situations similar to the following:

template<typename OriginFun, typename... Args>
/**
 * @brief unified implementation of socket IO series functions
 * @param[in] fd socket handle
 * @param[in] fun function pointer of system call function
 * @param[in] hook_fun_name hook function name
 * @param[in] event IO event (read event or write event)
 * @param[in] timeout_so sets the timeout for socket reception or transmission
 * @param[in] args Other parameters of the original system call except fd
 */
static ssize_t do_io(int fd, OriginFun fun, const char* hook_fun_name,
        uint32_t event, int timeout_so, Args & amp; & amp;... args) {
    // If this thread does not use hooks, then just use the original system call function directly.
    if(!sylar::t_hook_enable) {
        return fun(fd, std::forward<Args>(args)...);
    }
    //If the file handle class cannot be obtained, the original system call function will be used directly.
    sylar::FdCtx::ptr ctx = sylar::FdMgr::GetInstance()->get(fd);
    if(!ctx) {
        return fun(fd, std::forward<Args>(args)...);
    }
    //The fd has been closed. This fd cannot send or receive data. It returns -1 directly and sets the error code.
    if(ctx->isClose()) {
        errno = EBADF;
        return -1;
    }
    //If it is not a socket connection, or if the user actively sets the non-blocking function, the original system call function will be used directly. This is because it can guarantee all writing or reading at one time, even if the reading and writing are not completed, it will not matter.
    if(!ctx->isSocket() || ctx->getUserNonblock()) {
        return fun(fd, std::forward<Args>(args)...);
    }
    //Get the timeout time
    uint64_t to = ctx->getTimeout(timeout_so);
    std::shared_ptr<timer_info> tinfo(new timer_info);

retry:
    //Execute socket IO series functions
    ssize_t n = fun(fd, std::forward<Args>(args)...);
    //If the IO operation is interrupted by a signal, then execute the IO function repeatedly
    while(n == -1 & amp; & amp; errno == EINTR) {
        n = fun(fd, std::forward<Args>(args)...);
    }
    //If there is no data to read or write, but there is no read or write to EOF, execute the following branch
    if(n == -1 & amp; & amp; errno == EAGAIN) {
        sylar::IOManager* iom = sylar::IOManager::GetThis();
        sylar::Timer::ptr timer;
        std::weak_ptr<timer_info> winfo(tinfo);
        //If a timeout is set, add a conditional timer, execute it only once after the timeout, trigger and delete the event after the timeout expires, and set the t->cancelled timeout flag
        if(to != (uint64_t)-1) {
            timer = iom->addConditionTimer(to, [winfo, fd, iom, event]() {
                // If it returns due to timeout, execute the content of this callback function
                auto t = winfo.lock();
                if(!t || t->cancelled) {
                    return;
                }
                t->cancelled = ETIMEDOUT;
                iom->cancelEvent(fd, (sylar::IOManager::Event)(event));
            }, winfo);
        }
        //Add events to epoll and give up execution rights. Epoll will continue to monitor this fd and let the thread select the coroutine task in the task queue and execute it.
        int rt = iom->addEvent(fd, (sylar::IOManager::Event)(event));
        if(SYLAR_UNLIKELY(rt)) {
            SYLAR_LOG_ERROR(g_logger) << hook_fun_name << " addEvent("
                << fd << ", " << event << ")";
            if(timer) {
                timer->cancel();
            }
            return -1;
        } else {
            sylar::Fiber::GetThis()->yield();
            // Waiting for the callback function of the timeout execution timer or the return of epoll_wait indicates that the socket can be read and written.
            // If it times out and the winfo condition is valid, then set the timeout flag through winfo and trigger the read and write event. The coroutine returns from yield. After returning, set errno through the timeout flag and return -1
            // If epoll_wait returns, then cancel the timer and return success. When the timer is canceled, a timer callback will be executed. At this time, the connect_with_timeout coroutine has been executed, so the condition variable is released and the flag will no longer be set.
            if(timer) {
                timer->cancel();
            }
            //Set canceled as error code
            if(tinfo->cancelled) {
                errno = tinfo->cancelled;
                return -1;
            }
            goto retry;
            
        }
    }
    
    return n;
}



ssize_t read(int fd, void *buf, size_t count) {
    return do_io(fd, read_f, "read", sylar::IOManager::READ, SO_RCVTIMEO, buf, count);
}

ssize_t readv(int fd, const struct iovec *iov, int iovcnt) {
    return do_io(fd, readv_f, "readv", sylar::IOManager::READ, SO_RCVTIMEO, iov, iovcnt);
}

ssize_t recv(int sockfd, void *buf, size_t len, int flags) {
    return do_io(sockfd, recv_f, "recv", sylar::IOManager::READ, SO_RCVTIMEO, buf, len, flags);
}

ssize_t recvfrom(int sockfd, void *buf, size_t len, int flags, struct sockaddr *src_addr, socklen_t *addrlen) {
    return do_io(sockfd, recvfrom_f, "recvfrom", sylar::IOManager::READ, SO_RCVTIMEO, buf, len, flags, src_addr, addrlen);
}

ssize_t recvmsg(int sockfd, struct msghdr *msg, int flags) {
    return do_io(sockfd, recvmsg_f, "recvmsg", sylar::IOManager::READ, SO_RCVTIMEO, msg, flags);
}

ssize_t write(int fd, const void *buf, size_t count) {
    return do_io(fd, write_f, "write", sylar::IOManager::WRITE, SO_SNDTIMEO, buf, count);
}

ssize_t writev(int fd, const struct iovec *iov, int iovcnt) {
    return do_io(fd, writev_f, "writev", sylar::IOManager::WRITE, SO_SNDTIMEO, iov, iovcnt);
}

ssize_t send(int s, const void *msg, size_t len, int flags) {
    return do_io(s, send_f, "send", sylar::IOManager::WRITE, SO_SNDTIMEO, msg, len, flags);
}

ssize_t sendto(int s, const void *msg, size_t len, int flags, const struct sockaddr *to, socklen_t tolen) {
    return do_io(s, sendto_f, "sendto", sylar::IOManager::WRITE, SO_SNDTIMEO, msg, len, flags, to, tolen);
}

ssize_t sendmsg(int s, const struct msghdr *msg, int flags) {
    return do_io(s, sendmsg_f, "sendmsg", sylar::IOManager::WRITE, SO_SNDTIMEO, msg, flags);
}

2. sleep series delay function

This part of the function includes sleep, usleep, nanosleep, which are implemented through timers. The difference between these three timing functions is that the units are seconds, microseconds, and nanoseconds. Due to the limitation of epoll_wait, it can only support up to milliseconds. In fact, the accuracy of these three functions is the same. Take usleep as an example:

int usleep(useconds_t usec) {
    // If this thread does not use hooks, then just use the original system call function directly.
    if(!sylar::t_hook_enable) {
        return usleep_f(usec);
    }
    sylar::Fiber::ptr fiber = sylar::Fiber::GetThis();
    sylar::IOManager* iom = sylar::IOManager::GetThis();
    //Add a timer, execute it after usec/1000 milliseconds, and add scheduled tasks to the scheduler's task queue after usec/1000 milliseconds.
    iom->addTimer(usec / 1000, std::bind((void(sylar::Scheduler::*)
            (sylar::Fiber::ptr, int thread)) & sylar::IOManager::schedule
            ,iom, fiber, -1));
    sylar::Fiber::GetThis()->yield();
    return 0;
}

Three: Other functions related to network IO

//The only difference from the original system call function is that a file handle class is also created
int socket(int domain, int type, int protocol) {
    if(!sylar::t_hook_enable) {
        return socket_f(domain, type, protocol);
    }
    int fd = socket_f(domain, type, protocol);
    if(fd == -1) {
        return fd;
    }
    sylar::FdMgr::GetInstance()->get(fd, true);
    return fd;
}




//In this server framework, you can also customize the timeout of connect
int connect_with_timeout(int fd, const struct sockaddr* addr, socklen_t addrlen, uint64_t timeout_ms) {
    // If this thread does not use hooks, then just use the original system call function directly.
    if(!sylar::t_hook_enable) {
        return connect_f(fd, addr, addrlen);
    }
    // If fd is closed, set error and return
    sylar::FdCtx::ptr ctx = sylar::FdMgr::GetInstance()->get(fd);
    if(!ctx || ctx->isClose()) {
        errno = EBADF;
        return -1;
    }
    // If fd is not a socket or user-defined non-blocking, use the original api
    if(!ctx->isSocket()) {
        return connect_f(fd, addr, addrlen);
    }

    if(ctx->getUserNonblock()) {
        return connect_f(fd, addr, addrlen);
    }
    // If successful, return 0, if an error occurs, return an error code
    int n = connect_f(fd, addr, addrlen);
    if(n == 0) {
        return 0;
    } else if(n != -1 || errno != EINPROGRESS) {
        return n;
    }
    // If the connection cannot be established immediately due to non-blocking settings, execute the following statement
    sylar::IOManager* iom = sylar::IOManager::GetThis();
    sylar::Timer::ptr timer;
    std::shared_ptr<timer_info> tinfo(new timer_info);
    std::weak_ptr<timer_info> winfo(tinfo);
    //If connect sets a timeout, add a conditional timer, execute it only once after the timeout, trigger and delete the event after the timeout, and set the t->cancelled timeout flag
    if(timeout_ms != (uint64_t)-1) {
        timer = iom->addConditionTimer(timeout_ms, [winfo, fd, iom]() {
                auto t = winfo.lock();
                if(!t || t->cancelled) {
                    return;
                }
                t->cancelled = ETIMEDOUT;
                iom->cancelEvent(fd, sylar::IOManager::WRITE);
        }, winfo);
    }
    //Add events to epoll and give up execution rights. Epoll will continue to monitor this fd and let the thread select the coroutine task in the task queue and execute it.
    int rt = iom->addEvent(fd, sylar::IOManager::WRITE);
    if(rt == 0) {
        sylar::Fiber::GetThis()->yield();
        if(timer) {
            timer->cancel();
        }
        if(tinfo->cancelled) {
            errno = tinfo->cancelled;
            return -1;
        }
    } else {
        if(timer) {
            timer->cancel();
        }
        SYLAR_LOG_ERROR(g_logger) << "connect addEvent(" << fd << ", WRITE) error";
    }

    int error = 0;
    socklen_t len = sizeof(int);
    if(-1 == getsockopt(fd, SOL_SOCKET, SO_ERROR, & amp;error, & amp;len)) {
        return -1;
    }
    if(!error) {
        return 0;
    } else {
        errno = error;
        return -1;
    }
}
int connect(int sockfd, const struct sockaddr *addr, socklen_t addrlen) {
    return connect_with_timeout(sockfd, addr, addrlen, sylar::s_connect_timeout);
}


//The only difference from the original system call function is that the file handle class is also created
int accept(int s, struct sockaddr *addr, socklen_t *addrlen) {
    int fd = do_io(s, accept_f, "accept", sylar::IOManager::READ, SO_RCVTIMEO, addr, addrlen);
    if(fd >= 0) {
        sylar::FdMgr::GetInstance()->get(fd, true);
    }
    return fd;
}


//The close function also additionally encapsulates the cancellation and triggering of all events monitored by epoll once, as well as deletion of the file handle context class
int close(int fd) {
    if(!sylar::t_hook_enable) {
        return close_f(fd);
    }

    sylar::FdCtx::ptr ctx = sylar::FdMgr::GetInstance()->get(fd);
    if(ctx) {
        auto iom = sylar::IOManager::GetThis();
        if(iom) {
            iom->cancelAll(fd);
        }
        sylar::FdMgr::GetInstance()->del(fd);
    }
    return close_f(fd);
}

The knowledge points of the article match the official knowledge files, and you can further learn relevant knowledge. Cloud native entry-level skills treeHomepageOverview 16,929 people are learning the system