Linux log management-logrotate (crontab scheduled task, Ceph log dump)

Article directory

• 1. Overview of logrotate
• 2. Basic usage of logrotate
• 3. logrotate operating mechanism
• • logrotate parameters
• 4. How does logrotate achieve rolling logs without affecting the normal log output of the program?
• • Linux file operation mechanism
  • Option One
  • Option II
• 5. Logrotate in action–Ceph log dump
• refer to

1. Overview of logrotate

logrotate is a tool for managing log files on Linux systems. Its main purpose is to ensure that log files do not become too large, thereby taking up too much disk space and affecting system performance. When the system runs longer, more log files are generated, and these files may become very large, resulting in insufficient disk space. Additionally, if log files are too large, they may become unreadable, making it difficult to diagnose and resolve problems. logrotate can regularly rotate old log files and compress or delete them to free up disk space and keep log files readable. It can also create new log files during rotation to ensure that the system can still log important events and error information.

logrotate is designed to simplify the management of log files on systems that generate large amounts of log files. logrotate allows automatic rolling, compression, deletion and mailing of log files. logrotate can be set to process log files hourly, daily, weekly, monthly, or when the log file reaches a certain size. Therefore, logrotate is very important to maintain the stability and reliability of the system. It ensures that system administrators can detect and resolve potential problems in a timely manner and avoid performance issues caused by excessively large log files.

2. Basic usage of logrotate

logrotate [OPTION...] <configfile>
-d, --debug: debug mode, testing the configuration file for errors.
-f, --force: Force dump file.
-m, --mail=command: After compressing the log, send the log to the specified mailbox.
-s, --state=statefile: Use the specified state file.
-v, --verbose: Display the dump process.

The best option during troubleshooting is to run logrotate in staging mode using the -d option. To verify, without actually polling any log files, you can simulate drill log rotation and display its output.

logrotate -d configuration file

Even if the rotation conditions are not met, we can force logrotate to rotate the log file by using the -f option. The -v parameter provides detailed output.

logrotate -vf configuration file

3. logrotate operating mechanism

The operation of logrotate depends on the scheduled task command crontab on Linux: Linux crontab command

The crontab command will run logrotate regularly, usually once a day. crontab will execute the script in the /etc/cron.daily directory regularly every day, and there is a file called logrotate in this directory.

[root@ceph01 cron.daily]# cat logrotate
#!/bin/sh

/usr/sbin/logrotate /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
    /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit $EXITVALUE

Logrotate is installed by default on the Linux system, and its configuration file is:

/etc/logrotate.conf
/etc/logrotate.d/

Log rotation is done automatically by the system. When actually running, logrotate will call the configuration file /etc/logrotate.conf. You can place a customized configuration file in the /etc/logrotate.d/ directory to override the default value of logrotate.

View the /etc/logrotate.conf file

[root@ceph01 ceph]# cat /etc/logrotate.conf
# see "man logrotate" for details
# rotate log files weekly
weekly # By default, the rotate task is executed once a week

# keep 4 weeks worth of backlogs
rotate 4 # How many log files to keep, 4 are kept by default, 0 means no backup

# create new (empty) log files after rotating old ones
create # Automatically create a new log file. The new log file has the same permissions as the original file; because the log has been renamed, a new one must be created to continue to store the previous log.

# use date as a suffix of the rotated file
dateext # After cutting, the log file ends with the current date as the suffix, such as xxx.log-20230912. If commented out, the cut will be in numerical increment, that is, in the format of xxx.log-1

# uncomment this if you want your log files compressed
#compress # Whether to dump future log files through gzip compression, such as xxx.log-20131216.gz; if compression is not required, comment out

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d # Load all files in the /etc/logrotate.d/ directory

# system-specific logs may be also be configured here.

logrotate parameters

compress compresses the dumped logs via gzip
nocompress does not perform gzip compression processing
copytruncate is used to back up and truncate the current log file for a log file that is still open; it copies first and then clears it. There is a time difference between copying and clearing, and some log data may be lost.
nocopytruncate backs up the log file but does not truncate it
create mode owner group specifies the attributes to create new files during rotation, such as create 0777 nobody nobody
nocreate does not create a new log file
When delaycompress and compress are used together, the dumped log file will not be compressed until the next dump.
nodelaycompress overrides the delaycompress option, dumping is compressed simultaneously.
missingok If the log is missing, continue scrolling to the next log without reporting an error.
errors address Error messages during storage are sent to the specified email address
ifempty Rotates the log file even if it is empty. This is the default option of logrotate.
notifempty When the log file is empty, no rotation is performed
mail address Send the dumped log file to the specified e-mail address
nomail does not send log files when dumping
olddir directory The dumped log file is placed in the specified directory, which must be in the same file system as the current log file.
noolddir The dumped log file and the current log file are placed in the same directory.
sharedscripts runs the postrotate script, which is used to execute the script once after all logs are rotated. If this is not configured, the script will be executed after each log rotation
prerotate Instructions that need to be executed before logrotate dumps, such as modifying file attributes and other actions; must be in separate lines
postrotate Instructions that need to be executed after the logrotate dump, such as restarting (kill -HUP) a service! Must be independent
daily specifies the dump period to be every day
weekly specifies that the dump period is weekly
monthly specifies the dump period as monthly
rotate count specifies the number of dumps before deleting the log file, 0 means no backup, 5 means retaining 5 backups.
dateext uses the current date as the naming format
dateformat %Y%m%d%H.%s is used with dateext. It appears immediately on the next line and defines the file name after the file is cut. It must be used with dateext. Only the four parameters %Y %m %d %s are supported.
size (or minsize) log-size Dump only when the log file reaches the specified size. log-size can specify bytes (default) and KB (sizek) or MB (sizem).
Dump when the log file >= log-size. The following is the legal format: (I have not tried the unit cases in other formats)
size = 5 or size 5 (dump if >= 5 bytes)
size = 100k or size 100k
size = 100M or size 100M

4. How does logrotate achieve rolling logs without affecting the normal log output of the program?

Linux file operation mechanism

Detailed explanation, Linux kernel – file system (recommended collection) – directory storage

Directories are also files, and directory files store file names and corresponding inode numbers. The metadata and file contents of the file can be found through this inode number. The metadata of a file includes reference count, operation permissions, owner ID, creation time, last modification time, etc. The file name is not in the metadata but in the directory file. Therefore, renaming or moving files will not modify the file, but modify the directory file.

Every time a process opens a new file, the system will allocate a new file descriptor to the file. The file descriptor corresponds to a file table. The table stores the status information of the file (O_APPEND/O_CREAT/O_DIRECT…), the current file location and the inode information of the file. The system will create independent file descriptors and file tables for each process, and different processes will not share the same file table. Because of this, different processes can operate on different locations of the same file in different states at the same time. The file table stores inode information rather than file paths, so changes in file paths will not affect file operations.

Option 1

The idea of this solution is to rename the original log file and create a new log file. The detailed steps are as follows:

1. Rename the program currently outputting logs. Because renaming will only modify the contents of the directory file, and the process operates files based on the inode number, it does not affect the program’s continued output of logs.
2. Create a new log file with the same file name as the original log file. Although the name of the new log file is the same as the original log file, the inode number is different, so the log output by the program is still output to the original log file.
3. Notify the program in some way to reopen the log file. The program reopens the log file based on the file path rather than the inode number, so a new log file is opened.

How can I notify the program to reopen the log? The simple and crude method is to kill the process and reopen it. In many scenarios, this approach will affect online services, so some programs provide an interface to reopen the log, for example, nginx can be notified through signals. Various IPC methods are available, provided that the program itself supports this function.

It is worth mentioning that a program may output multiple log files that need to be rolled. Notifying the program to reopen all log files every time it scrolls is not cost-effective. There is a sharedscripts parameter that allows the program to notify you only once after renaming all logs.

Option 2

If the program does not support the function of reopening the log, and the program cannot be restarted rudely, how to roll the log? The copytruncate solution appears.

The idea of this solution is to copy the log being output and then clear (trucate) the original log. The detailed steps are as follows:

1. Copy the log file currently being output by the program and save the file name as the rolling result file name. During this period, the program outputs the log to the original file as usual, and the original file name has not changed.
2. Clear the log file that the program is outputting. After clearing, the log output by the program is still output to this log file, because clearing the file only deletes the content of the file, and the inode number of the file does not change. What changes is the file content information in the metainformation.
   As a result, the old log content is stored in the rolling file, and the new log is output to the empty file. Implemented log rolling.

There are two interesting things about this solution:

1. File clearing does not affect the file location information in the file table of the program that outputs the log, because the file table of each process is independent. Then after the file is cleared, the log output by the program should be output at the offset position of the previous log. This position will be filled with \0 before. But in fact, after logrotate clears the log file, the logs output by the program are written from the beginning of the file. how did you do that? I struggled with this problem for a long time, until one day I suddenly had a flash of inspiration. This was not done by logrotate, but by a mature way of writing logs, all written in the O_APPEND way. If the program does not use the O_APPEND method to open the log file, the front of the log file will be filled with a bunch of \0 after copytruncate.
2. During the period from when the log is copied to when the file is cleared, the logs output by the program are cleared without backup. Aren’t these logs lost? Yes, copytruncate runs the risk of losing part of the log content. So don’t use copytruncate if you can use create. Therefore, many programs provide the function of notifying me to update the open log file to support the create scheme, or do log rolling by themselves without relying on logrotate.

5. Logrotate practice – Ceph log dump

Take Ceph’s log file /var/log/ceph/*.log as an example:

Ceph’s log rolling is also implemented based on logrotate. The log configuration file is located at /etc/logrotate.d/ceph. The default log configuration file is as follows:

[root@ceph01 logrotate.d]# cat ceph
/var/log/ceph/*.log {<!-- -->
    rotate 7
    daily
    compress
    sharedscripts
    postrotate
        killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror || pkill -1 -x "ceph-mon|ceph-mgr|ceph-mds|ceph-osd|ceph -fuse|radosgw|rbd-mirror" || true
    endscript
    missingok
    notifempty
    su root ceph
}

By default, backup is performed once a day. To achieve daily backup, it depends on the crontab scheduled task. In the /etc/cron.daily/ directory, there is a logrotate script that is executed regularly every day. You can also move the script to the /etc/cron.hourly/ directory to do it. Log dumps are performed hourly.

What should I do if I want to dump according to the log file size?

1. Delete the logrotate script in the /etc/cron.daily/ directory (make a backup before deleting)

2. Modify the /etc/logrotate.d/ceph file, and set the parameters in it according to actual needs. Here I set the log to be dumped 7 times before deletion. The log file size exceeds 1M, and the time format is renamed to compress the dump. .

   [root@ceph01 etc]# cat /etc/logrotate.d/ceph
   /var/log/ceph/*.log {<!-- -->
       rotate 7
       size 1M
       dateext
       dateformat %Y%m%d%H.%s
       compress
       sharedscripts
       postrotate
           killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror || pkill -1 -x "ceph-mon|ceph-mgr|ceph-mds|ceph-osd|ceph -fuse|radosgw|rbd-mirror" || true
       endscript
       missingok
       notifempty
       su root ceph
   }
   

3. Add a custom scheduled task in /etc/crontab (see the link above for the meaning of the parameters). What is set here is to check whether the log file size exceeds 1M every minute. If it exceeds, execute the logrotate command to dump the log.

   [root@ceph01 etc]# cat /etc/crontab
   SHELL=/bin/bash
   PATH=/sbin:/bin:/usr/sbin:/usr/bin
   MAILTO=root
   
   # For details see man 4 crontabs
   
   # Example of job definition:
   # .---------------- minute (0 - 59)
   # | .------------- hour (0 - 23)
   # | | .---------- day of month (1 - 31)
   # | | | .------- month (1 - 12) OR jan,feb,mar,apr ...
   # | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
   # | | | | |
   # * * * * * user-name command to be executed
   * * * * * root /usr/sbin/logrotate /etc/logrotate.conf > /dev/null 2> & amp;1
   

Reference for some places that may confuse you: logrotate details and pitfalls

Reference

• https://www.cnblogs.com/txlsz/p/13126723.html
• https://zhuanlan.zhihu.com/p/90507023
• https://www.cnblogs.com/sailrancho/p/4784763.html
• https://blog.csdn.net/sinat_36358653/article/details/107390349
• https://blog.csdn.net/ht9999i/article/details/119001732
• https://blog.csdn.net/sinat_36358653/article/details/107390349