Fire channel occupancy detection system based on YOLOv5 based on GSConv+SlimNeck

1. Research background and significance

Project ReferenceAAAI Association for the Advancement of Artificial Intelligence

The smooth flow of fire escapes is crucial to the safety of personnel and property. However, in real life, due to various reasons, fire exits are often illegally occupied, resulting in fire vehicles being unable to reach the fire scene in time, seriously threatening people’s lives and property. Therefore, it is of great practical significance to develop an efficient and accurate fire escape passage occupancy detection system.

Currently, computer vision technology has made significant progress in the field of target detection. YOLOv5 is a target detection algorithm based on deep learning, which is fast and accurate. However, the traditional YOLOv5 has some problems in fire channel occupancy detection, such as poor detection of small targets and vulnerability to background interference. Therefore, it is of great research significance to improve YOLOv5 and improve its performance in fire passage occupancy detection.

Based on the research background, this study proposes a fire passage occupancy detection system based on YOLOv5 based on GSConv + SlimNeck. The system mainly includes two key technologies: GSConv and SlimNeck.

First of all, GSConv is an improved convolution operation that effectively improves the performance of target detection by introducing the ideas of group sparse convolution and group dense convolution. Traditional convolution operations tend to lose detailed information when processing large-scale targets, while GSConv can better retain the detailed information of the target and improve the accuracy of target detection.

Secondly, SlimNeck is a lightweight feature fusion module that reduces the complexity of the model and improves the running speed of the model by reducing the number of feature channels. In fire channel occupancy detection, since the targets are usually small, too many feature channels will lead to model redundancy and affect the detection effect. SlimNeck can effectively reduce the number of feature channels and improve the detection speed and accuracy of the model.

2. Picture demonstration

3. Demonstration video

Fire channel occupancy detection system based on YOLOv5 based on GSConv + SlimNeck_bilibili_bilibili

4. System advantages

First, the system can improve the accuracy of target detection. By introducing the GSConv operation, the detailed information of the target can be better retained, the accuracy of target detection can be improved, and the false detection rate and missed detection rate can be reduced.

Secondly, the system has a high operating speed. By introducing the SlimNeck module, the number of feature channels can be reduced, the complexity of the model can be reduced, the running speed of the model can be improved, and it is suitable for real-time fire channel occupancy detection.

Finally, the system has good versatility and scalability. Based on the YOLOv5 architecture, this system can be applied to other target detection tasks, such as pedestrian detection, vehicle detection, etc., and has good versatility and scalability.

In summary, the fire channel occupancy detection system based on GSConv + SlimNeck’s YOLOv5 has important research significance and practical application value. Through the development and application of this system, the accuracy and efficiency of fire passage occupancy detection can be improved and the safety of people’s lives and property can be guaranteed. At the same time, the research ideas and methods of this system can also provide reference for the research of other target detection tasks.

5. Core code explanation

5.1 GSConv + Slim Neck.py

# GSConv layer definition
class GSConv(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size, stride=1, padding=0):
        super(GSConv, self).__init__()
        self.main = nn.Conv2d(in_channels, out_channels, kernel_size, stride, padding)
        self.side = nn.Conv2d(in_channels, out_channels, 1)

    def forward(self, x):
        return self.main(x) + self.side(x)


#GSbottleneck definition
class GSbottleneck(nn.Module):
    def __init__(self, in_channels, out_channels):
        super(GSbottleneck, self).__init__()
        mid_channels = out_channels // 2
        self.conv1 = GSConv(in_channels, mid_channels, 1)
        self.conv2 = GSConv(mid_channels, out_channels, 3, padding=1)

    def forward(self, x):
        return self.conv2(self.conv1(x))


# VoV-GSCSP definition
class VoV_GSCSP(nn.Module):
    def __init__(self, in_channels, out_channels):
        super(VoV_GSCSP, self).__init__()
        self.conv = GSbottleneck(in_channels, out_channels)

    def forward(self, x):
        return self.conv(x)


#Slim-Neck
class SlimNeck(nn.Module):
    def __init__(self, in_channels, out_channels):
        super(SlimNeck, self).__init__()
        self.layer = VoV_GSCSP(in_channels, out_channels)

    def forward(self, x):
        return self.layer(x)

This program file is a neural network model for object detection. It contains several custom modules and layers.

The first is the GSConv layer, which is a convolutional layer with primary and auxiliary convolutions. The primary convolution and the auxiliary convolution perform convolution operations on the input respectively, and then add their results as the output.

Next is the GSbottleneck module, which consists of two GSConv layers. The first GSConv layer halves the number of input channels and then outputs it after passing through the second GSConv layer.

The VoV_GSCSP module is a module that uses the GSbottleneck module.

The SlimNeck module is a module that uses the VoV_GSCSP module.

GhostModule is a module with convolution operations with main convolutional and generative features. The main convolution performs a convolution operation on the input, then the convolution operation that generates the features performs a convolution operation on the output of the main convolution, and finally their results are spliced together as the output.

Finally, there is the YOLOv5Optimized module, which is a module using GhostModule and SlimNeck. It defines several GhostModule and SlimNeck layers, uses them for forward propagation, and finally outputs the detection results.

The entire program file defines an optimized YOLOv5 model for target detection tasks.

5.2 check_img.py

The following is the code encapsulated as a class:

class ImageProcessor:
    def __init__(self, path, train_file):
        self.path = path
        self.train_file = train_file

    def process_images(self):
        result = os.listdir(self.path)
        num = 0
        if not os.path.exists(self.train_file):
            os.mkdir(self.train_file)
        for i in result:
            try:
                image = cv2.imread(self.path + '/' + i)
                cv2.imwrite(self.train_file + '/' + 'Compressed' + i, image, [int(cv2.IMWRITE_JPEG_QUALITY), 100])
                num + = 1
            except:
                pass
        print('Data validity verification completed, the number of valid pictures is %d' % num)
        if num == 0:
            print('Your picture name has Chinese characters, it is recommended to unify it as 1 (1).jpg/png')

This class encapsulates the file processing logic in the process_images method, and the constructor __init__ receives the image folder path and output folder path that need to be processed. You can adjust the properties and methods of the class as needed.

The program file name is check_img.py. Its main function is to reduce noise and compress the pictures in the specified folder, and save the processed pictures to another folder.

The program first imported the cv2, numpy and os modules. cv2 is an OpenCV library for image processing; numpy is a numerical calculation library for Python; os is an operating system interface module for Python.

Next, a variable path is defined to store the path of the image folder that needs to be processed. Then use the os.listdir() function to get all the file names in the folder and save the results to the variable result.

A variable train_file is defined, which is used to store the folder path of the processed images. Then a variable num is defined to record the number of valid pictures.

Next, use the if statement to determine if the train_file folder does not exist, and then use the os.mkdir() function to create the folder.

Then use a for loop to iterate through each file name in the result. In the loop, use the cv2.imread() function to read the image data of each file and save the result to the variable image.

Then use the cv2.imwrite() function to save the processed image data to the train_file folder. The file name is ‘Compressed’ + i, that is, add ‘Compressed’ before the original file name.

In the try-except statement, if an exception occurs, the processing of the file is skipped.

Finally, print out the number of valid pictures.

If the number of valid pictures is 0, it is recommended to print out the picture naming as ‘1 (1).jpg/png’.

Summary: The function of this program is to reduce noise and compress the pictures in the specified folder, and save the processed pictures to another folder.

5.3 torch_utils.py

import datetime
import logging
import math
import os
import platform
import subprocess
import time
from contextlib import contextmanager
from copy import deepcopy
from pathlib import Path

import torch
import torch.distributed as dist
import torch.nn as nn
import torch.nn.functional as F
import torchvision

LOGGER = logging.getLogger(__name__)


@contextmanager
def torch_distributed_zero_first(local_rank: int):
    """
    Decorator to make all processes in distributed training wait for each local_master to do something.
    """
    if local_rank not in [-1, 0]:
        dist.barrier(device_ids=[local_rank])
    yield
    if local_rank == 0:
        dist.barrier(device_ids=[0])


def date_modified(path=__file__):
    # return human-readable file modification date, i.e. '2021-3-26'
    t = datetime.datetime.fromtimestamp(Path(path).stat().st_mtime)
    return f'{<!-- -->t.year}-{<!-- -->t.month}-{<!-- -->t.day}'


def git_describe(path=Path(__file__).

This program file is a tool file for PyTorch and contains some commonly used functions and classes. Here are some important parts of the file:

1. torch_distributed_zero_first: A context manager used in distributed training to make all processes wait for each local master process to perform certain operations.

2. date_modified: Returns the modified date of the file.

3. git_describe: Returns the description information of the git warehouse.

4. select_device: Select a device (CPU or GPU).

5. time_sync: Returns the accurate time.

6. profile: used to analyze the speed, memory and FLOPs of the model.

7. is_parallel: Determine whether the model is a parallel model.

8. de_parallel: Convert the model to a single GPU model.

9. intersect_dicts: Returns two words

6. Overall system structure

According to the analysis, overall, this project is a fire passage occupancy detection system based on YOLOv5 of GSConv + SlimNeck. It contains multiple program files, each with different functions, used to implement various parts of the entire system.

The following is a summary of the functions of each file:

td>

td>

< /table>

7.YOLOv5 target detection algorithm

The main improvements of the YOLOv5 algorithm based on YOLOv3 are: (1) Mosaic data enhancement is introduced at the input end, increasing the richness of the data; (2) The ideas in CSPNet are used in Backbone, the CSP structure is added, and The Focus structure is introduced to perform slicing operations on the feature tensor; (3) Multi-scale feature fusion is further improved in Neck, using the FPN structure and PAN structure; (4) GloU_Lossl2l is used as the loss function in Head.
The most special point is: Compared with the YOLO series algorithms mentioned in the previous chapter, YOLOv5 proposes four models: YOLOv5s, YOLOv5m, YOLOv51, and YOLOv5x by flexibly configuring different complexities. These four models have the same model components. , but the depth and width of the network are different. Among them, the CSP structure controls the depth of the network, while the number of convolution kernels in the Focus and CBS structures in the backbone network controls the width of the network. In terms of code implementation, the author controls the depth and width of the network through two scaling parameters: depth_multiple and width_multiple. Depth_multiple controls the number of executions of each module, and width_multiple controls the number of convolution kernels. Through these two parameters, the model design of different complexities of the YOLOv5 algorithm is realized.
The four models of YOLOv5: YOLOv5s, YOLOv5m, YOLOv51, and YOLOv5x have different depths and widths. Specifically, the Focus and CBS structures in the backbone network have different numbers of convolution kernels in the above four models. This difference makes the width of the models different. Among them, the number of convolution kernels in the Focus structure of YOLOv5s for 32. Moreover, the two CSP structures in the YOLOv5 model: CSP1 and CSP2, control the depth of the model. The X in CSP1_X and CSP2_X represents the X residual components used in the CSP structure. In different YOLOv5 models, the value of are different. The larger the value, the deeper the network model. As the network model deepens, the feature extraction and feature fusion capabilities of the network model will become stronger. As shown in Figure 3.3, YOLOv5s uses CSP1_1, CSP1_3, and CSP2_1. The value of X is the smallest among the four models, which means that the depth of YOLOv5s is the shallowest among the four models.

8.YOLOv5 algorithm improvement

GSConv

To speed up the computation of predictions, feed images in CNNs almost have to undergo a similar transformation process in Backbone: spatial information is transferred progressively towards the channels. And each spatial (width and height) compression and channel expansion of the feature map will result in partial loss of semantic information. Dense convolution calculations preserve the hidden connections between each channel to the maximum extent, while sparse convolution completely cuts off these connections.

GSConv preserves these connections whenever possible. But if it is used at all stages of the model, the network layer of the model will be deeper, and the deeper layers will increase the resistance to data flow and significantly increase the inference time. When these feature maps reach Neck, they have become elongated (the channel dimension reaches the maximum and the width and height dimensions reach the minimum) and no longer need to be transformed. So a better option is to use GSConv only on Neck (Slim-Neck + standard Backbone). At this stage, using GSConv to process concatenated feature maps is just right: there is less redundant repeated information, no compression is needed, and attention modules work better, such as SPP and CA.

This defect results in the feature extraction and fusion capabilities of DSC being much lower than that of SC. Excellent lightweight works, such as Xception, MobileNets and ShuffleNets, greatly improve the speed of detectors through DSC operations. But the lower accuracy of these models is concerning when applied to self-driving cars. In fact, these works have proposed some methods to alleviate this inherent defect of DSC (which is also a feature): MobileNets uses a large number of 1×1 dense convolutions to fuse independently calculated channel information; ShuffleNets uses channel shuffle to achieve channel information interactions, while GhostNet uses the halved SC operation to preserve the interaction information between channels. However, 1×1 dense convolution takes up more computing resources, and using the channel shuffle effect still does not touch the results of SC, and GhostNet is more or less back on the road of SC, and the impact may come from many aspects.

Many lightweight models use similar thinking to design the basic architecture: using only DSC from the beginning to the end of a deep neural network. But the flaws of DSC are amplified directly in the backbone, whether used for image classification or detection. The authors believe that SC and DSC can be used together. The feature maps generated only through the output channels of channel shuffle DSC are still “deeply separated”.

SlimNeck

First, the lightweight convolution method GSConv is used instead of SC. Its computational cost is about 60%~70% of SC, but its contribution to model learning ability is comparable to the latter. Then, GSbottleneck is introduced based on GSConv. Figure (a) shows the structure of the GSbottleneck module.

Likewise, a one-time aggregation approach is used to design the cross-level partial network (GSCSP) module VoV-GSCSP. The VoV-GSCSP module reduces the computational and network structure complexity but maintains sufficient accuracy. Figure 5(b) shows the structure of VoV-GSCSP. Notably, if we use VoV-GSCSP instead of Neck’s CSP, where the CSP layers consist of standard convolutions, FLOPs will be reduced by 15.72% on average compared to the latter.

Finally, three modules need to be used flexibly, GSConv, GSbottleneck and VoV-GSCSP.
YOLO series detectors are more widely used in the industry due to their high detection efficiency. The slim-neck module is used here to transform the Neck layer of Scaled-YOLOv4 and YOLOv5. The figure shows 2 slim-neck architectures.

8. Overall algorithm framework

With the goal of identifying fire truck passages and associated vehicles, it is necessary to detect fire truck passages and vehicles on fire truck passages. Because the difficulty, quantity, and quality of obtaining the fire truck passage data set are quite different from the vehicle data set, the experiment adopts a data training scheme for two detection subjects, and finally the detection results are merged to determine whether the fire truck passage is occupied.
The overall framework of the algorithm is shown in the figure. First, input the image and use the compensation module to process the input image. Then send the image to the detection network to obtain the detection results of the fire truck channel and the vehicle respectively. Since not all vehicles are above the fire truck channel area, we also designed In a screening stage, the vehicle information that has no intersection with the fire truck channel in the detection results is removed, and then the final result of whether occupancy occurs is obtained based on the proposed fire truck channel occupancy determination rule.

9. System integration

The complete source code & environment deployment video tutorial & custom UI interface shown below

Reference blog “Personnel Fall Detection System Based on OpenCV (Source Code & Deployment Tutorial)”

10. References

[1] Zheng Wang. Application of computer information technology in intelligent transportation systems [J]. Science and Informatization. 2022, (1).

[2] He Jian, Liu Xinyuan. Ground obstacle detection technology based on fusion of RGB-D and inertial sensors [J]. Journal of Computer-Aided Design and Graphics. 2022, 34(2). DOI: 10.3724/SP.J.1089.2022. 18870.

[3] Huang Kai, Li Xuewei, Xu Lei. Research on the impact of fire products on personnel evacuation based on FDS [J]. Journal of Dalian University for Nationalities. 2021, (3). DOI: 10.3969/j.issn.1009-315X.2021.03 .011.

[4] Yang Ayong, Zhao Jincheng, Hua Ying, et al. Research on evacuation of building fires considering the impact of pedestrian flow in corridors [J]. Computer Simulation. 2021, (1).

[5] Wu Hao, Ding Yuanchun, Weng Falu. Research on the impact of obstacles in shopping mall passages on personnel evacuation [J]. Technology and Innovation. 2021, (2). DOI: 10.15913/j.cnki.kjycx.2021.02.011.

[6] Wang Shouqi. Analysis of factors affecting personnel safety evacuation [J]. Science and Technology Wind. 2021, (12). DOI: 10.19392/j.cnki.1671-7341.202112059.

[7] Xu Degang, Wang Lu, Li Fan. A review of research on typical target detection algorithms of deep learning [J]. Computer Engineering and Applications. 2021, (8). DOI: 10.3778/j.issn.1002-8331.2012-0449.

[8] Fang Rongya, Tian Siao, Yang Yapu. Research on influencing factors of fire and emergency evacuation simulation of university student apartments [J]. Building Safety. 2020, (8).

[9] Zheng Yayu, Wang Jihao, Feng Jie. Detection algorithm of illegally parked vehicles on pedestrian paths based on improved instance segmentation network [J]. High-tech Communications. 2020, (6). DOI: 10.3772/j.issn.1002-0470.2020.06.003 .

[10] Liu Huimin, Guan Heng, Yu Minghe, et al. Research and implementation of a multi-feature fusion vehicle tracking algorithm [J]. Small Microcomputer Systems. 2020, (6).

File path	Function overview
check_img.py	Denoise and compress images in the specified folder
demo.py	Use YOLOv5 model for target detection
GSConv + Slim Neck.py	Defines modules and layers such as GSConv and SlimNeck
Interface.py	Load the model and perform target detection, draw the bounding box and display the image
location.py	Display the image and allow the user to pass Click the mouse to get the coordinates on the image
torch_utils.py	Contains some PyTorch utility functions and classes
train.py	Train YOLOv5 model
ui.py	User interface related functions
models\common.py	Contains some common model components
models\experimental.py	Contains some experimental model components
models\tf.py	Contains some TensorFlow-related models Component
models\yolo.py	Contains the definition of YOLOv5 model
models_< em>init_.py	Model-related initialization files
tools\activations.py	Contains Contains some activation functions
tools\augmentations.py	Contains some data enhancement methods
tools\autoanchor.py	Contains the method of automatic anchor box generation
tools\autobatch.py	Contains
tools\callbacks.py	Contains some callback functions
tools\datasets.py	Contains data set processing methods
tools\downloads.py	Contains Provides some methods for downloading data sets
tools\general.py	Contains some common tool functions
tools\loss.py	Contains some loss functions
tools\metrics.py	Contains some evaluation indicators
tools\plots.py	Contains some plotting functions
tools\torch_utils.py	Contains some PyTorch utility functions
tools_init_.py	Tool-related initialization files
tools\aws\resume.py	Contains the model recovery method on AWS
tools\aws_init_.py	AWS related initialization files
tools\flask_rest_api\example_request.py	Contains an example request for the Flask REST API
tools\flask_rest_api\restapi.py	Contains the implementation of Flask REST API
tools\loggers_init_.py	Log Logger related initialization file
tools\loggers\wandb\log_dataset.py	Contains the data set log method of WandB logger
tools\loggers\wandb\sweep.py	Contains the hyperparameter search method of WandB logger
tools\loggers\wandb\wandb_utils.py	Contains the utility functions of WandB logger
tools\loggers\wandb_init_.py	WandB logger related initialization file
utils\ activations.py	Contains some activation functions
utils\augmentations.py	Contains some data enhancement methods
utils\autoanchor.py	Contains the method of automatic anchor box generation
utils\ autobatch.py	Contains automatic batch processing methods
utils\callbacks.py	Contains some callback functions
utils\datasets.py	Contains data set processing methods
utils\ downloads.py	Contains some methods for downloading data sets
utils\general.py	Contains some common methods Utility functions
utils\loss.py	Contains some loss functions
utils\ \metrics.py	Contains some evaluation metrics
utils\plots.py	Contains some plotting functions
utils\torch_utils.py	Contains some PyTorch utility functions
utils_ init_.py	Tool-related initialization files
utils\aws\resume.py	Contains model recovery methods on AWS
utils\aws_init_.py	AWS-related initialization files
utils\flask_rest_api\example_request.py	Contains an example request for the Flask REST API