Deployment of yolov8 tensorrt model under Jetson Xavier NX (Jetpack5.1.2, CUDA11.4, Cudnn8.6.0, Tensorrt8.5.2)

Article directory

  • Preface
  • Jetson Xavier NX environment configuration
  • 1. TensorRT-Alpha source code download
    • 1. Source code download
    • 2.File settings
  • 2. Yolov8 model deployment
    • 1. Export yolov8 onnx model
    • 2. Use tensorrt to convert onnx files to trt files
    • 3. Source code modification
    • 4.Compile
    • 5. Run
  • Summarize
    • refer to

Foreword

Records of the deployment process and problem handling of the yolov8 tensorrt model under Jetson Xavier NX. This article uses the TensorRT-Alpha packaging library, which is based on tensorrt + cuda to implement GPU acceleration of the model.

Jetson Xavier NX environment configuration

Jetpack5.1.2, CUDA11.4, Tensorrt8.5.2, OpenCV 4.5.4, Ubuntu 20.04

1. TensorRT-Alpha source code download

1. Source code download

The code of TensorRT-Alpha can be downloaded directly from the github official website. The source code download address is https://github.com/FeiYull/TensorRT-Alpha. The code cloning instructions under Linux are as follows

$ git clone https://github.com/FeiYull/tensorrt-alpha

2. File settings

$ cd tensorrt-alpha/cmake
$ vim common.cmake

Replace the tensorrt path on line 20 in the file with the path in your development board. The general default path is: /usr/src/tensorrt
The path is set as follows:

set (TensorRT_ROOT /usr/src/tensorrt)

2. Yolov8 model deployment

1. Export yolov8 onnx model

Place your trained weights best.pt or the downloaded official pre-trained weights in the ultralytics-main home directory on the server or host, and enter the conda environment for training yolov8.
(If you have installed onnx version 1.12.0 and above and onnx-simplifier version 0.4.8 and above, please skip the installation below)

$ pip install onnx==1.12.0
$onnx-simplifier==0.4.8

Then export the model. The official code of yolov8 contains a model export command, so use the following command to export the model

$ yolo export model=best.pt format=onnx opset=12 dynamic=True simplify=True int8=True
# opset=12 default value
# dynamic=True The exported dimension is dynamic, which is an uncertain state. If not needed, it can be set to False
# simplify=True Simplify the model. It is recommended to set it to True. The performance of the development board is limited, so simplifying the model is necessary and can avoid some subsequent errors.
# int8=True INT8 quantization model, enable as needed

2. Use tensorrt to convert onnx files to trt files

#Place the onnx file exported in the previous step under the tensorrt-alpha/data/yolov8 folder of git on the development board just now

$ cd /tensorrt-alpha/data/yolov8

#Declare the tensorrt path used
#(Maybe it can be used without declaring it, because I tried to deploy it on the Linux server first, but there is no lib folder on the development board)
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/src/tensorrt/lib

$ /usr/src/tensorrt/bin/trtexec --onnx=best.onnx --saveEngine=best.trt --minShapes=images:1x3x640x640 --optShapes=images:4x3x640x640 --maxShapes=images:8x3x640x64

Due to the limited performance of the development board, the following warning message may appear, but you will still get the trt file in the end.

3. Source code modification

If deploying using official weights, please skip this step

This step can also be done after the modification is completed on the host computer, and then the corresponding files in the development board can be replaced.

$ cd tensorrt-alpha/yolov8
$ vim app_yolov8.cpp
#Modify the number of categories in line 8 to the number of categories in your own training set, for example:
initParameters.num_class = 2;

#Modify the header files and functions used
$ cd tensorrt-alpha/untils
$ vim untils.h

#Modify line 37 in the untils.h file and replace it with your own category, such as:
const std::vector<std::string> mask2 = {<!-- -->"non-mask", "mask"};
#After line 80 of the untils.h file, add your own category color, such as:
const std::vector<cv::Scalar> color2{<!-- -->
            cv::Scalar(148, 80, 189),cv::Scalar(105, 52, 238)
        };

#Modify untils.cpp file
$ vim untils.cpp

#Modify the show function in the untils.cpp file and add the code corresponding to your own category after line 144, such as:
if (classNames.size() == 2) // mask2
{<!-- -->
color = Colors::color2[box.label];
}
#Modify the save function in the untils.cpp file and add the code corresponding to your own category after line 198, such as:
if (classNames.size() == 2) // mask2
{<!-- -->
color = Colors::color2[box.label];
}

At this point, the source code has been modified and can be compiled.

4. Compile

$ cd tensorrt-alpha/yolov8
$ mkdir build
$ cd build
$ cmake ..
$ make -j10

If an error is reported after cmke, it may be that the cuda file cannot be found. You can declare it first and then compile it.

$ export PATH=/usr/local/cuda-11.4/bin/:$PATH

The following error may occur after executing the make -j10 command

/usr/bin/ld: libyolov8.so: undefined reference to `sample::splitToStringVec(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const & amp;, char)'
collect2: error: ld returned 1 exit status
make[2]: ***[CMakeFiles/app_yolov8.dir/build.make:161:app_yolov8] Error 1
make[1]: *** [CMakeFiles/Makefile2:78: CMakeFiles/app_yolov8.dir/all] Error 2
make: ***[Makefile:84:all] Error 2

This is caused by the tensorrt version being too high. The above error may not occur in the tensorrt8.4 version.
I am using tensorrt8.5.2 on my board. The splitToStringVec function is not defined in the sampleOptions.cpp file under this version of tensorrt and the above error will occur. Therefore, the following modifications need to be made
First, go to the /usr/src/tensorrt/samples/common folder, open the terminal, and execute the following command

$ sudo vim sampleUtils.h

In line 77, you can see a declaration about the splitToStringVec function, but it is not defined, so comment it out, as shown below:

// std::vector<std::string> splitToStringVec(std::string const & amp; option, char separator);

Then save and exit (Esc + shift: + wq), open the sampleOptions.cpp file, and add the definition of the splitToStringVec function here

$ sudo vim sampleOptions.cpp

#Add the following code in line 38 of the file, which is the namespace

std::vector<std::string> splitToStringVec(const std::string & amp; option, char separator)
{<!-- -->
    std::vector<std::string> options;

    for (size_t start = 0; start < option.length();)
    {<!-- -->
        size_t separatorIndex = option.find(separator, start);
        if (separatorIndex == std::string::npos)
        {<!-- -->
            separatorIndex = option.length();
        }
        options.emplace_back(option.substr(start, separatorIndex - start));
        start = separatorIndex + 1;
    }

    return options;
}

Then save and exit, and execute the make -j10 command again under the tensorrt-alpha/yolov8/build file to compile successfully.

5. Run

Continue to the previous step and execute the following command directly in the terminal

$ ./app_yolov8 --model=../../data/yolov8/best.trt --size=640 --batch_size=1 --img=../../data/bus.jpg - -show

$ ./app_yolov8 --model=../../data/yolov8/best.trt --size=640 --batch_size=1 --video=../../data/test.avi --show - -savePath=../

Other running instructions are available in the TensorRT-Alpha repository

Summary

This blog introduces the deployment of the yolov8 model that I trained on the Jetson Xavier NX embedded development board, and records the problems I encountered during the deployment process. Thank you for reading to the end.

Reference

yolov8
TensorRT-Alpha
yolov8 tensorrt model deployment under linux