Export onnx file Use script directly import torch from mmdet.apis import init_detector, inference_detector config_file = ‘./configs/yolo/yolov3_mobilenetv2_8xb24-ms-416-300e_coco.py’ checkpoint_file = ‘yolov3_mobilenetv2_mstrain-416_300e_coco_20210718_010823-f68a07b3.pth’ model = init_detector(config_file, checkpoint_file, device=’cpu’) # or device=’cuda:0′ torch.onnx.export(model, (torch.zeros(1, 3, 416, 416),), “yolov3.onnx”, opset_version=11) The exported onnx structure is as follows: The output is the output of three different levels of detection heads. If you […]
Tag: inference
[NLP] DeepSpeed-FastGen: High-throughput text generation for LLM through MII and DeepSpeed-Inference
1. Introduction Large language models (LLMs) such as GPT-4 and LLaMA have become dominant workloads serving AI applications at all levels. From general chat models to document summarization, from self-driving to co-piloting at every layer of the software stack, the need to deploy and serve these models at scale has skyrocketed. While frameworks such as […]
Slice-assisted super-inference for small object detection
Small object detection refers to the identification and localization of objects of relatively small size in digital images. These objects often have limited spatial extent and low pixel coverage, and can be difficult to detect due to their small appearance and low signal-to-noise ratio. Small target detection has many applications: Surveillance and Security: Identify and […]
Performing Inference In INT8 Precision
Performing Inference In INT8 Precision Table Of Contents Description How does this sample work? Configuring the builder to use INT8 without the INT8 calibrator Configuring the network to use custom dynamic ranges and set per-layer precision Building the engine Running the engine TensorRT API layers and ops Preparing sample data Running the sample Sample –help […]
OpenMMlab exports the mobilenet-v2 model and uses onnxruntime and tensorrt for inference
Export onnx file Use mmpretrain to export the onnx model of mobilenet-v2: import torch from mmpretrain import get_model model = get_model(‘mobilenet-v2_8xb32_in1k’, pretrained=’mobilenet_v2_batch256_imagenet_20200708-3b2dc3af.pth’, device=’cpu’) input = torch.zeros(1, 3, 224, 224) out = model(input) torch.onnx.export(model, input, “mobilenet-v2.onnx”, opset_version=11) If mmdeploy is installed, you can export it as follows: from mmdeploy.apis import torch2onnx from mmdeploy.backend.sdk.export_info import export2SDK img […]
LLM – GPU computing power evaluation during training and inference
Table of Contents I. Introduction 2. FLOPs and TFLOPs ◆ FLOPs [Floating point Opearation Per Second] ◆ TFLOPs [Tera Floating point Operation Per Second] 3. GPU consumption during training phase ◆ Factors affecting training ◆ GPT-3 training statistics ◆ Custom training GPU evaluation 4. GPU consumption during inference phase ◆ Factors affecting reasoning ◆ Custom […]
TypeScript (1) Type declaration, type inference, union type, interface, function, type assertion, type alias, enumeration
Table of Contents (1) Introduction 1.The difference between JavaScript and TypeScript 2. Advantages of TypeScript (2) Used in vue3 project (3) Type declaration 1.Basic data types (1)string (2) number (3)boolean (4)null and undefined 2. Reference data type (1)Array array (2) Tuple tuple (3)Object object 3.any and void types (1)any (2)void 4. Use typeof to determine […]
Ubuntu20 runs SegNeXt code to extract road water bodies (4) – successfully solved the problem of iou of 0 in training and inference of your own data set! !
In this blog post of mineUbuntu20 runs SegNeXt code to extract road water bodies (3) – SegNeXt training and inference on your own data set After a series of configurations The iou calculation is 0 After many attempts Finally, I tried out the correct configuration method! For specific configuration details, please see this article 1. […]
RK3588 uses npu to run onnx model inference
Article directory Preface 1.Install rknn-toolkit2 1.1. Create a virtual environment named rknn-toolkit2 1.2. Download [[rknn-toolkit2]](https://github.com/rockchip-linux/rknn-toolkit2/releases) 1.3. Open the console in this folder, activate the conda environment created earlier, and execute install 1.4. Verify installation 2. Convert onnx model to rknn model 3. Load the rknn model through rknpu2 to perform inference Foreword Let’s talk about […]
YOLOv5 onnx \tensorrt inference
1. Convert yolov5 pt model to onnx code: https://github.com/ultralytics/yolov5 python export.py –weights yolov5s.pt –include onnx 2. onnx reasoning import os import cv2 import numpy as np import onnxruntime import time CLASSES = [‘person’, ‘bicycle’, ‘car’, ‘motorcycle’, ‘airplane’, ‘bus’, ‘train’, ‘truck’ , ‘boat’, ‘traffic light’, ‘fire hydrant’, ‘stop sign’, ‘parking meter’, ‘bench’, ‘bird’, ‘cat’, ‘dog’, ‘horse’ […]