It’s such a simple thing, but I worked on it for several days for no reason. I was very depressed, so I just started without saying much.
First, you must have a data set in YoLo format. Starting from the simplest, you need image data and corresponding txt tag data of the yolo type. (The first folder in the screenshot below stores pictures, and the naming rule uses pure numbers. The latter folder is a txt file, corresponding to the detection label of each picture).
After having these two folders, the first step is to divide the data set. Divide the image dataset into 8:2. The code used is as follows:
import os
import shutil
import random
from tqdm import tqdm
"""
The annotation file is in yolo format (txt file)
Training set: Validation set (8:2)
"""
def split_img(img_path, label_path, split_list):
try: #Create dataset folder
Data = 'D:\datasets\small-object-detection-datasets\Solar_1103\ImageSets'
#os.mkdir(Data)
train_img_dir = Data + '/images/train'
val_img_dir = Data + '/images/val'
# test_img_dir = Data + '/images/test'
train_label_dir = Data + '/labels/train'
val_label_dir = Data + '/labels/val'
# test_label_dir = Data + '/labels/test'
#Create folder
os.makedirs(train_img_dir)
os.makedirs(train_label_dir)
os.makedirs(val_img_dir)
os.makedirs(val_label_dir)
#os.makedirs(test_img_dir)
# os.makedirs(test_label_dir)
except:
print('The file directory already exists')
train, val = split_list
all_img = os.listdir(img_path)
all_img_path = [os.path.join(img_path, img) for img in all_img]
# all_label = os.listdir(label_path)
# all_label_path = [os.path.join(label_path, label) for label in all_label]
train_img = random.sample(all_img_path, int(train * len(all_img_path)))
train_img_copy = [os.path.join(train_img_dir, img.split('')[-1]) for img in train_img]
train_label = [toLabelPath(img, label_path) for img in train_img]
train_label_copy = [os.path.join(train_label_dir, label.split('')[-1]) for label in train_label]
for i in tqdm(range(len(train_img)), desc='train ', ncols=80, unit='img'):
_copy(train_img[i], train_img_dir)
_copy(train_label[i], train_label_dir)
all_img_path.remove(train_img[i])
val_img = all_img_path
val_label = [toLabelPath(img, label_path) for img in val_img]
for i in tqdm(range(len(val_img)), desc='val ', ncols=80, unit='img'):
_copy(val_img[i], val_img_dir)
_copy(val_label[i], val_label_dir)
def _copy(from_path, to_path):
shutil.copy(from_path, to_path)
def toLabelPath(img_path, label_path):
img = img_path.split('')[-1]
label = img.split('.tiff')[0] + '.txt'
return os.path.join(label_path, label)
if __name__ == '__main__':
#Change to your own image path
img_path = 'D:\datasets\small-object-detection-datasets/aluminum_1103\JPEGImages'
#Modify to your own label path
label_path = 'D:\datasets\small-object-detection-datasets/aluminum_1103\labels'
split_list = [0.8, 0.2] # Data set division ratio [train:val]
split_img(img_path, label_path, split_list)
ok, now there is an additional folder
The structure inside is as follows:
| imageSets | |||
| images | |||
| train | The images of the training set are stored in it | ||
| val | Inside Store images of the verification set | ||
| labels | |||
| train | The labels of the training set are stored in it | ||
| val | The tags that store the verification set |
Next, train.txt and val.txt are generated (these two files are the files where yolov5 reads the image path). The code is as follows:
import os
#TODO: What needs to be changed is the three path addresses. Because train.txt and val.txt are to be generated, they need to be executed twice.
#Specify the path to the B folder
folder_B = 'D:\datasets\small-object-detection-datasets\Solar_1103\ImageSets\images/train'
# Get all image files in folder B
image_files = [f for f in os.listdir(folder_B) if f.lower().endswith(('.jpg', '.jpeg', '.png', '.gif', '.bmp', '. tiff'))]
#Create the train.txt file and write the image file name
with open('D:\datasets\small-object-detection-datasets\Solar_1103/train.txt', 'w') as file:
for image_file in image_files:
# image_files_myhouzhui = image_file[:6] + ".txt" #Split suffix
file.write("D:\datasets\small-object-detection-datasets\Solar_1103\JPEGImages/" + image_file + '\
')
print('train.txt file has been created and the image file name has been written')
After this you have five files:
The solution is the last step to generate the json file, run the yolo to COCO script, the code is as follows:
import os
import json
import random
import time
from PIL import Image
import csv
#TODO: The following four paths need to be modified accordingly. In addition, the contents of the class_names list should be modified to the categories of your own data.
coco_format_save_path = 'D:\datasets\small-object-detection-datasets\Solar_1103' # The folder where the standard coco format labels to be generated are located
yolo_format_annotation_path = 'D:\datasets\small-object-detection-datasets\Solar_1103\ImageSets\labels/val' # The folder where the yolo format labels are located
img_pathDir = 'D:\datasets\small-object-detection-datasets\Solar_1103\JPEGImages' # Folder where the picture is located
val_img_pathDir = 'D:\datasets\small-object-detection-datasets\Solar_1103\ImageSets\images/val'
# Category settings
categories = []
class_names = ['bl', 'ce', 'ch', 'cr', 'hi', 'ir', 'oi', 'pe'] #Need to change to your own category
for label in class_names:
categories.append({'id': class_names.index(label), 'name': label, 'supercategory': ""})
write_json_context = dict() # Write a large dictionary to the .json file
write_json_context['licenses'] = [{'name': "", 'id': 0, 'url': ""}]
write_json_context['info'] = {'contributor': "", 'date_created': "", 'description': "", 'url': "", 'version': "", 'year': ""}
write_json_context['categories'] = categories
write_json_context['images'] = []
write_json_context['annotations'] = []
# The following code mainly adds the key values of 'images' and 'annotations'
imageFileList = os.listdir(img_pathDir)
#Sort by numbers
def sort_list_by_number(strings):
def extract_number(s):
# Extract the numeric part from the string
return int(''.join(filter(str.isdigit, s)))
# Sort the string list using a custom sort key
sorted_strings = sorted(strings, key=extract_number)
return sorted_strings
imageFileList = sort_list_by_number(imageFileList)
'''
Create a new list with only val folders for secondary judgment! !
'''
val_imageFile = os.listdir(val_img_pathDir)
val_imageFile = sort_list_by_number(val_imageFile)
# Traverse all files in the folder and add all file names to the list
img_id = 0 #Image number
anno_id = 0 # mark the label
for i, imageFile in enumerate(imageFileList):
if '_' not in imageFile:
img_id + = 1
if imageFile in val_imageFile:
imagePath = os.path.join(img_pathDir, imageFile) # Get the absolute path of the image
image = Image.open(imagePath) # Read the image
W, H = image.size # Get the height and width of the image
img_context = {} # Use a dictionary to store the image information
# img_name=os.path.basename(imagePath)
img_context['id'] = img_id # Unique ID index for each image
img_context['width'] = W
img_context['height'] = H
img_context['file_name'] = imageFile
img_context['license'] = 0
img_context['flickr_url'] = ""
img_context['color_url'] = ""
img_context['date_captured'] = ""
write_json_context['images'].append(img_context) # Add the image information to the 'image' list
txtFile = imageFile.split('.')[0] + '.txt' # Get the txt file obtained by the image
with open(os.path.join(yolo_format_annotation_path, txtFile), 'r') as fr:
lines = fr.readlines() # Read each line of data from the txt file. lines2 is a list that contains all the annotation information of an image.
for j, line in enumerate(lines):
anno_id + = 1 # The annotated id starts from 1
bbox_dict = {} # Store each bounding box information in the dictionary
class_id, x, y, w, h = line.strip().split(' ') # Get detailed information of each label box
class_id, x, y, w, h = int(class_id), float(x), float(y), float(w), float(h) # Convert string type to calculable int and float types
# Coordinate conversion
xmin = (x - w / 2) * W
ymin = (y - h / 2) * H
xmax = (x + w / 2) * W
ymax = (y + h / 2) * H
w = w * W
h = h * H
height, width = abs(ymax - ymin), abs(xmax - xmin)
#Coordinate information of bounding box
bbox_dict['id'] = anno_id #The index of each annotation information
bbox_dict['image_id'] = img_id # ID index of the current image
bbox_dict['category_id'] = class_id #Category information
bbox_dict['segmentation'] = [[xmin, ymin, xmax, ymin, xmax, ymax, xmin, ymax]]
bbox_dict['area'] = height * width
bbox_dict['bbox'] = [xmin, ymin, w, h] # Note that the target category needs to be increased by one
bbox_dict['iscrowd'] = 0
bbox_dict['attributes'] = ""
write_json_context['annotations'].append(bbox_dict) # Add each bounding box information stored in the dictionary to the 'annotations' list
name = os.path.join(coco_format_save_path, "annotations" + '.json')
with open(name, 'w') as fw: # Write dictionary information into .json file
json.dump(write_json_context, fw, indent=4, ensure_ascii=False)
This way you can get the json file.
Explain this file and explain why this file is needed.
The json file is a description of the validation set in the data set (that is, the picture in the val folder, and the description of its coordinates – there are other things, but it is enough to know this for small target detection indicators).
This file is needed because when you use the val.py file to evaluate APs indicators, a best.json will be generated. That json file contains your detection information for the images in the verification set, which is actually the detection frame you generated. Information, compare best.json and annotations.json to get the detection indicators of APs.
Post a successful picture:
——Note that when calling val.py to execute COCO detection indicators, you need to modify three codes in val.py. I don’t have time today, so I’ll leave it for now (search it yourself, it’s easy to search), and wait until you have time. I’ll post the link, but I’m too lazy to look for it now.
——After the image is annotated, it will be a file in json format. To generate a txt file of yolo, you can refer to my blog to record the txt format label of yolo generated after labelme is annotated – CSDN Blog
——Also note that the folder of ImageSets can be deleted after generating the json file. You do not need to use it if you use yolo to run the detection.