Deep Neural Networks – Convert TensorFlow Classification Model and Release with OpenCV Python OpenCV v4.8.0

Original author	Anastasia Murzova
Compatibility	OpenCV >= 4.5

Goals

In this tutorial you will learn how to

• Get a frozen graph of a TensorFlow (TF) classification model
• Run the converted TensorFlow model using the OpenCV Python API
• Get evaluation results for TensorFlow and OpenCV DNN models

We will discuss the above points using the MobileNet architecture as an example.

Introduction

Let’s take a brief look at the key concepts involved in the TensorFlow model to OpenCV API conversion pipeline. The first step in converting a TensorFlow model to cv.dnn.Net is to obtain a frozen TF model graph. A frozen graph defines the combination of the model graph structure and the retained values of the required variables (such as weights). Frozen graphs are usually saved in protobuf (.pb) files. After the model .pb file is generated, it can be read using the cv.dnn.readNetFromTensorflow function.

Requirements

To experiment with the code below, you need to install a set of libraries. For this we will use a virtual environment with python3.7+:

virtualenv -p /usr/bin/python3.7 <env_dir_path>
source <env_dir_path>/bin/activate

To build OpenCV-Python from source, follow the instructions in Introduction to OpenCV.

Before you start installing the library, you can customize requirements.txt to exclude or include (for example, opencv-python) certain dependencies. The following line installs the startup requirements into the previously activated virtual environment:

pip install -r requirements.txt

Practice

In this part we will cover the following points:

1. Create a TF classification model transformation pipeline and provide inference
2. Evaluate and test TF classification models

If you only want to run an evaluation or test model pipeline, you can skip the “Model Transformation Pipeline” tutorial section.

Model conversion pipeline

The code for this subchapter is located in the dnn_model_runner module and can be executed through the following command line:

python -m dnn_model_runner.dnn_conversion.tf.classification.py_to_py_mobilenet

The following code contains instructions for the following steps:

1. Instantiate a TF model
2. Create TF freeze graph
3. Read TF frozen graph using OpenCV API
4. Prepare to enter data
5. provide reasoning

# Initialize TF MobileNet model
original_tf_model = MobileNet(
    include_top=True、
    weights="imagenet"
)
# Get the TF frozen map path
full_pb_path = get_tf_model_proto(original_tf_model)
# Use OpenCV API to read frozen images
opencv_net = cv2.dnn.readNetFromTensorflow(full_pb_path)
print("OpenCV model has been read successfully. Model layers: \
", opencv_net.getLayerNames())
# Get preprocessed image
input_img = get_preprocessed_img("../data/squirrel_cls.jpg")
# Get image network tags
imagenet_labels = get_imagenet_labels("../data/dnn/classification_classes_ILSVRC2012.txt")
# Get OpenCV DNN prediction results
get_opencv_dnn_prediction(opencv_net, input_img, imagenet_labels)
# Get TF model prediction results
get_tf_dnn_prediction(original_tf_model, input_img, imagenet_labels)

To provide model inference, we will use the following photo of a squirrel corresponding to ImageNet class ID 335 (licensed under CC0 license):

Fox squirrel, Eastern fox squirrel, Sciurus niger

Classification model input image

In order to decode the labels of the predictions, we also need the imagenet_classes.txt file, which contains the complete list of ImageNet classes.

Let’s take the pre-trained TF MobileNet as an example and dive into each step:

• Instantiate the TF model:

# Initialize TF MobileNet model
original_tf_model = MobileNet(
    include_top=True、
    weights="imagenet"
)

• Create TF freeze graph

# Define directory for .pb model
pb_model_path = "models" # Define the name of the .pb model
# Define the name of the .pb model
pb_model_name = "mobilenet.pb" # Create a directory for further converted models
# Create directories for further converted models
os.makedirs(pb_model_path, exist_ok=True)
# Get model TF graph
tf_model_graph = tf.function(lambda x: tf_model(x))
# Get specific functions
tf_model_graph = tf_model_graph.get_concrete_function(
    tf.TensorSpec(tf_model.inputs[0].shape, tf_model.inputs[0].dtype))
# Get the frozen specific function
frozen_tf_func = convert_variables_too_constants_v2(tf_model_graph)
# Get frozen graphics
frozen_tf_func.graph.as_graph_def()
# Save the complete tf model
tf.io.write_graph(graph_or_graph_def=frozen_tf_func.graph.as_graph_def() # Save the complete tf model
                  logdir=pb_model_path、
                  name=pb_model_name、
                  as_text=False)

After successfully executing the above code, we will get a frozen graph in models/mobilenet.pb.

• Use cv.dnn.readNetFromTensorflow to pass in the mobilenet.pb obtained in the previous step and read the TF frozen graph:

# Get TF frozen map path
full_pb_path = get_tf_model_proto(original_tf_model)

• Prepare input data using the cv2.dnn.blobFromImage function:

# Read image
input_img = cv2.imread(img_path, cv2.IMREAD_COLOR)
input_img = input_img.astype(np.float32)


# Define preprocessing parameters
mean = np.array([1.0, 1.0, 1.0]) * 127.5
scale= 1 / 127.5
# Prepare the input blob to fit the model input:
# Subtract the average
# 2. Adjust pixel value from 0 to 1
input_blob = cv2.dnn.blobFromImage(
    image=input_img、
    scalefactor=scale、
    size=(224, 224), # Image target size
    mean=mean、
    swapRB=True, # BGR -> RGB
    crop=True # Center crop
)
print("Input blob shape: {}\
".format(input_blob.shape))

Please note the preprocessing order in the cv2.dnn.blobFromImage function. First subtract the average and then multiply the pixel values by the defined scale. Therefore, to reproduce the image preprocessing flow in the TF mobilenet.preprocess_input function, we multiply the average by 127.5.

This results in a 4-dimensional input_blob:

Input blob shape: (1, 3, 224, 224)

• Provides OpenCV cv.dnn.Net inference:

# Set up OpenCV DNN input
opencv_net.setInput(preproc_img)
# OpenCV DNN inference
out = opencv_net.forward()
print("OpenCV DNN prediction: \
")
print("* shape: ", out.shape)
# Get the predicted class ID
imagenet_class_id = np.argmax(out)
# Get confidence
confidence = out[0][imagenet_class_id].
print("* class ID: {}, label: {}".format(imagenet_class_id, imagenet_labels[imagenet_class_id]))
print("* confidence: {:.4f}\
".format(confidence))

After executing the above code, we will get the following output:

OpenCV DNN prediction:
* shape: (1, 1000)
* Class ID: 335, label: fox squirrel, eastern fox squirrel, Sciurus niger
* Confidence level 0.9525

• Provides TF MobileNet inference:

# Inference
preproc_img = preproc_img.transpose(0, 2, 3, 1)
print("TF input blob shape: {}\
".format(preproc_img.shape))
out = original_net(preproc_img)
print("\
TensorFlow model prediction:\
")
print("* shape: ", out.shape)
# Get the predicted class ID
imagenet_class_id = np.argmax(out)
print("* class ID: {}, label: {}".format(imagenet_class_id, imagenet_labels[imagenet_class_id]))
# Get confidence
confidence = out[0][imagenet_class_id].
print("*confidence: {:.4f}".format(confidence))

To fit the TF model input, input_blob is transposed:

TF input blob shape: (1, 224, 224, 3)

The TF inference results are as follows:

TensorFlow model prediction:
* Shape: (1, 1000)
* Class ID: 335, label: fox squirrel, eastern fox squirrel, Sciurus niger
* Confidence level 0.9525

It can be seen from the experiments that the inference results of OpenCV and TF are the same.

Model Evaluation

The dnn/samples dnn_model_runner module allows running the complete evaluation pipeline on the ImageNet dataset and testing the following TensorFlow classification models:

• vgg16
• vgg19
• resnet50
• resnet101
• resnet152
• densenet121
• densenet169
• densenet201
• inceptionresnetv2
• inceptionv3
• mobilenet
• mobilenetv2
• nasnetlarge
• nasnetmobile
• xception

This list can also be extended with further appropriate evaluation pipeline configurations.

Evaluation Mode

The following line indicates that the module is running in evaluation mode:

python -m dnn_model_runner.dnn_conversion.tf.classification.py_to_py_cls--model_name<tf_cls_model_name>

The classification model selected from the list will be read into an OpenCV cv.dnn_Net object. Evaluation results (accuracy, inference time, L1) for TF and OpenCV models are written to log files. Inference time values are also displayed graphically to summarize the model information obtained.

The necessary evaluation configuration is defined in test_config.py and can be modified based on the actual path to the data location:

@dataclass
class TestClsConfig:
    batch_size: int = 50
    frame_size: int = 224
    img_root_dir: str = "./ILSVRC2012_img_val"
    # location of image-class matching
    img_cls_file: str = "./val.txt"
    bgr_to_rgb: bool = True

The values in TestClsConfig can be customized based on the selected model.

To start the TensorFlow MobileNet evaluation, run the following line:

python -m dnn_model_runner.dnn_conversion.tf.classification.py_to_py_cls --model_name mobilenet

After the script is started, a log file containing the evaluation data will be generated in dnn_model_runner/dnn_conversion/logs:

====== The model is being evaluated using the following parameters:
    * Value data location: ./ILSVRC2012_img_val
    * Log file location: dnn_model_runner/dnn_conversion/logs/TF_mobilenet_log.txt

Test Mode

The following line represents running the module in test mode, i.e. providing the steps for model inference:

python -m dnn_model_runner.dnn_conversion.tf.classification.py_to_py_cls --model_name <tf_cls_model_name> --test True --default_img_preprocess <True/False> --evaluate False

The default_img_preprocess keyword here defines whether you want to parameterize the model testing process with some specific values, or use default values such as scale, mean or std.

The test configuration is represented in the test_config.py TestClsModuleConfig class:

@dataclass
class TestClsModuleConfig:
    cls_test_data_dir: str = "../data"
    test_module_name: str = "classification"
    test_module_path: str = "classification.py"
    input_img: str = os.path.join(cls_test_data_dir, "squirrel_cls.jpg")
    model: str = ""
    frame_height: str = str(TestClsConfig.frame_size)
    frame_width: str = str(TestClsConfig.frame_size)
    scale: str = "1.0"
    mean: List[str] = field(default_factory=lambda: ["0.0", "0.0", "0.0"])
    std: List[str] = field(default_factory=list)
    crop: str = "False"
    rgb: str = "True"
    rsz_height: str = ""
    rsz_width: str = ""
    classes: str = os.path.join(cls_test_data_dir, "dnn", "classification_classes_ILSVRC2012.txt")

Default image preprocessing options are defined in default_preprocess_config.py. For example, for MobileNet

tf_input_blob = {<!-- -->
    "mean": ["127.5", "127.5", "127.5"],
    "scale": str(1 / 127.5),
    "std": [],
    "crop": "True",
    "rgb": "True"
}

The model testing basis represented in samples/dnn/classification.py is autonomous via the conversion model provided in --input and the parameters configured for cv.dnn.blobFromImage implement.

To reproduce the OpenCV steps described in “Model Conversion Pipeline” from scratch using dnn_model_runner, execute the following line:

python -m dnn_model_runner.dnn_conversion.tf.classification.py_too_py_cls --model_name mobilenet --test True --default_img_preprocess True --evaluate False

The network prediction results are displayed in the upper left corner of the output window: