Table of Contents
What is tqdm?
introduce
Install
Instructions
1. Pass in the iterable object
Use trange
2. Set a description for the progress bar
3.tqdm write method
4. How to use it in deep learning
Download training set data set
Create dataDataLoader
Create a neural network model
Model training and testing
Train a deep learning model
Output results
What is tqdm?
Introduction
tqdm is a fast and extensible Python progress bar that can add a progress prompt message in a Python long loop. Users only need to encapsulate any iterator tqdm (iterator). It can help usmonitor the progress of the program, estimate the running time, and even assist in debugging. It is useful to display progress on long-running tasks, as it lets the user know that the task is in progress.
Installation
pip install tqdm
How to use
1. Pass in the iterable object
import time from tqdm import * for i in tqdm(range(1000)): time.sleep(0.01) #The progress bar advances every 0.01s, the total time is 1000*0.01=10s
operation result:
usetrange
trange(i) is a simple way of writing tqdm(range(i))
import time from tqdm import trange for i in trange(1000): time.sleep(0.01)
operation result:
2. Set a description for the progress bar
Initialize tqdm outside the for loop, you can print other information
import time
from tqdm import tqdm
pbar = tqdm(["a","b","c","d"])
for char in pbar:
pbar.set_description("Processing %s" % char) # Set description
time.sleep(1) # Each task is allocated 1s
#The results are as follows
0%| | 0/4 [00:00<?, ?it/s]
Processing a: 0%| | 0/4 [00:00<?, ?it/s]
Processing a: 25%|██ | 1/4 [00:01<00:03, 1.01s/it]
Processing b: 25%|██ | 1/4 [00:01<00:03, 1.01s/it]
Processing b: 50%|█████ | 2/4 [00:02<00:02, 1.01s/it]
Processing c: 50%|█████ | 2/4 [00:02<00:02, 1.01s/it]
Processing c: 75%|███████ | 3/4 [00:03<00:01, 1.01s/it]
Processing d: 75%|███████ | 3/4 [00:03<00:01, 1.01s/it]
Processing d: 100%|██████████| 4/4 [00:04<00:00, 1.01s/it]
3.tqdm write method
bar = trange(10)
for i in bar:
time.sleep(0.1)
if not (i % 3):
tqdm.write("Done task %i" % i)
#The results are as follows
Done task 0
0%| | 0/10 [00:10<?, ?it/s]
0%| | 0/10 [00:00<?, ?it/s]
10%|████████▎ | 1/10 [00:00<00:01, 8.77it/s]
20%|████████████████ | 2/10 [00:00<00:00, 9.22it/s]
Done task 3
0%| | 0/10 [00:10<?, ?it/s]
30%|████████████████████████▉ | 3/10 [00:00<00:01, 6.91it/s]
| 9.17it/s]
50% | ███████████████████████████████████████ | 5/10 [ 00:00<00:00, 9.28it/s]
Done task 6
0%| | 0/10 [00:10<?, ?it/s]
60%|█████████████████████████████ ██▊ | 6/10 [00:00<00:00, 7.97it/s]
70%| ███████████ | 7/10 [00:00<00:00, 9.25it/s]
80%|█████████████████████████████ ▍ | 8/10 [00:00<00:00, 9.31it/s]
Done task 9
0%| | 0/10 [00:11<?, ?it/s]
90%| ▋ | 9/10 [00:01<00:00, 8.37it/s]
100%| 10/10 [00:01<00:00, 9.28it /s]
4. How to use it in deep learning
The following is a handwritten digital identification code
Download training set data set
import torch.cuda
from torch import nn #Import neural network module
from torch.utils.data import DataLoader #Data package management tool
from torchvision import datasets #Data processing tools, packages dedicated to image processing
from torchvision.transforms import ToTensor #Data conversion, tensor
'''Download the training set data set (including training images and labels)'''
#datasets.MNIST to load the MNIST dataset as a training dataset.
#root='data': Specify the root directory where the data set is stored, which can be changed as needed.
#train=True: Indicates loading the training data set
#download=True: If the dataset does not exist in the specified path, it will be automatically downloaded from the official source and saved.
#transform=ToTensor(): Specify the data conversion operation to convert the image data to the Tensor tensor format in PyTorch.
training_data = datasets.MNIST(
root='data',
train=True,
download=True,
transform=ToTensor(), #Tensor
) #The data that the pytorch library can recognize is generally tensor tensor
test_data = datasets.MNIST(
root='data',
train=False,
download=True,
transform=ToTensor()
)
Create data DataLoader
train_dataloader = DataLoader(training_data,batch_size=64) #64 pictures are one package
test_dataloader = DataLoader(test_data,batch_size=64)
for X,Y in train_dataloader: #X represents each packaged data packet
print(f'Shape of X[N,C,H,W]:{X.shape}')
print(f'Shape of Y:{Y.shape}{Y.dtype}')
break
'''Determine whether the current device supports GPU, where mps is the GPU of Apple's m-series chip'''
device = 'cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu'
print(f'Using {device} device')
Additional:
bath_size: Divide the data set into multiple parts, each part is bath_size data
Advantages: Can reduce memory usage and increase training speed
64: Represents the batch size (batch size), that is, the tensor contains 64 samples.
1: Indicates the number of channels (channel). This is a grayscale image, so the number of channels is 1.
28: Represents the height of the image (height), that is, each image has a height of 28 pixels.
28: Indicates the width of the image (width), that is, each image has a width of 28 pixels.
Create neural network model
class NeuralNetwork(nn.Module): #Defines a class named NeuralNetwork, inherited from the nn.Module class, which is the base class used to build neural networks in PyTorch.
def __init__(self):
super().__init__() #Call the constructor of the parent class (nn.Module) to ensure that the base class of the neural network is initialized.
self.flatten = nn.Flatten() #Create an expansion object flatten, used to flatten the input image data into a one-dimensional vector
self.hidden1 = nn.Linear(28*28,128) #Created the first fully connected layer, the input size is 28*28 (that is, the length of the image after expansion), and the output size is 128.
self.hidden2 = nn.Linear(128,64) #Created a second fully connected layer with an input size of 128 and an output size of 64.
self.hidden3 = nn.Linear(64,256) # Create a third fully connected layer with an input size of 64 and an output size of 64.
self.out = nn.Linear(256,10) #Created an output layer with an input size of 64 and an output size of 10 for classification tasks.
def forward(self,x): # Defines the forward propagation process of the model and specifies the path and operation of data flowing in the network.
x = self.flatten(x) #Flatten the input image data x.
x = self.hidden1(x) #Pass the flattened data x into the first fully connected layer hidden1 for linear transformation and activation function operations.
x = torch.relu(x) # Use the ReLU (Rectified Linear Unit) activation function to nonlinearize the output of the hidden1 layer
x = self.hidden2(x) # Pass the output x of the hidden1 layer to the second fully connected layer hidden2 for linear transformation and activation function operations.
x = torch.sigmoid(x)
x = self.hidden3(x)
x = torch.relu(x)
x = self.out(x)
return x
model = NeuralNetwork().to(device) #Create an instance of the neural network model and move it to the specified device,
Model training and testing
# Define the training function, the input parameters include data loader, model, loss function and optimizer
def train(dataloader,model,loss_fn,optimizer):
model.train() # Set the model to training mode
batch_size_num = 1 #Initialize the batch number to 1
for x,y in dataloader: # Traverse the data loader and provide a batch of data each time
x,y = x.to(device),y.to(device) #Move data to the specified device, such as GPU
pred = model.forward(x) # Calculate prediction results through the model
loss = loss_fn(pred,y) # Calculate the loss value between the predicted result and the actual result
optimizer.zero_grad() # Clear the optimizer's gradient cache
loss.backward() # Backpropagation, calculating gradient
optimizer.step() # Use the optimizer to update model parameters
loss_value = loss.item() # Get the item of the loss value for subsequent operations
#Print loss value and batch number
# print(f'loss:{loss_value:>7f}[num:{batch_size_num}]')
batch_size_num + = 1 # Increment the batch number by 1 so that the batch number is updated after each batch.
# Define the loss function, here we use the cross entropy loss function
loss_fn = nn.CrossEntropyLoss()
#Define the optimizer, here we use the Adam optimizer with a learning rate of 0.0015
optimizer = torch.optim.Adam(model.parameters(),lr=0.0015) #You can use Adam’s best model
# Call the training function for model training
train(train_dataloader,model,loss_fn,optimizer)
# Define the test function, the input parameters include data loader, model and loss function
def test(dataloader,model,loss_fn):
size = len(dataloader.dataset) # Get the number of test sets
num_batches = len(dataloader) # Get the number of batches of the data loader
model.eval() # Set the model to evaluation mode and turn off dropout and other factors that affect the results.
test_loss,correct = 0,0 # Initialize test loss and accuracy
with torch.no_grad(): # Turn off gradient calculation to save memory, because there is no need to backpropagate the gradient here.
for x,y in dataloader: # Traverse the data loader and provide a batch of data for testing each time
x,y = x.to(device),y.to(device) #Move data to the specified device, such as GPU
pred = model.forward(x) # Calculate prediction results through the model
test_loss + = loss_fn(pred,y).item() # Calculate the loss value between the predicted result and the actual result and add it to the total loss
correct + = (pred.argmax(1) == y).type(torch.float).sum().item() # Calculate the number of correct predictions and add them to the total
test_loss /= num_batches # Average loss value to get the final test loss
correct /= size # Average correct rate, get the final test correct rate
return test_loss, correct # Return test loss and correct rate as test results
Training deep learning model
# Define the number of rounds of training, that is, the number of iterations
epochs = 5
# Import the progress bar module, tqdm allows us to see the progress of each epoch during the training process
from tqdm import tqdm
# Loop through the specified number of rounds of training
for epoch in range(epochs):
# for data,targets in tqdm(train_loadr,leave=False) # Progress is displayed in one line
for data,targets in tqdm(train_dataloader):
# Move data to the specified device, if possible. For example, if you are using a GPU for calculations, this will move the data to the GPU.
data = data.to(device=device)
targets = targets.to(device=device)
# Forward propagation, the model calculates the input data and obtains the predicted results
scores = model(data)
# Calculate the loss. This loss is the difference between the model prediction result and the actual target.
loss = loss_fn(scores,targets)
# Back propagation, calculate the gradient based on the loss. This process is to calculate the gradient of the loss to the model parameters in each training step.
optimizer.zero_grad() # Clear the previous gradient cache
loss.backward() # Calculate gradient
# Use the optimizer to update the model parameters. This process is to update the model parameters based on the previously calculated gradient to achieve descent learning.
optimizer.step() # Use the optimizer to update model parameters
Output results
