1. Summary of common loss functions and usage scenarios in libtorch
1. CrossEntropyLoss:
CrossEntropyLoss (cross entropy loss) is mainly used for classification tasks. It is suitable for multi-classification problems where each sample belongs to only one class (mutually exclusive). This loss function compares the predicted probabilities with the one-hot vectors of the true labels and calculates the value of cross-entropy. It is usually used after the softmax operation on the output of the last layer of the neural network.
2. BCELoss:
BCELoss (Binary Cross Entropy Loss) is a loss function suitable for binary classification tasks (positive and negative classes), which compares the predicted probability with the true label and calculates the binary cross entropy element-by-element. Applicable to imbalanced binary classification problems.
3. NLLLoss:
NLLLoss (Negative Log Likelihood Loss) is used for multi-classification tasks and is suitable for situations where no softmax operation is performed. NLLLoss compares the output of log_softmax with the indices of the true labels and computes the negative of the log-likelihood.
4. MSELoss:
MSELoss (Mean Squared Error Loss) is a loss function for regression problems that compares the mean squared error between model predictions and true labels. Suitable for tasks that require prediction of continuous values.
5. L1Loss:
L1Loss (absolute value loss) is also a loss function for regression problems, which compares the absolute value error between the model’s predicted value and the true label. Compared with MSELoss, L1Loss pays more attention to outliers.
6. SmoothL1Loss:
SmoothL1Loss is a smooth L1 loss for regression problems. It is smoother than MSELoss and L1Loss, which can not only reduce the influence of outliers, but also reduce the sensitivity to large residuals while retaining good fit as much as possible.
7. CosineEmbeddingLoss:
CosineEmbeddingLoss is a loss function that measures the similarity between two samples. It mainly calculates the similarity between samples through the cosine similarity between two vectors. Suitable for related tasks, such as computing the similarity between text or images.
8. TripletMarginLoss:
TripletMarginLoss is a loss function for learning embedding vectors. It uses triplets (anchor, positive and negative examples) to train the model while minimizing the distance between anchors and positive examples and maximizing the distance between anchors and negative examples. It is suitable for novel category recognition, face recognition and other tasks.
In addition to the above loss functions that can be used alone, there are also some loss functions that can be combined to solve specific problems. For example, combine CrossEntropyLoss and Dice Loss to more accurately estimate the performance of the model in image segmentation problems, etc.
Various loss functions in the segmentation task and the corresponding libtorch implementation
-
DiceLoss
Dice Loss is a loss function widely used in medical image segmentation and other fields, and is suitable for binary or multi-classification problems. The main advantage of Dice Loss is that it works well when dealing with class imbalanced data, especially when the proportion of positive and negative samples is very unbalanced. Since Dice Loss pays more attention to pixel-level similarity, it usually produces more accurate prediction results, especially in segmentation tasks.
In general, Dice Loss is mainly suitable for pixel-level classification problems, especially for imbalanced data. It enables a finer segmentation between target instances and backgrounds, thus improving the performance of the model.
Ok, here is a basic code to implement DiceLoss:
torch::Tensor dice_loss(torch::Tensor input, torch::Tensor target)
{
torch::Tensor smooth = 1e-5;
torch::Tensor intersection = (input * target). sum();
torch::Tensor union_ = (input + target).sum();
torch::Tensor dice = (2 * intersection + smooth) / (union_ + smooth);
return 1 - dice;
}
This function takes an input tensor `input` and a target tensor `target`, and returns a `torch::Tensor` representing DiceLoss.
We first add a smoothing coefficient `smooth` to the loss function to prevent division errors when the numerator or denominator becomes zero. Then calculate the intersection and union, and calculate the Dice coefficient. Finally, subtract the Dice coefficient from 1 to get DiceLoss.
-
FocalLoss
Focal Loss is a loss function aimed at solving the problem of unbalanced data, which amplifies the samples that are difficult to classify with large contributions, so as to pay more attention to samples with poor classification effect. It is often used in tasks such as object detection and image segmentation.
Compared with the traditional cross-entropy loss, it can pay more attention to real positive samples for training in the presence of category imbalance. In Focal Loss, each positive sample is multiplied by a “difficulty factor”. The greater the difficulty factor, the greater the impact on classification, and the samples that are easy to be misclassified will get a relatively large weight, so that more attention can be paid to those with poor classification effect. samples, while ignoring samples with good classification effect.
The following is a simple Focal Loss function implementation for binary classification:
```c++
torch::Tensor focal_loss(torch::Tensor output, torch::Tensor target)
{
torch::Tensor alpha = 0.25;
torch::Tensor gamma = 2;
torch::Tensor pt = torch::sigmoid(output);
torch::Tensor p = (target * pt) + ((1 - target) * (1 - pt));
torch::Tensor w = alpha * target + (1 - alpha) * (1 - target);
w = w * torch::pow((1 - p), gamma);
torch::Tensor loss = -w * torch::log(pt + 1e-8);
return loss. mean();
}
```
This function takes an input tensor `output` and a target tensor `target`, and returns the loss used for Focal Loss. The `alpha` parameter is used to adjust the imbalance between positive and negative samples, and the `gamma` parameter is used to adjust the size of the “difficulty factor”. In this example, use the sigmoid function to change the output into a probability, then calculate the weight `w` and loss `loss` according to the formula, and finally return the average loss.
3. Jaccard Loss
Jaccard Loss, also known as Intersection Over Union (IoU) Loss or Jaccard Index Loss, is a commonly used loss function for image segmentation and object detection.
Jaccard Loss is based on the jaccard coefficient, also known as IoU (Intersection over Union), which measures the similarity between the predicted mask and the real mask. Jaccard Loss is calculated as 1 minus the intersection of two masks divided by their union.
In image segmentation, if the pixel value of the mask is 0 or 1, then IoU can be expressed as:
`IoU = (predict & target).sum() / (predict | target).sum()`
When the pixel value of the mask is 0 or 1, the calculation method of Jaccard Loss can be expressed as:
```c++
torch::Tensor jaccard_loss(torch::Tensor input, torch::Tensor target)
{
torch::Tensor smooth = 1e-5;
torch::Tensor intersection = (input * target). sum();
torch::Tensor union_ = (input + target).sum() - intersection;
torch::Tensor jaccard = (intersection + smooth) / (union_ + smooth);
return 1 - jaccard;
}
```
The above is an implementation of Jaccard Loss. The parameter `input` is the mask predicted by the model, and `target` is the real mask of the image. Calculate the intersection and union, and calculate the Jaccard coefficient according to the formula. Finally, subtract the Jaccard coefficient from 1 to get the Jaccard Loss.
4. Lovasz Loss
Lovasz Loss is a loss function for classification problems, which, like cross-entropy loss, is used to calculate the difference between the predicted results of the model and the real results. Unlike cross-entropy, Lovasz Loss does not need to convert the prediction result into a probability distribution through the Softmax function, so it is suitable for multi-label classification problems, and usually performs better than cross-entropy when the training set is imbalanced. The calculation method of Lovasz Loss is based on the Lovasz extension to measure the model’s contribution to the results of each sample, rather than simply considering the overall difference of the entire data set.
The following is a sample code to implement Lovasz Loss using libtorch:
```C++
#include <torch/torch.h>
torch::Tensor lovasz_hinge(const torch::Tensor & amp; logits, const torch::Tensor & amp; labels) {
// First sort the prediction results to facilitate subsequent calculations
torch::Tensor logits_sorted, dec_indices;
std::tie(logits_sorted, dec_indices) = torch::sort(logits, 1, true);
torch::Tensor labels_sorted = labels. index_select(0, dec_indices);
// Calculate the contribution value of each sample
torch::Tensor labels_f = labels_sorted;
torch::Tensor signs = 2 * labels_f - 1;
torch::Tensor errors = 1 - logits_sorted * signs;
torch::Tensor errors_sorted, permute_indices;
std::tie(errors_sorted, permute_indices) = torch::sort(errors, 1, true);
torch::Tensor permute_back = torch::argsort(permute_indices, 1);
torch::Tensor values = errors_sorted. clone();
torch::Tensor permutation = permute_back. clone();
values = values * permute_indices. sign();
permutation = permutation. index_select(1, permute_indices. abs() - 1);
// Calculate Lovasz Loss
int num_examples = logits. size(0);
torch::Tensor errors_flatten = values. flatten();
torch::Tensor labels_flatten = labels_f.index_select(0, permutation.flatten()).unsqueeze(1);
torch::Tensor signs_flatten = torch::ones_like(labels_flatten);
signs_flatten[labels_flatten <= 0] = -1;
torch::Tensor supergradient = torch::cumsum(signs_flatten, 0);
torch::Tensor nonzeros = torch::where(errors_flatten > 0.0, torch::ones_like(errors_flatten), torch::zeros_like(errors_flatten));
torch::Tensor gradient = 2 * supergradient - nonzeros;
torch::Tensor loss = torch::dot(torch::relu(errors_flatten), gradient) / num_examples;
return loss;
}
```
This code implements a function `lovasz_hinge()` that solves Lovasz Loss in multi-label classification problems. This function accepts two tensors as input, which are the predicted logits and the real label labels. It first sorts the logits, and calculates the contribution of each sample to the overall loss according to the prediction results, and finally calculates the overall loss value according to the Lovasz expansion.
5.MCCLoss
MCC (Matthews Correlation Coefficient) Loss is a loss function for classification problems. It is based on the MCC index to calculate the difference between the predicted results of the model and the real results, which can more accurately reflect the effect of the classification algorithm. The advantages of MCC Loss include: it is not affected by the uneven distribution of categories, it can be used for multi-label classification and binary classification problems, and it has good score regression performance. The value range of the MCC index is between -1 and 1, 1 means perfect forecast, 0 means random forecast, and -1 means complete reverse forecast. For the calculation of MCC Loss, the problem is usually converted into a multi-classification problem by binarizing the predicted results and the real results respectively, and finally the weighted average of the MCC values of all categories is calculated as the value of the loss function.
Ok, here is the code to implement MCC LOSS with libtorch:
```c++
#include <iostream>
#include <torch/torch.h>
#include <cmath>
// define multi-class Mcnemar loss
class McCrossEntropyLoss : public torch::nn::Module {
public:
McCrossEntropyLoss() = default;
torch::Tensor forward(torch::Tensor input, torch::Tensor target) {
int k = input. size(1);
int n = input. size(0);
torch::Tensor output = torch::zeros(k, k);
for (int i = 0; i < k; i ++ ) {
for (int j = i + 1; j < k; j + + ) {
torch::Tensor mask_i = (target == i);
torch::Tensor mask_j = (target == j);
torch::Tensor mask_ij = (mask_i + mask_j) > 0;
torch::Tensor logits_ij = input.index_select(0, mask_ij.nonzero().view(-1));
torch::Tensor labels_ij = target.index_select(0, mask_ij.nonzero().view(-1));
torch::Tensor num = (((labels_ij == i).int()) * ((logits_ij[:, j] > logits_ij[:, i]).int())) +
(((labels_ij == j).int()) * ((logits_ij[:, j] < logits_ij[:, i]).int()));
torch::Tensor den = (((labels_ij == i).int()) * ((logits_ij[:, j] < logits_ij[:, i]).int())) +
(((labels_ij == j).int()) * ((logits_ij[:, j] > logits_ij[:, i]).int()));
torch::Tensor freq_ij = (num - den) / mask_ij.sum().float();
output[i][j] = freq_ij. item<float>();
output[j][i] = -freq_ij.item<float>();
}
}
torch::Tensor loss = torch::trace(output.mm(output));
return loss;
}
};
int main() {
int classes = 3;
int instances = 10;
int dims = 5;
torch::Tensor inputs = torch::rand({instances, dims});
torch::Tensor targets = torch::randint(classes, {instances});
McCrossEntropyLoss mcc_loss = McCrossEntropyLoss();
torch::Tensor loss = mcc_loss(inputs, targets);
std::cout << "MCC loss: " << loss.item<float>() << std::endl;
return 0;
}
```
Among them, McCloss inherits from torch::nn::module, overloads the forward function, and realizes the calculation of MCC LOSS. In the main function, we tested whether the MCC LOSS calculation is correct using randomly generated input and target tensors.
6. SOFT BCE LOSS
Soft Binary Cross-Entropy (Soft BCE) Loss, is a loss function for multi-label classification tasks.
In multi-label classification tasks, each sample corresponds to multiple labels, so the cross-entropy loss function cannot be used like in single-label classification tasks. Soft BCE Loss treats samples and labels as a two-dimensional matrix, where each row represents all labels of a sample, and each column represents the same label in all samples. Then, the sigmoid function is applied to the predicted value of each label (squeezing the predicted value between [0,1]), and compared with the value of the real label, and finally the average loss value of all labels is calculated.
The formula of Soft BCE Loss is as follows:
$$ L_{i,j}=-w_j[y_{i,j}\log(p_{i,j}) + (1-y_{i,j})\log(1-p_{i, j})] $$
Among them, $w_j$ represents the weight of the $j$th label, $y_{i,j}$ represents the true value of the $j$th label of the $i$th sample, and $p_{i,j}$ represents The predicted value of the $j$th label of the $i$th sample (compressed by the sigmoid function).
The advantage of Soft BCE Loss is that it can handle multi-label classification tasks, and the prediction results are interpretable, that is, the predicted value of each label is between [0,1], representing the probability that the sample belongs to the label.
Ok, here is a sample code to implement soft bce loss using libtorch:
```c++
#include <torch/torch.h>
class SoftBCELoss : public torch::nn::Module {
public:
SoftBCELoss() {}
~SoftBCELoss() {}
torch::Tensor forward(torch::Tensor input, torch::Tensor target) {
torch::Tensor log_probs = torch::log_softmax(input, /*dim=*/1);
torch::Tensor loss = -target * log_probs;
loss = loss.mean(/*dim=*/0);
return loss;
}
};
```
Here we create a class `SoftBCELoss` inherited from `torch::nn::Module`, and its `forward` function implements the calculation logic of soft bce loss. Among them, `input` is the output predicted by the model, and `target` is the actual label. We first use the `log_softmax` function on `input`, then calculate the multiplication of `target` and the negative value of `log_probs`, and calculate the mean of each sample, and finally return the loss value.
Hope this sample code helps you.
7. SOFT CE LOSS
Soft cross-entropy (CE) loss is a commonly used multi-classification loss function. Different from ordinary CE-loss, soft CE-loss can treat the label as a probability distribution instead of a one-hot vector, making the training of the model smoother.
In soft CE-loss, the label is the probability value $y_i$ of each category, and their sum must be equal to 1. The output of the model is also a probability vector $p$ output by the softmax function, where each element $p_i$ represents the probability of being predicted as the $i$th class. The loss function of both is an extended form of cross-entropy loss:
$$
\text{CE}(p, y) = -\sum_{i=1}^C y_i\log(p_i),
$$
Where $C$ is the number of categories. When $y$ is a one-hot vector, the above formula is an ordinary CE-loss; and when $y$ is a probability distribution, $y_i$ represents the probability that the sample belongs to the class $i$, then the above formula is equivalent It is used to do a weighted average of the cross entropy of each category.
It should be noted that if the actual label $y$ of the sample is a one-hot vector, then using soft CE-loss will not make much difference compared to ordinary CE-loss, because the performance of the two is basically the same. However, when $y$ is a probability distribution, soft CE-loss can better guide the model to learn an appropriate probability distribution output, so as to achieve better results.
Ok, here is a sample code for soft cross-entropy loss using libtorch:
```c++
#include <torch/torch.h>
class SoftCELoss : public torch::nn::Module {
public:
SoftCELoss() {}
~SoftCELoss() {}
torch::Tensor forward(torch::Tensor input, torch::Tensor target) {
torch::Tensor log_probs = torch::log_softmax(input, /*dim=*/1);
torch::Tensor loss = -torch::sum(target * log_probs, /*dim=*/1);
loss = loss.mean(/*dim=*/0);
return loss;
}
};
```
Here we create a class `SoftCELoss` inherited from `torch::nn::Module`, and its `forward` function implements the calculation logic of soft cross-entropy loss. Among them, `input` is the output predicted by the model, and `target` is the probability distribution of the actual label. We first process the `input` using the `log_softmax` function, then calculate the cross-entropy loss for each sample and average them, and finally return the loss value.
Hope this sample code helps you.
8. Tversky LOSS
Tversky loss (Tversky loss) is a loss function used to calculate image segmentation tasks. It was proposed by Israeli psychologist Amos Tversky in 1977 to measure the similarity between two concepts in classification tasks.
In the field of image segmentation, Tversky loss is often used to measure the similarity between predicted labels (such as foreground and background) and ground truth labels. It does this mainly by computing the similarity of two sets, not by computing their intersection or union.
The Tversky loss function has better balance, it can balance the weight difference between different categories, and can avoid the balance problem between categories. Therefore, in image segmentation tasks, Tversky loss is widely used and achieves good performance.
Ok, here is a sample code to implement Tversky loss using C++ and libtorch:
```c++
#include <torch/torch.h>
torch::Tensor tversky_loss(torch::Tensor y_true, torch::Tensor y_pred, float alpha, float beta) {
// calculate true positives
auto true_pos = torch::sum(y_true * y_pred);
// calculate false positives
auto false_pos = torch::sum((1 - y_true) * y_pred);
// calculate false negatives
auto false_neg = torch::sum(y_true * (1 - y_pred));
// calculate Tversky index
auto tversky_index = true_pos / (true_pos + alpha * false_neg + beta * false_pos);
// calculate Tversky loss
auto tversky_loss = 1 - tversky_index;
// return Tversky loss as tensor
return tversky_loss;
}
```
Among them, `y_true` is the true label and `y_pred` is the predicted label. `alpha` and `beta` are two weight parameters used to balance the effects of false negatives and false positives. The function returns a `Tensor` object representing the Tversky loss.
It should be noted that to use this function, you need to install and configure the libtorch library in advance, and add the corresponding header files and link libraries to the code.