Deep neural networks have dominated computer vision and machine learning in recent years, and this has led to their widespread deployment in real-world systems [Cao et al., 2018, Chen et al., 2018, Kamilaris and Prenafeta-Boldú, 2018, Ker et al., 2018, Wang et al., 2018]. However, many current multi-class classification networks in particular are poorly calibrated, in the sense that the probability values that they associate with the class labels they predict overestimate the likelihoods of those class labels being correct in the real world. This is a major problem, since if networks are routinely overconfident, then downstream components cannot trust their predictions. The underlying cause is hypothesised to be that these networks’ high capacity leaves them vulnerable to overfitting on the negative log-likelihood (NLL) loss they conventionally use during training [Guo et al., 2017].
Given the importance of this problem, numerous suggestions for how to address it have been proposed. Much work has been inspired by approaches that were not originally formulated in a deep learning context, such as Platt scaling [Platt, 1999], histogram binning [Zadrozny and Elkan, 2001], isotonic regression [Zadrozny and Elkan, 2002], and Bayesian binning and averaging [Naeini et al., 2015, Naeini and Cooper, 2016]. As deep learning has become more dominant, however, various works have begun to directly target the calibration of deep networks. For example, Guo et al. [2017] have popularised a modern variant of Platt scaling known as temperature scaling, which works by dividing a network’s logits by a scalar T > 0 (learnt on a validation subset) prior to performing softmax. Temperature scaling has the desirable property that it can improve the calibration of a network without in any way affecting its accuracy. However, whilst its simplicity and effectiveness have made it a popular network calibration method, it does have downsides. For example, whilst it scales the logits to reduce the network’s confidence in incorrect predictions, this also slightly reduces the network’s confidence in predictions that were correct [Kumar et al., 2018]. Moreover, it is known that temperature scaling does not calibrate a model under data distribution shift [Ovadia et al., 2019].
By contrast, Kumar et al. [2018] initially eschew temperature scaling in favour of minimising a differentiable proxy for calibration error at training time, called Maximum Mean Calibration Error (MMCE), although they do later also use temperature scaling as a post-processing step to obtain better results than cross-entropy followed by temperature scaling [Guo et al., 2017]. Separately, Müller et al. [2019] propose training models on cross-entropy loss with label smoothing instead of one-hot labels, and show that label smoothing has a very favourable effect on model calibration.
In this paper, we propose a technique for improving network calibration that works by replacing the cross-entropy loss conventionally used when training classification networks with the focal loss proposed by Lin et al. [2017]. We observe that unlike cross-entropy, which minimises the KL divergence between the predicted (softmax) distribution and the target distribution (one-hot encoding in classification tasks) over classes, focal loss minimises a regularised KL divergence between these two distributions, which ensures minimisation of the KL divergence whilst increasing the entropy of the predicted distribution, thereby preventing the model from becoming overconfident. Since focal loss, as shown in §4, is dependent on a hyperparameter, , that needs to be cross-validated, we also provide a method for choosing automatically for each sample, and show that it outperforms all the baseline models.
The intuition behind using focal loss is to direct the network’s attention during training towards samples for which it is currently predicting a low probability for the correct class, since trying to reduce the NLL on samples for which it is already predicting a high probability for the correct class is liable to lead to NLL overfitting, and thereby miscalibration [Guo et al., 2017]. More formally, we show in §4 that focal loss can be seen as implicitly regularising the weights of the network during training by causing the gradient norms for confident samples to be lower than they would have been with cross-entropy, which we would expect to reduce overfitting and improve the network’s calibration.
Overall, we make the following contributions:
1. In §3, we study the link that Guo et al. [2017] observed between miscalibration and NLL overfitting in detail, and show that the overfitting is associated with the predicted distributions for misclassified test samples becoming peakier as the optimiser tries to increase the magnitude of the network’s weights to reduce the training NLL.
2. In §4, we propose the use of focal loss for training better-calibrated networks, and provide both theoretical and empirical justifications for this approach. In addition, we provide a principled method for automatically choosing for each sample during training.
3. In §5, we show, via experiments on a variety of classification datasets and network architectures, that DNNs trained with focal loss are more calibrated than those trained with cross-entropy loss (both with and without label smoothing), MMCE or Brier loss [Brier, 1950]. Finally, we also make the interesting observation that whilst temperature scaling may not work for detecting out-of-distribution (OoD) samples, our approach can. We show that our approach is better at detecting out-of-distribution samples, taking CIFAR-10 as the in-distribution dataset, and SVHN and CIFAR-10-C as out-of-distribution datasets.
Let denote a dataset consisting of N samples from a joint distribution D(X, Y), where for each sample is the input and is the ground-truth class label. Let be the probability that a neural network f with model parameters for a class y on a given input . The class that f predicts for is computed as and the predicted confidence as . The network is said to be perfectly calibrated when, for each sample , the confidence is equal to the model accuracy , i.e. the probability that the predicted class is correct. For instance, of all the samples to which a perfectly calibrated neural network assigns a confidence of 0.8, 80% should be correctly predicted.
A popular metric used to measure model calibration is the expected calibration error (ECE) [Naeini et al., 2015], defined as the expected absolute difference between the model’s confidence and its accuracy, i.e. . Since we only have finite samples, the ECE cannot in practice be computed using this definition. Instead, we divide the interval [0, 1] into M equispaced bins, where the bin is the interval. Let denote the set of samples with confidences belonging to the bin. The accuracy of this bin is computed as , where 1 is the indicator function, and and are the predicted and ground-truth labels for the sample. Similarly, the confidence bin is computed as is the average confidence of all samples in the bin. The ECE can be approximated as a weighted average of the absolute difference between the accuracy and confidence of each bin: A similar metric, the maximum calibration error (MCE) [Naeini et al., 2015], is defined as the maximum absolute difference between the accuracy and confidence of each bin: MCE =
AdaECE: One disadvantage of ECE is the uniform bin width. For a trained model, most of the samples lie within the highest confidence bins, and hence these bins dominate the value of the ECE. We thus also consider another metric, AdaECE (Adaptive ECE), for which bin sizes are calculated so as to evenly distribute samples between bins (similar to the adaptive binning procedure in Nguyen and O’Connor [2015]):
Classwise-ECE: The ECE metric only considers the probability of the predicted class, without considering the other scores in the softmax distribution. A stronger definition of calibration would require the probabilities of all the classes in the softmax distribution to be calibrated [Kull et al., 2019, Vaicenavicius et al., 2019, Widmann et al., 2019, Kumar et al., 2019]. This can be achieved with a simple classwise extension of the ECE metric: where K is the number of classes, denotes the set of samples from the class in the bin,
A common way of visualising calibration is to use a reliability plot [Niculescu-Mizil and Caruana, 2005], which plots the accuracies of the confidence bins as a bar chart (see Appendix Figure A.1). For a perfectly calibrated model, the accuracy for each bin matches the confidence, and hence all of the bars lie on the diagonal. By contrast, if most of the bars lie above the diagonal, the model is more accurate than it expects, and is under-confident, and if most of the bars lie below the diagonal, then it is over-confident.
We now discuss why high-capacity neural networks, despite achieving low classification errors on well-known datasets, tend to be miscalibrated. A key empirical observation made by Guo et al. [2017] was that poor calibration of such networks appears to be linked to overfitting on the negative log-likelihood (NLL) during training. In this section, we further inspect this observation to provide new insights.
For the analysis, we train a ResNet-50 network on CIFAR-10 with state-of-the-art performance settings [PyTorch-CIFAR]. We use Stochastic Gradient Descent (SGD) with a mini-batch of size 128, momentum of 0.9, and learning rate schedule of {0.1, 0.01, 0.001} for the first 150, next 100, and last 100 epochs, respectively. We minimise cross-entropy loss (a.k.a. NLL) , which, in a standard classification context, is , where is the probability assigned by the network to the correct class for the isample. Note that the NLL is minimised when for each training sample , whereas the classification error is minimised when for all . This indicates that even when the classification error is 0, the NLL can be positive, and the optimisation algorithm can still try to reduce it to 0 by further increasing the value of for each sample (see Appendix A).
To study how miscalibration occurs during training, we plot the average NLL for the train and test sets at each training epoch in Figures 1(a) and 1(b). We also plot the average NLL and the entropy of the softmax distribution produced by the network for the correctly and incorrectly classified samples. In Figure 1(c), we plot the classification errors on the train and test sets, along with the test set ECE.
(c) Figure 1: Metrics related to calibration plotted whilst training a ResNet-50 network on CIFAR-10.
Curse of misclassified samples: Figures 1(a) and 1(b) show that although the average train NLL (for both correctly and incorrectly classified training samples) broadly decreases throughout training, after the epoch (where the learning rate drops by a factor of 10), there is a marked rise in the average test NLL, indicating that the network starts to overfit on average NLL. This increase in average test NLL is caused only by the incorrectly classified samples, as the average NLL for the correctly classified samples continues to decrease even after the epoch. We also observe that after epoch 150, the test set ECE rises, indicating that the network is becoming miscalibrated. This corroborates the observation in Guo et al. [2017] that miscalibration and NLL overfitting are linked.
Peak at the wrong place: We further observe that the entropies of the softmax distributions for both the correctly and incorrectly classified test samples decrease throughout training (in other words, the distributions get peakier). This observation, coupled with the one we made above, indicates that for the wrongly classified test samples, the network gradually becomes more and more confident about its incorrect predictions.
Weight magnification: The increase in confidence of the network’s predictions can happen if the network increases the norm of its weights W to increase the magnitudes of the logits. In fact, cross-entropy loss is minimised when for each training sample , which is possible only when . Cross-entropy loss thus inherently induces this tendency of weight magnification in neural network optimisation. The promising performance of weight decay [Guo et al., 2017] (regulating the norm of weights) on the calibration of neural networks can perhaps be explained using this. This increase in the network’s confidence during training is one of the key causes of miscalibration.
As discussed in §3, overfitting on NLL, which is observed as the network grows more confident on all of its predictions irrespective of their correctness, is strongly related to poor calibration. One cause of this is that the cross-entropy objective minimises the difference between the softmax distribution and the ground-truth one-hot encoding over an entire mini-batch, irrespective of how well a network classifies individual samples in the mini-batch. In this work, we study an alternative loss function, popularly known as focal loss [Lin et al., 2017], that tackles this by weighting loss components generated from individual samples in a mini-batch by how well the model classifies them. For classification tasks where the target distribution is a one-hot encoding, it is defined as is a user-defined hyperparameter2.
Why might focal loss improve calibration? We know that cross-entropy forms an upper bound on the KL-divergence between the target distribution q and the predicted distribution , i.e. , so minimising cross-entropy results in minimising . Interestingly, a general form of focal loss can be shown to be an upper bound on the regularised KL-divergence, where the regulariser is the negative entropy of the predicted distribution , and the regularisation parameter is , the hyperparameter of focal loss (a proof of this can be found in Appendix B):
Figure 2: How metrics related to model calibration change whilst training several ResNet-50 networks on CIFAR-10, using either cross-entropy loss, or focal loss with set to 1, 2 or 3.
The most interesting property of this upper bound is that it shows that replacing cross-entropy with focal loss has the effect of adding a maximum-entropy regulariser [Pereyra et al., 2017] to the implicit minimisation that was previously being performed. In other words, trying to minimise focal loss minimises the KL divergence between and q, whilst simultaneously increasing the entropy of the predicted distribution . Note, in the case of ground truth with one-hot encoding, only the component of the entropy of corresponding to the ground-truth index, , will be maximised (refer Appendix B). Encouraging the predicted distribution to have higher entropy can help avoid the overconfident predictions produced by DNNs (see the ‘Peak at the wrong place’ paragraph of §3), and thereby improve calibration.
Empirical observations: To analyse the behaviour of neural networks trained on focal loss, we use the same framework as mentioned above, and train four ResNet-50 networks on CIFAR-10, one using cross-entropy loss, and three using focal loss with . Figure 2(a) shows that the test NLL for the cross-entropy model significantly increases towards the end of training (before saturating), whereas the NLLs for the focal loss models remain low. To better understand this, we analyse the behaviour of these models for correctly and incorrectly classified samples. Figure 2(b) shows that even though the NLLs for the correctly classified samples broadly-speaking decrease over the course of training for all the models, the NLLs for the focal loss models remain consistently higher than that for the cross-entropy model throughout training, implying that the focal loss models are relatively less confident than the cross-entropy model for samples that they predict correctly. This is important, as we have already discussed that it is overconfidence that normally makes deep neural networks miscalibrated. Figure 2(c) shows that in contrast to the cross-entropy model, for which the NLL for misclassified test samples increases significantly after epoch 150, the rise in this value for the focal loss models is much less severe. Additionally, in Figure 2(d), we notice that the entropy of the softmax distribution for misclassified test samples is consistently (if marginally) higher for focal loss than for cross-entropy (consistent with Equation 1).
Note that from Figure 2(a), one may think that applying early stopping when training a model on cross-entropy can provide better calibration scores. However, there is no ideal way of doing early stopping that provides the best calibration error and the best test set accuracy. For fair comparison, we chose 3 intermediate models for each loss function with the best val set ECE, NLL and accuracy, and observed that: a) for every stopping criterion, focal loss outperforms cross-entropy in both test set accuracy and ECE, b) when using val set ECE as a stopping criterion, the intermediate model for cross-entropy indeed improves its test set ECE, but at the cost of a significantly higher test error. Please refer to Appendix J for more details.
As per §3, an increase in the test NLL and a decrease in the test entropy for misclassified samples, along with no corresponding increase in the test NLL for the correctly classified samples, can be interpreted as the network starting to predict softmax distributions for the misclassified samples that are ever more peaky in the wrong place. Notably, our results in Figures 2(b), 2(c) and 2(d) clearly show that this effect is significantly reduced when training with focal loss rather than cross-entropy, leading to a better-calibrated network whose predictions are less peaky in the wrong place.
Theoretical justification: As mentioned previously, once a model trained using cross-entropy reaches high training accuracy, the optimiser may try to further reduce the training NLL by increasing the confidences for the correctly classified samples. It may achieve this by magnifying the network weights to increase the magnitudes of the logits. To verify this hypothesis, we plot the norm of the weights of the last linear layer for all four networks as a function of the training epoch (see Figure 2(e)). Notably, although the norms of the weights for the models trained on focal loss are initially higher than that for the cross-entropy model, a complete reversal in the ordering of the weight norms occurs between epochs 150 and 250. In other words, as the networks start to become miscalibrated,
Figure 3: (a): vs. p and (b-d): histograms of the gradient norms of the last linear layer for both cross-entropy and focal loss.
the weight norm for the cross-entropy model also starts to become greater than those for the focal loss models. In practice, this is because focal loss, by design, starts to act as a regulariser on the network’s weights once the model has gained a certain amount of confidence in its predictions. This behaviour of focal loss can be observed even on a much simpler setup like a linear model (see Appendix C). To better understand this, we start by considering the following proposition (proof in Appendix D):
Proposition 1. For focal loss and cross-entropy , the gradients is the focal loss hyperparameter, and w denotes the parameters of the last linear layer. Thus
Proposition 1 shows the relationship between the norms of the gradients of the last linear layer for focal loss and cross-entropy loss, for the same network architecture. Note that this relation depends on a function , which we plot in Figure 3(a) to understand its behaviour. It is clear that for every , there exists a (different) threshold such that for all , and for all . (For example, for .) We use this insight to further explain why focal loss provides implicit weight regularisation.
Implicit weight regularisation: For a network trained using focal loss with a fixed , during the initial stages of the training, when . This implies that the confidences of the focal loss model’s predictions will initially increase faster than they would for cross-entropy. However, as soon as crosses the threshold falls below 1 and reduces the size of the gradient updates made to the network weights, thereby having a regularising effect on the weights. This is why, in Figure 2(e), we find that the weight norms of the models trained with focal loss are initially higher than that for the model trained using cross-entropy. However, as training progresses, we find that the ordering of the weight norms reverses, as focal loss starts regularising the network weights. Moreover, we can draw similar insights from Figures 3(b), 3(c) and 3(d), in which we plot histograms of the gradient norms of the last linear layer (over all samples in the training set) at epochs 10, 100 and 200, respectively. At epoch 10, the gradient norms for cross-entropy and focal loss are similar, but as training progresses, those for cross-entropy decrease less rapidly than those for focal loss, indicating that the gradient norms for focal loss are consistently lower than those for cross-entropy throughout training.
Finally, observe in Figure 3(a) that for higher values, the fall in is steeper. We would thus expect a greater weight regularisation effect for models that use higher values of . This explains why, of the three models that we trained using focal loss, the one with outperforms (in terms of calibration) the one with , which in turn outperforms the model with . Based on this observation, one might think that, in general, a higher value of gamma would lead to a more calibrated model. However, this is not the case, as we notice from Figure 3(a) that for reduces to nearly 0 for a relatively low value of p (around 0.5). As a result, using values of too high will cause the gradients to die (i.e. reduce to nearly 0) early, at a point at which the network’s predictions remain ambiguous, thereby causing the training process to fail.
How to choose As discussed, focal loss provides implicit entropy and weight regularisation, which heavily depend on the value of . Finding an appropriate is normally done using cross-validation. Also, traditionally, is fixed for all samples in the dataset. However, as shown, the regularisation effect for a sample i depends on , i.e. the predicted probability for the ground truth label for the sample. It thus makes sense to choose for each sample based on the value of . To this end, we provide Proposition 2 (proof in Appendix D), which we use to find a solution to this problem:
Proposition 2. Given a , for for all
1996]. Moreover, for and , the equality holds only for and
It is worth noting that there exist multiple values of where for all . For a given , Proposition 2 allows us to compute ; and (iii) for . This allows us to control the magnitude of the gradients for a particular sample i based on the current value of , and gives us a way of obtaining an informed value of for each sample. For instance, a reasonable policy might be to choose s.t. if is small (say less than 0.25), and otherwise. Such a policy will have the effect of making the weight updates larger for samples having a low predicted probability for the correct class and smaller for samples with a relatively higher predicted probability for the correct class.
Following the aforementioned arguments, we choose a threshold , and use Proposition 2 to obtain a policy such that is observably greater than . In particular, we use the following schedule: if , otherwise (note that : see Figure 3(a)). We find this policy to perform consistently well across multiple classification datasets and network architectures. Having said that, one can calculate multiple such schedules for following Proposition 2, using the intuition of having a relatively high for low values of and a relatively low for high values of
We conduct image and document classification experiments to test the performance of focal loss. For the former, we use CIFAR-10/100 [Krizhevsky, 2009] and Tiny-ImageNet [Deng et al., 2009] , and train ResNet-50, ResNet-110 [He et al., 2016], Wide-ResNet-26-10 [Zagoruyko and Komodakis, 2016] and DenseNet-121 [Huang et al., 2017] models, and for the latter, we use 20 Newsgroups [Lang, 1995] and Stanford Sentiment Treebank (SST) [Socher et al., 2013] datasets and train Global Pooling CNN [Lin et al., 2014] and Tree-LSTM [Tai et al., 2015] models. Further details on the datasets and training can be found in Appendix E.
Baselines Along with cross-entropy loss, we compare our method against the following baselines: a) MMCE (Maximum Mean Calibration Error) [Kumar et al., 2018], a continuous and differentiable proxy for calibration error that is normally used as a regulariser alongside cross-entropy, b) Brier loss [Brier, 1950], the squared error between the predicted softmax vector and the one-hot ground truth encoding (Brier loss is an important baseline as it can be decomposed into calibration and refinement [DeGroot and Fienberg, 1983]), and c) Label smoothing [Müller et al., 2019] (LS): given a one-hot ground-truth distribution q and a smoothing factor (hyperparameter), the smoothed vector s is obtained as denote the elements of s and q respectively, and K is the number of classes. Instead of q, s is treated as the ground truth. We train models using and , but find to perform better. We thus report the results obtained from LS-
Focal Loss: As mentioned in §4, our proposed approach is the sample-dependent schedule FLSD-53 (for , and for ), which we find to perform well across most classification datasets and network architectures. In addition, we also train other focal loss baselines, including ones with , and also ones that have a training epoch-dependent schedule for . Among the focal loss models trained with a fixed , using validation set we find to perform the best. Details of all these approaches can be found in Appendix F.
Temperature Scaling: In order to compute the optimal temperature, we use two different methods: (a) learning the temperature by minimising val set NLL, and (b) performing grid search over temperatures between 0 and 10, with a step of 0.1, and finding the one that minimises val set ECE. We find the second approach to produce stronger baselines and report results obtained using this approach.
Performance Gains: We report ECE% (computed using 15 bins) along with optimal temperatures in Table 1, and test set error in Table 2. We report the other calibration scores (AdaECE, Classwise-ECE, MCE and NLL) in Appendix F. Firstly, for all dataset-network pairs, we obtain very competitive classification accuracies (shown in Table 2). Secondly, it is clear from Table 1, and Tables F.1 and F.2 in the appendix, that focal loss with sample-dependent and with outperform all the baselines: cross-entropy, label smoothing, Brier loss and MMCE. They broadly produce the
Figure 4: Bar plots with confidence intervals for ECE, AdaECE and Classwise-ECE, computed for ResNet-50 (first 3 figures) and ResNet-110 (last 3 figures) on CIFAR-10.
Table 1: ECE (%) computed for different approaches both pre and post temperature scaling (cross-validating T on ECE). Optimal temperature for each method is indicated in brackets. innately calibrated model.
Table 2: Test set error (%) computed for different approaches.
lowest calibration errors both before and after temperature scaling. This observation is particularly encouraging, as it also indicates that a principled method of obtaining values of for focal loss can produce a very calibrated model, with no need to use validation set for tuning . As shown in Figure 4, we also compute 90% confidence intervals for ECE, AdaECE and Classwise-ECE using 1000 bootstrap samples following Kumar et al. [2019], and using ResNet-50/110 trained on CIFAR-10 (see Appendix G for more results). Note that FLSD-53 produces the lowest calibration errors in general, and the difference in the metric values between FLSD-53 and other approaches (except Brier loss) is mostly statistically significant (i.e., confidence intervals don’t overlap), especially before temperature scaling. In addition to the lower calibration errors, there are other advantages of focal loss as well, which we explore next.
More advantages of focal loss: Behaviour on Out-of-Distribution (OoD) data: A perfectly calibrated model should have low confidence whenever it misclassifies, including when it encounters data which is OoD [Thulasidasan et al., 2019]. Although temperature scaling calibrates a model under the i.i.d. assumption, it is known to fail under distributional shift [Ovadia et al., 2019]. Since focal loss has implicit regularisation effects on the network (see §4), we investigate if it helps to learn representations that are more robust to OoD data. To do this, we use ResNet-110 and Wide-ResNet-26-10 trained on CIFAR-10 and consider the SVHN [Netzer et al., 2011] test set and CIFAR-10-C [Hendrycks and Dietterich, 2019] with Gaussian noise corruption at severity 5 as OoD data. We use the entropy of the softmax distribution as the measure of confidence or uncertainty, and report the corresponding AUROC scores both before and after temperature scaling in Table 3. For both SVHN
Confident and Calibrated Models: It is worth noting that focal loss with sample-dependent has optimal temperatures that are very close to 1, mostly lying between 0.9 and 1.1 (see Table 1). This property is shown by the Brier loss and label smoothing models as well, albeit with worse calibration errors. By contrast, the temperatures for cross-entropy and MMCE models are significantly higher, with values lying between 2.0 and 2.8. An optimal temperature close to 1 indicates that the model is innately calibrated, and cannot be made significantly more calibrated by temperature scaling. In fact, a temperature much greater than 1 can make a model underconfident in general, as it is applied irrespective of the correctness of model outputs. We observe this empirically for ResNet-50 and ResNet-110 trained on CIFAR-10. Although models trained with cross-entropy have much higher confidence before temperature scaling than those trained with focal loss, after temperature scaling, focal loss models are significantly more confident in their predictions. We provide quantitative and qualitative empirical results to support this claim in Appendix H.
In this paper, we have studied the properties of focal loss, an alternative loss function that can yield classification networks that are more naturally calibrated than those trained using the conventional cross-entropy loss, while maintaining accuracy. In particular, we show in §4 that focal loss implicitly maximises entropy while minimising the KL divergence between the predicted and the target distributions. We also show that, because of its design, it naturally regularises the weights of a network during training, reducing NLL overfitting and thereby improving calibration. Furthermore, we empirically observe that models trained using focal loss are not only better calibrated under i.i.d. assumptions, but can also be better at detecting OoD samples which we show by taking CIFAR-10 as the in-distribution dataset and SVHN and CIFAR-10-C as out-of-distribution datasets, something which temperature scaling fails to achieve.
Our work shows that using the right kind of loss function can lead to a calibrated model. This helps in improving the reliability of these models when used in real-world applications. It can help in deployment of the models such that users can be alerted when its prediction may not be trustworthy. We do not directly see a situation where calibrated neural networks can have a negative impact on society, but we do believe that research on making models more calibrated will help improve fairness and trust in AI.
This work was started whilst J Mukhoti was at FiveAI, and completed after he moved to the University of Oxford. V Kulharia is wholly funded by a Toyota Research Institute grant. A Sanyal acknowledges support from The Alan Turing Institute under the Turing Doctoral Studentship grant TU/C/000023. This work was also supported by the Royal Academy of Engineering under the Research Chair and Senior Research Fellowships scheme, EPSRC/MURI grant EP/N019474/1 and FiveAI.
Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 1950.
Chensi Cao, Feng Liu, Hai Tan, Deshou Song, Wenjie Shu, Weizhong Li, Yiming Zhou, Xiaochen Bo, and Zhi Xie. Deep Learning and Its Applications in Biomedicine. Genomics, Proteomics & Bioinformatics, 16(1):17–32, 2018.
Hongming Chen, Ola Engkvist, Yinhai Wang, Marcus Olivecrona, and Thomas Blaschke. The rise of deep learning in drug discovery. Drug Discovery Today, 23(36):1241–1250, 2018.
Robert M Corless, Gaston H Gonnet, David EG Hare, David J Jeffrey, and Donald E Knuth. On the Lambert W Function. Advances in Computational Mathematics, 5(1):329–359, 1996.
Morris H DeGroot and Stephen E Fienberg. The Comparison and Evaluation of Forecasters. Journal of the Royal Statistical Society: Series D (The Statistician), 1983.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR, 2009.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On Calibration of Modern Neural Networks. In ICML, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. arXiv preprint arXiv:1903.12261, 2019.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely Connected Convolutional Networks. In CVPR, 2017.
Andreas Kamilaris and Francesc X Prenafeta-Boldú. Deep learning in agriculture: A survey. Computers and Electronics in Agriculture, 147:70–90, 2018.
Justin Ker, Lipo Wang, Jai Rao, and Tchoyoson Lim. Deep Learning Applications in Medical Image Analysis. IEEE Access, 6:9375–9389, 2018.
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. Technical report, Department of Computer Science, University of Toronto, 2009.
Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with Dirichlet calibration. In NeurIPS, 2019.
Ananya Kumar, Percy S Liang, and Tengyu Ma. Verified Uncertainty Calibration. In NeurIPS, 2019.
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable Calibration Measures For Neural Networks From Kernel Mean Embeddings. In ICML, 2018.
Ken Lang. NewsWeeder: Learning to Filter Netnews. In Machine Learning Proceedings 1995, pages 331–339. Elsevier, 1995.
Min Lin, Qiang Chen, and Shuicheng Yan. Network In Network. In ICLR, 2014.
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal Loss for Dense Object Detection. In ICCV, 2017.
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When Does Label Smoothing Help? In NeurIPS, 2019.
Mahdi Pakdaman Naeini and Gregory F Cooper. Binary Classifier Calibration using an Ensemble of Near Isotonic Regression Models. In ICDM, 2016.
Mahdi Pakdaman Naeini, Gregory F Cooper, and Milos Hauskrecht. Obtaining Well Calibrated Probabilities Using Bayesian Binning. In AAAI, 2015.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading Digits in Natural Images with Unsupervised Feature Learning. In NeurIPS-W, 2011.
Ng. 20 Newsgroups. https://github.com/aviralkumar2907/MMCE, 2018. Accessed: 2019-05-22.
Khanh Nguyen and Brendan O’Connor. Posterior calibration and exploratory analysis for natural language processing models. arXiv preprint arXiv:1508.05154, 2015.
Alexandru Niculescu-Mizil and Rich Caruana. Predicting Good Probabilities With Supervised Learning. In ICML, 2005.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D Sculley, Sebastian Nowozin, Joshua V Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. In NeurIPS, 2019.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. GloVe: Global Vectors for Word Representation. In Empirical Methods in Natural Language Processing (EMNLP), 2014.
Gabriel Pereyra, George Tucker, Jan Chorowski, Łukasz Kaiser, and Geoffrey Hinton. Regularizing Neural Networks by Penalizing Confident Output Distributions. arXiv preprint arXiv:1701.06548, 2017.
John Platt. Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods. Advances in Large Margin Classifiers, 10(3):61–74, 1999.
PyTorch-CIFAR. Train CIFAR10 with PyTorch, 2019. Available online at https://github. com/kuangliu/pytorch-cifar (as of 22nd May 2019).
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In Empirical Methods in Natural Language Processing (EMNLP), 2013.
Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks. In Association for Computational Linguistics (ACL), 2015.
Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks. In NeurIPS, 2019.
TreeLSTM. TreeLSTM. https://github.com/ttpro1995/TreeLSTMSentiment, 2015. Accessed: 2019-05-22.
Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas B Schön. Evaluating model calibration in classification. In AISTATS, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In NeurIPS, 2017.
Jinjiang Wang, Yulin Ma, Laibin Zhang, Robert X Gao, and Dazhong Wu. Deep learning for smart manufacturing: Methods and applications. JMSY, 48:144–156, 2018.
David Widmann, Fredrik Lindsten, and Dave Zachariah. Calibration tests in multi-class classification: A unifying framework. In NeurIPS, 2019.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. In ICML, 2001.
Bianca Zadrozny and Charles Elkan. Transforming Classifier Scores into Accurate Multiclass Probability Estimates. In SIGKDD, 2002.
Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks. In BMVC, 2016.
Figure A.1: The confidence values for training samples at different epochs during the NLL training of a ResNet-50 on CIFAR-10 (see §3). Top row: reliability plots using 25 confidence bins; bottom row: % of samples in each bin. As training progresses, the model gradually shifts all training samples to the highest confidence bin. Notably, it continues to do so even after achieving 100% training accuracy by the 300 epoch point.
In §A, we provide some empirical evidence for the observation made in §3 in the main paper using reliability plots. In §B, we discuss the relation between focal loss and a regularised KL divergence, where the regulariser is the entropy of the predicted distribution. In §C, we discuss the regularisation effect of focal loss on a simple setup, i.e. a generalised linear model trained on a simple data distribution. In §D, we show the proofs of the two propositions formulated in the main text. We then describe all the datasets and implementation details for our experiments in §E. In §F, we discuss additional approaches for training using focal loss, and also the results we get from these approaches. We also provide the Top-5 accuracies of several models as we speculate that calibrated models should have a higher softmax probability on the correct class even when they misclassify, as compared to models which are less calibrated. We further provide the results of evaluating our models using various metrics other than ECE (like AdaECE, Classwise-ECE, MCE and NLL). Next, in §G, we provide additional results related to the confidence interval experiments performed in §5 of the main paper. In §H, we provide empirical and qualitative results to show that models trained using focal loss are calibrated, whilst maintaining their confidence on correct predictions. In §I, we provide a brief extension of our discussion about Figure 2(e) in the main paper, with a plot of norms of features obtained from the last ResNet block during training. In §J, we provide some empirical evidence to support the claims we make in §4 of the main paper about early stopping. Finally, in §K, we discuss the performance of focal loss on the downstream task of machine translation with beam search. We choose machine translation as the downstream task because in machine translation, softmax vectors from a model are directly fed into the beam search algorithm, and hence more calibrated probability vectors should intuitively produce better translations
In this section, we provide some empirical evidence to support the observation made in §3 of the main paper that a model, even after attaining perfect training accuracy, can reduce the training NLL (loss) further by increasing the prediction confidences to match the ground-truth one-hot encoding. To empirically observe this, we use the ResNet-50 network used for the analysis in §3. We divide the confidence range [0, 1] into 25 bins, and present reliability plots computed on the training set at training epochs 100, 200, 300 and 350 (see the top row of Figure A.1). In Figure A.1, we also show the percentage of samples in each confidence bin. It is quite clear from these plots that over time, the network gradually pushes all of the training samples towards the highest confidence bin. Furthermore, even though the network has achieved 100% accuracy on the training set by epoch 300, it still pushes some of the samples lying in lower confidence bins to the highest confidence bin by epoch 350.
Here we show why focal loss favours accurate but relatively less confident solutions. We show that it inherently provides a trade-off between minimizing the KL-divergence and maximizing the entropy, depending on the strength of . We use and to denote the focal loss with parameter and cross entropy between and q, respectively. K denotes the number of classes and denotes the ground-truth probability assigned to the class (similarly for ). We consider the following simple extension of focal loss:
L
We know that . Combining this equality with the above inequality leads to:
In the case of one-hot encoding (Delta distribution for q), focal loss will maximize (let y be the ground-truth class index), the component of the entropy of corresponding to the ground-truth index. Thus, it will prefer learning such that is assigned a higher value
Figure B.1: Optimal various values of q.
(because of the KL term), but not too high (because of the entropy term), and will ultimately avoid preferring overconfident models (by contrast to cross-entropy loss). Experimentally, we found the solution of the cross-entropy and focal loss equations, i.e. the value of the predicted probability which minimises the loss, for various values of q in a binary classification problem (i.e. K = 2), and plotted it in Figure B.1. As expected, focal loss favours a more entropic solution that is closer to 0.5. In other words, as Figure B.1 shows, solutions to focal loss (Equation 2) will always have higher entropy than those of cross-entropy, depending on the value of
The behaviour of deep neural networks is generally quite different from linear models and the problem of calibration is more pronounced in the case of deep neural networks, hence we focus on analysing the calibration of deep networks in the paper. However, weight norm analysis for the initial layers is complex due to batchnorm and weight decay. Hence, to see the effect of weight magnification on miscalibration, here we use a simple network without batchnorm or weight decay, which is a generalised linear model, and a simple data distribution.
Setup We consider a binary classification problem. The data matrix is created by assigning each class, two normally distributed clusters such that the mean of the clusters are linearly
Figure C.1: (a): Norm of logits (b): Norm of weights.
separable. The mean of the clusters are situated on the vertices of a two-dimensional hypercube of side length 4. The standard deviation for each cluster is 1 and the samples are randomly linearly combined within each cluster in order to add covariance. Further, for 10% of the data points, the labels were flipped. 4000 samples are used for training and 1000 samples are used for testing. The model consists of a simple 2-parameter logistic regression model. . We train this model using both cross-entropy and focal loss with
Weight Magnification We have argued that focal loss implicitly regularizes the weights of the model by providing smaller gradients as compared to cross-entropy. This helps in calibration as, if all the weights are large, the logits are large and thus the confidence of the network is large on all test points, even on the misclassified points. When the model misclassifies, it misclassifies with a high confidence. Figure C.1 shows, for a generalised linear model, that the norm of the logits and the weights of a network blows for Cross Entropy as compared to Focal Loss.
High Confidence for mistakes Figures C.2 (b) and (c) show that running gradient descent with cross-entropy (CE) and focal loss (FL) both gives the same decision regions i.e. the weight vector points in the same region for both FL and CE. However, as we have seen that the norm of the weights is much larger for CE as compared to FL, we would expect the confidence of misclassified test points to be large for CE as compared to FL. Figure C.2 (a) plots a histogram of the confidence of the misclassified points and it shows that CE misclassifies almost always with greater than 90% confidence whereas FL misclassifies with much lower confidence.
Figure C.2: (a): Confidence of mis-classifications (b): Decision boundary of linear classifier trained using cross entropy (c): Decision boundary of linear classifier trained using focal loss
Here we provide the proofs of both the propositions presented in the main text. While Proposition 1 helps us understand the regularization effect of focal loss, Proposition 2 provides us the values in a principled way such that it is sample-dependent. Implementing the sample-dependent is very easy as implementation of the Lambert-W function [Corless et al., 1996] is available in standard libraries (e.g. python scipy).
Proposition 1. For focal loss and cross-entropy , the gradients is the focal loss hyperparameter, and w denotes the parameters of the last linear layer. Thus
Proof. Let w be the linear layer parameters connecting the feature map to the logit s. Then, using the chain rule,
. Similarly, . The derivative of the focal loss with respect to , the softmax output of the network for the true class , takes the form
straightforward to verify that if , then
Proposition 2. Given a , for for all
1996]. Moreover, for and , the equality holds only for and
Proof. We derive the value of for a given . From Proposition 4.1, we already know that
where is focal loss, is cross entropy loss, is the probability assigned by the model to the ground-truth correct class for the sample, and
For , if we look at the function , then we can clearly see that as , and that when p = 1. To observe the behaviour of for intermediate values of p, we first take its derivative with respect to p:
In Equation 5, except when p = 1 (in which case ). Thus, to observe the sign of the gradient , we focus on the term
Dividing Equation 6 by , the sign remains unchanged and we get
We can see that as and as (using l’Hôpital’s rule). Furthermore, is monotonically decreasing for . Thus, as the gradient is positive initially starting from p = 0 and negative later till p = 1, we can say that first monotonically increases starting from 1 (as ) and then monotonically decreases down to 0 (at p = 1). Thus, if for some threshold and for some , then , . We now want to find a such that . First, let and
where . We know that the inverse of is defined as x = W(y), where W is the Lambert-W function [Corless et al., 1996]. Furthermore, the r.h.s. of the inequality in Equation 8 is always negative, with a minimum possible value of that occurs at . Therefore, applying the Lambert-W function to the r.h.s. will yield two real solutions (corresponding to a principal branch denoted by and a negative branch denoted by ). We first consider the solution corresponding to the negative branch (which is the smaller of the two solutions):
If we consider the principal branch, the solution is
which yields a negative value for that we discard. Thus Equation 9 gives the values of if . In other words, , and for any
We use the following image and document classification datasets in our experiments: 1. CIFAR-10 [Krizhevsky, 2009]: This dataset has 60,000 colour images of size , divided equally into 10 classes. We use a train/validation/test split of 45,000/5,000/10,000 images.
2. CIFAR-100 [Krizhevsky, 2009]: This dataset has 60,000 colour images of size equally into 100 classes. (Note that the images in this dataset are not the same images as in CIFAR-10.) We again use a train/validation/test split of 45,000/5,000/10,000 images.
3. Tiny-ImageNet [Deng et al., 2009]: Tiny-ImageNet is a subset of ImageNet with 64 x 64 dimensional images, 200 classes and 500 images per class in the training set and 50 images per class in the validation set. The image dimensions of Tiny-ImageNet are twice that of CIFAR-10/100 images.
4. 20 Newsgroups [Lang, 1995]: This dataset contains 20,000 news articles, categorised evenly into 20 different newsgroups based on their content. It is a popular dataset for text classification. Whilst some of the newsgroups are very related (e.g. rec.motorcycles and rec.autos), others are quite unrelated (e.g. sci.space and misc.forsale). We use a train/validation/test split of 15,098/900/3,999 documents.
5. Stanford Sentiment Treebank (SST) [Socher et al., 2013]: This dataset contains movie reviews in the form of sentence parse trees, where each node is annotated by sentiment. We use the dataset version with binary labels, for which 6,920/872/1,821 documents are used as the training/validation/test split. In the training set, each node of a parse tree is annotated as positive, neutral or negative. At test time, the evaluation is done based on the model classification at the root node, i.e. considering the whole sentence, which contains only positive or negative sentiment.
All our experiments required a single 12 GB TITAN Xp GPU. For training networks on CIFAR-10 and CIFAR-100, we use SGD with a momentum of 0.9 as our optimiser, and train the networks for 350 epochs, with a learning rate of 0.1 for the first 150 epochs, 0.01 for the next 100 epochs, and 0.001 for the last 100 epochs. We use a training batch size of 128. Furthermore, we augment the training images by applying random crops and random horizontal flips. For Tiny-ImageNet, we train for 100 epochs with a learning rate of 0.1 for the first 40 epochs, 0.01 for the next 20 epochs and 0.001 for the last 40 epochs. We use a training batch size of 64. It should be noted that for Tiny-ImageNet, we saved 50 samples per class (i.e., a total of 10000 samples) from the training set as our own validation set to fine-tune the temperature parameter (hence, we trained on 90000 images) and we use the Tiny-ImageNet validation set as our test set.
For 20 Newsgroups, we train the Global Pooling Convolutional Network [Lin et al., 2014] using the Adam optimiser, with learning rate 0.001, and betas 0.9 and 0.999. The code is a PyTorch adaptation of Ng. We used Glove word embeddings [Pennington et al., 2014] to train the network. We trained all the models for 50 epochs and used the models with the best validation accuracy.
For the SST Binary dataset, we train the Tree-LSTM [Tai et al., 2015] using the AdaGrad optimiser with a learning rate of 0.05 and a weight decay of , as suggested by the authors. We used the constituency model, which considers binary parse trees of the data and trains a binary Tree-LSTM on them. The Glove word embeddings [Pennington et al., 2014] were also tuned for best results. The code framework we used is inspired by TreeLSTM. We trained these models for 25 epochs and used the models with the best validation accuracy.
For all our models, we use the PyTorch framework, setting any hyperparameters not explicitly mentioned to the default values used in the standard models. For MMCE, we used for all our experiments as we found it to perform better over all the values we tried. A calibrated model which does not generalise well to an unseen test set is not very useful. Hence, for all the experiments, we set the training parameters in a way such that we get best test set accuracies on all datasets for each model.
In addition to the sample-dependent approach, we try the following focal loss approaches as well:
Focal Loss (Fixed : We trained models on focal loss with produce the best ECE among models trained using a fixed . This corroborates the observation we made in §4 of the main paper that should produce better results than or , as the regularising effect for is higher.
Focal Loss (Scheduled : As a simplification to the sample-dependent approach, we also tried using a schedule for during training, as we expect the value of to increase in general for all samples over time. In particular, we report results for two different schedules: (a) Focal Loss (scheduled for the first 100 epochs, for the next 150 epochs, and last 100 epochs, and (b) Focal Loss (scheduled for the first 100 epochs, next 150 epochs, and for the last 100 epochs. We also tried various other schedules, but found these two to produce the best results on the validation sets.
Finally, for the sample-dependent approach, we also found the policy: Focal Loss (sample-dependent 5,3,2) with for for and for to produce competitive results.
In Tables F.1 and F.2, we present the AdaECE and Classwise-ECE scores for all the baselines discussed in Table 1 of the main paper.
In Table 2 of the main paper, we present the classification errors on the test datasets for all the major loss functions we considered. Here we also report the classification errors for the different focal loss approaches in Table F.6. We also report the ECE, Ada-ECE and Classwise-ECE for all the focal loss approaches in Table F.3, Table F.4 and Table F.5 respectively.
Finally, calibrated models should have a higher logit score (or softmax probability) on the correct class even when they misclassify, as compared to models which are less calibrated. Thus, intuitively, such models should have a higher Top-5 accuracy. In Table F.7, we report the Top-5 accuracies for all our models on datasets where the number of classes is relatively high (i.e., on CIFAR-100 with
In addition to ECE, Ada-ECE and Classwise-ECE, we use various other metrics to compare the proposed methods with the baselines (i.e. cross-entropy, Brier loss, MMCE and Label Smoothing). We present the test NLL % before and after temperature scaling in Tables F.8 and F.9, respectively. We report the test set MCE % before and after temperature scaling in Tables F.10 and F.11, respectively.
We use the following abbreviation to report results on different varieties of Focal Loss. FL-1 refers to Focal Loss (fixed 1), FL-2 refers to Focal Loss (fixed 2), FL-3 refers to Focal Loss (fixed 3), FLSc-531 refers to Focal Loss (scheduled 5,3,1), FLSc-532 refers to Focal Loss (scheduled 5,3,2), FLSD-532 refers to Focal Loss (sample-dependent 5,3,2) and FLSD-53 refers to Focal Loss (sample-dependent
Table F.3: ECE (%) computed for different focal loss approaches both pre and post temperature scaling (cross-validating T on ECE). Optimal temperature for each method is indicated in brackets.
Table F.6: Error (%) computed for different focal loss approaches.
Table F.7: Top-1 and Top-5 accuracies computed for different approaches.
Table F.11: MCE (%) computed for different approaches post temperature scaling (cross-validating T on ECE).
In this section, we present additional results in reference to Figure 4 in the main paper. In particular, we compute 90% confidence intervals for ECE, AdaECE and Classwise-ECE using 1000 bootstrap samples following Kumar et al. [2019] and present the resulting confidence intervals as bar plots in Figures G.1, G.2, G.3 and G.4. These plots further corroboate the observations made in Section 5 of the main paper. We find that FLSD-53 broadly produces the lowest calibration errors, and in
Figure G.1: Bar plots with confidence intervals for ECE, AdaECE and Classwise-ECE computed for ResNet-50 (first 3 figures) and ResNet-110 (last 3 figures) on CIFAR-100.
Figure G.2: Bar plots with confidence intervals for ECE, AdaECE and Classwise-ECE computed for Wide-ResNet-26-10 (first 3 figures) and DenseNet-121 (last 3 figures) on CIFAR-100.
Figure G.3: Bar plots with confidence intervals for ECE, AdaECE and Classwise-ECE computed for Wide-ResNet-26-10 (first 3 figures) and DenseNet-121 (last 3 figures) on CIFAR-10.
Figure G.4: Bar plots with confidence intervals for ECE, AdaECE and Classwise-ECE computed for ResNet-50 on Tiny-ImageNet.
quite a few cases (especially before temperature scaling) the differences in calibration errors between cross-entropy and focal loss are statistically significant.
In extension to what we present in Section 5 of the main paper, we also follow the approach adopted in Kumar et al. [2018], and measure the percentage of test samples that are predicted with a confidence
Table H.1: Percentage of test samples predicted with confidence higher than 99% and the corresponding accuracy for Cross Entropy, MMCE and Focal loss computed both pre and post temperature scaling (represented in the table as pre T and post T respectively).
of 0.99 or more (we call this set of test samples S99). In Table H.1, we report |S99| as a percentage of the total number of test samples, along with the accuracy of the samples in S99 for ResNet-50 and ResNet-110 trained on CIFAR-10, using cross-entropy loss, MMCE loss, and focal loss. We observe that |S99| for the focal loss model is much lower than for the cross-entropy or MMCE models before temperature scaling. However, after temperature scaling, |S99| for focal loss is significantly higher than for both MMCE and cross-entropy. The reason is that with an optimal temperature of 1.1, the confidence of the temperature-scaled model for focal loss does not reduce as much as those of the models for cross-entropy and MMCE, for which the optimal temperatures lie between 2.5 to 2.8. We thus conclude that models trained on focal loss are not only more calibrated, but also better preserve their confidence on predictions, even after being post-processed with temperature scaling.
Figure H.1: Qualitative results showing the performance of Cross Entropy, Brier loss, MMCE and Focal Loss (sample-dependent 5,3) for a ResNet-50 trained on CIFAR-10. The first row of images have been correctly classified by networks trained on all four loss functions and the second row of images have all been incorrectly classified. For each image, we present the actual label, the predicted label and the confidence of the prediction both before and after temperature scaling.
In Figure H.1, we present some qualitative results to support this claim and show the improvement in the confidence estimates of focal loss in comparison to other baselines (i.e., cross entropy, MMCE and Brier loss). For this, we take ResNet-50 networks trained on CIFAR-10 using the four loss functions (cross entropy, MMCE, Brier loss and Focal loss with sample-dependent 5,3) and measure the confidence of their predictions for four correctly and four incorrectly classified test samples. We report these confidences both before and after temperature scaling. It is clear from Figure H.1 that for all the correctly classified samples, the model trained using focal loss has very confident predictions both pre and post temperature scaling. However, on misclassified samples, we observe a very low confidence for the focal loss model. The ResNet-50 network trained using cross entropy is very confident even on the misclassified samples, particularly before temperature scaling. Apart from focal loss, the only model which has relatively low confidences on misclassified test samples is the one trained using Brier loss. These observations support our claim that focal loss produces not only a calibrated model but also one which is confident on its correct predictions.
As an extension to the discussion related to Figure 2(e) in the main paper, we plot the norm of the features/activations obtained from the last ResNet block (right before the linear layer is applied on these features to get the logits). We plot these norms throughout the training period for networks trained on cross-entropy and focal loss with set to 1, 2 and 3 in Figure I.1. We observe that there is a distinct ordering of feature norms for the four models: cross-entropy has the highest feature norm, followed by focal loss with , followed by focal loss with and finally focal loss with . Furthermore, this ordering is preserved throughout training. As we saw from Figure 2(e) in the main paper, from epoch 150 onwards (i.e., the epoch from which the networks start getting miscalibrated), there is a flip in the ordering of weight norms of the last linear layer. From epoch 150 onwards, the weight norms also follow the exact same ordering that we observe from Figure I.1 here. This shows that throughout training the initial layer weights (before last linear layer) of the network trained using focal loss are also regularized to favor lower norm of the output features, thus possibly leading to less peakiness in final prediction as compared to that of cross-entropy loss (see the ‘Peak at the wrong place’ paragraph of Section 3 of the main paper).
From Figure 2(a), one may think that intermediate models (using early stopping) might provide better accuracy and calibration. However, there is no ideal approach for early stopping. For fair comparison, we train ResNet50, CIFAR-10 using cross-entrpy and focal loss with the best (in hindsight) possible early stopping. We train each model for 350 epochs and choose the 3 intermediate models with the best val set ECE, NLL and classification error, respectively. We present the test set performance in Table J.1.
Figure I.1: norm of features obtained from the last ResNet block (before the linear layer) of ResNet-50 averaged over entire training dataset of CIFAR-10 using a batch size of 128.
Table J.1: Classification errors and ECE scores obtained from ResNet-50 models trained using cross-entropy and focal loss with different early stopping criteria (best in hindsight ECE, NLL and classification error on the validation set) applied during training. In the table CE and FL stand for cross-entropy and focal loss respectively and the Full Criterion indicates models where early stopping has not been applied.
Figure K.1: Changes in test set BLEU score and validation set ECE with temperature, for models trained using (a) cross-entropy with hard targets (CE) (b) cross-entropy with label smoothing (LS) (), and (c) focal loss (FL) (
From the table, we can observe that: 1) On every early stopping criterion, the model trained on focal loss outperforms the one trained on cross-entropy in both error and ECE, 2) ECE as a stopping criterion provides better test set ECE, but increases the test error significantly, 3) even without early stopping, focal loss achieves consistently better error and ECE compared to cross-entropy using any stopping criterion.
In this section, we explore machine translation with beam search as a relevant downstream task for calibration. Following the setup in Müller et al. [2019], we train the Transformer architecture [Vaswani et al., 2017] on the WMT 2014 English-to-German translation dataset. The training settings (like optimiser, LR schedule, etc.) are the same as Vaswani et al. [2017]. We chose machine translation as a relevant task because the softmax vectors produced by the transformer model are directly fed into the beam search algorithm, and hence softmax outputs from a calibrated model should intuitively produce better translations and a better BLEU score.
We train three transformer models, one on cross-entropy with hard target labels, the second on cross-entropy with label smoothing (with smoothing factor ) and the third on focal loss with . In order to compare these models in terms of calibration, we report the test set ECE (%) both before and after temperature scaling in the first row of Table K.1. Furthermore, to evaluate their performance on the English-to-German translation task, we also report the test set BLEU score of these models in the second row of Table K.1. Finally, to study the variation of test set BLEU score and validation set ECE with temperature, we plot them against temperature for all three models in Figure K.1.
Table K.1: Test set ECE and BLEU score both pre and post temperature scaling for cross-entropy (CE) with hard targets, cross-entropy with label smoothing (LS) () and focal loss (FL) (
We observe from Table K.1 that the model trained on focal loss outperforms its competitors on ECE and also has a competitive edge over other methods on BLEU score as well. The focal loss model also has an optimal temperature of 1, just like the model trained on cross-entropy with label smoothing. From Figure K.1, we can see that the models obtain the highest BLEU scores at around the same temperatures at which they obtain low ECEs, thereby confirming our initial notion that a more calibrated model provides better translations. However, since the optimal temperatures are tuned on the validation set, they don’t often correspond to the best BLEU scores on the test set. On the test set, the highest BLEU scores we observe are 26.33 for cross-entropy, 26.36 for cross-entropy with label smoothing, and 26.39 for focal loss. Thus, the performance of focal loss on machine translation (a downstream task related to calibration) is also very encouraging.