Distributed Low Precision Training Without Mixed Precision

2019·arXiv

Abstract

1 Introduction

Deep convolutional networks have widely been applied to the non-trivial machine learning problems in computer vision task [19, 18], and there has been a gradual advance in the model representation. However, complex models are computationally intensive, which makes it hard to deploy on embedded systems. To address this issue, there has been a surge of interests recently in reducing the model complexity of the deep model, including light-weight deep learning models utilizing computation/memory efficient operations, exemplified by MobileNet [14], ShuffleNet [29], quantization [4, 31, 17, 23] and pruning [10, 12]. Among these approaches, quantization that reduce the precision requirements for the weights and activations by representing network weights with low-precision (lower then FP32), thus yielding highly compact deep models compared to their floating-point counterparts.

Meanwhile, the quantization not only requires less working memory and cache but accelerate the computation speed and cuts down the energy consumption of the computational device. [6] has proved the feasibility of low-precision training in a well-conditioned situation. It has proved that reduce the precision of the data would not significantly change the distribution of the data, so it is feasible to replace float32 data with fixed-point data, which has dramatically reduced performance and space consumption. The details of the energy consumption of the different operations in different precision are presented in Table 1. However, it is necessary to adjust the order values to avoid overflow dynamically. Unfortunately, the statistical effects of low-precision computation during training are still not well understood. Therefore, if the pure low-precision computation is used throughout training, it is often challenging to match the statistical accuracies of traditional, higher-precision hardware architectures. Hence, the mixed-precision strategy is becoming popular. [4] proposed training with binary weights, all other tensors and arithmetic were in full precision. [5] extended that work also to binarize the activations, but gradients were stored and computed in single precision. [15] considered quantization of weights and activations to 2, 4 and 6 bits, gradients were real numbers. Besides the mixed-precision training strategy, most of the works are not adopted on large scale datasets like ImageNet [7] and very deep models like ResNet[11]. Fixed precision training is a strategy that aims to tackle the challenge of the above problems. Currently, the state-of-the-art mixed-precision training strategy like [20] using the strategy of maintaining a master copy of weights in FP32, loss-scaling that minimizes gradient values becoming zeros, and FP16 arithmetic with accumulation in FP32 indicating that strategy is not a full FP16 training process. This strategy has significantly increased the training cost and hard to implement on a large-scale distributed system. What is more, a fully FP16 training strategy could also take advantage of the NVIDIA Pascal GPU, which support to accelerate the whole training process twice [13]. With mentioned above disadvantages, we viewed that the mixed-precision training strategy is an essential exploration to achieve pure FP16 training.

Table 1: The energy consumption of different operations in different precision

Training on FP16 reduces memory consumption by half compared with FP32. The main challenge of non-full FP16 training is that the FP16 has a narrower dynamic range than FP32. Some operations like Batch Normalization [16] require the computation of the sum of squares, square-root, and reciprocal operations to avoid zero variance, which requires high precision and a broad dynamic range. Thus, some works avoid to use Batch Normalization layer [25] or kept the precision of the parameters unchanged [30] to avoid facing this challenge. To tackle this problem, [1] has proposed Range Batch Normalization that normalizes inputs by the range of the input distribution. However, all the existing quantization solutions are based on 1-bit inference, which is insensitive to the gamma in the Batch Normalization layer before each ReLU.

In this work, we proposed to easily swap the position of the Gamma/Beta to solve this problem, which is a more straightforward approach to solve this problem. We also developed a low-precision training as an approach to accelerating the deep neural networks (DNN) training approach. There is no doubt that low-precision training would cut down the budget of the GPU storages, so scaling full low-precision training is also appealing. However, even for the traditional FP32 training model, scaling ImageNet-1k training would still waste great computational resources, communication bandwidth. Moreover, end-to-end training would be more efficient to train the models. [27] proposed Layer-wise Adaptive Rate Scaling(LARS) to address this issue.

Our specific contributions are:

1. Proposed a fully FP16 training without FP32 throughout the training process on ImageNet while maintaining model accuracy. To our knowledge, our proposed strategy is the first fully FP16 training solution without sacrificing accuracy.

2. We proved the scalability of our proposed FP16 training strategy over the large-scale distributed deep learning training cluster without accuracy sacrificed.

2 Related Works

Reduced precision methods for deep learning training is an approach to improving compute ef-ficiency by reducing the precision requirements for the weights and activations by representing network weights with low-precision. Quantized weights significantly reduce memory size and access bandwidth, improve power efficiency exploit hardware-friendly bitwise operations, and accelerate inference throughput. Moreover, since the bandwidth is the major bottleneck of the distributed system [27], the mitigated bandwidth would also help to improve the utilization of the computational resources, thus accelerate the model training process. [9] has demonstrated that deep neural networks can be trained with minimal loss in accuracy, using 16-bit fixed-point representation. However, most of the low-precision training strategies were not using deep models like ResNet [11] and tested on large-scale datasets like ImageNet [7] because most low-precision training strategies are highly unstable. Accumulation of errors will collapse the accuracy of the whole model with the updating process by back propagation. It is challenging to deal with low-precision weights and stimulate researchers’ interest in new training methods. The main challenge lies when the learning rate is very small, the stochastic gradient method updates the weight parameters. [1] applied their proposed algorithm to deep convolutional models where they analyzed the quantization sensitivity of the network for each layer and then manually decide the quantization bitwidths. [24] quantized the weights and activations of pre-trained deep networks using 8-bit fixed-point representation to improve inference speed. After each training iteration, the weights are binarized/discretized and rounded up, which leads to training stagnation [5]. Some updating gradients are not helpful for the training process [2]. Thus the straightforward approach that quantizing weights with a rounding procedure yields unsatisfactory results when weights are represented in low-precision numbers. The other approaches address this problem by using mixed-precision training strategies that using full precision and single precision together during the training procedure [21]. [20] that maintaining a master copy of weights in FP32, loss-scaling that minimizes gradient values becoming zeros, and FP16 arithmetic with accumulation in FP32.

3 Precision Standard

Figure 1: IEEE 754 Floating Point Format

The IEEE Standard for Floating-Point Arithmetic (IEEE 754) [3] is a technical standard for floating-point computation, which specifies the exchange and arithmetic format and method of binary and decimal floating-point operations in a computer programming environment. The IEEE 754 standard requires support for a handful of operations. These include the arithmetic operations add, subtract, multiply, divide, square root, fused multiply-add, remainder, conversion operations, scaling, sign operations, and comparisons.

The standard mandates binary floating-point data be encoded on three fields: a one-bit sign field, followed by exponent bits encoding the exponent offset by a numeric bias specific to each format, and bits encoding the significant (or fraction).

To ensure consistent computations across platforms and to exchange floating-point data, IEEE 754 defines basic and interchange formats. However, although Google has introduced a BFloat16 training on its designed chips, Tensor Processing Unit (TPU) [26]. However, the design of BFloat16 has sacrificed computational versatility.

4 Full Low-Precision Training Strategy

In this section, we introduce a pure FP16 training strategy which is more memory-efficient than BFloat16 and Float32, without sacrificing training accuracy. We have explored the limitation of low-precision training.

4.1 Swap the BatchNorm

We have found Mantissa Underflow Risk and Exponent Underflow Risk during the Batch Normalization Layer training process. The underflow exception shall be signaled when a tiny non-zero result is detected.

Batch Normalization(BN) [16] is a popular technique that normalizes the activation statistics at the output of every layer, reducing dependencies across layers while significantly improving model accuracy. However, Batch Normalization requires some operations that require high precision and a broad dynamic range, including the computation of the sum of squares, square-root, and reciprocal operations to avoid zero variance, which requires high precision and a broad dynamic range. What is more, we analyze the variance of each layer , denoted by (which is technically defined as the sum of the variance of all the coordinates of ). For the deep structure like ResNet, prevents from vanishing by forcing the variance to grow with depth, i.e. . This causes the output variance to explode exponentially with depth without normalization for positively homogeneous blocks, which are detrimental to learning because it can, in turn, cause gradient explosion.

The most exponent-risky term is the gamma () in the BN. For a layer with - dimensional input , the traditional batch normalization would normalizes each dimension

where the given is the expectation of the is the batch size, and the is the variance of the given input. and are a pair of parameters to scale and shift the normalized value. The terminvolves sums of squares, and the learned that could lead to numerical instability as well as to arithmetic overflow when dealing with large values. Since the is to scale on the related highly unstable term , the potential of the arithmetic overflow would be

enlarged.

To tackle the above challenges, an intuitive solution is to learn the instead of . However, we have found that learning instead of would improve the training stability with hurting preci- sion, since IEEE standard intrinsically defined the product of positive MAX and positive MIN =

To address this problem, we propose to swap the position of residual connection and gamma () in BatchNorm after the last convolution layer in each Residual Block means that less processing and less precision requirement when adding up results from 2 different branches in the computing chip.

4.2 Mantissa Underflow Risk - Exponent Term Dilemma

We found the fact that there is a dilemma between the Mantissa Underflow risk and the Exponent Term that the less Mantissa Underflow risk the more Exponent Term. The Mantissa Underflow is the process of aligning mantissa; digits may flow off the right end of the mantissa. In such a case truncation method such as chopping, the round is used. The mantissa underflow risk comes from the range of weights, which is proven not sensitive in [28].

However, we found that it can be easily tackled if introducing LARS-like method.

We do not use LARS, but we multiples output by 0.25 right after every BatchNorm in each Residual Block. Meanwhile, the BatchNorm in the stem is kept as it is.

5 Experiments

5.1 Setup

In this section, we adopted the ResNet50 [11] on [7] a large-scaled GPU cluster with 128 GPUs. Our framework is PyTorch [22]. Hence, we have adopted an ablation study with 8, 64, 128 GPU set up with different precision in the same experiment set up. To note that, the learning rate and the batch size has a strong correlation. In our experiment, when the batch size is multiplied by k, multiply the learning rate by k. All other hyper-parameters (weight decay, etc.) are kept unchanged. Because the training strategy of FP32 has many skills and data enhancement strategies to improve the training effect, to make a fair comparison, the training strategy of Resnet50 in FP32 precision was used from the public repository2. All our experiments were conducted under the same settings. For large-scale training, all batch normalization is implemented in a single GPU, which means synchronous batch normalization is not used throughout the pipeline.

The initialization strategy of our experiments is a zero weight decay in all Batch Normalization layers, and zero in the last Batch Normalization layer in every Residual Block. To avoid a sudden increase in the learning rate, allowing healthy convergence at the start of training, we adopted our learning rate schedule as the same as [8], which we set 5 round warm-up [11], decay the learning rate by 0.1 at 30th/60th/80th epochs. The total length of our training is 90 epochs, which are the same as [8].

5.2 Ablation Study

Table 2 depicts the ablation study of our proposed method. Our baseline (FP32) model is the singleprecision storage, which is used for activations, weights, and gradients. All arithmetic is also in FP32. The FP16 precision indicated that all arithmetic and its related activations, weights, and gradients are all stored in FP16 precision. There is no master copy of any parameters that are stored in high precision. The experimental results show that the proposed pure FP16 training strategy does not sacrifice much precision. Under the same experimental conditions, compared with the traditional FP32 training strategy, our Top-1% accuracy is only reduced by 0.7%, which is acceptable in the large-scale image classification challenge.

Our experiments show it can be harmful to precision when the batch size is over , and is fatal to convergence when batch size is over if we continuously use the "the batch size is multiplied by k, multiply the learning rate by k." strategy in [8] and this is even cannot be alleviated by using LARS [27].

Table 2: Large-scale ResNet-50 training results.

6 Conclusion

In this work, we have analysed the details of the pure FP16 training strategy. We have established that the communication cost reduction, model compression and large-scale distributed training are three coupled problems. This is the first full FP16 training strategy on ResNet50 without sacrificing the accuracy on the ImageNet dataset, which is also the first solution for 5 exponent lower-bound for a wide range of deep learning training schemes.

7 Acknowledgement

We want to express our special appreciation and thanks to Feng Xiong from Tsinghua University, who has given us advice about the quantization and Yang You from Google Brain, who has given us great support on large-scaled distributed training.

References

[1] Banner, R., Hubara, I., Hoffer, E., and Soudry, D. (2018). Scalable methods for 8-bit training of neural networks. In Advances in Neural Information Processing Systems, pages 5145–5153.

[2] Cheng, Z. and Xu, Z. (2019). Bandwidth reduction using importance weighted pruning on ring allreduce. arXiv preprint arXiv:1901.01544.

[3] Committee, I. et al. (2008). 754–2008 ieee standard for floating-point arithmetic. IEEE Computer Society Std, 2008.

[4] Courbariaux, M., Bengio, Y., and David, J.-P. (2015). Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in neural information processing systems, pages 3123–3131.

[5] Courbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., and Bengio, Y. (2016). Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830.

[6] De Sa, C., Leszczynski, M., Zhang, J., Marzoev, A., Aberger, C. R., Olukotun, K., and Ré, C. (2018). High-accuracy low-precision training. arXiv preprint arXiv:1803.03383.

[7] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248–255. Ieee.

[8] Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K. (2017). Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677.

[9] Gupta, S., Agrawal, A., Gopalakrishnan, K., and Narayanan, P. (2015). Deep learning with limited numerical precision. In International Conference on Machine Learning, pages 1737–1746.

[10] Han, S., Pool, J., Tran, J., and Dally, W. (2015). Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pages 1135–1143.

[11] He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778.

[12] He, Y., Zhang, X., and Sun, J. (2017). Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 1389–1397.

[13] Ho, N.-M. and Wong, W.-F. (2017). Exploiting half precision arithmetic in nvidia gpus. In 2017 IEEE High Performance Extreme Computing Conference (HPEC), pages 1–7. IEEE.

[14] Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861.

[15] Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., and Bengio, Y. (2017). Quantized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research, 18(1):6869–6898.

[16] Ioffe, S. and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167.

[17] Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. (2018). Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2704–2713.

[18] Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105.

[19] LeCun, Y., Bengio, Y., and Hinton, G. (2015). Deep learning. nature, 521(7553):436.

[20] Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., et al. (2017). Mixed precision training. arXiv preprint arXiv:1710.03740.

[21] Mishra, A., Nurvitadhi, E., Cook, J. J., and Marr, D. (2017). Wrpn: wide reduced-precision networks. arXiv preprint arXiv:1709.01134.

[22] Paszke, A., Gross, S., Chintala, S., and Chanan, G. (2017). Pytorch: Tensors and dynamic neural networks in python with strong gpu acceleration. PyTorch: Tensors and dynamic neural networks in Python with strong GPU acceleration, 6.

[23] Rastegari, M., Ordonez, V., Redmon, J., and Farhadi, A. (2016). Xnor-net: Imagenet classifica-tion using binary convolutional neural networks. In European Conference on Computer Vision, pages 525–542. Springer.

[24] Vanhoucke, V., Senior, A., and Mao, M. Z. (2011). Improving the speed of neural networks on cpus. In Deep Learning and Unsupervised Feature Learning NIPS Workshop,.

[25] Wu, S., Li, G., Chen, F., and Shi, L. (2018). Training and inference with integers in deep neural networks. arXiv preprint arXiv:1802.04680.

[26] Ying, C., Kumar, S., Chen, D., Wang, T., and Cheng, Y. (2018). Image classification at supercomputer scale. arXiv preprint arXiv:1811.06992.

[27] You, Y., Zhang, Z., Hsieh, C.-J., Demmel, J., and Keutzer, K. (2018). Imagenet training in minutes. In Proceedings of the 47th International Conference on Parallel Processing, page 1. ACM.

[28] Zhang, D., Yang, J., Ye, D., and Hua, G. (2018a). Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 365–382.

[29] Zhang, X., Zhou, X., Lin, M., and Sun, J. (2018b). Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6848–6856.

[30] Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., and Zou, Y. (2016). Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160.

[31] Zhu, C., Han, S., Mao, H., and Dally, W. J. (2016). Trained ternary quantization. arXiv preprint arXiv:1612.01064.

Designed for Accessibility and to further Open Science