Quantization, in general, and deep quantization (below eight bits), inparticular, aimtonotonlyreducethecomputerequirements of DNNs but also significantly reduce their memory footprint (Zhou et al., 2016; Judd et al., 2016b; Hubara et al., 2017; Mishra et al., 2018; Sharma et al., 2018). Nevertheless, without specialized training algorithms, quantization can diminish the accuracy. As such, the practical utility of quantization hinges upon addressing two fundamental challenges: (1) discovering the appropriate bitwidth of quantization for each layer while considering the accuracy; and (2) learning weights in the quantized domain for a given set of bitwidths.
This paper formulates both of these problems as a gradient-based joint optimization problem by introducing in the training loss an additional and novel sinusoidal regularization term, called WaveQ. The following two main insights drive this work. (1) Sinusoidal functions (sin2) have inherent periodic minima and by adjusting the period, the minima can be positioned on quantization levels corresponding to a bitwidth at per-layer granularity. (2) As such, sinusoidal period becomes a direct and continuous representation of the bitwidth. Therefore, WaveQ incorporates this continuous variable (i.e., period) as a differentiable part of the training loss in the form of a regularizer. Hence, WaveQ can piggy back on the stochastic gradient descent that trains the neural network to also learn the bitwidth (the period). Simultaneously this parametric sinusoidal regularizer pushes the weights to the quantization levels (sin2 minima).
By adding our sinusoidal regularizer to the original training objective function, our method automatically yields the
Figure 1. Geometrical sketch for a hypothetical loss surface (original task loss to be minimized) and an extra regularization term in 2-D weight space visualizing the induced 2-D gradients for (a) weight decay, and (b) WaveQ respectively. wopt is the optimal solution considering the original task loss alone.
bitwidths for each layer along with nearly quantized weights for those bitwidths. In fact, the original optimization procedure itself is harnessed for this purpose, which is enabled by the differentiability of the sinusoidal regularization term. As such, quantized training algorithms (Zhou et al., 2016; Mishra et al., 2018) that still use some form of backpropagation (Rumelhart et al., 1986) can effectively utilize the proposed mechanism by modifying their loss. Moreover, the proposed technique is flexible as it enables heterogenous quantization across the layers. The WaveQ regularization can also be applied for training a model from scratch, or for fine-tuning a pretrained model.
In contrast to the prior inspiring works (Uhlich et al., 2019; Esser et al., 2019), WaveQ is the only technique that casts finding the bitwidthes and the corresponding quantized weights as a simultaneous gradient-based optimization through sinusoidal regularization during the training process. We also prove a theoretical result providing insights on why the proposed approach leads to solutions preserving the original accuracy while being prone to quantization. We evaluate WaveQ using different bitwidth assignments across different DNNs (AlexNet, CIFAR-10, MobileNet, ResNet-18, ResNet-20, SVHN, and VGG-11). To show the versatility of WaveQ, it is used with two different quantized training algorithms, DoReFa (Zhou et al., 2016) and WRPN (Mishra et al., 2018). Over all the bitwidth assignments, the proposed regularization, on average, improves the top-1 accuracy of DoReFa by 4.8%. The reduction in the bitwidth, on average, leads to 77.5% reduction in the energy consumed during the execution of these networks.
Our proposed method WaveQ exploits weight regularization in order to automatically quantize a neural network while training. To that end, Sections 2.1 describes the role of regularization in neural networks and then Section 2.2
explains WaveQ in more details.
2.1. Background
Loss landscape of neural networks. Neural networks’ loss landscapes are known to be highly non-convex and generally poorly understood. It has been empirically verified that loss surfaces for large neural networks have many local minima that are essentially equivalent in terms of test error (Choromanska et al., 2015; Li et al., 2018). This opens up the possibility of adding soft constrains as extra custom objectives to optimize during the training process, in addition to the original objective (i.e., to minimize the accuracy loss). The added constraint could be with the purpose of increasing generalization performance or imposing some preference on the weights values.
Regularization in neural networks. Neural networks often suffer from redundancy of parameterization and consequently they commonly tend to overfit. Regularization is one of the commonly used techniques to enhance generalization performance of neural networks. Regularization effectively constrains weight parameters by adding a term (regularizer) to the objective function that captures the desired constraint in a soft way. This is achieved by imposing some sort of preference on weight updates during the optimization process. As a result, regularization seamlessly leads to unconditionally constrained optimization problem instead of explicitly constrained which, in most cases, is much more difficult to solve.
Classical regularization: weight decay. The most commonly used regularization technique is known as weight decay, which aims to reduce the network complexity by limiting the growth of the weights, see Figure 1 (a). It is realized by adding a regularization term R to the objective function E that penalizes large weight values as follows:
where w is the collection of all synaptic weights, Eo is the original loss function, and is a parameter governing how strongly large weights are penalized. The j-th synaptic weight in the i-th layer of the network is denoted by wij.
2.2. WaveQ Regularization
Proposed objective. Here, we propose our sinusoidal based regularizer, WaveQ, which consists of the sum of two terms defined as follows:
R
where is the weights quantization regularization strength which governs how strongly weight quantization errors are
Figure 2. (a) 3-D visualization of the proposed generalized objective WaveQ. (b) WaveQ 2-D profile, w.r.t weights, adapting for arbitrary bitwidths, (c) example of adapting to ternary quantization. (d) WaveQ 2-D profile w.r.t bitwidth. (e) Regularization strengths profiles, , across training iterations.
penalized, and is the bitwidth regularization strength. The parameter
is proportional to the quantization bitwidth as will be further elaborated on below. Figure 2 (a) shows a 3-D visualization of our regularizer, R. Figure 2 (b), (c) show a 2-D profile w.r.t weights (w), while (d) shows a 2-D profile w.r.t the bitwidth
Periodic sinusoidal regularization. As shown in Equation (2.2), the first regularization term is based on a periodic function (sinusoidal) that provides a smooth and differentiable loss to the original objective, Figure 2 (b), (c). The periodic regularizer induces a periodic pattern of minima that correspond to the desired quantization levels. Such correspondence is achieved by matching the period to the quantization step (1) based on a particular number of bits (
) for a given layer i. For the sake of simplicity and clarity, Figure 1(a) and (b) depict a geometrical sketch for a hypothetical loss surface (original objective function to be minimized) and an extra regularization term in 2-D weight space, respectively. For weight decay regularization (Figure 1 (a)), the faded circle shows that as we get closer to the origin, the regularization loss is minimized. The point wopt is the optimum just for the loss function alone and the overall optimum solution is achieved by striking a balance between the original loss term and the regularization loss term. In a similar vein, Figure 1(b) shows a representation of the proposed periodic regularization for a fixed bitwidth
. A periodic pattern of minima pockets are seen surrounding the original optimum point. The objective of the optimization problem is to find the best solution that is the closest to one of those minima pockets where weight values are nearly matching the desired quantization levels, hence the name quantization-friendly.
Quantizer. Before delving into how our sinusoidal regularizer is used for quantization, we discuss how quantization works. Consider a floating-point variable w f to be mapped into a quantized domain using (b + 1) bits. Let Q be a set of (2k + 1) quantized values, where k . Considering linear quantization, Q can be represented as
, where 1k is the size of the quantization bin. Now, w f can be mapped to the b-bit quantization (Zhou et al., 2016) space as follows:
where quantizeb, w f is a scalar, Wf is a vector, and wqo is a scalar in the range
. Then, practically, a scaling factor c is determined per layer to map the final quantized weight wq into the range
. As such, wq takes the form cwqo, where c
Learning the sinusoidal period. The parameter the period of the sinusoidal regularizer for layer i, thereby
is directly proportional to the actual quantization bitwidth (bi) of layer i as follows:
where is a scaling factor. Note that bi
is the only discrete parameter, while
is a continuous real valued variable, and
is the ceiling operator. While the first term in Equation ((2.2)) is only responsible for promoting quantized weights, the second term enforces small bitwidths achieving a good accuracy-quantization trade-off. The main insight here is that the sinusoidal period is a continuous valued parameter by definition. As such,
that defines the period serves as an ideal optimization objective and a proxy to minimize the actual quantization bitwidth bi. Therefore, WaveQ avoids the issues of gradient-based optimization for discrete valued parameters. Furthermore, the benefit of learning the sinusoidal period is two-fold. First, it provides a smooth differentiable objective for finding minimal bitwidths. Second, simultaneously learning the scaling factor (
) associated with the found bitwidth.
Putting it all together. Leveraging the sinusoidal properties, WaveQ learns the following two quantization parameters simultaneously: (i) a per-layer quantization bitwidth (bi) along with (ii) a scaling factor () through learning the period of the sinusoidal function. Additionally, by exploiting the periodicity, differentiability, and the local convexity profile in
Figure 3. Visualization for three variants of the proposed regularization objective using different normalizations and their respective first and second derivatives with respect to R1
sinusoidal functions WaveQ automatically propels network weights towards values that are inherently closer to quantization levels according to the jointly learned quantizer’s parameters bi, defined in Equation (2.4). These learned parameters can be mapped to the quantizer parameters explained for Equation (2.3) in paragraph Quantizer.. For
bits quantization, k is set to 2b
is set to 2
Bounding the gradients. The denominator in the first term of equation (2.2) is used to control the range of variation of the derivatives of the proposed regularization term with respect to and is chosen to limit vanishing and exploding gradients during training. To this end, we compared three variants of equation (2.2) with different normalization defined, for k=0, 1, and 2, as:
Figure 3 (a), (b), (c) provide a visualization on how each of the proposed scaled variants impact the first and second derivatives. For R0 and R2, there are regions of vanishing or exploding gradients. Only the regularization R1 (the proposed one) is free of such issues.
Setting the regularization strengths. The convergence behavior depends on the setting of the regularization strengths and
. Since our proposed objective seeks to learn multiple quantization parameterization in conjunction, we divide the learning process into three phases, as shown in Figure 2 (e). In Phase ( 1 ), we primarily focus on optimizing for the original task loss E0. Initially, the small
and
values allow the gradient descent to explore the optimization surface freely. As the training process moves forward, we transition to phase ( 2 ) where the larger
engage both the weights quantization regularization and the bitwidth regularization, respectively. Note that, for this to work, the strength of the weights quantization regularization
should be higher than the strength of the bitwidth regularization
such that a bitwidth per layer could be properly evaluated and eventually learned during this phase. After the bitwidth regularizer converges to a bitwidth for each layer, we transition to phase ( 3 ), where we fix the learned bitwidths and gradually decay
while we keep
our experiments, we choose
such that the original loss and the penalty terms have approximately the same magnitude. It is worth noting that this way of progressivly setting the regularization strenghts across a multi-phase optimization resembles the settings of classical optimization algorithms, e.g. simulated annealing, where the temperature is progressively decreased from an initial positive value to zero or transitioning from exploration to exploitation. The mathematical formula used to generate
and
profiles across iterations can be found in the appendix (Fig. 9).
The results of this section are motivated as follow. Intuitively, we would like to show that the global minima of E are very close to the minima of E0 that minimizes R. In other words, we expect to extract among the original solutions, the ones that are most prone to be quantized. To establish such result, we will not consider the minima of E
, but the sequence Sn of minima of En
defined for any sequence
of real positive numbers. The next theorem shows that our intuition holds true, at least asymptotically with n provided
Theorem 1. Let E0be continuous and assume that the set SE0 of the global minima of E0 is non-empty and compact. As SE0 is compact, we can also define SE0
as the set of minima of E0 which minimizes R. Let
be a sequence of real positive numbers, define En
and the sequence Sn
of the global minima of En. Then, the following holds true:
1. If
2. If then there is a subsequence
and a non-empty set S
where the convergence of sets, denoted by Sn , is defined as the convergence to 0 of their Haussdorff distance, i.e., lim
n
Proof. For the first statement, assume that Sn . We wish to show that S
. Assume that xn is a sequence of global minima of F
converging to x
. It suffices to show that x
. First let us observe that x
. Indeed, let
and assume that x
Thus, since E0 is continuous and xn we have that E0
which implies x
. Next, define
Let ˆx so that R
. Now observe that, by the minimality of xn we have that
Thus, Ris continuous and xn
we have that R
which implies that R
since x
. Thus, x
. The second statement follows from the standard theory of Hausdorff distance on compact metric spaces and the first statement.
Theorem 1 implies that by decreasing the strength of R, one recovers the subset of the original solutions that achives the smallest quantization loss. In practice, we are not interested in global minima, and we should not decrease much the strength of R. In our context, Theorem 1 should then be understood as a proof of concept on why the proposed approach leads the expected result. Experiments carried out in the next section will support this claim. For the interested reader, we provide a more detailed version of the above analysis in the Appendix B.
To demonstrate the effectiveness of our proposed WaveQ, we evaluated it on several deep neural networks with different image classification datasets (CIFAR10, SVHN, and ImageNet). We provide results for two different types of quantization. First, we show quantization results for learned heterogenous bitwidths using WaveQ and we provide different arguments to asses the quality of these learned bitwidth assignments. Second, we further provide results assuming a preset homogenous bitwidth assignment as a special setting of WaveQ, which in some cases is a practical assumption that might stem from particular hardware requirements or constraints. Table 1 provides a summary of the evaluated networks and datasets for both learned heterogenous bitwidths, and the special case of training preset homogenous bitwidth assignments. We compare our proposed WaveQ method with PACT (Choi et al., 2018a), LQ-Nets (Zhang et al., 2018), DSQ (Gong et al., 2019), and DoReFa, which are current state-of-the-art (SOTA) methods that show results with 3-, and 4-bit weight/activation quantization for various networks architectures (AlexNet, ResNet-18, and MobileNet).
4.1. Experimental Setup
We implemented our technique inside Distiller (Zmora et al., 2018), an open source framework for compression by Intel Nervana. The reported accuracies for DoReFa and WRPN are with the built-in implementations in Distiller, which may not exactly match the accuracies reported in their respective papers. However, an independent implementation from a major company provides an unbiased foundation for the comparisons. We consider quantizing all convolution and fully connected layers, except for the first and last layers which may use higher precision.
4.2. Learned Heterogenous Bitwidth Quantization
Quantization levels with WaveQ. As for quantizing both weights and activations, Table 1 shows that incorporating WaveQ into the quantized training process yields best accuracy results outperforming PACT, LQ-Net, DSQ, and DoReFa with significant margins. Furthermore, it can be seen that the learned heterogenous btiwidths yield better accuracy as compared to the preset 4-bit homogenous assignments, with lower, on average, bitwidh (3.85-, 3.57-, and 3.95- bits for AlexNet, ResNet-18, and MobileNet, respectively).
Figure 5 (a), (b) (bottom bar graphs) show the learned heterogenous weight bitwidths over layers for AlexNet and ResNet-18, respectively. As can be seen, WaveQ objective learning shows a spectrum of varying bitwidth assignments to the layers which vary from 2 bits to 8 bits with an irregular pattern. These results demonstrate that the proposed regularization objective, WaveQ, automatically distinguishes different layers and their varying importance with respect to accuracy while learning their respective bitwidths.
Although, we can observe slight correlation of learning small bitwidths for layers with many parameters, e.g., fully connected layers, due to the strong inter-dependence between layers of neural networks, the resulting bitwidth assignments are generally complex, thereby there is no simple heuristic that can be deduced. As such, it is important to develop techniques to automatically learn a near-optimal bitwidth assignment for a given deep neural network. To assess the quality of these bitwidths assignments, we conduct a sensitivity analysis to the relatively big networks, and a Pareto analysis on the DNNs for which we could populate the search space as shown below.
Superiority of heterogenous quantization. Figure 5 (a), (b) (top graphs) show various comparisons and sensitivity results for learned heterogenous bitwidth assignments for bigger networks (AlexNet and ResNet-18) that are infeasible
Table 1. Comparison with state-of-the-art quantization methods on ImageNet. The “ W/A ” values are the bitwidths of weights/activations.
Figure 4. Quantization space in terms of computation and accuracy for (a) CIFAR-10, (b) SVHN, and (c) VGG-11
to enumerate their respective quantization spaces. Compared to 4-bit homogenous quantization, it can be seen that learned heterogenous assignments achieve better accuracy with lower, on average, bitwidth 3.85 bits for AlexNet and 3.57 bits for ResNet-18. This demonstrates that a homogenous (uniform) assignment of the bits is not always the desired choice to preserve accuracy. Furthermore, Figure 5 also shows that decrementing the learned bitwidth for any single layer at a time results in 0.44% and 0.24% reduction in accuracy on average (across all layers of the network) for AlexNet and ResNet-18, respectively, which further demonstrates the learning quality of WaveQ.
Validation: Pareto analysis. Figure 4 (a) shows a sketch of the multi-objective optimization problem of layer-wise quantization of a neural network showing the underlying
design space and the different design components. Given a particular architecture and a training technique, different combinations of layer-wise quantization bitwidths form a network specific design space (possible solutions). The design space can be divided into two regions. Region 1 represents the set of combinations of layer-wise quantization bitwidths that preserves the accuracy. On the other side, region 2 represents the set of all the remaining combinations of layer-wise quantization bitwidths that are associated with some sort of accuracy loss. As number of bits (on average across layers) increases, the amount of compute increases (considering full precision solution ( 3 ) corresponds to the max amount of compute). The objective is to find the least (on average) combination of bitwidths that still preserves the accuracy (i.e., solution 4 ; the solution at the interface of the
Table 2. Comparing accuracies of different networks using plain WRPN, plain DoReFa and DoReFa + WaveQ on fixed homogenous weight quantization.
two regions). As mentioned, that is for a particular training technique. Utilizing an improved quantized training technique modifies the envelop of the design space by expanding region 1 . In other words, it pushes the desired solution (initially 4 ) to a lower combination of bitwidths 5 (i.e., preserves the accuracy with lower bitwidths on average). Note that this is just a sketch for illustration purposes and does not imply that the actual envelope of region 2 is linear. Figure 4 (b)-(d) depicts actual solutions spaces for three benchmarks (CIFAR10, SVHN, and VGG11) in terms of computation versus accuracy. Each point on these charts is a unique combination of bitwidths that are assigned to the layers of the network. The boundary of the solutions denotes the Pareto frontier and is highlighted by a dashed line. The solution found by WaveQ is marked out using an arrow and lays on the desired section of the Pareto frontier where the compute intensity is minimized while keeping the accuracy intact, which demonstrates the quality of the obtained solutions. It is worth noting that as a result of the moderate size of the three networks presented in this subsection, it was possible to enumerate the design space, obtain Pareto frontier and assess WaveQ quantization policy for each of the three networks. However, it is infeasible to do so for state-of-the-art deep networks (e.g., AlexNet and ResNet) which further stresses the importance of automation and efficacy of WaveQ.
Energy savings. To further demonstrate the energy savings of the solutions found by WaveQ, we evaluate it on Stripes (Judd et al., 2016a), a custom accelerator designed for DNNs, which exploits bit-serial computation to support flexible bitwidths for DNN operations. As shown in Table 1, the reduction in the bitwidth, on average, leads to 77.5% reduction in the energy consumed during the execution of these networks.
4.3. Preset Homogenous Bitwidth Quantization
Now, we consider a preset homogenous bitwidth quantization which can also be supported by the proposed WaveQ under special settings where we fix (to a preset bitwidth), thus only the first regularization term is engaged for weight quantization. Table 2 shows results comparison of different networks (SimpleNet-5, ResNet-20, VGG-11, and SVHN-8) using plain WRPN, plain DoReFa and DoReFa + WaveQ considering preset 3-, 4-, and 5- bitwdith assignments. As can be seen, These results concretely show the impact of incorporating WaveQ into existing quantized training techniques and how it outperforms previously reported accuracies of several SOTA methods.
Semi-quantized weight distributions. Figure 6 shows the evolution of weights distributions over fine-tuning epochs for different layers of CIFAR10, SVHN, AlexNet, and ResNet-18 networks. The high-precision weights form clusters and gradually converge around the quantization centroids as regularization loss is minimized along with the main accuracy loss.
We conduct an experiment that uses WaveQ for training fromscratch. Forthesakeofclarity, weareconsideringinthis experiment the case of preset bitwidth assignments (i.e., 0). Figure 7-Row(I)-Column(I) shows weight trajectories without WaveQ as a point of reference. Row(II)-Column(I) shows the weight trajectories when WaveQ is used with a constant
. As Figure 7-Row(II)-Column(I) illustrates, using a constant
results in the weights being stuck in a region close to their initialization, (i.e., quantization objective dominates the accuracy objective). However, if we dynamically change the
following the exponential curve in Figure 7-Row(III)-Column(I)) during the from-scratch training, the weights no longer get stuck. Instead, the weights traverse the space (i.e., jump from wave to wave) as illustrated in Figure 7-Columns(II) and (III) for CIFAR and SVHN, respectively. In these two columns, Rows (I), (II), (III), correspond to quantization with 3, 4, 5 bits, respectively. Initially, the smaller
values allow the gradient descent to explore the optimization surface freely, as the training process moves forward, the larger
gradually engages the sinusoidal regularizer, and eventually pushes the weights close to the quantization levels. Further convergence analysis is provided in the Appendix A.
This research lies at the intersection of (1) quantized training algorithms and (2) techniques that discover bitwidth for quantization. The following diuscusses the most related works in both directions. In contrast, WaveQ modifies the loss function of the training to simultaneously learn the period of an adaptive sinusoidal regularizer through the same stochastic
Figure 5. Quantization bitwidth assignments across layers. It can be seen that learned heterogenous assignments achieve (with lower on average bitwidth) better accuracy as compared to fixed homogenous assignments. (a) AlexNet (average bitwidth = 3.85 bits). (b) ResNet-18 (average bitwidth = 3.57 bits)
Figure 6. Evolution of weight distributions over training epochs (with the proposed regularization) at different layers and bitwidths for different networks. (a) CIFAR10, second convolution layer with 3 bits, top row: mid-rise type of quantization (shifting by half a step to exclude zero as a quantization level); bottom row: mid-tread type of quantization (zero is included as a quantization level). (b) SVHN, top row: first convolution layer with 4 bits quantization. (c) AlexNet, second convolution layer with 4 bits quantization, and (d) ResNet-18, , second convolution layer with 4 bits quantization.
gradient descent that trains the network. The diffrentionablity of the adaptive sinusoidal regaulrizer enables simultaneously learning both the bitwidthes and pushing the wight values to the quantization levels. As such, WaveQ can be used as a complementary method to some of these efforts, which is demonstrated by experiments with both DoReFa-Net (Zhou et al., 2016) and WRPN (Mishra et al., 2018).
Figure 7. Weight trajectories. The 10 colored lines in each plot denote the trajectory of 10 different weights.
Our preliminary efforts (Elthakeb et al., 2019b) and another work concurrent to it (Naumov et al., 2018) use a sinusoidal regularization to push the weights closer to the quantization levels. However, neither of these two works make the period a differentiable parameter nor find bitwidths during training.
Quantized training algorithms There have been several techniques (Zhou et al., 2016; Zhu et al., 2017; Mishra et al., 2018) that train a neural network in a quantized domain after the bitwidth of the layers is determined manually. DoReFaNet (Zhou et al., 2016) uses straight through estimator (Bengio et al., 2013) for quantization and extends it for any arbitrary k bit quantization. DoReFa-Net generalizes the method of binarized neural networks to allow creating a CNN that has arbitrary bitwidth below 8 bits in weights, activations, and gradients. WRPN (Mishra et al., 2018) is training algorithm that compensates for the reduced precision by increasing the number of filter maps in a layer (doubling or tripling). TTQ (Zhu et al., 2017) quantizes the weights to ternary values by using per layer scaling coefficients that are learnt during training. These scaling coefficients are used to scale the weights during inference. PACT (Choi et al., 2018a) pro- posesatechniqueforquantizingactivationsbyintroducingan activation clipping parameter . This parameter (
to represent the clipping level in the activation function and is learned via back-propagation during training. More recently, VNQ (Achterhold et al., 2018) uses a variational Bayesian approach for quantizing neural network weights during training. DCQ (Elthakeb et al., 2019a) employs sectional multibackpropagation algorithm that leverages multiple instances of knowledge distillation and intermediate feature representations to teach a quantized student through divide and conquer.
Loss-aware weight quantization. Recent works pursued loss-aware minimization approaches for quantization. (Hou etal.,2017)and(Hou&Kwok,2018)developedapproximate solutions using proximal Newton algorithm to minimize the loss function directly under the constraints of low bitwidth weights. One effort (Choi et al., 2018b) proposed to learn the quantization of DNNs through a regularization term of the mean-squared-quantization error. LQ-Net (Zhang et al., 2018) proposes to jointly train the network and its quantizers. The quantizer is a inner product between a basis vector and the binary coding vector. DSQ (Gong et al., 2019) employs a series of tanh functions to gradually approximate the staircase function for low-bit quantization (e.g., sign for 1-bit case), and meanwhile keeps the smoothness for easy gradient calculation. Although some of these techniques use regularization to guide the process of quantized training, none explores the use of adaptive sinusoidal regularizers for quantization. Moreover, unlike WaveQ, these techniques do not find the bitwidth for quantizing the layers.
Techniques for discovering quantization bitwidths. A recent line of research focused on methods which can also find the optimal quantization parameters, e.g., the bitwidth, the stepsize, in parallel to the network weights. Recent work (Ye et al., 2018) based on ADMM runs a binary search to minimize the total square quantization error in order to decide the quantization levels for the layers. They use a heuristic-based iterative optimization technique for fine-tuning. Most recently, (Uhlich et al., 2019) proposed to indirectly learn quantizer’s parameters via Straight Through Estimator (STE) (Bengio et al., 2013) based approach. In a similar vein, (Esser et al., 2019) has proposed to learn the quantization mapping for each layer in a deep network by approximating the gradient to the quantizer step size that is sensitive to quantized state transitions. On another side, recent works (Elthakeb et al., 2018) proposed a reinforcement learning based approach to find an optimal bitwidth assignment policy.
This paper provided a new approach in using sinusoidal regularizations to cast the two problems of finding bitwidth levels for layers and quantizing the weights as a gradient- based optimization through sinusoidal regularization. While this technique consistently improves the accuracy, WaveQ does not require changes to the base training algorithm or the neural network topology.
Achterhold, J., K¨ohler, J. M., Schmeink, A., and Genewein, T. Variational network quantization. In 6th ICLR, 2018.
Bengio, Y., L´eonard, N., and Courville, A. C. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432, 2013. URL http://arxiv.org/abs/1308.3432.
Choi, J., Wang, Z., Venkataramani, S., Chuang, P. I.-J., Srinivasan, V., and Gopalakrishnan, K. Pact: Parameterized clipping activation for quantized neural networks. CoRR, abs/1805.06085, 2018a.
Choi, Y., El-Khamy, M., and Lee, J. Learning low precision deep neural networks through regularization. CoRR, abs/1809.00095, 2018b.
Choromanska, A., Henaff, M., Mathieu, M., Arous, G. B., and LeCun, Y. The Loss Surfaces of Multilayer Networks. In Artificial Intelligence and Statistics, 2015.
Elthakeb, A. T., Pilligundla, P., Mireshghallah, F., Yazdanbakhsh, A., and Esmaeilzadeh, H. ReLeQ: A reinforcement learning approach for deep quantization of neural networks. Advances in Neural Information Processing Systems (NeurIPS) Workshop on Machine Learning for Systems, 2018. URL https://arxiv.org/abs/1811.01704.
Elthakeb, A. T., Pilligundla, P., and Esmaeilzadeh, H. Divide and Conquer: Leveraging intermediate feature representations for quantized training of neural networks. International Conference on Machine Learning (ICML) Workshop on Understanding and Improving Generalization in Deep Learning, 2019a. URL https://arxiv.org/abs/1906.06033.
Elthakeb, A. T., Pilligundla, P., and Esmaeilzadeh, H. SinReQ: Generalized sinusoidal regularization for low-bitwidth deep quantized training. International Conference on Machine Learning (ICML) Workshop on Understanding and Improving Generalization in Deep Learning, 2019b. URL https://arxiv.org/abs/1905.01416.
Esser, S. K., McKinstry, J. L., Bablani, D., Appuswamy, R., and Modha, D. S. Learned step size quantization. 8th ICLR, 2020, abs/1902.08153, 2019. URL http://arxiv.org/abs/1902.08153.
Gong, R., Liu, X., Jiang, S., Li, T., Hu, P., Lin, J., Yu, F., and Yan, J. Differentiable soft quantization: Bridging full-precision and low-bit neural
networks. CoRR, abs/1908.05033, 2019. URL http://arxiv.org/abs/1908.05033.
Hou, L. and Kwok, J. T. Loss-aware weight quantization of deep networks. In 6th ICLR, 2018.
Hou, L., Yao, Q., and Kwok, J. T. Loss-aware binarization of deep networks. In 5th ICLR, 2017.
Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., and Bengio, Y. Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations. J. Mach. Learn. Res., 2017.
Judd, P., Albericio, J., Hetherington, T. H., Aamodt, T. M., and Moshovos, A. Stripes: Bit-serial deep neural network computing. In 49th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO 2016, Taipei, Taiwan, October 15-19, 2016, pp. 19:1–19:12. IEEE Computer Society, 2016a. doi: 10.1109/MICRO.2016.7783722. URL https://doi.org/10.1109/MICRO.2016.7783722.
Judd, P., Albericio, J., Hetherington, T. H., Aamodt, T. M., and Moshovos, A. Stripes: Bit-serial deep neural network computing. 49th MICRO, pp. 1–12, 2016b.
Li, H., Xu, Z., Taylor, G., Studer, C., and Goldstein, T. Visualizing the Loss Landscape of Neural Nets. In NIPS, 2018.
Mishra, A. K., Nurvitadhi, E., Cook, J. J., and Marr, D. WRPN: Wide Reduced-Precision Networks. In ICLR, 2018.
Naumov, M., Diril, U., Park, J., Ray, B., Jablonski, J., and Tulloch, A. On periodic functions as regularizers for quantization of neural networks. CoRR, abs/1811.09862, 2018. URL http://arxiv.org/abs/1811.09862.
Rumelhart, D. E., Hinton, G. E., and Williams, R. J. Learning internal representations by error propagation. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Volume 1: Foundations, pp. 318–362. MIT Press, Cambridge, MA, 1986.
Sharma, H., Park, J., Suda, N., Lai, L., Chau, B., Chandra, V., and Esmaeilzadeh, H. Bit fusion: Bit-level dynamically composable architecture for accelerating deep neural network. ISCA, pp. 764–775, 2018.
Uhlich, S., Mauch, L., Yoshiyama, K., Cardinaux, F., Garc´ıa, J. A., Tiedemann, S., Kemp, T., and Nakamura, A. Mixed precision dnns: All you need is a good parametrization. abs/1905.11452, 2019. URL http://arxiv.org/abs/1905.11452.
Wang, K., Liu, Z., Lin, Y., Lin, J., and Han, S. HAQ: hardware-aware automated quantization with mixed precision. In IEEE Conference on Computer Vision and Pattern
Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 8612–8620. Computer Vision Foundation / IEEE, 2019. doi: 10.1109/CVPR.2019.00881. URL http://openaccess.thecvf.com/content CVPR 2019/ html/Wang HAQ Hardware-Aware Automated Quantization With Mixed Precision CVPR 2019 paper.html.
Ye, S., Zhang, T., Zhang, K., Li, J., Xie, J., Liang, Y., Liu, S., Lin, X., and Wang, Y. A unified framework of dnn weight pruning and weight clustering/quantization using admm. CoRR, abs/1811.01907, 2018.
Zhang, D., Yang, J., Ye, D., and Hua, G. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In ECCV, pp. 373–390, 2018. doi: 10.1007/978-3-030-01237-3\ 23.
Zhou, S., Ni, Z., Zhou, X., Wen, H., Wu, Y., and Zou, Y. DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients. CoRR, 2016.
Zhu, C., Han, S., Mao, H., and Dally, W. J. Trained Ternary Quantization. In ICLR, 2017.
Zmora, N., Jacob, G., and Novik, G. Neural network distiller, June 2018.
Figure 8. Convergence behavior: accuracy and WaveQ regularization loss over fine-tuning epochs for (a) CIFAR10, (b) SVHN. Comparing convergence behavior with and without WaveQ during training from scratch (c) accuracy, (d) training loss. Network: VGG-11, 2-bit DoReFa quantization
Figure 8 (a), (b) show the convergence behavior of WaveQ by visualizing both accuracy and regularization loss over finetuning epochs for two networks: CIFAR10 and SVHN. As can be seen, the regularization loss (WaveQ Loss) is minimized across the finetuning epochs while the accuracy is maximized. This demonstrates a validity for the proposed regularization being able to optimize the two objectives simultaneously. Figure 8 (c), (d) contrasts the convergence behavior with and without WaveQ for the case of training from scratch for VGG-11. As can be seen, at the onset of training, the accuracy in the presence of WaveQ is behind that without WaveQ. This can be explained as a result of optimizing for an extra objective in case of withWaveQ as compared to without. Shortly thereafter, the regularization effect kicks in and eventually achieves 6% accuracy improvement.
The convergence behavior, however, is primarily controlled by the regularization strengths . As briefly mentioned in Section 2.2,
is a hyperparameter that weights the relative contribution of the proposed regularization objective to the standard accuracy objective.
We reckon that careful setting of across the layers and during the training epochs is essential for optimum results (Choi et al., 2018b).
B.1. Motivation
The results of this section are motivated by the following question.
Question B.1. Suppose that a function F :has many global minima and that Q
is closed. How do we isolate the global minima of F that are closest to Q without actually computing the full set of global minima of F?
Intuitively, we would like to show that if is very small, then the global minima of the function
are very close to the global minima of F closest to Q. To achieve this we will have to introduce first the concept of convergence of sets and then we will show that our intuition is correct by proving that the set of global minima to the above relaxed function converges to a subset of global minima of F closest to Q.
B.2. Relevant Definitions
Definition B.2. If F : satisfies lim
, we will say that F is coercive.
Definition B.3. For a coercive function F : we let SF
be coercive.
Lemma B.4. Assume that F :is continuous and coercive. Then F has at least one global minimum. That is, SF is non-empty. Furthermore, SF is a compact set.
Definition B.5. Let Fare continuous and assume that F is coercive. Define
the minima of F which minimize G among the minima of F.
Definition B.6. Let Qbe a closed set and assume that x
. Define the distance from x to the set Q to be
Observe that since Q is a closed set we have that xonly if d(x,Q)=0 and otherwise d(x,Q)>0.
Definition B.7. Let Abe compact sets. We define the Hausdorff distance between A and B by
Observe that dH(A,B)=0 if and only if A=B.
Definition B.8. Let be a family of compact subsets of
. We say that lim
Lemma B.9. Let Sbe a family of compact subsets of
, then lim
if and only if the following two conditions hold.
1. If xconverges to x, then x
2. For every x , there exists a family x
with x
The lemma is just an exercise in the definition.
B.3. Statement of the Theorem
Theorem 2. Let Fare continuous and assume that F is coercive. Consider the sets SF
, the set of points at which F
is globally minimum. The following are true:
Figure 9. Math formula for setting during training iterations.
2. If then there is a subsequence
and a non-empty set S
Proof. The second statement follows from the standard theory of Hausdorff distance on compact metric spaces and the first statement. For the first statement, assume that SF. We wish to show that S
. Assume that xn is a sequence of global minima of F
converging to x
. It suffices to show that x
. First let us observe that x
. Indeed, let
and assume that x
Thus, since F is continuous and xn we have that F
which implies x
. Next, define
Let ˆx so that G
. Now observe that, by the minimality of xn we have that
Thus,
for all n. Since G is continuous and xn we have that G
which implies that G
x