LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression

2020·Arxiv

Abstract

Abstract

BERT is a cutting-edge language representation model pre-trained by a large corpus, which achieves superior performances on various natural language understanding tasks. However, a major blocking issue of applying BERT to online services is that it is memory-intensive and leads to unsatisfactory latency of user requests. Existing solutions leverage knowledge distillation frameworks to learn smaller models that imitate the behaviors of BERT. However, the training procedure of knowledge distillation is expensive itself as it requires sufficient training data to imitate the teacher model. In this paper, we address this issue by proposing a hybrid solution named LadaBERT (Lightweight adaptation of BERT through hybrid model compression), which combines the advantages of different model compression methods, including weight pruning, matrix factorization and knowledge distillation. LadaBERT achieves state-of-the-art accuracy on various public datasets while the training overheads can be reduced by an order of magnitude.

1 Introduction

The pre-trained language model, BERT (Devlin et al., 2018) has led to a big breakthrough in various kinds of natural language understanding tasks. Ideally, people can start from a pre-trained BERT checkpoint and fine-tune it on a specific downstream task. However, the original BERT models are memoryexhaustive and latency-prohibitive to be served in embedded devices or CPU-based online environments. As the memory and latency constraints vary in different scenarios, the pre-trained BERT model should be adaptive to different requirements with accuracy retained to the largest extent. Existing BERT-oriented model compression solutions largely depend on knowledge distillation (Hinton et al., 2015), which is inefficient and resource-consuming because a large training corpus is required to learn the behaviors of a teacher. For example, DistilBERT (Sanh et al., 2019) is re-trained on the same corpus as pre-training a vanilla BERT from scratch; and TinyBERT (Jiao et al., 2019) utilizes expensive data augmentation to fit the distillation target. The costs of these model compression methods are as large as pre-training, which are unaffordable for low-resource settings. Therefore, it is straight-forward to ask, can we design a lightweight method to generate adaptive models with comparable accuracy using significantly less time and resource consumption?

In this paper, we propose LadaBERT (Lightweight adaptation of BERT through hybrid model compression) to tackle this problem. Specifically, LadaBERT is based on an iterative hybrid model compression framework consisting of weighting pruning, matrix factorization and knowledge distillation. Initially, the architecture and weights of student model are inherited from the BERT teacher. In each iteration, the student model is first compressed by a small ratio based on weight pruning and matrix factorization, and is then fine-tuned under the guidance of teacher model through knowledge distillation. Because weight pruning and matrix factorization help to generate better initial and intermediate status for knowledge distillation, both accuracy and efficiency can be greatly improved.

We conduct extensive experiments on five public datasets of natural language understanding. As an example, the performance comparison of LadaBERT and state-of-the-art models on MNLI-m dataset is illustrated in Figure 1.

Figure 1: Accuracy comparison on MNLI-m dataset

We can see that LadaBERT outperforms other BERT-oriented model compression baselines at various model compression ratios. Especially, LadaBERT outperforms BERT-PKD sig-nificantly under compression ratio and outperforms TinyBERT under compression ratio while the training speed is accelerated by an order of magnitude.

The rest of this paper is organized as follows. First, we summarize the related works of model compression and their applications to BERT in Section 2. Then, the methodology of LadaBERT is introduced in Section 3, and experimental results are presented in Section 4. At last, we conclude this work and discuss future works in Section 5.

2 Related Work

Deep Neural Networks (DNNs) have achieved great success in many areas in recent years, but the memory consumption and computational cost expand greatly with the growing complexity of models. Thus, model compression has become an indispensable technique in practice, especially for low-resource scenarios. Here we review the current progresses of model compression techniques briefly, and present their application to pre-trained BERT models.

2.1 Model compression algorithms

Existing model compression algorithms can be divided into four categories, namely weight pruning, matrix factorization, weight quantization and knowledge distillation.

Numerous researches have shown that removing a large portion of connections or neurons does not cause significant performance drop in deep neural networks. For example, Han et al. (2015) proposed a method to reduce the storage and computation of neural networks by removing unimportant connections, resulting in sparse networks without affecting the model accuracy. Li et al. (2016) presented an acceleration method for convolution neural network by pruning whole filters together with their connecting filter maps. This approach does not generate sparse connectivity patterns and brings a much larger acceleration ratio with existing BLAS libraries for dense matrix multiplications.

Matrix factorization was also widely studied in the deep learning domain, the goal of which is to decompose a matrix into the product of two matrices in lower dimensions. Sainath et al (2013) explored a low-rank matrix factorization method of DNN layers for acoustic modeling. Xu et al. (2013; 2014) applied singular value decomposition to deep neural network acoustic models and achieved comparable performances with state-of-the-art models through much fewer parameters. GroupReduce (Chen et al., 2018) focused on the compression of neural language models and applied low-rank matrix approximation to vocabulary-partition. Winata et al. (2019) carried out experiments for low-rank matrix factorization on different NLP tasks and demonstrated that it was more effective in general than weight pruning.

Weight quantization is another common technique for compressing deep neural networks, which aims to reduce the number of bits to represent every weight in the model. With weight quantization, the weights can be reduced to at most 1-bit binary value from 32-bits floating point numbers. Zhou et al. (2016) showed that quantizing weights to 8-bits does not hurt the performance; Binarized Neural Networks (Hubara et al., 2016) contained binary weights and activations of only one bit; and Incremental Network Quantization (Zhou et al., 2017) converted a pre-trained full-precision neural network into low-precision counterpart through three interdependent operations: weight partition, groupwise quantization and re-training.

Knowledge distillation (Hinton et al., 2015) trains a compact and smaller model to approximate the

Figure 2: Overview of LadaBERT framework

function learned by a large and complex model. A preliminary step of knowledge distillation is to train a deep network (the teacher model) that automatically generates soft labels for training instances. This “synthetic” label is then used to train a smaller network (the student model), which assimilates the function learned by the teacher model. Chen et al. (2017) successfully applied knowledge distillation to object detection tasks by introducing several modifications, including a weighted cross-entropy loss, a teacher bounded loss, and adaptation layers to model intermediate teacher distributions. Li et al. (2017) developed a framework to learn from noisy labels, where the knowledge learned from a clean dataset and semantic knowledge graph were leveraged to correct the wrong labels.

To improve the performance of model compression, there are also numerous attempts to develop hybrid model compression methods that combine more than one category of algorithms. Han et al. (2016) combined quantization, hamming coding and weight pruning to conduct model compression on image classification tasks. Yu et al. (2017) proposed a unified framework for low-rank and sparse decomposition of weight matrices with feature map reconstructions. Polino et al. (2018) advocated a combination of distillation and quantization techniques and proposed two hybrid models, i.e., quantified distillation and differentiable quantization. Li et al., (2018) compressed DNN-based acoustic models through knowledge distillation and pruning.

2.2 BERT model compression

In the natural language processing community, there is a growing interest recently to study BERToriented model compression for shipping its performance gain into latency-critical or low-resource scenarios. Most existing works focus on knowledge distillation. For instance, BERT-PKD (Sun et al., 2019) is a patient knowledge distillation approach that compresses the original BERT model into a lightweight shallow network. Different from traditional knowledge distillation methods, BERT-PKD enables an exploitation of rich information in the teacher’s hidden layers by utilizing a layer-wise distillation constraint. DistillBERT (Sanh et al., 2019) pre-trains a smaller general-purpose language model on the same corpus as vanilla BERT. Distilled BiLSTM (Tang et al., 2019) adopts a single-layer BiLSTM as the student model and achieves comparable results with ELMo (Peters et al., 2018) through much fewer parameters and less inference time. TinyBERT (Jiao et al., 2019) exploits a novel attention-based distillation schema that encourages the linguistic knowledge in teacher to be well transferred into the student model. It adopts a two-stage learning framework, including general distillation (pre-training from scratch via distillation loss) and task-specific distillation with data augmentation. Both procedures require huge resources and long training times (from several days to weeks), which is cumbersome for industrial applications. Therefore, we are aiming to explore more lightweight solutions in this paper.

3 Lightweight Adaptation of BERT

3.1 Overview

The overall pipeline of LadaBERT (Lightweight Adaptation of BERT) is illustrated in Figure 2. As shown in the figure, the pre-trained BERT model (e.g., BERT-Base) is served as the teacher as well as the initial status of the student model. Then, the student model is compressed towards smaller parameter size iteratively through a hybrid model compression approach until the target size is reached. Concretely, in each iteration, the parameter size of student model is first reduced by based on weight pruning and matrix factorization, and then the parameters are fine-tuned by the loss function of knowledge distillation. The motivation behind is that matrix factorization and weight pruning are complementary to each other. Matrix factorization calculates the optimal approximation under a certain rank, while weight pruning introduces additional sparsity to the decomposed matrices. Moreover, both weight pruning and matrix factorization generate better initial and intermediate status of the student model, which improve the efficiency and effectiveness of knowledge distillation. In the following subsections, we will introduce the algorithms in detail.

3.2 Matrix factorization

We use Singular Value Decomposition (SVD) for matrix factorization. All parameter matrices, including the embedding ones, are compressed by SVD. Without loss of generality, we assume a matrix of parameters , the singular value decomposition of which can be written as:

where is a diagonal matrix composed of singular values and p is the full rank of W satisfying

To compress this weight matrix, we select a lower rank r < p. The diagonal matrix is truncated by selecting the top r singular values. i.e., are also truncated by selecting the top r columns and rows respectively, resulting in

Then, low-rank matrix approximation of W can be formulated as:

In this way, the original weight matrix W is decomposed to two smaller matrices, where A = . These two matrices are initialized by SVD and will be further fine-tuned during training.

Given a rank , the compression ratio of matrix factorization is defined as:

Therefore, for a target model compression ratio , the desired rank r can be calculated by:

3.3 Weight pruning

Weight pruning (Han et al., 2015) is an unstructured compression method that induces desirable sparsity for a neural network model. For a neural network with parameters , weight pruning finds a binary mask subject to a given sparsity ratio, . The neural network after pruning will be , where the non-zero parameter size is is the number of parameters in . For example, when , there are 70% zeros and 30% ones in the mask m. In our implementation, we adopt a simple pruning strategy (Frankle and Carbin, 2018): the binary mask is generated by setting the smallest weights to zeros.

To combine the benefits of weight pruning and matrix factorization, we leverage a hybrid approach that applies weight pruning on the basis of decomposed matrices generated by SVD. Following Equation (2), SVD-based matrix factorization for any weight matrix W can be written as: Then, weight pruning is applied on the decomposed matrices separately. The weight matrix after hybrid compression is formulated as:

where are binary masks derived by the weight pruning algorithm with compression ratio . The compression ratio after hybrid compression can be calculated by:

In LadaBERT, the hybrid compression produce is applied to each layer of the pre-trained BERT model. Given an overall model compression target P, the following constraint should be satisfied:

where is the total number of model parameters and P is the target compression ratio; the parameter number of embedding layer, which has a relative compression ratio of denotes the number of parameters of all layers in BERT encoder, which have a compression ratio of . The classification layers (MLP layers with Softmax activation) have a relative small number of parameters (), so they are not modified in model compression. In general, we have three flexible hyper-parameters for fine-grained compression: , which can be optimized by random search on the validation data.

3.4 Knowledge distillation

Knowledge distillation (KD) has been widely used to transfer knowledge from a large teacher model to a smaller student model. In other words, the student model mimics the behavior of the teacher model by minimizing the knowledge distillation loss functions. Various types of knowledge distillation can be employed at different sub-layers. Generally, all types of knowledge distillation can be modeled as minimizing the following loss function:

Where X denotes the training set and x is a sample input in the set. intermediate outputs or weight matrices for the student model and teacher model respectively. resents for a loss function which can be carefully designed for different types of knowledge distillation. We partly follow the recent technique proposed by TinyBERT (Jiao et al., 2019), which applies knowledge distillation constraints upon embedding, self-attention, hidden representation and prediction levels. Concretely, there are four types of knowledge distillation constraints as follows:

• Embedding-layer distillation is performed upon the embedding layer. for the word embedding output for input x, where n is the input word length and d is the dimension of word embedding. Mean Squared Error (MSE) is adopted as the loss function

• Attention-layer distillation is performed upon the self-attention sub-layer. represents the attention output for each self-attention sub-layer, and denotes MSE loss function.

• Hidden-layer distillation is performed at each fully-connected sub-layer in the Transformer architectures. f(x) denotes the output representation of the corresponding sub-layer, and adopts MSE loss function.

• Prediction-layer distillation makes the student model to learns the predictions from a teacher model directly. It is identical to a vanilla form of knowledge distillation (Hinton et al., 2015). It takes soft cross-entropy loss function, which can be formulated as:

where denotes Softmax function, are the predictive logits of teacher and student models respectively. t is a temperature value, which generally works well at t = 1 (Jiao et al., 2019).

4 Experiments

4.1 Datasets & Baselines

We compare LadaBERT with state-of-the-art model compression approaches on five public datasets of different tasks of natural language understanding, including sentiment classification (SST-2), natural language inference (MNLI-m, MNLI-mm, QNLI) and pairwise semantic equivalence (QQP). The statistics of these datasets are described in Table 1.

Table 1: Dataset Statistics

The baseline approaches are summarized below.

• Weight pruning and Matrix factorization are two simple baselines described in Section 3.3. We evaluate both pruning methods in an iterative manner until the target compression ratio is reached.

• Hybrid pruning is a combination of matrix factorization and weight pruning, which conducts iterative weight pruning on the basis of SVD-based matrix factorization. It is performed iteratively until the desired compression ratio is achieved.

• BERT-FT, BERT-KD and BERT-PKD are reported in (Sun et al., 2019), where BERT-FT directly fine-tunes the model via supervision labels, BERT-KD is the vanilla knowledge distillation algorithm (Hinton et al., 2015), and BERT-PKD stands for Patient Knowledge Distillation proposed in (Sun et al., 2019). The student model is composed of 3 Transformer layers, resulting in a compression ratio. Each layer has the same hidden size as the pre-trained teacher, so the initial parameters of student model can be inherited from the corresponding teacher.

• TinyBERT (Jiao et al., 2019) instantiates a tiny student model, which has totally 14.5M parameters (compression ratio) composed of 4 layers, 312 hidden units, 1200 intermediate size and 12 heads. For a fair comparison, we reproduce the TinyBERT pipeline1 without general distillation and data augmentation, which is time-exhaustive and resource-consuming.

• BERT-Small has the same model architecture as TinyBERT, but is directly pre-trained by the offi-cial BERT pipeline. The performance values are copied from (Jiao et al., 2019) for reference.

• Distilled-BiLSTM (Tang et al., 2019) leverages a single-layer bidirectional-LSTM as the student model, where the hidden units and intermediate size are set to be 300 and 400 respectively, resulting in a compression ratio. This model requires an expensive training process similar to vanilla BERT.

4.2 Setup

We leverage the pre-trained checkpoint of base-bert-uncased2 as the initial model for compression, which contains 12 layers, 12 heads, 110M parameters, and 768 hidden units per layer. Hyper-parameter selection is conducted on the validation data for each dataset. After training, the prediction results are submitted to the GLUE-benchmark evaluation platform3 to get the evaluation performance on test data.

For a comprehensive evaluation, we experiment with four settings of LadaBERT, namely LadaBERT-1, -2, -3 and -4, which reduce the model parameters of BERT-Base by 2.5, 5.0, 7.5 and 10.0 times respectively. In our experiment, we set the batch size as 32 and learning rate as 2e-5. The optimizer is BertAdam with the default setting (Devlin et al., 2018). Fine-grained compression ratios are optimized by random search on SST dataset and transferred to other datasets (shown in Table 2). Following (Jiao et al., 2019), the temperature value in distillation loss function is set as 1 in all experiments without tuning.

Table 2: Fine-grained compression ratios

4.3 Performance Comparison

Table 3: Performance comparison on various model sizes

The evaluation results of LadaBERT and state-of-the-art approaches are listed in Table 3, where the models are ranked by parameter sizes for feasible comparison. As shown in the table, LadaBERT consistently outperforms the strongest baselines under similar model sizes. In addition, the performance of LadaBERT demonstrates the superiority of a combination of SVD-based matrix factorization, weight pruning and knowledge distillation.

Figure 3: Learning curve on MNLI-m dataset.

Figure 4: Learning curve on QQP dataset.

With model size of reduction, LadaBERT-1 performs significantly better than BERT-PKD, boosting the performance by relative 8.9, 8.1, 6.1, 3.8 and 5.8 percentages on MNLI-m, MNLI-mm, SST-2, QQP and QNLI datasets respectively. Recall that BERT-PKD initializes the student model by selecting 3 of 12 layers in the pre-trained BERT-Base model. It turns out that the discarded layers have a huge impact on the model performance, which is hard to be recovered by knowledge distillation. On the other hand, LadaBERT generates the student model by iterative pruning on the pre-trained teacher. In this way, the original knowledge in the teacher model can be preserved to the largest extent, and the benefit is complementary to knowledge distillation.

LadaBERT-3 has a comparable size as TinyBERT with a compression ratio. As shown in the results, TinyBERT does not work well without expensive data augmentation and general distillation, hindering its application to low-resource settings. The reason is that the student model of TinyBERT is distilled from scratch, so it requires much more data to mimic the teacher’s behaviors. Instead, LadaBERT has better initial and intermediate status calculated by hybrid model compression, which is much more light-weighted and achieves competitive performances with much faster learning speed (learning curve comparison is shown in Section 4.4). Moreover, LadaBERT-3 outperforms BERT-Small on most of the datasets, which is pre-trained from scratch by the official BERT pipeline. This means that LadaBERT can quickly adapt to smaller model sizes and achieve competitive performance without expansive re-training on a large corpus.

Moreover, Distilled-BiLSTM performs well on SST-2 dataset with more than compression ratio, owing to good generalization ability of LSTM model on small datasets. Nevertheless, the performance of LadaBERT-4 is competitive on larger datasets such as MNLI and QQP. This is impressive as LadaBERT is much more efficient without exhaustive re-training on a large corpus. In addition, the inference speed of BiLSTM is slower than transformer-based models with similar parameter sizes.

4.4 Learning curve comparison

To further demonstrate the efficiency of LadaBERT, we visualize the learning curves on MNLI-m and QQP datasets in Figure 3 and 4, where LadaBERT-3 is compared to the strongest baseline, TinyBERT, under compression ratio. As shown in the figures, LadaBERT-3 achieves good performances much faster and results in a better convergence point. After training steps (batches) on MNLI-m dataset, the performance of LadaBERT-3 is already comparable to TinyBERT after convergence (approximately steps), achieving nearly 10 times acceleration. And on QQP dataset, both performance improvement and training speed acceleration are very significant. This clearly shows the superiority of combining matrix factorization, weight pruning and knowledge distillation in a collaborative manner. On the other hand, TinyBERT is based on pure knowledge distillation, so the learning speed is much slower.

4.5 Effect of low-rank + sparsity

In this paper, we demonstrate that a combination of matrix factorization and weight pruning is better than single solutions for BERT-oriented model compression. Similar phenomena has been reported in computer vision, showing that low-rank and sparsity are complementary to each other (Yu et al., 2017). Here we provide another explanation to support our observation.

element biases for a weight matrix in the neural network after pruning to 20% of its original parameter size. For illustration, we consider the matrix initialized by real pretrained BERT weights, and the pruning process is done at once. We define the biases to be calculated by denotes the

tion of biases generated by pure weight pruning, which has a sudden drop at the pruning threshold. The orange line represents for pure SVD pruning, which turns out to be smoother and is aligned with Gaussian distribution. The blue

line shows the result of hybrid pruning, which conducts weight pruning on the decomposed matrices. First, we apply SVD-based matrix factorization to reduce 60% of total parameters. Then, weight pruning is applied on the decomposed matrices by 50%, resulting in 20% parameters while the bias distribution changes slightly. As visualized in Figure 5, it has smaller mean and deviation of bias distribution than that of pure matrix factorization. In addition, it seems that a smoother weight distribution is more feasible for the fine-tuning procedure. Therefore, it is reasonable that a hybrid model compression approach is advantageous than pure weight pruning.

5 Conclusion

Model compression is a common way to deal with latency-critical or memory-intensive scenarios. Existing model compression methods for BERT are expansive as they require re-training on a large corpus to reserve the original performance. In this paper, we propose LadaBERT, a lightweight model compression pipeline that generates an adaptive BERT model efficiently based on a given task and specific constraint. It is based on a hybrid solution, which conducts matrix factorization, weight pruning and knowledge distillation in a collaborative fashion. The experimental results demonstrate that LadaBERT is able to achieve comparable performance with other state-of-the-art solutions using much less training data and computation budget. Therefore, LadaBERT can be easily plugged into various applications to achieve competitive performances with little training overheads. In the future, we would like to apply LadaBERT to large-scale industrial applications, such as search relevance and query recommendation.

References

Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. 2017. Learning efficient object detection models with knowledge distillation. In Advances in Neural Information Processing Systems, pages 742–751.

Patrick Chen, Si Si, Yang Li, Ciprian Chelba, and Cho-Jui Hsieh. 2018. Groupreduce: Block-wise low-rank approximation for neural language model shrinking. In Advances in Neural Information Processing Systems, pages 10988–10998.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.04805.

Jonathan Frankle and Michael Carbin. 2018. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635.

Song Han, Jeff Pool, John Tran, and William J Dally. 2015. Learning both weights and connections for efficient neural networks. pages 1135–1143.

Song Han, Huizi Mao, and William J Dally. 2016. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding.

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.

Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. 2016. Binarized neural networks. In Advances in neural information processing systems, pages 4107–4115.

Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351.

Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. 2016. Pruning filters for efficient convnets. arXiv: Computer Vision and Pattern Recognition.

Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. 2017. Learning from noisy labels with distillation. In Proceedings of the IEEE International Conference on Computer Vision, pages 1910– 1918.

Chenxing Li, Lei Zhu, Shuang Xu, Peng Gao, and Bo Xu. 2018. Compression of acoustic model via knowledge distillation and pruning. In 2018 24th International Conference on Pattern Recognition (ICPR), pages 2785– 2790. IEEE.

Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettle- moyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365.

Antonio Polino, Razvan Pascanu, and Dan Alistarh. 2018. Model compression via distillation and quantization. arXiv preprint arXiv:1802.05668.

Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. 2013. Low-rank matrix factorization for deep neural network training with high-dimensional output targets. In 2013 IEEE international conference on acoustics, speech and signal processing, pages 6655–6659. IEEE.

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.

Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for bert model compression. arXiv preprint arXiv:1908.09355.

Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019. Distilling task-specific knowledge from bert into simple neural networks. arXiv preprint arXiv:1903.12136.

Genta Indra Winata, Andrea Madotto, Jamin Shin, Elham J Barezi, and Pascale Fung. 2019. On the effectiveness of low-rank matrix factorization for lstm model compression. arXiv preprint arXiv:1908.09982.

Jian Xue, Jinyu Li, and Yifan Gong. 2013. Restructuring of deep neural network acoustic models with singular value decomposition. In Interspeech, pages 2365–2369.

Jian Xue, Jinyu Li, Dong Yu, Mike Seltzer, and Yifan Gong. 2014. Singular value decomposition based low- footprint speaker adaptation and personalization for deep neural network. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6359–6363. IEEE.

Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. 2017. On compressing deep models by low rank and sparse decomposition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7370–7379.

Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. 2016. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160.

Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. 2017. Incremental network quantization: Towards lossless cnns with low-precision weights. arXiv preprint arXiv:1702.03044.

designed for accessibility and to further open science