Not all computations in a deep neural network are of equal importance. In a typical deep learning pipeline, an expert crafts a neural architecture, which is trained using a prepared dataset. The success of training a deep model often requires trial and error, and such loop usually has little control on prioritizing the computations happening in the neural network. Recently researchers started to develop model-simplification methods for convolutional neural networks (CNNs), bearing in mind that some computations are indeed non-critical or redundant and hence can be safely removed from a trained model without substantially degrading the model’s performance. Such methods not only accelerate computational efficiency but also possibly alleviate the model’s overfitting effects.

Discovering which subsets of the computations of a trained CNN are more reasonable to prune, however, is nontrivial. Existing methods can be categorized from either the learning perspective or from the computational perspective. From the learning perspective, some methods use a dataindependent approach where the training data does not assist in determining which part of a trained CNN should be pruned, e.g. He et al. (2017) and Zhang et al. (2016), while others use a data-dependent approach through typically a joint optimization in generating pruning decisions, e.g., Han et al. (2015) and Anwar et al. (2017). From the computational perspective, while most approaches focus on setting the dense weights of convolutions or linear maps to be structured sparse, we propose here a method adopting a new conception to achieve in effect the same goal.

Instead of regarding the computations of a CNN as a collection of separate computations sitting at different layers, we view it as a network flow that delivers information from the input to the output through different channels across different layers. We believe saving computations of a CNN is not only about reducing what are calculated in an individual layer, but perhaps more importantly also about understanding how each channel is contributing to the entire information flow in the underlying passing graph as well as removing channels that are less responsible to such process. Inspired by this new conception, we propose to design a “gate” at each channel of a CNN, controlling whether its received information is actually sent out to other channels after processing. If a channel “gate” closes, its output will always be a constant. In fact, each designed “gate” will have a prior intention to close, unless it has a “strong” duty in sending some of its received information from the input to subsequent layers. We find that implementing this idea in pruning CNNs is unsophisticated, as will be detailed in Sec 4.

Our method neither introduces any extra parameters to the existing CNN, nor changes its computation graph. In fact, it only introduces marginal overheads to existing gradient training of CNNs. It also possess an attractive feature that one can successively build multiple compact models with different inference performances in a single round of resource-intensive training (as in our experiments). This eases the process to choose a balanced model to deploy in production. Probably, the only applicability constraint of our method is that all convolutional layers and fully-connected layer (except the last layer) in the CNN should be batch normalized (Ioffe & Szegedy, 2015). Given batch normalization has becomes a widely adopted ingredient in designing state-of-the-art deep learning models, and many successful CNN models are using it, we believe our approach has a wide scope of potential impacts.1

In this paper, we start from rethinking a basic assumption widely explored in existing channel pruning work. We point out several issues and gaps in realizing this assumption successfully. Then, we propose our alternative approach, which works around several numerical difficulties. Finally, we experiment our method across different benchmarks and validate its usefulness and strengths.

Reducing the size of neural network for speeding up its computational performance at inference time has been a long-studied topic in the communities of neural network and deep learning. Pioneer works include Optimal Brain Damage (LeCun et al., 1990) and Optimal Brain Surgeon (Hassibi & Stork, 1993). More recent developments focused on either reducing the structural complexity of a provided network or training a compact or simplified network from scratch. Our work can be categorized into the former, thus the literature review below revolves around reducing the structural complexity.

To reduce the structural complexity of deep learning models, previous work have largely focused on sparsifying the weights of convolutional kernels or the feature maps across multiple layers in a network (Anwar et al., 2017; Han et al., 2015). Some recent efforts proposed to impose structured sparsity on those vector components motivated from the implementation perspective on specialized hardware (Wen et al., 2016; Zhou et al., 2016; Alvarez & Salzmann, 2016; Lebedev & Lempitsky, 2016). Yet as argued by Molchanov et al. (2017), regularization-based pruning techniques require per layer sensitivity analysis which adds extra computations. Their method relies on global rescaling of criteria for all layers and does not require sensitivity estimation, a beneficial feature that our approach also has. To our knowledge, it is also unclear how widely useful those works are in deep learning. In Section 3, we discuss in details the potential issues in regularization-based pruning techniques potentially hurting them being widely applicable, especially for those that regularize high-dimensional tensor parameters or use magnitude-based pruning methods. Our approach works around the mentioned issues by constraining the anticipated pruning operations only to batch-normalized convolutional layers. Instead of posing structured sparsity on kernels or feature maps, we enforce sparsity on the scaling parameter in batch normalization operator. This blocks the sample-wise information passing through part of the channels in convolution layer, and in effect implies one can safely remove those channels.

A recent work by Huang & Wang (2017) used a similar technique as ours to remove unimportant residual modules in ResNet by introducing extra scaling factors to the original network. However, some optimization subtleties as to be pointed out in our paper were not well explained. Another recent work called Network-Slimming (Liu et al., 2017) also aims to sparsify the scaling parameters of batch normalization. But instead of using off-the-shelf gradient learning like theirs, we propose a new algorithmic approach based on ISTA and rescaling trick, improving robustness and speed of the undergoing optimization. In particular, the work of Liu et al. (2017) was able to prune VGG-A model on ImageNet. It is unclear how their work would deal with the rescaling effect and whether their approach can be adopted to large pre-trained models, such as ResNets and Inceptions. We experimented with the pre-trained ResNet-101 and compared to most recent work that were shown to work well with large CNNs. We also experimented with an image segmentation model which has an inception-like module (pre-trained on ImageNet) to locate foreground objects.

In most regularized linear regressions, a large-norm coefficient is often a strong indicator of a highly informative feature. This has been widely perceived in statistics and machine learning communities. Removing features which have a small coefficient does not substantially affect the regression errors. Therefore, it has been an established practice to use tractable norm to regularize the parameters in optimizing a model and pick the important ones by comparing their norms after training. However, this assumption is not unconditional. By using Lasso or ridge regression to select important predictors in linear models, one always has to first normalize each predictor variable. Otherwise, the result might not be explanatory. For example, ridge regression penalizes more the predictors which has low variance, and Lasso regression enforces sparsity of coefficients which are already small in OLS. Such normalization condition for the right use of regularization is often unsatisfied for nonconvex learning. For example, one has to carefully consider two issues outlined below. We provides these two cases to exemplify how regularization could fail or be of limited usage. There definitely exist ways to avoid the described failures.

Model Reparameterization. In the first case, we show that it is not easy to have fine-grained control of the weights’ norms across different layers. One has to either choose a uniform penalty in all layers or struggle with the reparameterization patterns. Consider to find a deep linear (convolutional) network subject to a least square with Lasso: for ,

The above formulation is not a well-defined problem because for any parameter set , one can always find another parameter set such that it achieves a smaller total loss while keeping the corresponding norm unchanged by actually setting

where . In another word, for any , one can always find a parameter set (which is usually non-sparse) that minimizes the first least square loss while having its second Lasso term less than .

We note that gradient-based learning is highly inefficient in exploring such model reparameterization patterns. In fact, there are some recent discussions around this (Dinh et al., 2017). If one adopts a pre-trained model, and augments its original objective with a new norm-based parameter regularization, the new gradient updates may just increase rapidly or it may take a very long time for the variables traveling along the model’s reparameterization trajectory. This highlights a theoretical gap questioning existing sparsity-inducing formulation and actual computational algorithms whether they can achieve widely satisfactory parameter sparsification for deep learning models.

Transform Invariance. In the second case, we show that batch normalization is not compatible with weight regularization. The example is penalizing - or -norms of filters in convolution layer which is then followed by a batch normalization: at the l-th layer, we let

where and are vectors whose length is the number of channels. Likewise, one can clearly see that any uniform scaling of which changes its - and -norms would have no effects on the output . Alternatively speaking, if one is interested in minimizing the weight norms of multiple layers together, it becomes unclear how to choose proper penalty for each layer. Theoretically, there always exists an optimizer that can change the weight to one with infinitesimal magnitude without hurting any inference performance. As pointed by one of the reviewers, one can tentatively avoid this issue by projecting the weights to the surface of unit ball. Then one has to deal with a non-convex feasible set of parameters, causing extra difficulties in developing optimization for data-dependent pruning methods. It is also worth noting that some existing work used such strategy in a layer-by-layer greedy way (He et al., 2017; Zhang et al., 2016).

Based on this discussion, many existing works which claim to use Lasso, group Lasso (e.g. Wen et al. (2016); Anwar et al. (2017)), or thresholding (e.g. Molchanov et al. (2017)) to enforce parameter sparsity have some theoretical gaps to bridge. In fact, many heuristic algorithms in neural net pruning actually do not naturally generate a sparse parameterized solution. More often, thresholding is used to directly set certain subset of the parameters in the network to zeros, which can be problematic. The reason is in essence around two questions. First, by setting parameters less than a threshold to zeros, will the functionality of neural net be preserved approximately with certain guarantees? If yes, then under what conditions? Second, how should one set those thresholds for weights across different layers? Not every layer contributes equally in a neural net. It is expected that some layers act critically for the performance but only use a small computation and memory budget, while some other layers help marginally for the performance but consume a lot resources. It is naturally more desirable to prune calculations in the latter kind of layers than the former.

In contrast with these existing approaches, we focus on enforcing sparsity of a tiny set of parameters in CNN — scale parameter s in all batch normalization. Not only placing sparse constraints on is simpler and easier to monitor, but more importantly, we have two strong reasons:

1. Every always multiplies a normalized random variable, thus the channel importance becomes comparable across different layers by measuring the magnitude values of ;

2. The reparameterization effect across different layers is avoided if its subsequent convolution layer is also batch-normalized. In other words, the impacts from the scale changes of parameter are independent across different layers.

Nevertheless, our current work still falls short of a strong theoretical guarantee. We believe by working with normalized feature inputs and their regularized coefficients together, one is closer to a more robust and meaningful approach. Sparsity is not the goal, but to find less important channels using sparsity inducing formulation is.

We describe the basic principle and algorithm of our channel pruning technique.

4.1 PRELIMINARIES

Pruning constant channels. Consider convolution with batch normalization:

For the ease of notation, we let . Note that if some element in is set to zero, say, , its output image becomes a constant , and a convolution of a constant image channel is almost everywhere constant (except for padding regions, an issue to be discussed later). Therefore, we show those constant image channels can be pruned while the same functionality of network is approximately kept:

where denotes the convolution operator which is only calculated along channels indexed by non-zeros of . Remark that sum reducedif .

Remark that the approximation () is strictly equivalence (=) if no padding is used in the convolution operator , a feature that the parallel work Liu et al. (2017) does not possess. When the original model uses padding in computing convolution layers, the network function is not strictly preserved after pruning. In our practice, we fine-tune the pruned network to fix such performance degradation at last. In short, we formulate the network pruning problem as simple as to set more elements in to zero. It is also much easier to deploy the pruned model, because no extra parameters or layers are introduced into the original model.

To better understand how it works in an entire CNN, imagine a channel-to-channel computation graph formed by the connections between layers. In this graph, each channel is a node, their inference dependencies are represented by directed edges. The parameter serves as a “dam” at each node, deciding whether let the received information “flood” through to other nodes following the graph. An end-to-end training of channel pruning is essentially like a flood control system. There suppose to be rich information of the input distribution, and in two ways, much of the original input information is lost along the way of CNN inference, and the useful part — that is supposed to be preserved by the network inference — should be label sensitive. Conventional CNN has one way to reduce information: transforming feature maps (non-invertible) via forward propagation. Our approach introduces the other way: block information at each channel by forcing its output being constant using ISTA.

ISTA. Despite the gap between Lasso and sparsity in the non-convex settings, we found that ISTA (Beck & Teboulle, 2009) is still a useful sparse promoting method. But we just need to use it more carefully. Specifically, we adopt ISTA in the updates of s. The basic idea is to project the parameter at every step of gradient descent to a potentially more sparse one subject to a proxy problem: let l denote the training loss of interest, at the (t + 1)-th step, we set

where is the derivative with respect to computed at step is the learning rate, is the penalty. In the stochastic learning, is estimated from a mini-batch at each step. Eq. (1) has closed form solution as

where proxsgn(x). The ISTA method essentially serves as a “flood control system” in our end-to-end learning, where the functionality of each is like that of a dam. When is zero, the information flood is totally blocked, while , the same amount of information is passed through in form of geometric quantities whose magnitudes are proportional to .

Scaling effect. One can also see that if is scaled by meanwhile is scaled by , that is,

the output is unchanged for the same input . Despite not changing the output, scaling of and also scales the gradients and by and , respectively. As we observed, the parameter dynamics of gradient learning with ISTA depends on the scaling factor if one decides to choose it other than 1.0. Intuitively, if is large, the optimization of is progressed much slower than that of .

4.2 THE ALGORITHM

We describe our algorithm below. The following method applies to both training from scratch or re-training from a pre-trained model. Given a training loss l, a convolutional neural net N, and hyper-parameters , our method proceeds as follows:

1. Computation of sparse penalty for each layer. Compute the memory cost per channel for each layer denoted by and set the ISTA penalty for layer l to . Here

• is the size of input image of the neural network. • is the kernel size of the convolution at layer l. Likewise, is the kernelsize of subsequent convolution at layer .

represents the set of the subsequent convolutional layers of layer l • denotes the channel size of the previous layer, which the l-th convolution operatesover; and denotes the channel size of one subsequent layer .

• is the image size of the feature map at layer l. 2. rescaling trick. For layers whose channels are going to get reduced, scale all s in batch normalizations by meanwhile scale weights in their subsequent convolutions by .

3. End-to-End training with ISTA on . Train N by the regular SGD, with the exception that s are updated by ISTA, where the initial learning rate is . Train N until the loss l plateaus, the total sparsity of s converges, and Lasso converges.

4. Post-process to remove constant channels. Prune channels in layer l whose elements in are zero and output the pruned model by absorbing all constant channels into subsequent layers (as described in the earlier section.).

5. rescaling trick. For s and weights in which were scaled in Step 2 before training,scale them by and respectively (scaling back).

6. Fine-tune using regular stochastic gradient learning.

Remark that choosing a proper as used in Steps 2 and 5 is necessary for using a large in ISTA, which makes the sparsification progress of s faster.

4.3 GUIDELINES FOR TUNING HYPER-PARAMETERS

We summarize the sensitivity of hyper-parameters and their impacts for optimization below:

• (learning rate): larger leads to fewer iterations for convergence and faster progress of sparsity. But if if too large, the SGD approach wouldn’t converge.

• (sparse penalty): larger leads to more sparse model at convergence. If trained with a very large , all channels will be eventually pruned.

• (rescaling): we use other than 1. only for pretrained models, we typically choose from {0.001, 0.01, 0.1, 1} and smaller warms up the progress of sparsity.

We recommend the following parameter tuning strategy. First, check the cross-entropy loss and the regularization loss, select such that these two quantities are comparable at the beginning. Second, choose a reasonable learning rate. Third, if the model is pretrained, check the average magnitude of s in the network, choose such that the magnitude of rescaled is around . We found as long as one choose those parameters in the right range of magnitudes, the optimization progress is enough robust. Again one can monitor the mentioned three quantities during the training and terminate the iterations when all three quantities plateaus.

There are several patterns we found during experiments that may suggest the parameter tuning has not been successful. If during the first few epochs the Lasso-based regularization loss keeps decreasing linearly while the sparsity of s stays near zero, one may decrease and restart. If during the first few epochs the sparsity of s quickly raise up to 100%, one may decrease and restart. If during the first few epochs the cross-entropy loss keeps at or increases dramatically to a non-informative level, one may decrease or and restart.

5.1 CIFAR-10 EXPERIMENT

We experiment with the standard image classification benchmark CIFAR-10 with two different network architectures: ConvNet and ResNet-20 (He et al., 2016). We resize images to and zero-pad them to . We pre-process the padded images by randomly cropping with size , randomly flipping, randomly adjusting brightness and contrast, and standardizing them such that their pixel values have zero mean and one variance.

ConvNet For reducing the channels in ConvNet, we are interested in studying whether one can easily convert a over-parameterized network into a compact one. We start with a standard 4-layer convolutional neural network whose network attributes are specified in Table 1. We use a fixed learning rate , scaling parameter , and set batch size to 125.

Model A is trained from scratch using the base model with an initial warm-up for 30k steps, and then is trained by raising up to 0.001. After the termination criterion are met, we prune the channels of the base model to generate a smaller network called model A. We evaluate the classification performance of model A with the running exponential average of its parameters. It is found that the test accuracy of model A is even better than the base model. Next, we start from the pre-trained model A to create model B by raising up to 0.002. We end up with a smaller network called model B, which is about 1% worse than model A, but saves about one third parameters. Likewise, we start from the pre-trained model B to create model C. The detailed statistics and its pruned channel size are reported in Table 1. We also train a reference ConvNet from scratch whose channel sizes are 32-64-64-128 with totally 224,008 parameters and test accuracy being 86.3%. The referenced model is not as good as Model B, which has smaller number of parameters and higher accuracy.

We have two major observations from the experiment: (1) When the base network is over-parameterized, our approach not only significantly reduces the number of channels of the base model but also improves its generalization performance on the test set. (2) Performance degradation seems unavoidable when the channels in a network are saturated, and our approach gives satisfactory tradeoff between test accuracy and model efficiency.

ResNet-20 We also want to verify our second observation with the state-of-art models. We choose the popular ResNet-20 as our base model for the CIFAR-10 benchmark, whose test accuracy is 92%. We focus on pruning the channels in the residual modules in ResNet-20, which has 9 convolutions in total. As detailed in Table 2, model A is trained from scratch using ResNet-20’s network structure as its base model. We use a warm-up for 30k steps and then train with . We are able to remove 37% parameters from ResNet-20 with only about 1 percent accuracy loss. Likewise, Model B is created from model A with a higher penalty .

Table 1: Comparisons between different pruned networks and the base network.

Table 2: Comparisons between ResNet-20 and its two pruned versions. The last columns are the number of channels of each residual modules after pruning.

5.2 ILSVRC2012 EXPERIMENT

We experiment our approach with the pre-trained ResNet-101 on ILSVRC2012 image classification dataset (He et al., 2016). ResNet-101 is one of the state-of-the-art network architecture in ImageNet Challenge. We follow the standard pipeline to pre-process images to for training ResNets. We adopt the pre-trained TensorFlow ResNet-101 model whose single crop error rate is 23.6% with about parameters.2 We set the scaling parameter , the initial learning rate , the sparsity penalty , and the batch size = 128 (across 4 GPUs). The learning rate is decayed every four epochs with rate 0.86. We create two pruned models from the different iterations of training ResNet-101: one has parameters and the other has parameters. We then fine-tune these two models using the standard way for training ResNet-101, and report their error rates. The Top-5 error rate increases of both models are less than 0.5%. The Top-1 error rates are summarized in Table 3. To our knowledge, only a few works have reported their performance on this very large-scale benchmark w.r.t. the Top-1 errors. We compare our approach with some recent works in terms of model parameter size, flops, and error rates. As shown in Table 3, our model v2 has achieved a compression ratio more than 2.5 while maintaining more than 1% lower error rates than that of other state-of-the-art models at comparable size of parameters.

In the first experiment (CIFAR-10), we train the network from scratch and allocate enough steps for both and W adjusting their own scales. Thus, initialization of an improper scale of is not really an issue given we optimize with enough steps. But for the pre-trained models which were originally optimized without any constraints of , the s scales are often unanticipated. It actually takes as many steps as that of training from scratch for to warm up. By adopting the rescaling trick setting to a smaller value, we are able to skip the warm-up stage and quick start to sparsify s. For example, it might take more than a hundred epoch to train ResNet-101, but it only takes about 5-10 epochs to complete the pruning and a few more epochs to fine-tune.

Table 3: Attributes of different versions of ResNet and their single crop errors on ILSVRC2012 benchmark. The last column means the parameter size of pruned model vs. the base model.

5.3 IMAGE FOREGROUND-BACKGROUND SEGMENTATION EXPERIMENT

As we have discussed about the two major observations in Section 5.1, a more appealing scenario is to apply our approach in pruning channels of over-parameterized model. It often happens when one adopts a pre-trained network on a large task (such as ImageNet classification) and fine-tunes the model to a different and smaller task (Molchanov et al., 2017). In this case, one might expect that some channels that have been useful in the first pre-training task are not quite contributing to the outputs of the second task.

We describe an image segmentation experiment whose neural network model is composed from an inception-like network branch and a densenet network branch. The entire network takes a image and outputs binary mask at the same size. The inception branch is mainly used for locating the foreground objects while the densenet network branch is used to refine the boundaries around the segmented objects. This model was originally trained on multiple datasets.

In our experiment, we attempt to prune channels in both the inception branch and densenet branch. We set , and batch size = 24. We train the pre-trained base model until all termination criterion are met, and build the pruned model for fine-tuning. The pruned model saves 86% parameters and 81% flops of the base model. We also compare the fine-tuned pruned model with the pre-trained base model across different test benchmark. Mean IOU is used as the evaluation metric.3 It shows that pruned model actually improves over the base model on four of the five test datasets with about , while it performs worse than the base model on the most challenged dataset DUT-Omron, whose foregrounds might contain multiple objects.

Table 4: mIOU reported on different test datasets for the base model and the pruned model.

We proposed a model pruning technique that focuses on simplifying the computation graph of a deep convolutional neural network. Our approach adopts ISTA to update the parameter in batch normalization operator embedded in each convolution. To accelerate the progress of model pruning, we use a rescaling trick before and after stochastic training. Our method cleverly avoids some possible numerical difficulties such as mentioned in other regularization-based related work, hence is easier to apply for practitioners. We empirically validated our method through several benchmarks and showed its usefulness and competitiveness in building compact CNN models.

Jose M Alvarez and Mathieu Salzmann. Learning the number of neurons in deep networks. In Advances in Neural Information Processing Systems, pp. 2270–2278, 2016.

Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks. ACM Journal on Emerging Technologies in Computing Systems (JETC), 13(3):32, 2017.

Amir Beck and Marc Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences, 2(1):183–202, 2009.

Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In Proceedings of International Conference on Machine Learning, 2017.

Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, pp. 1135–1143, 2015.

Babak Hassibi and David G Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in Neural Information Processing Systems, pp. 164–171, 1993.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778, 2016.

Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of International Conference on Computer Vision, 2017.

Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. arXiv preprint arXiv:1707.01213, 2017.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448–456, 2015.

Vadim Lebedev and Victor Lempitsky. Fast convnets using group-wise brain damage. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2554–2564, 2016.

Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in Neural Information Processing Systems, pp. 598–605, 1990.

Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations, 2017.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740–755. Springer, 2014.

Tie Liu, Zejian Yuan, Jian Sun, Jingdong Wang, Nanning Zheng, Xiaoou Tang, and Heung-Yeung Shum. Learning to detect a salient object. IEEE Transactions on Pattern analysis and machine intelligence, 33(2): 353–367, 2011.

Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. arXiv preprint arXiv:1708.06519, 2017.

Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural net- works for resource efficient transfer learning. In International Conference on Learning Representations, 2017.

Xiaoyong Shen, Aaron Hertzmann, Jiaya Jia, Sylvain Paris, Brian Price, Eli Shechtman, and Ian Sachs. Au- tomatic portrait segmentation for image stylization. In Computer Graphics Forum, volume 35, pp. 93–102. Wiley Online Library, 2016.

Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in Neural Information Processing Systems, pp. 2074–2082, 2016.

Chuan Yang, Lihe Zhang, Huchuan Lu, Xiang Ruan, and Ming-Hsuan Yang. Saliency detection via graph- based manifold ranking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3166–3173, 2013.

Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating very deep convolutional networks for classification and detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(10): 1943–1955, 2016.

Hao Zhou, Jose M Alvarez, and Fatih Porikli. Less is more: Towards compact cnns. In European Conference on Computer Vision, pp. 662–677. Springer, 2016.

Figure 1: Visualization of the number of pruned channels at each convolution in the inception branch. Colored regions represents the number of channels kept. The height of each bar represents the size of feature map, and the width of each bar represents the size of channels. It is observed that most of channels in the bottom layers are kept while most of channels in the top layers are pruned.