Structural Compression of Convolutional Neural Networks

2017·Arxiv

Abstract

Abstract

Deep convolutional neural networks (CNNs) have been successful in many tasks in machine vision, however, millions of weights in the form of thousands of convolutional filters in CNNs makes them difficult for human intepretation or understanding in science. In this article, we introduce CAR, a greedy structural compression scheme to obtain smaller and more interpretable CNNs, while achieving close to original accuracy. The compression is based on pruning filters with the least contribution to the classification accuracy. We demonstrate the interpretability of CAR-compressed CNNs by showing that our algorithm prunes filters with visually redundant functionalities such as color filters. These compressed networks are easier to interpret because they retain the filter diversity of uncompressed networks with order of magnitude less filters. Finally, a variant of CAR is introduced to quantify the importance of each image category to each CNN filter. Specifically, the most and the least important class labels are shown to be meaningful interpretations of each filter.

Index Terms—Convolutional neural networks, interpretation, compression, pruning

I. INTRODUCTION

DEEP convolutional neural networks (CNNs) achieve state-of-the-art performance for a wide variety of tasks in computer vision, such as image classification and segmentation [1], [2]. Recent studies have also shown that representations extracted from these networks can shed light on new tasks through transfer learning [3]. The superior performance of CNNs for large training datasets has led to their ubiquity in many industrial applications and to their emerging applications in science and medicine. Thus, CNNs are widely employed in many data-driven platforms such as cellphones, smart watches and robots. While the huge number of weights and convolutional filters in deep CNNs is key factor in their success, it makes them hard or impossible to interpret in general and especially for scientific and medical applications [4], [5]. Compressing CNNs or reducing the number of weights, while keeping prediction performance, thus facilitates interpretation, and understanding in science and medicine. Moreover, compression benefits the use of CNNs in platforms with limited memory and computational power.

In this paper, interpretability is defined as the ability to explain or to present the decisions made by the model in understandable terms to a human [6], say a biologist or a radiologist. Interpretability is typically studied from one of

Reza Abbasi-Asl is with the Department of Neurology and Weill Institute for Neuroscience, University of California, San Francisco, CA, 94158 USA, and Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, CA 94720, USA e-mail: Reza.AbbasiAsl@ucsf.edu. Bin Yu is with the Departments of Statistics and Electrical Engineering and Computer Sciences, University of California, Berkeley, CA, 94720 USA e-mail: binyu@berkeley.edu.

two perspectives. The first is algorithmic interpretablity and transparency of the learning mechanism. The other is post-hoc interpretability and explanation of the learned model using tools such as visualization. The first perspective attempts to answer the question that how the model learns and works, while the second perspective describes the predictions without explaining the learning mechanism. From the perspective of post-hoc interpretability, a CNN with fewer filters is easier to visualize and explain to human users, because CNNs are often visualized using graphical explanation of their filters [7]. Thus to make more interpretable CNNs, a compression scheme should reduce the number of filters while keeping the model accurate (predictively). We call such schemes ”structural compression”. In this paper, we argue that structurally compressed networks with fewer numbers of filters are easier to be investigated or interpreted by humans for possible domain knowledge gain.

The problem of compressing deep CNNs have been widely studied in literature, even though interpretability is not a motivating factor in majority of these studies. In the classical approach to compression of CNNs, individual weights, and not filters, are pruned and quantized [8]. We call these classical compression schemes ”weight compression”. Optimal brain damage [9], optimal brain surgeon [10], Deep Compression [8], binary neural networks [11]–[13] and most recently SqueezeNet [14] are some examples.

On the other hand, some studies have investigated pruning filters instead of weights, however, the interpretability of pruned networks has not been studied in details [15]–[24]. These studies are focused on high compression rates and low memory usage. In this paper, our goal is not to achieve state-of-the-art compression ratio or memory usage rates, but we aim to investigate the interpretability of a compressed network. However, to compare our compression ratio and computational cost to a baseline method, we chose the structural compression in [25], [26]. He et al. [25] and Li et al. [26] have studied structural compression based on removing filters and introduced importance indices based on average of incoming or outgoing weights to a filter.

Pruning activations or feature-maps to achieve faster CNNs has been also studied in [27]. Pruning activations can be viewed as removing filters in specific locations of the input, however, those filters almost always remain in other locations. Thus it rarely results in any compression of filters. On the other hand, pruning filters from the structure is equal to removing them for all the possible locations and avoiding to store them. Additionally, because of the simplified structure, filter-pruned networks are more interpretable compared to activationpruned ones, therefore more applicable in scientific and medical domains.

Pruning a fully-trained neural network has a number of advantages over training the network from scratch with fewer filters. A difficulty in training a network from scratch is not knowing which architecture or how many filters to start with. While several hyper-parameter optimization techniques [28]– [31] exist, the huge numbers of possible architectures and filters would lead to a high computational cost in a combinatorial manner as in other model selection problems [32]. Pruning provides a systematic approach to find the minimum number of filters in each layer required for accurate training. Furthermore, recent results suggest that for large-scale CNNs, the accuracy of the pruned network is slightly higher compared to a network trained from scratch ([33] for VGG and ResNet, [34] for AlexNet). For small-scale CNNs, it is possible to train a network from scratch that achieves the same accuracy as the pruned network even though the aforementioned computational cost is not trivial in this case. Additionally, in the majority of transfer learning applications based on well-trained CNNs, pruning algorithms achieve higher accuracies compared to training from scratch given the same architecture and number of filters [27], [35]. For example, [35] showed that a pruned AlexNet gains 47% more classification accuracy in bird species categorization compared to training the network from scratch.

Our main contributions in this paper are two folds. First, we introduce a greedy structural compression scheme to prune filters in CNNs. A filter importance index is defined to be the classification accuracy reduction (CAR) (similarly for regression or RAR) of the network after pruning that filter. This is similar in spirit to the regression variable importance measures in [36], [37]. We then iteratively prune filters in a greedy fashion based on the CAR importance index. Although achieving state of the art compression ratio is not the main goal in this paper, we show that our CAR structural compression scheme achieve higher classification accuracy in a hold-out test set compared to the baseline structural compression methods. CAR compressed AlexNet without retraining can achieve a compression ratio of 1.17 (for layer 1) to 1.5 (for layer 5) while having a close-to-original classification accuracy (54% top-1 classification accuracy compared to original 57%). This is 21% (for layer 1) to 43% (for layer 5) higher than the compression ratio from the benchmark method. If we fine-tune or retrain the CAR-compressed network, the compression ratio can be as high as 1.79 (for layer 3) when maintaining the same 54% classification accuracy. We take advantage of weight pruning, quantization and coding by combining our method with Deep Compression [8] and report considerably improved compression ratio. For AlexNet, we reduce the size of individual convolutional layers by factor of 8 (for layer 1) to 21 (for layer 3), while achieving close to original classification accuracy (or 54% compared to 57%) through retraining the network.

Our second contribution is bridging the compression and interpretation for CNNs. We demonstrate the ability of our CAR algorithm to remove functionally redundant filters such as color filters making the compressed CNNs more accessible to human interpreters without much classification accuracy loss. To our knowledge, such a connection between compression and functionality has not been reported previously. Furthermore, we introduce a variant of our CAR index that quantifies the importance of each image class to each CNN filter. This variant of our CAR importance index has been presented in [38], and is included in the Section 4 of this paper to establish the usefulness of the CAR index. Through this metric, a meaningful interpretation of each filter can be learned from the most and the least important class labels. This new interpretation of a filter is consistent with the visualized pattern selectivity of that filter.

The rest of the paper is organized as follows. In section 2, we introduce our CAR compression algorithm. The performance of the compression for the state-of-the-art CNNs in handwritten digit image and naturalistic image classification tasks is investigated in section 3.1. In section 3.2, we connect compression to the interpretation of CNNs by visualizing functionality of pruned and kept filters in a CNN. In section 4, a class-based interpretation of CNN filters using a variant of our CAR importance index is presented. The paper is concluded in section 5.

II. CAR-BASED STRUCTURAL COMPRESSION

A. Notation

We first introduce notations. Let tional filter in layer L of the network and the number of filters in this layer (). Each convolutional filter is a 3-dimensional tensor with the size of where is the size of spatial receptive field of the filter.

The activation or the feature map of filter i in layer L (

where is the nonlinear function in convolutional network (e.g. sigmoid or ReLU) and P denotes a block of activations from layer (i.e. the input to the neurons in layer L). The activation for the first layer could be patches of input images to the convolutional network.

Assuming network N is trained on classification task, top-1 classification accuracy of network N is defined as:

where are the number of correct and incorrect predicted classes, respectively.

In this paper, we use FLOPs to quantify the computational cost in each convolutional layer of the neural network. FLOPs for each layer of the network equals to the number of floating-point operations required in that layer to classify one image. Let’s assume is the input feature map and is the output feature map in layer L where is the spatial size. The FLOPs for this convolutional layer equals to . Additionally, the storage overhead for each convolutional layer of the network equals to bytes [39].

B. The proposed algorithm based on CAR importance index

In this section, we introduce our greedy algorithm to prune filters in layers of a CNN and structurally compress it. Figure

Fig. 1: Greedy compression of CNNs based on pruning filters

1 shows the process of greedy filter pruning. In each iteration, a candidate filter together with its connections to the next layer, gets removed from the network. The candidate filter should be selected based on an importance index of that filter. Therefore, defining an index of importance for a filter is necessary for any structural compression algorithm. Previous works used importance indices such as average of incoming and outgoing weights to and from a filter, but with unfortunately a considerable reduction of classification accuracy (e.g. 43% as mentioned earlier if one prunes only the first layer) for the compressed CNNs [25], [26]. To overcome this limitation, we define the importance measure for each filter in each layer as the classification accuracy reduction (CAR) when that filter is pruned from the network. This is similar in spirit to the importance measures defined for single variables in Random Forest [36] and distribution-free predictive inference [37]. Formally, we define CAR importance index for filter i in layer L of a convolutional neural network as:

where network is network N except that filter i from layer L together with all of its connections to the next layer are removed from the network. In our CAR structural (or filter pruning) compression algorithm, the filter with the least effect on the classification accuracy gets pruned in each iteration. The network can be retrained in each iteration and after pruning a filter. This process is regarded as fine tuning in this paper. We present details of our fine tuning procedure in the next section. Algorithm 1 shows the pseudo code of our CAR greedy structural compression algorithm. Here, and are, respectively, the number of remaining filters and compression ratio in the current iteration.

One possible drawback of the algorithm is the expensive computational cost of the early iterations. While this is a one-time computational cost for a CNN, it is still possible to significantly reduce this cost and increase the compression speed. To accomplish this, we propose the following two simple tweaks: 1. Pruning multiple filters in each iteration of the CAR algorithm. 2. Reducing the number of images for evaluating the accuracy in each iteration (i.e. batch size). Our experiments in Supplementary Materials (Figures S.1 and S.2) suggest that the accuracy remains close to the original CAR compression, when removing multiple filters at each iteration with a smaller batch size. While these tweaks increase the compression speed, the performance of the compressed network is slightly lower than Algorithm 1. The greedy process seems to allow for a better data and network adaptation and improves compression performance. That is, when pruning one filter at each iteration, we only remove the least important filter. In the next iteration, we update all the importance indexes using the new structure. This allows the algorithm to adapt to the new structure gradually and slightly improves the classification accuracy.

The CAR compression is designed to compress each individual layer separately. This is sufficient for the majority of the transfer learning and interpretability applications, because each layer is interpreted individually. However, it is possible to compress multiple layers together too. This has been discussed in Figure S.3 in Supplementary Materials.

III. RESULTS A. Compression rate and classification accuracy of the CAR compressed networks

To evaluate our proposed CAR structural compression algorithm, we have compressed LeNet [40] (with 2 convolutional layers and 20 filters in the first layer), AlexNet [1] (with 5 convolutional layers and 96 filters in the first layer) and ResNet-50 [41] (with 50 convolutional layers and 96 filters in the first layer). LeNet is a commonly used CNN trained for classification task on MNIST [40] consisting of 60,000 handwritten digit images. AlexNet and ResNet-50 are trained on the subset of ImageNet dataset used in ILSVRC 2012 competition [42] consisting of more than 1 million natural images in 1000 classes.

We used Caffe [43] to implement our compression algorithm for CNNs and fine tune them. The pre-trained LeNet and AlexNet are obtained from Caffe model zoo. All computations were performed on an NVIDIA Tesla K80 GPU. The CAR index is computed using half of the ImageNet test set. To avoid overfitting, the final performance of CAR compressed network is evaluated on the other half of the ImageNet test set. The running time of each pruning iteration depends on the number of filters remaining in the layer. On average, each iteration of CAR takes 45 minutes for the first layer of AlexNet. For 96 filters in this layer, the total compression time is 72 hours. However, in Supplemental Materials, we show that it is possible prune up to five filters in one iteration without loss in the accuracy. This reduces the total running time of the compression to 14 hours. Note that this is a one-time computational cost and much less than the time required to train AlexNet on our GPU which could take weeks.

For the fine-tuning, the learning rate has been set to 0 for the layer that is being compressed, 0.001 for the subsequent layer and 0.0001 for all other layers. The subsequent layer is directly affected by the compressed layer, therefore, requires higher learning rate. The network is retrained for 500 iterations. This is sufficent for the classification accuracy to converge to the final value.

1) LeNet on MNIST dataset: LeNet-5 is a four-layer CNN consisting of two convolutional layers and two fully-connected layers. CAR-compression has been performed on the convolutional layers and the performance on a hold-out test set is reported in Figure 2. We obtained classification accuracies (top-1) of the CAR-compression results (purple curve) and those from retraining or fine-tuning after CAR-compression on the same classification task (blue curve).

To compare the performance of our compression algorithm to benchmark filter pruning schemes, we have also implemented the compression algorithm based on pruning incoming and outgoing weights proposed in [25] and reported the classifi-cation accuracy curve in Figure 2. Furthermore, classification accuracy for random pruning of filters in LeNet has been shown in this figure. Candidate filters to prune are selected uniformly at random in this case. The error bar shows the standard deviation over 10 repeats of this random selection.

We conclude that our CAR-algorithm gives a similar classification accuracy to [25] for LeNet (using the outgoing weights in the first layer, and either weights for the second layer). Their accuracies are similar to the accuracy of the uncompressed, unless we keep very few filters for either layer. Fine-tuning improves the classification accuracy but there is not a considerable gap among performances (unless we keep very few filters, less than 8 among 20 for the first layer or less than 10 among 50 for the second layer). Among the 8 kept filters in the first layer, 4 of them are shared between the CAR-algorithm and that based on averaging outgoing weights in [25], while among the 10 kept filters in the second layer, 6 of them are shared.

2) AlexNet on ImageNet dataset: AlexNet consists of 5 convolutional layers and 3 fully-connected layers. Figure 3 shows the classification accuracy of AlexNet on a hold-out test set after each individual convolutional layer is compressed using our proposed CAR algorithms or benchmark compression schemes.

Comparing the accuracies of compressed networks in Figure 3, there are considerable gaps between our proposed CARalgorithm (purple curves) and the competing structural compression schemes that prune filters [25] for all five layers. Further considerable improvements are achieved by retraining or fine-tuning the CAR-compressed networks (see the blue curves in Figure 3).

Pruning half of the filters in either of the individual convolutional layers in AlexNet, our CAR algorithm achieves 16% (for the layer 5) to 25% (for the layer 2) higher classification accuracies compared to the best benchmark filter pruning scheme (pruning based on average outgoing weights). If we retrain the pruned network, it achieves 50% to 52% classification accuracy (compared with 57% of the uncompressed AlexNet). The superior performance of our algorithm for AlexNet is due to the proposed importance index for the filters in CNN. This figure demonstrates that our algorithm is able to successfully identify the least important filters for the purpose of classification accuracy. In section 5.2, we discuss the ability of our compression scheme to reduce functional redundancy in the structure of CNNs.

To present a different but equivalent quantitative comparison, we have reported the compression ratio and feed-forward speed up in Table I. Each individual convolutional filter is pruned while the classification accuracy dropped a relative 5% from the accuracy of uncompressed network (i.e. 54% compared to 57%). Results for CAR compression with and without fine tuning and compression based on average incoming and outgoing weights are presented in this table. The CAR algorithm (without retraining) can achieve a compression ratio of 1.17 (for layer 1) to 1.50 (for layer 5), which is 21% to 43% higher than those from the benchmark methods. If we fine-tune or retrain the CAR-compressed network, the compression ratio can be as high as 1.79 (for layer 3) when maintaining the same 54% classification accuracy.

3) Combination with Deep Compression: One advantage of our CAR-algorithm is that it is amenable to combination with weight based compression schemes to achieve substantial reduction in memory usage. Deep Compression [8] is a recent weight-based compression procedure that uses weight pruning, quantization and huffman coding. We have performed Deep Compression on top of our proposed compression algorithm and reported the compression ratio for AlexNet in Table II. Again, the filters are pruned while the classification accuracy is in the range of relative 5% from the accuracy of uncompressed network (54% compared to 57%). An additional 5 fold (for layer 1) to 12 fold (for layer 3) increase in compression ratio is acheived through joint CAR and Deep Compression. That is, further weight compression boosts the compression ratio by sparsifying weights of the kept filters, although the number of filters is the same as the CAR compression.

4) ResNet-50 on ImageNet dataset: First introduced by He et al. [41], deep residual networks take advantage of a residual block in their architecture (Figure 4, right panel) to achieve higher classification accuracy compared to a simple convolutional network. We have studied the performance of CAR compression on ResNet-50 architecture [41] with 50 layers of convolutional weights. Figure 4.A shows the classification accuracy of ResNet-50 after pruning first convolutional layer using CAR algorithm or benchmark compression schemes. Figure 4.B shows the classification accuracy after pruning the first convolutional layer in the first residual block (layer Conv a - Branch 2 in Figure 4). The performance for all other residual blocks are illustrated in Figure 4.C. CAR pruning of other convolutional layers in each residual block yields to similar figures and are not shown here. All of the accuracies are reported on the ILSVRC 2012 ImageNet hold out test set.

It is of great interest to compare at high compression ratio regimes where we keep less than 30 filters out of 64. In this situation and pruning layer Conv 1, the CAR algorithm (purple curve in Figure 4) outperforms the competitors based on incoming and outgoing weights. The higher the compression ratio, the higher the improvements by the CAR algorithm. For

Fig. 2: Performance of compression for LeNet. The top figure shows the overall classification accuracy of LeNet when the first convolutional layer is compressed. The bottom figure shows the classification accuracy when the second convolutional layer is compressed. The classification accuracy of uncompressed network is shown with a dashed red line. The purple curve shows the classification accuracy of our proposed CAR compression algorithm for various compression ratios. The accuracy for the fine tuned (retrained) CAR compression is shown in blue. The black and green curves shows the accuracy for compressed network based on outgoing and incoming weights, respectively. The red curve shows the accuracy when filters are pruned uniformly at random. The error bas is reported over 10 repeats of this random pruning process.

Fig. 3: Performance of compression for AlexNet. Each panel shows the classification accuracy of the AlexNet when an individual convolutional layer is compressed. In each panel, the classification accuracy of uncompressed network is shown with a dashed red line. The purple curve shows the classification accuracy of our proposed CAR compression algorithm for various compression ratios. The accuracy for the fine tuned (retrained) CAR compression is shown in blue. The black and green curves shows the accuracy for compressed network based on outgoing and incoming weights, respectively. The red curve shows the accuracy when filters are pruned uniformly at random. The error bas is reported over 10 repeats of this random pruning process.

low compression ratio regimes, the performances are similar. Compared to AlexNet, the gap between CAR and benchmark

TABLE I: Comparison of compression performance between our greedy CAR compression algorithm and benchmark schemes on AlexNet. For the compressed networks, the filters are pruned while the classification accuracy dropped a relative 5% from the accuracy of original network (i.e. 54% compared to 57%). FLOPs equals to the number of floating-point operations required in each layer to classify one image.

TABLE II: Compression performance of CAR-algorithm combined with Deep Compression

compressions is smaller for the first layer. This might be an evidence that ResNet has less redundant filters. Retraining (fine-tuning) the CAR-compressed network achieves further improvement in classification accuracy (blue curve in Figure 4). In fact, our CAR-algorithm achieves 72% classification accuracy (compared with the 75% for the uncompressed ResNet-50) when pruning half of the filters in the first layer of ResNet-50. This accuracy is 15% higher than that of filter pruning based on average outgoing or incoming weights.

For the residual block, we have pruned layer Conv a - Branch 2 and reported the classification accuracy in Figure 4. The accuracy of CAR algorithm is almost similar to the compression based on incoming and outgoing weights. Interestingly, the accuracy drops less than 15% if we fully prune the filters in this layer i.e. remove branch 2 from the residual block. The drop in accuracy is less than 5% for the fine-tuned network. The main reason for this is the existence of shortcuts in the residual module. The uncompressed branch 1 that is a parallel channel with the pruned filter allows the information to transfer through the residual layer. As a result of these parallel channels in the residual blocks, deep residual networks are more robust to pruning filters compared to simple convolutional networks.

B. CAR-compression algorithm prunes visually redundant filters

To study the ability of CAR compression in identifying redundant filters in CNNs, we take a closer look at the visualization of pruned filters. Filters in the first layer of a CNN can be visualized directly using their weights (weights in the first layer filters correspond to RGB channels of the input image). Figure 5 shows the the visualized filters in the first layer of AlexNet, ordered by their CAR importance index. Filters with higher CAR index tend to form a set of diverse patterns, spanning different orientations and spatial frequencies. Additionally, most of the filters with color selectivity tend to

Fig. 4: Performance of compression for ResNet-50. A. Classification accuracy of the ResNet-50 for the compression of first convolutional layer. The classification accuracy of uncompressed network is shown with a dashed red line. The purple curve shows the classification accuracy of our proposed CAR compression algorithm for various compression ratios. The accuracy for the fine tuned (retrained) CAR compression is shown in blue. The black and green curves shows the accuracy for compressed network based on outgoing and incoming weights, respectively. The red curve shows the accuracy when filters are pruned uniformly at random. The error bas is reported over 10 repeats of this random pruning process. B. Classification accuracy for the compression of first residual module (with the first layer untouched). C. Classification accuracy for the compression of each residual module in ResNet-50. D. The architecture of first layers in ResNet-50.

have lower CAR index. In fact, out of top 20 pruned filters, 15 of them in the first layer and 14 of them in the second layer correspond to the color filters, respectively. This finding points to the fact that shape is often first-order important for object recognition.

Unlike the first layer, visualization of filters in higher

convolutional layers is not trivial. To visualize the pattern selectivity of filters in these higher layers, we have fed one million image patch to the network and showed the top 9 image patch that activate each filter. This approach has been previously used to study functionality of filters in deep CNNs [7]. There are 256 filters in the layer 2 of AlexNet which

Fig. 5: Visualization of filters in the first layer of AlexNet, ordered by their CAR importance index.

makes it challenging to visualize all of these filters. Therefore, we manually grouped filters into subsets with visually similar pattern selectivity in Figure6. To investigate the ability of CAR compression in removing visually redundant filters in this layer, we continued to iterate the CAR algorithm while the classification accuracy is 54% or within a relative 5% from the accuracy of uncompressed network. This led to pruning 103 filters out of 256 filters in the second layer. A subset of the removed and remaining filters are visualized in Figure 6. The filters shown with red circles are pruned in the CAR process. Similar figures for other layers are shown in Supplementary Materials (Figures S.4 and S.5). Our algorithm tends to keep at least one filter from each group, suggesting that our greedy filter pruning process is able to identify redundant filters. This indicates that pruned filters based on the CAR importance index have in fact redundant functionality in the network.

To further investigate the effect of compression of each of the convolutional layers, we have shown the scatter plots of the classification accuracy for each of the 1000 classes in ImageNet in Figure 7. Although the total classification accuracy is about a relative 5% lower for the each compressed network, the accuracies for many of the categories are comparable between compressed and uncompressed networks. In fact, 37% (for layer 5) to 49% (for layer 2) of the categories have accuracies no larger than 3% below those for the uncompressed network.

IV. CLASS-BASED INTERPRETATION OF FILTERS

With a slight modification in the definition for the CAR importance index, we build a new index to interpret the filters via image class labels. This index has been introduced in [38] and also included in this section to demonstrate the merit of the CAR index. We define to be classification accuracy reduction in class c of images when filter i in layer L is pruned. quantifies the importance of each filter in predicting a class label. Therefore, for each filter, we can use to identify classes that are highly dependent on that filter (classification accuracy for these classes depends on the existence of that filter). These classes are the ones with highest among all other classes. Similarly, for each filter, the performance in classes with the lowest less dependency to that filter.

The labels of the two sets of classes with highest and lowest present a verbal interpretation of each filter in the network. This is particularly important in application of CNNs in scientific domains such as medicine, where it is necessary to provide a verbal explanation of the filters for the user. based interpretation is a better fit for the higher layers in the CNN because filters in these layers are more semantic and therefore more explainable by the class labels. For these layers, the interpretation of filters based on is consistent with the visualization of pattern selectivity for that filter. Figure 8 illustrates this consistency for layer 5 of AlexNet. We focus on three filters in layer 5 that are among the most important filters in this layer based on our original CAR pruning. Similar to Figure 7, the pattern selectivity of each filter is visualized in panel A using top 9 image patch activating that filter. Panels B and C show the top and bottom 5 classes with highest and lowest , respectively. Beside the class label, one sample image from that class is also visualized. Some of these classes are pointed out with green arrows in the scatter plot of classification accuracy for 1000 classes in ImageNet (panel D). Note that both CAR and indexes could be negative numbers, that is the pruned network has higher classification accuracy compared to the original network.

In Figure 8, the classes with highest share similar patterns with the top 9 patches activating each filter. For filter 1, the smooth elliptic curvature that consistently appears in the classes such as steep arch bridge or soup bowel is visible in the top activating patches (Panel A). On the other hand, less elliptic curvature patterns are expected in classes such as mailbag or altar. Filter 2 has higher for classes that contains patterns such as insect or bird’s head. Filter 3 is mostly selected by the classes that contain images of a single long tool, particularly musical instruments such as oboe or banjo.

V. DISCUSSION AND FUTURE WORK

Structural compression (or filter pruning) of CNNs has the dual purposes of saving memory cost and computational cost on small devices, and of resulted CNNs being more humanly interpretable in general and for scientific and medical applications in particular. In this paper, we proposed a greedy filter pruning based on the importance index of classification accuracy reduction (CAR). We have shown with AlexNet that the huge gain (8 to 21 folds) in compression ratio of CAR+Deep Compression schemes, without a serious loss of classification accuracy. Furthermore, we saw that the pruned filters have

Fig. 6: CAR compression removes filters with visually redundant functionality from second layer of AlexNet. To visualize each filter, we have fed one million image patch to the network and visualized each filter by 9 image patches with top response for that filter. We have manually clustered 256 filters in the second layer of AlexNet into 20 groups (11 of them visualized here) based on their pattern selectivity. We continue to iterate the CAR-based algorithm while the classification accuracy is in the relative range of 5% from the accuracy of uncompressed network. This leads to pruning 103 out of 256 filters in this layer. The pruned filters are specified with a red circle.

Fig. 7: Classification accuracy for each class of image in AlexNet after the first (left panel) or second layer (right panel) is compressed compared to the uncompressed network. Each point in plots corresponds to one of the 1000 categories of images in test set.

Fig. 8: The interpretation based on is consistent with the visualized pattern selectivity of each filter in layer 5 of AlexNet. Panel A shows The top 9 image patches that activate each filter [7]. Panel B and C show the top and bottom 5 classes with highest and lowest , respectively. Besides the class label, one sample image from that class is also visualized. Panel D shows the scatter plot of classification accuracy for each of the 1000 classes in ImageNet. Three of the top and bottom classes with highest and lowest are pointed out with green arrows. Each row corresponds to one filter in layer 5 of AlexNet.

redundant functionality for the AlexNet. In particular, for many categories in ImageNet, we found that the redundant filters are color-based instead of shape-based. This suggests the first order importance of shape for such categories.

However, a greedy algorithm is likely to be sub-optimal in identifying the best candidate filters to drop. The optimal solution may be to search through all possible subsets of filters to prune, but this can be computationally expensive and may lead to over-pruning. Procedures for subset selection, including genetic algorithms and particle swarm optimization, could be helpful in the compression of CNNs and will be investigated in future work. Even though the CAR compression of ResNet achieves state-of-the-art classification accuracy among other structural compressions by pruning the identity branch and identifying the redundant connections. ResNet compression merits further investigation because of the identity branches in the residual blocks.

We also proposed a variant of CAR index to compare

classification accuracies of original and pruned CNNs for each image class. In general, we can compare any two convolutional neural networks that are trained on the similar dataset through this index. The comparison could be done by looking into set of classes that are important for each filter in layer of each network. A similar class-based comparison for any two networks through our importance index is possible. This is a fruitful direction to pursue, particularly given the recent wave of various CNNs with different structures. Finally, we expect that our CAR structural compression algorithm for CNNs and related interpretations can be adapted to fully-connected networks with modifications.

ACKNOWLEDGEMENTS

This work is supported in part by National Science Foundation (NSF) Grants DMS-1613002, and IIS-1741340, and the Center for Science of Information, an NSF Science and Technology Center, under Grant Agreement CCF-0939370.

REFERENCES

[1] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, 2012.

[2] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3431– 3440, 2015.

[3] Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid-level image representations using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1717–1724, 2014.

[4] Gr´egoire Montavon, Wojciech Samek, and Klaus-Robert M¨uller. Methods for interpreting and understanding deep neural networks. Digital Signal Processing, 2017.

[5] Reza Abbasi-Asl, Yuansi Chen, Adam Bloniarz, Michael Oliver, Ben DB Willmore, Jack L Gallant, and Bin Yu. The deeptune framework for modeling and characterizing neurons in visual cortex area v4. bioRxiv, page 465534, 2018.

[6] Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. 2017.

[7] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818–833. Springer, 2014.

[8] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015.

[9] Yann LeCun, John S Denker, Sara A Solla, Richard E Howard, and Lawrence D Jackel. Optimal brain damage. In NIPS, volume 2, pages 598–605, 1989.

[10] Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In Neural Networks, 1993., IEEE International Conference on, pages 293–299. IEEE, 1993.

[11] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision, pages 525– 542. Springer, 2016.

[12] Jung H Kim and Sung-Kwon Park. The geometrical learning of binary neural networks. IEEE transactions on neural networks, 6(1):237–247, 1995.

[13] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in neural information processing systems, pages 3123–3131, 2015.

[14] Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size. arXiv preprint arXiv:1602.07360, 2016.

[15] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, pages 2736–2744, 2017.

[16] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 1389–1397, 2017.

[17] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems, pages 2074–2082, 2016.

[18] Jose M Alvarez and Mathieu Salzmann. Learning the number of neurons in deep networks. In Advances in Neural Information Processing Systems, pages 2270–2278, 2016.

[19] Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pages 5058–5066, 2017.

[20] Chenglong Zhao, Bingbing Ni, Jian Zhang, Qiwei Zhao, Wenjun Zhang, and Qi Tian. Variational convolutional neural network pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2780–2789, 2019.

[21] Hanyu Peng, Jiaxiang Wu, Shifeng Chen, and Junzhou Huang. Collaborative channel pruning for deep networks. In International Conference on Machine Learning, pages 5113–5122, 2019.

[22] Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Tim Kwang-Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. arXiv preprint arXiv:1903.10258, 2019.

[23] Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. In Advances in Neural Information Processing Systems, pages 2130–2141, 2019.

[24] Yuchao Li, Shaohui Lin, Baochang Zhang, Jianzhuang Liu, David Doermann, Yongjian Wu, Feiyue Huang, and Rongrong Ji. Exploiting kernel sparsity and entropy for interpretable cnn compression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2800–2809, 2019.

[25] Tianxing He, Yuchen Fan, Yanmin Qian, Tian Tan, and Kai Yu. Reshaping deep neural network for fast decoding by node-pruning. In Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on, pages 245–249. IEEE, 2014.

[26] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710, 2016.

[27] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. 2017.

[28] Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In Advances in neural information processing systems, pages 2951–2959, 2012.

[29] Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08734, 2017.

[30] Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint arXiv:1711.09846, 2017.

[31] Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. The Journal of Machine Learning Research, 18(1):6765–6816, 2017.

[32] Russell Reed. Pruning algorithms-a survey. IEEE transactions on Neural Networks, 4(5):740–747, 1993.

[33] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710, 2017.

[34] Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Compression of deep convolutional neural networks for fast and low power mobile applications. In international Conference on Learning Representations, 2016.

[35] Steve Branson, Grant Van Horn, Serge Belongie, and Pietro Perona. Bird species categorization using pose normalized deep convolutional nets. arXiv preprint arXiv:1406.2952, 2014.

[36] Leo Breiman. Random forests. Machine learning, 45(1):5–32, 2001.

[37] Jing Lei, Max GSell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113(523):1094–1111, 2018.

[38] Reza Abbasi-Asl and Bin Yu. Interpreting convolutional neural networks through compression. arXiv preprint arXiv:1711.02329, 2017.

[39] Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng. Quantized convolutional neural networks for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4820–4828, 2016.

[40] Yann LeCun, L´eon Bottou, Yoshua Bengio, and Patrick Haffner. Gradientbased learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.

[41] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.

[42] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211–252, 2015.

[43] Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014.

Supplementary Materials

While CAR compression constructs a more interpretable network with sufficiently high classification performance, the main drawback is the expensive computational cost. Here, we propose two following tweaks to increase the compression speed: 1. Pruning multiple filters in each iteration of the CAR algorithm. 2. Reducing the number of images for evaluating the accuracy in each iteration (i.e. batch size). Our experiments suggest that the accuracy remains close to the original CAR compression, when removing up to 5 filters at each iteration with a batch size of 128 for LeNet. Removing 5 filters at each iteration increased the computational speed by a factor of 5. Using the batch size of 128 instead of the 5000 increased the speed by a factor of 12. In total, the compression speed increased by a factor of 60 for LeNet while keeping the accuracy close to original CAR compression. Figures S.1.A illustrates the classification accuracy as a function of number of filters pruned in LeNet layer 1 when removing 1, 2, 4, or 5 filters at each iteration. The accuracy remains close to original CAR compression when pruning 5 filters at each iteration. Panels B, C, D, and E in Figure S.1 compares the accuracy curves between different batch sizes. Batch size in this figure equals to the number of images from the validation set used in each iteration of CAR. For LeNet layer 1, batch size does not have a considerable effect on the curve when pruning up to 5 filters at each iteration of the algorithm. Figure S.2 illustrates the accuracy curves for LeNet layer 2. Similar to layer 1, the accuracy remains close to original CAR compression when pruning up to 5 filters at each iteration with batch size 512. However, for LeNet Layer 2, batch size of 64 degrades the accuracy.

To study the accuracy of CAR when multiple layers are compressed together, we used CAR to compre Figure S.3 shows the classification accuracy curves as a function of portion of remaining filters in the network. As expected, the classification accuracy decreases as we increase the number of layers involved in the compression.

III. CAR PRUNED FILTERS FOR LAYERS 3 AND 4 OF ALEXNET

To further elaborate on the ability of CAR compression in identifying redundant filters, we have visualized filters in layers 3 and 4 of AlexNet in Figures S.4 and S.5, respectively. Similar to Figure 6 (in the main text) for layer 2, the filters are manually clustered based on their pattern selectivity. We continue to iterate the CAR pruning while the classification accuracy is in the relative range of 5% from the accuracy of uncompressed network. The pruned filters are specified with a red circle. Similar to layer 2, the CAR algorithm tend to keep filters with diverse functionality in the network.

0 10 20 Number of filters pruned

0 10 20 Number of filters pruned

Fig. S.1: Removing multiple filters at each iteration of CAR algorithm boosts compression speed without degrading the accuracy for LeNet layer 1. A Classification accuracy as a function of number of filters pruned in LeNet layer 1 when removing 1, 2, 4, or 5 filters at each iteration. Batch size is set to 512. The accuracy remains close to original CAR compression when pruning 5 filters at each iteration. B Comparison of the accuracy curves between different batch sizes when removing 1 filter at each iteration of the algorithm. C, D, and E Similar to B but when removing 2, 4, or 5 filters at each iteration of the algorithm, respectively. Batch size does not have a considerable effect on the curves.

0 25 50 Number of filters pruned

0 25 50 Number of filters pruned

Fig. S.2: Removing multiple filters at each iteration of CAR algorithm boosts compression speed without degrading the accuracy for LeNet layer 2. A Classification accuracy as a function of number of filters pruned in LeNet layer 2 when removing 1, 2, 4, or 5 filters at each iteration. Batch size is set to 512. The accuracy remains close to original CAR compression when pruning 5 filters at each iteration. B Comparison of the accuracy curves between different batch sizes when removing 1 filter at each iteration of the algorithm. C, D, and E Similar to B but when removing 2, 4, or 5 filters at each iteration of the algorithm, respectively. Batch size of 64 degrades the accuracy.

Fig. S.3: Classification accuracy when compressing multiple layers together. Classification accuracy curves as a function of portion of remaining filters in the network are shown for compressing multiple combination of layers. The classification accuracy decreases as we increase the number of layers involved in the compression.

Fig. S.4: CAR compression removes filters with visually redundant functionality from third layer of AlexNet. To visualize each filter, we have fed one million image patch to the network and visualized each filter by 9 image patches with top response for that filter. We have manually clustered filters in the third layer of AlexNet based on their pattern selectivity. We continue to iterate the CAR-based algorithm while the classification accuracy is in the relative range of 5% from the accuracy of uncompressed network. This leads to pruning 170 out of 384 filters in this layer. The pruned filters are specified with a red circle.

Fig. S.5: CAR compression removes filters with visually redundant functionality from fourth layer of AlexNet. To visualize each filter, we have fed one million image patch to the network and visualized each filter by 9 image patches with top response for that filter. We have manually clustered filters in the third layer of AlexNet based on their pattern selectivity. We continue to iterate the CAR-based algorithm while the classification accuracy is in the relative range of 5% from the accuracy of uncompressed network. This leads to pruning 159 out of 384 filters in this layer. The pruned filters are specified with a red circle.

designed for accessibility and to further open science