Single image super-resolution (SISR) is a classical problem in low-level computer vision, which reconstructs a high-resolution (HR) image from a low-resolution (LR) image. Actually, an infinite number of HR images can get the same LR image by downsampling. Hence, the SR problem is inherently ill-posed and no unique solution exists. In order to mitigate this problem, numerous SISR methods have been proposed in the literature, including interpolation-based methods, reconstruction-based methods
Figure 1. Speed and accuracy trade-off. The average PSNR and the average inference time for upscaling on Set5. The IDN is faster than other methods and achieves the best performance at the same time.
and example-based methods. Since the former two kinds of methods usually suffer dramatically drop in restoration performance with larger upscaling factors, the recent SR methods fall into the example-based methods which try to learn prior knowledge from LR and HR pairs.
Recently, due to the strength of deep convolutional neural network (CNN), many CNN-based SR methods try to train a deep network to gain better reconstruction performance. Kim et al. propose a 20-layer CNN model known as VDSR [12], which adopts residual learning and adaptive gradient clipping to ease training difficulty. To control the model parameters, the authors construct a deeply-recursive convolutional network (DRCN) [13] by adopting recursive layer. To mitigate training difficulty, Mao et al. propose a very deep residual encoder-decoder network (RED) [17], which consists of a series of convolutional and subsequent transposed convolution layers to learn end-to-end mappings from the LR images to the ground truths. Tai et al. propose a deep recursive residual network (DRRN) [22], which employs parameters sharing strategy to alleviate the require- ment of enormous parameters of the very deep networks.
Although achieving prominent performance, most of deep networks still have some drawbacks. Firstly, in order to achieve better performance, deepening or widening the network has been a design trend. But the result is that these methods demand large computational cost and memory consumption, which are less applicable in practice, such as mobile and embedded vision applications. Moreover, the traditional convolutional networks usually adopt cascaded network topologies, e.g., VDSR [12] and DRCN [13]. In this way, the feature maps of each layer are sent to the sequential layer without distinction. However, Hu et al. [9] experimentally demonstrate that adaptively recalibrating channel-wise features responses can improve the representational power of a network.
To address these drawbacks, we propose a novel information distillation network (IDN) with lightweight parameters and computational complexity as illustrated in Figure 2. In the proposed IDN, a feature extraction block (FBlock) first extracts features from the LR image. Then, multiple information distillation blocks (DBlocks) are stacked to progressively distill residual information. Finally, a reconstruction Block (RBlock) aggregates the obtained HR residual representations to generate the residual image. To get a HR image, we implement an element-wise addition operation on the residual image and the upsampled LR image.
The key component of IDN is the information distillation block, which contains an enhancement unit and a compression unit. The enhancement unit mainly comprises two shallow convolutional networks as illustrated in Figure 3. Each of them is a three-layer shallow module. The feature maps of the first module are extracted through a short path (3-layer). Thus, they can be regarded as the local short-path features. Considering that the deep networks have more expressive power, we send a portion of the local short-path features to another module. By this way, the feature maps of the second module naturally become the local long-path features. Different from the approach in [9], we divide feature maps into two parts. One part represents reserved short-path features and another expresses the short-path features that will be enhanced. After getting long and short-path feature maps, we aggregate these two types of features for gaining more abundant and efficient information. In summary, the enhancement unit is mainly to improve the representation power of the network. As for the compression unit, we adopt a simple convolutional layer to compress the redundancy information in features of the enhancement unit.
The main contributions of this work are summarized as follows:
• The proposed IDN extracts feature maps directly from LR images and employs multiple cascaded DBlocks
to generate the residual representations in HR space. In each DBlock, the enhancement unit gathers more information as much as possible and the compression unit distills more useful information. As a result, IDN achieves competitive results in spite of using less number of convolutional layer.
• Due to the concise structure of the proposed IDN, it is much faster than several CNN-based SR methods, e.g., VDSR [12], DRCN [13], LapSRN [15], DRRN [22] and MemNet [23] as illustrated in Figure 1. Only the proposed method achieves real-time speed and maintains better reconstruction accuracy.
Single image super-resolution has been extensively studied in these years. In this section, we will focus on recent example-based and neural network based approaches.
2.1. Self-example based methods
The self-example based methods exploit the self-similarity property and extract example pairs merely from the LR image across different scales. This type of methods usually works well in the images containing repetitive patterns and textures but lacks the richness of image structures outside the input image and thus fails to generate satisfactory prediction for images of other classes. Huang et al. [10] extend self-similarity based SR to handle the affine and perspective deformation.
2.2. External-example based methods
The external-example based methods learn a mapping between LR and HR patches from external datasets. This type of approaches usually focuses on how to learn a compact dictionary or manifold space to relate LR/HR patches, such as nearest neighbor [7], manifold embedding [2], random forest [20] and sparse representation [25, 26]. While these approaches are effective, the extracted features and mapping functions are not adaptive, which may not be optimal for generating high-quality SR images.
2.3. Convolutional neural networks based methods
Recently, inspired by the achievement of many computer vision tasks tackled with deep learning, neural networks have been achieved dramatic improvement in SR. Dong et al. [3, 4] first exploit a three-layer convolutional neural network, named SRCNN, to jointly optimize the feature extraction, non-linear mapping and image reconstruction stages in an end-to-end manner. Afterwards Shi et al. [21] propose an efficient sub-pixel convolutional neural network (ESPCN), which extracts feature maps in the LR space and replaces the bicubic upsampling operation with an efficient
Figure 2. Architecture of the proposed network.
sub-pixel convolution. Dong et al. [5] adopt deconvolution to accelerate SRCNN in combination with smaller filter sizes and more convolution layers. Kim et al. [12] propose a very deep CNN model with global residual architecture to achieve superior performance, which utilizes contextual information over large image regions. Another network designed by Kim et al. [13], which has recursive convolution with skip connection to avoid introducing additional parameters when the depth is increasing. Mao et al. [17] tackle the general image restoration problem with encoder-decoder networks and symmetric skip connections. Lai et al. [15] propose the laplacian pyramid super-resolution network (LapSRN) to address the speed and accuracy of SR problem, which takes the original LR images as input and progressively reconstructs the sub-band residuals of HR images. Tai et al. [22] propose the deep recursive residual network to effectively build a very deep network structure for SR, which weighs the model parameters against the accuracy. The authors also present a very deep end-to-end persistent memory network (MemNet) [23] for image restoration task, which tackles the long-term dependency problem in the previous CNN architectures. Sajjadi et al. [19] propose a novel combination of automated texture synthesis with a perceptual loss focusing on creating realistic textures at a high magnification ratio of 4.
In this section, we first describe the proposed model architecture and then suggest the enhancement unit and the compression unit, which are the core of the proposed method.
3.1. Network structure
The proposed IDN, as shown in Figure 2, consists of three parts: a feature extraction block (FBlock), multiple stacked information distillation blocks (DBlocks) and a reconstruction block (RBlock). Here, we denote x and y as the input and the output of IDN. With respect to FBlock, two convolutional layers are utilized to extract the feature maps from the original LR image. This procedure can be expressed as
where f represents the feature extraction function and denotes the extracted features and servers as the input to the following stage. The next part is composed of multiple information distillation blocks by using chained mode. Each block contains an enhancement unit and a compression unit with stacked style. This process can be formulated as
where denotes the k-th DBlock function,
and
indicate the input and output of the k-th DBlock respectively. Finally, we take a transposed convolution without activation function as the RBlock. Hence, the IDN can be expressed as
where R, U denote the RBlock and bicubic interpolation operation respectively.
3.1.1 Loss function
We consider two loss functions that measure the difference between the predicted HR image and the corresponding ground-truth I. The first one is mean square error (MSE), which is the most widely used loss function for general image restoration as defined below:
However, Lim et al. [16] experimentally demonstrate that training with MSE loss is not a good choice. The second loss function is mean absolute error (MAE), which is formulated as follows:
We empirically found that our model with MSE loss can improve performance of a trained network with MAE loss. Therefore, we first train the network with MAE loss and then fine-tune it by MSE loss.
Figure 3. The architecture of enhancement unit in the proposed model. Orange circle represents slice operation and purple circle indicates concatenation operation in channel dimension.
3.2. Enhancement unit
As shown in Figure 3, enhancement unit can be roughly divided into two modules, one is the above three convolutions and another is the below three convolutions. The above module has three convolutions, each of them is followed by a leaky rectified linear unit (LReLU) activation function, which is omitted here. Let’s denote the feature map dimensions of the i-th layer as
. In that way, the relationship of the convolutional layers can be expressed as
where d denotes the difference between the first layer and the second layer or between the first layer and the third layer. Similarly, the dimension of channels in the below module also has this relation and can be described as follows:
where . The above module is composed of three cascaded convolution layers with LReLUs, and the output of the third convolution layer is sliced into two segments. Supposing the input of this module is
, we have
where denotes the output of previous block and meanwhile is the input of present block,
indicates chained convolutions operation and
is the output of the above module in the k-th enhancement unit. The feature maps with
dimensions of
and the input of the first convo- lutional layer are concatenated in the channel dimension,
where C, S represent concatenation operation and slice operation respectively. Specifically, we know the dimension of is
. Therefore,
denotes that
dimen- sions features are fetched from
. Moreover,
concatenates features with
in channel dimension. The purpose is to combine the previous information with some current information. It can be regarded as partially retained local short-path information. We take the rest of local short-path information as the input of the below module, which mainly further extracts long-path feature maps,
where ,
are the output and stacked convolution oper- ations of the below module respectively. Finally, as shown in Figure 3, the input information, the reserved local short-path information and the local long-path information are aggregated. Therefore, the enhancement unit can be formulated as
where is the output of enhancement unit. At this point, local long-path features
and the combination of local short-path features and the untreated features
are utilized without exception by a compression unit.
3.3. Compression unit
We achieve compression mechanism by taking advantage of a convolution layer. Concretely, the outputs of the enhancement unit are sent to a
convolution layer, which acts as dimensionality reduction or distilling relevant information for the later network. Thus, the compression unit can be formulated as
where denotes the function of the
convolution layer (
denotes the activation function and
is the weight parameters).
4.1. Datasets
4.1.1 Training datasets
By following [12, 15, 22, 23], we use 91 images from Yang et al. [26] and 200 images from Berkeley Segmentation Dataset (BSD) [18] as the training data. As in [22], to make full use of the training data, we apply data augmen- tation in three ways: (1) Rotate the images with the degree of and
. (2) Flip images horizontally. (3) Downscale the images with the factor of 0.9, 0.8, 0.7 and 0.6.
4.1.2 Testing datasets
The proposed method is evaluated on four widely used benchmark datasets: Set5 [1], Set14 [27], BSD100 [18], Urban100 [10]. Among these datasets, Set5, Set14 and BSD100 consist of natural scenes and Urban100 contains challenging urban scenes images with details in different frequency bands. The ground truth images are downscaled by bicubic interpolation to generate LR/HR image pairs for both training and testing datasets. We convert each color image into the YCbCr color space and only process the Ychannel, while color components are simply enlarged using bicubic interpolation.
Table 1. The sizes of training and fine-tuning sub-images for dif- ferent scaling factors.
4.2. Implementation details
For preparing the training samples, we first downsample the original HR images with upscaling factor m (m = 2, 3, 4) by using the bicubic interpolation to generate the corresponding LR images and then crop the LR training images into a set of size sub-images. The corresponding HR training images are divided into
size sub-images. As the proposed model is trained using the Caffe package [11], its transposed convolution fil-ters will generate the output with size
instead of
. So we should crop
-pixel borders on the HR sub-images. Since the minimum size picture “t20” in the 291 dataset is a
size image, the maximum size of the sub-image we can crop on the LR image is
for maintaining data integrity when scaling factor m = 3. However, the training process will be unstable due to the larger size training samples equipped with the larger learning rate by using Caffe package. Therefore,
training pairs are generated for training stage and
LR/HR sub-images pairs are utilized for fine-tuning phase. The learning rate is initially set to
and decreases by the factor of 10 during fine-tuning phase. In this way, the sizes of training and fine-tuning samples are shown in Table 1.
Taking into account the trade-off between the execution time and the reconstruction performance, we construct a 31-layer network that denoted as IDN. This model has 4 DBlocks, and the parameters and s of enhancement unit in each block are set to 64, 16 and 4 respectively. To reduce the parameters of network, we use the grouped convolution layer [6, 24] in the second and fourth layers in each enhancement unit with 4 groups. In addition, the transposed convolution adopts
filters for all scaling factors and the negative scope of LReLU is set as 0.05. We initialize the weights by using the method proposed in [8] and the biases are set to zero. The proposed network is optimized using Adam [14]. We set the parameters of mini-batch size and weight decay to 64 and
respectively. In order to get better initialization parameters, we empirically pre-train the proposed model with
iterations and take these parameters as the initial values of the IDN. Training a IDN roughly takes a day with a TITAN X GPU on the
model.
Figure 4. The residual image and its data distribution of the “but-terfly” image from Set5 dataset.
Figure 5. Visualization of the average feature maps.
4.3. Network analysis
The proposed model with a global residual structure mainly learns a residual image. As show in Figure 4(a), the ground truth residual image mainly contains details and texture information and its normalized pixel value ranges from -0.4 to 0.5. From Figure 4(b), we find that there are positive and negative values in the residual image, and the number of positive pixels is intuitively similar to that of negative ones. Obviously, the number of zero value and its neighbors is the
Table 2. Average PSNR/SSIMs for scale . Red color indicates the best and blue color indicates the second best performance.
Table 3. Average IFCs for scale . Red color indicates the best and blue color indicates the second best performance.
most, which suggests that smooth region in residual image is almost eliminated. Therefore, the task of our network is to gradually subtract the smooth area of the original input image. In order to verify our intuition, we need inspect the outputs of enhancement and compression units. For better visualizing the intermediary of the proposed model, we consider an operation T that can transform a 3D tensor A to a flattened 2D tensor defined over the spatial dimensions, which can be formulated as follows:
Specifically, in this work, we will consider the mean of the feature maps in channel dimension, which can be described by
where (using Matlab notation). The average feature map can roughly represent the situations of the whole feature maps. To explore the functions of enhancement unit and compression unit, we visualize the outputs of each enhancement unit and compression unit by utilizing above-mentioned method. As illustrated in Figure 5(a), from the first subpicture to the third subpicture, average feature maps gradually reduce the pixel values, especially in smooth areas. According to Figure 5(a), we can easily see that the first subfigure holds larger pixel values but has rough outline of the butterfly. The second and the third sub-figures show that the later enhancement units continue decreasing the pixel values to obtain the features with a relatively clear contour profile. In addition, the last subfig-ure obviously surpasses the former figures, which brings the better inputs for the sequential compression unit that directly connects to RBlock. In summary, the function of the enhancement unit mainly enhances the outline areas of input LR image. As for the effect of compression unit, comparing Figure 5(a) with Figure 5(b), we find that the pixel values of features are mapped into a smaller range through the compression unit. From the second subfigure in Figure 5(b) and the third subfigure in Figure 5(a), we can see some regions of the average feature map of compression unit are enhanced by the following enhancement unit. This indicates that the process of the first three stacked blocks is to reduce the pixel value as a whole, while the last block greatly enhances the contrast between the contour and the
Table 4. Comparison the running time (sec) on the 4 benchmark datasets with scale factors . Red color indicates the fastest algorithm and blue color indicates the second fastest method. Our IDN achieves the best time performance.
Figure 6. The “barbara” image from the Set14 dataset with an upscaling factor 4.
smooth areas.
The RBlock, a transposed convolution layer, assembles the output of the final DBlock to generate the residual image. The bias term of this transposed convolution can automatically adjust the central value of the residual image data distribution to approach the ground-truth.
4.4. Comparisons with state-of-the-arts
We compare the proposed method with other SR methods, including bicubic, SRCNN [3, 4], VDSR [12], DRCN [13], LapSRN [15], DRRN [22] and MemNet [23]. Table 2 shows the average peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) values on four benchmark datasets. The proposed method performs favorably against state-of-the-art results on most datasets. In addition, we measure all methods with information fidelity criterion (IFC) metric, which assesses the image quality based on natural scene statistics and correlates well with human perception of image super-resolution. Table 3 shows the proposed method achieves the best performance and outperforms MemNet [23] by a considerable margin.
Figure 6, 7 and 8 show visual comparisions. The “barbara” image has serious artifacts in the read box due to the loss of high frequency information, which can be seen from the result of bicubic interpolation. Only the proposed method recovers roughly the outline of several stacked books as shown in Figure 6. From Figure 7, we can obviously see that the proposed method gains clearer contour without serious artifacts while other methods have different degrees of the fake information. In Figure 8, the building structure on image “img085” of Urban100 dataset is relatively clear in the proposed method.
From Table 2, the performance of the proposed IDN is lower than that of MemNet in Urban100 dataset and ,
Figure 7. The “8023” image from the BSD100 dataset with an upscaling factor 4.
Figure 8. The “img085” image from the Urban100 dataset with an upscaling factor 4.
scale factors, while our IDN can achieve slightly better performance in other benchmark datasets. The main reason is that MemNet takes an interpolated LR image as its input so that more information is fed into the network and the process of the SR only needs to correct the interpolated image. The algorithms that take the original LR image as input demand predicting more pixels from scratch, especially in larger images and larger magnification factors.
As for inference time, we use the public codes of the compared algorithms to evaluate the runtime on the machine with 4.2GHz Intel i7 CPU (32G RAM) and Nvidia TITAN X (Pascal) GPU (12G memory). Since we note that official implementations of MemNet and DRRN have the condition of out of the GPU memory when testing the images in BSD100 and Urban100 datasets, we divide 100 images into several parts and evaluate on these parts and then collect them for these two datasets. Table 4 shows the average execution time on four benchmark datasets. It is noteworthy that the proposed IDN is approximately 500 times faster than MemNet [23] with magnification on the Ur-ban100 dataset.
In this paper, we propose a novel network that employs distillation blocks to gradually extract abundant and effi-cient features for the reconstruction of HR images. The proposed approach achieves competitive results on four benchmark datasets in terms of PSNR, SSIM and IFC. Meanwhile the inference time substantially exceeds the state-of-the-art methods such as DRRN [22] and MemNet [23]. This compact network will be more widely applicable in practice. In the future, this approach of image super-resolution will be explored to facilitate other image restoration problems such
as denosing and compression artifacts reduction.
[1] M. Bevilacqua, A. Roumy, C. Guillemot, and M. L. Alberi- Morel. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In BMVC, 2012. 5
[2] H. Chang, D.-Y. Yeung, and Y. Xiong. Super-resolution through neighbor embedding. In CVPR, 2004. 2
[3] C. Dong, C. C. Loy, K. He, and X. Tang. Learning a deep convolutional network for image super-resolution. In ECCV, pages 184–199, 2014. 2, 7
[4] C. Dong, C. C. Loy, K. He, and X. Tang. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2):295–307, 2016. 2, 7
[5] C. Dong, C. C. Loy, and X. Tang. Accelerating the super- resolution convolutional neural network. In ECCV, pages 391–407, 2016. 3
[6] C. Francois. Xception: Deep learning with depthwise sepa- rable convolutions. In CVPR, pages 1251–1258, 2017. 5
[7] W. T. Freeman, T. R. Jones, and E. C. Pasztor. Example- based super-resolution. IEEE Computer Graphics and Applications, 22(2):56–65, 2002. 2
[8] K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In ICCV, pages 1026–1034, 2015. 5
[9] J. Hu, L. Shen, and G. Sun. Squeeze-and-excitation networks. In arXiv:1709.01507, 2017. 2
[10] J.-B. Huang, A. Singh, and N. Ahuja. Single image super- resolution from transformed self-exemplars. In CVPR, pages 5197–5206, 2015. 2, 5
[11] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Gr- ishick, S. Guadarrama, and T. Darrell. Caffe: convolutional architecture for fast feature embedding. In ACMMM, pages 675–678, 2014. 5
[12] J. Kim, J. K. Lee, and K. M. Lee. Accurate image super- resolution using very deep convolutional networks. In CVPR, pages 1646–1654, 2016. 1, 2, 3, 4, 6, 7
[13] J. Kim, J. K. Lee, and K. M. Lee. Deeply-recursive convolu- tional network for image super-resolution. In CVPR, pages 1637–1645, 2016. 1, 2, 3, 6, 7
[14] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2014. 5
[15] W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. In CVPR, pages 624–632, 2017. 2, 3, 4, 6, 7
[16] B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee. Enhanced deep residual networks for single image super-resolution. In CVPR Workshop, pages 136–144, 2017. 3
[17] X.-J. Mao, C. Shen, and Y.-B. Yang. Image restoration us- ing very deep convolutional encoder-decoder networks with symmetric skip connections. In NIPS, 2016. 1, 3
[18] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In CVPR, pages 416–423, 2001. 4, 5
[19] M. S. M. Sajjadi, B. Scholkopf, and M. Hirsch. Enhancenet: Single image super-resolution through automated texture synthesis. In ICCV, pages 4491–4500, 2017. 3
[20] S. Schulter, C. Leistner, and H. Bischof. Fast and accurate image upscaling with super-resolution forests. In CVPR, pages 3791–3799, 2015. 2
[21] W. Shi, J. Caballero, F. Husz´ar, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR, pages 1874–1883, 2016. 2
[22] Y. Tai, J. Yang, and X. Liu. Image super-resolution via deep recursive residual network. In CVPR, pages 3147–3155, 2017. 1, 2, 3, 4, 6, 7, 8
[23] Y. Tai, J. Yang, X. Liu, and C. Xu. Memnet: A persistent memory network for image restoration. In ICCV, pages 3147–3155, 2017. 2, 3, 4, 6, 7, 8
[24] S. Xie, R. Girshick, P. Dollar, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. In CVPR, pages 1492–1500, 2017. 5
[25] J. Yang, J. Wright, T. Huang, and Y. Ma. Image super-resolution as sparse representation of raw image patches. In CVPR, 2008. 2
[26] J. Yang, J. Wright, T. S. Huang, and Y. Ma. Image super- resolution via sparse representation. IEEE Transactions on Image Processing, 19(11):2861–2873, 2010. 2, 4
[27] R. Zeyde, M. Elad, and M. Protter. On single image scale-up using sparse-representations. In Curves and Surfaces, pages 711–730, 2010. 5