Implementation of the VBM3D Video Denoising Method and Some Variants

2020·Arxiv

Abstract

Abstract

VBM3D is an extension to video of the well known image denoising algorithm BM3D, which takes advantage of the sparse representation of stacks of similar patches in a transform domain. The extension is rather straightforward: the similar 2D patches are taken from a spatio-temporal neighborhood which includes neighboring frames. In spite of its simplicity, the algorithm offers a good trade-off between denoising performance and computational complexity. In this work we revisit this method, providing an open-source C++ implementation reproducing the results. A detailed description is given and the choice of parameters is thoroughly discussed. Furthermore, we discuss several extensions of the original algorithm: (1) a multi-scale implementation, (2) the use of 3D patches, (3) the use of optical flow to guide the patch search. These extensions allow to obtain results which are competitive with even the most recent state of the art.

I. INTRODUCTION

VBM3D was proposed by [7] as an adaptation to video denoising of BM3D, the successful image denoising algorithm [9]. The algorithme is designed for additive white Gaussian noise with zero mean and standard deviation , i.e.

where is a position in the video domain (a pixel), v is the noisy video and u the unknown clean video. The denoising principle of VBM3D (and BM3D) is based on the redundancy of similar patches. Groups of similar 2D patches are assembled in a 3D stack of patches. A separable 3D transform is applied to this stack. The stack is denoised by applying a shrinkage operator to the coefficients in the transformed domain. The algorithm follows four basic steps:

1) Search for similar patches in the sequence, grouping them in a 3D stacks, 2) Apply a 3D linear domain transform to the 3D block, 3) Shrink the transformed coefficients, 4) Apply the inverse transform, 5) Aggregate the resulting patches in the video. The underlying idea here is that, due to the high redundancy of a stack of similar patches, the energy will be concentrated in a few coefficients of the transform, while the noise is spread evenly among all coefficients. This allows to jointly denoise the patches of each stack. The reconstruction of the estimated video is obtained by aggregating for each pixel the estimated patches that contain it. This principle is applied twice. The first time the patches are denoised using a hard threshold in the transformed domain. In the second iteration a Wiener filter is used, with Wiener coefficients computed using the output of the first step as oracle. There exist other adaptations to video and 3D images of this framework: BM4D [15] and VBM4D [14]. These methods stack similar 3D spatio-temporal patches in a 4D stack. The main difference between them is that VBM4D uses motion compensated patches whereas BM4D aims at denoising volumetric images (such as those appearing in medical imaging), where the 3rd dimension is just another spatial dimension. In [1] however, it was shown that even non-motion compensated 3D patches provide a very good denoising performance. In this article, we revisit the VBM3D method, providing an open source implementation and we discuss some variants introduced in [1], such as 3D patches (as in BM4D [15]), optical flow guided patch search, and a multiscale implementation. In the next section, the algorithm itself is reviewed. In the following sections we consider three possible extensions: multiscale, spatio-temporal patches and motion compensated patch search using an optical flow. Finally the performance of the algorithm is compared against other state of the art algorithms. a) Notation : The noisy input video denoted v consists of f + 1 frames written for i between 0 and f. The clean unknown video is denoted by u. A patch is a small rectangular piece of the video, for example of size pixels width and height, 3 pixels in the temporal dimension). Patches are represented by bold lowercase letters, e.g. p. If we need to emphasize the location of the patch we write p(x), where x represents the location on the video of the top-left-front pixel of the patch. For the parameters of the method we will use the same notation as in [7]. The different parameters for the patch search are the number N of patches to return, the number of temporal frames that are being searched, the search region for the

Both authors are with CMLA, CNRS, ENS Paris-Saclay, Université Paris-Saclay. e-mail: thibaud.ehret@cmla.ens-cachan.fr Work partly financed by IDEX Paris-Saclay IDI 2016, ANR-11-IDEX-0003-02, Office of Naval research grant N00014-17-1-2552, DGA Astrid project «filmer la Terre» nANR-17-ASTR-0013-01, MENRT.

reference frame , the search region for the other frames , the maximum number of patches to compute for each local search, a correcting factor d and a maximum distance threshold . The threshold parameter in the first step is . We also write for the element-wise product. Similar parameters will be used for the hard thresholding first step and for the Wiener filtering second step. Because they can have different values for the different steps, they’ll be differentiated using the subscript hard and wien respectively.

II. VBM3D : THE ALGORITHM

VBM3D performs two steps, as its image counterpart BM3D. The first hard-thresholding step computes a a basic estimate which is refined in the Wiener filtering step, yielding the final estimate. Figure 1 represents the global structure of the algorithm. The pseudocode of the core of the algorithm is presented in Algorithm 5.

Fig. 1. Scheme of the core of the VBM3D algorithm.

A. The patch search

The groups of similar patches are built by selecting a reference patch and searching around it for its nearest neighbors. This patch search is the main difference between BM3D and VBM3D. The image version of the algorithm searches a square 2D window centered at the reference patch. While the same could be done in a space-time volume for videos, Dabov et al. [7] propose a predictive search heuristic to reduce the size of the search window. The idea behind the proposed search is to track patches in the video. Let p and q be two patches at positions (x, y, t) and respectively. The distance between p and q used to find the nearest neighbors for a given regularizing factor d is

This distance regularizes the patch trajectories by favoring non moving patches. Suppose that the reference patch is p located at (x, y, t). The goal of the proposed search is to find (at most) N patches in a window of frames around frame t. The steps are the following:

1) Find the nearest neighbors to p in frame t in a square search region of size centered at the location of the reference patch (x, y, t). Let be the set of found patches.

2) Find the nearest neighbors to p in frames . The search region at is the union of square regions centered at positions of the candidates in , where . This is depicted in Figure 2.

3) Similarly, find nearest neighbors to p in frames . This time the search region is the union of squares centered at the positions of the patches in .

4) Remove the candidates with distance (1) larger than from

5) Keep the best N candidates from L. 6) Because the next steps of the algorithm (in particular the domain transforms) require a number of patches that is a power of 2, only the largest power of 2 smaller or equal to N is kept.

These steps are summarized in Algorithms 1 and 2.

Fig. 2. The patch search starts with a local spatial search in the current frame of size best candidates are then used in the following frame to predict where to search. This leads to local spatial searches (each centered at the candidates from the previous frame), of size practice,

B. Patch stack filtering: hard-threshold step

The first step processes the image by filtering stacks of similar patches and aggregating them in an output image. The reference patches for the stacks are all patches on a sub-grid of step . For each group there is first an estimation, followed by an aggregation. a) Estimation.: For a given reference patch p, we first find its similar patches L as described in Section II-A. These patches are stacked in a 3D volume P(p), of size . The first spatial slice of this stack is the reference patch, and the remaining ones are the similar patches in L ordered by their distance to p. A 3D domain transform is first applied to the group of patches followed by a thresholding of the resulting spectrum (excepting the DC component of every patch). The inverse 3D domain transform is then applied to the thresholded coefficients to obtain the estimation, i.e.

where HT is defined by

In practice the 3D transforms are chosen to be separable, consisting of a 2D spatial transform applied directly on the patches of P(p), typically a bi-orthogonal wavelet transform, followed by an 1D transform along the third dimension of the stack, typically a Haar transform. The pseudocode of the estimation for the first step is presented in Algorithm 3.

b) Aggregation.: An output pixel is estimated several times, since it belongs to several patches and each patch can be estimated multiple times (once for each group it belongs to). To compute the output frame these estimates are aggregated. For a pixel of the frame, this aggregation is performed as

where a patch p has a support of the size of the image and is zero everywhere except on the actual location of the patch. The weights used during the aggregation are computed in part during the estimation. The part computed during the estimation is where corresponds to the number of coefficients that have not been thresholded during the estimation of . Each coefficient is properly defined because is always positive as the DC component is never thresholded. The rest of the weight comes from a 2D Kaiser window K(p) of size (see Eq. (6)) located on the position of p applied to avoid boundary effects from the patch. The Kaiser window of size

input : A patch p center of the search region, a reference patch , the search region size s, the size of the patch k, the

output: L the list of the patches closest to in the region described by p and s and their distance

1 for each patch q of size in the spatial region of size centered on p do

2 if q is at the same spatial position than p then

3 Add to L

4 else

5 Add to L

6 Sort L according to the value of the distance

7 Keep the best elements in L

8 return L

of parameter is

Finally the (patch) weight is defined by . The pseudocode with the aggregation step can be found in Algorithm 5.

C. Patch stack filtering: Wiener filtering step

The second step of the algorithm uses the basic estimate computed during the first step for the patch search, and to compute the coefficients of a Wiener shrinkage of the transformed coefficients. The video is processed by building groups of patches around reference patches from a coarse subgrid with step .

patches extracted from the basic estimate for computing the patch distance. Two sets of patches are extracted: one from the noisy sequence and the other one from the basic estimate at the same locations. Once the sets of candidates, P(p) from the noisy sequence and for the basic sequence, have been computed, a Wiener filtering step is applied. The coefficients for the filtering are computed using but it’s P (p) which is used for the estimation. Just like the first step, a 3D domain transform is first applied to the group of patches before the application of the Wiener filter. The computation is done following Equation (7).

where WF on a frequency (with corresponding to the same frequency in the basic estimation) is defined by

In practice the 3D transforms are chosen as a 2D transform applied directly on the patches of P(p) and , typically a DCT, followed by an 1D transform along the third dimension of the group, typically a Haar transform. The pseudocode of the estimation for the first step is presented in Algorithm 4.

b) Aggregation.: Just as with the first step, the estimation gives multiple estimates per pixel. Therefore the different estimates are aggregated using the same principle as in Section II-B, defined by Equation (4). The only difference lies in the weights. The weights used during the aggregation are computed in part during the estimation. The part computed

during the estimation is

coefficients used for the filtering. The rest of the weight come from a 2D Kaiser window K(p) of size applied to avoid boundary effects from the patches. Finally the pixel weight is defined by . The pseudocode with the aggregation step can be found in Algorithm 5.

D. Reproducing the original VBM3D

We now compare the results obtained with our implementation to those obtained from the binaries released by the authors of the original VBM3D [7]1. Throughout this work we will use a test set of seven grayscale test sequences obtained from the Derf’s Test Media collection2. The original sequences are of higher resolution and in RGB. They have been downscaled and converted to grayscale.

Table I compares the PSNR obtained by our implementation with the original binaries. Our results below are those of the original implementation. The average gap starts at 0.27dB for and reduces to 0.18dB for . Station and sunflower are the sequences where the gap is larger, reaching 0.68dB for the later at .

A visual comparison of both results is shown in Figure 3, for noise . Both results are visually very similar, with the same type of artifacts. A closer inspection reveals that the result of the original VBM3D is slightly smoother, resulting in less noticeable DCT artifacts and on some textures with decreased contrast (see for instance the grass in the rightmost figure).

III. EXTENSIONS

A. Using optical flow to guide the search

The displacement vector can be sub-pixel, in which case the image A needs to be interpolated. We decided to use the TVL1 optical flow method [21], in particular the implementation provided in [20]. We compute the optical flow in a downscaled version of the video by a factor of 4, and scale it back to the original resolution. This reduces the running time and the impact of the noise while still having a reasonable precision, as shown in [11].

input : A video v, the noise variance , the number of similar patches to compute N, a number of temporal frames

1 Kaiser window of size and coefficient

2 Kaiser window of size and coefficient

3 for each p on the grid of step do

4 L, N, N, N, N, N, k, d, τ

5 ht_filtering

6 for do

7 aωK

8 wωK

9 for each pixel x do

10

11 for each p on the grid of step do

12 L, N, N, N, N, N, k, d, τ

13 patches from v at the same position than the one from

14 wiener_filtering

15 for do

16 aωK

17 wωK

18 for each pixel x do

19

TABLE I COMPARISON OF THE DENOISING QUALITY WITH THE BINARY PROGRAM PROVIDED BY DABOV et al. WITH [8]. RESULTS WERE BOTH COMPUTED USING THE NORMAL PROFILE ’NP’ PARAMETERS.

Fig. 3. Top: results of VBM3D [7], with the original authors’ implementation. Bottom: result obtained with our implementation. The noise level is The contrast has been linearly scaled for better visualization.

We add the optical flow to VBM3D as a guide, the same way it is done for VNLB [2]. The spatio-temporal search region is defined as two sequences of square windows of size (plus the window corresponding to the reference frame), whose centers follow the motion trajectory of the reference patch. The trajectory is estimated using the forward and backward optical flow. We use the center corresponding to the position of the reference patch propagated in previous, respectively following, frames using the backward, respectively forward, optical flow. The forward half of the trajectory passing through (x, y, t) is computed by integrating the forward optical flow (also indexed by time on top of the spatial position) as follows:

where and denote the round and floor operators. The backward half of the trajectory is defined analogously using the backward optical flow

This also means that only one center needs to be tracked, in contrast to the required by a regular VBM3D search. Parameters are kept the same as in the original algorithm. The guided version of the search is summarized in Algorithm 6.

We also tested setting the parameter d to zero for both steps since one can assume that it would be redundant with the regularization of the patch search offered by the optical flow. PSNR results are shown in Table II. Using the optical flow as a guide greatly improves the quality of the denoising. The non-zero d yields better results both with and without the optical flow guide, thus we decided to keep it in our final version.

Figure 4 shows results obtained with the different extensions that will be described in this section. The first column corresponds to VBM3D with the default parameters, and the 3rd column to the one using an optical flow guided search region (denoted “VBM3D OF”). Guiding the patch search allows to recover more details. This is because the tracking of patches is more robust to motion than the block-matching suggested for the original VBM3D and therefore provides better matches. Since the optical flow is used only as guide for the center of the search region, computing the optical flow from the noisy data is not a problem as it is not required to be very precise.

B. Spatio-temporal patches

The patch similarity is determined based on the (squared) Euclidean distance between patches. Due to the noise, the Euclidean distance follows a non-central distribution, with variance

where are the noisy versions of the patches , and m denotes the number of pixels in the patch. The noise in the distance can be reduced by considering larger patches. However, increasing the size of the patch also increases the distances between patches and reduces the likelihood of finding similar ones. The additional temporal dimension in a spatio-temporal patch allows to increase the number of pixels in the patch, without increasing its spatial size. Due to the high redundancy of the video in the temporal dimension, increasing the temporal size of the patch causes a much lower increase in the patch distances.

TABLE II COMPARISON OF THE DENOISING QUALITY WITH AND WITHOUT GUIDING WITH AN OPTICAL FLOW. GUIDING THE SEARCH LEADS TO MUCH BETTER RESULTS. IT IS ALSO BETTER IN GENERAL TO KEEP THE ADDITIONAL REGULARIZATION GIVEN BY

When the motion is known or can be estimated, then it is natural to consider motion compensated spatio-temporal patches (see for instance [14], [6]). Alternatively, rectangular spatio-temporal patches with no motion compensation have been also used [18], [2]. For more complex types of motion, using rectangular spatio-temporal patches will result in a larger variability in the set of nearest neighbors of a given patch, due to the fact that both the spatial texture and the motion pattern may vary. At least in principle, better results should be obtained using motion compensation. However, in practice, for higher levels of noise the bad quality of the estimated motion can undermine the final result.

The principle of BM3D has been applied to 3D patches with and without motion compensation. VBM4D, introduced in [14], uses motion compensated spatio-temporal patches for video denoising (the “V” stands for video). The motion is estimated using block matching. BM4D uses cubic patches without motion compensation (of size or ), aiming at filtering volumetric images [15]. In [15] the authors compare the performance of both VBM4D and BM4D on video denoising concluding that VBM4D was the best video filtering strategy.

However, in [2], [1] it is shown that rectangular spatio-temporal patches with a temporal size of only two frames improve the denoising quality and still provide higher temporal consistency than a 2D patch. Based on those results, in this work we evaluate rectangular spatio-temporal patches of size in the first step and in the second (i.e we keep the spatial patch size).

In Table III we compare the quantitative results obtained by using spatial and spatio-temporal patches (denoted by “ST” in the table). We also consider the effect of the optical-flow-guided patch search, indicated as “OF”.

The first four columns of Figure 4 show results with/out motion-compensated search and spatio-temporal patches. From a qualitative point of view, using spatio-temporal patches provides better temporal consistency. In addition, the patch distance is more reliable since the number of pixels in the patches is doubled. This help retrieve details and texture for regions with a simple motion (e.g. translational). For low noise levels, the effect of these 3d patches is mixed. While the results seem more consistent temporally, they are blurrier for sequences with complex motions. This explains the drop in PSNR observed for pedestrians, sunflower and touchdown between VBM3D and VBM3D ST for . As the noise level increases this detail loss is out-weighted by the increased robustness to noise of the patch matching.

When spatio-temporal patches are used in conjunction with the optical-flow-guided search, their positive impact is magnified. Although the patches are not themselves motion-compensated, having a motion-compensated search region helps find better matches, even in sequences with more complex motion patterns. The motion-compensated search region also improves the temporal consistency, although to a lesser extent than the spatio-temporal patches. The best result, both in terms of PSNR and temporal consistency, is obtained when both strategies are used together (VBM3D ST+OF).

TABLE III QUANTITATIVE DENOISING RESULTS (PSNR AND SSIM) FOR SEVEN GRAYSCALE TEST SEQUENCES OF SIZE Derf’s Test Media collection FOR SEVERAL VARIANTS OF VBM3D. WE HIGHLIGHTED THE BEST PERFORMANCE IN BLACK AND THE SECOND BEST IN BROWN.

C. Multiscale video denoising

Multiscale approaches have shown to both reduce the residual noise but also improve the visual quality of the result. Indeed, most denoising algorithms work by processing local regions of the image/video (a patch, a group of patches, the receptive field of a neuron, etc). As a result, these methods fail to remove the lower frequencies of the noise. This results in smooth bumps mostly noticeable in large non-textured areas. Multiscale approaches are able to reduce this artifact by applying the denoising algorithm at different scales.

There are two main approaches for multi-scale denoising in the literature. The first one consists in modifying the denoising algorithm to consider several scales of the image/video. See for instance the multiscale version of the EPLL method [22] proposed by Papyan and Elad [16]. Another approach proposed in [12], [17] considers the denoising algorithm as a black box, applying it as is at multiple scales. The result is then obtained by merging the results of each scale. This has the benefit that it can be applied to any denoising method, without any adaptation needed.

The multiscaler of [12] first creates a pyramid from the noisy input image by applying a downscaling operator D

At each scale, the denoising algorithm is applied and yields denoised images . These are then recomposed into a single multiscale output . The recomposition is recursive. The recursion is initialized at the coarsest level by defining , and then proceeds as follows:

where U is the upscaling operator, and is a low-pass filter with cutoff frequency parameterized by This equation substitutes the low frequencies of the single-scale result with the multiscale solution , computed using the coarser scales s + 1, ..., S. The recomposition parameter determines which low frequencies are substituted. In one extreme, the low-pass filter lets all frequencies pass, in which case the whole coarse solution is used. At the other end, filters out all frequencies: the solution at the coarser level is discarded, and the output of the recomposition is the single scale denoising . In [12] it is found that the optimum is to filter out some of the high frequencies of the coarser level . However, the exact amount depends on the denoiser.

To apply the multiscaler on a video, we apply it spatially, i.e. the temporal dimension is not downscaled. We first create a spatial pyramid of the entire video by creating a pyramid of each frame. We then denoise these videos, and recompose them by applying Eq. (12) to each frame. This is summarized in Algorithm 7.

The parameters of the multiscaler are the number of scales, the downsampling ratio and the recomposition parameter . We shall set the downscaling ratio to 2 (the default), and try different values for the number of scales and the recomposition factor. There are different possibilities for the down/upscaling operators and the low-pass filter.

DCT pyramid.The DCT multiscaler uses a DCT pyramid which guarantees white Gaussian noise at all scales. The downscaling is performed by computing the DCT transform of the image and keeping only the quadrant of the image corresponding to the lowest frequencies. The upscaling is done by zero-padding in the DCT domain, and the low-pass filtering zeroes out the highest frequencies in the DCT domain. In this case represents the ratio of frequencies that are left: corresponds to an all-pass filter, where as filters out all the image.

Fig. 5. Effect of the DCT multiscaler for VBM3D without extensions (square markers) and VBM3D ST+OF (round markers). Each plot shows the average PSNR over our seven test sequences obtained when varying the recomposition factor for different number of scales. From left to right, The multiscaler has a positive effect on the original VBM3D, but not on the improved VBM3D ST+OF.

We set a = 3. For the downscaling, to reduce aliasing, the image is downsampled using a scaled version of the kernel (as described in [19]).3 The low pass filter used is the Gaussian filter of width . When we obtain the single scale output, and if no frequencies from the coarser solutions are discarded. Figures 5 and 6 show plots of the average PSNR for our seven test sequences obtained with the two multiscalers varying the number of scales and the recomposition cutoff parameter . In each figure, we show the results of the multiscaler applied to the standard VBM3D, and to the one with statio-temporal patches and guided patch search (VBM3D ST+OF). Both multiscalers have a positive impact when applied to the standard VBM3D. The gain can be up to 0.3dB for noise 10, 0.5dB for noise 20 and 0.8dB for noise 40. However, when applied to the VBM3D ST+OF version of the algorithm, the multiscaler does not improve the result. In fact, for noise 10 and 20, the best PSNR is attained by the single scale algorithm and the PSNR deteriorates as the cutoff frequency of the low-pass filter is increased (i.e. as more frequency components from the coarse solution are used). The multiscaler achieves a better denoising of large objects with smooth textures by removing low-frequency noise left by the denoiser. This low-frequency noise is much stronger for the VBM3D denoiser than for the VBM3D ST+OF version. Hence, the improvement provided by the multiscaler is smaller for the latter. This also depends on the characteristics of the sequence. Frames with smooth objects occupying larger areas will benefit from the multiscaler. Yet, the multiscaler introduces artifacts penalizing the PSNR (particularly additional ringing for the DCT multiscaler). These artifacts are not temporally coherent and can therefore be quite noticeable. Based on these plots we chose to select the Lanczos3 multiscaler. We use 2 scales and a recomposition factor when applied to VBM3D ST+OF (i.e. VBM3D ST+OF+MS) and 3 scales with recomposition factor when applied to the standard VBM3D (VBM3D MS). Table IV shows the obtained PSNRs. The visual results of VBM3D ST+OF+MS are shown in Figure 4. The impact of the multiscaler can be noticed in the top row (results for pedestrian) as a reduction of the low-frequency noise in the smooth areas in the image. For the other sequences, since they are highly textured, the effect of the multiscaler is subtle. A careful examination reveals some texture loss.

IV. COMPARISON WITH THE STATE OF THE ART

In Table V, we compare the PSNR of different recent denoising methods with VBM3D as well as with versions of VBM3D modified using different combinations of the improvements suggested in Sections III-C, III-B, III-A. VBM3D combined with spatio-temporal patches and a patch-search guided with an optical flow gives very competitive results even compared to the latest state of the art, and especially for higher noises. When combined with the other improvements, it seems that multiscaling does not increase the quality of the denoising and can even degrade it in terms of PSNR. Visual examples are shown in Figures 4, where we compare results obtained with the original VBM3D with our implementation, using 3D patches and an optical flow to guide the search region.

Fig. 6. Effect of the Lanczos multiscaler for VBM3D without extensions (square markers) and VBM3D ST+OF (round markers). Each plot shows the average PSNR over our seven test sequences obtained when varying the recomposition factor for different number of scales. From left to right, The multiscaler has a positive effect on the original VBM3D, and on the improved VBM3D ST+OF for

TABLE IV QUANTITATIVE DENOISING RESULTS (PSNR AND SSIM) FOR SEVEN GRAYSCALE TEST SEQUENCES OF SIZE Derf’s Test Media collection FOR SEVERAL VARIANTS OF VBM3D. WE HIGHLIGHTED THE BEST PERFORMANCE IN BLACK AND THE SECOND BEST IN BROWN.

TABLE V QUANTITATIVE DENOISING RESULTS (PSNR AND SSIM) FOR SEVEN GRAYSCALE TEST SEQUENCES OF SIZE Derf’s Test Media collection ON SEVERAL STATE-OF-THE-ART VIDEO DENOISING ALGORITHMS. WE HIGHLIGHTED THE BEST PERFORMANCE IN BLACK AND THE SECOND BEST IN BROWN.

REFERENCES

[1] P. Arias, G. Facciolo, and J.-M. Morel. A comparison of patch-based models in video denoising. In 2018 IEEE 13th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP), pages 1–5, June 2018.

[2] P. Arias and J.-M. Morel. Video denoising via empirical bayesian estimation of space-time patches. Journal of Mathematical Imaging and Vision, 60(1):70–93, Jan 2018.

[3] P. Arias and J.-M. Morel. Kalman filtering of patches for frame recursive video denoising. In 2019 IEEE Conf. on Comp. Vision and Pattern Recognition (CVPR) Workshops (NTIRE), 2019.

[4] J. C. Brailean and A. K. Katsaggelos. Simultaneous recursive displacement estimation and restoration of noisy-blurred image sequences. IEEE Transactions on Image Processing, 4(9):1236–1251, 1995.

[5] A. Buades, B. Coll, and J. M. Morel. Denoising image sequences does not require motion estimation. In Proceedings of the IEEE Conference on Advanced Video and Signal Based Surveillance, pages 70–74, 2005.

[6] A. Buades, J.-L. Lisani, and M. Miladinovi´c. Patch-based video denoising with optical flow estimation. IEEE Transactions on Image Processing, 25(6):2573–2586, June 2016.

[7] K. Dabov, A. Foi, and K. Egiazarian. Video denoising by sparse 3D transform-domain collaborative filtering. In EUSIPCO, 2007.

[8] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian. Color image denoising via sparse 3d collaborative filtering with grouping constraint in luminance- chrominance space. In Proc. IEEE Int. Conf. Image Process., 2007.

[9] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian. Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Transactions on image processing, 2007.

[10] A. Davy, T. Ehret, G. Facciolo, J.-M. Morel, and P. Arias. Non-local video denoising by cnn. arXiv preprint arXiv:1811.12758, 2018.

[11] T. Ehret, J.-M. Morel, and P. Arias. Non-local kalman: A recursive video denoising algorithm. In 2018 25th IEEE International Conference on Image Processing (ICIP), pages 3204–3208. IEEE, 2018.

[12] G. Facciolo, N. Pierazzo, and J. Morel. Conservative scale recomposition for multiscale denoising (the devil is in the high frequency detail). SIAM Journal on Imaging Sciences, 10(3):1603–1626, 2017.

[13] C. Liu and W. T. Freeman. A high-quality video denoising algorithm based on reliable motion estimation. In ECCV, pages 706–719, 2010.

[14] M. Maggioni, G. Boracchi, A. Foi, and K. Egiazarian. Video denoising, deblocking, and enhancement through separable 4-D nonlocal spatiotemporal transforms. IEEE Transactions on Image Processing, 2012.

[15] M. Maggioni, V. Katkovnok, K. Egiazarian, and A. Foi. A nonlocal transform-domain filter for volumetric data denoising and reconstruction. IEEE Transactions on Image Processing, 22(1):119–133, 2013.

[16] V. Papyan and M. Elad. Multi-scale patch-based image restoration. IEEE Transactions on image processing, 25(1):249–261, 2016.

[17] N. Pierazzo, J.-M. Morel, and G. Facciolo. Multi-Scale DCT Denoising. Image Processing On Line, 7:288–308, 2017. https://doi.org/10.5201/ipol.2017. 201.

[18] M. Protter and M. Elad. Image sequence denoising via sparse and redundant representations. IEEE Transactions on Image Processing, 18(1):27–35, 2009.

[19] D. Schumacher. General filtered image rescaling. In D. KIRK, editor, Graphics Gems III (IBM Version), pages 8 – 16. Morgan Kaufmann, San Francisco, 1992.

[20] J. Sánchez Pérez, E. Meinhardt-Llopis, and G. Facciolo. TV-L1 Optical Flow Estimation. Image Processing On Line, 3:137–150, 2013. https: //doi.org/10.5201/ipol.2013.26.

[21] C. Zach, T. Pock, and H. Bischof. A duality based approach for realtime tv-l 1 optical flow. In Joint Pattern Recognition Symposium. Springer, 2007.

[22] D. Zoran and Y. Weiss. From learning models of natural image patches to whole image restoration. In Computer Vision (ICCV), 2011 IEEE International Conference on, pages 479–486, Nov 2011.

designed for accessibility and to further open science