MOPS-Net: A Matrix Optimization-driven Network forTask-Oriented 3D Point Cloud Downsampling

2020·Arxiv

Abstract

Abstract

This paper explores the problem of task-oriented downsampling over 3D point clouds, which aims to downsample a point cloud while maintaining the performance of subsequent applications applied to the downsampled sparse points as much as possible. Designing from the perspective of matrix optimization, we propose MOPS-Net, a novel interpretable deep learning-based method, which is fundamentally different from the existing deep learning-based methods due to its interpretable feature. The optimization problem is challenging due to its discrete and combinatorial nature. We tackle the challenges by relaxing the binary constraint of the variables, and formulate a constrained and differentiable matrix optimization problem. We then design a deep neural network to mimic the matrix optimization by exploring both the local and global structures of the input data. MOPS-Net can be end-to-end trained with a task network and is permutation-invariant, making it robust to the input. We also extend MOPS-Net such that a single network after one-time training is capable of handling arbitrary downsampling ratios. Extensive experimental results show that MOPS-Net can achieve favorable performance against state-of-the-art deep learning-based methods over various tasks, including classification, reconstruction, and registration. Besides, we validate the robustness of MOPS-Net on noisy data.

1 INTRODUCTION

WITH recent advances in three-dimensional (3D) sens-ing technology (e.g., LiDAR scanning devices), 3D point clouds can be easily obtained. Compared with other 3D representations such as multi-view images, voxel grids and polygonal meshes, point clouds are a raw 3D representation, containing only 3D samples which are located on the scanned surface. Powered by deep learning techniques, the performance of many point cloud applications, such as classification, segmentation and reconstruction, has been improved significantly in recent years. However, processing large-scale and/or dense 3D point clouds is still challenging due to the high cost of computation, storage, and communication load.

Point cloud downsampling is a popular and effective technique to reduce information redundancy, hereby improving the runtime performance of the downstream applications and saving storage space and transmission bandwidth. The classic downsampling approaches such as farthest point sampling (FPS) [1] and Poisson disk sampling (PDS) [2] iteratively generate uniformly distributed samples on the input shape, and thus they can preserve the geometry

Fig. 1. Illustrations of task-independent and task-oriented point cloud downsampling using classification as an example. (a) A deep learning based point cloud classifier is trained on dense point clouds. (b) Classic downsampling methods, such as FPS, generate sparse point clouds without considering the nature of the task, hereby may compromise the performance of the classifier in (a) significantly. (c) The classification-oriented downsampling method produces sparse point clouds that maintain the performance of the classifier in (a). To develop such a classification-oriented downsampling, we take both the geometry of the input shape and the performance of the classifier into account.

well. Such sampling methods, however, focus on reducing the geometry loss only and are completely independent of the downstream applications. Although voxelization is computationally efficient, it suffers from quantization errors. As a result, the downsampled point clouds yielded by the classic methods would degrade the performance of the subsequent applications severely.

An alternative way for downsampling is to generate samples that optimize the performance of a particular task, i.e., the resulting sparse point clouds will maintain the task performance as much as possible. Due to the task centric nature, we call it task-oriented downsampling. Moreover, an effective downsampling method should allow the user to freely specify the downsampling ratio to balance the task performance and computation efficiency.

As the deep learning technologies have proven effective in point cloud classification [3], [4], [5], [6], [7] and segmentation [8], [9], [10], [11], it is highly desired to combine downsampling methods with the deep neural networks. However, extending the existing network architectures to point cloud downsampling is non-trivial, since point selection process is discrete and non-differentiable. Recently, Dovrat et. al. pioneered a deep learning approach, called S-Net [12], that takes downsampling as a generative task and uses the extracted global features to generate samples. S-Net is flexible in that it can be combined with task-specific networks to produce an end-to-end network trained by a joint loss. Thanks to its task-oriented nature, S-Net outperforms FPS in various applications. Hereafter, Lang et al. [13] improved S-Net by introducing additional projection module to encourage the generated points closer to original point clouds, learning SampleNet. However, as S-Net and SampleNet solely relies on global features in its generative process, they can not utilize the point-wise high-dimensional local features, which limits the quality of synthesized points.

In this paper, we propose a novel deep learning approach, called MOPS-Net, for task-oriented point cloud downsampling. In contrast to the existing methods, we propose MOPS-Net from the matrix optimization perspective. Viewing downsampling as a selection process, we first formulate a discrete and combinatorial optimization problem. As it is difficult to solve the 0-1 integer program directly, we relax the integer constraint for each variable, in which a constrained and differentiable matrix optimization problem with an implicit objective function is formulated. We then design a deep neural network architecture to mimic the matrix optimization problem by exploring both the local and the global structures of the input data. With a task network, MOPS-Net can be end-to-end trained. MOPS-Net is permutation-invariant, making it robust to input data. Moreover, by restricting the invoking columns of the soft sampling matrix, we also extend MOPS-Net such that a single network with one-time training is capable of handling arbitrary downsampling ratios. Extensive results show that MOPS-Net achieves much better performance than state-of-the-art deep learning-based methods and traditional methods in various tasks, including point cloud classification, reconstruction, and registration. We also provide comprehensive ablation studies and analysis on model robustness and time efficiency.

The main contributions of this paper are summarized as follows:

• we explicitly formulate the problem of task-oriented point cloud downsampling from the matrix optimization perspective;

• we propose an interpretable deep learning-based method named MOPS-Net, which mimics the above formulation, making it fundamentally different from the existing deep learning-based methods;

• we extend MOPS-Net and propose FMOPS-Net, which is capable of handling arbitrary downsampling ratios after only one-time training; and

• we propose a new compact deep-learning framework for large-scale point cloud reconstruction, which enables the validation of the scalability of MOPS-Net on relatively large-scale point clouds; and

• we conduct various experiments and comprehensive ablation studies to demonstrate the advantages of our methods over state-of-the-art methods.

The rest of this paper is organized as follows. Section 2 reviews classic point cloud downsampling methods and recent deep learning techniques for 3D point clouds. In Section 3, we formulate task-oriented downsampling as a constrained and differentiable optimization problem, followed by an end-to-end task-oriented downsampling deep neural network which mimics the resulting optimization in Section 4. Section 5 presents extensive experimental results, comparisons with the state-of-the-art, as well as comprehensive robustness tests and ablation studies. Section 6 finally concludes this paper.

2 RELATED WORK

2.1 Traditional Downsampling Methods

The traditional methods, such as farthest point sampling (FPS) and Poisson disk sampling (PDS), generate samples in an iterative manner. Starting from a random sample, FPS repeatedly places the next sample point in the center of the least-sampled area. Using efficient geodesic distance computation tools (such as the fast marching method [14]), FPS generates m samples on a n-vertex mesh in O(n log n) time, where f(x) = O(x log x). FPS is easy to implement and becomes popular in designing neural networks that aggregate local features [3], [5], [15]. Poisson disk sampling produces samples that are tightly-packed, but no closer to each other than a specified minimum distance, resulting in a more uniform distribution than FPS. There are efficient implementations of PDS with linear time complexity in Euclidean spaces [16], [17]. However, it is expensive to generate Poisson disk samples on curved surfaces due to frequent computation of geodesic distances [2]. Voxelization is also a commonly used technique to downsample or resample point clouds, which quantizes point clouds into regular voxels in 3D space with predefined resolution. Compared with FPS and PDS, voxelization is more efficient due to its non-iterative manner. However, voxelization often suffers from quantization error and cannot yield results that are the exact subsets of the input. Moreover, the traditional approaches focus only on preserving the geometry of the input shape and they do not consider the downstream tasks at all.

2.2 Deep Learning for 3D Point Clouds

Due to the irregular and unordered nature of point clouds, the widely used convolutional neural networks (CNNs) on 2D images/videos [18], [19], [20] cannot be applied directly. PointNet, proposed by Qi et al. [4], maps a 3D point to a high dimensional space by point-wise multilayer perceptrons (MLPs) and aggregates global features by

Fig. 2. MOPS-Net is a matrix optimization driven deep learning method for task-oriented 3D point cloud downsampling. It first extracts high- dimensional features, which contain both global and local geometric information, from point coordinates (Sec. 4.2). It then utilizes the features to learn a differentiable sampling matrix, which is multiplied to the input dense point cloud to obtain the sampled points (Sec 4.3). Finally, it feeds the downsampled sparse points to a task network. The whole network is trained by jointly optimizing the task loss and the subset loss (Sec. 4.5).

a symmetric function, named max-pooling. As the first deep neural network that works for 3D raw points without projecting or parameterizing them to regular domains, PointNet quickly gained popularity and was successfully used as the fundamental feature extraction for point clouds. However, PointNet processes the points individually and does not consider the spatial relation among points. The follow-up works, such as PointNet++ [3], DGCNN [7] and PointCNN [5], improve PointNet by taking local geometry into account.

Inspired by the success of PointNet on classification, many other point cloud applications were studied in recent years, such as retrieval [21], [22], segmentation [8], [9], [10], [11], reconstruction [23], [24], [25], registration [26], [27], [28], [29], and object detection [30], [31], [32], [33], [34], just name a few. Although they have different problem settings, these networks can be combined with the point cloud down-sampling network and jointly trained. In this paper, we evaluate the performance of the proposed downsampling framework on classification, registration, and reconstruction.

Opposite to downsampling, point cloud upsampling [35], [36], [37], [38] has also been investigated recently. Upsampling can be treated as either a 3D version of image super-resolution, or the inverse process of downsampling. Despite the common word “sampling”, the two tasks are completely different. Upsampling, as a generative task, requires informative feature expansion and can be trained by ground truth dense point clouds. In contrast, point cloud downsampling is close to feature selection, where a differentiable end-to-end framework should be carefully designed. Moreover, due to lack of ground truth, the downsampled points should be learned to optimize a specific task loss.

2.3 Deep Learning-based Point Cloud Downsampling

It is an emerging topic, on which there are only a few works. Nezhadary et al. [39] proposed to use critical points invoked in max-pooling as sampled points. In order to improve classification accuracy, Yang [40] adopted a gumbel softmax layer to integrate high level features. Recently, Dovrat et al. [12] proposed a data-driven point cloud downsampling framework named S-Net. After the point-wise feature extraction by PointNet, a global feature was obtained by the max-pooling operation. Then 3D coordinates of fewer points were regressed by fully-connected layers. Followed by a pre-trained task network, S-Net can be trained end-to-end. Lang et al. [13] proposed SampleNet, which improves SNet by introducing a soft projection module that adopts an anealing schedule to encourage generated points to be close to original points. However, both S-Net and SampleNet regress coordinates from global features directly and do not consider spatial correlation among sampled points, which plays an important role in downsampling, since spatially close points have the tendency to be represented by the same downsampled point. In sharp contrast to S-Net and SampleNet, we propose a novel framework by exploring the local geometry of the input data from a perspective of matrix optimization.

3 PROBLEM FORMULATION

where indicates the task-tailored loss, whose explicit form will be discussed in Section 4.5. We rewrite Eq. (1) in a more explicit manner

where and Q = are the matrix representations of P and Q, respectively, constructed by stacking each point as a column in an unodered manner1; is the (i, j)-th entry of is the column vector with all elements equal to 1; and refers to the identity matrix of size . The constraints force S to be an ideal sampling matrix, i.e., S only contains m columns of a permutation matrix of size . More precisely, the orthogonal constraint is used to avoid repeated columns (indicating that an identical point is selected multiple times) in S.

The challenge for solving Eq. (2) comes from the discrete and binary characteristics of matrix S. To tackle this challenge, we relax the binary constraints Eq. (2) in a soft and continuous manner, i.e., the elements of S are continuous, ranging between 0 and 1. The relaxed variables indicate the probabilities of the corresponding points that will be sampled. After this relaxation, the resulting points in Q may not be the subset of P. To mitigate this effect for a meaningful sampling process, we further introduce a metric to quantitatively measure the distant between two point clouds, and minimizing will promote Q to be a subset of P as much as possible. We will explain the explicit form of in Section 4.5.

We finally express the relaxed and continuous optimization problem for task-oriented point cloud downsampling as

where is the Frobenious norm of a matrix, is a threshold, and is the penalty parameter to balance the two terms.

Remarks. Relaxing the binary matrix S produces a point cloud which is not a subset of the input . This can be explained from the perspective of geometry processing. When presenting an object using two point clouds of different resolutions, the one with fewer points is generally not fully overlapping with the larger one since we have to re-distribute the points in order to preserve the geometry. Although , the objective function penalizes the points that are away from the input shape. Therefore, the points of Q are either on or close to the underlying object surface. If the subsequent task requires , we can assign each point of Q the closest point in the input data. In Section 5, we will quantitatively analyze the effect of these postprocessing operations.

4 PROPOSED METHOD

This section presents MOPS-Net, a novel end-to-end deep neural network that mimics the formulated optimization problem in Eq. (3) for task-oriented point cloud downsampling.

4.1 Overview

Figure 2 illustrates the flowchart of MOPS-Net. Given a point cloud P and a pre-trained task network, MOPS-Net uses a feature extraction module to encode each point with high dimensional and informative features by exploring both the local and the global structures of P (Section 4.2).

Fig. 3. The flowchart of differentiable sampling module.

Based on the high dimensional features, MOPS-Net estimates a differentiable sampling matrix S under the guidance of the constraints in Eq. (3). Multiplying the learned sampling matrix to original dense point clouds, we can obtain the sampled points. Together with a fixed task network, MOPS-Net can be end-to-end trained with a joint loss (Section 4.5). Such a joint loss simultaneously penalizes the degradation of task performance and regularizes the distribution of sampled points, which is consistent with the objective function in Eq. (3). We show that MOPS-Net is flexible and it can be extended so that a single network with one-time training is capable of handling arbitrary downsampling ratio (Section 4.6). Last but not the least, we prove that MOPS-Net is permutation-invariant, which is a highly desired feature for point cloud applications.

Remarks. The proposed MOPS-Net is fundamentally different from the existing deep learning framework S-Net [12] and SampleNet [13]. They formulate the sampling process as a point generation problem from global features, while MOPS-Net is designed from matrix optimization perspective to utilize the informative local (or point-wise) features. Experimental results demonstrate the advantages of MOPSNet over S-Net and SampleNet on point cloud classification, reconstruction and registration. See Section 5.

4.2 Feature Extraction

Given a point cloud , we extract d- dimensional point-wise features, denoted by . Let denote the ma- trix form of pointwise features. We utilize PointNet [4], a basic feature extraction backbone over 3D point clouds, to extract pointwise local features via a shared MLP, and a permutation-invariant global feature can be derived by applying the max-pooling operation to the resulting point-wise features along the spatial dimension. extract permutation-invariant features. We finally concatenate the global feature to each pointwise local feature to form F.

It is also worth noting that other advanced feature extraction techniques, such as PointNet++ [3], DGCNN [7], and KPConv [41] , could be adopted to further boost the performance of our method. , Here we adopt the basic PointNet for fair comparisons with S-Net and SampleNet.

4.3 Learning Differentiable Sampling Matrix

As analyzed in Section 3, an ideal sampling matrix, which is a submatrix of a permutation matrix, is discrete and non-differentiable, making it challenging to optimize. Accordingly, such a non-differentiable sampling matrix cannot be implemented in a deep neural network. Fortunately, the relaxation on the sampling matrix in Eq. (3) leads to a

where is the i-th row of S. In our experiments, is realized by a 4-layer MLP of size [512, 256, 128, m]. To satisfy the constraints on the sampling matrix in Eq. (3), we further apply the softmax with temperature annealing operation on each element of S:

where and is the (i, j)-th entry of the final sampling matrix S, and the value of the temperature gradually decreases from 1 to during training, and it is fixed to during inference. Such an operation ensures to be non-negative and encourages each column of S to be dominated by a single element, especially when is small. We experimentally found that such an operation is able to realize the constraints in Eq. (3) well. As visualized in Figure 4, the learned sampling matrix S is extremely sparse and close to a sample matrix. Besides, is also close to an identity matrix, although we do not explicitly minimize this constraint.

(a) (b) Fig. 4. Visualization of (a) learned sampling matrix m = 64. The two subfigures share the same colorbar.

4.4 Efficient Regression of the Sampled Set

Having obtained the sampling matrix S, we can naturally deduce the corresponding downsampled point set Q by

Analytically, the downsampling procedure described in Eq. (6) requires explicitly storing the sampling matrix and performing dot-product with P in O(nm) time complexity, which seems to be a major computational bottleneck faced with large-scale point clouds. Fortunately,

TABLE 1 Statistic of the percentage of non-zero elements in learned sampling matrix S (n = 1024). r is the threshold.

benefiting from the structural sparsity of S, we can perform the matrix multiplication operation in a much more efficient manner during inference.

More specifically, driven by the temperature annealing operation in Eq. (5), the learned sampling matrix S highly approximates an ideal sampling matrix, i.e., a columntruncated binary permutation matrix with only m non-zero entries, and thus most entries of S are extremely small (see Figure 4) and make negligible contributions to the actual subset selection. In practice, we design a deterministic matrix simplification rule by setting all entries of S smaller than an appropriate threshold to zero, after which we can expect that the ratio of non-zero entries should be as small as . Particularly, with the threshold set to 0.01 in our implementation, we counted the percentage of non-zero elements in S. As listed in Table 1, we can observe that the number of non-zero elements of the quantized S is about cm, where c is around 4. In our implementation, we adopt the Coordinate format (COO) [42] to store the highly sparse matrix S, which has O(cm) memory cost, where . Therefore, performing dot-product on only non-zero elements has time complexity O(cm).

As relaxation cannot guarantee , we can adopt an optional post-matching operation that maps each point of Q to its nearest point in P to obtain the downsampled subset. If the number of points contained in the matched point set is smaller than the specified value, we adopt FPS to complete it, i.e., points with farthest distances to the matched point set are iteratively selected from the original dense point clouds, until the target number is achieved. In Section 5, we will demonstrate the performance of the three cases.

In addition to differentiability, our design of the sampling matrix learning is also permutation-invariant. That is, the sampling result is independent of the point order of the input data. Proposition 1. MOPS-Net is permutation-invariant.

Proof. Let be an arbitrary permutation matrix (), and be the permutated input. Denote by and the corresponding high-dimensional point-wise features and sampled points when feeding into MOPS-Net, respectively.

Notice that the non-linear function extracts point-wise features with shared MLPs. Moreover, the global feature is aggregated via max-pooling, which is a symmetric function2, hence

Similarly, we obtain due to the permutation-invariant property of softmax, implying

TABLE 2 Comparisons of the classification accuracy by different downsampling methods. The larger, the better.

512 84.76 73.82 86.06 81.69 71.43 85.66 57.82 57.82 86.63 86.67 85.25 86.75 86.18 85.07 86.35

256 76.94 73.50 83.06 82.94 72.08 82.78 83.23 83.02 84.48 86.63 85.53 86.10 86.51 85.17 86.14

128 62.32 68.15 72.85 83.31 72.24 73.18 84.04 83.14 82.58 86.06 84.64 85.29 86.02 84.4 84.89

64 36.75 58.31 56.69 78.81 66.00 63.82 82.21 80.19 79.78 85.25 83.95 84.00 83.39 81.44 81.08

32 15.07 20.02 34.08 78.16 59.89 60.05 75.45 72.89 72.49 84.28 79.01 79.74 76.58 71.76 70.42

16 6.36 13.94 20.22 68.56 43.60 43.44 54.42 50.04 50.00 81.40 64.99 64.83 67.22 56.77 55.71 8 4.70 3.85 11.02 45.99 20.50 20.5 29.58 25.53 25.57 52.39 32.62 32.62 40.96 32.62 32.58

indicating that the sampled point set from is identical to those from P, which completes the proof.

4.5 Joint Training Loss

As analyzed in the objective function (3), two types of losses are needed to train MOPS-Net, i.e., the task loss and the subset loss . Specifically, aims to promote the network to learn downsampled point clouds that are able to maintain the high performance for a specific task. Let be the network for a typical task, which was trained with the original dense point cloud data, and we have

where is the corresponding ground-truth data for Q. Specifically, will be the class label and the input point cloud when the task is classification and reconstruction, respectively.

The subset loss regularizes the network to learn downsampled point clouds that are close to subsets of inputs, which is expressed as

Therefore, the total loss for end-to-end training of MOPS-Net is written as

where balances the two terms. Figure 16 shows the effect of the value of on performance.

4.6 Flexible MOPS-Net for Arbitrary Ratios

In the previous sections, we construct MOPS-Net with a predefined sample size m, and a different network has to be trained for each m, which is tedious and unpractical for real-world applications. To solve this issue, we extend MOPS-Net and propose flexible MOPS-Net (FMOPS-Net), which is a single network that can achieve 3D point cloud downsampling with arbitrary sampling ratios after only one-time training.

Specifically, we consider learning a relatively large matrix with the same network architecture as MOPS-Net. Given an arbitrary sample size , we select the m left-most columns of to form the sampling matrix , producing a point cloud with m points according to Eq. (6). Such a manner is equivalent to indirectly sorting the points of P according to their importance in a downsampled point cloud. To enable flexibility,

Fig. 5. Quantitative comparisons of classification by different down- sampling methods. Note that FMOPS-Net is applicable for arbitrary downsampled size.

we train MOPS-Net by randomly picking the downsampled number and minimizing at each iteration. Note that the computational complexity and the memory consumption during the inference phase are identical to Section 4.4 by replacing m with .

5 EXPERIMENTS

We validated the effectiveness of MOPS-Net and FMOPSNet on three typical tasks, i.e., point cloud classification, reconstruction, and registration. The task networks and experiment settings will be discussed in each section in detail. We used three widely used traditional downsampling methods, i.e., random sampling (RS), voxelization (Voxel) and farthest point sampling (FPS), as baselines. We also compared with S-Net [12] and SampleNet [13], which are state-of-the-art deep learning-based task-oriented downsampling methods. Note that for S-Net, SampleNet, and MOPS-Net, a network was trained for a downsampling size. For deep learning-based methods, we examined the performance of three types of downsampled sets, i.e., (1) Generated (G) sets: the point sets are generated directly by deep learning-based methods; (2) Matched (M) sets: the directly generated sets are post-processed via the matching operation, making the point sets be subsets of input ones; and (3) Completed (C) sets: the matched sets are further completed via FPS if their numbers of points are less than the specified value. In the following visual results, we visualized the Generated (G),

Fig. 6. Visual comparisons of the generated point sets by the three deep learning-based task-oriented downsampling methods with m = 64 over three classes: (a) Bottles (b) Chairs (c) Lamps. Prominent regions within each class are highlighted in red.

Fig. 7. Visualization of sampled point clouds by different methods for m = 256, 128, and 32. (a1) generated sets by S-Net; (a2) matched and completed sets by S-Net; (b1) generated sets by SampleNet; (b2) matched and completed sets by SampleNet; (c1) generated sets by MOPS-Net; (c2) matched and completed sets by MOPS-Net.

Matched (M), and Completed (C) sets with blue, red, and orange colors, respectively, to distinguish them.

5.1 Classification-oriented Downsampling

5.1.1 Classification of small-scale point clouds

Following S-Net [12] and SampleNet [13], we used the pre-trained PointNet vanilla [4] performing on ModelNet40 [43]

as the classification task network. Note that the pre-trained PointNet vanilla trained on point clouds with 1024 points achieves 87.1% overall accuracy when classifying point clouds with 1024 points each to 40 categories. The task loss refers to the cross entropy between the predicted and ground-truth labels. During training of our MOPS-Net, we set and , and we initialized the learning rate to and exponentially decreased it to within 250 epochs. Quantitative comparisons. From Table 2, we can observe that task-oriented downsampling methods, including S-Net, SampleNet and MOPS-Net, achieve much better performance than task-independent methods, including random sampling (RS), voxelization, and FPS. Note that the accuracy of S-Net drops significantly from generated sets to the matched subsets because the generative-based S-Net fails to obey the subset constraint. By replacing the repeated points in matched subsets by FPS points, the accuracy of completed subsets by S-Net can be improved, especially for relatively large downsampled sizes m =256 and 512. SampleNet improves S-Net by projecting the generated sets to nearest neighbors in original point clouds, and thus can minimize the performance gap between the generated set and matched subset. However, because of the additional restriction for projection, the accuracy of generated points by SampleNet is inferior to that by S-Net for relatively small downsampled sizes m = 8, 16, 32. Moreover, SampleNet fails to generate meaningful points for m = 512, and it mainly relies on additional FPS postprocessing to obtain the completed set which can result in comparable performance. The proposed MOPS-Net consistently achieves the best performance over all cases, which is credited to real sampling process-like modeling of MOPS-Net. The flexibility of FMOPS-Net is achieved at the cost of slight degradation of the accuracy of MOPS-Net; however, FMOPS-Net still outperforms S-Net and SampleNet under almost all cases. Figure 5 further demonstrates the flexibility and advantage of FMOPS-Net by showing the accuracy of more downsampling sizes. Visual comparisons. Figure 6 visually illustrates sampled point clouds (m = 64) by the three deep learning-based task-oriented downsampling methods, i.e. S-Net, SampleNet, and MOPS-Net, on three classes. From Figure 6, it can be seen that the two generative-based S-Net and SampleNet tend to generate points close to the centers of shapes and fail to capture prominent regions. By contrast, our MOPS-Net can successfully select points near the contours of shapes and focus on prominant regions. Besides, for different point clouds of a typical class, MOPS-Net focus on selecting points corresponding to identical semantics, e.g. the bottleneck of bottles, the legs of chairs, and the lampshade of lamps. These observations also explain why the downsampled point clouds by MOPS-Net can be classified with higher accuracy than S-Net and SampleNet. Besides, Figure 7 visualizes the generated, matched, and

completed sets by S-Net, SampleNet, and MOPS-Net under various m. where it can be seen that S-Net fails to directly generate downsampled points close to the input points once the shape is complex, and SampleNet tends to directly generate points clustered around the shape center and omit the footrest of the stool, which is assumed to be important

TABLE 3 Comparisons of the classification accuracy on downsampling large-scale point clouds. The larger, the better.

n = 10, 000 n = 100, 000 m RS FPS MOPS-Net RS FPS MOPS-Net

for shape identification. By contrast, our MOPS-Net can consistently capture these discriminative regions for any downsampled size.

5.1.2 Classification of large-scale point clouds

To demonstrate the potential of MOPS-Net for processing large-scale point clouds, we further applied MOPS-Net for downsampling point clouds with 10,000 and 100,000 points each. We first pre-trained a PointNet-vanilla classifier on ModelNet40 with 100,000 points each model. The other settings, including the optimized loss function, training epoch and the training strategies, were identical to the experiments in Section 5.1.1. The pre-trained classifier can achieve 90.11% overall accuracy. By fixing the pre-trained classifier, we then trained MOPS-Net for downsampling point clouds with 10,000 points each. The experiment settings were kept identical to our previous experiments on small-scale point clouds. As MOPS-Net is built upon point-wise MLPs and the Softmax operator, which are independent of the number of input points, we can directly apply MOPS-Net trained on 10,000 points for downsampling larger-scale point clouds with 100,000 points each.

Table 3 lists the classification accuracy of MOPS-Net for downsampling 10,000 points and 100,000 points. The high classification accuracy demonstrates the ability for the proposed MOPS-Net on downsampling large-scale point clouds. In particular, MOPS-Net trained on point clouds with 10,000 points each can be successfully extended to downsample larger-scale point clouds with 100,000 points each, without any modification or fine-tune, which also demonstrates its flexibility.

5.2 Reconstruction-oriented Downsampling

5.2.1 Reconstruction of small-scale point clouds

In this scenario, we followed the settings of S-Net and SampleNet to evaluate our method. The task network was achieved by a pre-trained MLP-based reconstruction network [23], where a 3-layer MLP of size is utilized to reconstruct point clouds with 1024 points from a 128-dimensional global feature. We obtained the 128-dimensional global features of downsampled point clouds by applying max-pooling on the pointwise features extracted by a 5-layer MLP of size [64, 128, 128, 256, 128]. A single class of ShapeNetCore [44] was used for training and testing. The task loss was set as the combination of the Chamfer distance (CD) and earth-mover distance (EMD). During training of our MOPS-Net, we set and initialized the learning rate to and exponentially decreased it to within 250 epochs. We quantitatively measured the reconstruction performance of different downsampling methods using the normalized

Fig. 8. Quantitative comparisons of the distortion of reconstructed dense point clouds from the corresponding downsampled ones by different downsampling methods. Reconstruction from (a) generated sets (b) matched sets (c) completed sets.

reconstruction error (NRE) for CD and EMD, which are defined as

where and compute the CD and EMD, respectively. The values of and are lower bounded by 1, and the smaller, the better.

Quantitative comparisons. Figures 8(a) and (b) show the and values of the reconstructed point clouds from generated and matched sets by different donwsampling methods under various downsampling sizes, where it can be seen that except the extremely small downsampled size (m = 8 and 16), our MOPS-Net and FMOPSNet produce much lower distortion than S-Net and SampleNet. For the reconstruction from completed sets shown in Figure 8(c), S-Net and SampleNet are even worse than FPS when m > 64. The reason may be that FPS is able to produce uniformly distributed downsampled points; however, the uniform distribution cannot be guaranteed by S-Net and SampleNet as they are generative methods. However, our MOPS-Net and FMOP-Net consistently achieve the best and second best performance under all downsampled sizes, respectively, and FMOP-Net is even comparable to MOPSNet.

Visual comparisons. Figure 9 visually compares the reconstructed dense point clouds from sparse ones obtained by different downsampling methods, where it can be seen that the reconstructed point clouds from our MOPS-Net are much better than those from other downsampling methods and are much closer to ground-truth ones. This advantage is is credited to that the downsampled points by our MOPSNet can well capture the contour and salient features of 3D shapes.

5.2.2 Reconstruction of large-scale point clouds

To demonstrate the ability of our MOPS-Net on large-scale point clouds, we also examined MOPS-Net on point clouds with 40,960 points each. Unfortunately, the MLP-based reconstruction framework (see Figure 11(a)) employed in Section 5.2.1 cannot well adapt to large-scale point cloud reconstruction because the network size is linearly proportional to the number of output points. To this end, we also propose a new framework for reconstructing large-scale point clouds, whose network size is independent of the output point number.

The proposed framework for large-scale point cloud reconstruction, namely Multi-FoldingNet (M-FoldingNet), is motivated by FoldingNet [45] shown in Figure 11(b). As illustrated in Figure 11(d), instead of concatenating the global feature to the 2D coordinates of a single regular grid, we first segment such a global feature into a set of M local features with a smaller and equal feature dimension, and it is expected that each local feature encodes the high-level semantic information of a local patch on a point cloud . We then concatenate each local feature to the coordinates of a 2D regular grid separately, which are fed into a shared folding operator. Note that the number of output points can be varied by adjusting the dimensions of the 2D regular grids. Compared with FoldingNet, which can be thought of as a special case of M-FoldingNet with M = 1, MFoldingNet with fewer network parameters can allow the folding operator to focus on local regions which are easier to be reconstructed. Although AtlasNet [46] illustrated in Figure 11(c) also realizes reconstruction in a local manner, it adopts multiple independent folding operators, leading to the significant increase of network parameters, compared with FoldingNet.

We first evaluated and compared the proposed MFoldingNet with other reconstruction frameworks on both small-scale point clouds (1024 points) and large-scale point clouds (10,000 points each)3. The settings, including the dataset, the generation of codewords/global features, and the task loss, were kept identical to the MLP-based reconstruction framework in Section 5.2.1. For fair comparisons, we used an identical codeword dimension for all reconstruction frameworks, i.e., the codeword dimension equals to

Fig. 9. Visual comparisons of downsampled point clouds (1rows) and reconstructed point clouds (2rows) by different downsampling methods with m = 32. (a) Original point clouds and corresponding reconstructions. (b) Random sampling; (c) Voxelization; (d) FPS; (e) S-Net; (f) SampleNet; (g) MOPS-Net.

Fig. 10. Visual comparisons of the reconstructed large-scale point clouds via different reconstruction frameworks. (a) Original point cloud with 10, 000 points; Reconstructed by (b) FoldingNet; (c) M-FoldingNet (M = 4); (d) M-FoldingNet (M = 8); (e) M-FoldingNet (M = 16); (f) M-FoldingNet (M = 64).

TABLE 4 Quantitative comparisons of different reconstruction frameworks. The smaller, the better. Model sizes were measured under the same parameter precesion.

Fig. 11. Illustration of the differences between different frameworks for point cloud reconstruction. (a) MLP-based; (b) FoldingNet [45]; (c) AtlasNet [46]; (d) Proposed M-FoldingNet. Note that the multiple folding operators of the AtlasNet are independent. For FoldingNet, AtlasNet, and M-FoldingNet, the number of reconstructed points depends on the dimensions of the 2D grids.

d = 128 (resp. d = 512) for reconstructing point clouds with n = 1024 (resp. n = 10, 000) points. As listed in Table 4, we can observe that the proposed M-FoldingNet can achieve better performance than FoldingNet. Specifically, M-FoldingNet with M = 4 can achieve best performance on the reconstruction of small-scale point clouds. The reconstruction quality decreases with M increasing because the segmented local features have a limited dimension to fully embed local part information. As expected, more pieces of local features are needed to achieve best reconstruction performance. Besides, our M-FoldingNet is more compact than FoldingNet and MLP-based. Figure 10 visually compares the reconstructed point clouds by M-FlodingNet and FoldingNet, which also demonstrates the superiority of the proposed reconstruction framework

We evaluated the performance of MOPS-Net for down-sampling real scanned data with 40,960 points each [?], where the pre-trained M-FoldingNet (M = 128 and d = 2048 (or )) was used as the task network. The settings of MOPS-Net were the same as those in Section 5.2.1. Ta-

TABLE 5 Quantitative comparisons of the distortion of reconstructed point clouds from downsampled sparse point clouds by different methods. The original point clouds consist of 40,960 points each. The smaller, the better.

m RS FPS MOPS RS FPS MOPS

ble 5 quantitatively compares the distortion of reconstructed point clouds from the downsampled sparse point clouds by random sampling, FPS, and MOPS-Net 4. From Table 5, we can see that the and values of the reconstruction from the downsampled points by MOPSNet are much smaller than those of the reconstruction from downsampled points by RS and FPS under all cases, which demonstrates the ability of MOPS-Net in handling large-scale point clouds. Besides, in Figure 12, we visualized the reconstructed dense point clouds from downsampled m = 256 points via different methods. The high quality of reconstructed 40,960 dense point clouds from MOPSNet demonstrates the superiority of the proposed method for downsampling large-scale point clouds. The memory and computational complexity for downsampling large-scale point clouds can also be found in Section 5.6. Last but not least, for a very large-scale point cloud in practice, a promising solution is to partition it to several regions with smaller points each and then downsample the regions in parallel or sequentially.

5.3 Registration-oriented Downsampling

Registration aims to predict rigid transformations between two point clouds, including a rotation and a translation, which can well align them. As an overdetermined problem, only a few key points, which can well capture the shape information of a point cloud, are usually extracted, and the registration will be conducted on the key points rather than original point clouds to save memory and computational complexity. Thus, the registration accuracy also depends on the quality of selected key points. In this section, we

Fig. 12. Visual comparisons of reconstructed large-scale point clouds by different downsampling methods with m = 256. (a) Original 40,960 real scanned data. Reconstructed dense point clouds from 256 points sampled by (b) Random sampling; (c) FPS; (d) MOPS-Net; (e) Reconstructed dense point cloud by extracting the codeword from the original point cloud. Colors are assigned by the pointwise depths for better visualization.

TABLE 6 Quantitative comparisons of MREs of donwsampled point clouds by different methods used for registration. The smaller, the better.

examined the performance of MOPS-Net with registration as the subsequent task.

We utilized point clouds with 1024 points each in Model-Net40 as the original data. The paired data were generated by applying random rotations and translations to training point clouds. We adopted PCRNet [47] with one iteration as the task network, whose loss function is the L2 difference between the predicted quaternions and ground-truth ones. We quantitatively evaluated different downsampling methods by using the mean rotation error (MRE) between the predicted rotations and ground-truth ones. Note that the MRE of PCRNet trained and tested with original point clouds is 7.21. For MOPS-Net, we set and initialized the learning rate to and exponentially decreased to within 250 epochs.

Quantitative and visual comparisons. Table 6 lists MRE values of different downsampling methods with the task network fixed to the pre-trained PCRNet, where we can observe that the proposed MOPS-Net can achieve best performance than the traditional methods, S-Net, and SampleNet for all settings, and FMOPS-Net even achieves better performance than MOPS-Net over the generated sets. Besides, SNet suffers from significant performance degradation once the generated point sets are restricted to be subsets of original point clouds (i.e. the matched sets). Figure 13 visually compares different methods, where the advantage of our MOPS-Net is verified again.

Joint training. In all the above experiments for classi-fication, reconstruction, and registration, the task networks were fixed to be the pre-trained models. Here, taking the registration task as an example, we illustrated the advantage of joint training i.e., the task network PCRNet and the downsampling network MOPS-Net are jointly trained. As listed in Table 7, such a joint training manner can further improve the registration accuracy.

TABLE 7 Quantitative comparisons of MREs for pre-trained and jointly trained registration networks. The smaller, the better.

Pre-trained PCRNet Jointly trained PCRNet m G M C G M C

5.4 Robustness Analysis

We also evaluated the robustness of the proposed MOPSNet to noise over the classification task. We added various levels of the Gaussian noise to input point clouds. As shown in Figure 14, even the input point clouds are highly contaminated, i.e., the noise level in each dimension is 10%, MOPS-Net still remains high accuracy which is comparable to that of MOPS-Net with clean input, demonstrating its robustness. Besides, the accuracy of MOPS-Net with noisy input is still higher than that of S-Net and SampleNet with clean input.

In Figure 15, we visually illustrated the downsampled points by our MOPS-Net over noisy data, where it can be seen that the proposed MOPS-Net can capture the important regions (head, hands, legs) and the locations of sampled points remain consistent at different noise levels.

5.5 Ablation Study

Taking the reconstruction task in Section 5.2.1 as an example, we investigated the effect of the hyper-parameters contained in the loss functions of our MOPS-Net and SampleNet [13]. The loss function used by SampleNet is given as

which contains two hyper-parameters and . Note that in the experiments of Section 5.2.1, the hyper-parameters of both MOPS-Net and SampleNet have been tuned to be

Fig. 13. Visual comparisons of registered point clouds by different downsampling methods (m = 8). (a) Non-registered input pair; Registered results on the key points extracted by (b) Random sampling; (c) Voxelization; (d) FPS; (e) S-Net; (f) SampleNet; and (g) MOPS-Net.

Fig. 14. Comparison of the classification performance of different down-sampling methods applied to point clouds with various levels of noise. (a) performance on the generated sets (b) performance on the completed sets Note that the noise level refers to that added to each dimensional of 3D point cloud data.

almost optimal, i.e., and for MOPS-Net and and for SampleNet.

As shown in Figure 16, we can see that our MOPSNet can consistently achieve almost optimal performance in wide ranges of and , demonstrating its stability. However, the performance of SampleNet varies largely as the values of and change. Besides, under the best parameter settings, SampleNet is still worse than MOPS-Net in terms of both and .

5.6 Complexity Analysis

Table 8 reports the running time of various methods applied to downsample point clouds with 1024 points each. All methods were implemented on GTX 2080Ti GPU, and we reported the average inference time per shape. From Table 8, we observe that the running time of FPS is linearly proportional to the downsampled size m, while the other methods are insensitive to m. Besides, S-Net and MOPS-Net are even faster than random sampling. Compared with SNet, SampleNet requires an additional projection operation involving k-NN search, and thus takes more time than SNet.

Besides, we also analyzed the complexity of down-sampling large-scale point clouds. The running time and memory consumptions were recorded during the inference period on Quadro RTX 8000 GPU. Table 9 lists the running time of our MOPS-Net applied to downsample point clouds with 40,960 points each. We also provided the running time of random sampling and FPS as a reference. We observe that our MOPS-Net works efficiently and is faster than FPS. Table 10 lists the memory consumption and running time for each step when downsampling 4096 points from 40,960 points, where it can be observed that performing the standard feature extraction via PointNet on large-scale point clouds consumes most of the inference time. By contrast, the proposed sampling modules, including the learning of the sampling matrix and regression of the sampled set are very efficient. Besides, it is worth noticing that the MLP operators which formulate the feature extraction module and learning of the sampling matrix module, requires large memory consumption when dealing with large-scale point clouds.

TABLE 8 Average running time (seconds) for downsampling a 1024-point model.

TABLE 9 Time efficiency (seconds) for downsampling 40,960 points.

6 CONCLUSION & FUTURE WORK

In this paper, we presented MOPS-Net, a novel end-to-end deep learning framework for task-oriented point cloud

Fig. 15. Visualization of sampled point clouds by MOPS-Net on clean data and noisy data with various levels of noise. (a) clean; (b) 1% noise ; (c) 2% noise; (d) 5% noise; (e) 10% noise. Note that the noise level refers to that added to each dimensional of 3D point clouds.

Fig. 16. Illustration of the effect of the hyper-parameters involved in the loss functions of our MOPS-Net and SampleNet on reconstruction performance (m = 32). SampleNet’s performance is highly sensitive to the hyper-parameters, whereas ours is not.

TABLE 10 Memory consumption and running time of MOPS-Net to downsample m = 4096 points from large-scale point clouds with n = 40, 960 points.

downsampling. In contrast to the existing methods, we designed MOPS-Net from the perspective of matrix optimization. As the original discrete and combinatorial optimization problem is difficult to solve, we obtained a continuous and differentiable form by relaxing the 0-1 constraint of each variable. MOPS-Net elegantly mimics the function of the resulting matrix optimization problem by exploring both local and global structures of input data. MOPS-Net is permutation invariant and can be end-to-end trained with a task network. We applied MOPS-Net to three typical applications, including 3D point cloud classification, reconstruction, and registration, and observed that MOPS-Net produces better results than state-of-the-art methods. Moreover, MOPS-Net is flexible in that with simple modification, a single network with one-time training can handle arbitrary downsampling ratios. We justified our optimization driven design principle and demonstrated the efficacy of MPOSNet through extensive evaluations and comparisons.

The promising results of MOPS-Net inspire several interesting future directions. For example, it can replace the widely used FPS in feature extraction of current networks to boost performance. Though MOPS-Net is designed for point cloud downsampling, increasing the dimension of differential sampling matrix allows us to handle upsampling as well. Moreover, MOPS-Net opens the door to apply matrix optimization in deep learning. We believe the matrix optimization idea is general and can work for other selection and ranking problems, such as key frame selection in videos [48], [49], band selection in hyperspectral images [50], [51] and view selection in light field images [52].

REFERENCES

[1] Y. Eldar, M. Lindenbaum, M. Porat, and Y. Y. Zeevi, “The farthest point strategy for progressive image sampling,” IEEE Transactions on Image Processing, vol. 6, no. 9, pp. 1305–1315, 1997. 1

[2] X. Ying, S. Xin, Q. Sun, and Y. He, “An intrinsic algorithm for parallel poisson disk sampling on arbitrary surfaces,” IEEE Trans. Vis. Comput. Graph., vol. 19, no. 9, pp. 1425–1437, 2013. 1, 2

[3] C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “Pointnet++: Deep hierar- chical feature learning on point sets in a metric space,” in Advances in neural information processing systems, 2017, pp. 5099–5108. 2, 3, 4

[4] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 652–660. 2, 4, 5, 7

[5] Y. Li, R. Bu, M. Sun, W. Wu, X. Di, and B. Chen, “Pointcnn: Con- volution on x-transformed points,” in Advances in neural information processing systems, 2018, pp. 820–830. 2, 3

[6] Y. Shen, C. Feng, Y. Yang, and D. Tian, “Mining point cloud local structures by kernel correlation and graph pooling,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4548–4557. 2

[7] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon, “Dynamic graph cnn for learning on point clouds,” ACM Transactions on Graphics (TOG), vol. 38, no. 5, pp. 1–12, 2019. 2, 3, 4

[8] J. Li, B. M. Chen, and G. Hee Lee, “So-net: Self-organizing network for point cloud analysis,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 9397–9406. 2, 3

[9] L. Tchapmi, C. Choy, I. Armeni, J. Gwak, and S. Savarese, “Seg- cloud: Semantic segmentation of 3d point clouds,” in 2017 international conference on 3D vision (3DV). IEEE, 2017, pp. 537–547. 2, 3

[10] H. Su, V. Jampani, D. Sun, S. Maji, E. Kalogerakis, M.-H. Yang, and J. Kautz, “Splatnet: Sparse lattice networks for point cloud processing,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 2530–2539. 2, 3

[11] Q. Hu, B. Yang, L. Xie, S. Rosa, Y. Guo, Z. Wang, N. Trigoni, and A. Markham, “Randla-net: Efficient semantic segmentation of large-scale point clouds,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 11 108–11 117. 2, 3

[12] O. Dovrat, I. Lang, and S. Avidan, “Learning to sample,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 2760–2769. 2, 3, 4, 6, 7, 12, 13

[13] I. Lang, A. Manor, and S. Avidan, “Samplenet: differentiable point cloud sampling,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 7578–7588. 2, 3, 4, 6, 7, 12, 13

[14] R. Kimmel and J. A. Sethian, “Computing geodesic paths on manifolds,” Proceedings of the National Academy of Sciences, vol. 95, no. 15, pp. 8431–8435, 1998. 2

[15] W. Wu, Z. Qi, and L. Fuxin, “Pointconv: Deep convolutional networks on 3d point clouds,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 9621–9630. 2

[16] R. Bridson, “Fast poisson disk sampling in arbitrary dimensions.” SIGGRAPH sketches, vol. 10, pp. 1 278 780–1 278 807, 2007. 2

[17] L.-Y. Wei, “Parallel poisson disk sampling,” ACM Transactions on Graphics (TOG), vol. 27, no. 3, pp. 1–9, 2008. 2

[18] M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi, “Xnornet: Imagenet classification using binary convolutional neural networks,” in European conference on computer vision. Springer, 2016, pp. 525–542. 2

[19] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classi-fication with deep convolutional neural networks,” in Advances in neural information processing systems, 2012, pp. 1097–1105. 2

[20] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778. 2

[21] M. Angelina Uy and G. Hee Lee, “Pointnetvlad: Deep point cloud based retrieval for large-scale place recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 4470–4479. 3

[22] Z. Kuang, J. Yu, J. Fan, and M. Tan, “Deep point convolutional approach for 3d model retrieval,” in 2018 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2018, pp. 1–6. 3

[23] P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. Guibas, “Learning representations and generative models for 3d point clouds,” in International conference on machine learning. PMLR, 2018, pp. 40– 49. 3, 8

[24] Y. Sun, Y. Wang, Z. Liu, J. Siegel, and S. Sarma, “Pointgrow: Au- toregressively learned point cloud generation with self-attention,” in The IEEE Winter Conference on Applications of Computer Vision, 2020, pp. 61–70. 3

[25] C.-L. Li, M. Zaheer, Y. Zhang, B. Poczos, and R. Salakhutdinov, “Point cloud gan,” arXiv preprint arXiv:1810.05795, 2018. 3

[26] G. Elbaz, T. Avraham, and A. Fischer, “3d point cloud registration for localization using a deep neural network auto-encoder,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 4631–4640. 3

[27] Y. Aoki, H. Goforth, R. A. Srivatsan, and S. Lucey, “Pointnetlk: Robust & efficient point cloud registration using pointnet,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 7163–7172. 3

[28] Z. J. Yew and G. H. Lee, “3dfeat-net: Weakly supervised local 3d features for point cloud registration,” in European Conference on Computer Vision. Springer, 2018, pp. 630–646. 3

[29] A. Zaganidis, L. Sun, T. Duckett, and G. Cielniak, “Integrating deep semantic segmentation into 3-d point cloud registration,” IEEE Robotics and Automation Letters, vol. 3, no. 4, pp. 2942–2949, 2018. 3

[30] Y. Zhou and O. Tuzel, “Voxelnet: End-to-end learning for point cloud based 3d object detection,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 4490–4499. 3

[31] B. Li, “3d fully convolutional network for vehicle detection in point cloud,” in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017, pp. 1513–1518. 3

[32] S. Shi, X. Wang, and H. Li, “Pointrcnn: 3d object proposal gen- eration and detection from point cloud,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 770– 779. 3

[33] M. Engelcke, D. Rao, D. Z. Wang, C. H. Tong, and I. Posner, “Vote3deep: Fast object detection in 3d point clouds using efficient convolutional neural networks,” in 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2017, pp. 1355–1361. 3

[34] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom, “Pointpillars: Fast encoders for object detection from point clouds,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 12 697–12 705. 3

[35] L. Yu, X. Li, C.-W. Fu, D. Cohen-Or, and P.-A. Heng, “Pu-net: Point cloud upsampling network,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 2790–2799. 3

[36] W. Yifan, S. Wu, H. Huang, D. Cohen-Or, and O. Sorkine-Hornung, “Patch-based progressive 3d point set upsampling,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 5958–5967. 3

[37] R. Li, X. Li, C.-W. Fu, D. Cohen-Or, and P.-A. Heng, “Pu-gan: a point cloud upsampling adversarial network,” in Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 7203– 7212. 3

[38] Y. Qian, J. Hou, S. Kwong, and Y. He, “Pugeo-net: A geometry- centric network for 3d point cloud upsampling,” in European Conference on Computer Vision. Springer, 2020, pp. 752–769. 3

[39] E. Nezhadarya, E. Taghavi, R. Razani, B. Liu, and J. Luo, “Adaptive hierarchical down-sampling for point cloud classification,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 12 956–12 964. 3

[40] J. Yang, Q. Zhang, B. Ni, L. Li, J. Liu, M. Zhou, and Q. Tian, “Modeling point clouds with self-attention and gumbel subset sampling,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 3323–3332. 3

[41] H. Thomas, C. R. Qi, J.-E. Deschaud, B. Marcotegui, F. Goulette, and L. J. Guibas, “Kpconv: Flexible and deformable convolution for point clouds,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 6411–6420. 4

[42] Y. Saad, Iterative methods for sparse linear systems. SIAM, 2003. 5

[43] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao, “3d shapenets: A deep representation for volumetric shapes,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1912–1920. 7

[44] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su et al., “Shapenet: An information-rich 3d model repository,” arXiv preprint arXiv:1512.03012, 2015. 8

[45] Y. Yang, C. Feng, Y. Shen, and D. Tian, “Foldingnet: Point cloud auto-encoder via deep grid deformation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 206– 215. 9, 11

[46] T. Groueix, M. Fisher, V. G. Kim, B. C. Russell, and M. Aubry, “A papier-mâché approach to learning 3d surface generation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 216–224. 9, 11

[47] V. Sarode, X. Li, H. Goforth, Y. Aoki, R. A. Srivatsan, S. Lucey, and H. Choset, “Pcrnet: point cloud registration network using pointnet encoding,” arXiv preprint arXiv:1908.07906, 2019. 12

[48] Z. Wu, C. Xiong, C.-Y. Ma, R. Socher, and L. S. Davis, “Adaframe: Adaptive frame selection for fast video recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 1278–1287. 14

[49] S. Mei, M. Ma, S. Wan, J. Hou, Z. Wang, and D. D. F. Feng, “Patch based video summarization with block sparse representation,” IEEE Transactions on Multimedia, vol. PP, pp. 1–1, 04 2020. 14

[50] B. Guo, S. R. Gunn, R. I. Damper, and J. D. Nelson, “Band selection for hyperspectral image classification using mutual information,” IEEE Geoscience and Remote Sensing Letters, vol. 3, no. 4, pp. 522–526, 2006. 14

[51] Q. Wang, J. Lin, and Y. Yuan, “Salient band selection for hyperspec- tral image classification via manifold ranking,” IEEE transactions on neural networks and learning systems, vol. 27, no. 6, pp. 1279–1289, 2016. 14

[52] J. Jin, J. Hou, J. Chen, H. Zeng, S. Kwong, and J. Yu, “Deep coarse- to-fine dense light field reconstruction with flexible sampling and geometry-aware fusion,” IEEE Annals of the History of Computing, no. 01, pp. 1–1, 2020. 14

designed for accessibility and to further open science