RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving

2020·Arxiv

Abstract

Abstract

In this work, we propose an efficient and accurate monocular 3D detection framework in single shot. Most successful 3D detectors take the projection constraint from the 3D bounding box to the 2D box as an important component. Four edges of a 2D box provide only four constraints and the performance deteriorates dramatically with the small error of the 2D detector. Different from these approaches, our method predicts the nine perspective keypoints of a 3D bounding box in image space, and then utilize the geometric relationship of 3D and 2D perspectives to recover the dimension, location, and orientation in 3D space. In this method, the properties of the object can be predicted stably even when the estimation of keypoints is very noisy, which enables us to obtain fast detection speed with a small architecture. Training our method only uses the 3D properties of the object without the need for external networks or supervision data. Our method is the first real-time system for monocular image 3D detection while achieves state-of-the-art performance on the KITTI benchmark. Code will be released at https://github.com/Banconxuan/RTM3D.

1. Introduction

3D object detection is an essential component of scene perception and motion prediction in autonomous driving [2, 10]. Currently, most powerful 3D detectors heavily rely on 3D LIDAR laser scanners for the reason that it can provide scene locations [9, 48, 43, 31]. However, the LiDAR-based systems are expensive and not conducive to embedding into the current vehicle shape. In comparison, monocular camera devices are cheaper and convenient which makes it drawing an increasing attention in many application scenarios [7, 28, 42]. In this paper, the scope of

Figure 1. Overview of proposed method: We first predict ordinal keypoints projected in the image space by eight vertexes and a central point of a 3D object. We then reformulate the estimation of the 3D bounding box as the problem of minimizing the energy function by using geometric constraints of perspective projection.

our research lies in 3D object detection from only monocular RGB image.

Monocular 3D object detection methods can be roughly divided into two categories by the type of training data: one utilizes complex features, such as instance segmentation, vehicle shape prior and even depth map to select best proposals in multi-stage fusion module [7, 8, 42]. These features require additional annotation work to train some stand-alone networks which will consume plenty of computing resources in the training and inferring stages. Another one only employs 2D bounding box and properties of a 3D object as the supervised data [35, 4, 22, 44]. In this case, an intuitional idea is to building a deep regression network to predict directly 3D information of the object. This can cause performance bottlenecks due to the large search space. For this reason, recent works have clearly pointed out that apply geometric constraints from 3D box vertexes to 2D box edges to refine or directly predict object parameters [30, 25, 4, 22, 28]. However, four edges of a 2D bounding box provide only four constraints on recovering a 3D bounding box while each vertex of a 3D bounding box might correspond to any edges in the 2D box, which will takes 4,096 of the same calculations to get one result [28]. Meanwhile, the strong reliance on the 2D box causes a sharp decline in 3D detection performance when predictions of 2D detectors even have a slight error. Therefore, most of these methods take advantage of two-stage detectors [12, 11, 34] to ensure the accuracy of 2D box prediction, which limit the upper-bound of the detection speed.

In this paper, we propose an efficient and accurate monocular 3D detection framework in the form of one-stage, which be tailored for 3D detection without relying on 2D detectors. The framework can be divided into two main parts, as shown in Fig. 1. First, we perform a one-stage fully convolutional architecture to predict 9 of the 2D keypoints which are projected points from 8 vertexes and central point of 3D bounding box. This 9 keypoints provides 18 geometric constrains on the 3D bounding box. Inspired by CenterNet [47], we model the relationship between the eight vertexes and the central point to solve the keypoints grouping and the vertexes order problem. The SIFT, SUFT and other traditional keypoint detection methods [26, 1]computed an image pyramid to solve the scale-invariant problem. A similar strategy was used by CenterNet as a post-processing step to further improve detection accuracy, which slows the inference speed. Note that the Feature Pyramid Network(FPN) [23] in 2D object detection is not applicable to the network of keypoint detection, because adjacent keypoints may overlap in the case of smallscale prediction. We propose a novel multi-scale pyramid of keypoint detection to generate a scale-space response. The final activate map of keypoints can be obtained by means of the soft-weighted pyramid. Given the 9 projected points, the next step is to minimize the reprojection error over the perspective of 3D points that parameterized by the location, dimension, and orientation of the object. We formulate the reprojection error as the form of multivariate equations in space, which can generate the detection results accurately and efficiently. We also discuss the effect of different prior information on our keypoint-based method, such as dimension, orientation, and distance. The prerequisite for obtaining this information is not to add too much computation so as not to affect the final detection speed. We model these priors and reprojection error term into an overall energy function in order to further improve 3D estimation.

To summarize, our main contributions are the following:

• We formulate the monocular 3D detection as the keypoint detection problem and combine the geometric constrains to generate properties of 3D objects more

• We propose a novel one-stage and multi-scale network for 3D keypoint detection which provide the accurate project points for multi-scale object.

• We propose an overall energy function that can jointly optimize the prior and 3D object information.

• Evaluation on the KITTI benchmark, We are the first real-time 3D detection method using only images and achieves better accuracy under the same running time in comparing other competitors.

2. Related Work

The 3D detection can be divided into two groups by the type of data: LiDAR-, and image-based methods.

LiDAR-based method. LiDAR-based systems can provide accuracy and reliable point cloud of object surfaces in 3D scene. Therefor, most of the recent 3D object detection employ it in different representation to obtain the state-of-the-art model [48, 3, 9, 36, 21].

Extra Data or Network for Image-based 3D Object Detection. In the last years, many studies develop the 3D detection in an image-based method for the reason that camera devices are more convenient and much cheaper. To complement the lacked depth information in image-based detection, most of the previous approaches heavily relied on the stand-alone network or additional labeling data, such as instance segmentation, stereo, wire-frame model, CAD prior , and depth, as shown in Table. 1. Among them, monocular 3D detection is a more challenging task due to the difficulty of obtaining reliable 3D information from a single image. One of the first examples [7] enumerate a multitude of 3D proposals from pre-defined space where the objects may appear as the geometrical heuristics. Then it takes the other complex prior, such as shape, instance segmentation, contextual feature, to filter out dreadful proposals and scoring them by a classifier. To make up for the lack of depth, [42] embed a pre-trained stand-alone module to estimate the disparity and 3D point cloud. The disparity map concatenates the front view representation to help the 2D proposal network and the 3D detection can be boosted by fusing the extracted feature after RoI pooling and point cloud. As a followup, [27] combines the 2D detector and monocular depth estimation model to obtain the 2D box and corresponding point cloud. The final 3D box can be obtained by the regression of PointNet [32] after the aggregation of the image feature and 3D point information through attention mechanism, which achieves the best performance in the monocular image. Intuitively, these methods would certainly increase the accuracy of the detection, but the additional network and annotated data would lead to more computation and labor-intensive work.

Table 1. Comparison of the real-time status and the requirements of additional data in different image-based detection approaches.

Image-only in Monocular 3D Object Detection. Recent works have tried to fully explore the potency of RGB images for 3D detection. Most of them include geometric constraints and 2D detectors to explicitly describe the 3D information of the object. [28] uses CNN to estimate the dimension and orientation extracted feature from the 2D box, then it proposes to obtain the location of an object by using the geometric constraints of the perspective relationship between 3D points and 2D box edges. This contribution is followed by most image-based detection methods either in re-finement step or as direct calculation on 3D objects [22, 4]. All we know in this constraint is that certain 3D points are projected onto 2D edges, but the corresponding relationship and the exact location of the projection are not clear. Therefore, it needs to exhaustively enumerate config-urations to determine the final correspondence and can only provide four constraints, which is not sufficient for fully 3D representation in 9 parameters. It led to the need to estimate other prior information. Nevertheless, possible inaccuracies in the 2D bounding boxes may result in a grossly inaccurate solution with a small number of constraints. Therefore, most of these methods obtain more accurate 2D box through a two-stage detector, which is difficult to get real-time speed.

Keypoints in Monocular 3D Object Detection. It is believed that the detection accuracy of occluded and truncated objects can be improved by deducing complete shapes from vehicle keypoints [6, 46, 29]. They represent the regularshape vehicles as a wire-frame template , which is obtained from a large number of CAD models. To train the keypoint detection network, they need to re-label the data set and even use depth maps to enhance the detection capability. [14] is most related to our work, which also considers the wire-frame model as prior information. Furthermore, It jointly optimizes the 2D box, 2D keypoints, 3D orientation, scale hypotheses, shape hypotheses, and depth with four different networks. This has limitations in run time. In contrast to prior work, we reformulate the 3D detection as the coarse keypoints detection task. Instead of predicting the 3D box based on an off-the-shelf 2D detectors or other data generators, we build a network to predict 9 of 2D keypoints projected by vertexes and center of 3D bounding box while minimize the reprojection error to find an optimal result.

3. Proposed Method

In this section. We first describe the overall architecture for keypoint detection. Then we detail how to find the 3D vehicles from the generated keypoints.

3.1. Keypoint Detection Network

Our keypoint detection network takes an only RGB image as the input and generates the perspective points from vertexes and center of the 3D bounding box. As shown in Fig. 2, it consists of three components: backbone, keypoint feature pyramid, and detection head. The main architecture adopts a one-stage strategy that shares a similar layout with the anchor-free 2D object detector [38, 16, 47, 19], which allows us to get a fast detection speed. Details of the network are given below.

Backbone. For the trade-off between speed and accuracy, we use two different structures as our backbones: ResNet-18[13] and DLA-34 [45]. All models take a single RGB image and downsample the input with factor S = 4. The ResNet-18 and DLA-34 build for image classification network, the maximal downsample factor is . We upsample the bottleneck thrice by three bilinear interpolations and convolutional layer. Before the upsampling layers, we concatenate the corresponding feature maps of the low level while adding one convolutional layers for channel dimension reduction. After three upsampling layers, the channels are 256, 128, 64, respectively.

Keypoint Feature Pyramid. Keypoint in the image have no difference in size. Therefore, the keypoint detection is not suitable for using the Feature Pyramid Network(FPN) [23], which detect multi-scale 2D box in different pyramid layers. We propose a novel approach Keypoint Feature Pyramid Network (KFPN) to detect scale-invariant keypoints in the point-wise space, as shown in Fig. 3. Assuming we have F scale feature maps, we first resize each

Figure 2. An overview of proposed keypoint detection architecture: It takes only the RGB images as the input and outputs main center heatmap, vertexes heatmap, and vertexes coordinate as the base module to estimate 3D bounding box. It can also predict other alternative priors to further improve the performance of 3D detection.

scale f, 1 < f < F back to the size of maximal scale, which yields the feature maps . Then, we generate soft weight by a softmax operation to denote the importance of each scale. The finally scale-space score map is obtained by linear weighing sum. In detail, it can be defined as:

where denote element-wise product. Detection Head. The detection head is comprised of three fundamental components and six optional components which can be arbitrarily selected to boost the accuracy of 3D detection with a little computational consumption. Inspired by CenterNet [47], we take a keypoint as the maincenter for connecting all features. Since the 3D projection point of the object may exceed the image boundary in the case of truncation, the center point of the 2D box will be

Figure 3. Illustration of our keypoint feature pyramid net- work(KFPN).

selected more appropriately. The heatmap can be define as , where C is the number of object categories. Another fundamental component is the heatmap of nine perspective points projected by vertexes and center of 3D bounding box. For keypoints association of one object, we also regress an local offset from the maincenter as an indication. Keypoints of V closest to the coordinates from are taken as a group of one object.

Although the 18 constraints by the 9 keypoints have an ability to recover the 3D information of the object, more prior information can provide more constraints and further improve the detection performance. We offer a number of options to meet different needs for accuracy and speed. The center offset and vertexes offset are discretization error for each keypoint in heatmaps. The dimension of 3D object have a smaller variance, which makes it easy to predict. The rotation of an object only by parametrized by orientation (yaw) in the autonomous driving scene. We employ the Multi-Bin based method [28] to regress the local orientation. We classify the probability with cosin and sine offset of the local angle in one bin, which generates feature map of orientation with two bins. We also regress the depth of 3D box center, which can be used as the initialization value to speed up the solution in Sec .3.2. Training. The all heatmaps of keypoint training strategy follow the [47, 19]. The loss solves the imbalance of positive and negative samples with focal loss [24]:

where K is the channels of different keypoints, K = C in maincenter and K = 9 in vertexes. N is the number of maincenter or vertexes in an image, and and are the hyper-parameters to reduce the loss weight of negative and easy positive samples. We set is and in all experiments following [47, 19]. can be de-fined by Gaussian kernel centered by ground truth keypoint . For , we find the max area and min area of 2D box in training data and set two hyper-parameters and . We then define the for a object with size A. For regres- sion of dimension and distance, we define the residual term as:

We set , where and are the mean and standard deviation dimensions of training data. denotes if maincenter appears in position x, y. The offset of maincenter, vertexes are trained with an L1 loss following [47]:

where are the position of maincenter and vertexes in the original image. The regression coordinate of vertexes with an L1 loss as:

Finial, we define the multi-task loss for keypoint detection as:

We empirical set and in our experimental.

3.2. 3D Bounding Box Estimate

Consider an image I, a set of i = 1...N object are represented by 9 keypoints and other optional prior, given by our keypoint detection network. We define this keypoints as for , dimension as , orientation as , and distance as . The corresponding 3D bounding box can be defined by its rotation , position , and dimensions . Our goal is to estimate the 3D bounding box , whose projections of center and 3D vertexes on the image space best fit the corresponding 2D keypoints . This can be solved by minimize the reprojection error of 3D keypoints and 2D keypoints. We formulate it and other prior errors as a nonlinear least squares optimization problem:

where are measurement error of camera-point, dimension prior and orientation prior respectively. We set and in our experimental. is the covariance matrix of keypoints projection error. It is the confidence extracted from the heatmap corresponding to the keypoints:

In the rest of the section, we will first define this error item, and then introduce the way to optimize the formulation.

Camera-Point. Following the [10], we define the homogeneous coordinate of eight vertexes and 3D center as:

Given the camera intrinsics matrix K, the projection of these 3D points into the image coordinate is:

where and exp maps the into space. The projection coordinate should fit tightly into 2D keypoints detected by the detection network. Therefore, the camera-point error is then defined as:

Minimizing the camera-point error needs the Jacobians in space. It is given by:

Rotation-Prior: We define in SE3 space and use log to map the error into its tangent vector space:

These multivariate equations can be solved via the Gaussnewton or Levenberg-Marquardt algorithm in the g2o library [18]. A good initialisation is mandatory using this optimization strategy. We adopt the prior information generated by keypoint detection network as the initialization value, which is very important in improving the detection speed.

4. Experimental

4.1. Implementation Details

We evaluated our experiments on the KITTI 3D detection benchmark [10], which has a total of 7481 training images and 7518 test images. We follow the [8] and [41] to split the training set as and respectively. We comprehensively compare our framework and other method on this two validation as well as test set.

We implemented our network using PyTorch, with the machine i7-8086K CPU and 2 1080Ti GPUs. We pad the original image to for training and testing. We project the 3D bounding box of Ground Truths in the left and right images to obtain Ground Truth keypoints and use the flipping as image augmentation, which makes our dataset is quadruple with the origin training set. We run Adam [15] optimizer with a base learning rate of 0.0002 for 300 epochs and reduce at 150 and 180 epochs. For standard deviation of Gaussian kernel, we set and . Based on the statistics of KITTI dataset, we set and . In the inference step, af- ter max pooling, we filter the maincenter and keypoints with threshold 0.4 and 0.1 respectively, and only keypoints that in the image size range is sent into the geometric constraint module. The backbone networks are initialized by a classification model pre-trained on the ImageNet data set. Finally, The ResNet-18 takes about three days with batch size 16 and DLA-34 for four days with batch size 30 in training.

4.2. Comparison with Other Methods

To fully evaluate the performance of our keypoint-based method, for each task three official evaluation metrics be reported in KITTI: average precision for 3D intersection-over-union (), average precision for Birds Eye View (), and Average Orientation Similarity (AOS) if 2D bounding box available. We evaluate our method at three difficulty settings: easy, moderate, and hard, according to the object’s occlusion, truncation, and height in the image space [10].

and . We compare our method with current image-based SOTA approaches and also provide a comparison about running time. However, it is not realistic to list the running times of all previous methods because most of them do not report their efficiency. The results and running time are shown in Table 2 and 3, respectively. ResNet-18 as the backbone achieves the best speed while our accuracy outperforms most of the image-only method. In particular, it is more than 100 times faster than Mono3D [7] while outperforms over 10% for both and across all datasets. In addition, our ResNet-18 method is more than 75 times faster while

Table 2. Comparison of our method to image-based 3D detection frameworks for car category evaluated using metric of KITTI data set. Extra means the extra data used in training.

Table 3. Comparison of our method to image-based 3D detection frameworks for car category, evaluated using metric of KITTI data set.

having a comparable accuracy than 3DOP [8], which employs stereo images as the input. DLA-34 as the backbone achieves the best accuracy while having relatively good speed. It is faster about 3 times than the recently proposed M3D-RPN [4] while achieves the improvement in most of the metrics. Note that comparing our method with this all approaches is unfair because most of these approaches rely on extra stand-alone network or data in addition to monocular images. Nevertheless, we achieve the best speed with better performance.

Results on the KITTI testing set. We also evaluate our results on the KITTI testing set, as shown in Table. 4. More details can be found on the KITTI website 1.

4.3. Qualitative Results

Fig. 4 shows some qualitative results of our method. We visualize the keypoint detection network outputs, geometric constraint module outputs and BEV images. The results of the projected 3D box on image demonstrate than our method can handle crowded and truncated objects. The

Table 4. Comparing 3D detection on KITTI testing set. We use the DLA-34 as the backbone.

results of the BEV image show that our method has an accuracy localization in different scenes.

4.4. Ablation Study

Effect of Optional Components. Three optional components be employed to enhance our method: dimension, orientation, distance and keypoints offset. We experiment with different combinations to demonstrate their effect on 3D detection. The results are shown in Table.5, we train our network with DLA-34 backbone and evaluate it using and . The combinations of dimension, orientation, distance and keypoints offset achieve the best accuracy meanwhile have a faster running speed. This is be-

Figure 4. Qualitative results of our 3D detection. From top to bottom are keypoints, projections of the 3D bounding box and bird’s eye view image, ground truths in green and our results in blue. The crimson arrows, green arrows, and red arrows point to occluded, distant, and truncated objects, respectively.

cause we take the output predicted by our network as the initial value of the geometric optimization module, which can reduce the search space of the gradient descent method.

Table 5. Ablation study of different optional selecting results on set. We use the DLA-34 as the backbone.

Effect of Keypoint FPN. We propose keypoint FPN as a strategy to improve the performance of multi-scale keypoint detection. To better understand its effect, we compare the and with and without KFPN. The details are shown in Table. 6, using KFPN achieves the improvement across all sets while no significant change in time consumption.

Table 6. Comparing 3D detection of w/o KFPN and w/ KFPN for car category on set. We use the DLA-34 as the backbone.

2D Detection and Orientation. Although our focus is on 3D detection, we also compare the performance of our methods in 2D detection and orientation evaluation. We report the AOS and AP with a threshold IoU=0.7 for comparison. The results are shown in Table. 7, the Deep3DBox train MS-CNN [5] in KITTI to produce 2D bounding box and adopt VGG16 [37] for orientation prediction, which gives him the highest accuracy. Deep3Dbox takes advantage of better 2D detectors, however, our outperforms it by about 20% in moderate sets, which emphasize the importance of customizing the network specifically for 3D detection. Another interesting finding is that the 2D accuracy of back-projection 3D results is better than the direct prediction, thanks to our method that can infer the occlusive area of the object.

Table 7. Comparing of 2D detection entation AOS results for car category evaluated on KITTI data set. Only the results under the moderate criteria are shown. Ours(2D) represents the results from the keypoint detection network, and Ours(3D) is the 2D bounding box of the projected 3D box.

5. Conclusion

In this paper, we have proposed a faster and more accurate monocular 3D object detection method for autonomous driving scenarios. We reformulate 3D detection as the keypoint detection problem and show how to recover the 3D bounding box by using keypoints and geometric constraints. We specially customize the point detection network for 3D detection, which can output keypoints of the 3D box and other prior information of the object using only images. Our geometry module formulates this prior to easy-to-optimize loss functions. Our approach generates a stable and accurate 3D bounding box without containing stand-alone networks, additional annotation while achieving real-time running speed.

References

[1] Herbert Bay, Tinne Tuytelaars, and Luc Van Gool. Surf: Speeded up robust features. In European conference on computer vision, pages 404–417. Springer, 2006.

[2] Aseem Behl, Omid Hosseini Jafari, Siva Karthik Mustikovela, Hassan Abu Alhaija, Carsten Rother, and Andreas Geiger. Bounding boxes, segmentations and object coordinates: How important is recognition for 3d scene flow estimation in autonomous driving scenarios? In Proceedings of the IEEE International Conference on Computer Vision, pages 2574–2583, 2017.

[3] Jorge Beltr´an, Carlos Guindel, Francisco Miguel Moreno, Daniel Cruzado, Fernando Garcia, and Arturo De La Escalera. Birdnet: a 3d object detection framework from lidar information. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pages 3517–3523. IEEE, 2018.

[4] Garrick Brazil and Xiaoming Liu. M3d-rpn: Monocular 3d region proposal network for object detection. In Proceedings

of the IEEE International Conference on Computer Vision, Seoul, South Korea, 2019.

[5] Zhaowei Cai, Quanfu Fan, Rogerio S Feris, and Nuno Vas- concelos. A unified multi-scale deep convolutional neural network for fast object detection. In european conference on computer vision, pages 354–370. Springer, 2016.

[6] Florian Chabot, Mohamed Chaouch, Jaonary Rabarisoa, C´eline Teuli`ere, and Thierry Chateau. Deep manta: A coarse-to-fine many-task network for joint 2d and 3d vehicle analysis from monocular image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2040–2049, 2017.

[7] Xiaozhi Chen, Kaustav Kundu, Ziyu Zhang, Huimin Ma, Sanja Fidler, and Raquel Urtasun. Monocular 3d object detection for autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2147–2156, 2016.

[8] Xiaozhi Chen, Kaustav Kundu, Yukun Zhu, Huimin Ma, Sanja Fidler, and Raquel Urtasun. 3d object proposals using stereo imagery for accurate object class detection. IEEE transactions on pattern analysis and machine intelligence, 40(5):1259–1272, 2017.

[9] Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1907–1915, 2017.

[10] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 3354–3361. IEEE, 2012.

[11] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440–1448, 2015.

[12] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580–587, 2014.

[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.

[14] Tong He and Stefano Soatto. Mono3d++: Monocular 3d ve- hicle detection with two-scale 3d hypotheses and task priors. arXiv preprint arXiv:1901.03446, 2019.

[15] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv: Learning, 2014.

[16] Tao Kong, Fuchun Sun, Huaping Liu, Yuning Jiang, and Jianbo Shi. Foveabox: Beyond anchor-based object detector. arXiv preprint arXiv:1904.03797, 2019.

[17] Jason Ku, Alex D Pon, and Steven L Waslander. Monoc- ular 3d object detection leveraging accurate proposals and shape reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11867–11876, 2019.

[18] Rainer Kummerle, Giorgio Grisetti, Hauke Strasdat, Kurt Konolige, and Wolfram Burgard. G 2 o: A general framework for graph optimization. pages 3607–3613, 2011.

[19] Hei Law and Jia Deng. Cornernet: Detecting objects as paired keypoints. In Proceedings of the European Conference on Computer Vision (ECCV), pages 734–750, 2018.

[20] Buyu Li, Wanli Ouyang, Lu Sheng, Xingyu Zeng, and Xiao- gang Wang. Gs3d: An efficient 3d object detection framework for autonomous driving. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.

[21] Bo Li, Tianlei Zhang, and Tian Xia. Vehicle detection from 3d lidar using fully convolutional network. 12, 2016.

[22] Peiliang Li, Xiaozhi Chen, and Shaojie Shen. Stereo r-cnn based 3d object detection for autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7644–7652, 2019.

[23] Tsung-Yi Lin, Piotr Doll´ar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.

[24] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Doll´ar. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.

[25] Lijie Liu, Jiwen Lu, Chunjing Xu, Qi Tian, and Jie Zhou. Deep fitting degree scoring network for monocular 3d object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1057–1066, 2019.

[26] David G Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60(2):91–110, 2004.

[27] Xinzhu Ma, Zhihui Wang, Haojie Li, Pengbo Zhang, Wanli Ouyang, and Xin Fan. Accurate monocular 3d object detection via color-embedded 3d reconstruction for autonomous driving. In Proceedings of the IEEE International Conference on Computer Vision, pages 6851–6860, 2019.

[28] Arsalan Mousavian, Dragomir Anguelov, John Flynn, and Jana Kosecka. 3d bounding box estimation using deep learning and geometry. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7074– 7082, 2017.

[29] J Krishna Murthy, GV Sai Krishna, Falak Chhaya, and K Madhava Krishna. Reconstructing vehicles from a single image: Shape priors for road scene understanding. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 724–731. IEEE, 2017.

[30] Andretti Naiden, Vlad Paunescu, Gyeongmo Kim, Byeong- Moon Jeon, and Marius Leordeanu. Shift r-cnn: Deep monocular 3d object detection with closed-form geometric constraints. 2019 IEEE International Conference on Image Processing (ICIP), pages 61–65, 2019.

[31] Charles R Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J Guibas. Frustum pointnets for 3d object detection from rgbd data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 918–927, 2018.

[32] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 652–660, 2017.

[33] Zengyi Qin, Jinglu Wang, and Yan Lu. Monogrnet: A geo- metric reasoning network for monocular 3d object localization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8851–8858, 2019.

[34] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pages 91–99, 2015.

[35] Cosimo Rubino, Marco Crocco, and Alessio Del Bue. 3d object localisation from multi-view image detections. IEEE transactions on pattern analysis and machine intelligence, 40(6):1281–1294, 2017.

[36] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointr- cnn: 3d object proposal generation and detection from point cloud. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–779, 2019.

[37] Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. Computer Science, 2014.

[38] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object detection. arXiv preprint arXiv:1904.01355, 2019.

[39] Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariha- ran, Mark Campbell, and Kilian Q Weinberger. Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8445–8453, 2019.

[40] Yu Xiang, Wongun Choi, Yuanqing Lin, and Silvio Savarese. Data-driven 3d voxel patterns for object category recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1903–1911, 2015.

[41] Yu Xiang, Wongun Choi, Yuanqing Lin, and Silvio Savarese. Subcategory-aware convolutional neural networks for object proposals and detection. pages 924–933, 2017.

[42] Bin Xu and Zhenzhong Chen. Multi-level fusion based 3d object detection from monocular images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2345–2353, 2018.

[43] Bin Yang, Wenjie Luo, and Raquel Urtasun. Pixor: Real- time 3d object detection from point clouds. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7652–7660, 2018.

[44] Shichao Yang and Sebastian Scherer. Cubeslam: Monocular 3-d object slam. IEEE Transactions on Robotics, 2019.

[45] Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Dar- rell. Deep layer aggregation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2403–2412, 2018.

[46] Muhammad Zeeshan Zia, Michael Stark, and Konrad Schindler. Are cars just 3d boxes?-jointly estimating the 3d

shape of multiple objects. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3678–3685, 2014.

[47] Xingyi Zhou, Dequan Wang, and Philipp Kr¨ahenb¨uhl. Ob- jects as points. In arXiv preprint arXiv:1904.07850, 2019.

[48] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4490–4499, 2018.