Shared Cross-Modal Trajectory Prediction for Autonomous Driving

2020·arXiv

Abstract

1. Introduction

Future trajectory prediction has become the central challenge to succeed in the safe operation of autonomous vehicles designed to cooperate with interactive agents (i.e., pedestrians, cars, cyclists, etc.). It can benefit to the deployment of applications in autonomous navigation and driving assistance systems with advanced motion planning and decision making. Based on the fact that multi-modal sensors (e.g., LiDAR scanner, RGB cameras, radar, etc.) are equipped in autonomous vehicles, we propose a cross-modal embedding framework that demonstrates the efficacy of the use of multiple sensor data for motion prediction.

Figure 1 illustrates an overview of the proposed approach. At training time, we embed multiple feature representations encoded from individual sensor data into a single

*Co-first author. †Work done during Jiachen’s internship at Honda Research Institute

Figure 1: Given a set of multi-modal data (e.g., LiDAR data, RGB images, etc.) obtained from an autonomous vehicle, the model is trained to embed complementary representations of different input modalities into a shared latent space. Output predictions are generated from different perspectives using a latent variable sampled from the learned embedding space. At test time, the proposed method takes a single input modality (e.g., LiDAR data, red-dashed arrow) and predicts the future motion in the same space (i.e., LiDAR-captured world space, red-solid arrow).

shared latent space. Our model jointly optimizes the objective functions across different input modalities, so that the evidence lower bound of multiple input data over the likelihood can be jointly maximized. We provide a derivation of the objective of shared cross-modal embedding and its implementation using a CVAE-based generative model. At test time, the model takes a single input modality (e.g., LiDAR data) and generates a future trajectory from the input perspective (i.e., top-down view) using a latent variable sampled from the shared embedding space. In this way, we can benefit to the model training from the use of multiple input modalities1, while keeping the same computational time for trajectory generation as if the single modality had been used. To the best of our knowledge, we are the first to employ multi-modal sensor data from a single framework for trajectory prediction. Note that existing works solve the problem either in top-down view [23, 36, 9] with LiDAR data or in frontal view [46, 4, 30] with RGB images.

The proposed framework is clearly distinguishable to studies on a multi-modal pipeline for scene understanding such as detection [6, 22, 27], tracking [12, 50], and semantic segmentation [17, 41]. They have presented more accurate models by simply fusing different representations extracted from several sensor modalities. The generation of such joint representations, however, would not be desirable in driving automation systems due to the following issues: (i) during inference, it inherently increases the computation time proportional to the number of input modalities used; and (ii) with the anomalous LiDAR data, the model would fail in finding a solution, which is critical to operate self-driving vehicles. For the former issue, our proposed cross-modal embedding takes only a single input data during inference and thus does not influence the computational time, while it still benefits from the model trained with multiple input modalities. In the latter, our model provides alternative prediction solutions in frontal view using the RGB data, which will activate driving assistance functions (i.e., ADAS) for safe vehicle operation, even with a sensor failure.

To this end, we generate multiple modes of future trajectories by sampling several latent variables from the learned latent space. However, such random sampling-based strategy [23, 9] is likely to predict similar trajectories, ignoring the random variables while generating predictions from the decoder. This posterior collapse2 problem of VAEs is particularly critical to future prediction as it mitigates the diverse modes of system outputs. Therefore, we introduce a regularizer (i) that pushes the model to rely on the latent variables, predicting diverse modes of future motion; and (ii) that does not weaken the prediction capability of the decoder while preventing the performance degradation.

We address the following ideas in the proposed method:

• The objective of shared cross-modal embedding to jointly approximate a real distribution using multiple input sources is mathematically derived using the Kullback-Leibler divergence (Sec. 3.2).

• Shared cross-modal embedding is implemented based on our derivation to benefit from the use of multiple input modalities, while keeping the same computational time as if the single modality had been used (Sec. 3.2).

• The regularizer is designed for future prediction to mitigate posterior collapse of VAEs and to predict more diverse modes of motion behavior (Sec. 3.3). In addition, we design an interaction graph with a graph-level target (Sec. 3.1), introduce a new evaluation metric to measure prediction success (Sec. 4.2), and propose to use absolute motions in frontal view (Sec. 4.1).

Throughout the paper, we use the word ‘multi-modality’ to denote two different sources. First, multi-modal input represents input data obtained from different types of sensors. Second, multi-modal prediction depicts predicted trajectory outputs with multiple variations.

2. Related Work

Pedestrian Trajectory Prediction A majority of research on trajectory prediction [1, 15, 43, 49] has been conducted toward modeling the interactive behavior between humans. These works first encode the temporal information of individual humans and then find their correlation through a social module. Recently, social interactions have been modeled from the graph structure in [42, 19, 31]. Although these methods may be successful in interaction modeling, they overlook the environmental influences that may cause prediction failures in structured environments with stationary obstacles. Therefore, the subsequent work [7, 21] takes images as input to constrain their model using scene context.

Vehicle Trajectory Prediction in Top-down View Similar interaction modules are applied for vehicle trajectory prediction. Some approaches only consider the past motion of road agents [11, 33, 28, 25], and thus result in large errors with a complex road environment in traffic scenes. To alleviate such problems, [23, 36, 24, 9, 37, 39] input additional visual cues to condition their model on the road topology. However, they overlook the vehicle interactions against pedestrians, which is most critical to model the natural behavior of vehicles on the road for safe driving. We thus do not limit our scope to ‘vehicle’ trajectories and its interactions. Instead, we explicitly discover interactions of heterogeneous entities using the proposed interaction graph. Vehicle Trajectory Prediction in Frontal View [4, 46, 30, 29] aim to predict the future trajectory of vehicles in a frontal view image space. They predict a target agent’s relative trajectory with respect to the potential motion of ego-vehicle. Therefore, the predictions are valid only if the accurate ego-future is available. In practice, however, prediction of ego-motion is an another research topic [18] in the transportation domain, which makes hard to simply apply such systems to the real world driving applications. Therefore, we predict the absolute coordinates of trajectories with no effect of unknown ego-future in frontal view.

Multi-Modal Learning Learning representations of multiple input modalities have been explored in recent years. As described in [32], multi-modal learning can be categorized into three types. Multi-modal fusion takes multiple modalities as input and learns their joint representations. Basically, the same set of input types should be provided at test time as in [20, 45]. Cross-modal learning tries to learn more descriptive representations from one modality when auxiliary modalities are given at training time. During inference, the auxiliary modalities are not necessary as in [16, 8]. Shared representation learning learns the representation from one modality and performs the test on the other modality as shown in [47, 35]. The proposed cross-modal embedding aligns in between cross-modal learning and shared representation learning, similar in spirit to [2]. We aim to benefit from different modalities that are correlated to each other. However, rather than learning common representations, we train the model to embed different representations into the shared cross-modal latent space.

3. Proposed Method

Given a scenario with the trajectory data {1, ..., K}} of K traffic agents, we split into a past trajectory for the first obser- vation time steps and a future trajectory for the next time steps, where rep- resents a 2D position of an arbitrary agent i at time t. Assuming that a visual sequence I is available during observation time steps, we compute the optical flow O by running TV-L1 [48] and segmentation map S from DeepLab-V2 [5] trained on Cityscapes [10]. Given {O, S} and , our goal is to generate a trajectory prediction of the target agent k. To achieve this, we build a feature extraction module in Sec. 3.1 upon graph neural networks (GNNs) in order to learn social behaviors of the target k toward all other traffic agents (e.g., pedestrians, vehicles, etc.) as well as surrounding road structures. Then, we derive the objective of the proposed shared cross-modal embedding and show its implementation within CVAE in Sec. 3.2. The encoder is learned to embed into the latent space, conditioning on the observed social behavior . The following decoder reconstructs the future locations using with a latent sample z. Finally, in Sec. 3.3 we provide a solution for mode diversifi-cation addressing the posterior collapse issue.

3.1. Social Behavior Encoding

Input Layer for External Features The importance of external constraints on trajectory prediction is particularly pronounced for traffic agents in driving scenes. To model such environmental influences, the system should be able to recognize each object’s static/dynamic states as well as the semantic context of the scene.

The image sequence I captured during the past time steps is used to generate two types of representations: a set of optical flow images O and a segmentation map S. The temporal changes of the objects from O are processed using the 3D convolutional neural network by extracting temporal representations along the time axis:

where is the learnable weight parameters.

In addition, a pixel-level segmentation map is obtained at the first time step of the given scenario. Among the estimated labels, we only leave the background structures such as road, sidewalk, vegetation, etc. to extract visual features from the stationary environment. The 2D convolutional neural network is used in this stream to take advantage of its spatial feature encoding:

where is the learnable parameters.

We merge the temporal states of static/dynamic objects with the spatial features of the stationary context to generate spatio-temporal features

We further convert to the external feature matrix for the graph. K entities (of size ) are taken from one of cells in a grid of , where the cell location corresponds to each agent i’s original pixel location at time . For example, an agent shown in the first 3232 sub-region of an original 256256 image takes the feature vector from the (1, 1)-th cell in a 88 grid of .

Input Layer for Node Features Using the past motion history of traffic agents, we encode the node features. Assuming the task is to predict the future motion of the target agent k, we first discover its own intent by preforming the following procedure. The past states is encoded into high dimensional feature representations through the multi-layer perceptron (MLP). The encoded features are then combined with the local perception that contains mid-level semantic context (nearby areas of ) from for- mer . By adding spatial locality, interactions of the target toward the local environment further constrain its motion intent. The subsequent LSTM captures the temporal dependency of motion states on the local environment by

where and is the learnable parameters of MLP and LSTM layer, and denotes the hidden state of LSTM at time t. We define the last hidden state as and use it to initialize the node features of the target in the graph.

We run a different feature encoding procedure for the rest of the agents to model their relative motion toward the target agent k as follows:

where and is the learnable parameters of MLP and LSTM, and denotes the hidden state of LSTM at time t. This process is simple yet effective to infer temporal changes of interactive behavior of individual agents. We use

Figure 2: The illustration of the proposed feature encoder. Using the past image sequence, we model spatio-temporal factors given by external environments. The internal and social factors of the target agent is encoded from its past motion as well as surrounding local perceptual context. In the following GNN layer, we model agent-specific social behavior.

the last hidden state of each agent j as for the graph.

GNN Layer The social behavior of the target agent is modeled from each agent’s features and external environment features. We define a graph G = (H, F), where , is a node feature matrix representing K node embeddings of size . is an external feature matrix {1, ..., K}}, where each entity represents outside influence on each node in the graph. Following the general message passing phases [14], we construct a GNN architecture:

where M is the message propagation function that takes the node feature matrix updated by l times of the message passing phase. We initialize using the hidden states obtained from the input layer.

The proposed GNN structure for social behavior modeling can be considered as a family of pair message passing neural networks [3], where the function M takes a concatenation of two nodes as a pair. We design our model on top of this process with an additional graph-level target:

where is the learnable parameters of MLP, Concat(,,) denotes concatenation, k is a target agent, and i and j are the rest of agents. During the message passing phase, the relation between two nodes i and j is encoded with respect to the target node k by considering their external influences , . A summation operation generates messages invariant to the permutation of the nodes. Then, the features of the target node in the graph are updated by a non-linearity function such as ReLU using the messages . After L updates, the output social behavior features are gener-

ated by another MLP during the readout phase:

where is the learnable parameters. We use for a cer- tain input i. For notational brevity, we drop the target indicator k in the following sections. The input layer and GNN layer is illustrated in Figure 2, and details of the network architecture are shown in the supplementary material.

3.2. Shared Cross-Modal Framework

The main contribution of this work is that we propose a cross-modal embedding framework for future prediction. It aims to benefit from the use of multiple input modalities, while keeping the same computational complexity as if the single data type had been used for trajectory prediction. To implement such functionality, we derive our model within the CVAE framework to embed various types of representations into a single shared latent space. Instead of learning the latent space manifold from a single input, several complementary representations extracted from multiple data sources simultaneously characterize the cross-modal space at training time. By jointly learning the same scenario from different input perspectives, the generative process becomes more descriptive, which results in increasing the performance. At test time, a single modal input is used to sample the latent variables from the learned cross-modal space, taking advantages with other sensor modalities.

In the followings, we mathematically derive the objective function of shared cross-modal embedding and extend its derivation toward a generative model conditioned on the input observation.

Joint Optimization The objective of cross-modal embedding is to jointly approximate a real distribution p(z) using a posterior of multiple input sources LiDAR, RGB, ...}, where is the sample data point of input modality i, and z is the latent variable. Exploiting the fact that

the Kullback-Leibler (KL) divergence associated with mul-

By applying Baye’s theorem and employing1, Eqn. (10) can be revised as

Using the definition of the KL divergence and expected value and simple math, Eqn. (11) is converted to

Therefore, maximizing the evidence lower bound (ELBO) of multiple input data over the likelihood jointly maximizes their evidence probability.

Cross-Modal Embedding The proposed cross-modal embedding framework is trained to jointly learn the shared latent space conditioned on multiple input observations such as . The variational lower bound of the log-likelihood can be extended as a conditional form by

where and is implemented as a pair of an encoder and decoder for i-th input modality following the reparameterization trick of CVAE. is the conditional observation. The full derivation is provided in the supplementary material. We draw the loss to minimize the negative ELBO while training the model as follows:

The network parameters of the encoder are learned to minimize the KL divergence between the prior distribution and the approximates . The second term is the log-likelihood of samples, which is considered as the reconstruction loss of the decoder. The decoder generates trajectories using the latent variables z sampled from the prior that is modeled as Gaussian distribution .

3.3. Multi-modal Prediction

In practice, the optimization of VAE and its variants is challenging itself because of the posterior collapse problem. The strong autoregressive power of the decoder often ignores the random variable z sampled from the learned latent space. Thus, the output is dominantly generated using the conditional input c, still satisfying the minimization of the KL divergence and maximization of the log-likelihood in Eqn. (14). Such a problem alleviates the multi-modal nature of future prediction where multiple plausible trajectories are generated given the same past motion. To address posterior collapse, we consider the following challenges: (i) our technique helps to generate diverse responses from the decoder, which enables multi-modal prediction and (ii) it does not physically weaken the decoder to alleviate its prediction capability. In this sense, we design an auxiliary regularizer that makes the decoder to rely on the latent variable.

At training time, we assume that there exist N modes of trajectories for each query. Then, the latent variables are sampled from the normal distribution with the mean and variance , where . We consider the trajectories generated using these latent variables as N modes of prediction outputs. To maximize the diversity among predictions, the pair-wise similarity is evaluated using Gaussian kernel by

where D measures a distance between predictions and with and is the hyper-parameter of this kernel function. We find a maximum similarity and minimize it during training. The regularizer then enforces the model to maximize the diversity among N predicted trajectories through the optimization without losing the prediction capability of the decoder.

As a result, the total objective function of the proposed approach is drawn as follows:

where LiDAR,RGB,...} is an indicator for input data modalities and balances multi-modality and accuracy (is used). To optimize the first term in Eqn. (16), we find of the mode n that best reconstructs the ground truth y. In this way, the log-likelihood in Eqn. (14) encourages the decoder to generate accurate results, while preserving the mode diversity with the regularizer.

4. Experiments

4.1. Input Modalities

Any set of sensory data can be used as input to the proposed framework. For demonstration, however, we use two exemplary data types that are easily accessible from the existing benchmark datasets: (i) LiDAR data provide 3D scanning of the surrounding environment. Using 3D point clouds, we project every single point onto the ground plane in top-down view and predict trajectories of traffic agents in the LiDAR-captured world coordinates. (ii) RGB images captured from a frontal-facing camera provide rich and dense representations. We predict the trajectories from the egocentric perspective in the image space. Unlike relative trajectories in [44, 46], we propose to predict trajectories using the absolute locations, eliminating the effect of uncertain ego-future [30]. We provide its details with our data preparation in the supplementary material.

4.2. Datasets and Evaluation Metrics

Datasets Two benchmark driving datasets (KITTI [13] and H3D [34]) are used to evaluate the proposed approach comparing to self-generated baselines and state-of-the-art methods. The KITTI dataset was introduced for trajectory forecast in [23] to predict future motions of road agents in top-down view, and then [46] found their future locations in frontal view using this dataset. As in [23], we generate a set of trajectory segments with 6 sec long (2 sec for observation and 4 sec for prediction) using Road and City scenes in the Raw subset. We divide all videos into five sets and conduct 5-fold cross validation, following the split of [7]. In addition, the H3D [34] dataset is used to further validate the proposed approach on heterogeneous agents in highly congested urban environments. For evaluation, we divide 160 scenarios of H3D into the training (75%) and test set (25%) and use the same observation / prediction time as KITTI.

Metrics For the performance comparison, we mainly follow the standard evaluation metrics:

• Average Distance Error (ADE) is computed using L2 distance between the predicted trajectory and the ground truth for a certain time duration.

• Final Distance Error (FDE) shows L2 distance between the predicted location and the ground truth at a certain time step.

Both ADE and FDE are reported with 1 sec interval at future time steps. For multi-modal prediction, we sample 20 trajectories and find the best one with a minimum ADE at 4 sec in future. Note that the single- and multi-modal models are respectively denoted by a different suffix S and M.

In addition, we introduce a new metric that measures the rate of prediction success:

• Success Rate (SR) finds the fraction of scenarios where L2 distance between the predicted endpoint and ground truth is within a certain threshold value .

Under the assumption that the prediction would be successful if the error at the endpoint is within a certain threshold,

Table 1: Ablation study on the KITTI [13] dataset. ADE / FDE is reported in meters. Refer to Sec. 4.3 for description.

this metric plots how many scenarios can be considered as ‘success prediction’. SR thus is a more practical evaluation metric that tests the overall robustness of the algorithm.

4.3. Ablative Study

We first demonstrate our design choices through ablative studies conducted in top-down view using KITTI. We evaluate the baseline models on the following components:

• Env: External features (fand f);

• Soc: Social influences of other agents;

• Mul: Multi-modal learning. Fus: multi-modal fusion with feature aggregation, Emb: proposed shared cross-modal embedding.

Table 1 compares ADE and FDE of six baseline models that are designed by adding (+) or dropping (-) these components. When one or more of information is missing, a significant performance drop is observed. The error of the model without any components is particularly larger than others by a huge margin. By considering environmental influences (Env), the performance improves toward long-term prediction (4sec). It clearly demonstrates the effectiveness of the environmental constraints on more distant areas. We observe that adding Soc outperforms previous baselines, which implies the role of social behavior encoding for trajectory prediction with the significant improvement at the short-term time steps. The impressive error drop is found by taking both Env and Soc into account. It demonstrates the validity of the proposed feature extractor. We highlight the efficacy of the use of multiple sensor modalities from Mul, where both Fus and Emb further improve the performance. Interestingly, the proposed cross-modal embedding (Emb) achieves even lower error than fusion-based counterpart (Fus). It indicates that our model benefits from complementary input modalities with cross-modal embedding, even though a single input data is used during inference.

Additionally, we show the efficacy of the proposed regularizer for multi-modal prediction in the bottom of Table 2. Without the proposed regularizer (S-CM 10 w/o reg), the performance improvement of the model with 10 samples is minimal against single-modal prediction (S-CM 1), which is interpreted as a posterior collapse problem. However, the model with the regularizer (S-CM 10) highly improves the

Table 2: Quantitative comparison (ADE / FDE in meters) of our approach with the state-of-the-art methods. The KITTI dataset [38] is used to predict trajectories in top-down view. N denotes the number of samples used.

Table 3: ADE / FDE is evaluated in pixels. The KITTI [13] dataset is used to predict trajectories in frontal view. denotes the evaluation on relative motion from ego-vehicle. N denotes the number of samples used.

accuracy generating diverse output responses. We conclude that the proposed regularizer can ease posterior collapse for future prediction.

4.4. Quantitative Results

We first compare the performance of the proposed approach with the state-of-the-art methods using KITTI. In Table 2, we observe from single-modal prediction (N=1) that our S-CM 1 outperforms all compared single-modal approaches including social interaction oriented methods [1] as well as scene context oriented methods [23, 7]. For multi-modal prediction, the proposed approach (SCM 10) with N=10 already achieves overall lower ADE and FDE than other competitors in top-down view trajectory forecast. By sampling N=20 modes, we improve FDE at 4.0 sec over 19% against [39].

Using the same cross-modal model, we examine the frontal view prediction capability in Table 3. Note that Conv1D [44] and FVL [46] predicts relative motion with respect to the future ego-motion. Their poor performance might be caused by the prediction difficulties with unknown ego-future. Although the proposed method (S CM 1) further improved the accuracy without affecting the inference time, the effect seems less significant compared to that

Table 4: Quantitative results (ADE / FDE) are reported in meters. We use H3D [34] to evaluate the proposed method in top-down view. N denotes the number of samples used.

Table 5: Our approach is evaluated on ADE / FDE (in pixels) using the H3D [34] dataset. The proposed absolute motions (with ego-future elimination) are used to compute errors in frontal view. N denotes the number of samples used.

shown in top-down view (Table 1). Our insight is as follows: (i) the use of complementary features obtained from different input modality is not as impactful as it was for top-down prediction; and (ii) the performance improvement achieved by other aspects (e.g., social behavior, semantic context, etc.) is already exceptional in frontal view, which makes the improvement with embedding less compelling. Nevertheless, the proposed method with cross-modal embedding generally shows higher accuracy against others.

We further evaluate our work using the H3D dataset in its highly congested environments. In top-down view as in Table 4, we found that the proposed model with a single sample (S-CM 1) already achieves the lower error than most of methods including very recent graph-based models ([31] w/o scene and [39] w/ scene). Our explicit modeling of relational interactions together with cross-modal embedding enables us to explore more discriminative behavior representations over these graph-based methods. The performance is further improved by sampling multiple predictions with the regularizer (S-CM 20). Compared to the best state-of-the-art method [26] that finds the dynamic evolution of interactions, our work improves the performance over 10% on FDE at 4.0 sec. Such lower errors demonstrates the generation of highly diverse yet acceptable future motions using our model, considering the road topology.

Subsequently, we evaluate our trajectory prediction framework for the task of frontal view forecast. In Table 5,

Figure 3: We visualize the top-1 prediction from highly interactive scenarios among heterogeneous traffic agents, such as human-human, human-vehicle, and vehicle-vehicle interactions.

Figure 4: The success rate (SR) is plotted as a fraction of success prediction scenarios with respect to maximum allowed distance error, which is indicative of overall robustness of algorithms. SR is evaluated in top-down view on FDE at 4.0 sec. (a) KITTI and (b) H3D.

we observe that the performance of our single-modal prediction model (S-CM 1) is on par with multi-modal prediction model of Social-GAN (S-GAN with N=20). It implies that the prediction capability of the proposed framework is being at the level of the state-of-the-art. The significant improvement of error from our multi-modal prediction model S-CM 20 further demonstrates the effectiveness of our objective function for optimization.

4.5. Evaluation with Success Rate

The standard evaluation metrics such as ADE and FDE do not capture the success or failure of predictions. We thus introduce SR that plots the proportion of scenarios that can be considered as ‘successful prediction’ with respect to the definition of success. We use the error threshold on the xaxis and measure the rate of success scenarios by FDE at 4.0 sec. Figure 4 compares our approach with two state-of-the-art methods [31, 39]. We observe from 4a that our approach performs better than others in terms of the correctness of predictions. Assuming that the real driving application is designed with a small prediction tolerance (), our model is more reliable and credible with considerably higher success rate (63% compared to [39] of 33% or [31] of 29%). We also plot SR using the H3D dataset in 4b, which indicates that our prediction model can achieve much smaller errors in the majority of scenarios. Our method shows consistently higher success rate, validating the robustness of our prediction capability.

4.6. Qualitative Results in Top-down View

Figure 3 visualizes the top-1 prediction result of the proposed approach. Each scenario contains the heterogeneous agents (i.e., cars, bus, pedestrians, cyclist, etc.) interactive one to another. We robustly forecast their future motions by taking advantages of the proposed social behavior modeling and cross-modal embedding. In between pedestrians, our approach models their motion behaviors and generates socially acceptable trajectories (dotted oval in the second column). In the last column, our model predicts that the car would turn left, which influences the behavior of on-coming vehicle that slows its speed (i.e., yielding; dotted arrow). We conclude that the proposed graph accordingly considers relational interactions while predicting future motions. We provide the results of 20 prediction samples as well as qualitative results in frontal view in the supplementary material.

5. Conclusion

We proposed a solution to future trajectory forecast in driving scenarios. Assuming that the multiple sensory data is available for autonomous driving, our approach can ben-efit from the model trained using multiple input modalities. First, the GNN-based feature encoder extracts social behaviors of the target agent, considering its interactions toward all other traffic agents as well as surrounding road structures. Then, the relational behaviors obtained from multiple perspectives are embedded into a shared cross-modal latent space. We provided its derivation that jointly optimizes objective functions using the generative variational models. Finally, we designed an auxiliary regularizer to ease the posterior collapse problem for future prediction. We analyzed the significance of the proposed approach through the extensive evaluation, showing the improvement of the performance against the state-of-the-art methods.

References

[1] Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social lstm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 961–971, 2016.

[2] Yusuf Aytar, Lluis Castrejon, Carl Vondrick, Hamed Pirsi- avash, and Antonio Torralba. Cross-modal scene networks. IEEE transactions on pattern analysis and machine intelligence, 40(10):2303–2314, 2017.

[3] Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. In Advances in neural information processing systems, pages 4502–4510, 2016.

[4] Apratim Bhattacharyya, Mario Fritz, and Bernt Schiele. Long-term on-board prediction of people in traffic scenes under uncertainty. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4194– 4202, 2018.

[5] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834–848, 2018.

[6] Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1907–1915, 2017.

[7] Chiho Choi and Behzad Dariush. Looking to relations to future trajectory forecast. In Computer Vision (ICCV), 2019 IEEE International Conference on. IEEE, 2019.

[8] Chiho Choi, Sangpil Kim, and Karthik Ramani. Learning hand articulations by hallucinating heat distribution. In Proceedings of the IEEE International Conference on Computer Vision, pages 3104–3113, 2017.

[9] Chiho Choi, Srikanth Malla, Abhishek Patil, and Joon Hee Choi. Drogon: A trajectory prediction model based on intention-conditioned behavior reasoning. In Proceedings of the Conference on Robot Learning (CoRL), 2020.

[10] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016.

[11] Nachiket Deo and Mohan M Trivedi. Multi-modal trajec- tory prediction of surrounding vehicles with maneuver based lstms. In 2018 IEEE Intelligent Vehicles Symposium (IV), pages 1179–1184. IEEE, 2018.

[12] Davi Frossard and Raquel Urtasun. End-to-end learning of multi-sensor 3d tracking by detection. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 635–642. IEEE, 2018.

[13] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The Inter-

national Journal of Robotics Research, 32(11):1231–1237, 2013.

[14] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In ICML, 2017.

[15] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.

[16] Saurabh Gupta, Judy Hoffman, and Jitendra Malik. Cross modal distillation for supervision transfer. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2827–2836, 2016.

[17] Qishen Ha, Kohei Watanabe, Takumi Karasawa, Yoshitaka Ushiku, and Tatsuya Harada. Mfnet: Towards real-time semantic segmentation for autonomous vehicles with multispectral scenes. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5108–5115. IEEE, 2017.

[18] Xin Huang, Stephen McGill, Brian C Williams, Luke Fletcher, and Guy Rosman. Uncertainty-aware driver trajectory prediction at urban intersections. 2019 IEEE International Conference on Robotics and Automation (ICRA), 2019.

[19] Yingfan Huang, Huikun Bi, Zhaoxin Li, Tianlu Mao, and Zhaoqi Wang. Stgat: Modeling spatial-temporal interactions for human trajectory prediction. In Proceedings of the IEEE International Conference on Computer Vision, pages 6272– 6281, 2019.

[20] Ashesh Jain, Avi Singh, Hema S Koppula, Shane Soh, and Ashutosh Saxena. Recurrent neural networks for driver activity anticipation via sensory-fusion architecture. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 3118–3125. IEEE, 2016.

[21] Vineet Kosaraju, Amir Sadeghian, Roberto Mart´ın-Mart´ın, Ian Reid, Hamid Rezatofighi, and Silvio Savarese. Socialbigat: Multimodal trajectory forecasting using bicycle-gan and graph attention networks. In Advances in Neural Information Processing Systems, pages 137–146, 2019.

[22] Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven L Waslander. Joint 3d proposal generation and object detection from view aggregation. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1–8. IEEE, 2018.

[23] Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B Choy, Philip HS Torr, and Manmohan Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 336–345, 2017.

[24] Jiachen Li, Hengbo Ma, and Masayoshi Tomizuka. Conditional generative neural system for probabilistic trajectory prediction. In 2019 IEEE Conference on Robotics and Systems (IROS), 2019.

[25] Jiachen Li, Hengbo Ma, and Masayoshi Tomizuka. Interaction-aware multi-agent tracking and probabilistic behavior prediction via adversarial learning. In 2019 IEEE In-

ternational Conference on Robotics and Automation (ICRA). IEEE, 2019.

[26] Jiachen Li, Fan Yang, Masayoshi Tomizuka, and Chiho Choi. Evolvegraph: Multi-agent trajectory prediction with dynamic relational reasoning. In Advances in neural information processing systems, 2020.

[27] Ming Liang, Bin Yang, Yun Chen, Rui Hu, and Raquel Urta- sun. Multi-task multi-sensor fusion for 3d object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7345–7353, 2019.

[28] Yuexin Ma, Xinge Zhu, Sibo Zhang, Ruigang Yang, Wen- ping Wang, and Dinesh Manocha. Trafficpredict: Trajectory prediction for heterogeneous traffic-agents. 2019.

[29] Srikanth Malla, Behzad Dariush, and Chiho Choi. Titan: Future forecast using action priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11186–11196, 2020.

[30] Srikanth Malla, Isht Dwivedi, Behzad Dariush, and Chiho Choi. Nemo: Future object localization using noisy ego priors. arXiv preprint arXiv:1909.08150, 2019.

[31] Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian Claudel. Social-stgcnn: A social spatio-temporal graph convolutional neural network for human trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14424– 14432, 2020.

[32] Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multimodal deep learning. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 689–696, 2011.

[33] Seong Hyeon Park, ByeongDo Kim, Chang Mook Kang, Chung Choo Chung, and Jun Won Choi. Sequence-to-sequence prediction of vehicle trajectory via lstm encoderdecoder architecture. In 2018 IEEE Intelligent Vehicles Symposium (IV), pages 1672–1678. IEEE, 2018.

[34] Abhishek Patil, Srikanth Malla, Haiming Gang, and Yi-Ting Chen. The h3d dataset for full-surround 3d multi-object detection and tracking in crowded urban scenes. In 2019 International Conference on Robotics and Automation (ICRA), pages 9552–9557. IEEE, 2019.

[35] Yuxin Peng, Xin Huang, and Jinwei Qi. Cross-media shared representation by hierarchical learning with multiple deep networks. In IJCAI, pages 3846–3853, 2016.

[36] Nicholas Rhinehart, Kris M Kitani, and Paul Vernaza. R2p2: A reparameterized pushforward policy for diverse, precise generative path forecasting. In Proceedings of the European Conference on Computer Vision (ECCV), pages 772–788, 2018.

[37] Nicholas Rhinehart, Rowan McAllister, Kris Kitani, and Sergey Levine. Precog: Prediction conditioned on goals in visual multi-agent settings. In Proceedings of the IEEE International Conference on Computer Vision, pages 2821–2830, 2019.

[38] Alexandre Robicquet, Amir Sadeghian, Alexandre Alahi, and Silvio Savarese. Learning social etiquette: Human trajectory understanding in crowded scenes. In European conference on computer vision, pages 549–565. Springer, 2016.

[39] Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Multi-agent generative trajectory forecasting with heterogeneous data for control. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.

[40] Christoph Sch¨oller, Vincent Aravantinos, Florian Lay, and Alois Knoll. What the constant velocity model can teach us about pedestrian motion prediction. IEEE Robotics and Automation Letters, 5(2):1696–1703, 2020.

[41] Abhinav Valada, Rohit Mohan, and Wolfram Burgard. Self- supervised model adaptation for multimodal semantic segmentation. International Journal of Computer Vision, pages 1–47, 2019.

[42] Anirudh Vemula, Katharina Muelling, and Jean Oh. Social attention: Modeling attention in human crowds. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1–7. IEEE, 2018.

[43] Yanyu Xu, Zhixin Piao, and Shenghua Gao. Encoding crowd interaction with deep neural network for pedestrian trajectory prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5275– 5284, 2018.

[44] Takuma Yagi, Karttikeya Mangalam, Ryo Yonetani, and Yoichi Sato. Future person localization in first-person videos. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.

[45] Xiaodong Yang, Pavlo Molchanov, and Jan Kautz. Multilayer and multimodal fusion of deep neural networks for video classification. In Proceedings of the 24th ACM international conference on multimedia, pages 978–987. ACM, 2016.

[46] Yu Yao, Mingze Xu, Chiho Choi, David J Crandall, Ella M Atkins, and Behzad Dariush. Egocentric vision-based future vehicle localization for intelligent driving assistance systems. In IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2019.

[47] Dong Yi, Zhen Lei, and Stan Z Li. Shared representation learning for heterogenous face recognition. In 2015 11th IEEE international conference and workshops on automatic face and gesture recognition (FG), volume 1, pages 1–7. IEEE, 2015.

[48] Christopher Zach, Thomas Pock, and Horst Bischof. A du- ality based approach for realtime tv-l 1 optical flow. In Joint pattern recognition symposium, pages 214–223. Springer, 2007.

[49] Pu Zhang, Wanli Ouyang, Pengfei Zhang, Jianru Xue, and Nanning Zheng. Sr-lstm: State refinement for lstm towards pedestrian trajectory prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12085–12094, 2019.

[50] Wenwei Zhang, Hui Zhou, Shuyang Sun, Zhe Wang, Jian- ping Shi, and Chen Change Loy. Robust multi-modality multi-object tracking. In Proceedings of the IEEE International Conference on Computer Vision, pages 2365–2374, 2019.

Supplementary Material A. Derivation of Cross-Modal Embedding

In this section, we provide a complete derivation of the objective function of shared cross-modal embedding. Based on Eqn. (9), the KL divergence approximating the conditional posteriors is written as follows:

where y and c respectively denotes the data point and condition. Assuming multiple input sources LiDAR, RGB, ...} are available, we derive the objective for an arbitrary number of modalities, yielding

which is still under the premise that the KL divergence is non-negative. We apply Baye’s theorem

to Eqn. (18) and employ

yielding

By transferring integral to the other side, we get

Therefore, the evidence lower bound of multiple data over the conditional likelihood is given by

which completes the proof.

B. Dataset Preprocessing

B.1. Frontal View Input

Segmentation Map Frontal view RGB images (top row in Figure. 5) are used to estimate the semantic labels in the scene. We first run the DeepLab-V2 [5] model trained on the Cityscapes dataset [10]. Then, we leave the background labels with stationary structures (i.e., road, sidewalk, building, etc.) to get the background map as shown in the second row of Figure 5. For the static stream, we directly use the background map at the first observation time of each scenario as segmentation input S to the model.

Ego-future Elimination To eliminate the effect of ego-future from frontal view prediction, we introduce the absolute coordinate where the motion of traffic agents is not influenced by the ego-vehicle. To eliminate the ego-motion for a dynamic stream, we define the new coordinates at the first observation time step t = 1 for every trajectory segments (of length ), and all foreground objects are projected into this space. For this, we conduct the following procedure. First, the point cloud in the world coordinates is projected into the image space to grab the corresponding RGB information. Next, we transform the point cloud to the first frame using GPS/IMU position estimates. Finally, the transformed point cloud is projected into the blank image and displayed using previously acquired RGB information. The transformed RGB images generated from this procedure are shown in the third row of Figure 5.

Optical Flow Images We first locate the ground truth bounding box in the image space. Then, we go through ego-future elimination steps using these images with corresponding point clouds. Output foreground images (last row in Figure 5) are used to compute optical flow O by running the TV-L1 [48] algorithm. The positions of each traffic agent is also computed by this procedure. Images have a dimension of 414 125.

B.2. Top-down View Image

Segmentation Map From each trajectory segment, we grab the segmentation label for the point cloud from the RGBbased background map. Then, the point cloud is transformed onto the top-down image space that is discretized with a resolution of 0.5 m as illustrated in the second row

Figure 5: Frontal view input.

Figure 6: Top-down view input.

of Figure 6. We use the map at time t = 1 as S in top-down view.

Ego-future Elimination Similar to frontal view images, we first transform each point cloud to the local coordinates at t = 1 using GPS/IMU position estimates. The transformed point clouds are projected into the top-down view image space with a resolution of 0.5 m. The third row of Figure 6 shows the transformed point clouds that are created using the original point cloud in the first row of Figure 6.

Optical Flow Images The ground truth bounding box of objects are first processed to eliminate the effect of ego-future. Then, they are drawn in the top-down view image space as displayed in the last row of Figure 6. The final output has a dimension of that corresponding to 80 m to the longitudinal direction and to the lateral direction.

B.3. Absolute vs Relative Motions in Frontal View

In Figure 8, we compare the absolute motions introduced in the proposed method with the relative motions used in other approaches. The absolute trajectory (in red dots) is intuitive by eliminating the ego-future. In contrast, the relative trajectory (in orange dots) is not interpretable without extra information of future ego-motion that should be separately predicted (as in [18]), considering the uncertainty [30].

C. Additional Evaluation

C.1. Qualitative Results in Frontal View

We additionally show the qualitative results of the proposed method in frontal view using the KITTI [13] dataset. As in Figure 7, our frontal view prediction is based on the

Figure 7: Additional qualitative results evaluated using KITTI in frontal view.

Figure 8: Comparison between absolute (red trajectory) and relative (orange trajectory) motions of a cyclist.

absolute locations (with ego-future elimination) in the local coordinates of the first frame of each trajectory segment. The proposed approach recognizes the road layouts and accurately predict interactive future motions of different types of road agents. Note that we visualize top-1 prediction in this figure.

C.2. Quantitative Results in Top-down View

Table 6 shows an additional study on extra metrics. We provide FDE at 4.0 sec with standard deviation computed from five experiments. It demonstrates that our approach consistently provides robust prediction capabilities with lower standard deviation.

C.3. Additional Top-down View Videos

We provide additional video clips (scenario1.mp4, sce-nario2.mp4, scenario3.mp4) to visualize entire 20 predictions of all traffic agents in the scene. The videos are generated using three interactive scenarios of the H3D dataset. Please check attached videos for additional qualitative evaluation (Color codes: Blue - past observation, Green -ground truth, Red - our predictions).

D. Implementation

We share details of our model architectures.

Table 6: Additional study using H3D. FDE at 4.0 sec with standard deviation is reported in meters.

D.1. Social Behavior Encoding

Input Layer for External Features We design a 3D CNN module to extract temporal representations . This module is constructed using 4 sets of [3D conv - 1D conv]. The first two sets have 3D conv layer with a fil-ter size of 53 and stride of 31, and the last two 3D conv layers have a stride of 21. The final layer merges time channels as one, which results in the output feature of size batch512 for frontal view and that of batch512 for top-down view.

The spatial features are extracted from the stationary environment using a 2D CNN module . We use 4 sets of [2D conv - 1D conv] where the filter sizes are same as those of without depth channel of 3D conv. Note that we use the output of the third set as mid-level semantic context to capture the local environment while encoding the motion of the target agent k .

Input Layer for Node Features Past motion history of the target agent k is encoded into high-dimensional feature representations through MLP with 2 fully connected layers. The resulting features of size batch512 are added to the corresponding local perception . We run LSTM and use the last hidden state of size batchto initialize the node feature of the agent .

For the rest of agents, we follow the similar procedure using MLP and LSTM with the relative motion information. Each individual last hidden state of size batchis used to initialize .

GNN Layer The message generation is implemented using three layers of MLP, all with size 256. We use an additional layer of MLP with a same size during the readout phase.

D.2. Cross-Modal Embedding

Encoder The ground-truth future motion of size batch2 is first reshaped to batch20 and processed through six fully connected layers, each with output size 1024, 6400, 25600, 6400, 1024, and 512. Each layer has a subsequent leaky ReLU layer, and the intermediate features are conditioned by . From the output of the penultimate layer, the first 256 dimension is used as mean and the next 256 dimension is used as standard deviation.

Decoder The sampled latent variable is decoded through the 8 fully connected layers. The output size is 1024, 6400, 25600, 102400, 25600, 6400, 1024, and 20, respectively. All layers come together with a ReLU activation function, and each intermediate output is conditioned by .

Designed for Accessibility and to further Open Science