Joint distribution matching (JDM) problem, which aims to learn bidirectional mappings to match joint distributions of two domains, occurs in many machine learning and computer vision applications. This problem, however, is very difficult due to two critical challenges: (i) it is often difficult to exploit sufficient information from the joint distribution to conduct the matching; (ii) this problem is hard to formulate and optimize. In this paper, relying on optimal transport theory, we propose to address JDM problem by minimizing the Wasserstein distance of the joint distributions in two domains. However, the resultant optimization problem is still intractable. We then propose an important theorem to reduce the intractable problem into a simple optimization problem, and develop a novel method (called Joint Wasserstein Distribution Matching (JWDM)) to solve it. In the experiments, we apply our method to unsupervised image translation and cross-domain video synthesis. Both qualitative and quantitative comparisons demonstrate the superior performance of our method over several state-of-the-arts.

Joint distribution matching (JDM) seeks to learn the bidirectional mappings to match the joint distributions of unpaired data in two different domains. Note that this problem has many applications in computer vision, such as image translation [39, 22] and video synthesis [3, 33]. Compared to the learning of marginal distribution in each individual domain, learning the joint distribution of two domains is more difficult and has the following two challenges.

The first key challenge, from a probabilistic modeling perspective, is how to exploit the joint distribution of unpaired data by learning the bidirectional mappings between two dif- ferent domains. In the unsupervised learning setting, there are two sets of samples drawn separately from two marginal distributions in two domains. Based on the coupling theory [21], there exist an infinite set of joint distributions given two marginal distributions, and thus infinite bidirectional mappings between two different domains may exist. Therefore, directly learning the joint distribution without additional information between the marginal distributions is a highly ill-posed problem. Recently, many studies [39, 36, 16] have been proposed to learn the mappings in two domains separately, which cannot learn cross-domain correlations. Therefore, how to exploit sufficient information from the joint distribution still remains an open question.

The second critical challenge is how to formulate and optimize the joint distribution matching problem. Most existing methods [19, 26] do not directly measure the distance between joint distributions, which may result in the distribution mismatching issue. To address this, one can directly apply some statistics divergence, e.g., Wasserstein distance, to measure the divergence of joint distributions. However, the optimization may result in intractable computational cost and statistical difficulties [8]. Therefore, it is important to design a new objective function and an effective optimization method for the joint distribution matching problem.

In this paper, we propose a Joint Wasserstein Distribution Matching (JWDM) method. Specifically, for the first challenge, we use the optimal transport theory to exploit geometry information and correlations between different domains. For the second challenge, we apply Wasserstein distance to measure the divergence between joint distributions in two domains, and optimize it based on an equivalence theorem.

The contributions are summarized as follows:

• Relying on optimal transport theory, we propose a novel JWDM to solve the joint distribution matching problem. The proposed method is able to exploit sufficient information for JDM by learning correlations between domains instead of learning from individual domain.

• We derive an important theorem so that the intractable primal problem of minimizing Wasserstein distance between joint distributions can be readily reduced to a simple optimization problem (see Theorem 1).

• We apply JWDM to unsupervised image translation and cross-domain video synthesis. JWDM obtains highly qualitative images and visually smooth videos in two domains. Experiments on real-world datasets show the superiority of the proposed method over several state-of-the-arts.

Image-to-image translation. Recently, Generative adversarial networks (GAN) [10, 4, 29, 5, 11], Variational Auto-Encoders (VAE) [18] and Wasserstein Auto-Encoders (WAE) [31] have emerged as popular techniques for the image-to-image translation problem. Pix2pix [13] proposes a unified framework which has been extended to generate high-resolution images [34]. Besides, recent studies also attempt to tackle the image-to-image problem on the unsupervised setting. CycleGAN [39], DiscoGAN [16] and DualGAN [36] minimize the adversarial loss and the cycle-consistent loss in two domains separately, where the cycle-consistent constraint helps to learn cross-domain mappings. SCANs [20] also adopts this constraint and proposes to use multi-stage translation to enable higher resolution image-to-image translation.

To tackle the large shape deformation difficulty, Gokaslan et al. [9] designs a discriminator with dilated convolutions. More recently, HarmonicGAN [37] introduces a smoothness term to enforce consistent mappings during the translation. Besides, Alami et al. [1] introduces unsupervised attention mechanisms to improve the translation quality. Unfortunately, these works minimize the distribution divergence of images in two domains separately, which may induce a joint distribution mismatching issue. Other methods focus on learning joint distributions. CoGAN [23] learns a joint distribution by enforcing a weight-sharing constraint. However, it uses noise variable as input thus can not control the outputs, which limits it’s effect of applications. Moreover, UNIT [22] builds upon CoGAN by using a shared-latent space assumption and the same weight-sharing constraint.

Video synthesis. In this paper, we further consider cross-domain video synthesis problem. Since existing image-to-image methods [39, 16, 36] cannot be directly used in the video-to-video synthesis problem, we combine some video frame interpolation methods [38, 14, 25, 24] to synthesize video. Note that one may directly apply existing UNIT [22] technique to do cross-domain video synthesis, which, however, may result in significantly incoherent videos with low visual quality. Recently, a video-to-video translation method [3] translates videos from one domain to another domain, but it cannot conduct video frame interpolation. Moreover, Vid2vid[33] proposes a video-to-video synthesis method, but it cannot work for the unsupervised setting.

Notations. We use calligraphic letters (e.g., X) to denote the space, capital letters (e.g., X) to denote random variables, and bold lower case letter (e.g., x) to be their corresponding values. We denote probability distributions with capital letters (i.e., P(X)) and the corresponding densities with bold lower case letters (i.e., p(x)). Let be the domain, be the marginal distribution over X, and P(X) be the set of all the probability measures over X.

Wasserstein distance. Recently, optimal transport [32] has many applications [2, 35]. Based on optimal transport theory, we provide the definition of Wasserstein distance and then develop our proposed method. Given the distributions and , where is generated by a generative model G, the Monge-Kantorovich problem is to find a transport plan (i.e., a joint distribution)

where is a cost function and set of all joint distributions with the marginals and , respectively. In next section, we use Wasserstein distance to measure the distribution divergence and develop our method.

In this paper, we focus on the joint distribution matching problem which has broad applications like unsupervised image translation [39, 22] and cross-domain video synthesis [33, 3]. For convenience, we first give a formal definition of joint distribution and the problem setting as follows.

Joint distribution matching problem. Let and be two domains, where are the marginal distributions over X and Y, respectively. In this paper, we seek to learn two cross-domain mappings and to construct two joint distributions defined as:

where . In this paper, our goal is to match the joint distributions as follows.

Joint distribution matching problem. Relying on optimal transport theory, we employ Wasserstein distance (1) to measure the distance between joint distributions and , namely . To match these two joint distributions, we seek to learn the cross-domain mappings f and g by minimizing the following Wasserstein distance, i.e.,

where is the set of couplings composed of all joint distributions, and c is any measurable cost function. In the following, we propose to show how to solve Problem (3).

Figure 1. General training and testing scheme of Joint Wasserstein Distribution Matching for unsupervised image translation and cross-domain video synthesis. Training scheme: given real data X and Y , we learn cross-domain mappings (i.e., generate samples such that the generated distributions can be close to real distribution. Moreover, latent distributions generated by Auto-Encoder (i.e., ) should be close to each other. Testing scheme: given two video frames (i.e., ) in the source domain, we extract their embeddings , then we synthesize two videos in two domains by mapping linear interpolations to the corresponding frames. More details for these two tasks can be found in Subsection 4.1.

In practice, directly optimizing Problem (3) would face two challenges. First, directly optimizing Problem (3) would incur intractable computational cost [8]. Second, how to choose an appropriate cost function is very difficult. To address these, we seek to reduce Problem (3) into a simpler optimization problem using the following theorem.

Theorem 1 (Problem equivalence) Given two deterministic models and as Dirac measures, i.e., and for all , we can rewrite Problem (3) as follows:

where we define and as the sets of all probabilistic encoders, respectively, where Q satisfies the set

Proof See supplementary materials for the proof.

Optimization problem. Based on Theorem 1, we are able to optimize Wasserstein distance by optimizing the reconstruction losses of two auto-encoders when the generated joint distributions can match real joint distributions. Specifi-cally, let be the reconstruction losses of two auto-encoders (and ), respectively, and let the models be , we optimize Problem (4) as follows:

where and are real distributions, and and are generated distributions. By minimizing Problem (5), the two reconstruction losses and will be minimized, meanwhile the generated distributions can match real distributions. The details of objective function and optimization are given below.

Objective function. With the help of Theorem 1, intractable Problem (3) can be turned into a simple optimization problem. To optimize Problem (5), we propose to enforce the constraints (i.e., ) by introducing distribution divergences to measure the distance between generated and real distribution. Specifically, given two AutoEncoders and , and let , we can instead optimize the following problem for joint distribution matching problem:

where is arbitrary distribution divergence between two distributions, and are hyper-parameters. Note that the proposed objective function involves two kinds of functions, namely the reconstruction loss (i.e., and distribution divergence (i.e., and ). We depict each of them as follows.

(i) Reconstruction loss. To optimize Problem (6), the reconstruction loss should be small, it means that the reconstruction of any input should be close to the input for the source and the target domain. As shown in Figure 1, the reconstruction of x in the source domain can be derived from Auto-Encoder (i.e., ) and cycle mapping (i.e., ). Similarly, the reconstruction in the target domain can be learned in the same way. Taking the source domain as an example, given an input x, we minimize the following reconstruction loss

where is the empirical expectation. Note that the first term is the Auto-Encoders reconstruction loss and the second term is the cycle consistency loss [39, 22]. For the target domain, the loss can be constructed similarly.

(ii) Distribution divergence. From Theorem 1, the constraints enforce that the generated distributions should be

Algorithm 2 Inference for cross-domain video synthesis.

Input: Testing data in the source domain:

equal to the real distributions in the source and target domain. Moreover, the latent distributions generated by two Auto-Encoders (i.e., ) should be close to the prior distribution. Therefore, there are three distribution divergence for the optimization, As shown in Figure 1. Note that they are not limited to some specific distribution divergence, e.g., Adversarial loss (such as original GAN and WGAN [2]) and Maximum Mean Discrepancy (MMD), etc. In this paper, we use original GAN to measure the divergence between real and generated distribution. Taking as an example, we minimize the following loss function,

where x denotes a sample drawn from real distribution denotes a sample drawn from generated distribution , and is a discriminator w.r.t. the source domain. Similarly, the losses can be constructed in the same way. 1

4.1. Connection to Applications

JWDM can be applied to unsupervised image translation and cross-domain video synthesis problems.

(i) Unsupervised image translation. As shown in Figure 1, we can apply JWDM to solve unsupervised image translation problem. Specifically, given real data x and y in the source and target domain, respectively, we learn cross-domain mappings (i.e., ) to generate samples . By minimizing the distribution divergence we learn cross-domain mappings

where are Encoders, and are Decoders. The detailed training method is shown in Algorithm 1.

(ii) Cross-domain video synthesis. JWDM can be applied to conduct cross-domain video synthesis, which seeks to produce two videos in two different domains by performing linear interpolation based video synthesis [14]. Specifically, given two input video frames and in the source domain, we perform a linear interpolation between two embeddings extracted from and . Then the interpolated latent representations are decoded to the corresponding frames in the source and target domains (see Figure 1 and Algorithm 2). The interpolated frames in two domains are

where denotes the interpolated frame in the source domain and denotes the translated frame in the target domain.

In this section, we first apply our proposed method on unsupervised image translation to evaluate the performance of joint distribution matching. Then, we further apply JWDM on cross-domain video synthesis to evaluate the interpolation performance on latent space.

Implementation details. Adapted from CycleGAN [39], the Auto-Encoders (i.e. and ) of JWDM are composed of two convolutional layers with the stride size of two for downsampling, six residual blocks, and two transposed convolutional layers with the stride size of two for upsampling. We leverage PatchGANs [13] for the discriminator network. 2 We follow the experimental settings in CycleGAN. For the optimization, we use Adam solver [17] with a mini-batch size of 1 to train the models, and use a learning rate of 0.0002 for the first 100 epochs and gradually decrease it to zero for the next 100 epochs. Following [39], we set in Eqn. (6). By default, we set in our experiments (see more details in Subsection 5.3).

Datasets. We conduct experiments on two widely used benchmark datasets, i.e., Cityscapes [7] and SYNTHIA [27].

• Cityscapes [7] contains street scene video of several German cities and a portion of ground truth semantic segmentation in the video. In the experiment, we perform scene segmentation translation in the unsupervised setting.

• SYNTHIA [27] contains many synthetic videos in different scenes and seasons (i.e., spring, summer, fall and winter). In the experiment, we perform season translation from winter to the other three seasons.

Evaluation metrics. For quantitative comparisons, we adopt Inception Score (IS), Fr´echet Inception Distance (FID) and video variant of FID (FID4Video) to evaluate the generated samples.

• Inception Score (IS) [28] is a widely used metric for generative models. By using the class predition information of Inception-V3 [30], IS can be used to evaluate the quality and diversity of the generated samples.

• Fr´echet Inception Distance (FID) [12] is another widely used metric for generative models. FID can evaluate the quality of the generated images because it captures the similarity of the generated samples to real ones and correlates well with human judgements.

• Video variant of FID (FID4Video) [33] evaluates both visual quality and temporal consistency of synthesized videos. Specifically, we use a pre-trained video recognition CNN (i.e. I3D [6]) as a feature extractor. Then, we use CNN to extract a spatio-temporal feature map for each video. Last, we calculate FID4Video using the formulation in [33].

In general, the higher IS means the better quality of translated images or videos. For both FID and FID4Video, the lower score means the better quality of translated images or videos.

5.1. Results on Unsupervised Image Translation

In this section, we compare the performance of the proposed JWDM with the following baseline methods on unsupervised image translation.

• CoGAN [23] conducts image translation by finding a latent representation that generates images in the source domain and then rendering this latent representation into the target domain.

• CycleGAN [39] uses an adversarial loss and a cycle-consistent loss to learn a cross-domain mapping for unsupervised image-to-image translation.

• UNIT [22] learns the joint distribution of images in different domains. It is trained with the images from the marginal distributions in the individual domain.

• AGGAN [1] introduces unsupervised attention mechanisms that are jointly trained with the generators and discriminators to improve the translation quality.

Comparisons with state-of-the-art methods. We conduct experiments on five image translation tasks on Cityscapes and SYNTHIA, such as scene2segmentation, seg-mentation2scene, winter2spring, winter2summer and win-ter2fall. Quantitative and visual results are shown in Table 1 and Figure 2, respectively.

Figure 2. Comparisons with different methods for unsupervised image-to-image translation on Cityscape and SYNTHIA.

Table 1. IS and FID scores of different methods for unsupervised image-to-image translation on Cityscape and SYNTHIA.

Quantitative Comparisons. From Table 1, JWDM is able to learn a good joint distribution and consistently outperforms the considered methods on all the image translation tasks. On the contrary, CoGAN gets the worst results because it cannot directly translate the input images but samples latent variables to generate the target images. For CycleGAN and AGGAN, because they cannot exploit the cross-domain information, the translated images may lose some information. Besides, UNIT is hard to learn a good joint distribution so that the results are worse than our JWDM.

Visual Comparisons. From Figure 2. CoGAN translates images with the worst quality and contains noises. The translated images of CycleGAN and AGGAN are slightly worse and lose some information. Besides, the translation result of UNIT is better than the above three comparison methods, but its translated image is not sharp enough. Compared with these methods, JWDM generates more accurate translated images with sharp structure by exploiting sufficient cross-domain information. These results demonstrate the effectiveness of our method in directly learning the joint distribution between different domains.

5.2. Results on Cross-domain Video Synthesis

In this experiment, we apply our proposed method to cross-domain video synthesis by performing video frame interpolation and translation in two different domains. Specifi-cally, we use the first and ninth video frames in the source domain as input and interpolate the intermediate seven video frames in two domains simultaneously.

We consider several state-of-the-art baseline methods, including UNIT [22] with latent space interpolation and several constructed variants of CycleGAN [39] with different

Figure 3. Comparisons of different methods for scene segmentation translation on Cityscapes. We first synthesize a video of street scene and then translate it to the segmentation domain (Top), and vice versa for the mapping from segmentation to street scene (Bottom).

Table 2. Comparisons of different methods for cross-domain image-to-video synthesis on Cityscapes and SYNTHIA.

view synthesis methods. For these constructed baselines, we first conduct video frame interpolation and then perform image-to-image translation. All considered baselines are shown as follows.

• UNIT [22] is an unsupervised image-to-image translation method. Because of the usage of a shared latent space, UNIT is able to conduct interpolations on the latent code of two domains to perform video synthesis.

• DVF-Cycle combines the view synthesis method DVF [24] with CycleGAN. To be specific, DVF produces videos by video interpolation in one domain, and CycleGAN translates the videos from one domain to another domain.

• DVM-Cycle uses a geometrical view synthesis DVM [14] for video synthesis, and then uses CycleGAN to translate the generated video to another domain.

• AdaConv-Cycle combines a state-of-the-art video interpolation method AdaConv [25] with a pre-trained CycleGAN model. We term it AdaConv-Cycle in the following experiments.

• Slomo-Cycle applies the video synthesis method Super Slomo [15] to perform video interpolation in the source domain. Then, we use pre-trained CycleGAN to translate the synthesized video frames into the target domain. We term it Slomo-Cycle in this paper.

Quantitative Comparisons. We compare the performance on Cityscapes and SYNTHIA and show the results in Table 2. For IS, our JWDM achieves comparative performance on scene2segmentation task, while achieves the best performance on other four tasks. Moreover, JWDM consistently outperforms the baselines in terms of both FID and FID4Video scores. It means that our method produces

Figure 4. Comparisons of different methods for season translation on SYNTHIA. Top row: the synthesized video in the winter domain. Rows 2-4: The corresponding translated video in the domains of the other three seasons, i.e., spring, summer and fall.

frames and videos of promising quality by exploiting the cross-domain correlations. The above observations demonstrate the superiority of our method over other methods.

Visual Comparisons. We compare the visual results of different methods on the following three tasks. 3

(i) Visual results on Cityscapes. We first interpolate videos in the cityscape domain and then translate them to the segmentation domain. In Figure 3, we compare the visual quality of both the interpolated and the translated images. From Figure 3 (top), our proposed method is able to produce sharper cityscape images and yields more accurate results in the semantic segmentation domain, which significantly outperforms the baseline methods. Besides, we can drawn the same conclusions in Figure 3 (bottom).

(ii) Visual results on SYNTHIA. We further evaluate the performance of our method on SYNTHIA. Specifically, we synthesize videos among the domains of four seasons shown in Figure 4. First, our method is able to produce sharper images when interpolating the missing in-between frames (see top row of Figure 4). Second, the translated frames in the spring, summer and fall domains are more photo-realistic than other baseline methods (see the shape of cars in Figure 4). These results demonstrate the efficacy of our method in producing visually promising videos in different domains.

(iii) High-frame-rate cross-domain video synthesis. We investigate the performance of high-frame-rate video synthesis on Cityscapes. Unlike traditional video synthesis studied in the previous section, we use two consecutive video frames in the source domain as input, and interpolate the intermediate 7 frames, i.e., 8frame rate up-conversion. The synthesized results are shown in Figure 5. 4

In this experiment, we take segmentation frames as the source domain and scene frames as the target domain to perform cross-domain video synthesis. First, we compare the quality of interpolated images in the source domain (see bottom row of Figure 5). It is clear that our method is able to interpolate better frames than most video synthesis methods trained with the ground-truth intermediate frames. Second, we also compare the translated frames in the target domain (see middle row of Figure 5). From Figure 5, our method is able to generate sharper images in the target domain with the help of well learned joint distribution. Last, we show the entire video sequence in the top row of Figure 5. The proposed JWDM is able to produce smooth video sequence in two domains simultaneously.

Figure 5. Comparisons of different methods for high-frame-rate image-to-video synthesis, segmentation2scene. Top: generated 8frame-rate video sequence by JWDM. Middle: Interpolated frames of different methods in the target domain. Bottom: Interpolated frames of different methods in the source domain.

Table 3. Influence of for the adversarial loss on Z. We compare the results of wintersummer in terms of FID and FID4Video.

5.3. Influence of for Adversarial Loss on Z

In this section, we evaluate the influences of the trade-off parameter over the adversarial loss on Z in Eqn. (6). Specifically, we compare FID and FID4Video with different on the wintersummer image translation task. The value of is selected among [0.01, 0.1, 1, 10]. The quantitative results are shown in Table 3.

From Table 3, JWDM achieves the best performance when setting to 0.1. When increasing the value of to 1 and 10, the performance degrades gradually. The same phenomenon happens when decreasing to 0.01. This means that when setting , we can achieve better trade-off between the optimization over latent space and data space, and thus obtain better performance.

In this paper, we have proposed a novel joint Wasserstein Distribution Matching (JWDM) method to match joint distributions in different domains. Relying on optimal transport theory, JWDM is able to exploit cross-domain correlations to improve the performance. Instead of directly optimizing the primal problem of Wasserstein distance between joint distributions, we derive an important theorem to solve a simple optimization problem. Extensive experiments on unsupervised image translation and cross-domain video synthesis demonstrate the superiority of the proposed method.

[1] Y. Alami Mejjati, C. Richardt, J. Tompkin, D. Cosker, and K. I. Kim. Unsupervised attention-guided image-to-image translation. In Neural Information Processing Systems 31, pages 3693–3703, 2018. 2, 5, 6

[2] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein gan. In International Conference on Machine Learning, 2017. 2, 4

[3] D. Bashkirova, B. Usman, and K. Saenko. Unsupervised video-to-video translation. arXiv preprint arXiv:1806.03698, 2018. 1, 2

[4] J. Cao, Y. Guo, Q. Wu, C. Shen, J. Huang, and M. Tan. Adversarial learning with local coordinate coding. In International Conference on Machine Learning, 2018. 2

[5] J. Cao, L. Mo, Y. Zhang, K. Jia, C. Shen, and M. Tan. Multimarginal wasserstein gan. In Advances in Neural Information Processing Systems, pages 1774–1784, 2019. 2

[6] J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In Computer Vision and Pattern Recognition, pages 6299–6308, 2017. 5

[7] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In Computer Vision and Pattern Recognition, 2016. 5

[8] A. Genevay, G. Peyr´e, and M. Cuturi. Learning generative models with sinkhorn divergences. In Artificial Intelligence and Statistics, 2018. 1, 3

[9] A. Gokaslan, V. Ramanujan, D. Ritchie, K. In Kim, and J. Tompkin. Improving shape deformation in unsupervised image-to-image translation. In European Conference on Computer Vision, 2018. 2

[10] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. WardeFarley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Neural Information Processing Systems, 2014. 2

[11] Y. Guo, Q. Chen, J. Chen, J. Huang, Y. Xu, J. Cao, P. Zhao, and M. Tan. Dual reconstruction nets for image superresolution with gradient sensitive loss. arXiv preprint arXiv:1809.07099, 2018. 2

[12] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Neural Information Processing Systems, 2017. 5

[13] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. In Computer Vision and Pattern Recognition, 2017. 2, 5

[14] D. Ji, J. Kwon, M. McFarland, and S. Savarese. Deep view morphing. In Computer Vision and Pattern Recognition, 2017. 2, 5, 7

[15] H. Jiang, D. Sun, V. Jampani, M.-H. Yang, E. Learned-Miller, and J. Kautz. Super slomo: High quality estimation of multiple intermediate frames for video interpolation. In Computer Vision and Pattern Recognition, pages 9000–9008, 2018. 7

[16] T. Kim, M. Cha, H. Kim, J. K. Lee, and J. Kim. Learning to discover cross-domain relations with generative adversarial networks. In International Conference on Machine Learning, 2017. 1, 2

[17] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference for Learning Representations, 2015. 5

[18] D. P. Kingma and M. Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014. 2

[19] C. Li, H. Liu, C. Chen, Y. Pu, L. Chen, R. Henao, and L. Carin. Alice: Towards understanding adversarial learning for joint distribution matching. In Neural Information Processing Systems, pages 5495–5503, 2017. 1

[20] M. Li, H. Huang, L. Ma, W. Liu, T. Zhang, and Y. Jiang. Unsupervised image-to-image translation with stacked cycle-consistent adversarial networks. In European Conference on Computer Vision, 2018. 2

[21] T. Lindvall. Lectures on the Coupling Method. Courier Corporation, 2002. 1

[22] M.-Y. Liu, T. Breuel, and J. Kautz. Unsupervised image-to-image translation networks. In Neural Information Processing Systems, 2017. 1, 2, 4, 5, 6, 7

[23] M.-Y. Liu and O. Tuzel. Coupled generative adversarial networks. In Neural Information Processing Systems, 2016. 2, 5, 6

[24] Z. Liu, R. A. Yeh, X. Tang, Y. Liu, and A. Agarwala. Video frame synthesis using deep voxel flow. In International Conference on Computer Vision, 2018. 2, 7

[25] S. Niklaus, L. Mai, and F. Liu. Video frame interpolation via adaptive separable convolution. In International Conference on Computer Vision, 2017. 2, 7

[26] Y. Pu, S. Dai, Z. Gan, W. Wang, G. Wang, Y. Zhang, R. Henao, and L. C. Duke. Jointgan: Multi-domain joint distribution

learning with generative adversarial nets. In International Conference on Machine Learning, 2018. 1

[27] G. Ros, L. Sellart, J. Materzynska, D. Vazquez, and A. M. Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In Computer Vision and Pattern Recognition, 2016. 5

[28] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans. In Neural Information Processing Systems, 2016. 5

[29] T. Salimans, H. Zhang, A. Radford, and D. Metaxas. Improving GANs using optimal transport. In International Conference on Learning Representations, 2018. 2

[30] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016. 5

[31] I. Tolstikhin, O. Bousquet, S. Gelly, and B. Schoelkopf. Wasserstein auto-encoders. In International Conference on Learning Representations, 2017. 2

[32] C. Villani. Optimal Transport: Old and New. Springer Science & Business Media, 2008. 2

[33] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, G. Liu, A. Tao, J. Kautz, and B. Catanzaro. Video-to-video synthesis. In Neural Information Processing Systems, 2018. 1, 2, 5

[34] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In Computer Vision and Pattern Recognition, 2018. 2

[35] Y. Yan, M. Tan, Y. Xu, J. Cao, M. Ng, H. Min, and Q. Wu. Oversampling for imbalanced data via optimal transport. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5605–5612, 2019. 2

[36] Z. Yi, H. R. Zhang, P. Tan, and M. Gong. Dualgan: Unsupervised dual learning for image-to-image translation. In International Conference on Computer Vision, 2017. 1, 2

[37] R. Zhang, T. Pfister, and J. Li. Harmonic unpaired image-to-image translation. In International Conference on Learning Representations, 2019. 2

[38] T. Zhou, S. Tulsiani, W. Sun, J. Malik, and A. A. Efros. View synthesis by appearance flow. In European conference on computer vision, 2016. 2

[39] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In International Conference on Computer Vision, 2017. 1, 2, 4, 5, 6

Proof We denote by and the set of all joint distributions of and with marginals and , respectively, and denote by the set of all joint distribution of and . Recall the definition of Wasserstein distance

Line (11) holds by the definition of . Line (12) uses the fact that the variable pair is independent of the variable pair , and and are the marginals on and induced by joint distributions in . In Line (13), if are Dirac measures (i.e.,

We consider certain sets of joint probability distributions and of three random variables and , respectively. We denote by and the set of all joint distributions of and with marginals and , respectively. The set of all joint distributions such that and , and likewise for . We denote by and the sets of marginals on and induced by distributions in , respectively, and likewise for . For the further analyses, we have

where and are the set of all probabilistic encoders, where are the marginal distributions of , respectively.

Line (14) uses the tower rule of expectation and Line (15) holds by the conditional independence property of . In line (16), we take the expectation w.r.t. and , respectively, and use the total probability. Line (17) follows the fact that and since and depend on the choice of conditional distributions , while does not, and likewise for distributions w.r.t. Y and . In line (18), the generative model can be derived from two cases where can be sampled from and , and likewise for the generative model

In this section, we discuss some details of optimization for distribution divergence. In the training, we use original GAN to measure the divergence , respectively. For , we optimize the following minimax problem:

where is a discriminator w.r.t. For , we optimize the following minimax problem:

where is a discriminator w.r.t. For , it contains simultaneously, we optimize the following minimax problem:

where is a discriminator w.r.t. Z, and z is drawn from the prior distribution . Similarly, the another term can be written as

The network architectures of JWDM are shown in Tables 4 and 5. We use the following abbreviations: N: the number of output channels, : the number of channels for latent variable(set to 256 by default), K: kernel size, S: stride size, P: padding size, IN: instance normalization.

Table 4. Auto-Encoder architecture.

Embedding Layer , K8x8, S1, P0), IN, ReLU

Embedding Layer DECONV-(N256, K8x8, S1, P0), IN, ReLU

Table 5. Discriminator network architecture.

Hidden Layer CONV-(N128, K4x4, S2, P1), Leaky ReLU Hidden Layer CONV-(N256, K4x4, S2, P1), Leaky ReLU Hidden Layer CONV-(N512, K4x4, S2, P1), Leaky ReLU Hidden Layer CONV-(N1024, K4x4, S2, P1), Leaky ReLU Hidden Layer CONV-(N2048, K4x4, S2, P1), Leaky ReLU

4.1. Results of Unsupervised Image-to-image Translation

winter2summer

Figure 6. Comparisons with different methods for unsupervised image-to-image translation on Cityscape and SYNTHIA.

winter2summer

Figure 7. Comparisons with different methods for unsupervised image-to-image translation on Cityscape and SYNTHIA.

4.2. Results of Unsupervised Cross-domain Video Synthesis

In this section, we provide more visual results for cross-domain video synthesis on SYNTHIA and Cityscapes datasets. Note that we use the first and eighth video frames in the source domain as input and interpolate the intermediate seven video frames in two domains simultaneously.

Figure 8, Figure 9 and Figure 10 are video synthesis results on winter spring, summer and fall}. Figure 11, Figure 12 and Figure 13, Figure 14 are two sets of video synthesis results on photosegmentation respectively. Evidently, both visual results on SYNTHIA and Cityscapes datasets show that our method produces frames and videos of promising quality and consistently outperforms all the baselines.

Figure 8. Comparisons of different methods for season winterspring translation on SYNTHIA dataset. The figure shows all frames of a video synthesized and translated by these mehtods.

Figure 9. Comparisons of different methods for season wintersummer translation on SYNTHIA dataset. The figure shows all frames of a video synthesized and translated by these mehtods.

Figure 10. Comparisons of different methods for season winterfall translation on SYNTHIA dataset. The figure shows all frames of a video synthesized and translated by these mehtods.

Figure 11. Comparisons of different methods for photosegmentation translation on Cityscapes dataset. The figure shows all frames of a video synthesized and translated by these mehtods.

Figure 12. Comparisons of different methods for segmentationphoto translation on Cityscapes dataset. The figure shows all frames of a video synthesized and translated by these mehtods.

Figure 13. Comparisons of different methods for photosegmentation translation on Cityscapes dataset. The figure shows all frames of another video synthesized and translated by these mehtods.

Figure 14. Comparisons of different methods for segmentationphoto translation on Cityscapes dataset. The figure shows all frames of another video synthesized and translated by these mehtods.