Learning a compact and interpretable representation of data without supervision is important to improve our understanding of data and machine learning systems. Recently, it is suggested that a disentangled representation, which represents data using independent factors of variations in data can improve the interpretability and transferability of the representation [5, 1, 50]. Among various use-cases of disentangled representation, we are particularly interested in its application to generative models, since it allows users to specify the desired properties in the output by controlling the generative factors encoded in each latent dimension. There are increasing demands on such generative models in various domains, such as image manipulation [20, 30, 27], drug discovery [15], ML fairness [11, 35], etc.

Most prior works on unsupervised disentangled representation learning formulate the problem as constrained generative modeling task. Based on well-established frameworks, such as the Variational Autoencoder (VAE) or the Generative Adversarial Network (GAN), they introduce additional regularization to encourage the axes of the latent manifold to align with independent generative factors in the data. Approaches based on VAE [17, 9, 25, 7] augment its objective function to favor a factorized latent representation by adding implicit [17, 7] or explicit penalties [25, 9]. On the other hand, approaches based on GAN [10] propose to regularize the generator such that it increases the mutual information between the input latent code and its output.

One major challenge in the existing approaches is the trade-off between learning disentangled representations and generating realistic data. VAE-based approaches are effective in learning useful disentangled representations in various tasks, but their generation quality is generally worse than the state-of-the-arts, which limits its applicability to the task of realistic synthesis. On the other hand, GANbased approaches can achieve the high-quality synthesis with a more expressive decoder and without explicit likelihood estimation [10]. However, they tend to learn comparably more entangled representations than the VAE counterparts [17, 25, 9, 7] and are notoriously difficult to train, even with recent techniques to stabilize the training [25, 53].

To circumvent this trade-off, we propose a simple and generic framework to combine the benefits of disentangled representation learning and high-fidelity synthesis. Unlike the previous approaches that address both problems jointly by a single objective, we formulate two separate, but successive problems; we first learn a disentangled representation using VAE, and distill the learned representation to GAN for high-fidelity synthesis. The distillation is performed from VAE to GAN by transferring the inference model, which provides a meaningful latent distribution, rather than a simple Gaussian prior and ensures that both models are aligned to render the same generative factors. Such decomposition also naturally allows a layered approach to learn latent representation by first learning major disentangled factors by VAE, then learning missing (entangled) nuisance factors by GAN. We refer the proposed method as the Information Distillation Generative Adversarial Network (ID-GAN).

Despite the simplicity, the proposed ID-GAN is extremely effective in addressing the previous challenges, achieving high-fidelity synthesis using the learned disentangled representation (e.g. 10241024 image). We also show that such decomposition leads to a practically efficient model design, allowing the models to learn the disentangled representation from low-resolution images and transfer it to synthesize high-resolution images.

The contributions of this paper are as follows:

• We propose ID-GAN, a simple yet effective framework that combines the benefits of disentangled representation learning and high-fidelity synthesis.

• The decomposition of the two objectives enables plug-and-play-style adoption of state-of-the-art models for both tasks, and efficient training by learning models for disentanglement and synthesis using low- and high-resolution images, respectively.

• Extensive experimental results show that the proposed method achieves state-of-the-art results in both disentangled representation learning and synthesis over a wide range of tasks from synthetic to complex datasets.

Disentanglement learning. Unsupervised disentangled representation learning aims to discover a set of generative factors, whose element encodes unique and independent factors of variation in data. To this end, most prior works based on VAE [17, 25, 9] and GAN [10, 21, 33, 32] focused on designing the loss function to encourage the factorization of the latent code. Despite some encouraging results, however, these approaches have been mostly evaluated on simple and low-resolution images [40, 36]. We believe that improving the generation quality of disentanglement learning is important, since it not only increases the practical impact in real-world applications, but also helps us to better assess the disentanglement quality on complex and natural images where the quantitative evaluation is dif-ficult. Although there are increasing recent efforts to improve the generation quality with disentanglement learning [21, 44, 33, 32], they often come with the degraded disentanglement performance [10], rely on a specific inductive bias (e.g. 3D transformation [44]), or are limited to low-resolution images [33, 32, 21]. On the contrary, our work aims to investigate a general framework to improve the generation quality without representation learning trade-off, while being general enough to incorporate various methods and inductive biases. We emphasize that this contribution is complementary to the recent efforts for designing better inductive bias or supervision for disentanglement learning [52, 43, 47, 37, 8]. In fact, our framework is applicable to a wide variety of disentanglement learning methods and can incorporate them in a plug-and-play style as long as they have an inference model (e.g. nonlinear ICA [24]).

Combined VAE/GAN models. There have been extensive attempts in literature toward building hybrid models of VAE and GAN [28, 4, 19, 54, 6]. These approaches typically learn to represent and synthesize data by combining VAE and GAN objectives and optimizing them jointly in an end-to-end manner. Our method is an instantiation of this model family, but is differentiated from the prior work in that (1) the training of VAE and GAN is decomposed into two separate tasks and (2) the VAE is used to learn a specific conditioning variable (i.e. disentangled representation) to the generator while the previous methods assume the availability of an additional conditioning variable [4] or use VAE to learn the entire (entangled) latent distribution [28, 19, 54, 6]. In addition, extending the previous VAE-GAN methods to incorporate disentanglement constraints is not straightforward, as the VAE and GAN objectives are tightly entangled in them. In the experiment, we demonstrate that applying existing hybrid models on our task typically suffers from the suboptimal trade-off between the generation quality and the disentanglement performance, and they perform much worse than our method.

The objective of unsupervised disentanglement learning is to describe each data x using a set of statistically independent generative factors z. In this section, we briefly review prior works and discuss their advantages and limitations.

The state-of-the-art approaches in unsupervised disentanglement learning are largely based on the Variational Autoencoder (VAE). They rewrite their original objective and derive regularizations that encourage the disentanglement of the latent variables. For instance, -VAE [17] proposes to optimize the following modified Evidence LowerBound (ELBO) of the marginal log-likelihood:

where setting reduces to the original VAE. By forcing the variational posterior to be closer to the factorized prior (), the model learns a more disentangled representation, but with a sacrifice of generation quality, since it also decreases the mutual information between z and x [9, 25]. To address such trade-off and improve the generation quality, recent approaches propose to gradually anneal the penalty on the KL-divergence [7], or decompose it to isolate the penalty for total correlation [51] that encourages the statistical independence of latent variables [1, 9, 25].

Approaches based on VAE have shown to be effective in learning disentangled representations over a range of tasks from synthetic [40] to complex datasets [34, 3]. However, their generation performance is generally insufficient to achieve high-fidelity synthesis, even with recent techniques isolating the factorization of the latent variable [25, 9]. We argue that this problem is fundamentally attributed to two reasons: First, most VAE-based approaches assume the fully-independent generative factors [17, 25, 9, 36, 50, 39]. This strict assumption oversimplifies the latent manifold and may cause the loss of useful information (e.g. correlated factors) for generating realistic data. Second, they typically utilize a simple generator, such as the factorized Gaussian decoder, and learn a uni-modal mapping from the latent to input space. Although this might be useful to learn meaningful representations [7] (e.g. capturing a structure in local modes), such decoder makes it difficult to render complex patterns in outputs (e.g. textures).

Our objective is to build a generative model X that produces high-fidelity output with an interpretable latent code (i.e. disentangled representation). To achieve this goal, we build our framework upon VAE-based models due to their effectiveness in learning disentangled representations. However, discussions in the previous section suggest that disentanglement learning in

Figure 2. Overall framework of the proposed method (ID-GAN).

VAE leads to the sacrifice of generation quality due to the strict constraints on fully-factorized latent variables and the utilization of simple decoders. We aim to improve the VAEbased models by enhancing generation quality while maintaining its disentanglement learning performance.

Our main idea is to decompose the objectives of learning disentangled representation and generating realistic outputs into separate but successive learning problems. Given a disentangled representation learned by VAEs, we train another network with a much higher modeling capacity (e.g. GAN generator) to decode the learned representation to a realistic sample in the observation space.

Figure 2 describes the overall framework of the proposed algorithm. Formally, let z = (s, c) denote the latent variable composed of the disentangled variable c and the nuisance variable s capturing independent and correlated factors of variation, respectively. In the proposed framework, we first train VAE (e.g. Eq. (1)) to learn disentangled latent representations of data, where each observation x can be projected to c by the learned encoder after the training. Then in the second stage, we fix the encoder and train a generator for high-fidelity synthesis while distilling the learned disentanglement by optimizing the following objective:

where is the aggregated poste- rior [38, 18, 49] of the encoder network1. Similar to [10], Eq. (4) corresponds to the variational lower-bound of mutual information between the latent code and the generator output I(c; G(s, c)), but differs in that (1) c is sampled from the aggregated posterior instead of the prior p(c) and (2) it is optimized with respect to the generator only. Note that we treat as a constant since is fixed in Eq.(4). We refer the proposed model as the Information Distillation Generative Adversarial Network (ID-GAN).

4.1. Analysis

In this section, we provide in-depth analysis of the proposed method and its connections to prior works.

Comparisons to -VAEs [17, 9, 25]. Despite the simplicity, the proposed ID-GAN effectively addresses the problems in -VAEs with generating high-fidelity outputs; it augments the latent representation by introducing a nuisance variable s, which complements the disentangled variable c by modeling richer generative factors. For instance, the VAE objective tends to favor representational factors that characterize as much data as possible [7] (e.g. azimuth, scale, lighting, etc.), which are beneficial in representation learning, but incomprehensive to model the complexity of observations. Given the disentangled factors discovered by VAEs, the ID-GAN learns to encode the remaining generative factors (such as high-frequency textures, face identity, etc.) into nuisance variable s. (Figure 8). This process shares a similar motivation with a progressive augmentation of latent factors [31], but is used for modeling disentangled and nuisance generative factors. In addition, ID-GAN employs a much more expressive generator than a simple factorized Gaussian decoder in VAE, which is trained with adversarial loss to render realistic and convincing outputs. Combining both, our method allows the generator to synthesize various data in a local neighborhood defined by c, where the specific characteristics of each example are fully characterized by the additional nuisance variable s.

Comparisons to InfoGAN [10]. The proposed method is closely related to InfoGAN, which optimizes the variational lower-bound of mutual information I(c; G(s, c)) for disentanglement learning. To clarify the difference between the proposed method and InfoGAN, we rewrite the regularization for both methods using the KL divergence as follows:

where summarizes all regularization terms in our method 2. See the Appendix A.1 for detailed derivations.

Eq. (5) shows that InfoGAN optimizes the forward KL divergence between the prior p(c) and the approximated posterior . Due to the zero-avoiding characteristics of forward KL [42], it forces all latent code c with non-zero prior to be covered by the posterior . Intuitively, it implies that InfoGAN tries to exploit every dimensions in c to encode each (unique) factor of variations. It becomes problematic when there is a mismatch between the

Figure 3. Comparison of disentanglement vs. generation performance on dSprites dataset.

number of true generative factors and the size of latent variable c, which is common in unsupervised disentanglement learning. On the contrary, VAE optimizes the reverse KL divergence (Eq. (6)), which can effectively avoid the problem by encoding only meaningful factors of variation into certain dimensions in c while collapsing the remainings to the prior. Since the encoder training in our method is only affected by Eq. (6), it allows us to discover the ambient dimension of latent generative factors robust to the choice of latent dimension |c|.

In addition, Eq. (5) shows that InfoGAN optimizes the encoder using the generated distributions, which can be problematic when there exists a sufficient discrepancy between the true and generated distributions (e.g. modecollapse may cause learning partial generative factors.). On the other hand, the encoder training in our method is guided by the true data (Eq. (6)) together with maximum likelihood objective, while the mutual information (Eq. (7)) is enforced only to the generator. This helps our model to discover comprehensive generative factors from data while guiding the generator to align its outputs to the learned representation.

Practical benefits. The objective decomposition in the proposed method also offers a number of practical advantages. First, it enables plug-and-play-style adoption of the state-of-the-art models for disentangled representation learning and high-quality generation. As shown in Figure 3, it allows our model to achieve state-of-the-art performance on both tasks. Second, such decomposition also leads to an efficient model design, where we learn disentanglement from low-resolution images and distill the learned representation to the task of high-resolution synthesis with a much higher-capacity generator. We argue that it is practically reasonable in many cases since VAEs tend to learn global structures in disentangled representation, which can be captured from low-resolution images. We demonstrate this in the high-resolution image synthesis task, where we use the disentangled representation learned with images for the synthesis of or images.

Table 1. Quantitative comparison results on synthetic datasets.

Figure 4. Qualitative results on synthetic datasets. Both -VAE and ID-GAN share the same latent code, but ID-GAN exhibits substantailly higher generation quality.

In this section, we present various results to show the effectiveness of ID-GAN. Refer to the Appendix for more comprehensive results and figures.

5.1. Implementation Details

Compared methods. We compare our method with state-of-the-art methods in disentanglement learning and generation. We choose -VAE [17], FactorVAE [25], and InfoGAN [10] as baselines for disentanglement learning. For fair comparison, we choose the best hyperparameter for each model via extensive hyper-parameter search. We also report the performance by training each method over five different random seeds and averaging the results.

Network architecture. For experiments on synthetic datasets, we adopt the architecture from [36] for all VAEbased methods (VAE, -VAE, and FactorVAE). For GANbased methods (GAN, InfoGAN, and ID-GAN), we employ the same decoder and encoder architectures in VAE as the generator and discriminator, respectively. We set the size of disentangled latent variable to 10 for all methods, and exclude the nuisance variable in GAN-based methods for a fair comparison with VAE-based methods. For experiments on complex datasets, we employ the generator and discriminator in the state-of-the-art GAN [41, 46]. For VAE architectures, we utilize the same VAE architecture as in the synthetic datasets. We set the size of disentangled and nuisance variables to 20 and 256, respectively.

Evaluation metrics We employ three popular evaluation metrics in the literature: Factor-VAE Metric (FVM) [25], Mutual Information Gap (MIG) [9], and Fr´echet Inception Distance (FID) [16]. FVM and MIG evaluate the disentanglement performance by measuring the degree of axisalignment between each dimension of learned representations and ground-truth factors. FID evaluates the generation quality by measuring the distance between the true and the generated distributions.

5.2. Results on Synthetic Dataset.

For quantitative evaluation of disentanglement, we employ the dSprites dataset [40], which contains synthetic images generated by randomly sampling known generative factors, such as shape, orientation, size, and x-y position. Since the complexity of dSprites is limited to analyze the disentanglement and generation performance, we adopt three variants of dSprites, which are generated by adding color [25] (Color-dSprites) or background noise [36] (Noisy- and Scream-dSprites).

Table 1 and Figure 4 summarize the quantitative and qualitative comparison results with existing disentangle-

Table 2. Comparison of approaches using a joint and decomposed objective for disentanglement learning and synthesis.

ment learning approaches, respectively. First, we observe that VAE-based approaches (i.e. -VAE and FactorVAE) achieve the state-of-the-art disentanglement performance across all datasets, outperforming the VAE baseline and InfoGAN with a non-trivial margin. The qualitative results in Figure 4 show that the learned generative factors are wellcorrelated with meaningful disentanglement in the observation space. On the other hand, InfoGAN fails to discover meaningful disentanglement in most datasets. We observe that information maximization in InfoGAN often leads to undesirable factorization of generative factors, such as encoding both shape and position into one latent code, but factorizing latent dimensions by different combinations of them (e.g. Color-dSprites in Figure 4). ID-GAN achieves state-of-the-art disentanglement through the distillation of the learned latent code from the VAE-based models. Appendix B.3 also shows that ID-GAN is much more stable to train and insensitive to hyper-parameters than InfoGAN.

In terms of generation quality, VAE-based approaches generally perform much worse than GAN baseline. This performance gap is attributed to the strong constraints on the factorized latent variable and weak decoder in VAE, which limits the generation capacity. This is clearly observed in the results on the Noisy-dSprites dataset (Figure 4), where the outputs from -VAE fail to render the high-dimensional patterns in the data (i.e. uniform noise). On the other hand, our method achieves competitive generation performance to the state-of-the-art GAN using a much more flexible generator for synthesis, which enables the modeling of complex patterns in data. As observed in Figure 4, ID-GAN performs generation using the same latent code with -VAE, but produces much more realistic outputs by capturing accurate object shapes (in Color-dSprites) and background patterns (in Scream-dSprites and NoisydSprites) missed by the VAE decoder. These results suggest that our method can achieve the best trade-off between disentanglement learning and high-fidelity synthesis.

5.3. Ablation Study

This section provides an in-depth analysis of our method.

Is two-step training necessary? First, we study the impact of two-stage training for representation learning and synthesis. We consider two baselines: (1) VAE-GAN [28] as an extension of -VAE with adversarial loss, and (2) end-to-end training of ID-GAN. Contrary to ID-GAN that learns

Table 3. Comparison of two-step approaches for generation (FID) and alignment (and GILBO)3performance.

Figure 5. Qualitative comparisons of various two-step approaches. All samples share the same disentangled code c, but different nuisance variable s. (1) First column: output of -VAE decoder. (2) Second to fourth columns: images generated by different nuisance variables s using various methods (rows).

to represent () and synthesize (G) data via separate objectives, these baselines learn a single, entangled objective for both tasks. Table 2 summarizes the results in the dSprites dataset.

The results show that VAE-GAN improves the generation quality of -VAE with adversarial learning. The generation quality is further improved in the end-to-end version of ID-GAN by employing a separate generator for synthesis. However, the improved generation quality in both baselines comes with the cost of degraded disentanglement performance. We observe that updating the encoder using adversarial loss hinders the discovery of disentangled factors, as the discriminator tends to exploit high-frequency details to distinguish the real images from the fake images, which motivates the encoder to learn nuisance factors. This suggests that decomposing the representation learning and generation objective is important in the proposed framework (ID-GAN two-step), which achieves the best performance in both tasks.

Is distillation necessary? The above ablation study jus-tifies the importance of two-step training. Next, we compare different approaches for two-step training that perform conditional generation using the representation learned by -VAE. Specifically, we consider two baselines: (1) cGAN and (2) ID-GAN trained without distillation (ID-GAN w/o distill). We opt to consider cGAN as the baseline since we find that it implicitly optimizes (see Appendix A.2 for the proof). In the experiments, we train all models in the CelebA 128x128 dataset using the same -VAE trained on the resolution, and compare the generation quality (FID) and a degree of alignment between the disentangled code c and generator output G(s, c). For comparison of the alignment, we measure (Eq. (7)) and GILBO 4 [2], both of which are valid lower-bounds of mutual information I(c; G(s, c)). Note that the comparison based on the lower-bound is still valid as its relative order has shown to be insensitive to the tightness of the bound [2]. Table 3 and Figure 5 summarize the quantitative and qualitative results, respectively.

As shown in the table, all three models achieve comparable generation performances in terms of FID. However, we observe that their alignments to the input latent code vary across the methods. For instance, ID-GAN (w/o distill) achieves very low and GILBO, indicating that the generator output is not accurately reflecting the generative factors in c. The qualitative results (Figure 5) also show considerable mismatch between the c and the generated images. Compared to this, cGAN achieves much higher degree of alignment due to the implicit optimization of , but its association is much loose than our method (e.g. changes in gender and hairstyle). By explicitly constraining the generator to optimize , ID-GAN achieves the best alignment.

5.4. Results on Complex Dataset

To evaluate our method with more diverse and complex factors of variation, we conduct experiments on natural image datasets, such as CelebA [34], 3D Chairs [3], and Cars [26]. We first evaluate our method on images, and extend it to higher resolution images using the CelebA () and CelebA-HQ [23] () datasets.

Comparisons to other methods. Table 4 summarizes quantitative comparison results (see Appendix A.4 for qualitative comparisons). Since there are no ground-truth factors available in these datasets, we report the performance based on generation quality (FID). As expected, the generation quality of VAE-based methods is much worse in natural images. GAN-based methods, on the contrary, can generate more convincing samples exploiting the expressive generator. However, we observe that the baseline GAN taking only nuisance variables ends up learning highly-entangled generative factors. ID-GAN achieves disentanglement via disentangled factors learned by VAE, and generation performance on par with the GAN baseline.

To better understand the disentanglement learned by GAN-based methods, we present latent traversal results in Figure 6. We generate samples by modifying values of each dimension in the disentangled latent code c while fixing the

Table 4. Quantitative results based on FID (

Figure 6. Comparisons of latent traversal between GAN-based approaches. Although both methods achieve comparable generation quality, ID-GAN learns much more meaningful disentanglement.

Figure 7. Comparisons of VAE and ID-GAN outputs (top-rows: VAE, bottom-rows: ID-GAN). Note that both outputs are generated from the same latent code, but using different decoders. Both decoders are aligned well to render the same generative factors, but ID-GAN produces much more realistic outputs.

rest. We observe that the InfoGAN fails to encode meaningful factors into c, and nuisance variable z dominates the generation process, making all generated images almost identical. On the other hand, ID-GAN learns meaningful disentanglement with c and generates reasonable variations.

Extension to high-resolution synthesis. One practical benefit of the proposed two-step approach is that we can incorporate any VAE and GAN into our framework. To demonstrate this, we train ID-GAN for high-resolution images (e.g. and ) while distilling the

Figure 8. Analysis on the learned disentangled variables and nuisance variables of ID-GAN on CelebA (256256). The samples in the first row are generated by the VAE decoder and the rest are generated by ID-GAN. Each captures the most salient factors of variation (e.g., azimuth, hairstructure, etc.) while contributes to the local details (e.g., and for curvy and straight hair, respectively).

-VAE encoder learned with much smaller images5. This allows us to easily scale up the resolution of synthesis and helps us to better assess the disentangled factors.

We first adapt ID-GAN to the image synthesis task. To understand the impact of distillation, we visualize the outputs from the VAE decoder and the GAN generator using the same latent code as inputs. Figure 7 summarizes the results. We observe that the outputs from both networks are aligned well to render the same generative factors to similar outputs. Contrary to blurry and low-resolution () VAE outputs, however, ID-GAN produces much more realistic and convincing outputs by introducing a nuisance variable and employing more expressive decoder trained on higher-resolution (). Interestingly, synthesized images by ID-GAN further clarify the disentangled factors learned by the VAE encoder. For instance, the first row in Figure 7 shows that the ambiguous disentangled factors from the VAE decoder output is clari-fied by ID-GAN, which is turned out to capture the style of a cap. This suggests that ID-GAN can be useful in assessing the quality of the learned representation, which will broadly benefit future studies.

To gain further insights on the learned generative factors by our method, we conduct qualitative analysis on the latent variables (c and s) by generating samples by fixing one variable while varying another (Figure 8). We observe that varying the disentangled variable c leads to variations in the holistic structures in the outputs, such as azimuth, skin

Figure 9. Results on the CelebA-HQ dataset (ages).

color, hair style, etc, while varying the nuisance variable s leads to changes in more fine-grained facial attributes, such as expression, skin texture, identity, etc. It shows that ID-GAN successfully distills meaningful and representative disentangled generative factors learned by the inference network in VAE, while producing diverse and high-fidelity outputs using generative factors encoded in the nuisance variable.

Finally, we further conduct experiments on the more challenging task of mega-pixel image synthesis. In the experiments, we base our ID-GAN on the VGAN architecture [46] and adapt it to synthesize CelebA-HQ images given factors learned by -VAE. Figure 9 presents the results, where we generate images by changing one values in one latent dimension in c. We observe that IDGAN produces high-quality images with nice disentanglement property, where it changes one factor of variation in the data (e.g. azimuth and hair-style) while preserving the others (e.g. identity).

We propose Information Distillation Generative Adversarial Network (ID-GAN), a simple framework that combines the benefits of the disentanglement representation learning and high-fidelity synthesis. We show that we can incorporate the state-of-the-art for both tasks by decomposing their objectives while constraining the generator by distilling the encoder. Extensive experiments on synthetic and complex datasets validate that the proposed method can achieve the best trade-off between realism and disentanglement, outperforming the existing approaches with substantial margin. We also show that such decomposition leads to efficient and effective model design, allowing high-fidelity synthesis with disentanglement on high-resolution images.

[1] A. Achille and S. Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018. 1, 3

[2] Alexander A Alemi and Ian Fischer. Gilbo: One metric to measure them all. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, NeurIPS, pages 7037–7046. Curran Associates, Inc., 2018. 6, 7

[3] Mathieu Aubry, Daniel Maturana, Alexei Efros, Bryan Rus- sell, and Josef Sivic. Seeing 3D chairs: exemplar part-based 2D-3D alignment using a large dataset of CAD models. In CVPR, 2014. 3, 7, 23

[4] J. Bao, D. Chen, F. Wen, H. Li, and G. Hua. Cvae-gan: Fine- grained image generation through asymmetric training. In ICCV. 2

[5] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Rep- resentation Learning: A review and new perspectives. PAMI, 2013. 1

[6] Andrew Brock, Theodore Lim, James M. Ritchie, and Nick Weston. Neural photo editing with introspective adversarial networks. In ICLR, 2017. 2

[7] Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in preprint arXiv:1804.03599, 2018. 1, 2, 3, 4, 23

[8] Junxiang Chen and Kayhan Batmanghelich. Weakly supervised disentanglement by pairwise similarities. arXiv preprint arXiv:1906.01044, 2019. 2

[9] Tian Qi Chen, Xuechen Li, Roger Grosse, and David Du- venaud. Isolating sources of disentanglement in variational autoencoders. In NeurIPS, 2018. 1, 2, 3, 4, 5, 23

[10] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets. In NeurIPS, 2016. 1, 2, 3, 4, 5, 11, 16

[11] Elliot Creager, David Madras, Joern-Henrik Jacobsen, Marissa Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. Flexibly fair representation learning by disentanglement. In ICML, pages 1436–1445, 2019. 1

[12] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR, 2009. 23

[13] Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks. In NeurIPS. 2016. 22

[14] Maurice Fr´echet. Sur la distance de deux lois de probabilit´e. COMPTES RENDUS HEBDOMADAIRES DES SEANCES DE L ACADEMIE DES SCIENCES, 1957. 23

[15] Rafael Gmez-Bombarelli, Jennifer N. Wei, David Duvenaud, Jos Miguel Hernndez-Lobato, Benjamn Snchez-Lengeling, Dennis Sheberla, Jorge Aguilera-Iparraguirre, Timothy D. Hirzel, Ryan P. Adams, and Aln Aspuru-Guzik. Automatic chemical design using a data-driven continuous representation of molecules. ACS Central Science, 2018. 1

[16] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Gnter, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a nash equilibrium. In NeurIPS, 2017. 5, 23

[17] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. -VAE: Learning basic visual concepts with a constrained variational framework. In ICLR, 2017. 1, 2, 3, 4, 5, 16

[18] Matthew D. Hoffman and Matthew J. Johnson. ELBO surgery: yet another way to carve up the variational evidence lower bound. In NeurIPS, 2016. 3

[19] Huaibo Huang, zhihang li, Ran He, Zhenan Sun, and Tie- niu Tan. Introvae: Introspective variational autoencoders for photographic image synthesis. In NIPS. 2018. 2

[20] Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image translation. In ECCV, 2018. 1

[21] Insu Jeon, Wonkwang Lee, and Gunhee Kim. IB-GAN: Disentangled representation learning with information bottleneck GAN, 2019. 2

[22] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016. 22

[23] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In ICLR, 2018. 7, 23

[24] Ilyes Khemakhem, Diederik Kingma, and Aapo Hyvrinen. Variational autoencoders and nonlinear ica: A unifying framework. arXiv preprint arXiv:1907.04809, 2019. 2

[25] Hyunjik Kim and Andriy Mnih. Disentangling by factoris- ing. In ICML, 2018. 1, 2, 3, 4, 5, 16, 23

[26] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), 2013. 7, 23

[27] Guillaume Lample, Neil Zeghidour, Nicolas Usunier, An- toine Bordes, Ludovic DENOYER, and Marc Aurelio Ranzato. Fader networks:manipulating images by sliding attributes. In NeurIPS. 2017. 1

[28] Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoencoding beyond pixels using a learned similarity metric. In ICML, 2016. 2, 6

[29] C. Ledig, L. Theis, F. Huszr, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi. Photo-realistic single image super-resolution using a generative adversarial network. In CVPR, 2017. 22

[30] Hsin-Ying Lee, Hung-Yu Tseng, Jia-Bin Huang, Ma- neesh Kumar Singh, and Ming-Hsuan Yang. Diverse image-to-image translation via disentangled representations. In ECCV, 2018. 1

[31] Jos Lezama. Overcoming the disentanglement vs reconstruc- tion trade-off via jacobian supervision. In ICLR, 2019. 4

[32] Zinan Lin, Kiran Koshy Thekumparampil, Giulia C. Fanti, and Sewoong Oh. Infogan-cr: Disentangling generative adversarial networks with contrastive regularizers. arXiv preprint arXiv:1906.06034, 2019. 2

[33] Bingchen Liu, Yizhe Zhu, Zuohui Fu, Gerard de Melo, and Ahmed Elgammal. OOGAN: disentangling GAN with onehot sampling and orthogonal regularization. arXiv preprint arXiv:1905.10836, 2019. 2

[34] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, 2015. 1, 3, 7, 23

[35] Francesco Locatello, Gabriele Abbati, Tom Rainforth, Ste- fan Bauer, Bernhard Sch¨olkopf, and Olivier Bachem. On the fairness of disentangled representations. arXiv preprint arXiv:1905.13662, 2019. 1

[36] Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Sch¨olkopf, and Olivier Bachem. Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations. arXiv preprint arXiv:1811.12359, 2018. 2, 3, 5, 23

[37] Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar R¨atsch, Bernhard Sch¨olkopf, and Olivier Bachem. Disentangling factors of variation using few labels. arXiv preprint arXiv:1905.01258, 2019. 2

[38] Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. In ICLR, 2016. 3

[39] Emile Mathieu, Tom Rainforth, N. Siddharth, and Yee Whye Teh. Disentangling disentanglement in variational auto-encoders. In Bayesian Deep Learning Workshop, NeurIPS, 2018. 3

[40] Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dSprites: Disentanglement testing Sprites dataset. https://github.com/deepmind/dsprites-dataset/, 2017. 2, 3, 5, 23

[41] Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. Which training methods for gans do actually converge? In ICML, 2018. 5, 16, 24, 25

[42] Tom Minka et al. Divergence measures and message pass- ing. Technical report, Technical report, Microsoft Research, 2005. 4

[43] Siddharth Narayanaswamy, T. Brooks Paige, Jan-Willem van de Meent, Alban Desmaison, Noah Goodman, Pushmeet Kohli, Frank Wood, and Philip Torr. Learning disentangled representations with semi-supervised deep generative models. In NeurIPS. 2017. 2

[44] Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Hologan: Unsupervised learning of 3d representations from natural images. In ICCV, 2019. 2

[45] Diederik P Kingma and Max Welling. Auto-encoding varia- tional bayes. In ICLR, 2014. 16

[46] Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, and Sergey Levine. Variational discriminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining information flow. In ICLR, 2019. 5, 8, 17, 18, 25

[47] Adria Ruiz, Oriol Mart´ınez, Xavier Binefa, and Jakob Verbeek. Learning disentangled representations with reference-based variational autoencoders. arXiv preprint arXiv:1901.08534, 2019. 2

[48] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016. 23

[49] Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bern- hard Schoelkopf. Wasserstein auto-encoders. In ICLR, 2018. 3

[50] Michael Tschannen, Olivier Frederic Bachem, and Mario Lui. Recent advances in autoencoder-based representation learning. In Bayesian Deep Learning Workshop, NeurIPS, 2018. 1, 3

[51] S. Watanabe. Information Theoretical Analysis of Multivari- ate Correlation. IBM Journal of Research and Development, 1960. 3

[52] Nicholas Watters, Lo¨ıc Matthey, Christopher P. Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes. arXiv preprint arXiv:1901.07017, 2019. 2

[53] Xiang Wei, Zixia Liu, Liqiang Wang, and Boqing Gong. Improving the improved training of wasserstein GANs. In ICLR, 2018. 2

[54] Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Dar- rell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. In NIPS, 2017. 2

A.1. InfoGAN optimizes Forward-KL Divergence

This section provides the derivation of Eq. (5) in the main paper. Consider G(s, c) as a mapping function , where s and c denote nuisance and disentangled variables, respectively. Also, assuming G is a deterministic function of (s, c), the conditional distribution can be approximated to a dirac distribution . Then, the marginal distribution can be described as below:

ps, c)dcds s, cs, c)}]. (A.1)

Then, the variational lower-bound of mutual information optimized in InfoGAN [10] can be rewritten as follows:

G, q, = p(s) p(c) ps, cdxdcds + H(c), (A.2)

= p(s) p(c) 1{x = G(s, cdxdcds + H(c), (A.3)

= p(s) ps, c))dcds + H(c), (A.4)

= p(s) ps, c))dcds p(c) log p(c)dc, (A.5)

= p(s) ps, c))dcds p(c) log p(c)dc p(s)ds, (A.6)

= p(s) p(c) log qs, c)) p(c) dcds, (A.7)

s, c))], (A.8)

where the Eq. (A.8) corresponds to Eq. (5) of the main paper. Similiarly, we can rewrite the distillation regularization (Eq. (4) in the main paper) as follows:

, = p(s) q1{x = G(s, cdxdcds , (A.9)

= p(s) qs, c))dcds p(s) qdcds, (A.10)

s, c))], (A.11)

where Eq. (A.11) corresponds to Eq. (7) in the main paper. As discussed in the paper, both Eq. (A.8) and (A.11) correspond to the forward KLD; regularization in InfoGAN (Eq. (A.8)) is optimized with respect to both the encoder q and the generator G, which is problematic due to the zero-avoiding characteristics of forward KLD and the potential mismatch between the true and generated data distributions. On the other hand, our method can effectively avoid this problem by optimizing Eq. (A.11) with only respect to the generator while encoder training is guided by reverse KLD using the true data distribution (Eq. (6) in the main paper).

A.2. cGAN implicitly maximizes

In Section 5.3 of the main paper, we define cGAN as the baseline that also optimizes implicitly in its objective function. This section provides its detailed derivation. Formally, we consider cGAN that minimizes a JensenShannon divergence (JSD) between two joint distributions DJS, where and

denote real and fake joint distributions, respectively. Then, from DJSis derived as follows:

2DJSx, cx, c)) = DKLx, cx, c)) + DKLx, cx, c)), (A.12)

= DKLx, cx, c)) + px, c) log px, cx, c) dxdc, (A.13)

= DKLx, cx, c)) + qqdxdc, (A.14)

+ qpdxdc qdxdc, (A.15)

+ qppdxdc s, c) log qdxdcds, (A.16)

= DKL()) + [DKL((A.17)

Eq. (A.17) implies that the cGAN objective also implicitly maximizes . However, Eq. (A.17) is guaranteed only when the discriminator converges to (near-)optimal with respect to real and fake joint distributions, which makes the optimization of highly dependent on the quality of the discriminator. On the other hand, ID-GAN maximizes explicitly by directly computing from the learned encoder , which leads to a higher degree of alignment between the input latent code and the generated output (Table 3 and Figure 5 in the main paper).

B.1. Additional Results on Synthetic Dataset

We present additional qualitative results on the synthetic dataset, which corresponds to Section 5.2 in the main paper. Figure A.1 presents the randomly generated images by the proposed ID-GAN. We observe that the generated images are sharp and realistic, capturing complex patterns in the background (Screem-dSprites and Noise-dSprites datasets). We also observe that it generates convincing foreground patterns, such as color and shape of the objects, while covering diverse and comprehensive patterns in real objects.

Figure A.2 and A.3 present additional qualitative comparison results with -VAE and InfoGAN by manipulating the disentangled factors, which correspond to Figure 4 in the main paper. We observe that -VAE captures the meaningful disentangled factors, such as location and color of the object, but overlooks several complex but important patterns in the background (Screem-dSprites and Noise-dSprites datasets) as well as foreground (e.g. detailed shape and orientation). On the other hand, InfoGAN generates more convincing images by employing a more expressive decoder, but learns much more entangled representations (e.g. changing location and color of the objects in Color-dSprites dataset). By combining the benefits of both approaches, ID-GAN successfully learns meaningful disentangled factors and generates realistic patterns.

Figure A.1. Random generated samples of ID-GAN on dSprites and its variants.

Figure A.2. Latent traversals of InfoGAN, -VAE, and ID-GAN on (a) dSprites and (b) Color-dSprites datasets. Each represents a single dimension of c.

Figure A.3. Latent traversals of InfoGAN, -VAE, and ID-GAN on (a) Noisy-dSprites and (b) Scream-dSprites datasets. Each (j = 1, . . . , 10) represents a single dimension of c.

B.2. Additional Results on a Complex Dataset

Qualitative results of Table 4. Here we compare the qualitative samples generated by each model in Table 4, i.e. VAE, -VAE, FactorVAE, GAN, InfoGAN, and ID-GAN. The qualitative results are shown in Figure A.4. Although VAE-based methods learn to represent global structures or salient factors of data in all datasets, generated samples are often blurry and lack textural or local details. On the other hand, GAN-based approaches (i.e. GAN, InfoGAN and ID-GAN) generate sharp and realistic samples thanks to the implicit density estimation and expressive generators. However, as shown in Figure A.5, InfoGAN generally fails to capture meaningful disentangled factors into c since it exploits the nuisance variable s to encode the most salient factors of variations. On the other hand, ID-GAN successfully captures major disentangled factors into c while encoding only local details into the nuisance variable s.

Figure A.4. Random samples generated by VAE [45], ], FactorVAE [25], GAN [41], InfoGAN [10], and ID-GAN on CelebA, 3D Chairs, and Cars datasets (64

Figure A.5. Analysis on the learned factors of variations in ID-GAN and InfoGAN. We sample 5 points for each c and s and visualize how they are used during the generative process of ID-GAN and InfoGAN. The samples in each row are generated from a single c with 5 different s. Also, we show the generated samples of -VAE from each c at the first column of the panes of ID-GAN. As shown above, ID-GAN successfully learns to decode the global structures of data from -VAE, while utilizing s as additional sources of variations for modeling local details. In InfoGAN, however, the most salient factors of variations are dominated by s while c acts as the nuisance.

Additional results on high-resolution synthesis. Here we provide more qualitative results of ID-GAN on high-resolution image synthesis (CelebA 256256 and CelebA-HQ datasets). We first present the results on the CelebA-HQ dataset composed of mega-pixel images (10241024 pixels). Figure A.6 presents the randomly generated samples by ID-GAN. We observe that ID-GAN produces sharp and plausible samples on high resolution images, showing on par generation performance with the state-of-the-art GAN baseline [46] employed as a backbone network of ID-GAN. We argue that this is due to the separate decoder and generator scheme adopted in ID-GAN, which is hardly achievable in the VAE-based approaches using a factorized Gaussian decoder for explicit maximization of data log-likelihood.

Next, we analyze the learned factors of variations in ID-GAN by investigating the disentangled and nuisance variable c and s, respectively. Similarly to Figure 8 in the main paper, we compare the samples generated by fixing one latent variable while varying another. The results are summarized in Figure A.7. Similar to Figure 8, we observe that the disentangled variable c contributes to the most salient factors of variations (e.g. azimuth, shape, or colour of face and hair, etc.) while the nuisance variable s contributes to the remaining fine details (e.g. identity, hair style, expression, background, etc.). For instance, we observe that fixing the disentangled variable c leads to consistent global face structure (e.g. black male facing slightly right (first column), blonde female facing slightly left (fourth column)), while fixing nuisance variable s leads to consistent details (e.g. horizontally floating hair (third row), smiling expression (fourth and fifths rows)). These results suggest that the generator in ID-GAN is well-aligned with the VAE decoder to render the same disentangled variable c into similar observations, but with more expressive and realistic details by exploiting the nuisance variable s.

Finally, to further visualize the learned disentangled factors in c, we present the latent traversal results in Figure A.8 as an extension to Figures 1 and 9 in the main paper. We also visualize the results on CelebA images in Figure A.9, where we observe a similar behavior.

Figure A.6. Random samples generated by ID-GAN on the CelebA-HQ dataset (). ID-GAN is based on VGAN architecture [46] and is trained to render learned disentangled representation c of -VAE trained on much smaller image resolution.

Figure A.7. Similar visualizations as Figure 8, but on a more challenging CelebA-HQ () dataset. We can clearly observe the different contributions of disentangled variable c and nuisance variable s to the generative process G(s, c); disentangled variable c captures the most salient factors of variations in the data (e.g. azimuth and overall structure/color of face/hair are largely determined by c); nuisance variable s contributes to the remaining fine details (e.g. identity, hair style, expression, background, etc.).

Figure A.8. Latent traversal results of ID-GAN on CelebA-HQ () dataset.

Figure A.9. Latent traversals of ID-GAN on the CelebA dataset ().

B.3. Sensitivity of Generation Performance (FID) on the Hyperparameter

Figure A.10. Sensitivity of the generation performance (FID) on

To better understand the sensitivity of our model to its hyperparmeter (in Eq. (2)), we conduct an ablation study by measuring the generation performance (FID) of our models trained with various . Figure A.10 summarizes the results on the dSprite dataset. First, we observe that the proposed ID-GAN performs well over a wide range of hyper-parameters () while the performance of InfoGAN is affected much sensitively to the choice of . Interestingly, increasing the in our method also leads to the improved generation quality over a certain range of . We suspect that it is because the information maximization in Eq. (4) using the pre-trained encoder also behaves as the perceptual loss [22, 13, 29], regularizing the generator to match the true distribution in more meaningful feature space (i.e. disentangled representation).

C.1. Evaluation Metrics

FactorVAE Metric (FVM). FVM [25] measures the accuracy of a majority-vote classifier, where the encoder network to be evaluated is used for constructing the training data of this classifier. A single training data, or vote, is generated as follows: we first extract encoder outputs from the entire samples of a synthetic dataset; estimate empirical variances of each latent dimension from the extracted outputs; sort out collapsed latent dimensions of variances smaller than 0.05; synthesize 100 samples with a single factor fixed and the other factors varying randomly; extract encoder outputs from synthesized samples; compute variances of each latent dimension divided by the empirical variances computed beforehand; then finally get a single vote which is a pair of the index of the fixed factor and the index of the latent dimension with the smallest normalized variance. We generate 800 votes to train the majority-vote classifier and report its train accuracy as the metric score.

Mutual Information Gap (MIG). MIG [9] is an information-theoretic approach to measure the disentanglement of representations. Specifically, assuming K generative factors and D-dimensional latents , it computes a normalized empirical mutual information to measure the information-theoretic preferences of towards each , or vice versa. Then, it aggregates the differences, or gap, between the top two preferences for each and averages them to compute MIG, i.e. , where . For implementation details, we directly follow the settings6 in [36].

Fr´echet Inception Distance (FID). We employ Fr´echet Inception Distance (FID) [16] to evaluate the generation quality of each model considered in our experiments. FID measures the Frchet distance [14] between two Gaussians, constructed by generated and real images, respectively, in the feature space of a pre-trained deep neural network. For each model, we compare 50,000 generated images and 50,000 real images to compute FID. For dSprites and its variants, we use a manually trained ConvNet trained to predict true generative factors of dSprites and its varaints. For the CelebA, 3D Chairs RGB, and Cars datasets, we use Inception V3 [48] pre-trained on the ImageNet [12] dataset. We use the publicly available code7 to compute FID.

C.2. Dataset

Table A.1. Descriptions on datasets.

dSprites [40] 737,280 binary 64x64 images of 2D sprites with 5 ground-truth factors, including shape (3), scale (6), orientation (40), x-position (32), and y-position (32).

Color-dSprites [7, 36] The sprite is filled with a random color. We randomly sample intensities of each color channel from 8 discrete values, linearly spaced between [0, 1].

Scream-dSprites [36] The background of each dSprites sample is replaced with a randomly-cropped patch of The Scream painting [] and the sprite is colored with the inverted color of the patch over the pixel regions of the sprite.

CelebA, CelebA-HQ [34, 23] CelebA dataset contains 202,599 RGB images of celebrity faces, which is composed of 10,177 identities, 5 landmark locations, and 40 annotated attributes of human faces. We use the aligned&cropped version of the dataset with the image size of 64256. CelebA-HQ is the subset of the in-the-wild version of the CelebA dataset, which is composed of 30,000 RGB 10241024 high-resolution images.

Cars [26] 16,185 RGB images of 196 classes of cars. We crop and resize each image into the size of 64using the bounding-box annotations provided.

C.3. Architecture

Table A.2. Architectures of -VAE and FactorVAE for all datasets. Note that Discriminator is needed only when training FactorVAE.

Input: 64 64 # channels Input: FC 1000, leaky ReLU 44 conv 32, ReLU, stride 2 FC 256, ReLU FC 1000, leaky ReLU 44 conv 32, ReLU, stride 2 FC 464, ReLU FC 1000, leaky ReLU 44 conv 64, ReLU, stride 2 44 upconv 64, ReLU, stride 2 FC 1000, leaky ReLU 44 conv 64, ReLU, stride 2 44 upconv 32, ReLU, stride 2 FC 1000, leaky ReLU FC 256, FC 210 44 upconv 32, ReLU, stride 2 FC 1000, leaky ReLU 44 upconv # channels, stride 2 FC 2

Table A.3. Architectures of Generator and Discriminator networks for ID-GAN and InfoGAN on dSprites, Color-dSprites, Noisy-dSprites, and Scream-dSprites datasets. The encoder and the decoder networks are specified in Table A.2.

Input: Input: 64 64 # channels FC 256, ReLU 44 conv 32, ReLU, stride 2 FC 464, ReLU 44 conv 32, ReLU, stride 2 44 upconv 64, ReLU, stride 2 44 conv 64, ReLU, stride 2 44 upconv 32, ReLU, stride 2 44 conv 64, ReLU, stride 2 44 upconv 32, ReLU, stride 2 FC 256, FC 1

Table A.4. Architectures of Generator and Discriminator networks for ID-GAN and InfoGAN on CelebA, 3D Chairs, and Cars () datasets. We directly follow the architecture proposed in [41]. The encoder and the decoder networks are specified in Table A.2.

Input: Input: 64 64 3 FC 4512 33 conv 64, stride 1 ResBlock 512, NN Upsampling ResBlock 64, AVG Pooling ResBlock 256, NN Upsampling ResBlock 128, AVG Pooling ResBlock 128, NN Upsampling ResBlock 256, AVG Pooling ResBlock 64, NN Upsampling ResBlock 512, AVG Pooling ResBlock 64, 44 conv 3, stride 1 FC 1

Table A.5. Architectures of Generator and Discriminator networks for ID-GAN (w/o distill), cGAN, and ID-GAN on the CelebA (128128) dataset. We directly follow the architecture proposed in [41]. The encoder and the decoder networks are specified in Table A.2.

Input: Input: 128 128 3

FC 4512 33 conv 64, stride 1

ResBlock 512, NN Upsampling ResBlock 64, AVG Pooling

ResBlock 512, NN Upsampling ResBlock 128, AVG Pooling ResBlock 512, NN Upsampling ResBlock 256, AVG Pooling ResBlock 256, NN Upsampling ResBlock 512, AVG Pooling ResBlock 128, NN Upsampling ResBlock 512, AVG Pooling ResBlock 128, 44 conv 3, stride 1 FC 1

Table A.6. Architectures of Generator and Discriminator networks for ID-GAN on the CelebA (256256) dataset. We directly follow the architecture proposed in [41]. The encoder and the decoder networks are specified in Table A.2.

Input: Input: 256 256 3

FC 4512 33 conv 64, stride 1

ResBlock 512, NN Upsampling ResBlock 64, AVG Pooling

ResBlock 512, NN Upsampling ResBlock 128, AVG Pooling ResBlock 512, NN Upsampling ResBlock 256, AVG Pooling ResBlock 256, NN Upsampling ResBlock 512, AVG Pooling ResBlock 128, NN Upsampling ResBlock 512, AVG Pooling ResBlock 64, NN Upsampling ResBlock 512, AVG Pooling ResBlock 64, 44 conv 3, stride 1 FC 1

Table A.7. Architectures of Generator and Discriminator networks for ID-GAN on the CelebA-HQ (10241024) dataset. We directly follow the architecture proposed in [46]. The encoder and the decoder networks are specified in Table A.2.

Input: Input: 1024 1024 3 FC 4512 ResBlock 16, AVG Pooling ResBlock 512, NN Upsampling ResBlock 32, AVG Pooling ResBlock 512, NN Upsampling ResBlock 64, AVG Pooling ResBlock 512, NN Upsampling ResBlock 128, AVG Pooling ResBlock 512, NN Upsampling ResBlock 256, AVG Pooling ResBlock 256, NN Upsampling ResBlock 512, AVG Pooling ResBlock 128, NN Upsampling ResBlock 512, AVG Pooling ResBlock 64, NN Upsampling ResBlock 512, AVG Pooling ResBlock 32, NN Upsampling 11 conv 2512, Sampling 512 ResBlock 16,44 conv 3, stride 1 FC 1