Progressively-Growing AmbientGANs For Learning Stochastic Object Models From Imaging Measurements
2020·Arxiv
ABSTRACT

The objective optimization of medical imaging systems requires full characterization of all sources of randomness in the measured data, which includes the variability within the ensemble of objects to-be-imaged. This can be accomplished by establishing a stochastic object model (SOM) that describes the variability in the class of objects to-be-imaged. Generative adversarial networks (GANs) can be potentially useful to establish SOMs because they hold great promise to learn generative models that describe the variability within an ensemble of training data. However, because medical imaging systems record imaging measurements that are noisy and indirect representations of object properties, GANs cannot be directly applied to establish stochastic models of objects to-be-imaged. To address this issue, an augmented GAN architecture named AmbientGAN was developed to establish SOMs from noisy and indirect measurement data. However, because the adversarial training can be unstable, the applicability of the AmbientGAN can be potentially limited. In this work, we propose a novel training strategy—Progressive Growing of AmbientGANs (ProAGAN)—to stabilize the training of AmbientGANs for establishing SOMs from noisy and indirect imaging measurements. An idealized magnetic resonance (MR) imaging system and clinical MR brain images are considered. The proposed methodology is evaluated by comparing signal detection performance computed by use of ProAGAN-generated synthetic images and images that depict the true object properties.

Keywords: stochastic object model, generative adversarial networks, signal detection, objective assessment of image quality

It has been widely accepted that the optimization of medical imaging systems should be guided by objective measures of image quality (IQ) that quantify the performance of an observer at specific tasks.1–5The performance of a numerical observer ideally should account for all sources of randomness in the measured data, which includes the variability within a group of objects being imaged.1–7To achieve this, one can establish a stochastic object model (SOM) that can sample numerous realizations of objects from the ensemble of objects to-be-imaged. To establish a SOM that describes the to-be-imaged distribution, it is desirable to establish it from experimental measurements. Moreover, the established SOM should be independent of the imaging system. Kupinski et al. developed a method to fit SOMs by use of noisy imaging measurements.6However, the reported applications have been limited to lumpy and clustered lumpy object models.6In order to establish more complicated SOMs, Zhou et al. implemented a deep learning method using an augmented generative adversarial network (GAN) architecture called an AmbientGAN8to learn SOMs from noisy and indirect measurements.9In that proof-of-concept study, a simple lumpy object model was considered.9However, it is well-known that training of GANs

image

can be unstable and, therefore, in practice, it can be difficult to learn the statistical properties of the to-be-imaged distribution.

A recently developed GAN training methodology—Progressive Growing of GANs (ProGANs)—holds great promise to stabilize the training of GANs and has been successfully employed to generate high-quality synthetic images with high resolutions.10Unlike the conventional training of GANs, in which all scales of the image distribution are learned simultaneously, the training of ProGANs starts with low-resolution images. Subsequently, network layers are added progressively to the generator and discriminator in synchrony to increase the image resolution for learning finer scale details of the image distribution. Here, the generator and the discriminator have mirrored architectures. It has been shown that GAN training stability and the quality of the produced synthetic images can be improved significantly by use of this training strategy.10However, because medical imaging systems record noisy and indirect measurements of object properties, ProGANs cannot be immediately applied to establish stochastic models of objects to-be-imaged.

In this study, inspired by the ProGAN10and the AmbientGAN,8we propose a novel training methodology— Progressive Growing of AmbientGANs (ProAGANs)—to stably train AmbientGANs to establish SOMs from noisy and indirect imaging measurements. An idealized magnetic resonance (MR) imaging system and clinical MR brain images from NYU fastMRI Initiative database11were considered. The proposed methodology was evaluated by comparing signal detection performance computed by use of ProAGAN-generated synthetic images and images that depict the true object properties.

Consider a linear discrete-to-discrete imaging system:

image

where  f ∈ Rn2denotes a vector of an object image having dimension of  n×n, H : Rn2 → Rmdenotes an imaging operator,  g ∈ Rmdenotes a vector of imaging measurement data, and  n ∈ Rmdenotes measurement noise.

2.1 GANs and AmbientGANs

Generative adversarial neural networks (GANs) were developed to learn data distributions by training a generator through an adversarial process with a discriminator.12Here, the generator is represented by a deep neural network having a set of weight parameters  θGand a mapping function  G(· ; θG): Rk → Rn2that maps a latent vector z ∈ Rkto a synthetic image ˆf = G(z; θG). The discriminator is represented by another deep neural network having a set of weight parameters  θDand a mapping function  D(· ; θD): Rn2 → Rthat maps a real or synthetic image to a real-valued scalar s that can be used to distinguish between real and synthetic images. However, because medical imaging systems acquire noisy and indirect imaging measurements g, GANs cannot be directly employed to establish SOMs.

An augmented GAN architecture named AmbientGAN was developed to learn SOMs from noisy and indirect measurements of object properties.8,9Similar to GANs, AmbientGANs comprise a generator and a discriminator. The generator is trained to produce synthetic digital images that depict the object properties. Synthetic imaging measurements are subsequently simulated by applying the corresponding imaging operator H to the finite-dimensional representations of the object property produced by the generator. The discriminator that maps Rm → Ris applied to distinguish between real and synthetic imaging measurements. The training process of AmbientGANs can be represented by a two-player minimax game:

image

Here, ˆg = Hˆf +n, where ˆf = G(z; θG), and  l(·) represents an objective function. It has been shown that when  pfuniquely induces  pg, and both the discriminator and the generator possess sufficient capacity,  pˆfapproximates pfwhen the global optimal of the minimax game is achieved.8,12

2.2 Progressive Growing of GANs

Progressive Growing of GANs (ProGANs) were developed to stabilize the training of GANs.10Unlike the conventional GANs, in which the generator and the discriminator are trained to learn all scales of image distribution by use of full resolution images through the whole training process, the training of ProGANs start with training the first few layers of the generator and discriminator on low-resolution images that are down-scaled from the original images. Higher-resolution images are subsequently employed and more layers of the generator and discriminator are progressively included in the training to learn finer scale details of the image distribution. It has been shown that the ProGAN training strategy results in significantly improved training stability and produces synthetic images with the state-of-the-art image quality.10However, like other GANs, ProGANs cannot be immediately applied to learn SOMs from noisy and/or indirect imaging measurements.

In this work, we propose a novel training strategy named Progressive Growing of AmbientGANs (ProAGANs) to stably train the AmbientGANs for establishing SOMs from noisy and indirect imaging measurements. A MR imaging system that fully samples k-space data was considered: g = F(f) + n, where  F(·) denotes a 2D discrete Fourier transform (DFT). The generator in the proposed ProAGAN was trained to synthesize images that depict object properties in the to-be-imaged distribution. However, because the MR imaging system records noisy k-space measurement data, the discriminator cannot be applied directly to distinguish between real and synthetic object images. In our proposed architecture of ProAGAN, a 2D inverse discrete Fourier transform (IDFT)  F−1is included in the training of ProAGAN to reconstruct object images from k-space measurements, and the discriminator is trained to distinguish between real and synthetic reconstructed images. The goal is to learn the distribution of images that depict to-be-imaged object properties by progressively training a generator by competing against a discriminator that distinguishes between real and synthetic reconstructed images. This training process is illustrated in Fig. 1.

image

Figure 1: ProAGAN training for the considered MR imaging system. The training started with low-resolution images, and then increased the image resolution progressively by including more layers to the generator and the discriminator in the training. The synthetic k-space measurements were simulated by use of the DFT. The IDFT was applied to reconstruct real and synthetic object images. The discriminator was trained to distinguish between real and synthetic reconstructed images.

A simulation study was conducted. MR brain images in the NYU fastMRI Initiative database11(https:// fastmri.med.nyu.edu/) were employed to form an ensemble of images that depict object properties sampled from the unknown SOM. Specifically, 3000 T1 weighted brain MR images corresponding to the magnetic field strength of 3T were selected, and these 3000 images were resized to the dimension of 128  ×128 to be employed as real object images. Fully-sampled MR k-space data of these 3000 object images were simulated, and complex Gaussian noise were added to the k-space data. These 3000 noisy k-space measurement data formed the training dataset. An example of MR brain images, its corresponding k-space measurement data and the reconstructed image (i.e., IDFT of k-space measurement data) are shown in Fig. 2.

image

Figure 2: (a) A MR brain image. (b) k-space measurement data corresponding to (a). (c) The IDFT of (b). A central region in (a) and (c) was displayed to better visualize the noise in the reconstructed image.

4.1 Training details

The proposed ProAGAN was implemented in Tensorflow13by modifying the ProGAN codes to apply the architecture illustrated in Fig. 1. The codes for the ProGAN can be found at https://github.com/tkarras/ progressive_growing_of_gans. The ProAGAN was trained by use of 4 NVIDIA Tesla V100 GPUs. The Adam algorithm,14which is a stochastic gradient algorithm, was employed to train ProAGANs.

4.2 Hotelling observer validation studies

Signal-known-exactly (SKE) binary signal detection tasks were conducted to assess the learned SOM in a taskdependent way. The considered signal detection tasks require an observer to classify a MR brain image as satisfying signal-absent hypothesis (H0) or signal-present hypothesis (H1):

image

where s denotes the signal to be detected, and n is independent and identically distributed Gaussian noise. Two different signals represented by two artificial tumors were considered. These signals are shown in Fig. 3.

image

Figure 3: An example of signal-absent images, two signals, and the corresponding signal-present images. The region of interest (ROI) where the signals may be presented is displayed.

The signal detection tasks were performed on a region of interest (ROI) that was centrally located in the images. The ROIs had dimension of 32  ×32. The covariance matrices corresponding to the real and ProAGANproduced images were computed by use of 3000 ROIs extracted from the real and ProAGAN-produced images, respectively. These covariance matrices were subsequently employed to computed the Hotelling observer (HO) by use of a covariance matrix decomposition.1The performances of the HOs were evaluated on a testing dataset that comprised 500 pairs of signal-absent and signal-present images. The receiver operating characteristic (ROC) curves and AUC values corresponding to the ProAGAN-produced images were compared to those corresponding to the real images. The proper binormal model15was employed to fit the ROC curves.

Examples of real and synthetic images of object properties produced by the ProAGAN are shown in Fig. 4. The synthetic images are promising in terms of visually mimicking the real images.

image

Figure 4: (a)-(e) Real MR brain images. (f)-(j) Synthetic MR brain images produced by the ProAGAN.

To illustrate the progressive growing of ProAGAN, synthetic images at different training steps with different resolution levels are shown in Fig. 5. It is demonstrated that the generator was progressively established in a stable way.

image

Figure 5: (a)-(f) Synthetic images produced by the generator at different training steps corresponding to the resolution of 4  ×4, 8  ×8, 16  ×16, 32  ×32, 64  ×64, and 128  ×128.

The Hotelling templates that were computed by use of the real images and the ProAGAN-produced images are compared in Fig. 6. The ROC curves and AUC values corresponding the real images and the ProAGANproduced images are compared in Fig. 6 (g), in which the solid curves and the dashed curves correspond to the HOs computed by use of real images and synthetic images produced by the ProAGAN, respectively. The ROC curves are almost identical.

image

Figure 6: (a) ROI corresponding to signal 1. (b) Hotelling template corresponding to signal 1 that was computed by use of ROIs of real images. (c) Hotelling template corresponding to signal 1 that was computed by use of ROIs of synthetic images produced by the ProAGAN. (d) ROI corresponding to signal 2. (e) Hotelling template corresponding to signal 2 that was computed by use of ROIs of real images. (f) Hotelling template corresponding to signal 2 that was computed by use of ROIs of synthetic images produced by the ProAGAN. (g) ROC curves and AUC values corresponding to real and synthetic images. The ROC curves almost overlap.

This study provides a novel training methodology—Progressive Growing of AmbientGANs—to stably train AmbientGANs for learning SOMs from noisy and indirect measurement data. In this preliminary study, an idealized magnetic resonance (MR) imaging system was considered, and a task-based validation study was conducted by use of the Hotelling observer. It is demonstrated that the proposed method is able to establish a SOM by use of noisy k-space measurement data. From the perspective of the HO, for the signal detection tasks considered, it was found that the images synthesized by the learned SOM contained nearly the same task-specific information as the true object property images.

This research was supported in part by NIH awards EB020604, EB023045, NS102213, EB028652, and NSF award DMS1614305.

[1] Barrett, H. H. and Myers, K. J., [Foundations of Image Science], John Wiley & Sons (2013).

[2] Zhou, W. and Anastasio, M. A., “Learning the Ideal Observer for SKE detection tasks by use of convolutional neural networks,” in [Medical Imaging 2018: Image Perception, Observer Performance, and Technology Assessment], 10577, 1057719, International Society for Optics and Photonics (2018).

[3] Zhou, W. and Anastasio, M. A., “Learning the ideal observer for joint detection and localization tasks by use of convolutional neural networks,” in [Medical Imaging 2019: Image Perception, Observer Performance, and Technology Assessment], 10952, 1095209, International Society for Optics and Photonics (2019).

[4] Zhou, W., Li, H., and Anastasio, M. A., “Learning the hotelling observer for ske detection tasks by use of supervised learning methods,” in [Medical Imaging 2019: Image Perception, Observer Performance, and Technology Assessment], 10952, 1095208, International Society for Optics and Photonics (2019).

[5] Zhou, W., Li, H., and Anastasio, M. A., “Approximating the ideal observer and hotelling observer for binary signal detection tasks by use of supervised learning methods,” IEEE transactions on medical imaging (2019).

[6] Kupinski, M. A., Clarkson, E., Hoppin, J. W., Chen, L., and Barrett, H. H., “Experimental determination of object statistics from noisy images,” JOSA A 20(3), 421–429 (2003).

[7] Dolly, S. R., Lou, Y., Anastasio, M. A., and Li, H., “Learning-based stochastic object models for character- izing anatomical variations,” Physics in Medicine & Biology 63(6), 065004 (2018).

[8] Bora, A., Price, E., and Dimakis, A. G., “Ambientgan: Generative models from lossy measurements,” in [International Conference on Learning Representations (ICLR)], (2018).

[9] Zhou, W., Bhadra, S., Brooks, F., and Anastasio, M. A., “Learning stochastic object model from noisy imaging measurements using AmbientGANs,” in [Medical Imaging 2019: Image Perception, Observer Performance, and Technology Assessment], 10952, 109520M, International Society for Optics and Photonics (2019).

[10] Karras, T., Aila, T., Laine, S., and Lehtinen, J., “Progressive Growing of GANs for improved quality, stability, and variation,” arXiv preprint arXiv:1710.10196 (2017).

[11] Zbontar, J., Knoll, F., Sriram, A., Muckley, M. J., Bruno, M., Defazio, A., Parente, M., Geras, K. J., Katsnelson, J., Chandarana, H., et al., “fastmri: An open dataset and benchmarks for accelerated mri,” arXiv preprint arXiv:1811.08839 (2018).

[12] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y., “Generative adversarial nets,” in [Advances in neural information processing systems], 2672–2680 (2014).

[13] Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al., “Tensorflow: a system for large-scale machine learning.,” in [OSDI], 16, 265–283 (2016).

[14] Kingma, D. P. and Ba, J., “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 (2014).

[15] Pesce, L. L. and Metz, C. E., “Reliable and computationally efficient maximum-likelihood estimation of “proper” binormal roc curves,” Academic radiology 14(7), 814–829 (2007).