Myocardial infarction (MI) arises due to lasting damage suffered by cardiomyocytes as a result of blockage in blood circulation over an extended period of time to an area within the heart. In the course of the next few weeks following MI, permanently damaged cardiomyocytes are slowly restored by collagen containing scar tissue mainly comprising of fi-broblasts and extracellular matrix (ECM) proteins because of the lack of cardiomyocyte regeneration [1]. The passage of myocardial scar repair post infarction is time-dependent and is comprised of three different phases, namely, inflam-mation/necrosis, fibrosis/proliferation, and continued left ventricular remodeling/maturation [1]. Improper function of left ventricle due to MI and left ventricular remodeling can result in cardiac failure post MI [1].
The increased collagen contents [2] in fibrosis is traditionally analyzed with tedious histological and immunohistochemical methods which needs irreversible isolation of cardiac tissue specimens, dissection into thin sections using microtomes (1-10 m), embedding, fixation, and staining procedures. Moreover, acquiring myocardial tissue images in the clinical environments is difficult, as pretreatment procedures acting on sample processing can not only corrupt but also can induce structural modifications in tissue specimens.
Biopsy has been the gold standard for myocardial fibro-sis. Manual chemical staining and histopathology of these biopsy slides often lead to inter-observer variability, inadequate quantification of accuracy, and lack of high-confidence in the data. Another problem with large volume of samples is the labor cost and time of analysis. In order to reduce time and cost, pathologists often utilize partial sampling strategy where instead of analyzing all the slides, only few randomly chosen slides are analyzed. Since, the diagnosis and therapy rely on these histopathological assessments, the current procedures lead to inconsistent, inaccurate, and unreliable results leading to unfavorable patient outcomes [3]. Hence, there is an unmet need to develop an automated technique to perform label free assessment of cardiac tissues using their inherent molecular contrast.
Recent years have seen the rise of non-linear optical microscopy techniques like second harmonic generation (SHG) microscopy [4] as powerful techniques for non-invasive imaging of extracellular matrix (ECM) and cellular structures [4] of thick tissue specimens in their native environments [4]. Second Harmonic Generation (SHG) [4] prvides a coherent second order nonlinear optical scattering method usually by molecules having asymmetric structures which allows conversion of two lower energy photons into single photon with twice the incident energy of an excitation laser light. The SHG photon is nonisotropic hence polarization-sensitive and can be synthesized within femto-seconds. This allows using femtosecond pulsed lasers, to produce SHG signals with comparable reaction time scales. In contrast to conventional confocal imaging penetration depth (m), SHG has higher penetration depth (
m) to image thick tissue specimens and intrinsic optical sectioning capability [5]. SHG microscopy is a label-free technique for imaging native structure of tissue specimens without any requirement for staining, thus preventing any corruption due to staining [4].
In this study, we infer highly relevant second harmonic
Fig. 1. Architecture of our proposed framework.
generation (SHG) cues from collagen fibers exhibiting highly non-centrosymmetric assembly together with two-photon excited cellular autofluorescence in infarcted mouse heart to quantitatively probe fibrosis, especially targeted at an early stage after MI. Employing a computational framework comprising robust one-shot machine learning algorithms, we determine 2D assembly of collagen with high spatial resolution along with its structural arrangement in heart tissues post-MI with spectral specificity and sensitivity (we limit this study to 2D as obtaining high resolution 3D SHG images through collection of Z-stacks of images on whole mount or thick tissue specimens would require lot of time and resources). More precisely, the framework uses a detection network for localizing the left ventricle in an SHG microscopy image of a heart and a generative adversarial network (GAN) [6] to infer a segmentation mask for the infarction area (in the left ventricle), by estimating the heat map representing the anomaly between a single SHG microscopy image of a normal heart and that of an infarcted one. The segmented infarction area is used to compute an infarction score to quantitatively assess the fibrosis extent. Detection, evaluation, and precise quantification of fibrosis extent at early stage would guide one to develop treatment therapies that may prevent further progression and heart transplant needs for patient survival.
2.1. Framework Overview
Fig. 1 presents the architecture for the one-shot machine learning and image processing pipeline that constitutes the basis of our framework. The pipeline consists of two components: a detection network for localizing the left ventricle in an SHG microscopy image of a heart and a GAN to learn a segmentation mask for the infarction area (in the left ventricle), by estimating the heat map representing the anomaly [7] between an SHG microscopy image of a normal heart and that of an infarcted one. A GAN [6, 8] trains two models simultaneously. The first model G, called the generator, generates samples by attempting to capture the data distribution of the training set (patches from a single SHG microscopy image of a normal, i.e., non-infarcted, heart); and the second model D, called the discriminator, discriminates between a created sample by G and “normal” images from the training distribution. Finally, we use the segmented infarction area to compute the infarction score, a measure that we have proposed (see below) to quantitatively assess the heart risk due to the infarction.
Segmenting the infarction area in the SHG microscopy image of an infarcted heart is challenging due to the lack of training data. Standard deep learning-based segmentation methods [9, 10, 11] require large volumes of labeled training data and hence are unsuitable for the present purpose. Hence, we adopted an unsupervised learning pipeline that uses a GAN to efficiently learn features for computing the segmentation mask for the infarction area, by estimating the heat map representing the anomaly between an SHG microscopy image of a normal heart and that of an infarcted one. Existing threshold-based segmentation techniques cannot automatically estimate the anomaly between the normal heart and that of an infarcted one to quantify the risk of a cardiac attack.
2.2. Detection Network
It is known that infarction in the left ventricle is critical with respect to quantitative assessment of heart risk [12]. From a raw SHG microscopy image of a heart, we first use a detection network to determine the region of interest (ROI) focusing on the left ventricle. For the detection network, we adopted a pretrained (on the Coco dataset) object detection framework, known as YOLO [13]. The detection network computes the ROI that will be used for segmentation of the infarcted area.
Fig. 2. Segmentation of infarcted region in left ventricle. The top row shows the detection of the region of interest (left ventricle) using detection network. The second row presents the left ventricular region. Third row exhibits the residual difference heat map for infarction area computed as segmentation mask from the GAN. And the bottom row, shows the segmentation of the infarction area based on the segmentation mask.
2.3. GAN-based Segmentation Mask Estimation
Patches from the SHG microscopy image of a single normal, i.e., non-infarcted, heart are used as training data for GAN-based segmentation mask estimation. The SHG microscopy images of infarcted heart collected at different timepoints were used for testing. The GAN architecture of our framework is similar to that in [7]. Let Mic be the set of N SHG microscopy images of a heart collected at different timepoints (before and after surgical ligation; see below), where each image , is of size of
, with
being the set of pixel intensities. Let
be the SHG microscopy image of a normal (non-infarcted) heart. We randomly extract M image patches of size
from I. These patches constitute our training dataset X. All input images are resized to a fixed input size of
. Being an unsupervised setting, we did not use any labels for training. The loss function for the GAN consists of two components.
Residual Loss For a given patch input to the discriminator, where x follows the probability distribution
, the generator G will generate an image G(z) with a random noise vector z (following the distribution
) as input. We use the residual loss function to ensure that the generated image G(z) is similar to the patch x input to the discriminator. The residual loss function [7] defined by,
, where E represents expectation.
Feature Matching Loss Similar to [7], we use the feature matching loss in the feature space of discriminator without using labels in an unsupervised setting. The discriminator works as a feature extractor rather than a classifier. The feature matching loss defined by, , where E represents expectation,
represents the feature extractor for the intermediate layer of the discriminator. Finally, the overall loss for GAN based on both losses, defined by,
, where
(we use
in the experiments below).
Segmentation Since, in the SHG microscopic image of an infarcted heart, the accumulation of collagen is signalled by the green channel, the segmentation of the infarction area (in the left ventricle) of the SHG microscopic image of an infarcted heart entails identification of the “green” pixels within the ROI (designating the left ventricle). For a predicate p de-fined over the set of pixels x, y in an image I, define the function that evaluates to 1 if the predicate p is true on the pixel (x, y), and 0 otherwise. Let
be the generator of the GAN after training converges. For an image I with ROI
, let
(resp
) be the mean green (resp. red) channel intensity within
. Then for a query image
with ROI
(left ventricle), the green segmentation mask is given by
where denotes the green channel value at pixel (x, y) of
is the ROI for the image
generated by the generator
, and z is a noise vector. Similarly, the red segmentation mask for
is given by
where denotes the red channel value at pixel x, y of
.
2.4. Infarction Score
For assessment of the risk associated with an infarcted heart, we propose an Infarction Score. Since, in the SHG micro-
Fig. 3. Infarction score corresponding to temporal stages after introduction of myocardial infarction in left ventricle.
Fig. 4. t-SNE visualization of the bottleneck repre- sentations from discriminator for infarcted and normal images.
scopic image of an infarcted heart, the accumulation of collagen is signalled by the green channel, the infarction score will measure the proportion of “green” pixels in the segmented infarction area. For a query image , let
and
. Then the infarction score for
is given by
.
In this section, we evaluate our proposed framework on our collected SHG Heart Imagery Dataset.
SHG Heart Imagery Dataset We induced MI in mice by permanently blocking flow of blood in the left coronary artery using surgical ligation permanent surgical ligation [14]1. Heart tissue sections were processed and obtained following protocols published elsewhere [15]. SHG and Two-photon excited fluorescence imaging was performed using a Leica SP5 resonant scanning multiphoton confocal microscope (Leica Microsystem), coupled to a Spectra Physics Mai-Tai tunable pulsed near-IR laser (690 - 1040 nm) using 70 femtosecond pulses with 84 MHz repetition rate. We carried out all SHG experiments with an illumination wavelength of 860 nm. We collected SHG emission in the epi direction using the 320-430 nm range band pass filters while in case of aut-ofluorescence 486-506 nm bandpass filter was used. After the surgical ligation procedure, SHG microscopic imaging was performed after 24 hours, three days, four days, two weeks, and four weeks [2, 16]. The size of the images were around on the average. For separate validation of our MI model through collagen quantification, we analyzed SHG microscopy picture of picrosirius red-stained infarcted mouse heart tissues. The stained images have the same size as that of the unstained ones and were collected for the same timepoints after the surgical ligation procedure.
Experimental Setup The generator of the GAN consisted of one dense layer (128 neurons) followed by two conv-transpose layers (64 neurons each). The discriminator consisted of two convolutional layers (first: 64 neurons, second 128 neurons) followed by a dense layer (1 neuron). We used the bottleneck features of the discriminator. Due to the limited size of our dataset, we only used a single unstained image of normal non-infarcted heart for training the GAN (trained 500 epochs). We randomly extracted 1000 patches with patch size of from this training image. When testing, the input test image was be resized to
to fit into the input size of the discriminator for computing the mean green channel value for further segmentation. For picrosirius red-stained images too, we trained the GAN using a a single stained image of normal non-infarcted heart.
Experimental Results In Fig.2, we can see that our proposed method based on one-shot learning shows good performance for the segmentation task. Fig.4 shows the representations learned by the discrimination. Fig.4 shows separation between features of SHG images of normal heart (red dots) and those corresponding to images of infarcted heart. Since the discriminator has learned enriched feature information for normal images, the feature variety results in the red dots scattered throughout the space. Fig.3 shows the temporal variation in the infarction score after the surgical ligation procedure. Over the course of a week, there have been drastic changes in collagen concentration in the left ventricle. For unstained data (indicated by the blue line in Fig.3), on the first day after MI, the infarction score was around 0.31 increasing to 0.34 after three days. However, by 4 days, it had gone down to 0.24. By the 28th day however, it increases to 0.41. This is shown in the images as well (see Fig.2). By the second week, there is a significant portion of green channel. The levels of red (myofibers) start to go down to thinner and thinner levels as well. The results for the stained data (the red line in Fig.3) shows similar trend as that for the unstained data.
In this study, we presented a one-shot machine learning approach that is able to segment fibrillar collagens in two-photon excited cellular auto-fluorescence in infarcted mouse heart images in an unsupervised fashion [17] using a deep generative adversarial network. Our framework was able to segment the anomalies presented in the left ventricle due to myocardial infarction by identifying fibrillar collagens and was able to assess the risk using an infarction score.
[1] William J Richardson, Samantha A Clarke, T Alexan- der Quinn, and Jeffrey W Holmes, “Physiological implications of myocardial scar structure,” Comprehensive Physiology, vol. 5, no. 4, pp. 1877, 2015.
[2] Nagahiro Nishikawa, Tohru Masuyama, Kazuhiro Ya- mamoto, Yasushi Sakata, Toshiaki Mano, Takeshi Miwa, Motoaki Sugawara, and Masatsugu Hori, “Longterm administration of amlodipine prevents decompensation to diastolic heart failure in hypertensive rats,” Journal of the American College of Cardiology, vol. 38, no. 5, pp. 1539–1545, 2001.
[3] S. Sitharama Iyengar, Xin Li, Huanhuan Xu, Supratik Mukhopadhyay, N. Balakrishnan, Amit Sawant, and Puneeth Iyengar, “Toward more precise radiotherapy treatment of lung tumors,” IEEE Computer, vol. 45, no. 1, pp. 59–65, 2012.
[4] Valentina Caorsi, Christopher Toepfer, Markus B Sikkel, Alexander R Lyon, Ken MacLeod, and Mike A Ferenczi, “Non-linear optical microscopy sheds light on cardiovascular disease,” PLoS One, vol. 8, no. 2, pp. e56136, 2013.
[5] Thomas Abraham, Jeremy A Hirota, Samuel Wadsworth, and Darryl A Knight, “Minimally invasive multiphoton and harmonic generation imaging of extracellular matrix structures in lung airway and related diseases,” Pulmonary pharmacology & therapeutics, vol. 24, no. 5, pp. 487–496, 2011.
[6] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio, “Generative adversarial nets,” in Advances in Neural Information Processing Systems 27, Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, Eds., pp. 2672– 2680. Curran Associates, Inc., 2014.
[7] Thomas Schlegl, Philipp Seeb¨ock, Sebastian M Wald- stein, Ursula Schmidt-Erfurth, and Georg Langs, “Unsupervised anomaly detection with generative adversarial networks to guide marker discovery,” in International Conference on Information Processing in Medical Imaging. Springer, 2017, pp. 146–157.
[8] Qun Liu, Edward Collier, and Supratik Mukhopadhyay, “Pcgan-char: Progressively trained classifier generative adversarial networks for classification of noisy handwritten bangla characters,” in International Conference on Asian Digital Libraries. Springer, 2019, pp. 3–15.
[9] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla, “Segnet: A deep convolutional encoderdecoder architecture for image segmentation,” IEEE
transactions on pattern analysis and machine intelligence, vol. 39, no. 12, pp. 2481–2495, 2017.
[10] Manohar Karki, Robert DiBiano, Saikat Basu, and Supratik Mukhopadhyay, “Core sampling framework for pixel classification,” in Artificial Neural Networks and Machine Learning - ICANN 2017 - 26th International Conference on Artificial Neural Networks, Alghero, Italy, September 11-14, 2017, Proceedings, Part II, 2017, pp. 617–625.
[11] Edward Collier, Kate Duffy, Sangram Ganguly, Geri Madanguit, Subodh Kalia, Shreekant Gayaka, Ramakrishna R. Nemani, Andrew R. Michaelis, Shuang Li, Auroop R. Ganguly, and Supratik Mukhopadhyay, “Progressively growing generative adversarial networks for high resolution semantic segmentation of satellite images,” in 2018 IEEE International Conference on Data Mining Workshops, ICDM Workshops, Singapore, Singapore, November 17-20, 2018, 2018, pp. 763–769.
[12] Anja M van der Laan, Matthias Nahrendorf, and Jan J Piek, “Healing and adverse remodelling after acute myocardial infarction: role of the cellular immune response,” Heart, vol. 98, no. 18, pp. 1384–1390, 2012.
[13] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi, “Youonly look once: Unified, real-time object detection,” arXivpreprint arXiv:1506.02640, 2015.
[14] Jop H Van Berlo, Onur Kanisicak, Marjorie Mail- let, Ronald J Vagnozzi, Jason Karch, Suh-Chin J Lin, Ryan C Middleton, Eduardo Marb´an, and Jeffery D Molkentin, “C-kit+ cells minimally contribute cardiomyocytes to the heart,” Nature, vol. 509, no. 7500, pp. 337, 2014.
[15] Xing Fu, Hadi Khalil, Onur Kanisicak, Justin G Boyer, Ronald J Vagnozzi, Bryan D Maliken, Michelle A Sargent, Vikram Prasad, I˜nigo Valiente-Alandi, Burns C Blaxall, et al., “Specialized fibroblast differentiated states underlie scar formation in the infarcted mouse heart,” The Journal of clinical investigation, vol. 128, no. 5, 2018.
[16] JP Cleutjens, MJ Verluyten, JF Smiths, and MJ Daemen, “Collagen remodeling after myocardial infarction in the rat heart.,” The American journal of pathology, vol. 147, no. 2, pp. 325, 1995.
[17] Qun Liu and Supratik Mukhopadhyay, “Unsupervised learning using pretrained cnn and associative memory bank,” in 2018 International Joint Conference on Neural Networks (IJCNN). IEEE, 2018, pp. 01–08.