A latent topic model for mining heterogenous non-randomly missing electronic health records data

2018·arXiv

Abstract

1 Introduction

With recent healthcare initiatives, many hospitals routinely generate large number of electronic health records (EHR) data. In the United States, the number of non-federal acute care hospitals with basic digital systems increased from 9.4 to 96% over the 7 year period between 2008 and 2015 [1–3]. Furthermore, the amount of comprehensive EHR data recording not only clinical notes but also other data types increased from only 1.6% in 2008 to 40% in 2015 [3]. These data types are largely standardized into systematic codes such as Logical Observation Identifiers Names and Codes (LOINC) lab test codes, Diagnosis-Related Groups (DRG) code, International Classifica-tion of Diseases (ICD)-9 code, and prescription for electronic drug information exchange code (RxNorm). However, these raw EHR data present several challenges including interpretation of high-dimensional, heterogeneous and extremely sparse clinical features, and extremely biased measurements based on patient diagnoses and self-reported condition, formally known as non-missing at random (NMAR) [4].

To address these challenges, we developed a Bayesian unsupervised learning approach that builds on the concepts of collaborative filtering [5–9] and latent topic models [10–15]. While it is often infeasible to directly model the joint distribution over the entire EHR feature space, it is possible to formulate a latent topic model over discrete data. Each patient, hereby considered as a “document", exhibits a mixture of “memberships" over a set of latent “disease topics" each prescribing a distinct spectrum of disease frequency. In this paper, we present MixEHR for modeling heterogeneous EHR data. MixEHR can simultaneously model an arbitrary number of EHR categories with separate discrete distributions while jointly modeling the lab records and the lab results to account for the NMAR observation biases. For efficient Bayesian learning, we develop a variational inference algorithm that scales to large-scale patient EHR data. We apply MixEHR to the largest public dataset called Medical Information Mart for Intensive Care (MIMIC)-III [1]. We first used MIMIC-III to develop a simulation framework to evaluate MixEHR. We find that the clinical features emerging across EHR categories under common disease topics are biologically meaningful and reveal new insights into disease comorbidity. We also find these disease topics to be highly predictive of mortality.

2 Related work and our contributions

Our method is related to the widely popular text-mining method latent Dirichlet allocation (LDA) [10]. However, LDA is not well suited to EHR phenotyping because (1) LDA assumes that the clinical “terms" are completely observed in the patient “document", whereas as mentioned above EHR data such as lab tests are often missing; (2) LDA is not suitable to jointly model heterogeneous EHR data categories with distinct distributions; (3) even variations of LDA that deal with missing data do not cope with NMAR mechanism. Our method is also related to several EHR phenotyping studies focusing on matrix factorization and model interpretability [16–20]. These unsupervised approaches differ from the recent supervised deep learning methods that primarily focus on prediction performance of target clinical outcomes [21–27]. Several recent reviews are helpful in gaining perspectives on the recent machine learning advancements in healthcare [28–30]. [18] proposed a multi-view LDA with Gibbs sampling for a fixed set of data types. [31] proposed a single-view nonprobabilistic modeling of clinical notes, requiring predefined data to learn interpretable disease topics. [32] jointly model multiple data matrices but require expert-curated tagging data. [33] proposed a collective matrix factorization approach on multi-source EHR data. In contrast to these methods, we jointly model lab tests and lab results while inferring missing lab results to account for NMAR bias. Moreover, our joint collapsed variational Bayesian (CVB) inference algorithm provides fast convergence to good local solutions compared to the Gibbs sampling in [18]. We extend CVB for LDA only [13–15] by inferring the joint expectations of latent topics and lab results not only for the observed but also missing lab tests, which enables us to account for the NMAR observation bias common in the EHR data. We implemented MixEHR in C++ as a standalone Unix command-line software using OpenMP for multi-threaded inference. It allows arbitrary number of discrete data types and discrete states per EHR feature. On a 8-core Ubuntu server, MixEHR takes 2 hours to learn 100 disease topics from 40,000 patient records over 50,000 EHR features across 6 data types.

3 MixEHR: a latent topic model for EHR data

We model EHR data using a generative latent topic model (Fig. 1). Suppose there are K latent disease topics, each topic under data type prescribes a vector of unknown weights for EHR features, which follows a Dirichlet distri- bution with unknown hyperparameter . Additionally, each topic is also characterized by a set of unknown Dirichlet-distributed weights with distinct values. For each patient , the disease mixture membership is generated from the K-dimensional Dirichlet

Figure 1: Proposed MixEHR model. See main text for details.

Figure 2: Averaged predictive log likelihood on held-out patients in 5-fold cross-validation.

distribution with unknown asymmetric hyperparameters . To generate EHR feature i for patient j, a latent topic under data type t is first drawn from multinomial distribution with rate . Given the latent topic , a clinical feature is drawn from multinomial distribution with rate equal to .

For lab data, we use a generative process where each patient has a variable for the result for every lab test regardless whether the results is observed. One important assumption we make here is that the lab results and lab observation for patient j are conditionally independent given the latent topic , namely that the probability of taking the test and the value of that test both depend on the true disease status, but not on each other. In terms of the generative process, we first sample the latent variable from the multinomial distribution with rate . Conditioned on the latent variable, we then concurrently sample (1) the lab result from -dimensional Dirichlet distribution with hyperparameters over values and (2) the lab observation indicator from Binomial distribution1 with rate , which follows a Beta distribution with unknown shape and scale hyperparameters and . The hyperparameters ’s follow unknown Gamma distributions.

Formally, we first generate global variables for disease topics:

We then generate local variables for the binary EHR features of each patient:

and the local variables for the lab data including latent topic, lab result, and observation indicator:

where Gam(.), Dir(.), Bet(.), Mul(.), and Bin(.) denote Gamma, Dirichlet, Beta, Multinomial, and Binomial distributions, respectively. The notations are in Supplementary Information Table 1.

4 Variational Bayesian learning of MixEHR model

4.1 Marginalized likelihood integrating out

Treating the latent variables as missing data, we can express the complete joint likelihood based on our model as follows:

At this point, we could formulate a variational inference algorithm by optimizing an evidence lower bound of the marginal likelihood with respect to the model parameters [34]. However, due to the respective conjugacy of Dirichlet variables to the multinomial likelihood variables x, y, {z, h}, and the conjugacy of Beta variable to the binomial lab observation indicator variable r, we can achieve better inference by first integrating out the Dirichlet and Beta variables and then inferring

directly the distribution of the latent variables {z, h} as follows [14,35]:

where the sufficient statistics are

Note that we use to denote the frequency of observing the lab test l of outcome v for patient j and as binary indicator of a single test. Detailed derivation is in Supplementary Information.

4.2 Joint collapsed variational Bayesian inference

The marginal likelihood can be approximated by evidence lower bound (ELBO):

Maximizing is equivalent to minimizing Kullback-Leibler (KL) divergence:

Under mean-field factorization, the proposed distribution of latent variables z and h are defined as:

Maximizing (6) with respect to the variational parameter and is equivalent to calculating the expectation of and with respect to all of the other latent variables [14,36]:

Normalizing the distribution of and gives

We can approximate these expectations by first deriving the conditional distribution for and (Supplementary Information) and then approximating the sufficient statistics by the summation of the variational parameters [14,37]:

To infer , we will need to separately consider whether the lab test l is observed or missing. In particular, we can infer the topic distribution of an observed lab test at value v as:

For unobserved lab tests, we infer the joint distribution of latent topic and lab result :

where the notation indicate the exclusion of variable index i, j and the sufficient statistics are

Furthermore, we update the hyperparameters by maximizing the marginal likelihood under the variational expectations via empirical Bayes fixed-point updates [37, 38]). For example, the update for is . Other hyperparameters updates are similar. The learning algorithm therefore follows expectation-maximization: E-step infers ’s; M-step updates model parameters. To learn MixEHR from massive- scale EHR data in the future, we also developed a stochastic variational inference algorithm [39] (Supplementary Information).

5 Experiments

5.1 MIMIC-III dataset

We applied MixEHR to MIMIC-III data [1], which contain 39,000 patients each with a single admission and 7500 patients each with multiple admissions (Table 2 in Supplementary Information). We used MixEHR to jointly model 6 data categories including unstructured text in clinical notes, ICD-9, DRG, current procedural terminology (CPT), prescriptions, and lab tests, together comprising of 53,000 clinical features and 12.8 million total clinical observations. For the 564 lab tests, we used the “flag" column in the MIMIC LABEVENT data file to record the lab results, which contains empty value indicating normal results or “abnormal" or “delta". We set the results of “delta" values to “abnormal" as they represent only a small fraction of the patients. As a result, each lab test takes either normal or abnormal values. The lab tests that are not observed for each patient are the missing lab tests. Notably, although we used only two-value system to represent the

Figure 3: Predictive log likelihood of missing lab test results using four different MixEHR models.

lab results, MixEHR is able to handle arbitrary number of lab values. We trained and cross-validated our model using the 39k single-admission records and evaluated the trained model using the earlier admission of the 7500 test patients to predict mortality outcomes in their last admissions.

To evaluate model learning and monitor empirical convergence, we performed 5-fold cross-validation. For each patient in the validation fold, we used randomly selected 50% of their EHR features to infer their disease mixtures and then used the other 50% of the features to evaluate the log predictive likelihood, which is a common metric to evaluate topic models [10,15,39]:

where we inferred the variational expectations of the disease mixture and disease topics as:

Sensible models should demonstrate improved predictive likelihood on the held-out patients. We evaluated the predictive log likelihood of models with and set K=75 as it gave the highest predictive likelihood (Fig. 2).

5.2 Simulation to evaluate robustness to non-randomly missing lab tests

To evaluate the novel NMAR aspect of MixEHR, we needed to know the underlying results of the truly missing lab tests for each patient, which is obviously not available. To this end, we simulated EHR data from the presented generative model. Specifically, we ran MixEHR on the full MIMIC data until convergence with the topic number set to 75. We then used the patients’ mixtures and the trained model parameters to generate lab data and the other 5 EHR data types. In particular, multinomial latent variable assignments for each EHR variable and each patient were sampled from the given patient mixtures, which were shared across all EHR variables. The missing indicators of each lab test on each patient were simulated from a binomial distribution with the trained beta parameters specific to the latent variable assignments.

We performed 5-fold cross-validation using the simulated data. Because NMAR is only relevant to the lab data, we focused on evaluating the predictive log likelihood on the missing lab test (i.e., ). Specifically, we used the observed data to infer the mixture of the held-out patients and then used the inferred patient mixtures and the model parameters to impute the missing lab tests. Because the predictive log likelihood is the product of the normalized mixtures and normalized imputed lab results, it is independent from the model complexity and therefore an appropriate evaluation metric. As comparisons, we evaluated four different variations of the model: (1) MixEHR_nmar modeling NMAR using lab view only (labview); (2) MixEHR_nmar using all EHR (mixview); (3) & (4) MixEHR_mar assuming the lab results are missing at random (MAR) using lab (labview) and all EHR (mixview), respectively. For the MAR model, we did not

Figure 4: Top five EHR features and lab tests for select topics from a 75-topic MixEHR model.

estimate the distribution of the missing indicator and we only used the observed lab results to update the models as opposed to using both the observed and expected missing lab results.

While the training likelihoods increased in all models, only the NMAR models achieved increasing averaged predictive log likelihood on the missing lab tests for the held-out patients (Fig. 3). Moreover, integrating both lab tests and auxiliary EHR data (i.e., clinical notes, ICD-9, prescriptions, etc) provided further improvements. Therefore, our model is robust to NMAR lab results.

5.3 Multi-view disease topics reveal meaningful pathology

We examined the clinical relevance of the learned disease topics (i.e., ) by taking the top 5 EHR binary features and the top 5 lab tests from the same topics. Taking a common index k from those 3 topic matrices gives us one “disease topic" distribution over the clinical terms across categories. Here we represented the disease topics over lab data using an empirical score calculated as the abnormal lab results weighted by the observation bias normalized over all lab tests for the same topics: . Here we sought to intuitively examine how diversified and distinct the disease topics were and how they might help interpreting abnormal lab results.

For demonstration purpose, we took disease topics that exhibit the highest probabilities over a broad range of diseases including ICD-9 with prefixes {250, 296, 331, 042, 205, 415, 571}, which represent diabetes, psychoses, neurodegeneration,HIV, myeloid leukemia, acute pulmonary heart disease, chronic liver disease and cirrhosis, respectively. Notably, these ICD-9 codes are not necessarily the highest ranked features for each topic but we expected prominent enrichments of related clinical terms. Indeed, we observed salient contrast of each topic in terms of the top representative EHR features across diverse EHR categories (colored bars on the right; Fig. 4a). Notably, if we were to model the data without modeling distinct distributions over each data category using LDA, the disease topics would have been dominated by clinical notes as they contain overwhelmingly larger number of features (i.e., words) than the other categories. Moreover, we observed interesting connections between abnormal lab results and specific diseases. For instance, topic M31 is enriched for alcohol cirrhosis and blood tests in ascites typical for cirrhosis patients; pulmonary embolism that manifests as blood clotting in lung is linked with proliferation of blood cells via topics M35; under M42, HIV is related with abnormal counts of immune cells such as CD4/CD8 and lymphocytes; diabetes tends to rank high with acetone level under M48, and psychoses topic M62 enriched for opioid abuse is associated with valproic acid and lithium (known markers for psychiatry). Interestingly, topic M59 associated with neurodegenerative diseases such as Alzheimer’s disease (AD) is strongly associated with vitamin B12, which was shown to exhibit differential pattern in AD patients [40].

5.4 Mortality predictions

Predicting patients’ mortality risks in Intensive Care Unit (ICU) is vital to assessing the severity of illness or trauma and facilitating subsequent urgent care. Because we were able to distill meaningful disease topics from heterogeneous EHR data, we sought to assess the predictive power of these disease topics for mortality predictions. For each patient, we labelled them as “deceased" if their

Figure 5: Mortality prediction. a. Five-fold cross-validation on predicting mortality of patients with single-admission. b. Predicting patients’ mortality outcomes in the last admission using mixture embedding of EHR early admission. c. Topic clinical features of the most and least predictive topics.

death times were recorded in the admission data. Out of the 39000 patients with single admission, 4351 (11%) were deceased. We experimented with K=50 and 75 topics. As baseline, we also evaluated LDA implemented in R package “topicmodels" [41] on flattened EHR matrix of all 6 data types.

We first performed a 5-fold cross-validation (CV) on patients with single admission by stratifying the patients into five folds with equal proportion of alive and deceased patients. For each CV round, we trained MixEHR on four folds to learn the disease mixture topics. We then applied the trained model to infer the disease mixtures of the training folds, which were then used as predictors to train the supervised LASSO regression on mortality response variable. Finally, we used the trained LASSO to predict the mortality of the validation fold and repeated the same process for the five folds. Note: (1) mortality labels were not used in learning the disease topics; (2) the unsupervised model were not trained on the test folds. For all CV folds, we recorded the predictions and mortality labels to generate the ROC and precision-recall curves (PRC) and evaluated the area under these curves (AUC). With 75 (50) topics, we obtained 92% (90.9%) AUROC and 63% (59.7%) AUPRC whereas LDA obtained only 85% (83%) AUROC and 51% (45.2%) AUPRC (Fig. 5a; Table 1).

Moreover, we also performed evaluation on the 7500 patients with multiple admissions. We first trained MixEHR or LDA and LASSO models on the full training set of 39,000 single-admission records. We then inferred the disease mixtures of the test patients from their EHR data in the earliest admission only, predicted mortality outcomes of the patients, and compared the predictions with the death labels in their last admissions. We used only the patients whose last admission was after 30 days but within 6 months of their first admissions, which gave us 458 deceased and 6079 alive patients (7% positive labels). With 75 (50) topics, we obtained 79.8% (85%) AUROC and 21.3% (31.2%) AUPRC compared favorably to LDA (Fig. 5b; Table 1). Based on the LASSO linear coefficients, the top 3 most predictive disease topics associated with mortality are enriched for cardiac arrest, acute liver failure, and septicemia, whereas the least predictive topic is related with newborn (Fig. 5c).

Table 1: Mortality prediction

6 Conclusion and Future Work

In this paper, we present MixEHR to jointly model heterogeneous EHR data while accounting for NMAR. Our approach demonstrates robustness to NMAR lab results and the ability to distill meaningful disease topics, linking lab results to other EHR features across categories. We also achieved promising mortality predictions. We expect that our method will have broad applications in healthcare. We only demonstrated MixEHR on ICU EHR due to data availability but expect it to also work well on out-patient data. As an extension of MixEHR, we will incorporate longitudinal aspects of the EHR data to model disease progressions (e.g., [42,43]).

References

[1] Alistair E W Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. MIMICIII, a freely accessible critical care database. Scientific Data, 3:160035–9, May 2016.

[2] Dustin Charles, Meghan Gabriel, and Michael F Furukawa. Adoption of electronic health record systems among US non-federal acute care hospitals: 2008-2012. ONC data brief, 9:1– 9, 2013.

[3] J Henry, Yuriy Pylypchuk, Talisha Searcy, and Vaishali Patel. Adoption of electronic health record systems among US non-federal acute care hospitals: 2008-2015. The Office of National Coordinator for Health Information Technology, 2016.

[4] Roderick J A Little and Donald B Rubin. Statistical Analysis with Missing Data. John Wiley & Sons, August 2014.

[5] R Salakhutdinov, A Mnih, and G Hinton. Restricted Boltzmann machines for collaborative filtering. In Proceedings of the 24th international conference on Machine learning, pages 791–798, New York, New York, USA, 2007. ACM Press.

[6] Andriy Mnih and Ruslan R Salakhutdinov. Probabilistic matrix factorization. In Advances in neural information processing systems, pages 1257–1264, 2008.

[7] J M Hernandez-lobato, N Houlsby, and Z Ghahramani. Probabilistic Matrix Factorization with Non-random Missing Data. ICML, 2014.

[8] Benjamin M Marlin and Richard S Zemel. Collaborative prediction and ranking with non-random missing data. In the third ACM conference, pages 5–12, New York, New York, USA, 2009. ACM Press.

[9] Gary Fraser and Ru Yan. Collaborative Filtering and the Missing at Random Assumption. Epidemiology, 18(2):1–9, May 2016.

[10] David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. J. Mach. Learn. Res., 3:993–1022, March 2003.

[11] Jon D Mcauliffe and David M Blei. Supervised topic models. In J C Platt, D Koller, Y Singer, and S T Roweis, editors, Advances in Neural Information Processing Systems 20, pages 121– 128. Curran Associates, Inc., 2008.

[12] David M Blei. Probabilistic topic models. Commun. ACM, 55(4):77–84, April 2012.

[13] Thomas L Griffiths and Mark Steyvers. Finding scientific topics. Proceedings of the National Academy of Sciences of the United States of America, 101 Suppl 1(Supplement 1):5228–5235, April 2004.

[14] Yee W Teh, David Newman, and Max Welling. A collapsed variational bayesian inference algorithm for latent dirichlet allocation. In B Schölkopf, J C Platt, and T Hoffman, editors, Advances in Neural Information Processing Systems 19, pages 1353–1360. MIT Press, 2007.

[15] Arthur Asuncion, Max Welling, Padhraic Smyth, and Yee Whye Teh. On smoothing and inference for topic models. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI ’09, pages 27–34, Arlington, Virginia, United States, 2009. AUAI Press.

[16] Yoni Halpern, Steven Horng, and David Sontag. Clinical Tagging with Joint Probabilistic Models. arXiv. org, August 2016.

[17] Shalmali Joshi, Suriya Gunasekar, David Sontag, and Ghosh Joydeep. Identifiable phenotyping using constrained Non-Negative matrix factorization. In Machine Learning for Healthcare Conference, pages 17–41, December 2016.

[18] Rimma Pivovarov, Adler J Perotte, Edouard Grave, John Angiolillo, Chris H Wiggins, and Noémie Elhadad. Learning probabilistic phenotypes from heterogeneous EHR data. J. Biomed. Inform., 58(C):156–165, December 2015.

[19] Suriya Gunasekar, Joyce C Ho, Joydeep Ghosh, Stephanie Kreml, Abel N Kho, Joshua C Denny, Bradley A Malin, and Jimeng Sun. Phenotyping using structured collective matrix factorization of multi–source EHR data. arXiv, September 2016.

[20] Patrick Flaherty, Guri Giaever, Jochen Kumm, Michael I Jordan, and Adam P Arkin. A latent variable model for chemogenomic profiling. Bioinformatics, 21(15):3286–3293, August 2005.

[21] Narges Razavian and David Sontag. Temporal Convolutional Neural Networks for Diagnosis from Lab Tests. arXiv. org, November 2015.

[22] Y Cheng, F Wang, P Zhang, and J Hu. Risk prediction with electronic health records: A deep learning approach. 2016 SIAM International Conference, pages 432–440, 2016.

[23] Zachary C Lipton, David C Kale, Charles Elkan, and Randall Wetzel. Learning to Diagnose with LSTM Recurrent Neural Networks. arXiv. org, November 2015.

[24] Edward Choi, Mohammad Taha Bahadori, Andy Schuetz, Walter F Stewart, and Jimeng Sun. Doctor AI: Predicting Clinical Events via Recurrent Neural Networks. JMLR Workshop Conf. Proc., 56:301–318, August 2016.

[25] Phuoc Nguyen, Truyen Tran, Nilmini Wickramasinghe, and Svetha Venkatesh. Deepr: A Convolutional Net for Medical Records. arXiv. org, pages 22–30, July 2016.

[26] Riccardo Miotto, Li Li, Brian A Kidd, and Joel T Dudley. Deep Patient: An Unsupervised Representation to Predict the Future of Patients from the Electronic Health Records. Sci. Rep., 6(1):1–10, May 2016.

[27] Harini Suresh, Peter Szolovits, and Marzyeh Ghassemi. The Use of Autoencoders for Discov- ering Patient Phenotypes. arXiv. org, March 2017.

[28] Alistair E W Johnson, Mohammad M Ghassemi, Shamim Nemati, Katherine E Niehaus, David A Clifton, and Gari D Clifford. Machine learning and decision support in critical care. Proc. IEEE Inst. Electr. Electron. Eng., 104(2):444–466, February 2016.

[29] Jyotishman Pathak, Abel N Kho, and Joshua C Denny. Electronic health records-driven phenotyping: challenges, recent advances, and perspectives. J. Am. Med. Inform. Assoc., 20(e2):e206–11, December 2013.

[30] George Hripcsak and David J Albers. Next-generation phenotyping of electronic health records. J. Am. Med. Inform. Assoc., 20(1):117–121, January 2013.

[31] Shalmali Joshi, Suriya Gunasekar, David Sontag, and Joydeep Ghosh. Identifiable Phenotyping using Constrained Non-Negative Matrix Factorization. arXiv.org, August 2016.

[32] Yoni Halpern, Steven Horng, and David Sontag. Clinical Tagging with Joint Probabilistic Models. arXiv.org, August 2016.

[33] Suriya Gunasekar, Joyce C Ho, Joydeep Ghosh, Stephanie Kreml, Abel N Kho, Joshua C Denny, Bradley A Malin, and Jimeng Sun. Phenotyping using Structured Collective Matrix Factorization of Multi–source EHR Data. arXiv.org, September 2016.

[34] Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An Intro- duction to Variational Methods for Graphical Models. In Learning in Graphical Models, pages 105–161. Springer Netherlands, Dordrecht, 1998.

[35] Thomas L Griffiths and Mark Steyvers. Finding scientific topics. Proc. Natl. Acad. Sci. U. S. A., 101 Suppl 1:5228–5235, April 2004.

[36] C Bishop. Pattern recognition and machine learning. library.wisc.edu, 2006.

[37] Arthur Asuncion, Max Welling, Padhraic Smyth, and Yee Whye Teh. On smoothing and inference for topic models. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI ’09, pages 27–34, Arlington, Virginia, United States, 2009. AUAI Press.

[38] T Minka. Estimating a Dirichlet distribution. Technical Report, 2000.

[39] M D Hoffman, D M Blei, C Wang, and J W Paisley. Stochastic variational inference. Journal of Machine Learning Research, 2013.

[40] Alicia Osimani, Andrea Berger, Joshua Friedman, Bat-Sheba Porat-Katz, and Jacob M. Abar- banel. Neuropsychologyof vitamin b12 deficiency in elderly dementia patients and control subjects. Journal of Geriatric Psychiatry and Neurology, 18(1):33–38, 2005. PMID: 15681626.

[41] Bettina Grün and Kurt Hornik. topicmodels: An r package for fitting topic models. Journal of Statistical Software, Articles, 40(13):1–30, 2011.

[42] Peter Schulam and Suchi Saria. A framework for individualizing predictions of disease tra- jectories by exploiting multi-resolution structure. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 748–756. Curran Associates, Inc., 2015.

[43] Peter Schulam and Suchi Saria. Integrative analysis using coupled latent variable models for individualizing prognoses. Journal of Machine Learning Research, 17(234):1–35, 2016.

Designed for Accessibility and to further Open Science