Learning Optimal Dynamic Treatment Strategies from Temporal Monitoring Health Data
Contact: Dr. Li-wei Lehman
A fundamental challenge of treatment decision
making in a medical setting is that treatment strategies are complex and
dynamic. They involve sequential decisions at many timepoints, each based on
evolving patient history. We refer to such treatment strategies as Dynamic
Treatment Regimes (or DTRs). Standard reinforcement learning (RL) approaches to
estimating optimal DTRs from observational data, such as Q-learning or
A-learning, face challenges in that the learned DTRs have high-variance and the
model estimates can be biased and un-interpretable. We are working
along multiple threads, all aimed at understanding, mitigating and solving
these problems.
Building Interpretable Probabilistic
Models for Informed Treatment Decision Making:
In our AAAI 2022 paper “Knowledge
Distillation via Constrained Variational Inference,” we leverage deep
learning to build more powerful probabilistic models that can simultaneously
identify interpretable latent structure in medical data and accurately predict
patient outcomes. In our NeurIPS 2022 workshop paper “Treatment-RSPN: Recurrent Sum-Product
Networks for Sequential Treatment Regimes,” we
introduced a probabilistic deep generative approach, Treatment-RSPN, that
leverages Recurrent Sum Product Networks (RSPNs) for joint modelling of
treatment decision-making and patient treatment response. As part of our
framework, we develop a method for transforming conventional probabilistic
graphical models (PGMs), such as dynamic Bayesian networks (DBNs), into RSPNs,
allowing us to bootstrap our models with a structure informed by domain
knowledge and the specific task.
Counterfactual Prediction under
Dynamic and Time-Varying Treatment Regimes: We are developing sequential deep learning
approaches to estimate expected
patient trajectories and treatment outcomes under dynamic and time-varying
treatment strategies.
Please read our paper "G-Net: A
Recurrent-Network Approach to G-computation for Counterfactual Outcome
Prediction Under Dynamic Treatment Regimes" for one of the approaches
that we have developed in this framework. G-Net is based on G-computation, a
causal inference method that can be used to estimate the average effect of a
DTR on the population, or the conditional effect given observed patient
history. We are extending and applying
this methodology to make personalized counterfactual predictions of patient
outcomes under alternative fluid administration and mechanical ventilation
strategies given patient history.
Learning
to Treat COVID-19 Patients from Observational ICU Data: We are developing an AI tool based on causal
inference methods to facilitate mechanical ventilation decision making for
COVID-19 patients with acute respiratory distress syndrome (ARDS) in the ICUs.
Our approach consists of a sequential modeling framework that would allow
clinicians to explore various “what-if” scenarios to estimate both individual
and population-level effects of alternative mechanical ventilation strategies
for ARDS patients.
Dynamic
Marginal Structural Models (DynMSMs) to
Estimate Time-Varying Treatment Effects: In this line of work, we
collaborate closely with clinicians to define clinical questions pertaining to
simplified classes of treatment strategies. We do not attempt to estimate the
overall optimal DTR across all possible functions of patient history. Instead,
we estimate the optimal DTR within a restricted class (or in one case just
estimate the effects of a few separate treatment strategies). In our paper Fluid-limiting treatment strategies among sepsis
patients in the ICU: a retrospective causal analysis, we use DynMSM to estimate the effect of different fluid resuscitation strategies on sepsis
patient outcomes.
Safe
and Robust Machine Learning Models for Health:
One important aspect of responsible AI is to evaluate the safety and robustness
of ML models in the context of clinical treatment decision making to avoid
unintended harm. We evaluated the safety
and robustness of machine learning models for clinical treatment decision
making in our AMIA 2020 Annual Symposium paper "Is Deep Reinforcement Learning
Ready for Practical Applications in Healthcare? A Sensitivity Analysis of
Duel-DDQN for Hemodynamic Management in Sepsis Patients." Our work systematically explored the
sensitivity of a deep reinforcement learning (DRL) technique for sepsis
treatment, and uncovered several important areas of caution in adopting DRL in
a healthcare setting.
· Our
paper G-Net
was featured on the MIT News! Read this MIT news
article about G-Net, a new deep learning approach
developed by our team to simulate treatment outcomes under time-varying and
dynamic treatment strategies.
· Congratulations
to our team for winning the Distinguished Paper Award at the 2020
American Medical Informatics Association (AMIA)’s annual symposium! In
our AMIA 2020 Annual Symposium paper "Is
Deep Reinforcement Learning Ready for Practical Applications in Healthcare? A
Sensitivity Analysis of Duel-DDQN for Hemodynamic Management in Sepsis
Patients," we evaluated the safety and robustness of deep
reinforcement learning (DRL) models for clinical treatment decision
making. Our work systematically explored
the sensitivity of a DRL technique for sepsis treatment, and uncovered several
important areas of caution in adopting DRL in a healthcare setting.
We
are seeking talented students, interns, and collaborators to join our team. For more details,
please see here.
Our team consists of
an interdisciplinary group of researchers from
MIT and MIT-IBM Watson AI Lab, bringing
together expertise from machine learning, causal inference, physiological
modeling, and medical informatics to solve
challenging problems in dynamic treatment regimes for
clinical medicine.
MIT
Principal Investigator (PI): Dr. Li-wei
Lehman (Ph.D.)
MIT Co-PI: Professor Roger Mark
(M.D., Ph.D.)
MIT
Students: Adam Dejl, Amy Hu, Jenny Moralejo
MIT Alumni: Rui Li
(MEng Student), Dr. Jun Li (Post-doc), Fengyi Andy
Tang (Research Intern), Kechun Huang (Research
Fellow), MingYu Lu (Research Fellow), Stephanie Hu
(MEng), Yuria Utsumi
(UROP), Michelle Yin (UROP), Nicholas Baginski
(UROP), Ardavan Saeedi
(Ph.D.)
IBM Team: Dr. Zach Shahn
(Ph.D., PI), Dr. Daby Sow (Ph.D., PI), Dr. Prithwish Chakraborty (Ph.D.), Dr.
Mohamed Ghalwash (Ph.D.), Piyush Madan (MSc)
Clinical
Collaborators (BIDMC): Dr. Daniel Talmor, Dr. Nate Shapiro, Dr.
Elias Kassis, Dr. Somnath Bose
·
Treatment-RSPN: Recurrent Sum-Product
Networks for Sequential Treatment Regimes, Adam Dejl,
Harsh Deep, Jonathan Fei, Ardavan Saeedi,
Li-wei Lehman, ML4H Symposium and NeurIPS Time Series for Health Workshop, New Orleans,
2022. Poster.
· Knowledge
Distillation via Constrained Variational Inference, Ardavan Saeedi, Yuria Utsumi, Li Sun, Kayhan Batmanghelich, Li-wei H. Lehman, Proceedings of the Thirty-Sixth AAAI
Conference on Artificial Intelligence, February 2022.
·
Retaining
Privileged Information for Multi-Task Learning, Fengyi Tang, Cao Xiao, Fei Wang, Jiayu
Zhou, Li-wei H. Lehman, Proceedings of the 25th
SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), August 2019,
Anchorage, Alaska USA. Research Track.
·
Robust Low-Rank
Discovery of Data-Driven Partial Differential Equations, Jun Li, Gan Sun, Guoshuai
Zhao, Li-wei Lehman, The Thirty-Fourth AAAI
Conference on Artificial Intelligence, 2020, New York, USA.
·
A
Physiological Time Series Dynamics-Based Approach to Patient Monitoring and
Outcome Prediction, Li-wei H. Lehman, Ryan
P. Adams, Louis Mayaud, George B. Moody,
Atul Malhotra, Roger G. Mark, Shamim Nemati, IEEE Journal of Biomedical and
Health Informatics, 19(3):1068-1076, May 2015. doi:10.1109/JBHI.2014.2330827. Preprint.
·
Bayesian
nonparametric learning of switching dynamics in cohort physiological time
series: application in critical care patient monitoring, Li-wei H. Lehman, Matthew
J. Johnson, Shamim Nemati, Ryan P. Adams, Roger G. Mark, Chapter in Advanced State Space Methods for Neural and Clinical
Data, Cambridge University Press, 2015. Publisher's
Version.
·
A
Model-Based Machine Learning Approach to Probing Autonomic Regulation from
Nonstationary Vital-Sign Time Series, Li-wei H.
Lehman, Roger G. Mark, Shamim Nemati, IEEE Journal of Biomedical and Health Informatics,
Vol. 22, No. 1, January 2018. doi:10.1109/JBHI.2016.2636808. Preprint
· G-Net: a Recurrent
Network Approach to G-Computation for Counterfactual Prediction Under a Dynamic
Treatment Regime, Rui Li, Stephanie Hu, Mingyu
Lu, Yuria Utsumi, Prithwish Chakraborty, Daby M.
Sow, Piyush Madan, Mohamed Ghalwash, Zach
Shahn, Li-wei H. Lehman , Proceedings of
Machine Learning for Health, PMLR 158:280-297, 2021. An earlier ArXiv version of this work is available at arXiv:2003.10551.
·
Fluid-limiting treatment strategies among sepsis
patients in the ICU: a retrospective causal analysis, Zach Shahn, Nathan I.
Shapiro, Patrick D. Tyler, Daniel Talmor and Li-wei
H. Lehman, Journal of Critical Care, 24:62, 2020.
·
Titration
of Ventilator Settings to Target Driving Pressure and Mechanical Power,
Elias Baedorf Kassis,
Stephanie Hu, MingYu Lu, Alistair Johnson, Somnath
Bose, Maximilian S Schaefer, Daniel Talmor, Li-wei H
Lehman, Zach Shahn, Respiratory Care, July 2022.
·
Delaying
initiation of diuretics in critically ill patients with recent vasopressor use
and high positive fluid balance, Zach
Shahn, Li-wei H Lehman, Roger G. Mark, Daniel
Talmor, Somnath Bose, British Journal of Anaesthesia
(BJA), Vol. 127, Issue 4, 2021, pages 569-576.
·
Should
Diuretic Initiation be Delayed in ICU Patients with Recent Vasopressor Use? A
Causal Analysis, Somnath Bose, Li-wei H. Lehman, Kechun Huang, Daniel Talmor, Zach Shahn, Critical Care
Medicine, Volume 48, Issue 1, p 733, January 2020.
·
Efficient
estimation of optimal regimes under a no direct effect assumption, Lin Liu,
Zach Shahn, James M. Robins, Andrea Rotnitzky,
Journal of the American Statistical Association (JASA), 2020. ArXiv version https://arxiv.org/abs/1908.10448.
·
Estimating Optimal Dynamic Treatment
Regimes Under Resource Constraints Using Dynamic Marginal Structural Models,
Ellen Caniglia, Eleanor Murray, Miguel Hernan, and Zach Shahn, Statistics in
Medicine, in revision. https://arxiv.org/abs/1903.06488
·
Latent Class Mixture
Models of Treatment Effect Heterogeneity, Zach Shahn and David Madigan,
Bayesian Analysis, 12(3), pp831-854, 2017.
·
Is Deep Reinforcement Learning Ready for
Practical Applications in Healthcare? A Sensitivity Analysis of Duel-DDQN for
Sepsis Treatment," MingYu Lu, Zachary Shahn,
Daby Sow, Finale Doshi-Velez, Li-wei
H. Lehman, arXiv:2005.04301, AMIA Annual Symposium 2020.
·
Evaluating Reinforcement Learning
Algorithms in Observational Health Setting, Omer Gottesman, Fredrik
Johansson, Joshua Meier, Jack Dent, Donghun Lee, Srivatsan Srinivasan, Linying
Zhang, Yi Ding, David Wihl, Xuefeng
Peng, Jiayu Yao, Isaac Lage, Christopher Mosch, Li-wei H. Lehman, Matthieu
Komorowski, Aldo Faisal, Leo Anthony Celi, David Sontag, Finale Doshi-Velez,
https://arxiv.org/abs/1805.12298v1, 2018.
·
Improving
Sepsis Treatment Strategies by Combining Deep and Kernel-Based Reinforcement
Learning, Xuefeng Peng, Yi Ding, David Wihl, Omer Gottesman, Matthieu Komorowski,
Li-wei H. Lehman, Andrew Ross, Aldo Faisal, Finale
Doshi-Velez, Proceedings of the AMIA Annual Symposium, 2018.