Home Mathematics Bayesian inference for optimal dynamic treatment regimes in practice
Article
Licensed
Unlicensed Requires Authentication

Bayesian inference for optimal dynamic treatment regimes in practice

  • Daniel Rodriguez Duque EMAIL logo , Erica E. M. Moodie and David A. Stephens
Published/Copyright: May 17, 2023
The International Journal of Biostatistics
From the journal The International Journal of Biostatistics Volume 19 Issue 2

Abstract

In this work, we examine recently developed methods for Bayesian inference of optimal dynamic treatment regimes (DTRs). DTRs are a set of treatment decision rules aimed at tailoring patient care to patient-specific characteristics, thereby falling within the realm of precision medicine. In this field, researchers seek to tailor therapy with the intention of improving health outcomes; therefore, they are most interested in identifying optimal DTRs. Recent work has developed Bayesian methods for identifying optimal DTRs in a family indexed by ψ via Bayesian dynamic marginal structural models (MSMs) (Rodriguez Duque D, Stephens DA, Moodie EEM, Klein MB. Semiparametric Bayesian inference for dynamic treatment regimes via dynamic regime marginal structural models. Biostatistics; 2022. (In Press)); we review the proposed estimation procedure and illustrate its use via the new BayesDTR R package. Although methods in Rodriguez Duque D, Stephens DA, Moodie EEM, Klein MB. (Semiparametric Bayesian inference for dynamic treatment regimes via dynamic regime marginal structural models. Biostatistics; 2022. (In Press)) can estimate optimal DTRs well, they may lead to biased estimators when the model for the expected outcome if everyone in a population were to follow a given treatment strategy, known as a value function, is misspecified or when a grid search for the optimum is employed. We describe recent work that uses a Gaussian process ( G P ) prior on the value function as a means to robustly identify optimal DTRs (Rodriguez Duque D, Stephens DA, Moodie EEM. Estimation of optimal dynamic treatment regimes using Gaussian processes; 2022. Available from: https://doi.org/10.48550/arXiv.2105.12259). We demonstrate how a G P approach may be implemented with the BayesDTR package and contrast it with other value-search approaches to identifying optimal DTRs. We use data from an HIV therapeutic trial in order to illustrate a standard analysis with these methods, using both the original observed trial data and an additional simulated component to showcase a longitudinal (two-stage DTR) analysis.

Keywords: Bayesian inference; dynamic treatment regimes; precision medicine; R package.
Corresponding author: Daniel Rodriguez Duque, Department of Epidemiology & Biostatistics, McGill University, Montréal, QC, Canada, E-mail:

  1. Author contribution: All the authors have accepted responsibility for the entire content of this submitted manuscript and approved submission.

  2. Research funding: DRD is supported by a doctoral fellowship from the Fonds de recherche du Québec (FRQ), Nature et technologie. EEMM and DAS acknowledge support from Discovery Grants from the Natural Sciences and Engineering Research Council of Canada (NSERC). EEMM is a Canada Research Chair (Tier 1) in Statistical Methods for Precision Medicine and acknowledges the support of a chercheur de mérite career award from the FRQ, Santé.

  3. Conflict of interest statement: The authors declare no conflicts of interest regarding this article.

References

1. Murphy, SA, van der Laan, MJ, Robins, JM. Marginal mean models for dynamic regimes. J Am Stat Assoc 2001;96:1410–23. https://doi.org/10.1198/016214501753382327.Search in Google Scholar PubMed PubMed Central

2. Robins, JM. A new approach to causal inference in mortality studies with a sustained exposure period—application to control of the healthy worker survivor effect. Math Model 1986;7:1393–512. https://doi.org/10.1016/0270-0255(86)90088-6.Search in Google Scholar

3. Robins, JM. Analytic methods for estimating HIV-treatment and cofactor effects. In: Ostrow, DG, Kessler, RC, editors. Methodological issues in AIDS behavioral research. Boston: Springer; 1993:213–87 pp.10.1007/0-306-47137-X_12Search in Google Scholar

4. Zhao, Y, Kosorok, MR, Zeng, D. Reinforcement learning design for cancer clinical trials. Stat Med 2009;28:3294–315. https://doi.org/10.1002/sim.3720.Search in Google Scholar PubMed PubMed Central

5. Orellana, L, Rotnitzky, A, Robins, JM. Dynamic regime marginal structural mean models for estimation of optimal dynamic treatment regimes, part I: main content. Int J Biostat 2010;6:1–47. https://doi.org/10.2202/1557-4679.1200.Search in Google Scholar

6. Zhao, Y, Zeng, D, Rush, J, Kosorok, MR. Estimating individualized treatment rules using outcome weighted learning. J Am Stat Assoc 2012;107:1106–18. https://doi.org/10.1080/01621459.2012.695674.Search in Google Scholar PubMed PubMed Central

7. Saarela, O, Arjas, E, Stephens, DA, Moodie, EEM. Predictive Bayesian inference and dynamic treatment regimes. Biom J 2015;57:941–58. https://doi.org/10.1002/bimj.201400153.Search in Google Scholar PubMed

8. Murray, TA, Yuan, Y, Thall, PF. A Bayesian machine learning approach for optimizing dynamic treatment regimes. J Am Stat Assoc 2018;113:1255–67. https://doi.org/10.1080/01621459.2017.1340887.Search in Google Scholar PubMed PubMed Central

9. Arjas, E, Saarela, O. Optimal dynamic regimes: presenting a case for predictive inference. Int J Biostat 2010;6:1–91. https://doi.org/10.2202/1557-4679.1204.Search in Google Scholar PubMed PubMed Central

10. Xu, Y, Müller, P, Wahed, AS, Thall, PF. Bayesian nonparametric estimation for dynamic treatment regimes with sequential transition times. J Am Stat Assoc 2016;111:921–50. https://doi.org/10.1080/01621459.2015.1086353.Search in Google Scholar PubMed PubMed Central

11. Hua, W, Mei, H, Zohar, S, Giral, M, Xu, Y. Personalized dynamic treatment regimes in continuous time: a Bayesian approach for optimizing clinical decisions with timing. Bayesian Anal 2022;17:849–78. https://doi.org/10.1214/21-ba1276.Search in Google Scholar

12. Rodriguez Duque, D, Stephens, DA, Moodie, EEM, Klein, MB. Semiparametric Bayesian inference for dynamic treatment regimes via dynamic regime marginal structural models. Biostatistics; 2022. (In Press).10.1093/biostatistics/kxac007Search in Google Scholar PubMed

13. Rodriguez Duque, D, Stephens, DA, Moodie, EEM. Estimation of optimal dynamic treatment regimes using Gaussian processes; 2022. Available from: https://doi.org/10.48550/arXiv.2105.12259.Search in Google Scholar

14. Saarela, O, Stephens, DA, Moodie, EEM, Klein, MB. On Bayesian estimation of marginal structural models. Biometrics 2015;71:279–88. https://doi.org/10.1111/biom.12269.Search in Google Scholar PubMed

15. Wallace, MP, Moodie, EEM, Stephens, DA, Simoneau, G, Schulz, J. DTRreg: DTR estimation and inference via g-estimation, dynamic WOLS, Q-learning, and dynamic weighted survival modeling (DWSurv); 2020. R package version 1.7.Search in Google Scholar

16. Chen, Y, Liu, Y, Zeng, D, Wang, Y. DTRlearn2: statistical learning methods for optimizing dynamic treatment regimes; 2020. R package version 1.1.Search in Google Scholar

17. Holloway, ST, Laber, EB, Linn, KA, Zhang, B, Davidian, M, Tsiatis, AA. DynTxRegime: methods for estimating optimal dynamic treatment regimes; 2020. R package version 4.9.Search in Google Scholar

18. Artman, W. SMARTbayesR: Bayesian set of best dynamic treatment regimes and sample size in SMARTs for Binary Outcomes; 2021. R package version 2.0.0.10.32614/CRAN.package.SMARTbayesRSearch in Google Scholar

19. Roustant, O, Ginsbourger, D, Deville, Y. Dicekriging, diceoptim: two R packages for the analysis of computer experiments by kriging-based metamodelling and optimization. J Stat Software 2012;51:1–55.10.18637/jss.v051.i01Search in Google Scholar

20. Kundu, MG. LongCART: recursive partitioning for longitudinal data and right censored data using baseline covariates; 2021. R package version 3.1.Search in Google Scholar

21. Walker, SG. Bayesian nonparametric methods: motivation and ideas. In: Hjort, NL, Holmes, C, Müller, P, Walker, SG, editors. Bayesian nonparametrics. New York: Cambridge University Press; 2010. Chapter 1.10.1017/CBO9780511802478.002Search in Google Scholar

22. Robins, JM, Hernan, MA, Brumback, B. Marginal structural models and causal inference in epidemiology. Epidemiology 2000;11:550–60. https://doi.org/10.1097/00001648-200009000-00011.Search in Google Scholar PubMed

23. Ghosal, S, van der Vaart, A. Fundamentals of nonparametric Bayesian inference, Volume 44. Cambridge, United Kingdom: Cambridge University Press; 2017.10.1017/9781139029834Search in Google Scholar

24. Rubin, DB. The Bayesian bootstrap. Ann Stat 1981;9:130–4. https://doi.org/10.1214/aos/1176345338.Search in Google Scholar

25. Stephens, DA, Nobre, WS, Moodie, EEM, Schmidt, AM. Causal inference under mis-specification: adjustment based on the propensity score; 2022. Available from: https://doi.org/10.48550/arXiv.2201.12831.Search in Google Scholar

26. Graham, DJ, McCoy, EJ, Stephens, DA. Approximate Bayesian inference for doubly robust estimation. Bayesian Anal 2016;11:47–69. https://doi.org/10.1214/14-ba928.Search in Google Scholar

27. Austin, PC, Stuart, EA. Moving towards best practice when using inverse probability of treatment weighting (iptw) using the propensity score to estimate causal treatment effects in observational studies. Stat Med 2015;34:3661–79. https://doi.org/10.1002/sim.6607.Search in Google Scholar PubMed PubMed Central

28. Myers, JA, Rassen, JA, Gagne, JJ, Huybrechts, KF, Schneeweiss, S, Rothman, KJ, et al.. Effects of adjusting for instrumental variables on bias and precision of effect estimates. Am J Epidemiol 2011;174:1213–22. https://doi.org/10.1093/aje/kwr364.Search in Google Scholar PubMed PubMed Central

29. Cain, LE, Robins, JM, Lanoy, E, Logan, R, Costagliola, D, Hernán, MA. When to start treatment? A systematic approach to the comparison of dynamic regimes using observational data. Int J Biostat 2010;6:1–24. https://doi.org/10.2202/1557-4679.1212.Search in Google Scholar PubMed PubMed Central

30. Saarela, O, Belzile, LR, Stephens, DA. A Bayesian view of doubly robust causal inference. Biometrika 2016;103:667–81. https://doi.org/10.1093/biomet/asw025.Search in Google Scholar

31. Tsiatis, AA, Davidian, M, Holloway, ST, Laber, EB. Dynamic treatment regimes: statistical methods for precision medicine. New York: Chapman and Hall/CRC; 2019.10.1201/9780429192692Search in Google Scholar

32. Huang, D, Allen, TT, Notz, WI, Zeng, N. Global optimization of stochastic black-box systems via sequential kriging meta-models. J Global Optim 2006;34:441–66. https://doi.org/10.1007/s10898-005-2454-3.Search in Google Scholar

33. Santner, TJ, Williams, BJ, Notz, W, Williams, BJ. The design and analysis of computer experiments, 2nd ed. New York: Springer; 2018.10.1007/978-1-4939-8847-1Search in Google Scholar

34. Guan, Q, Reich, BJ, Laber, EB, Bandyopadhyay, D. Bayesian nonparametric policy search with application to periodontal recall intervals. J Am Stat Assoc 2020;115:1066–78. https://doi.org/10.1080/01621459.2019.1660169.Search in Google Scholar PubMed PubMed Central

35. Forrester, AI, Keane, AJ, Bressloff, NW. Design and analysis of “noisy” computer experiments. AIAA J 2006;44:2331–9. https://doi.org/10.2514/1.20068.Search in Google Scholar

36. Freeman, NL, Browder, SE, McGinigle, KL, Kosorok, MR. Dynamic treatment regime characterization via value function surrogate with an application to partial compliance. arXiv preprint arXiv:2212.00650 2022.Search in Google Scholar

37. O’Hagan, A, Kennedy, MC, Oakley, JE. Uncertainty analysis and other inference tools for complex computer codes. In: Bayesian statistics 6: proceedings of the sixth valencia international meeting. Oxford University Press; 1999:503–24 pp.10.1093/oso/9780198504856.003.0022Search in Google Scholar

38. Park, J-S, Baek, J. Efficient computation of maximum likelihood estimators in a spatial linear model with power exponential covariogram. Comput Geosci 2001;27:1–7. https://doi.org/10.1016/s0098-3004(00)00016-9.Search in Google Scholar

39. Williams, CK, Rasmussen, CE. Gaussian processes for machine learning, Volume 2. Cambridge, MA: MIT Press; 2006.10.7551/mitpress/3206.001.0001Search in Google Scholar

40. Lizotte, DJ. Practical Bayesian optimization [Ph.D. thesis]. Edmonton, AB, Canada: University of Alberta; 2008.Search in Google Scholar

41. Picheny, V, Wagner, T, Ginsbourger, D. A benchmark of kriging-based infill criteria for noisy optimization. Struct Multidiscip Optim 2013;48:607–26. https://doi.org/10.1007/s00158-013-0919-4.Search in Google Scholar

42. Jones, DR, Schonlau, M, Welch, WJ. Efficient global optimization of expensive black-box functions. J Global Optim 1998;13:455–92. https://doi.org/10.1023/a:1008306431147.10.1023/A:1008306431147Search in Google Scholar

43. Frazier, PI, Wang, J. Bayesian optimization for materials design. In: Lookman, T, Alexander, FJ, Rajan, K, editors. Information science for materials discovery and design. New York: Springer; 2016:45–75 pp.10.1007/978-3-319-23871-5_3Search in Google Scholar

44. Locatelli, M. Bayesian algorithms for one-dimensional global optimization. J Global Optim 1997;10:57–76. https://doi.org/10.1023/a:1008294716304.10.1023/A:1008294716304Search in Google Scholar

45. MebaneJr.WR, Sekhon, JS. Genetic optimization using derivatives: the rgenoud package for R. J Stat Software 2011;42:1–26.10.18637/jss.v042.i11Search in Google Scholar

46. Hernán, MA, Robins, JM. Causal inference: what if. Boca Raton: Chapman & Hall/CRC; 2020.Search in Google Scholar

47. Xiao, Y, Abrahamowicz, M, Moodie, EEM. Accuracy of conventional and marginal structural Cox model estimators: a simulation study. Int J Biostat 2010;6:1–28. https://doi.org/10.2202/1557-4679.1208.Search in Google Scholar PubMed

48. Hammer, SM, Katzenstein, DA, Hughes, MD, Gundacker, H, Schooley, RT, Haubrich, RH, et al.. A trial comparing nucleoside monotherapy with combination therapy in HIV-infected adults with CD4 cell counts from 200 to 500 per cubic millimeter. N Engl J Med 1996;335:1081–90. https://doi.org/10.1056/nejm199610103351501.Search in Google Scholar PubMed

49. Henmi, M, Eguchi, S. A paradox concerning nuisance parameters and projected estimating functions. Biometrika 2004;91:929–41. https://doi.org/10.1093/biomet/91.4.929.Search in Google Scholar

50. Shortreed, SM, Ertefaie, A. Outcome-adaptive lasso: variable selection for causal inference. Biometrics 2017;73:1111–22. https://doi.org/10.1111/biom.12679.Search in Google Scholar PubMed PubMed Central

Supplementary Material

This article contains supplementary material (https://doi.org/10.1515/ijb-2022-0073).

Received: 2022-06-20
Accepted: 2023-03-21
Published Online: 2023-05-17

© 2023 Walter de Gruyter GmbH, Berlin/Boston

Articles in the same Issue

  1. Frontmatter
  2. Part-1: SMAC 2021 Webconference
  3. Statistics, philosophy, and health: the SMAC 2021 webconference
  4. Part-2: Regular Articles
  5. “Show me the DAG!”
  6. Causal inference for oncology: past developments and current challenges
  7. The EBM+ movement
  8. Bayesianism from a philosophical perspective and its application to medicine
  9. Bayesian inference for optimal dynamic treatment regimes in practice
  10. Agent-based modeling in medical research, virtual baseline generator and change in patients’ profile issue
  11. Agent based modeling in health care economics: examples in the field of thyroid cancer
  12. A copula-based set-variant association test for bivariate continuous, binary or mixed phenotypes
  13. Detection of atypical response trajectories in biomedical longitudinal databases
  14. Potential application of elastic nets for shared polygenicity detection with adapted threshold selection
  15. Error analysis of the PacBio sequencing CCS reads
  16. A SIMEX approach for meta-analysis of diagnostic accuracy studies with attention to ROC curves
  17. Statistical modelling of COVID-19 and drug data via an INAR(1) process with a recent thinning operator and cosine Poisson innovations
  18. The balanced discrete triplet Lindley model and its INAR(1) extension: properties and COVID-19 applications
https://doi.org/10.1515/ijb-2022-0073
Keywords for this article
Bayesian inference; dynamic treatment regimes; precision medicine; R package.

Articles in the same Issue

  1. Frontmatter
  2. Part-1: SMAC 2021 Webconference
  3. Statistics, philosophy, and health: the SMAC 2021 webconference
  4. Part-2: Regular Articles
  5. “Show me the DAG!”
  6. Causal inference for oncology: past developments and current challenges
  7. The EBM+ movement
  8. Bayesianism from a philosophical perspective and its application to medicine
  9. Bayesian inference for optimal dynamic treatment regimes in practice
  10. Agent-based modeling in medical research, virtual baseline generator and change in patients’ profile issue
  11. Agent based modeling in health care economics: examples in the field of thyroid cancer
  12. A copula-based set-variant association test for bivariate continuous, binary or mixed phenotypes
  13. Detection of atypical response trajectories in biomedical longitudinal databases
  14. Potential application of elastic nets for shared polygenicity detection with adapted threshold selection
  15. Error analysis of the PacBio sequencing CCS reads
  16. A SIMEX approach for meta-analysis of diagnostic accuracy studies with attention to ROC curves
  17. Statistical modelling of COVID-19 and drug data via an INAR(1) process with a recent thinning operator and cosine Poisson innovations
  18. The balanced discrete triplet Lindley model and its INAR(1) extension: properties and COVID-19 applications
Sign up now to receive a 20% welcome discount
Institutional Access
How does access work?
Winner of the OpenAthens UX Award 2026
Winner of the OpenAthens UX Award 2026
Have an idea on how to improve our website?
Please write us.
© 2026 De Gruyter Brill
Downloaded on 10.3.2026 from https://www.degruyterbrill.com/document/doi/10.1515/ijb-2022-0073/html
Scroll to top button