Bayesian learners in gradient boosting for linear mixed models
-
Boyao Zhang
,
Colin Griesbach
Abstract
Selection of relevant fixed and random effects without prior choices made from possibly insufficient theory is important in mixed models. Inference with current boosting techniques suffers from biased estimates of random effects and the inflexibility of random effects selection. This paper proposes a new inference method “BayesBoost” that integrates a Bayesian learner into gradient boosting with simultaneous estimation and selection of fixed and random effects in linear mixed models. The method introduces a novel selection strategy for random effects, which allows for computationally fast selection of random slopes even in high-dimensional data structures. Additionally, the new method not only overcomes the shortcomings of Bayesian inference in giving precise and unambiguous guidelines for the selection of covariates by benefiting from boosting techniques, but also provides Bayesian ways to construct estimators for the precision of parameters such as variance components or credible intervals, which are not available in conventional boosting frameworks. The effectiveness of the new approach can be observed via simulation and in a real-world application.
Funding source: Volkswagen Foundation
-
Author contributions: All the authors have accepted responsibility for the entire content of this submitted manuscript and approved submission.
-
Research funding: This work was supported by the Freigeist-Fellowships of Volkswagen Stiftung, project “Bayesian Boosting – A new approach to data science, unifying two statistical philosophies”. Boyao Zhang performed the present work in partial fulfilment of the requirements for obtaining the degree “Dr. rer. pol.” at the Georg-August-Universität Göttingen.
-
Conflict of interest statement: The authors declare no conflicts of interest regarding this article.
References
1. Laird, NM, Ware, JH. Random-effects models for longitudinal data. Biometrics 1982;38:963–74. https://doi.org/10.2307/2529876.Search in Google Scholar
2. Bates, JCPDM, Pinheiro, JC, Pinheiro, J, Bates, D. Mixed-effects models in S and S-PLUS. New York: Springer; 2000. Available from: https://books.google.de/books?id=N3WeyHFbHLQC.10.1007/978-1-4419-0318-1Search in Google Scholar
3. Gumedze, F, Dunne, T. Parameter estimation and inference in the linear mixed model. Linear Algebra Appl 2011;435:1920–44. https://doi.org/10.1016/j.laa.2011.04.015.Search in Google Scholar
4. Heagerty, PJ, Kurland, BF. Misspecified maximum likelihood estimates and generalised linear mixed models. Biometrika 2001;88:973–85. https://doi.org/10.1093/biomet/88.4.973.Search in Google Scholar
5. Litière, S, Alonso, A, Molenberghs, G. The impact of a misspecified random-effects distribution on the estimation and the performance of inferential procedures in generalized linear mixed models. Stat Med 2008;27:3125–44. https://doi.org/10.1002/sim.3157.Search in Google Scholar PubMed
6. Breslow, NE, Clayton, DG. Approximate inference in generalized linear mixed models. J Am Stat Assoc 1993;88:9–25. https://doi.org/10.2307/2290687.Search in Google Scholar
7. Breslow, NE, Lin, X. Bias correction in generalised linear mixed models with a single component of dispersion. Biometrika 1995;82:81–91. https://doi.org/10.1093/biomet/82.1.81.Search in Google Scholar
8. Lin, X, Zhang, D. Inference in generalized additive mixed models by using smoothing splines. J R Statist Soc B 1999;61:381–400. https://doi.org/10.1111/1467-9868.00183.Search in Google Scholar
9. Fahrmeir, L, Lang, S. Bayesian inference for generalized additive mixed models based on Markov random field priors. Appl Statist 2001;50:201–20. https://doi.org/10.1111/1467-9876.00229.Search in Google Scholar
10. Zhao, Y, Staudenmayer, J, Coull, BA, Wand, MP. General design Bayesian generalized linear mixed models. Stat Sci 2006;21:35–51. https://doi.org/10.1214/088342306000000015.Search in Google Scholar
11. Fong, Y, Rue, H, Wakefield, J. Bayesian inference for generalized linear mixed models. Biostatistics 2010;11:397–412. https://doi.org/10.1093/biostatistics/kxp053.Search in Google Scholar PubMed PubMed Central
12. Schelldorfer, J, Bühlmann, P, de Geer, S. Estimation for high-dimensional linear mixed-effects models using L1-penalization. Scand J Stat 2011;38:197–214. https://doi.org/10.1111/j.1467-9469.2011.00740.x.Search in Google Scholar
13. Groll, A, Tutz, G. Variable selection for generalized linear mixed models by L1-penalized estimation. Stat Comput 2014;24:137–54. https://doi.org/10.1007/s11222-012-9359-z.Search in Google Scholar
14. Tutz, G, Groll, A. Generalized linear mixed models based on boosting. In: Statistical modelling and regression structures. Heidelberg: Springer; 2010:197–215 pp.10.1007/978-3-7908-2413-1_11Search in Google Scholar
15. Tutz, G, Binder, H. Generalized additive modeling with implicit variable selection by likelihood-based boosting. Biometrics 2006;62:961–71. https://doi.org/10.1111/j.1541-0420.2006.00578.x.Search in Google Scholar PubMed
16. Friedman, JH. Greedy function approximation: a gradient boosting machine. Ann Stat 2001;29:1189–232.10.1214/aos/1013203451Search in Google Scholar
17. Bühlmann, P, Yu, B. Boosting with the L2 loss: regression and classification. J Am Stat Assoc 2003;98:324–39. https://doi.org/10.1198/016214503000125.Search in Google Scholar
18. Hothorn, T, Bühlmann, P, Kneib, T, Schmid, M, Hofner, B. Model-based boosting 2.0. J Mach Learn Res 2010;11:2109–13.Search in Google Scholar
19. Hofner, B, Mayr, A, gamboostLSS, SM. An R package for model building and variable selection in the GAMLSS framework. J Stat Softw 2016;74:1–31.10.18637/jss.v074.i01Search in Google Scholar
20. Griesbach, C, Groll, A, Bergherr, E. Addressing cluster-constant covariates in mixed effects models via likelihood-based boosting techniques. PLoS One 2021;16:e0254178. https://doi.org/10.1371/journal.pone.0254178.Search in Google Scholar PubMed PubMed Central
21. Griesbach, C, Säfken, B, Waldmann, E. Gradient boosting for linear mixed models. Int J Biostat 2021;17:317–29. https://doi.org/10.1515/ijb-2020-0136.Search in Google Scholar PubMed
22. Hepp, T, Schmid, M, Mayr, A. Significance tests for boosted location and scale models with linear base-learners. Int J Biostat 2019;15:20180110. https://doi.org/10.1515/ijb-2018-0110.Search in Google Scholar PubMed
23. Mayr, A, Schmid, M, Pfahlberg, A, Uter, W, Gefeller, O. A permutation test to analyse systematic bias and random measurement errors of medical devices via boosting location and scale models. Stat Methods Med Res 2017;26:1443–60. https://doi.org/10.1177/0962280215581855.Search in Google Scholar PubMed
24. Akaike, H. Information theory and an extension of the maximum likelihood principle. In: Selected papers of Hirotugu Akaike. New York: Springer; 1973:199–213 pp.10.1007/978-1-4612-1694-0_15Search in Google Scholar
25. Vaida, F, Blanchard, S. Conditional Akaike information for mixed-effects models. Biometrika 2005;92:351–70. https://doi.org/10.1093/biomet/92.2.351.Search in Google Scholar
26. Liang, H, Wu, H, Zou, G. A note on conditional AIC for linear mixed-Effects models. Biometrika 2008;95:773–8. https://doi.org/10.1093/biomet/asn023.Search in Google Scholar PubMed PubMed Central
27. Greven, S, Kneib, T. On the behaviour of marginal and conditional AIC in linear mixed models. Biometrika 2010;97:773–89. https://doi.org/10.1093/biomet/asq042.Search in Google Scholar
28. Thomas, J, Hepp, T, Mayr, A, Bischl, B. Probing for sparse and fast variable selection with model-based boosting. In: Computational and mathematical methods in medicine; 2017. Available from: http://nbn-resolving.de/urn/resolver.pl? urn=nbn:de:bvb:19-epub-55643-1.10.1155/2017/1421409Search in Google Scholar PubMed PubMed Central
29. Fahrmeir, L, Kneib, T, Lang, S, Marx, B. Regression: models, methods and applications. Berlin: Springer-Verlag; 2013.10.1007/978-3-642-34333-9Search in Google Scholar
30. Säfken, B, Rügamer, D, Kneib, T, Greven, S. Conditional model selection in mixed-effects models with cAIC4. J Stat Softw 2021;99:1–30.10.18637/jss.v099.i08Search in Google Scholar
31. Higham, NJ. Computing the nearest correlation matrix—a problem from finance. IMA J Numer Anal 2002;22:329–43. https://doi.org/10.1093/imanum/22.3.329.Search in Google Scholar
32. Allen, DM. The relationship between variable selection and data agumentation and a method for prediction. Technometrics 1974;16:125–7. https://doi.org/10.1080/00401706.1974.10489157.Search in Google Scholar
33. Stone, M. Cross-validatory choice and assessment of statistical predictions. J R Statist Soc B 1974;36:111–33. https://doi.org/10.1111/j.2517-6161.1974.tb00994.x.Search in Google Scholar
34. Stone, M. An asymptotic equivalence of choice of model by cross-validation and Akaike’s criterion. J R Statist Soc B 1977;39:44–7. https://doi.org/10.1111/j.2517-6161.1977.tb01603.x.Search in Google Scholar
35. Belitz, C, Brezger, A, Klein, N, Kneib, T, Lang, S, Umlauf, N. BayesX – Software for Bayesian inference in structured additive regression models. Version 3.0.2; 2015. Available from: http://www.bayesx.org.Search in Google Scholar
36. Bates, D, Mächler, M, Bolker, B, Walker, S. Fitting linear mixed-Effects models using lme4. J Stat Software 2015;67:1–48. https://doi.org/10.18637/jss.v067.i01.Search in Google Scholar
37. Meinshausen, N, Meier, L, Bühlmann, P. P-values for high-dimensional regression. J Am Stat Assoc 2009;104:1671–81. https://doi.org/10.1198/jasa.2009.tm08647.Search in Google Scholar
38. Lin, L, Drton, M, Shojaie, A. Statistical significance in high-dimensional linear mixed models. In: Proceedings of the 2020 ACM-IMS on foundations of data science conference; 2020:171–81 pp.10.1145/3412815.3416883Search in Google Scholar PubMed PubMed Central
39. Javanmard, A, Montanari, A. Confidence intervals and hypothesis testing for high-dimensional regression. J Mach Learn Res 2014;15:2869–909.Search in Google Scholar
40. Bühlmann, P, Kalisch, M, Meier, L. High-dimensional statistics with a view toward applications in biology. Annu. Rev. Stat. Appl. 2014;1:255–78. https://doi.org/10.1146/annurev-statistics-022513-115545.Search in Google Scholar
Supplementary Material
The online version of this article offers supplementary material (https://doi.org/10.1515/ijb-2022-0029).
© 2022 Walter de Gruyter GmbH, Berlin/Boston
Articles in the same Issue
- Frontmatter
- Research Articles
- Survival analysis using deep learning with medical imaging
- Using a population-based Kalman estimator to model the COVID-19 epidemic in France: estimating associations between disease transmission and non-pharmaceutical interventions
- Approximate reciprocal relationship between two cause-specific hazard ratios in COVID-19 data with mutually exclusive events
- Sensitivity of estimands in clinical trials with imperfect compliance
- Highly robust causal semiparametric U-statistic with applications in biomedical studies
- Hierarchical Bayesian bootstrap for heterogeneous treatment effect estimation
- Penalized logistic regression with prior information for microarray gene expression classification
- Bayesian learners in gradient boosting for linear mixed models
- Unequal allocation of sample/event sizes with considerations of sampling cost for testing equality, non-inferiority/superiority, and equivalence of two Poisson rates
- HiPerMAb: a tool for judging the potential of small sample size biomarker pilot studies
- Heterogeneity in meta-analysis: a comprehensive overview
- On stochastic dynamic modeling of incidence data
- Power of testing for exposure effects under incomplete mediation
- Exact correction factor for estimating the OR in the presence of sparse data with a zero cell in 2 × 2 tables
- Right-censored partially linear regression model with error in variables: application with carotid endarterectomy dataset
- Assessing HIV-infected patient retention in a program of differentiated care in sub-Saharan Africa: a G-estimation approach
- Prediction-based variable selection for component-wise gradient boosting
Articles in the same Issue
- Frontmatter
- Research Articles
- Survival analysis using deep learning with medical imaging
- Using a population-based Kalman estimator to model the COVID-19 epidemic in France: estimating associations between disease transmission and non-pharmaceutical interventions
- Approximate reciprocal relationship between two cause-specific hazard ratios in COVID-19 data with mutually exclusive events
- Sensitivity of estimands in clinical trials with imperfect compliance
- Highly robust causal semiparametric U-statistic with applications in biomedical studies
- Hierarchical Bayesian bootstrap for heterogeneous treatment effect estimation
- Penalized logistic regression with prior information for microarray gene expression classification
- Bayesian learners in gradient boosting for linear mixed models
- Unequal allocation of sample/event sizes with considerations of sampling cost for testing equality, non-inferiority/superiority, and equivalence of two Poisson rates
- HiPerMAb: a tool for judging the potential of small sample size biomarker pilot studies
- Heterogeneity in meta-analysis: a comprehensive overview
- On stochastic dynamic modeling of incidence data
- Power of testing for exposure effects under incomplete mediation
- Exact correction factor for estimating the OR in the presence of sparse data with a zero cell in 2 × 2 tables
- Right-censored partially linear regression model with error in variables: application with carotid endarterectomy dataset
- Assessing HIV-infected patient retention in a program of differentiated care in sub-Saharan Africa: a G-estimation approach
- Prediction-based variable selection for component-wise gradient boosting
