Home Mathematics Heterogeneity in meta-analysis: a comprehensive overview
Article
Licensed
Unlicensed Requires Authentication

Heterogeneity in meta-analysis: a comprehensive overview

  • Dimitris Stogiannis , Fotios Siannis and Emmanouil Androulakis ORCID logo EMAIL logo
Published/Copyright: March 27, 2023
The International Journal of Biostatistics
From the journal The International Journal of Biostatistics Volume 20 Issue 1

Abstract

In recent years, meta-analysis has evolved to a critically important field of Statistics, and has significant applications in Medicine and Health Sciences. In this work we briefly present existing methodologies to conduct meta-analysis along with any discussion and recent developments accompanying them. Undoubtedly, studies brought together in a systematic review will differ in one way or another. This yields a considerable amount of variability, any kind of which may be termed heterogeneity. To this end, reports of meta-analyses commonly present a statistical test of heterogeneity when attempting to establish whether the included studies are indeed similar in terms of the reported output or not. We intend to provide an overview of the topic, discuss the potential sources of heterogeneity commonly met in the literature and provide useful guidelines on how to address this issue and to detect heterogeneity. Moreover, we review the recent developments in the Bayesian approach along with the various graphical tools and statistical software that are currently available to the analyst. In addition, we discuss sensitivity analysis issues and other approaches of understanding the causes of heterogeneity. Finally, we explore heterogeneity in meta-analysis for time to event data in a nutshell, pointing out its unique characteristics.

Keywords: graphical methods; heterogeneity; individual patient data; meta-analysis; time-to-event data
Corresponding author: Emmanouil Androulakis, Mathematical Modeling and Applications Laboratory, Section of Mathematics, Hellenic Naval Academy, Piraeus, Greece, E-mail:

Acknowledgment

We would like to thank the Associate Editor and the Referees for their useful comments which led to a substantial improvement in the content and the presentation of the article.

  1. Author contributions: All the authors have accepted responsibility for the entire content of this submitted manuscript and approved submission.

  2. Research funding: None declared.

  3. Conflict of interest statement: The authors declare no conflicts of interest regarding this article.

Appendix: Available software to conduct meta-analysis and assess heterogeneity

At this point it is appropriate to draw attention to the amount of progress that has occurred regarding software availability for meta-analysis [137, 138]. The Cochrane Collaboration software, RevMan [139], continues to be constantly developed with the capability of applying strategies for addressing heterogeneity. Stand-alone packages have also been presented, i.e. the commercial Comprehensive Meta Analysis [140], a software which is considered quite sophisticated, yet in the same time user-friendly, with a clear and intuitive user interface, that can be used to analyze datasets with heterogeneous data and demonstrate the implications of heterogeneity. Apart from that, MetaDiSc [141] allows exploration of heterogeneity with a variety of available statistics, such as I 2, whereas it implements meta-regression techniques, performs sensitivity testing using fixed-effects and random-effects, and produces figures of high quality. As a result, it has proved to be very useful for carrying out meta-analyses of diagnostic data. Except for the available commercialized products, independent users have also developed collections of macros, e.g. for STATA or SAS [142, 143]. Note that SAS does not offer an out-of-the-box solution devoted solely on meta-analysis, yet it has a comprehensive set of powerful and thoroughly-tested software offerings called procedures that are abbreviated in SAS syntax as “proc”. These procs can be used for a variety of meta-analysis purposes; indicatively they allow fitting mixed linear models with normal or non-normal response and random effects present, complex hierarchical models, nonlinear conditional mean functions and censored and survival analysis data. For example, the METAANAL macro in SAS produces the DSL estimators for fixed-effects or random-effects and tests for between studies heterogeneity. In addition to that, a user-friendly general package providing standard methods for meta-analysis in R is meta [144] for assessing heterogeneity with random-effects models. Bax et al. [145] assessed the differences in features, results and usability of available programs for meta-analysis up to 2007. Thereafter, Kontopantelis and Reeves [146], introduced the command metaan for conducting meta-analysis in STATA, which can be used to perform either fixed or random-effects meta-analysis. Metaan offers a wide choice of available models and reports a variety of heterogeneity measures, including Cochran’s Q, I 2, H 2, and the between-studies variance estimate τ ̂ 2 . A forest plot and a graph of the maximum likelihood function can also be generated. A more comprehensive package best suited for relatively more complex models, implemented in R, is metafor [147]. Furthermore, a newly developed package in R which aims to provide additional tools and functionalities for researchers conducting meta-analyses is dmetar [148]. In terms of software used to conduct analysis under the Bayesian perspective, WinBUGS [149] is a fully extensible modular framework for constructing and analyzing Bayesian full probability models. Lunn et al. [150] provided a practical introduction to BUGS, the most popular software for Bayesian analysis. Moreover, the “bayesmeta” R package [26] provides readily accessible tools to perform Bayesian meta-analyses and generate plots and summaries. It allows for flexible prior specification for the heterogeneity parameter τ and instant access to posterior distributions, including prediction and shrinkage estimation and facilitating quick sensitivity checks.

References

1. Ioannidis, J, Fanelli, D, Dunne, DD, Goodman, SN. Meta-research: evaluation and improvementof research methods and practices. PLoS Biol 2015;13:1–7. https://doi.org/10.1371/journal.pbio.1002264.Search in Google Scholar PubMed PubMed Central

2. Rothman, KJ, Greenland, S, Lash, T. Modern epidemiology, 3rd ed. Philadelphia, USA: Lippincott Williams and Wilkins; 2008.Search in Google Scholar

3. Pearson, K. Report on certain enteric fever inoculation statistics. Br Med J 1904;3:1243–6.10.1136/bmj.2.2288.1243Search in Google Scholar

4. Sutton, AJ, Higgins, JPT. Recent developments in meta-analysis. Stat Med 2008;27:625–50. https://doi.org/10.1002/sim.2934.Search in Google Scholar PubMed

5. Hartung, J, Knapp, G, Sinha, B. Statistical meta-analysis with applications. New Jersey, USA: Wiley; 2008.10.1002/9780470386347Search in Google Scholar

6. Petticrew, M. Systematic reviews from astronomy to zoology: myths and misconceptions. Br Med J 1904;322:98–101. https://doi.org/10.1136/bmj.322.7278.98.Search in Google Scholar PubMed PubMed Central

7. Moher, D, Cook, DJ, Eastwood, S, Olkin, I, Rennie, D, Stroup, D. Improving the quality of reporting of meta-analysis of randomised controlled trials: the quorum statement. Lancet 1999;354:1896–900. https://doi.org/10.1016/s0140-6736(99)04149-5.Search in Google Scholar PubMed

8. Moher, D, Schulz, KF, Altman, DG. The consort statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 2001;357:1191–4. https://doi.org/10.1016/s0140-6736(00)04337-3.Search in Google Scholar

9. Von Elm, E, Egger, M. The scandal of poor epidemiological research. Br Med J 2004;329:868–9. https://doi.org/10.1136/bmj.329.7471.868.Search in Google Scholar PubMed PubMed Central

10. Thompson, SG. Why sources of heterogeneity in meta-analysis should be investigated. Br Med J 1994;309:1351–5. https://doi.org/10.1136/bmj.309.6965.1351.Search in Google Scholar PubMed PubMed Central

11. Higgins, JP, Thomas, J, Chandler, J, Cumpston, M, Li, T, Page, MJ, .Cochrane handbook for systematic reviews of interventions. New Jersey, USA: John Wiley and Sons; 2019.10.1002/9781119536604Search in Google Scholar

12. Debray, TPA, Moons, KGM, Abo-Zaid, GMA, Koffijberg, H, Riley, RD. Individual participant data meta-analysis for a binary outcome: one-stage or two-stage? PLoS One 2013;8:1–10. https://doi.org/10.1371/journal.pone.0060650.Search in Google Scholar PubMed PubMed Central

13. Scotti, L, Rea, F, Corrao, G. One-stage and two-stage meta-analysis of individual participant data led to consistent summarized evidence: lessons learned from combining multiple databases. J Clin Epidemiol 2018;95:19–27. https://doi.org/10.1016/j.jclinepi.2017.11.020.Search in Google Scholar PubMed

14. Kontopantelis, E. A comparison of one-stage vs two-stage individual patient data meta-analysis methods: a simulation study. Res Synth Methods 2018;9:417–30. https://doi.org/10.1002/jrsm.1303.Search in Google Scholar PubMed PubMed Central

15. Burke, DL, Ensor, J, Riley, RD. A comparison of one-stage vs two-stage individual patient data meta-analysis methods: a simulation study. Stat Med 2017;36:855–75. https://doi.org/10.1002/sim.7141.Search in Google Scholar PubMed PubMed Central

16. Trikalinos, TA, Ioannidis, JPA. Predictive modeling and heterogeneity of baseline risk in meta-analysis of individual patient data. J Clin Epidemiol 2001;54:245–52. https://doi.org/10.1016/s0895-4356(00)00311-5.Search in Google Scholar PubMed

17. Smith, CT, Williamson, PR, Marson, AG. Investigating heterogeneity in an individual patient data meta-analysis of time to event outcomes. Stat Med 2005;24:1307–19. https://doi.org/10.1002/sim.2050.Search in Google Scholar PubMed

18. Simmonds, MC, Higgins, JPT, Stewart, LA, Tierney, JF, Clarke, MJ, Thompson, SG. Meta-analysis of individual patient data from randomized trials: a review of methods used in practice. Clin Trials 2005;2:209–17. https://doi.org/10.1191/1740774505cn087oa.Search in Google Scholar PubMed

19. Riley, RD, Lambert, PC, Zaid, GA. Meta-analysis of individual participant data: rationale, conduct, and reporting. BMJ 2010;340:221–7. https://doi.org/10.1136/bmj.c221.Search in Google Scholar PubMed

20. Siannis, F, Barrett, JK, Farewell, VT, Tierney, JF. One-stage parametric meta-analysis of time-to-event outcomes. Stat Med 2010;29:3030–45. https://doi.org/10.1002/sim.4086.Search in Google Scholar PubMed PubMed Central

21. Borenstein, M, Hedges, LV, Higgins, JPT, Rothsteind, HR. A basic introduction to fixed-effect and random-effects models for meta-analysis. Res Synth Methods 2010;1:97–111. https://doi.org/10.1002/jrsm.12.Search in Google Scholar PubMed

22. Kontopantelis, E, Springate, DA, Reeves, D. Re-analysis of the cochrane library data: the dangers of unobserved heterogeneity in meta-analyses. PLoS One 2013;8:1–4.10.1371/journal.pone.0069930Search in Google Scholar PubMed PubMed Central

23. DerSimonian, R, Laird, N. Meta-analysis in clinical trials. Contr Clin Trials 1986;7:177–88. https://doi.org/10.1016/0197-2456(86)90046-2.Search in Google Scholar PubMed

24. Sutton, AJ, Abrams, KR. Bayesian methods in meta-analysis and evidence synthesis. Stat Methods Med Res 2001;10:277–303. https://doi.org/10.1191/096228001678227794.Search in Google Scholar

25. Borenstein, M, Hedges, LV, Higgins, JPT, Rothstein, HR. Introduction to meta-analysis. West Sussex, England: John Wiley and Sons; 2009.10.1002/9780470743386Search in Google Scholar

26. Röver, C. Bayesian random-effects meta-analysis using the bayesmeta r package. J Stat Software 2020;93:1–51. https://doi.org/10.18637/jss.v093.i06.Search in Google Scholar

27. Röver, C, Friede, T. Dynamically borrowing strength from another study through shrinkage estimation. Stat Methods Med Res 2020;29:293–308. https://doi.org/10.1177/0962280219833079.Search in Google Scholar PubMed

28. Ott, M, Plummer, M, Roos, M. How vague is vague? How informative is informative? Reference analysis for Bayesian meta-analysis. Stat Med 2021;40:4505–21. https://doi.org/10.1002/sim.9076.Search in Google Scholar PubMed PubMed Central

29. Maritz, JS, Lwin, T. Empirical Bayes methods. London, England: Chapman and Hall; 1989.Search in Google Scholar

30. Smith, TC, Spiegelhalter, DJ, Thomas, A. Bayesian approaches to random-effects meta-analysis: a comparative study. Stat Med 1995;14:2685–99. https://doi.org/10.1002/sim.4780142408.Search in Google Scholar PubMed

31. Gagnier, JJ, Moher, D, Boon, H, Beyene, J, Bombardier, C. Investigating clinical heterogeneity in systematic reviews: a methodologic review of guidance in the literature. BMC Med Res Methodol 2012;12:111. https://doi.org/10.1186/1471-2288-12-111.Search in Google Scholar PubMed PubMed Central

32. Sutton, AJ, Cooper, NJ, Jones, DR, Lambert, PC, Thompson, JR, Abrams, KR. Evidence-based sample size calculations based upon updated meta-analysis. Stat Med 2007;26:2479–500. https://doi.org/10.1002/sim.2704.Search in Google Scholar PubMed

33. Cabrera, JA, Higgins, JPT. Graphical displays for meta-analysis: an overview with suggestions for practice. Res Synth Methods 2010;1:66–80. https://doi.org/10.1002/jrsm.6.Search in Google Scholar PubMed

34. Kossmeier, M, Tran, US, Voracek, M. Charting the landscape of graphical displays for meta-analysis and systematic reviews: a comprehensive review, taxonomy, and feature analysis. BMC Med Res Methodol 2020;20:1–24. https://doi.org/10.1186/s12874-020-0911-9.Search in Google Scholar PubMed PubMed Central

35. Sterne, JA, Sutton, AJ, Ioannidis, JP, Terrin, N, Jones, DR, Lau, J, et al.. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ Br Med J 2011;343:1–8. https://doi.org/10.1136/bmj.d4002.Search in Google Scholar PubMed

36. Galbraith, RF. A note on graphical presentation of estimated odds ratios from several clinical trials. Stat Med 1988;7:889–94. https://doi.org/10.1002/sim.4780070807.Search in Google Scholar PubMed

37. Copas, J, Lozada, CC. The radial plot in meta-analysis: approximations and applications. J Roy Stat Soc: C Appl Stat 2009;58:329–44. https://doi.org/10.1111/j.1467-9876.2008.00650.x.Search in Google Scholar

38. Song, F. Exploring heterogeneity in meta-analysis: is the l’abbe plot useful? J Clin Epidemiol 1999;52:725–30. https://doi.org/10.1016/s0895-4356(99)00066-9.Search in Google Scholar PubMed

39. L’Abbé, KA, Detsky, AS, O’Rourke, K. Meta-analysis in clinical research. Ann Intern Med 1987;2:224–33. https://doi.org/10.7326/0003-4819-107-2-224.Search in Google Scholar PubMed

40. Schild, AH, Voracek, M. Finding your way out of the forest without a trail of bread crumbs: development and evaluation of two novel displays of forest plots. Res Synth Methods 2015;6:74–86. https://doi.org/10.1002/jrsm.1125.Search in Google Scholar PubMed

41. Crowther, MJ, Riley, RD, Staessen, JA, Wang, J, Gueyffier, F, Lambert, PC. Individual patient data meta-analysis of survival data using Poisson regression models. BMC Med Res Methodol 2012;12:1–14. https://doi.org/10.1186/1471-2288-12-34.Search in Google Scholar PubMed PubMed Central

42. Olkin, I, Dahabreh, IJ, Trikalinos, TA. Gosh–a graphical display of study heterogeneity. Res Synth Methods 2012;3:214–23. https://doi.org/10.1002/jrsm.1053.Search in Google Scholar PubMed

43. Kulinskaya, E, Koricheva, J. Use of quality control charts for detection of outliers and temporal trends in cumulative meta-analysis. Res Synth Methods 2012;1:297–307. https://doi.org/10.1002/jrsm.29.Search in Google Scholar PubMed

44. Thompson, CG. Graphing effects as fuzzy numbers in meta-analysis. J Mod Appl Stat Methods 2016;15:957–77. https://doi.org/10.22237/jmasm/1462078080.Search in Google Scholar

45. Krahn, U, Binder, H, König, J. A graphical tool for locating inconsistency in network meta-analyses. BMC Med Res Methodol 2013;13:1–18. https://doi.org/10.1186/1471-2288-13-35.Search in Google Scholar PubMed PubMed Central

46. Walker, AM, Martin-Moreno, JM, Artalejo, FR. Odd man out: a graphical approach to meta-analysis. Am J Publ Health 1988;78:961–6. https://doi.org/10.2105/ajph.78.8.961.Search in Google Scholar PubMed PubMed Central

47. Baujat, B, Mahe, C, Pignon, JP. A graphical method for exploring heterogeneity in meta-analyses: application to a meta-analysis of 65 trials. Stat Med 2002;21:2641–52. https://doi.org/10.1002/sim.1221.Search in Google Scholar PubMed

48. Petitti, DB. Approaches to heterogeneity in meta-analysis. Stat Med 2001;20:3625–33. https://doi.org/10.1002/sim.1091.Search in Google Scholar PubMed

49. Cochran, WG. The combination of estimates from different experiments. Biometrics 1954;10:101–29. https://doi.org/10.2307/3001666.Search in Google Scholar

50. Hoaglin, DC. Misunderstandings about q and ’Cochran’s q test’ in meta-analysis. Stat Med 2016;35:485–95. https://doi.org/10.1002/sim.6632.Search in Google Scholar PubMed

51. Veroniki, AA, Jackson, D, Viechtbauer, W, Bender, R, Bowden, J, Knapp, G, et al.. Methods to estimate the between-study variance and its uncertainty in meta-analysis. Res Synth Methods 2016;7:55–79. https://doi.org/10.1002/jrsm.1164.Search in Google Scholar PubMed PubMed Central

52. DerSimonian, R, Kacker, R. Random-effects model for meta-analysis of clinical trials: an update. Contemp Clin Trials 2007;28:105–14. https://doi.org/10.1016/j.cct.2006.04.004.Search in Google Scholar PubMed

53. Biggerstaff, BJ, Tweedie, RL. Incorporating variability in estimates of heterogeneity in the random effects model in meta-analysis. Stat Med 1997;16:753–68. https://doi.org/10.1002/(sici)1097-0258(19970415)16:7<753::aid-sim494>3.0.co;2-g.10.1002/(SICI)1097-0258(19970415)16:7<753::AID-SIM494>3.0.CO;2-GSearch in Google Scholar

54. Brockwell, SE, Gordon, IR. A comparison of statistical methods for meta-analysis. Stat Med 2001;20:825–40. https://doi.org/10.1002/sim.650.Search in Google Scholar

55. Bohning, D, Malzahn, U, Dietz, E, Schlattmann, P. Some general points in estimating heterogeneity variance with the dersimonian–laird estimator. Biostatistics 2002;3:445–57. https://doi.org/10.1093/biostatistics/3.4.445.Search in Google Scholar

56. Sidik, K, Jonkman, JN. Simple heterogeneity variance estimation for meta-analysis. J Roy Stat Soc C Appl Stat 2005;54:367–85. https://doi.org/10.1111/j.1467-9876.2005.00489.x.Search in Google Scholar

57. Jackson, D, Bowden, J, Baker, R. How does the dersimonian and laird procedure for random effects meta-analysis compare with its more efficient but harder to compute counterparts? J Stat Plann Inference 2010;140:961–70. https://doi.org/10.1016/j.jspi.2009.09.017.Search in Google Scholar

58. Rukhin, AL. Estimating heterogeneity variance in meta-analysis. J Roy Stat Soc B Stat Methodol 2013;75:451–69. https://doi.org/10.1111/j.1467-9868.2012.01047.x.Search in Google Scholar

59. Hartung, J, Knapp, G. A refined method for the meta-analysis of controlled clinical trials with binary outcome. Stat Med 2001;30:3875–9. https://doi.org/10.1002/sim.1009.Search in Google Scholar

60. Amatya, A, Bhaumikb, DK, Normandc, SL, Greenhoused, J, Kaizare, E, Neelonf, B, et al.. Likelihood-based random effect meta-analysis of binary events. J Biopharm Stat 2014;25:984–1004. https://doi.org/10.1080/10543406.2014.920348.Search in Google Scholar

61. Steyerberg, EW, Nieboer, D, Debray, TP, Van Houwelingen, HC. Assessment of heterogeneity in an individual participant data meta-analysis of prediction models: an overview and illustration. Stat Med 2019;38:4290–309. https://doi.org/10.1002/sim.8296.Search in Google Scholar

62. Sidik, K, Jonkman, JN. A comparison of heterogeneity variance estimators in combining results of studies. Stat Med 2007;26:1964–81. https://doi.org/10.1002/sim.2688.Search in Google Scholar

63. Hedges, L. A random effects model for effect sizes. Psychol Bull 1983;93:388. https://doi.org/10.1037/0033-2909.93.2.388.Search in Google Scholar

64. Hardy, RJ, Thompson, SG. A likelihood approach to meta-analysis with random effects. Stat Med 1996;15:619–29. https://doi.org/10.1002/(sici)1097-0258(19960330)15:6<619::aid-sim188>3.0.co;2-a.10.1002/(SICI)1097-0258(19960330)15:6<619::AID-SIM188>3.0.CO;2-ASearch in Google Scholar

65. Thompson, SG, Sharp, SJ. Explaining heterogeneity in meta-analysis: a comparison of methods. Stat Med 1999;18:2693–708. https://doi.org/10.1002/(sici)1097-0258(19991030)18:20<2693::aid-sim235>3.0.co;2-v.10.1002/(SICI)1097-0258(19991030)18:20<2693::AID-SIM235>3.0.CO;2-VSearch in Google Scholar

66. Harville, DA. Maximum likelihood approaches to variance component estimation and to related problems. J Am Stat Assoc 1977;72:320–38. https://doi.org/10.1080/01621459.1977.10480998.Search in Google Scholar

67. Raudenbush, SW, Bryk, AS. Empirical bayes meta-analysis. J Educ Stat 1985;10:75–98. https://doi.org/10.2307/1164836.Search in Google Scholar

68. Morris, CN. Parametric empirical bayes inference: theory and applications. J Am Stat Assoc 1983;78:47–55. https://doi.org/10.1080/01621459.1983.10477920.Search in Google Scholar

69. Kontopantelis, E, Reeves, D. Performance of statistical methods for meta-analysis when true study effects are non-normally distributed: a simulation study. Stat Methods Med Res 2012;21:409–26. https://doi.org/10.1177/0962280210392008.Search in Google Scholar

70. Follmann, DA, Proschan, MA. Valid inference in random effects meta-analysis. Biometrics 1999;55:732–7. https://doi.org/10.1111/j.0006-341x.1999.00732.x.Search in Google Scholar

71. Kontopantelis, E, Reeves, D. Letter: performance of statistical methods for meta-analysis when true study effects are non-normally distributed: a comparison between dersimonian-laird and restricted maximum likelihood. Stat Methods Med Res 2012;21:657–9. https://doi.org/10.1177/0962280211413451.Search in Google Scholar

72. IntHout, J, Ioannidis, JP, Borm, GF. The hartung-knapp-sidik-jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard dersimonian-laird method. BMC Med Res Methodol 2014;14:1–12. https://doi.org/10.1186/1471-2288-14-25.Search in Google Scholar

73. Langan, D, Higgins, JP, Jackson, D, Bowden, J, Veroniki, AA, Kontopantelis, E, et al.. A comparison of heterogeneity variance estimators in simulated random-effects meta-analyses. Res Synth Methods 2019;10:83–98. https://doi.org/10.1002/jrsm.1316.Search in Google Scholar PubMed

74. Paule, RC, Mandel, J. Consensus values and weighting factors. J Res Natl Bur Stand 1982;87:377–85. https://doi.org/10.6028/jres.087.022.Search in Google Scholar PubMed PubMed Central

75. Hartung, J, Makambi, KH. Reducing the number of unjustified significant results in meta-analysis. Commun Stat Simulat Comput 2003;32:1179–90. https://doi.org/10.1081/sac-120023884.Search in Google Scholar

76. Sidik, K, Jonkman, JN. A simple confidence interval for meta-analysis. Stat Med 2002;21:3153–9. https://doi.org/10.1002/sim.1262.Search in Google Scholar PubMed

77. Knapp, G, Biggerstaff, BJ, Hartung, J. Assessing the amount of heterogeneity in random-effects meta-analysis. Stat Med 2006;48:271–85. https://doi.org/10.1002/bimj.200510175.Search in Google Scholar PubMed

78. Viechtbauer, W. Confidence intervals for the amount of heterogeneity in meta-analysis. Br Med J 2007;26:37–52. https://doi.org/10.1002/sim.2514.Search in Google Scholar PubMed

79. Switzer, FS, Paese, PW, Drasgow, F. Bootstrap estimates of standard errors in validity generalization. J Appl Psychol 1992;77:123. https://doi.org/10.1037/0021-9010.77.2.123.Search in Google Scholar

80. Turner, RM, Omar, RZ, Yang, M, Goldstein, H, Thompson, SG. A multilevel model framework for meta-analysis of clinical trials with binary outcomes. Stat Med 2000;19:3417–32. https://doi.org/10.1002/1097-0258(20001230)19:24<3417::aid-sim614>3.0.co;2-l.10.1002/1097-0258(20001230)19:24<3417::AID-SIM614>3.0.CO;2-LSearch in Google Scholar

81. Jackson, D. Confidence intervals for the between-study variance in random effects meta-analysis using generalised Cochran heterogeneity statistics. Res Synth Methods 2013;4:220–9. https://doi.org/10.1002/jrsm.1081.Search in Google Scholar

82. Veroniki, AA, Jackson, D, Bender, R, Kuss, O, Langan, D, Higgins, J, et al.. Methods to calculate uncertainty in the estimated overall effect size from a random-effects meta-analysis. Res Synth Methods 2019;10:23–43. https://doi.org/10.1002/jrsm.1319.Search in Google Scholar

83. Knapp, G, Hartung, J. Improved tests for a random effects meta-regression with a single covariate. Stat Med 2003;22:2693–710. https://doi.org/10.1002/sim.1482.Search in Google Scholar

84. Rhodes, KM, Turner, RM, Higgins, JP. Predictive distributions were developed for the extent of heterogeneity in meta-analyses of continuous outcome data. J Clin Epidemiol 2015;68:52–60. https://doi.org/10.1016/j.jclinepi.2014.08.012.Search in Google Scholar

85. Kulinskaya, E, Dollinger, MB, Bjørkestøl, K. On the moments of Cochran’s q statistic under the null hypothesis, with application to the meta-analysis of risk difference. Res Synth Methods 2011;2:254–70. https://doi.org/10.1002/jrsm.54.Search in Google Scholar

86. Kulinskaya, E, Dollinger, MB, Bjørkestøl, K. Testing for homogeneity in meta-analysis i. the one-parameter case: standardized mean difference. Biometrics 2011;67:203–12. https://doi.org/10.1111/j.1541-0420.2010.01442.x.Search in Google Scholar

87. Hardy, RJ, Thompson, SG. Detecting and describing heterogeneity in meta-analysis. Stat Med 1998;17:841–56. https://doi.org/10.1002/(sici)1097-0258(19980430)17:8<841::aid-sim781>3.0.co;2-d.10.1002/(SICI)1097-0258(19980430)17:8<841::AID-SIM781>3.0.CO;2-DSearch in Google Scholar

88. Jackson, D. The power of the standard test for the presence of heterogeneity in meta-analysis. J Clin Epidemiol 2006;25:2688–99. https://doi.org/10.1002/sim.2481.Search in Google Scholar

89. Higgins, JP, Thompson, SG. Quantifying heterogeneity in a meta-analysis. Stat Med 2002;21:1539–58. https://doi.org/10.1002/sim.1186.Search in Google Scholar

90. Higgins, JPT, Thompson, SG, Deeks, JJ, Altman, DG. Measuring inconsistency in meta-analyses. Br Med J 2003;327:557–60. https://doi.org/10.1136/bmj.327.7414.557.Search in Google Scholar

91. Borenstein, M, Higgins, J, Hedges, LV, Rothstein, H. Basics of meta-analysis: i2 is not an absolute measure of heterogeneity. Res Synth Methods 2017;8:5–18. https://doi.org/10.1002/jrsm.1230.Search in Google Scholar

92. Bohning, D, Lerdsuwansri, R, Holling, H. Some general points on the i2-measure of heterogeneity in meta-analysis. Metrika 2017;80:685–95. https://doi.org/10.1007/s00184-017-0622-3.Search in Google Scholar

93. Medina, TBH, Meca, JS, Martınez, FM, Botella, J. Assessing heterogeneity in meta-analysis: Q statistic or i2 index? Psychol Methods 2006;11:193–206. https://doi.org/10.1037/1082-989x.11.2.193.Search in Google Scholar

94. Mittlbock, M, Heinzl, H. A simulation study comparing properties of heterogeneity measures in meta-analyses. Stat Med 1998;25:4321–33. https://doi.org/10.1002/sim.2692.Search in Google Scholar

95. Ioannidis, JP, Patsopoulos, N, Evangelou, E. Uncertainty in heterogeneity estimates in meta-analyses. BMJ 2007;335:914. https://doi.org/10.1136/bmj.39343.408449.80.Search in Google Scholar

96. Rucker, G, Schwarzer, G, Carpenter, JR, Schumacher, M. Undue reliance on i2 in assessing heterogeneity may mislead. BMC Med Res Methodol 2008;8:79. https://doi.org/10.1186/1471-2288-8-79.Search in Google Scholar

97. Pathak, M, Dwivedi, SN, Deo, SVS, Sreenivas, V, Thakur, B. Which is the preferred measure of heterogeneity in meta-analysis and why? A revisit. Biostat Biometrics Open Acc 2017;1:1–7. https://doi.org/10.19080/bboaj.2017.01.555555.Search in Google Scholar

98. Crippa, A, Khudyakov, P, Wang, M, Orsini, N, Spiegelman, D. A new measure of between-studies heterogeneity in meta-analysis. Stat Med 2016;35:3661–75. https://doi.org/10.1002/sim.6980.Search in Google Scholar

99. Lin, L, Chu, H, Hodges, JS. Alternative measures of between-study heterogeneity in meta-analysis: reducing the impact of outlying studies. Biometrics 2017;73:156–66. https://doi.org/10.1111/biom.12543.Search in Google Scholar

100. Ma, X, Lin, L, Qu, Z, Zhu, M, Chu, H. Performance of between-study heterogeneity measures in the cochrane library. Epidemiology 2018;29:821–4. https://doi.org/10.1097/ede.0000000000000857.Search in Google Scholar

101. Bradburn, MJ, Deeks, JJ, Berlin, JA, Localio, R. Much ado about nothing: a comparison of the performance of meta-analytical methods with rare events. Stat Med 2007;26:53–77. https://doi.org/10.1002/sim.2528.Search in Google Scholar

102. Abrams, K, Sanso, B. Approximate Bayesian inference for random effects meta-analysis. Stat Med 1998;17:201–18. https://doi.org/10.1002/(sici)1097-0258(19980130)17:2<201::aid-sim736>3.0.co;2-9.10.1002/(SICI)1097-0258(19980130)17:2<201::AID-SIM736>3.0.CO;2-9Search in Google Scholar

103. Larose, DT, Dey, DK. Grouped random effects models for Bayesian meta-analysis. Stat Med 1997;16:1817–29. https://doi.org/10.1002/(sici)1097-0258(19970830)16:16<1817::aid-sim621>3.0.co;2-n.10.1002/(SICI)1097-0258(19970830)16:16<1817::AID-SIM621>3.0.CO;2-NSearch in Google Scholar

104. Pauler, DK, Wakefield, J. Modeling and implementation issues in Bayesian meta-analysis. In: Meta-analysis in medicine and health policy. Boca Raton, USA: CRC Press; 2000:190–214 pp.10.1201/9780203909935.ch9Search in Google Scholar

105. Lambert, PC, Sutton, AJ, Burton, PR, Abrams, KR, Jones, DR. How vague is vague? A simulation study of the impact of the use of vague prior distributions in MCMC using WinBUGS. Stat Med 2005;24:2401–28. https://doi.org/10.1002/sim.2112.Search in Google Scholar

106. Spiegelhalter, DJ, Abrams, KR, Myles, JP. Bayesian approaches to clinical trials and health-care evaluation. West Sussex, England: Wiley; 2003.10.1002/0470092602Search in Google Scholar

107. Abrams, KR, Gillies, CL, Lambert, PC. Meta-analysis of heterogeneously reported trials assessing change from baseline. Stat Med 2005;24:3823–44. https://doi.org/10.1002/sim.2423.Search in Google Scholar

108. Ashby, D. Bayesian statistics in medicine: a 25 year review. Stat Med 2006;25:3589–631. https://doi.org/10.1002/sim.2672.Search in Google Scholar

109. Warn, DE, Thompson, SG, Spiegelhalter, DJ. Bayesian random effects meta-analysis of trials with binary outcomes: methods for the absolute risk difference and relative risk scales. Stat Med 2002;21:1601–23. https://doi.org/10.1002/sim.1189.Search in Google Scholar

110. Bai, O, Chen, M, Wang, X. Bayesian estimation and testing in random effects meta-analysis of rare binary adverse events. Stat Biopharm Res 2016;8:49–59. https://doi.org/10.1080/19466315.2015.1096823.Search in Google Scholar

111. Bhaumik, DK, Amatya, A, Normand, SLT, Greenhouse, J, Kaizar, E, Neelon, B, et al.. Meta-analysis of rare binary adverse event data. J Am Stat Assoc 2012;107:555–67. https://doi.org/10.1080/01621459.2012.664484.Search in Google Scholar

112. Turner, RM, Davey, J, Clarke, MJ, Thompson, SG, Higgins, JPT. Predicting the extent of heterogeneity in meta-analysis, using empirical data from the cochrane database of systematic reviews. Int J Epidemiol 2012;41:818–27. https://doi.org/10.1093/ije/dys041.Search in Google Scholar

113. Bodnar, O, Link, A, Arendacká, B, Possolo, A, Elster, C. Bayesian estimation in random effects meta-analysis using a non-informative prior. Stat Med 2017;36:378–99. https://doi.org/10.1002/sim.7156.Search in Google Scholar PubMed

114. Bodnar, O, Link, A, Elster, C. Objective Bayesian inference for a generalized marginal random effects model. Bayesian Anal 2016;11:25–45. https://doi.org/10.1214/14-ba933.Search in Google Scholar

115. Jones, HE, Ades, AE, Sutton, AJ, Welton, NJ. Use of a random effects meta-analysis in the design and analysis of a new clinical trial. Stat Med 2018;37:4665–79. https://doi.org/10.1002/sim.7948.Search in Google Scholar PubMed PubMed Central

116. Moreno, E, Vázquez-Polo, FJ, Negrín, MA. Bayesian meta-analysis: the role of the between-sample heterogeneity. Stat Methods Med Res 2018;27:3643–57. https://doi.org/10.1177/0962280217709837.Search in Google Scholar PubMed

117. Hatala, R, Keitz, S, Wyer, P, Guyatt, G. Tips for learners of evidence-based medicine: 4. assessing heterogeneity of primary studies in systematic reviews and whether to combine their results. Can Med Assoc J 2005;172:661–5. https://doi.org/10.1503/cmaj.1031920.Search in Google Scholar PubMed PubMed Central

118. Patsopoulos, NA, Evangelou, E, Ioannidis, JPA. Sensitivity of between-study heterogeneity in meta-analysis: proposed metrics and empirical evaluation. Int J Epidemiol 2008;37:1148–57. https://doi.org/10.1093/ije/dyn065.Search in Google Scholar PubMed PubMed Central

119. Panityakul, T, Bumrungsup, C, Knapp, G. On estimating residual heterogeneity in random-effects meta-regression: a comparative study. J Stat Theor Appl 2013;12:253–65. https://doi.org/10.2991/jsta.2013.12.3.4.Search in Google Scholar

120. Simmonds, MC, Higgins, JP. Covariate heterogeneity in meta-analysis: criteria for deciding between meta-regression and individual participant data. Stat Med 2007;26:2982–99. https://doi.org/10.1002/sim.2768.Search in Google Scholar PubMed

121. Cooper, NJ, Sutton, AJ, Morris, D, Ades, AE, Welton, NJ. Addressing between-study heterogeneity and inconsistency in mixed treatment comparisons: application to stroke prevention treatments in individuals with non-rheumatic atrial fibrillation. Genet Epidemiol 2009;28:1861–81. https://doi.org/10.1002/sim.3594.Search in Google Scholar PubMed

122. Thomas, D, Radji, S, Benedetti, A. Systematic review of methods for individual patient data meta-analysis with binary outcomes. BMC Med Res Methodol 2014;14:1–9. https://doi.org/10.1186/1471-2288-14-79.Search in Google Scholar PubMed PubMed Central

123. Teramukai, S, Matsuyama, Y, Mizuno, S, Sakamoto, J. Individual patient-level and study-level meta-analysis for investigating modifiers of treatment effect. Jpn J Clin Oncol 2004;34:717–21. https://doi.org/10.1093/jjco/hyh138.Search in Google Scholar PubMed

124. Broeze, KA, Opmeer, BC, Van der Veen, F, Bossuyt, PM, Bhattacharya, S, Mol, BW. Individual patient data meta-analysis: a promising approach for evidence synthesis in reproductive medicine. Hum Reprod Update 2010;16:561–7. https://doi.org/10.1093/humupd/dmq043.Search in Google Scholar PubMed

125. Williamson, PR, Tudur Smith, C, Hutton, JL, Marson, AG. Aggregate data meta-analysis with time-to-event outcomes. Stat Med 2002;21:3337–51. https://doi.org/10.1002/sim.1303.Search in Google Scholar PubMed

126. Guevara, JP, Berlin, JA, Wolf, FM. Meta-analytic methods for pooling rates when follow-up duration varies: a case study. BMC Med Res Methodol 2004;4:17. https://doi.org/10.1186/1471-2288-4-17.Search in Google Scholar PubMed PubMed Central

127. Batson, S, Greenall, G, Hudson, P. Review of the reporting of survival analyses within randomised controlled trials and the implications for meta-analysis. PLoS One 2016;11:1–14. https://doi.org/10.1371/journal.pone.0154870.Search in Google Scholar PubMed PubMed Central

128. Wei, Y, Royston, P, Tierney, J, Parmar, M. Meta-analysis of time-to-event outcomes from randomized trials using restricted mean survival time: application to individual participant data. Stat Med 2015;34:2881–98. https://doi.org/10.1002/sim.6556.Search in Google Scholar PubMed PubMed Central

129. Michiels, S, Piedbois, P, Burdett, S, Syz, N, Stewart, L, Pignon, JP. Meta-analysis when only the median survival times are known: a comparison with individual patient data results. Int J Technol Assess Health Care 2005;21:119–25. https://doi.org/10.1017/s0266462305050154.Search in Google Scholar PubMed

130. Fiocco, M, Stijnen, T, Putter, H. Meta-analysis of time-to-event outcomes using a hazard-based approach: comparison with other models, robustness and meta-regression. Comput Stat Data Anal 2012;56:1028–37. https://doi.org/10.1016/j.csda.2011.05.009.Search in Google Scholar

131. Lindsay, BG. Composite likelihood methods. Contemp Math 1988;80:221–39.10.1090/conm/080/999014Search in Google Scholar

132. Moodie, PF, Nelson, NA, Koch, GG. A non-parametric procedure for evaluating treatment effect in the meta-analysis of survival data. Stat Med 2004;23:1075–93. https://doi.org/10.1002/sim.1696.Search in Google Scholar PubMed

133. Barrett, JK, Farewell, VT, Siannis, F, Tierney, JF, Higgins, JPT. Two-stage meta-analysis of survival data from individual participants using percentile ratios. Stat Med 2012;31:4296–308. https://doi.org/10.1002/sim.5516.Search in Google Scholar PubMed PubMed Central

134. Meddis, A, Latouche, A, Zhou, B, Michiels, S, Fine, J. Meta-analysis of clinical trials with competing time-to-event endpoints. Biom J 2020;62:712–23. https://doi.org/10.1002/bimj.201900103.Search in Google Scholar PubMed

135. Latouche, A, Allignol, A, Beyersmann, J, Labopin, M, Fine, JP. A competing risks analysis should report results on all cause-specific hazards and cumulative incidence functions. J Clin Epidemiol 2013;66:648–53. https://doi.org/10.1016/j.jclinepi.2012.09.017.Search in Google Scholar PubMed

136. Jansen, J, Cope, S. Meta-regression models to address heterogeneity and inconsistency in network meta-analysis of survival outcomes. BMC Med Res Methodol 2012;12:1–16. https://doi.org/10.1186/1471-2288-12-152.Search in Google Scholar PubMed PubMed Central

137. Sterne, JAC, Egger, M, Sutton, AJ. Meta–analysis software. In: Systematic reviews in health care: meta-analysis in context. London, England: BMJ Publishing Group; 2001:336–46 pp.10.1002/9780470693926.ch17Search in Google Scholar

138. Sutton, AJ, Lambert, PC, Hellmich, M, Abrams, KR, Jones, DR. Meta analysis in practice: a critical review of available software. In: Meta-analysis in medicine and health policy. New York, USA: Marcel Dekker; 2000.10.1201/9780203909935.ch15Search in Google Scholar

139. The Cochrane, Collaboration. RevMan analyses; computer program. Oxford, England: The Cochrane Collaboration; 2011.Search in Google Scholar

140. Borenstein, M, Hedges, L, Higgins, J, Rothstein, H. Comprehensive meta analysis, version 2. Englewood: Biostat; 2005.Search in Google Scholar

141. Zamora, J, Abraira, V, Muriel, A, Khan, K, Coomarasamy, A. Meta-disc: a software for meta-analysis of test accuracy data. BMC Med Res Methodol 2006;6:1–12. https://doi.org/10.1186/1471-2288-6-31.Search in Google Scholar PubMed PubMed Central

142. Sterne, JAC, Bradburn, MJ, Egger, M. Meta–analysis in stata. In: Systematicreviews in health care: meta-analysis in context. London, England: BMJ Publishing Group; 2001:347–69 pp.10.1002/9780470693926.ch18Search in Google Scholar

143. Wang, MC, Bushman, BJ. Integrating results through meta-analytic review using SAS software. North Carolina, USA: SAS Institute Inc; 1999.Search in Google Scholar

144. Schwarzer, G. meta: an r package for meta-analysis. R News 2007;7:40–5.Search in Google Scholar

145. Bax, L, Yu, LM, Ikeda, N, Moons, KG. A systematic comparison of software dedicated to meta-analysis of causal studies. Med Res Methodol 2007;7:1–9. https://doi.org/10.1186/1471-2288-7-40.Search in Google Scholar PubMed PubMed Central

146. Kontopantelis, E, Reeves, D. metaan: random-effects meta-analysis. STATA J 2010;10:395–407. https://doi.org/10.1177/1536867x1001000307.Search in Google Scholar

147. Viechtbauer, W. Conducting meta-analyses in r with the metafor package. J Stat Software 2010;36:1–48. https://doi.org/10.18637/jss.v036.i03.Search in Google Scholar

148. Harrer, M, Cuijpers, P, Furukawa, TA, Ebert, DD. Doing meta-analysis in R: a hands-on guide. New York, USA: CRC Press; 2021.10.1201/9781003107347Search in Google Scholar

149. Spiegelhalter, DJ, Thomas, A, Best, NG. WinBUGS version 14 user manual. Cambridge, England: MRC Biostatistics Unit; 2000.Search in Google Scholar

150. Lunn, D, Jackson, C, Best, N, Thomas, A, Spiegelhalter, D. The BUGS book: a practical introduction to Bayesian analysis. London, England: Chapman and Hall; 2012.10.1201/b13613Search in Google Scholar
Received: 2022-06-14
Accepted: 2023-02-10
Published Online: 2023-03-27

© 2023 Walter de Gruyter GmbH, Berlin/Boston

Articles in the same Issue

  1. Frontmatter
  2. Research Articles
  3. Survival analysis using deep learning with medical imaging
  4. Using a population-based Kalman estimator to model the COVID-19 epidemic in France: estimating associations between disease transmission and non-pharmaceutical interventions
  5. Approximate reciprocal relationship between two cause-specific hazard ratios in COVID-19 data with mutually exclusive events
  6. Sensitivity of estimands in clinical trials with imperfect compliance
  7. Highly robust causal semiparametric U-statistic with applications in biomedical studies
  8. Hierarchical Bayesian bootstrap for heterogeneous treatment effect estimation
  9. Penalized logistic regression with prior information for microarray gene expression classification
  10. Bayesian learners in gradient boosting for linear mixed models
  11. Unequal allocation of sample/event sizes with considerations of sampling cost for testing equality, non-inferiority/superiority, and equivalence of two Poisson rates
  12. HiPerMAb: a tool for judging the potential of small sample size biomarker pilot studies
  13. Heterogeneity in meta-analysis: a comprehensive overview
  14. On stochastic dynamic modeling of incidence data
  15. Power of testing for exposure effects under incomplete mediation
  16. Exact correction factor for estimating the OR in the presence of sparse data with a zero cell in 2 × 2 tables
  17. Right-censored partially linear regression model with error in variables: application with carotid endarterectomy dataset
  18. Assessing HIV-infected patient retention in a program of differentiated care in sub-Saharan Africa: a G-estimation approach
  19. Prediction-based variable selection for component-wise gradient boosting
https://doi.org/10.1515/ijb-2022-0070
Keywords for this article
graphical methods; heterogeneity; individual patient data; meta-analysis; time-to-event data

Articles in the same Issue

  1. Frontmatter
  2. Research Articles
  3. Survival analysis using deep learning with medical imaging
  4. Using a population-based Kalman estimator to model the COVID-19 epidemic in France: estimating associations between disease transmission and non-pharmaceutical interventions
  5. Approximate reciprocal relationship between two cause-specific hazard ratios in COVID-19 data with mutually exclusive events
  6. Sensitivity of estimands in clinical trials with imperfect compliance
  7. Highly robust causal semiparametric U-statistic with applications in biomedical studies
  8. Hierarchical Bayesian bootstrap for heterogeneous treatment effect estimation
  9. Penalized logistic regression with prior information for microarray gene expression classification
  10. Bayesian learners in gradient boosting for linear mixed models
  11. Unequal allocation of sample/event sizes with considerations of sampling cost for testing equality, non-inferiority/superiority, and equivalence of two Poisson rates
  12. HiPerMAb: a tool for judging the potential of small sample size biomarker pilot studies
  13. Heterogeneity in meta-analysis: a comprehensive overview
  14. On stochastic dynamic modeling of incidence data
  15. Power of testing for exposure effects under incomplete mediation
  16. Exact correction factor for estimating the OR in the presence of sparse data with a zero cell in 2 × 2 tables
  17. Right-censored partially linear regression model with error in variables: application with carotid endarterectomy dataset
  18. Assessing HIV-infected patient retention in a program of differentiated care in sub-Saharan Africa: a G-estimation approach
  19. Prediction-based variable selection for component-wise gradient boosting
Sign up now to receive a 20% welcome discount
Institutional Access
How does access work?
Have an idea on how to improve our website?
Please write us.
© 2025 De Gruyter Brill
Downloaded on 12.12.2025 from https://www.degruyterbrill.com/document/doi/10.1515/ijb-2022-0070/html
Scroll to top button