Home Two-sample t α -test for testing hypotheses in small-sample experiments
Article
Licensed
Unlicensed Requires Authentication

Two-sample t α -test for testing hypotheses in small-sample experiments

  • Yuan-De Tan EMAIL logo
Published/Copyright: June 24, 2022
Become an author with De Gruyter Brill
Submit Manuscript Author Information
The International Journal of Biostatistics
From the journal The International Journal of Biostatistics Volume 19 Issue 1

Abstract

It has been reported that about half of biological discoveries are irreproducible. These irreproducible discoveries were partially attributed to poor statistical power. The poor powers are majorly owned to small sample sizes. However, in molecular biology and medicine, due to the limit of biological resources and budget, most molecular biological experiments have been conducted with small samples. Two-sample t-test controls bias by using a degree of freedom. However, this also implicates that t-test has low power in small samples. A discovery found with low statistical power suggests that it has a poor reproducibility. So, promotion of statistical power is not a feasible way to enhance reproducibility in small-sample experiments. An alternative way is to reduce type I error rate. For doing so, a so-called t α -test was developed. Both theoretical analysis and simulation study demonstrate that t α -test much outperforms t-test. However, t α -test is reduced to t-test when sample sizes are over 15. Large-scale simulation studies and real experiment data show that t α -test significantly reduced type I error rate compared to t-test and Wilcoxon test in small-sample experiments. t α -test had almost the same empirical power with t-test. Null p-value density distribution explains why t α -test had so lower type I error rate than t-test. One real experimental dataset provides a typical example to show that t α -test outperforms t-test and a microarray dataset showed that t α -test had the best performance among five statistical methods. In addition, the density distribution and probability cumulative function of t α -statistic were given in mathematics and the theoretical and observed distributions are well matched.

Keywords: cumulative distribution function; hypothesis test; power; small samples; type I error; type II error
Corresponding author: Yuan-De Tan, Statistics core, Dan L. Duncan Cancer Center, Baylor College of Medicine, Houston, TX, USA, E-mail:

Acknowledgments

We are thankful to Dr.Li Qin for providing her original data for comparison between t-test and t α -test.

  1. Author contribution: All the authors have accepted responsibility for the entire content of this submitted manuscript and approved submission.

  2. Research funding: None declared.

  3. Conflict of interest statement: The authors declare no conflicts of interest regarding this article.

References

1. Gosset, WS. The probable error of a mean. Biometrika 1908;6:1–25. https://doi.org/10.2307/2331554.Search in Google Scholar

2. Box, JF. Guinness, gosset, fisher, and small samples. Stat Sci 1987;2:34–52. https://doi.org/10.1214/ss/1177013437.Search in Google Scholar

3. Lehmann, EL. Student and small-sample theory. Stat Sci 1999;14:1–30. https://doi.org/10.1214/ss/1009212520.Search in Google Scholar

4. Cohen, J. Approximate power and sample size determination for common one-sample and two- sample hypothesis tests. Educ Psychol Meas 1970;30:811–31. https://doi.org/10.1177/001316447003000404.Search in Google Scholar

5. Rossi, JS. Statistical power of psychological research: what have we gained in 20 years? J Consult Clin Psychol 1990;58:646–56. https://doi.org/10.1037/0022-006x.58.5.646.Search in Google Scholar

6. de Winter, JCF. Using the Student’s t-test with extremely small sample sizes. Practical Assess Res Eval 2013;18:1531–7714.Search in Google Scholar

7. Rost, DH. Effect strength vs. statistical significance: a warning against the danger of small samples: a comment on Gefferth and Herskovits’s article “Leisure activities as predictors of giftedness”. Eur J High Abil 1991;2:236–43. https://doi.org/10.1080/0937445910020212.Search in Google Scholar

8. Kochetkova, M, McKenzie, OL, Bais, AJ, Martin, JM, Seshadri, R, Powell, JA, et al.. CBFA2T3 (MTG16) is a putative breast tumor suppressor gene from the breast cancer loss of heterozygosity region at 16q24.3. Cancer Res 2002;62:4599–604.Search in Google Scholar

9. Chen, EG, Chen, Y, Dong, LL, Zhang, JS. Effects of SASH1 on lung cancer cell proliferation, apoptosis, and invasion in vitro. Tumor Biol 2012;33:1393–401. https://doi.org/10.1007/s13277-012-0387-2.Search in Google Scholar PubMed

10. Lin, S, Zhang, J, Xu, J, Wang, H, Sang, Q, Xing, Q, et al.. Effects of SASH1 on melanoma cell proliferation and apoptosis in vitro. Mol Med Rep 2012;6:1243–8. https://doi.org/10.3892/mmr.2012.1099.Search in Google Scholar PubMed

11. Meng, Q, Zheng, M, Liu, H, Song, C, Zhang, W, Yan, J, et al.. SASH1 regulates proliferation, apoptosis, and invasion of osteosarcoma cell. Mol Cell Biochem 2012;373:201–10. https://doi.org/10.1007/s11010-012-1491-8.Search in Google Scholar PubMed

12. Nasrallah, A, Saykali, B, Al Dimassi, S, Khoury, N, Hanna, S, El-Sibai, M. Effect of StarD13 on colorectal cancer proliferation, motility and invasion. Oncol Rep 2013;31:505–15. https://doi.org/10.3892/or.2013.2861.Search in Google Scholar PubMed

13. Hanna, S, Khalil, B, Nasrallah, A, Saykali, BA, Sobh, R, Nasser, S, et al.. StarD13 is a tumor suppressor in breast cancer that regulates cell motility and invasion. Int J Oncol 2014;44:1499–511. https://doi.org/10.3892/ijo.2014.2330.Search in Google Scholar PubMed PubMed Central

14. Ishibashi, M, Yokosuka, T, Yanagimachi, MD, Iwasaki, F, Tsujimoto, SI, Sasaki, K, et al.. Clinical courses of two pediatric patients with acute megakaryoblastic leukemia harboring the cbfa2t3-GLIS2 fusion gene. Turk J Haematol 2016;33:331–4. https://doi.org/10.4274/tjh.2016.0008.Search in Google Scholar PubMed PubMed Central

15. Altman, N, Krzywinski, M. Interpreting P values. Nat Methods 2017;14:213–4. https://doi.org/10.1038/nmeth.4210.Search in Google Scholar

16. Altman, N, Krzywinski, M. P values and the search for significance. Nat Methods 2017;14:4. https://doi.org/10.1038/nmeth.4120.Search in Google Scholar

17. Aarts, A, Anderson, J, Anderson, C, Attridge, P, Attwood, A, Axt, J, et al.. PSYCHOLOGY Estimating the reproducibility of psychological science. Science 2015;349:aac4716.10.1126/science.aac4716Search in Google Scholar PubMed

18. Baker, M. Reproducibility crisis: blame it on the antibodies. Nature 2015;521:274–6. https://doi.org/10.1038/521274a.Search in Google Scholar PubMed

19. Baker, M. Biotech giant posts negative results. Nature 2016;530:141. https://doi.org/10.1038/nature.2016.19269.Search in Google Scholar PubMed

20. Begley, CG, Ellis, LM. Drug development: Raise standards for preclinical cancer research. Nature 2012;483:531–3. https://doi.org/10.1038/483531a.Search in Google Scholar PubMed

21. Schooler, JW. Metascience could rescue the ‘replication crisis’. Nature 2014;515:9. https://doi.org/10.1038/515009a.Search in Google Scholar PubMed

22. Colquhoun, D. The reproducibility of research and the misinterpretation of p-values. R Soc Open Sci 2018;4:171085. https://doi.org/10.1098/rsos.171085.Search in Google Scholar PubMed PubMed Central

23. Baldi, P, Long, AD. A Bayesian framework for the analysis of microarray expression data: regularized t -test and statistical inferences of gene changes. Bioinformatics 2001;17:509–19. https://doi.org/10.1093/bioinformatics/17.6.509.Search in Google Scholar PubMed

24. Tusher, VG, Tibshirani, R, Chu, G. Significance analysis of microarrays applied to the ionizing radiation response. Proc Natl Acad Sci U S A 2003;98:5116–21. https://doi.org/10.1073/pnas.091062498.Search in Google Scholar PubMed PubMed Central

25. Cui, X, Churchill, GA. Statistical tests for differential expression in cDNA microarray experiments. Genome Biol 2003;4:210. https://doi.org/10.1186/gb-2003-4-4-210.Search in Google Scholar PubMed PubMed Central

26. Anders, S, Huber, W. Differential expression analysis for sequence count data. Genome Biol 2001;11:R106. https://doi.org/10.1186/gb-2010-11-10-r106.Search in Google Scholar PubMed PubMed Central

27. Efron, B, Tibshirani, R, Storey, SD, Tusher, V. Empirical bayes analysis of a microarray experiment. J Amer Statist Assoc 2001;96:1151–60.10.1198/016214501753382129Search in Google Scholar

28. Love, MI, Huber, W, Anders, S. Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2. Genome Biol 2014;15:550. https://doi.org/10.1186/s13059-014-0550-8.Search in Google Scholar PubMed PubMed Central

29. Opgen-Rhein, R, Strimmer, K. Accurate ranking of differentially expressed genes by a distribution-free shrinkage approach. Stat Appl Genet Mol Biol 2007;6:9. https://doi.org/10.2202/1544-6115.1252.Search in Google Scholar PubMed

30. Robinson, MD, McCarthy, DJ, Smyth, GK. edgeR: a bioconductor package for differential expression analysis of digital gene expression data. Bioinfortmatics 2009;26:139–40. https://doi.org/10.1093/bioinformatics/btp616.Search in Google Scholar PubMed PubMed Central

31. Robinson, MD, Smyth, GK. Small-sample estimation of negative binomial dispersion, with applications to SAGE data. Biostatistics 2008;9:321–32. https://doi.org/10.1093/biostatistics/kxm030.Search in Google Scholar PubMed

32. Tan, YD, Chandler, AM, Chaudhury, A, Neilson, JR. A powerful statistical approach for large-scale differential transcription analysis. Plos One 2015. https://doi.org/10.1371/journal.pone.0123658. In this issue.Search in Google Scholar PubMed PubMed Central

33. Satterthwaite, FE. An approximate distribution of estimates of variance components. Biometrics Bull 1946;2:110–4. https://doi.org/10.2307/3002019.Search in Google Scholar

34. Forero, LO. Wilcoxon-Mann-Whitney test and a small sample size; 2013. Available from: https://www.blopig.com/blog/2013/10/wilcoxon-mann-whitney-test-and-a-small-sample-size/.Search in Google Scholar

35. Fornage, M, Chiang, YA, O’Meara, ES, Psaty, BM, Reiner, AP, Siscovick, DS, et al.. Biomarkers of inflammation and MRI-defined small vessel disease of the brain: the cardiovascular health study. Stroke 2008;39:1952–9. https://doi.org/10.1161/strokeaha.107.508135.Search in Google Scholar

36. Qin, L, Wu, YL, Toneff, MJ, Li, D, Liao, L, Gao, X, et al.. NCOA1 directly targets M-CSF1 expression to promote breast cancer metastasis. Cancer Res 2014;74:3477–88. https://doi.org/10.1158/0008-5472.can-13-2639.Search in Google Scholar

37. Ritchie, ME, Phipson, B, Wu, D, Hu, Y, Law, CW, Shi, W, et al.. Limma powers differential expression analyses for RNA-sequencing and microarray studies. Nucleic Acids Res 2015;43:e47. https://doi.org/10.1093/nar/gkv007.Search in Google Scholar PubMed PubMed Central

38. Onder, TT, Gupta, PB, Mani, SA, Yang, J, Lander, ES, Weinberg, RA. Loss of E-cadherin promotes metastasis via multiple downstream transcriptional pathways. Cancer Res 2008;68:3645–54. https://doi.org/10.1158/0008-5472.can-07-2938.Search in Google Scholar

39. Smyth, GK. limma: Linear Models for Microarray Data. New York: Springer; 2005.Search in Google Scholar

40. Siegel, SE. Nonparametric statistics for the behavioral sciences. New York: McGraw-Hill; 1956.Search in Google Scholar

41. Siegel, SE. Nonparametric statistics. Am Statistician 1957;11:13–9. https://doi.org/10.1080/00031305.1957.10501091.Search in Google Scholar

Supplementary Material

The online version of this article offers supplementary material (https://doi.org/10.1515/ijb-2021-0047).

Received: 2021-06-02
Accepted: 2022-05-29
Published Online: 2022-06-24

© 2022 Walter de Gruyter GmbH, Berlin/Boston

Articles in the same Issue

  1. Frontmatter
  2. Research Articles
  3. Two-sample t α -test for testing hypotheses in small-sample experiments
  4. Estimating risk and rate ratio in rare events meta-analysis with the Mantel–Haenszel estimator and assessing heterogeneity
  5. Estimating population-averaged hazard ratios in the presence of unmeasured confounding
  6. Commentary
  7. Comments on ‘A weighting analogue to pair matching in propensity score analysis’ by L. Li and T. Greene
  8. Research Articles
  9. Variable selection for bivariate interval-censored failure time data under linear transformation models
  10. A quantile regression estimator for interval-censored data
  11. Modeling sign concordance of quantile regression residuals with multiple outcomes
  12. Robust statistical boosting with quantile-based adaptive loss functions
  13. A varying-coefficient partially linear transformation model for length-biased data with an application to HIV vaccine studies
  14. Application of the patient-reported outcomes continual reassessment method to a phase I study of radiotherapy in endometrial cancer
  15. Borrowing historical information for non-inferiority trials on Covid-19 vaccines
  16. Multivariate small area modelling of undernutrition prevalence among under-five children in Bangladesh
  17. The optimal dynamic treatment rule superlearner: considerations, performance, and application to criminal justice interventions
  18. Estimators for the value of the optimal dynamic treatment rule with application to criminal justice interventions
  19. Efficient estimation of pathwise differentiable target parameters with the undersmoothed highly adaptive lasso
https://doi.org/10.1515/ijb-2021-0047
Keywords for this article
cumulative distribution function; hypothesis test; power; small samples; type I error; type II error

Articles in the same Issue

  1. Frontmatter
  2. Research Articles
  3. Two-sample t α -test for testing hypotheses in small-sample experiments
  4. Estimating risk and rate ratio in rare events meta-analysis with the Mantel–Haenszel estimator and assessing heterogeneity
  5. Estimating population-averaged hazard ratios in the presence of unmeasured confounding
  6. Commentary
  7. Comments on ‘A weighting analogue to pair matching in propensity score analysis’ by L. Li and T. Greene
  8. Research Articles
  9. Variable selection for bivariate interval-censored failure time data under linear transformation models
  10. A quantile regression estimator for interval-censored data
  11. Modeling sign concordance of quantile regression residuals with multiple outcomes
  12. Robust statistical boosting with quantile-based adaptive loss functions
  13. A varying-coefficient partially linear transformation model for length-biased data with an application to HIV vaccine studies
  14. Application of the patient-reported outcomes continual reassessment method to a phase I study of radiotherapy in endometrial cancer
  15. Borrowing historical information for non-inferiority trials on Covid-19 vaccines
  16. Multivariate small area modelling of undernutrition prevalence among under-five children in Bangladesh
  17. The optimal dynamic treatment rule superlearner: considerations, performance, and application to criminal justice interventions
  18. Estimators for the value of the optimal dynamic treatment rule with application to criminal justice interventions
  19. Efficient estimation of pathwise differentiable target parameters with the undersmoothed highly adaptive lasso
Sign up now to receive a 20% welcome discount
Institutional Access
How does access work?
Have an idea on how to improve our website?
Please write us.
© 2025 De Gruyter Brill
Downloaded on 27.9.2025 from https://www.degruyterbrill.com/document/doi/10.1515/ijb-2021-0047/html
Scroll to top button