Setting Statistical Hurdles for Publishing in Accounting

Appendix: ASA Statement on Statistical Significance and P-Values

We summarize the six principles, labeled P1 through P6, from the 2016 ASA Statement on Statistical Significance and P-values in this appendix and highlight common mistakes of accounting researchers in implementing them.

1. P-Values can indicate how incompatible the data are with a specified statistical method.

   A small p-value (usually < 0.05) means that there is strong evidence against the null hypothesis (e.g. there is no association between x and y variables) with the current data so one can reject the null. However, a common mistake by accounting researchers is to conclude that a large p-value (>0.05) means that there is x is unrelated to y when all that can be concluded is that the evidence is inconclusive as it is compatible with the null hypothesis being either true or false (Cready et al. 2022).

1. P-Values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.

   The researcher should be reminded that a p-value does not tell one whether the null hypothesis is true or not. It only tells one how likely it is that the data would have turned out the way it did if the (null) hypothesis is actually true. A small p-value suggests that it is unlikely that the data would have turned out as observed if x and y were unrelated, but it does not prove that x and y are related.

1. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.

   This is a common mistake of researchers making conclusions from the statistical findings in empirical research. Principle 3 of the ASA Statement says that a p-value should not be used as the sole basis for making accounting policies driven by accounting research. A p-value is just one piece of information that researchers use to make decisions about the data. Other factors such as study design, measurement quality, external evidence, and assumptions underlying data analysis should also be discussed when making inferences from data. Using a “bright-line” rule like “p < 0.05” to justify scientific claims can lead to incorrect beliefs and poor decision-making.

1. Proper inference requires full reporting and transparency.

   This is key concern because journals have incentives to publish significant results and often reject non-significant results. Researchers should not cherry pick positive (significant) findings only, a practice called variously data dredging, significance chasing or questing, selective inference or p-hacking. Harvey’s (2017) presidential address to the American Finance Association highlighted this concern and suggests strategies for discouraging this practice. The ASA recommends that researchers disclose the number of hypotheses explored during the study, all data collection decisions, all statistical analyses conducted, and all p-values computed. Note that a step towards transparency in research for experimental studies is pre-registering a study. This practice may bring advantages also to benefit archival studies.

1. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.

   We quote the ASA statement directly – “Statistical significance is not equivalent to scientific or economic significance. Smaller p-values do not necessarily imply the presence of larger or more important effects, and conversely larger p-values do not imply a lack of importance or even lack of effect.”

   Also, researchers need to be reminded that any effect, no matter how tiny, can produce a small p-value if the sample size is high enough, and conversely large effects may produce unimpressive p-values if the sample size is small. Similarly, identical estimated effects will have different p-values if the measurement error of the variables differs.” See Johannesson, Ohlson and Zhai (2023) for the application of this point in the accounting context.

1. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.

   Principle 6 of the ASA Statement says that researchers should rely on p-values with context, use other approaches and robustness tests for proper inferences. Examples of other approaches include confidence or prediction intervals; Bayesian methods; likelihood ratios or Bayes factors; false discovery rates. The additional approaches can also provide additional evidence about size or uncertainty of an effect.

   A final remark going beyond the six principles: we would like to remind researchers that the logical reasoning for the null and alternative hypotheses must pass what we refer to as the “smell test.” If the reasonings for the mechanisms to produce the proposed hypotheses are too fanciful, Occam’s razor rule suggests that evidence from testing the hypothesis is unlikely to yield meaningful inferences.