How to reduce scientific irreproducibility: the 5-year reflection

It seems these days everybody is talking about irreproducibility in science [1]. But is this a new problem? Certainly not! The increased awareness is due to the fact that the number of irreproducible papers published in high-impact journals is on the rise [2]. In 2006, one of us (EPD) wrote a mentorship paper addressing this problem [3]. Also, John Ioannidis [4] and others published repeatedly on the same issue, including one paper with the attention catching title “Why most published research findings are false”. Can this menace to science be eliminated? Likely not, but it could be significantly reduced with some measures (please see below).

We emphasize here that this issue is of great interest to laboratory professionals and other readers of CCLM because a significant number of papers describing disease biomarkers fall into the category of irreproducible science.

In this paper, we identify six types of irreproducible results or papers in science (Table 1).

Type 1 irreproducibility is due to fraud, which is presumed to be rare. Some highly publicized cases can be found in the cited literature [5], [6], [7], [8].

Type 2 irreproducible papers are usually written by prominent authorities, such as Nobel Prize winners, usually in an area that is unrelated to their Nobel-winning discovery. Some of these authors may suffer from a disease that one of us coined recently as “Nobelitis”, which is related to another megalomania-related malady known as “Hubris syndrome” [9], [10]. In such cases, these authors overestimate their abilities due to their fame and publish papers that are not well-founded. An example is the case of the brilliant Chemist Linus Pauling (a double Nobel Prize winner) who claimed in the 1970s that mega-doses of vitamin C can prevent or cure cancer. Specialists in this field worked for years examining this suggestion and eventually discredited it. As expected, Pauling never accepted the verdict. Even Einstein published blunders. For example, in his work about relativity, he added an (incorrect) cosmological constant to balance his equations [11].

A third type of irreproducibility is false discovery due to bias (pre-analytical/analytical/post-analytical) [12]. Bias can lead to statistically significant differences between the comparison groups (e.g. control subjects and cancer patients) that are not due to the disease at hand but to something else. Ideally, scientists should control for every factor in the comparison groups but this is easier said than done. There are numerous examples of papers published in top-tier journals where there was obvious bias between the comparison groups. For example, comparing a test for prostate cancer diagnosis between a control group of 20–30-year-old men to prostate cancer patients (who are usually 60–70 years old) can be very problematic as differences between the groups could be due to cancer or to the large age difference. Although this appears to be an obvious bias, we previously identified a study in a prestigious journal which utilized this troublesome comparison [13]. Other biases can be very subtle. For example, biotechnology company “Atairgin”, formed in 1996 with millions of dollars of investment, aimed to commercialize a test for early ovarian cancer diagnosis [14]. The company folded a few years later after realizing that the observed differences between their controls and ovarian cancer patients in a study to develop their test was merely due to minor differences in the centrifugation speed of blood samples collected at two different locations. This is a prime example of a very subtle bias which was very difficult to spot but had massive impact on the results. Other examples are cited in our previous review [14].

Irreproducibility type 4 is usually due to technical deficiencies of those who do the experiments, especially unawareness of the limitations of the techniques used. We have plenty of experience with our own graduate students working with mass spectrometry instrumentation. These trainees are familiar with the routine operation of the machines but they generally lack in-depth knowledge of the technology and its limitations. For example, we have seen students using the wrong parameters to search their results in public databases while others used the wrong Zip-Tips (a solid-phase extraction device) for sample preparation. These small technical details can lead to significant errors in data collection and interpretation. As we often tell our students “a small hole can sink a big ship”. A newly surfaced reason for paper irreproducibility, included in the technical deficiencies category, is what is known as “counterfeit science”. In this case, manufacturers sell fake reagents, which generate misleading (and irreproducible) results [15], [16].

Type 5 irreproducibility is on the rise and it is a byproduct of fragmented science. Mega projects are now executed in pieces in various laboratories, sometimes on an international scale, and the results are then knitted together, usually by a single principal investigator. As it is virtually impossible to know exactly what is happening in other sites, the PI accepts external results based on faith. This practice can lead to disasters, as was the case with the 2014 pluripotent stem cell case, where prominent scientists participated in a fraudulent (and later retracted) publication in a top tier journal [7], [17]. As fragmented science is becoming more common, this problem will likely become more frequent.

Irreproducibility type 6 is associated with the recent trend towards big data generation and analysis by bioinformatics. With big data, it is next to impossible to be sure that the bioinformatics tools are analyzing the information with great robustness. Small glitches in the tools may lead to false conclusions. An example of this type of error is the case of a well-respected crystallographer who was forced to retract five high-impact papers in 2007 (three in Science, one in PNAS and one in J. Mol. Biol.) due to a very small computer program glitch [18].

Is there a radical solution to these irreproducibility issues in science? Likely not, but some recent efforts to minimize the problem are worth mentioning.

The Reproducibility Project was launched in 2013 with the goal of reproducing data from 50 cancer papers published in high-impact journals [19]. Due to financial constraints, the project was subsequently downsized and data on five papers are now published [20]. The results were not clear-cut, precluding firm conclusions. In these authors’ opinion, this project is not viable, for four major reasons. This process is costly, frequently inconclusive, technically demanding and slow.

Recently, Mogil and Macleod [21] proposed inclusion of very strong preclinical data before publishing a paper with perceived impact. Though this suggestion could improve scientific irreproducibility, we doubt that it will be effective as generating such preclinical data will be expensive, time consuming and will likely delay publication of critical information. Also recently, top tier journals have adopted new policies for submitting manuscripts, requiring more explicit details of the experimental and statistical procedures [22].

We propose an alternative solution to address the issue of irreproducibility of papers with translational or commercial importance (perceived as “breakthroughs”). We suggest authors of such papers should be invited to provide a 5-year (and perhaps a 10-year) reflection on their papers. With this system, authors will be obliged to sign a declaration form at publication (like the commonly used conflict of interest forms) where they will agree to publish a reflection on their work; including information such as whether the invention was translated or commercialized, the state of its development and any other insights (especially identification of errors, misinterpretations or other roadblocks). We proposed this idea to some authors of such “high-impact” papers a few years back, but none of them agreed to write such a reflection. Nonetheless, with the advent of electronic publishing, it is now possible for journals to easily publish such a 5–10-year reflection, which could be an addendum to the original manuscript. This is a no cost procedure which will allow readers to very quickly assess the impact and success or failures of these papers.

We believe the obligation of authors to write a reflection on their high-impact papers will make them be more careful when executing their experiments and more conservative when publishing their work. For example, they will likely avoid overselling their results. Furthermore, we believe that this obligation will likely improve upon other stages of any research project, such as idea conception and experimental planning.

Work by psychologists Jessica Lerner and Philip Tetlock [23] suggests that simple steps meant to hold people accountable for their judgment can actually improve their judgment. In this spirit, in Figure 1, we describe the thought process of a scientist who is executing an experiment, anticipating that he/she may or may not be writing a reflection on the outcome of his/her work in the near future. We postulate that the experiment will likely be planned, executed and published with a different state of mind between the two scenarios. The future accountability, represented by the reflection, will likely lead to more sound and reproducible results.

Figure 1:

Behavioral modification of investigators who are expected to write (lower panel) or not write (upper panel) a 5-year reflection. Note that all steps of the investigation, from idea conception to publishing, will likely be improved with the 5-year reflection. This thought modification will likely lead to better and more reproducible science. This behavioral modification has been proposed earlier by Lerner and Tetlock [23].

We hope that high-impact journals, especially in the clinical sciences, will adopt this proposal as a mandatory publication requirement for papers with seemingly large translational potential. We also suggest that funding agencies adopt similar requirements for their grantees and include such reflections in the reporting process for their awarded grants. The outcome of our proposal, to make scientists reflect, could be evaluated in the future as a no cost measure for its effectiveness in reducing scientific irreproducibility.