Think twice before starting a new trial; what is the impact of recommendations to stop doing new trials?

Design

We analysed the number of trials registered on the World Health Organisation (WHO) International Clinical Trials Registry Platform (ICTRP) as this is the most complete trial registry, in 4-years’ time windows before and after the recommendation not to conduct further trials.

Selection criteria

We used the same selection criteria as the two papers that recommended not to conduct further trials (Appendix 1) [6], [7]. For the first recommendation registered trial protocols were included if they met the following criteria: (1) RCT comparing at least two interventions; (2) participants with chronic LBP (≥3 months) randomly allocated to either exercise or a control group of no or minimal intervention; and (3) measured pain or disability as an outcome. For the second recommendation, registered trial protocols were included if they met the following criteria: (1) RCT comparing at least two interventions; (2) participants with chronic pain (≥3 months) that were randomly allocated to CBT or a control group of no intervention or a non-psychotherapy alternative intervention; and (3) measured pain or disability as an outcome.

Trials evaluating psychological treatments with a component of CBT were included, e.g. internet-based CBT, acceptance and commitment therapy (ACT) and other modifications of CBT, both individual as group-based. We excluded physiotherapy as a control arm, assuming that this would be an active intervention.

Search strategy

We searched the WHO-ICTRP (http://apps.who.int/trialsearch/) for trial protocols using the selection criteria. The papers that made the recommendations were published in 2012. Therefore, we searched for trials registered in an a priori set period of 4 years between 1st January 2008 and 31st December 2011 and for trials registered between 1st January 2015 and 31st December 2018. The a priori set period after the recommendation was chosen based on the idea that researchers should have been able to read the recommendation prior to planning and conducting a RCT.

We were unable to use “exercise” or “CBT” as search terms to limit the yield (Appendix 2) and therefore the yield in CBT for chronic pain was much larger than in exercise for LBP. To ensure we had comparable numbers we decided to limit the period for CBT to 2-year periods (1st January 2009 and 31st December 2010 and between 1st January 2016 and 31st December 2017).

Screening

Two authors independently screened all titles and abstracts for relevance (IS and LvR), and a third author (AV) resolved the conflicts. We extracted the data from the registry, and in case of uncertainty, we checked the original registration and all available (published) data. In addition, the third assessor performed a 10% random check of all titles and abstracts from the original registration.

Data extraction

Data were extracted on: (1) the number of trials registered that evaluated exercise/CBT compared to no intervention in patients with LBP and chronic pain, and absolute numbers of registered trials on LBP and chronic pain in the trial registry; (2) trial-specific factors (e.g. condition, setting, country, sample size, ethical approval and granting body [last check August 2019]); and (3) factors related to the publication (e.g. journal of publication [impact factor], strength of recommendation, Altmetrics score, keywords used). The Altmetrics score, literally “alternative metrics”, measures and monitors the reach and impact of research publications through online interactions, next to the traditional measurements of academic success such as citation counts, impact factor, and author H-index (www.altmetrics.com). While the Almetric score does not measure how many times a particular article is read it does provide an indication of the extent of the readership.

Data synthesis

First, we calculated frequencies of the absolute and relative numbers of included trial protocols before and after 2012 both for exercise in LBP and for CBT in chronic pain. We calculated the relative number by dividing the number of trials that met our selection criteria (included trials) by the total number of trials that evaluated exercise/CBT in patients with LBP/chronic pain in the trial registry during the search period. Next, the relative risk ratio, absolute change and relative change were calculated. For our primary aim, we decided a priori that we consider a change in the number of trials registered that was a relative positive or negative change, when this change was >10%. If the relative change was between −10 and 10% we would consider this as no change.

Search

There were in total 1,574 trials registered for exercise in LBP: 459 before 2012 and 1,115 after (Figure 1) and 5,037 trials on chronic pain: 1,564 before 2012 and 3,473 after (Figure 2). There was a 90.7% agreement between the two authors in the selection of included trials. We present all trial-specific factors in Appendix 3.

Figure 1:

PRISMA flowchart for trials evaluating exercise for low back pain (LBP) in period 2008–2011 (pre-recommendation) and 2015–2018 (post-recommendation).

Figure 2:

PRISMA flowchart for trials evaluating cognitive behavioural therapy (CBT) for chronic pain in period 2009–2010 (pre-recommendation) and 2016–2017 (post-recommendation).

Impact of recommendations

Characteristics of included trials

Specific factors related to the recommendation

Main findings

We found a marked reduction in clinical trials of CBT for chronic pain, but a marked increase in clinical trials of exercise for LBP, following clear recommendations that neither intervention required further investigation. Our data demonstrated that researchers can, but don’t always, follow recommendations from other researchers as regards the value of conducting further RCTs.

The increase in the number of LBP trials registered may not have been influenced by the publication of this recommendation. The publication itself did not have a clear statement on exercise in LBP in the title or abstract, and there were no keywords provided [6]. Although it was published in a relatively high impact journal, it did not result in much media exposure (Altmetrics score). Also, an overview of Cochrane reviews on exercise in adults with chronic pain stated that further research is needed [13]. Although this overview was not specifically targeted at LBP, just three out of 21 included reviews were on LBP, and conclusions were not specific for LBP either, it could have influenced the increase in exercise for LBP protocols.

By contrast, the chronic pain recommendation in the Cochrane review (also a high impact journal) had a firmer recommendation, was easier to find due to the inclusion of keywords, and had a higher Altmetrics score [7]. The Altmetrics score could be a more realistic reflection of the attention of research output and its impact on researchers (and their plans to conduct new trials) than journal impact. It shows that research may have more impact, if it is available, accessible and has a clear message. We conclude that the difference in adherence to the recommendation between exercise for LBP and CBT for chronic pain was not related to the impact factor of the journal in which the recommendation was published, but probably more related to the strength of the recommendation, the visibility (i.e. Altmetrics score) of the paper and probably also to the use of keywords.

Strengths and limitations

This study is the first study to investigate the impact of recommendations not to conduct further trials on the registration of new trials. It identifies specific factors related to the impact of the recommendations.

This study nevertheless has some limitations. It was difficult to search the WHO-ICTRP, which might be a result of the format of the search boxes in the trial register. The search parameters were restrictive making it difficult to conduct an extensive search, but this influenced both searches equally.

The wording of the recommendations differs between studies. As for exercise in LBP it was not really a “recommendation”, which might explain why it had less impact than a “clear” recommendation stated in the conclusion of a Cochrane systematic review.

In addition, we had trouble extracting data from the trial protocols, as protocols were often incomplete, and provided unclear descriptions of the intervention and control conditions. To make sure we did not miss any relevant protocols, we searched for the full text for final selection. This shows again the importance of a high-quality protocols addressing the study methodology in full, particularly a clear description of the intervention [8]. Studies of published trial reports showed that the poor description of interventions meant that 40–89% were non-replicable and hence increase research waste [9]. Therefore, authors are advised to follow a standardized format (e.g. SPIRIT statement), regularly update the trial protocol, and report all results. Almost half of the studies on LBP before 2012 published their results. This is in line with a recent European study that showed that half of all trials are non-compliant with reporting results to the EU Clinical trials register [10]. This is comparable to another study that found that half of all non-registered studies are not published [11]. In addition, the current study investigated only one domain of science-allied health interventions for chronic pain. We cannot be certain that the same pattern would emerge in other domains of science such as RCTs on medications. We do feel however that the recommendations below are applicable to all domains and research areas.

Recommendations

To reduce research waste and increase research impact, researchers should have a clear message to other researchers. In addition, one may consider a targeted media campaign to address policy makers, funding bodies and society. Researchers should be encouraged to thoroughly research the available, published and unpublished literature before designing a new trial [12]. When a meta-analysis of existing trials does not provide clear findings about whether an intervention has worthwhile effects, extended funnel plots can be used to explore the potential impact of a new trial on the updated meta-analysis and chance of changing clinical recommendations [6]. It is hoped that the use of living systematic reviews will help to inform researchers when a clinical question has been adequately addressed and further deter researchers from planning and conducting RCTS that do add to the overall existing evidence. We acknowledge that the current research landscape encourages and rewards early career researchers, including PhD candidates, who have conducted RCTs. An investigation into the rationale, not only for early career researchers but all researchers, for conducting RCTs would assist in understanding how to ensure only necessary RCTs are conducted. We encourage researchers and policymakers to actively spread and follow recommendations to increase their impact and to reduce research waste. Also funding bodies and ethical committees should take some responsibility for this, especially in a competitive research environment.