Are randomized controlled trials in urology being conducted with justification?

Oversight, transparency, and reproducibility

This study did not involve humans and, thus, did not meet the regulatory definition for human subjects research per the U.S. Code of Federal Regulations and was not subject to institutional review board oversight. The methods for this study were adapted from previously published works [11], [12], [13], [14]. No protocol was developed prior to conducting this study.

Eligibility criteria

For an article to be included in our study, it must have met the following criteria: (1) a full length RCT (published abstracts and posters were excluded); (2) published between November 30, 2014 and November 30, 2019 in one of the following journals: European Urology, BJU international, The Journal of Urology, and Urology; (3) only contain human subjects; and (4) accessible in English.

Search strategy

On December 13, 2019, one member of our research team (B.J.) conducted a PubMed search for RCTs published in the top four urology journals according to the Google Scholar h5 index. The Google Scholar h5 index is based on metrics reported in Google Scholar which generates a Hirsch index (h index) score for journals using an algorithm based on article citations for articles published within the prior five complete years [15], [16]. We used the Google Scholar h5 index as it reflects a journal’s research productivity and impact of all publications over the prior five years; using the h5 index allowed us to identify the journals that have been consistently publishing impactful research in the field of urology. Among these journals were European Urology, BJU International, The Journal of Urology, and Urology. We included trials and protocols published between November 30, 2014 and November 30, 2019 from the following search string: (((((“randomized controlled trial” [Publication Type] OR “randomized controlled trials as topic” [MeSH Terms] OR “randomized controlled trial” [All Fields] OR “randomized controlled trial” [All Fields]) AND “0302-2838” [Journal]) OR “0022-5347” [Journal]) OR “1464-410X” [Journal]) OR “0090-4295” [Journal]) OR “1078-1439” [Journal] AND (Clinical Trial[ptyp] AND (“2014/11/30” [PDAT]: “2019/11/30” [PDAT])). Journals were searched by International Standard Serial Number rather than title.

Training

Two of the investigators (S.S., A.W.) underwent training by two senior investigators (M.V., B.J.) who have published previous research in this subject area prior to article screening and data extraction. Both trained authors (S.S., A.W.) were instructed on the screening platform Rayyan (Rayyan Systems, Inc.) and a pilot tested Google Form (Google, Inc.) for extracting data which was adapted from previous studies [12], [14].

Screening

After receiving the search string results, the two trained authors (S.S., A.W.) screened all identified articles in a masked and duplicate fashion using the Rayyan screening tool to ensure they were RCTs. Upon completion of title and abstract screening, the first of two resolution meetings was held to finalize the list of included articles for full text screening and data extraction by agreement between both investigators (S.S., A.W.); the two training authors (B.J., M.V.) were available for third party adjudication.

Data extraction

Data from RCTs selected for inclusion were then extracted by two authors (S.S., A.W.) in a masked and duplicated fashion using the pilot tested Google Form (Google, Inc.). All data extraction was recorded and analyzed through Google Sheets (Google, Inc.). The following data points were extracted from each included RCT: manuscript title, year of publication, journal title, type of interventions categorized as pharmaceutical, medical device, procedure, and “other” (which consisted of education, psychosocial intervention, exercise programs, and online therapy including behavioral and psychosocial counseling as well as group therapies), funding source (government, hospital/university, industry, mixed) type of trial (parallel groups, crossover, cluster), and the total number of participants reported in each RCT. The number of participants was categorized into tiers based on quartiles. The number of SRs or metaanalyses cited in each RCT’s Introduction, Methods, or Discussion section was recorded, as was information about whether the RCT used these citations to justify the trial. Justification of trial was recorded as “yes – verbatim,” “yes – inferred,” or “not used as justification.” The rationale for each classification was recorded for the data extraction resolution meeting. Upon completion of data extraction, a second resolution meeting was held to reconcile the data that had been extracted; if any disagreements between the two authors responsible for data extraction (S.S., A.W.) were not able to be resolved, the training authors (B.J., M.V.) were available for third party adjudication.

Statistical analysis

We calculated frequencies and percentages of study characteristics, including journal, funding source, intervention, and participant tiers. Next, we tallied the number and percentage of studies that used an SR as a justification for the study and where the justification was located within the manuscript. Last, we used chi square tests to determine associations between an RCT’s use of a SR for justification (or lack thereof) and study characteristics including journal of publication, funding source, intervention, and participant tiers. We conducted a post hoc analysis consisting of two chi square tests: one to determine whether there was an association between journal of publication and presence of SR citation, regardless of whether it was used for justification, and another to determine whether there was an association between journal of publication and the location of SR citation within the body of the manuscript. Data analysis was conducted using Stata 16.1 (StataCorp, College Station, TX, USA).

General characteristics

Our search retrieved 561 bibliographic records indexed as RCTs from PubMed. After screening, 276 (49.2%) were included (Figure 1). The two most common reasons for a study to be excluded were nonrandomized methods (123; 21.9%) and secondary analysis (81; 14.4%; Figure 1). Most included RCTs were parallel group trials (262; 94.9%), followed by crossover trials (12; 4.3%) and cluster trials (two; 0.8%) The most commonly studied interventions were pharmaceutical therapies (118; 42.8%), followed by procedures (93; 33.7%), medical devices (34; 12.3%) and other (31; 11.2%). The most common funding source was industry (101; 36.6%), followed by mixed funding (37; 13.4%) and government (29; 10.5%). A large number of trials did not report their funding source (80; 28.9%). The journal with the highest number of RCTs included in our sample was The Journal of Urology with 82 (29.7%).

Figure 1:

Preferred reporting items for systematic reviews and meta-analyses (PRISMA) flow diagram demonstrating exclusion criteria for randomized controlled trials examined in this study.

SR citations were most common in the Discussion section (205 of 403 total SR citations; 50.8%); 15 of 403 (3.7%) SR citations were found in the Methods section of included trials (Table 1). A total of 241 SRs were cited for verbatim trial justification; 87 (36.1%) of those citations appeared in Journal of European Urology, 53 (22.0%) appeared in both BJU International and Urology, and 48 (19.9%) of the SRs cited for verbatim trial justification appeared in The Journal of Urology (Table 2). Of the 107 RCTs without any SR citations, 33 (30.8%) were published in The Journal of Urology (Table 1).

Factors associated with SR justification in published urology trials

Of the 276 included RCTs, 150 (54.3%) cited an SR as either verbatim (108;39.1%), or inferred (42;5.2%) trial justification. The remaining 126 (45.7%) RCTs did not cite an SR as justification. Of those 126 RCTs, 19 (15.1%) cited an SR but did not explicitly mention the SR in the context of justifying the trial. Of the 126 studies without RCT justification, 107 (84.9%) did not cite an SR anywhere in the manuscript. Table 2 shows displays associations by justification and characterized by journal, funding source, intervention, and corresponding participant tiers. There was no significant association between SR justification and journal of publication (X²=6.57, p=0.68), with 33 of 72 (45.8%) RCTs published in European Urology having verbatim SR justification, 29 of 82 (35.4%) in The Journal of Urology, compared with 23 of 66 (34.9%) in Urology, and 22 of 56 (39.3%) in BJU International having verbatim SR justification. No significant associations were found between funding sources and SR justification, either; 38 of 101 (37.6%) industry funded RCTs were found to have verbatim SR citation justification compared with 38 of 101 (37.6%) industry funded RCTs with no SR citation at all. There was a significant association between verbatim justification and type of intervention (X²=20.23, p=0.017); 18 of 31 (58.1%) RCTs that described “other” interventions (i.e. education, psychosocial intervention, exercise programs, and online therapy including behavioral and psychosocial counseling, and group therapies) cited an SR as verbatim justification for the trial, but only 39 of 118 (33.1%) of pharmaceutical RCTs had SR verbatim justification. Our post hoc analysis to assess the association between journal of publication and presence of SR citation (regardless of whether it was used for justification) showed no statistically significant association(X²=0.76, p=0.86). The second post hoc analysis also showed no statistically significant association between journal of publication and the location of SR citation within the body of the manuscript (X²=5.98, p=0.43).

Importance of using SRs for RCT justification

Funding from the National Institutes of Health for urologic diseases increased from $541 million in 2018 to $587 million in 2020 [20]. Ensuring efficient research production can be accomplished by minimizing research waste, which is often preventable [5]. The use of SRs to develop, refine, and justify research questions when designing RCTs will serve to prevent research redundancy, avoid research waste, and ensure that research aims add value to the current literature base [6]. The inconsistent use of SRs for justifying urology RCTs may be related to the poor methodological quality known to plague urology RCTs [21]. Methodological research has provided the tools necessary to identify discrepancies in RCT methodology and RCT quality [22], [23]. As such, the relatively new high quality research practices informed by methodological research are not yet ubiquitous [24]. Therefore, educating students, researchers, and physicians on the importance of using SRs when formulating RCT research questions may lead to higher quality research in the field of urology.

Education on research methodology is particularly important for osteopathic medical students and residents, as osteopathic medical education has historically placed a low priority on fostering a research culture [25], [26], [27]. However, efforts are underway to positively change the culture of osteopathic medicine to include a heavier emphasis on research. Poor quality research practices have long been a topic of discussion that is not limited to osteopathic physicians or trainees; D.G. Altman, an English statistician known for making great strides in medical research quality and reporting, posited that “We need less research, better research, and research done for the right reasons.” [28] Our study emphasizes the need for academic medical researchers in the field of urology to use SRs more frequently in RCT design. Informing osteopathic professionals on the poor rates of SR use in urology RCTs could serve to enhance research quality in the field of urology and across subspecialties.

Clinicians in the field of urology must regularly appraise findings from RCTs and consider their implications to effectively integrate evidence based conclusions into daily practice. Translating research findings into practice may be hindered by unclear study aims, which can be resolved by reporting research in the context of previous findings [6]. Our results suggest that RCTs in the field of urology often fail to present aims within the context of the previous literature like SRs. Moreover, SR citation patterns were not strongly correlated with many of the other factors we analyzed, making their use seem rather random and potentially making the purpose and significance of a study’s findings less clear to physicians. Therefore, clinicians and physician–scientists participating in urology research can help reduce low rates of RCT justification by including context in manuscripts, ideally by citing SRs when available.

Placing our data in the context of existing evidence suggests that more work is needed to understand the full utility of SRs, which are not solely for the synthesis of sound evidence, but may also serve as drivers for the development of novel research questions. While trialists, researchers, and healthcare professionals may be aware of the utility of SRs for guiding clinical decision making, promoting the use of SRs for justifying RCTs is needed. Potential solutions include (1) funding agencies requiring statements from researchers attesting that they have consulted the literature for available SRs when writing grant applications; (2) researchers incorporating guidance into the Consolidated Standards of Reporting Trials (CONSORT) Statement [9] – the widely adopted reporting guideline for clinical trials – and within its Explanation and Elaboration document [9]; and (3) journals adopting procedures requiring authors of clinical trials to include a clear summary of previous research findings with an explanation of how their trial’s findings apply to previous literature, as journals such as The Lancet [29] have done. Bridging newly synthesized evidence with existing evidence is encouraged, as happens when SRs and metaanalyses are properly cited in the context of new research.

Limitations

Our study has both strengths and limitations. We used a duplicate and blinded data extraction process, as recommended by the Cochrane Handbook for Systematic Reviews of Interventions. [30] The methodology, though adapted, has been previously validated. Future methodological research is warranted to compare our current findings with results from future RCTs to determine any change in the proportion of urology trials citing SRs for justifying their conduct. Regarding study limitations, we conducted a cross sectional investigation that was limited to only a few journals across a limited time period, and therefore, our results are only applicable to our sample. Our results should be interpreted descriptively rather than inferentially, and our results are not generalizable to other fields of medicine. Furthermore, we acknowledge that SRs are not available for every topic; therefore, clinical trialists may have searched for existing SRs and, having found none, omitted reporting this information in the final publication. Thus, our results may underreport the number of trials that attempted to justify their RCT with an SR. Also, researchers seeking to confirm the results of other clinical trials may abstain from performing a literature search, as the findings may not alter their decision to run a confirmatory trial. A balance between confirmatory trials and exploratory trials is imperative for ensuring the reliability of trial results, especially when used in clinical decision making [31]. Therefore, the use of SRs for trial justification may not be applicable for confirmatory trials.