Evaluating the underreporting of patient-reported outcomes in carpal tunnel syndrome randomized controlled trials

Study design

Our design is a meta-epidemiological study analyzing data extracted from published RCTs concerning CTS. Our study was not subject to Institutional Review Board oversight because it did not meet the regulatory definition of a human participant study. We followed reporting guidelines for meta-epidemiological studies established by Murad and Wang [12].

Search strategy

One investigator (B.H.), in consultation with a medical research librarian, utilized the Ovid interface to search the Cochrane Central Register of Controlled Trials (CENTRAL), Embase, and MEDLINE for published RCTs about CTS. The Cochrane highly sensitive search strategy was utilized to maximize sensitivity for identifying RCTs, which is a validated filter for OVID interfaces [13]. Our protocol, including our search string, is available on Open Science Framework (OSF) [14].

Eligibility

Included RCTs were published between 2006 and 2020 and address CTS with at least one PRO measure as a primary or secondary outcome. Only studies published in the English language were included. Observational studies, animal studies, case reports, meta-analyses, systematic reviews, clinical trial protocols, cost-effective studies, secondary analysis, letters to the editor, and trials without a PRO measure were excluded.

Selection process

Following the literature search, collected data was combined and uploaded into Rayyan (https://rayyan.qcri.org/), a systematic review screening platform. Duplicates were removed, and two investigators independently performed title and abstract screening in a duplicated, masked fashion. Following the initial screening, investigators then resolved disagreements by discussion and, when necessary, through collaboration with a third investigator.

Data collection process

Investigators were trained to extract data with the CONSORT-PRO adaptation utilizing work published by the guideline authors [11, 15]. As a calibration exercise, we performed masked, duplicate extraction from three RCTs not included in our sample until consensus was achieved. The data extraction was performed utilizing a pilot-tested Google Forms form by two investigators, also in a masked, duplicate fashion. Investigators were trained to assess risk of bias (RoB) utilizing material provided by Cochrane [16]. The RoB evaluation was performed by two investigators in the same fashion. Following data extraction and the RoB evaluation, the investigators resolved all discrepancies; a third investigator (B.H.) was available to resolve disagreements.

Data items

We assessed completion of the CONSORT-PRO checklist adaptation developed by Mercieca-Bebber et al. [15], for all RCTs as our primary objective reported as the mean percent completion (see scoring of CONSORT-PRO adaptation). Our secondary objective assessed relationships between the mean completeness of PRO reporting and trial characteristics. The trial characteristics analyzed were: (1) year of publication (before or after 2014, a year following the publication of the CONSORT reporting guidelines); (2) intervention of RCT (e.g., drug or surgical technique); (3) conflict of interest statement; (4) journal endorsement of CONSORT-PRO; (5) citation of CONSORT-PRO within the publication; (6) whether an RCT utilized a PRO as a primary or secondary outcome; (7) RoB assessed by the Cochrane RoB 2.0 tool (see Evaluating RoB); (8) the length of PRO follow-up time; and (9) sample size of the trial.

The publishing journal’s endorsement of the CONSORT guidelines were coded as either not mentioned, recommended, or required. This data item was evaluated by screening the instructions to the authors’ pages for mention of EQUATOR, CONSORT, or CONSORT-PRO guidelines.

RoB was assessed by evaluating the following bias domains: (1) bias arising from the randomization process; (2) bias due to deviations from the intended interventions; (3) bias due to missing outcome data; (4) bias in measurement of the outcome; (5) bias in selection of the reported result; and (6) overall RoB appraisal for each RCT.

Scoring CONSORT-PRO

Scoring methodology has been adapted from Mercieca-Bebber et al. [15]. We omitted Item 4a of CONSORT-PRO (the use of PROs in eligibility or stratification) because there is difficulty in the verification of this criterion. Instead, we recorded whether a study described adherence to this item as a ‘yes’ or ‘no.’ We allocated a maximum value of 0.5 or 1 when information for an item was present. Items that received the maximum value (1 or 0.5 if the item was double-barreled) were considered ‘complete,’ whereas items that failed to reach the maximum value are reported as ‘not complete.’ Item P1b was scored as partially complete if an RCT reported the PRO measure utilized in the study but failed to identify whether the PRO was a primary or secondary outcome. Therefore, Item P1b could be scored as 0, 0.5, or 1 based on the information available. Item 7a was dependent on the PRO measure being reported as a primary outcome. Because of this dependency, RCTs with a primary PRO outcome had a maximum score of 15, whereas RCTs with a secondary outcome had a maximum score of 14. A percent completeness of the checklist per RCT was calculated by adding the items and dividing by the total of possible items.

Evaluating risk of bias

We utilized a decision algorithm developed by the Cochrane Collaboration to evaluate RoB. If investigators had partially divergent assessments on bias domains (e.g., one investigator answered ‘yes’ and another investigator answered ‘partial yes’), the overall RoB judgment for the trial outcome was not altered. We evaluated the overall RoB domain per the Microsoft Excel tool provided by Cochrane as ‘high,’ ‘some concerns,’ or ‘low’ risk [17], [18], [19]

Data analysis

To address our primary outcome, we calculated the mean completion percentage of the CONSORT-PRO adaptation across all RCTs in our sample and by PROs as a primary or secondary outcome. Next, we reported frequencies and percentages for the trial characteristics listed in the Data Items section. To address our secondary outcome, we utilized bivariate regression models to determine the association between mean completion percentage of the CONSORT-PRO adaptation and the trial characteristics listed in the Data Items section. Finally, we reported on the frequency and percentage of individual items on the CONSORT-PRO adaptation for all RCTs, and by PROs as a primary or secondary outcome within the RCT. All analyses were performed utilizing STATA 16.1 (StataCorp, LLC, College Station, TX).

Reproducibility

Our study protocol, data sheets, analysis scripts, data dictionaries, and extraction forms are publicly available on OSF as a means to promote the transparency and reproducibility of our study. This investigation was conducted in tandem with other studies addressing completeness of reporting in other fields of medicine utilizing similar methodology.

Systematic search and screening

Our systematic search returned 374 records, 243 of which remained after removing duplicates. Following title and abstract review, 39 studies were included for full-text review. After full-text evaluation, 31 published RCTs were included for data extraction. Rationales for the excluded studies are provided in Figure 1.

Figure 1:

Preferred reporting items for systematic reviews and meta-analyses (PRISMA) flow diagram.

RCT characteristics

Among the included RCTs, 61.3% (19/31; Table 1) were published prior to 2014. The most common intervention type was a combination of interventions (12/31, 38.7%). There were no conflicts of interest reported in 54.8% (17/31) of the RCTs, while 45.2% (14/31) did not include conflict of interest statements. Twenty (of 31, 60.6%) studies were published in journals that recommended or required CONSORT guidelines, and 20 (of 31, 64.5%) endorsed CONSORT guidelines within the article. A total of 26 RCTs included a PRO as a primary outcome, and five RCTs included PROs as secondary outcomes. The percentage of RCTs in each overall RoB categories are as follows: 35.5% (11/31) were ‘high’ risk, 58.1% (18/31) were evaluated as ‘some concerns,’ and 6.5% (2/31) were ‘low’ risk. Nineteen (of 31, 61.3%) RCTs had PRO follow-up times of 3 months or less. There were no RCTs that performed follow-up appraisals later than 12 months. The mean sample size of the RCTs in our sample was 83.7 (SD=47.6).

Completeness of reporting according to CONSORT-PRO adaptation

The overall mean completion percent for the CONSORT-PRO checklist adaptation among our sample was 40.9% (SD=13.5). RCTs that listed PRO as a primary outcome had a mean completeness of 38.3% (SD=7.07) and was not significantly different from RCTs with a PRO as a secondary outcome (41.2%; SD=14.1) (t₁=−0.28, p=0.79).

Among all RCTs in our sample, item 17aii results include an estimate of precision––was the most consistently reported item overall (28/31; 90.3%; Table 2). Conversely, item P2bii––PRO domains specified in the hypothesis—was reported least often (1/31; 3.2%), which was also the lowest reported item among primary studies (1/31; 3.9%). For RCTs with PROs as a primary outcome, item 17aii was also the most consistently reported item (23/26, 88.5%).

RCTs that included a PRO as a secondary outcome consistently reported the following: results include an estimate of precision (17aii), study limitations (P20), and implications of PRO results for generalizability, clinical practice (P21). No RCTs with a secondary PRO outcome reported the following items: rationale for PRO outcome (2a), and PRO hypothesis is present and domains specified in the hypothesis (P2bi and P2bii).

Associations between PRO outcomes, completion, and study characteristics

Lastly, our bivariate regression analyses revealed that RCTs reporting no conflicts of interest among authors were 13.5% (SE=6.39) more complete than RCTs that did not include a conflict of interest statement (t₁=2.11; p=0.043). Furthermore, RCTs that were published in a journal that endorsed CONSORT reporting guidelines were 23.5% (SE=6.68) more complete compared to RCTs published in journals that did not mention reporting guidelines (t₁=3.51; p=0.002). Additional relationships are demonstrated in Table 1.

Study limitations

Our study has several strengths. First, to facilitate reproducibility and transparency, we uploaded our protocol, data sheets, and data extraction forms to OSF, a free and open source archive. In addition, investigators completed a training course for either CONSORT-PRO and the Cochrane Collaboration RoB 2.0 tool to standardize data extraction. Lastly, data were extracted in duplicate by two independent, masked researchers. This practice is in accordance with Cochrane Collaboration guidelines and is considered the gold standard for data extraction [32], further heightening the validity of this study.

Our study is not without limitations, however. We extracted all CTS trials examining PROs as either a primary or secondary outcome between a specified date range, which led to a reduced sample size. Additionally, our study is cross-sectional by design, and thus, these findings are not generalizable to other neurological or orthopedic topics. Finally, we utilized three bibliographic databases, MEDLINE, Embase, and CENTRAL to locate RCTs containing PROs. However, these are not all-encompassing of available trials.