Reliability and measurement error of exercise-induced hypoalgesia in pain-free adults and adults with musculoskeletal pain: A systematic review

2.1 Eligibility criteria

We used the framework provided by the COSMIN initiative [33] to determine the following eligibility criteria:

• Population: Pain-free adults or adults with MSK pain (acute, chronic, and any MSK location) of any age, gender, and ethnicity. We excluded studies involving individuals younger than 18 years of age, animal models, or studies including adult individuals with medical conditions different from MSK pain or with “secondary” MSK pain (e.g., associated with Parkinson’s disease, due to persistent inflammation or systemic diseases, among others).

• Construct of interest: EIH. It refers to an acute reduction in experimental pain (i.e., pain elicited by experimental means) after a single bout of exercise [1].

• Way of operationalization of the measurement protocol: Studies that used a single bout of exercise (providing mode, intensity, and duration) combined with an experimental pain induction procedure (pain threshold, pain tolerance, or pain ratings of experimentally induced painful stimuli) measured within 2 h upon exercise cessation – we chose this time-frame, also used by Bonello et al. [34], to target only studies on the effects of acute exercise while excluding those that considered longer lasting effects of exercise. We excluded studies that did not adequately characterize their exercise intervention in terms of mode, intensity, and duration (unless the authors provided the missing information) and studies that assessed pain ratings of clinical pain (i.e., patients with MSK pain reporting a change in their usual pain level after exercise), used repeated exercise sessions in between measurements, did not include a pain-related measure within 2 h upon exercise cessation or combined exercise with another intervention (e.g., manual therapy, neuromuscular electrical stimulation, medication, or supplementation).

• Language: No limits.

• Study type: Due to the nature of our research question, we targeted both observational studies and experimental studies – parallel or cross-over design – from any year. Both phases of the cross-over studies were considered. We excluded narrative reviews, SRs, meta-analyses, commentaries, book chapters, and conference reports. However, the references for these studies were searched.

• Measurement properties of interest: Reliability and measurement error of EIH. Details of these metrics are provided below.

2.2 Information sources and search strategy

The search strategy was designed and conducted by the principal investigator (VA). Team members provided feedback in terms of key words and search strings (peer-reviewed by SA and DS). Five databases were searched using both controlled vocabulary specific to each database, natural language terms related to the different components of the research question and a validated search filter for finding studies on measurement properties designed by Terwee et al. [35]. The details of the search strings for each database can be found in Supplementary material: 3. Search strategy. The following databases were searched initially on November 11, 2021 and updated on February 15 and 16, 2024:

• PUBMED from 1978 to February 2024,

• EMBASE from 1974 to February 2024,

• SCOPUS from 1966 to February 2024,

• PSYCINFO from 1932 to February 2024,

• CINAHL Complete from 1983 to February 2024.

No limits were applied. In total, 4,762 records were exported to a reference manager software (Endnote 20; endnote.com) and were imported, after duplicates removal, into the Covidence software (covidence.org). After completion of the study selection process, VA conducted manual searches examining the grey literature (March 27, 2022, September 04, 2023, and February 20, 2024) and the reference lists of the included studies (March 18, 2022).

2.3 Selection process

Two independent reviewers (VA and DS) screened – after a pilot phase on 25 references and using SR software (covidence.org) – first, the titles and abstracts of the studies considered for inclusion (first screening round), and then the full text of the studies (second screening round) using the criteria set up above. After each screening round, they discussed their decisions until they reached a consensus. A third reviewer (SA) settled any disagreement.

2.4 Data collection process

The principal investigator (VA) collected – after a pilot phase on three studies with the study team – data from the included studies using a standardized extraction form (Supplementary material: 4. Data extraction form). The data collected were reviewed and discussed with the study team until a consensus was reached. SA settled any disagreement. When information was missing, VA contacted the authors by email. The details of this correspondence can be found in Supplementary material: 5. Correspondence with the authors.

2.5 Data items

We collected data related to the following categories (the full list of the collected variables and outcomes is in Supplementary material: 4. Data extraction form):

• Report information (e.g., authors, year of publication, journal),

• Study information (e.g., study objectives, study question, study design, target population, sample size, eligibility criteria),

• Information regarding exercise (e.g., mode, intensity, duration),

• Test modalities information (i.e., how the authors assessed experimental pain. e.g., stimulus type, test modality, site of test, modulation by exercise),

• EIH determination methods (e.g., scoring of EIH, statistical analysis used, data processing and storage [i.e., “All actions undertaken on the raw data to store it in a usable (electronic) form for later data manipulation”[33]]),

• Information regarding reliability or measurement error (e.g., type of reliability, repeated component of the procedures, reliability parameters),

• Information regarding the Risk of Bias (RoB) assessment,

• Outcomes

1. Related to reliability and measurement error: intraclass correlation coefficient (ICC), Pearson’s r, Cohen’s kappa, standard error of measurement (SEM).

2. Related to EIH: report of EIH, baseline values for each test modality and location, absolute and relative change values, and numbers of “EIH non-responders/responders” per session.

We defined reliability and measurement error, according to the COSMIN initiative [32], as “The proportion of the total variance in the measurements which is because of ‘true’ differences among patients” and “the systematic and random error of a patient’s score that is not attributed to true changes in the construct to be measured,” respectively. In this definition, “true change” refers to “the average score that would be obtained if the [measurements] were given an infinite number of times” [36]. We collected all the reliability parameters used by the authors of the included studies, but we only reported those that the COSMIN initiative [33] recommends for reliability or measurement error – in the case where ICC and Pearson’s r were calculated by the authors, we only reported the ICC. When studies used control and experimental conditions, we collected data related to EIH (e.g., baseline PPT) only from the experimental conditions. A meta-analysis regarding the effect size of EIH was not the aim of this review, and these data were deemed unnecessary.

2.6 Study RoB assessment

We assessed the RoB of the included studies with the COSMIN RoB tool [33], which is recognized as the best current tool to assess the RoB of studies investigating the reliability and measurement error of an outcome measurement instrument. The COSMIN RoB Tool uses a four-point system – “very good,” “adequate,” “doubtful,” or “inadequate” – to rate six common standards for reliability and measurement error – five on design requirements and one on “other flaws” – and additional standards for preferred statistical methods: three for reliability and two for measurement error (Supplementary material: 6. COSMIN RoB tool). We summarized the RoB for each standard and determined the overall assessment for each study, as recommended by the COSMIN initiative, with the worst-score-counts method: the lowest rating of all standards determined the study RoB. When a study involved multiple test modalities (e.g., PPT and cuff pain tolerance), the RoB was assessed for each of them – we considered that RoB could depend on how each test modality was applied and that within-studies variations were possible. Two independent reviewers (VA and DS) assessed each study and discussed their results until they reached a consensus. A third reviewer (SA) settled any disagreement. We created plots (using R package “robvis” [37]) to report the RoB assessments.

2.7 Data synthesis

We created evidence tables (using R packages “flextable” [38] and “officedown” [39]) to describe the study characteristics, the way of operationalization of EIH, the elements of reliability and measurement error of each individual study, and the outcomes. When studies reported the SEM, we computed the Smallest Detectable Change with a 95% confidence interval (SDC 95; it is a change score that will be observed in only 5% of the individuals that are “truly unchanged” [40]) as SEM*√2*1.96 [41]. When relevant, we organized the data according to the test modality used and its site of application, which we divided into two categories: local (i.e., the stimulus was applied to a limb involved in the exercise) and remote (i.e., the stimulus was applied to a limb not, or less, involved in the exercise). We also created plots (using R package “ggplot2”[42]) to display graphically the values of ICC, SEM, and SDC 95.

We rated the results of the statistics used in each study to assess reliability and measurement error as “+,” “−,” or “?,” respectively “sufficient,” “insufficient,” and “indeterminate,” according to the criteria for good reliability and measurement error from the COSMIN initiative [33] (Supplementary material: 7. COSMIN criteria for good reliability).

We assessed inconsistency in the results (i.e., ratings of “ +” or “−” account for less than 75% of all ratings) and compared the designs (e.g., way of operationalizing EIH), populations, and statistical computations of the included studies. Although one of our objectives was to pool reliability parameters from included studies, this was not possible because EIH protocols and exercise parameters greatly differed in each study.

Two independent reviewers (VA and DS) graded the level of evidence of each individual result as “high,” “moderate,” “low,” or “very low” with the modified Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach published by the COSMIN initiative [32]. A third reviewer (SA) settled any disagreement. GRADE assessments (details about the criteria used for GRADE are provided Supplementary material: 8. COSMIN modified GRADE approach) were based on RoB in included studies, inconsistency in the ratings of the results of the included studies, imprecision (based on the total sample size of the summarized studies), and indirectness (i.e., the population of the summarized studies differed from the target population of this review). We reported the assessment of each outcome in an evidence table.

3.1 Study selection

We identified 4,762 records (i.e., the title and/or abstract of a report, which supplies information about a study [43]) through database searching. After duplicate removal, 2,443 records remained, and 2,383 were excluded after screening titles and abstracts. We reviewed 58 full reports; two potentially eligible reports [44,45] were excluded as only their conference report could be found. The details of the reasons for exclusion after reading the full texts can be found in Supplementary material: 9. Reasons for exclusion of reports. We finally included five studies [4,5,20,23,46]. Then, we searched the references of the included studies and the grey literature, but no new eligible article was found. Figure 1 displays the PRISMA flow diagram of the study selection process.

Figure 1

PRISMA flow diagram of included studies (EIH, exercise-induced hypoalgesia).

3.2 Study characteristics

The characteristics of the included studies are described in Tables 1–4.

3.3 RoB in studies

The RoB assessments from studies investigating the reliability and measurement error of EIH are summarized in Figure 2. In all studies [4,5,20,23,46], we considered that subjects were stable between the sessions as only healthy participants were included; thus, we rated R1 and M1 – related to the clinical stability of the population – “very good.” The time interval between sessions ranged from 1 (all studies [4,5,20,23,46]) to 3 weeks (one study [4]); we considered that domains R2 and M2 were “very good.” Four studies [4,20,23,46] stated that measurement sessions were identical – the domains R3 and M3 were judged “very good”; in one study [5], we considered that the authors could have provided more evidence that sessions were identical and we deemed the domains R3 and M3 “adequate.” We considered all the studies [4,5,20,23,46] to be “doubtful” regarding the domains R4, R5, M4, and M5 because the assessor was not blinded [4,5,20,23,46] to the scores of the participants from the previous session when s/he administered the measurement and attributed a score. We considered the domains R6 and M6 as “doubtful” for the following reasons: lack of control of factors potentially affecting exercise performance, exercise metabolism, and/or nociceptive processing, such as menstrual cycle phase[47,48,49] (all studies [4,5,20,23,46]), pre-exercise nutritional status [50,51] (all studies [4,5,20,23,46]), and consumption of other substances than analgesics (four studies) [5,20,23,46]; lack of control of factors known to affect EIH such as subjects expectations/beliefs [52,53] (all studies [4,5,20,23,46]). We judged the domain R7 as follows: “very good” (one study [4]); the authors used ICC 3.1 agreement to assess the test–retest reliability of EIH as a continuous score; “adequate” (four studies [5,20,23,46]), ICC 3.1 consistency was used to assess test–retest reliability and the authors provided evidence that no systematic errors occurred (similar EIH display in all sessions, no or minor differences between exercise parameters across sessions); “doubtful” (two studies [5,20]), ICC 3.1 consistency was used to assess test–retest reliability but differences across sessions in the magnitude of EIH were observed – reflecting a lack of agreement between sessions, potentially arising from bias. Domain R9 was considered “very good” in all studies [4,5,20,23,46] – unweighted kappa was used to measure the reliability of EIH as a dichotomous variable. The domain M7 was judged as follows: “adequate” in two studies [4,23], SEM consistency was used to determine measurement error and the authors provided evidence that no systematic errors occurred; “doubtful” in one study [20], SEM consistency was used and the manifestation of EIH varied across sessions. The overall RoB was deemed “doubtful” for all studies [4,5,20,23,46].

Figure 2

RoB in studies investigating the (a) reliability and (b) measurement error of EIH. Domains: R1/M1, Were patients stable in the time between the repeated measurements on the construct to be measured? R2/M2, Was the time interval between the repeated measurements appropriate? R3/M3, Were the measurement conditions similar for the repeated measurements-except for the condition being evaluated as a source of variation? R4M4, Did the professional(s) administer the measurement without knowledge of scores or values of other repeated measurement(s) in the same patients? R5/M5, Did the professional(s) assign scores or determine values without knowledge of the scores or values of other repeated measurement(s) in the same patients? R6/M7, Were there any other important flaws in the design or statistical methods of the study?; R7, For continuous scores: was an intraclass correlation coefficient (IC) calculated? R8, For ordinal scores: was a (weighted) kappa calculated? R9, For dichotomous/nominal scores: was kappa calculated for each category against the other categories combined? M7, For continuous scores: was the SEM, SDC, limits of agreement or coefficient of variation calculated? M8, For dichotomous/nominal/ordinal scores: Was the percentage specific (e.g. positive and negative) agreement calculated? (cPPT, cuff pressure pain threshold; PPT, pressure pain threshold; cPTT, cuff pressure pain tolerance).

3.4 Results of individual studies

3.4.1 Reliability and measurement error

Figures 3 and 4, respectively, display the results of reliability and measurement error from the included studies for EIH considered as a continuous score. ICCs for EIH computed as relative and absolute change ranged from 0 to 0.61; kappa for the reliability of EIH considered as a dichotomous score ranged from 0.01 to 0.46; SEM of EIH computed as absolute and relative change, respectively, ranged from 30.1 to 105 kPa and from 10.4 to 21%; SDC 95 of EIH computed as absolute and relative change, respectively, ranged from 83.54 to 291.05 kPa and from 28.83 to 58.21%. Table 5 summarizes this information and the reliability of EIH computed as a dichotomous score. Based on the COSMIN criteria for good reliability and measurement error, we rated all the ICCs and kappa values as “insufficient” (“−”; ICCs and kappas < 0.7) and the SDC 95 values as “indeterminate” (“?”; minimal important change [MIC] is not defined).

Figure 3

Reliability of EIH. EIH was computed as (a) absolute change and (b) relative change. The ICCs determined in the included studies are reported with their 95% confidence intervals. They vary according to the site of the test and the test modality applied, which was further categorized as local (i.e., the stimulus was applied to a limb involved in the exercise) and remote (i.e., the stimulus was applied to a limb not, or less, involved in the exercise). All values are below the COSMIN criteria for good reliability (ICC > 0.7). (cPPT, cuff pressure pain threshold; PPT, pressure pain threshold; cPTT, cuff pressure pain tolerance).

Figure 4

Measurement error of EIH. EIH was computed as (a and b) absolute change and (c and d) relative change. The SEMs of (a) absolute change (SEM abs) and (c) relative change (SEM rel) determined in the included studies are reported. We computed the SDC 95% of (b) absolute change (SDC 95 abs) and (d) relative change (SDC 95 rel). They vary according to the site of the test and the test modality applied, which was further categorized as local (i.e., the stimulus was applied to a limb involved in the exercise) and remote (i.e., the stimulus was applied to a limb not, or less, involved in the exercise). To compare the SEM abs and SDC 95 bs across studies, we divided them by the mean of the mean baseline (mean b) values across sessions and multiplied by 100. (cPPT, cuff pressure pain threshold; PPT, pressure pain threshold; cPTT, cuff pressure pain tolerance).

Table 5

Results of individual studies, ratings, and GRADE

Author (Year)	Test modality			Reliability							Measurement error
				EIH continuous				EIH dichomotous		Grade	Absolute change (kPa)			Relative change (%)			Grade
	Type (units)	Site	Category	ICC absolute change [95% CI]	Rating	ICC relative change [95% CI]	Rating	Kappa [95% CI]	Rating		SEM	SDC 95%	Rating	SEM	SDC 95%	Rating
Gomolka et al. (2019) [4]	PPT (kPa)	Biceps femoris	Local	0.54 [0.22,0.75]	–	0.27 [−0.1,0.58]	–	0.28 [NR,NR]	–	Very low	65.23	180.81	?	15.44	42.80	?	Very low
	PPT (kPa)	Lower back	Remote	0.4 [0.03,0.67]	–	0.33 [−0.01,0.6]	–	0.27 [NR,NR]	–	Very low	64.54	178.90	?	14.63	40.55	?	Very low
	PPT (kPa)	Hand	Remote	0.32 [−0.02,0.6]	–	0.33 [−0.02,0.61]	–	0.32 [NR,NR]	–	Very low	30.14	83.54	?	13.19	36.56	?	Very low
Hviid et al. (2019) [5]	PPT (kPa)	Quadriceps	Local	0.29 [−0.41,0.64]	–	NR	?	0.07 [−0.24,0.37]	–	Very low	NA	NA	NA	NA	NA	NA	NA
	PPT (kPa)	Upper trapezius	Remote	0 [−0.29,0.01]	–	NR	?	0.11 [−0.24,0.47]	–	Very low	NA	NA	NA	NA	NA	NA	NA
	cPPT (kPa)	Calf	Local	0.6 [0.21,0.8]	–	NR	?	0.34 [0.03,0.65]	–	Very low	NA	NA	NA	NA	NA	NA	NA
	cPTT (kPa)	Calf	Local	0.61 [0.2,0.8]	–	NR	?	0.19 [−0.15,0.53]	–	Very low	NA	NA	NA	NA	NA	NA	NA
Vaegter et al. (2018) [20,23]	PPT (kPa)	Quadriceps	Local	0.45 [−0.1,0.73]	–	0.4 [−0.1,0.7]	–	0.24 [−0.08,0.56]	–	Very low	86.00	238.38	?	14.00	38.81	?	Very low
	PPT (kPa)	Upper trapezius	Remote	0.46 [−0.1,0.73]	–	0.27 [−0.4,0.64]	–	0.01 [−0.33,0.34]	–	Very low	65.00	180.17	?	21.00	58.21	?	Very low
Vaegter (2019) [46]	PPT (kPa)	Quadriceps	Local	0.45 [0.1,0.72]	–	NR	?	0.46 [0.17,0.75]	–	Very low	NA	NA	NA	NA	NA	NA	NA
	PPT (kPa)	Upper trapezius	Remote	0.57 [0.1,0.79]	–	NR	?	0.43 [0.11,0.74]	–	Very low	NA	NA	NA	NA	NA	NA	NA
Vaegter (2019) [23]	PPT (kPa)	Quadriceps	Local	0.08 [−0.71,0.46]	–	0.33 [−0.33,0.66]	–	0.08 [−0.25,0.65]	–	Very low	105.00	291.05	?	15.00	41.58	?	Very low
	PPT (kPa)	Upper trapezius	Remote	0.03 [−0.9,0.51]	–	0.43 [−0.13,0.71]	–	0.13 [−0.2,0.46]	–	Very low	53.00	146.91	?	10.40	28.83	?	Very low

NA, not applicable; NR, not reported; EIH, exercise-induced hypoalgesia; ICC, intraclass correlation coefficient; SEM, standard error of measurement; kPa, kilopascal; CI, confidence interval; RoB, Risk of Bias; SDC 95, smallest detectable change with a 95% confidence interval; PPT, pressure pain threshold; cPPT, cuff pressure pain threshold; cPTT, cuff pressure pain tolerance. Rating: rating of the reliability value according to the COSMIN criteria: “–,” insufficient; “?,” indeterminate. Grade: level of the quality of the evidence according to the modified GRADE approach from the COSMIN initiative.

The bold letters refer to the rating of reliability and measurement error according to COSMIN.

4.1 Main results

To our knowledge, this is the first SR on the reliability and measurement error of EIH. We found five studies [4,5,20,23,46] conducted in pain-free human adults but none in adults with MSK pain. Our results suggest, with a very low level of certainty, that the between sessions test–retest reliability and the measurement error of EIH are, respectively, “insufficient” (i.e., ICCs and kappas < 0.7) and “indeterminate” (unknown MIC) in pain-free human adults.

4.2 General interpretation of the results

4.3 Limitations of the evidence included in the review

The evidence included [4,5,20,23,46] in this SR came from a single research group. It is mainly limited by its overall RoB, which was deemed “doubtful” in all studies. Our main concerns were the lack of blindness of the assessor, the lack of control of potential sources of error, and the statistical methods.

4.4 Limitations of the review process

This review has used a comprehensive search strategy and an in-depth data collection process. Because of the heterogeneity in the methods of the included studies (i.e., clinical heterogeneity), we could not quantitatively pool their results. Also, because we only included five studies, we could not provide a statistical or graphical (e.g., funnel plot) assessment of publication bias – we would have lacked statistical power [63]. However, we think that our results are not at high risk of publication bias: our search strategy was not limited to electronic databases; we carefully searched references and grey literature [64], and the reliability estimates of all the included studies were below the criteria for good reliability of the COSMIN initiative [33], whereas publication bias tends to be related to “positive” results [65].

4.5 Implications for future research

We have provided evidence that EIH might be unreliable, which can weaken the conclusions of the numerous studies on the topic [1]. Despite their limitations, our results demonstrate that the EIH literature must be interpreted with caution, highlighting a knowledge gap regarding EIH reliability (especially in MSK pain sufferers), and can be used to provide recommendations for further research in this field. We suggest that future studies prioritize investigating the reliability and measurement error of EIH in patients with MSK pain and the MIC of EIH in these populations. Furthermore, we recommend that future studies tailor the intensity of the exercise conditions, explore other means of experimental pain induction than PPT, blind the assessors/raters to the scores of the participants, and consider the participants’ beliefs and expectations regarding pain and exercise. Also, additional studies are needed to shed light on the sources of variations of EIH and its underlying mechanism(s). For reliability studies, we consider that two-way random-effects ICC models with formula for agreement should be used by researchers – it would allow them to generalize their results to randomly selected raters (and thus to other research teams) and prevent an overestimation of reliability in the presence of bias. Finally, when more studies on the reliability of EIH using a homogenous methodology are published, future SRs on the topic could consider a metanalysis by subgrouping studies according to the exercise type, exercise intensity, and testing site to improve the precision of the reliability and measurement error estimates – thus increasing the certainty of the results.