Painful differences between different pain scale assessments: The outcome of assessed pain is a matter of the choices of scale and statistics

2.1 The pain datasets

The VDS categories of pain intensity in the ODI version 2.0 are as follows: I have no pain just now, I have very mild pain just now, I have moderate pain just now, I have rather severe pain just now, I have very severe pain just now, and I have completely unbearable pain just now. The pain scale categories in the EQ-5D are: I have neither pain nor complaints, I have moderate pain or complaints, and I have severe pain or complaints. The perceived low back pain just now is marked on the horizontal 100 mm VAS line with the anchor points being no pain (0) and maximal pain (100), which provides 101 possible VAS pain position scores.

In this study, the datasets of pain assessments that were completed by 101 patients the evening before planned surgery at the Clinic of Spinal Surgery were used [7]. The median age was 57 (Q₁: 41, Q₃:69) years, and ranged from 19 to 85 years. The patients had one of the following diagnoses: disc herniation or stenosis in the lumbar region, segmental discogenic pain, or isthmic spondylolisthesis in the lumbar region. The perceived pain was regarded as stable. Majority of the patients, i.e., 54 patients have assessed rather severe or very severe low back pain in the last 4 weeks on the BIS [7]. The patients gave their informed consent to participate and to use the data in studies.

2.2 Statistical methods

The single pain datasets were described by the median, quartiles, and range. The distributions of the VAS pain data and the comparing (VAS,VDS) pairs of pain datasets were described by scatter plots [13,14,34]. In classical rank-based methods for paired data, identical observations within each single dataset are assigned the mean rank of the ranks they share. The ranks are tied to each single set of identical data.

The Spearman rank-order correlation coefficient (r _s) adjusted for ties, and the 95% confidence interval, 95% CI, of the possible relationship, association, between the continuous VAS data and each of the discrete verbal descriptive sets of data from the ODI and the EQ-5D were calculated [35,36]. However, an inter-scale association does not imply comparability of the comparing scales [33,34,35].

Contrary to the classical single ranking approach, in the paired ranking approach by Svensson, each pair of data are allocated bivariate ranks, which means that only identical pairs of data are regarded as tied. Hence, each pair will be regarded as ordered, disordered, or tied with respect to the other pairs of the dataset [13,14,15,25,26,27]. The proportion disordered (VAS,VDS)-pairs among all possible different combinations of pairs was calculated and expressed as the measure of disorder, D, (0 < D < 1).

Irrespective of the scaling and of the frequency distributions of the two datasets, called marginal distributions, the level of agreement in ordering of the pairs is defined by the coefficient of monotonic agreement, MA = 1–2D, ranging from (−1) to 1 [33,34,35,37]. A high level of agreement in ordering, MA, is required for regarding two sets of scale scores as comparable. MA = 1 when each pair of scale scores have the same rank-order regardless of the type of comparing scales and of the possible the number of scores. This complete agreement in rank-ordering of all the pairs of data means that the comparing paired data set is rank-transformable, given the frequency distributions. Such a paired distribution of a rank-transformable data set is an important tool for identifying which categories of two scales correspond to each other in the case of complete agreement in ordering. Therefore, the Rank-Transformable Pattern of Agreement, (the RTPA), was used for identifying which categories of two comparing scales that correspond to each other in the case of complete agreement in ordering, D = 0 and MA = 1 [33,34,37].

2.3 Unbiased inter-scale (VAS,VDS) calibration

There are different ways to discretize continuous VAS data into categories. The bivariate ranking approach allows for an unbiased calibration of scales with different number of response alternatives, which is valuable in inter-scale comparisons concerning the cut-offs between interchangeable sets of scores. The RTPA is uniquely defined and constructed by pairing off the two comparing single sets of frequency distributions of the data. In the RTPA, the bivariate rank ordering of all individuals will be unchanged when changing scales, which is the case of interchangeability of scale assessments, then MA = 1 [26,27,34,35].

In this study, the cut-off positions on the VAS pain scale that correspond to each of the verbal pain categories of the ODI and the EQ-5D were identified by studying the rank-transformable data-pairs of (VAS,ODI) and (VAS, EQ-5D), respectively. The ordered marks on the VAS that divide the continuous line into categorical VAS intervals having the same frequency distribution as the comparing discrete scale are defined by the steps of the rank-transformable pattern of agreement. Therefore, the categories of the discretized VAS and of the VDS data will have identical frequency distributions, provided no more than one single observation on the categorical cut-off positions on the VAS [13,14,34,37].

For comparative reasons, the paired distributions of the (equidistant VAS, ODI) and of the (equidistant VAS, EQ-5D) pain datasets were also evaluated. The percentage agreement, PA, the measures of disorder, D, and of order consistency, MA, were calculated. Presence of inter-scale bias is expressed by the measures of systematic disagreement in position (RP) and in concentration (RC) of the scale distributions, and were calculated by using a free software program [25,26,27,35,38,39,40].

3.1 Statistical descriptions

As evident by Figure 1, the distribution of the patients’ VAS-pain assessments was skewed, median: 59, (Q₁: 38; Q₃: 76).

Figure 1

The frequency distribution of perceived low back pain just now assessed by the 101 patients on the visual analogue pain scale (VAS), the anchors being 0 (no pain) and 100 (maximal pain) just now.

The frequency distributions of the verbal descriptive pain assessments on the ODI and the EQ-5D are shown in Table 1. The median ODI pain was rather severe, (Q₁: moderate, Q₃: rather severe) pain, and the median EQ-5D pain was moderate pain (Q₁: moderate pain; Q₃: severe pain or complaints).

The paired distributions of (VAS, ODI)-pain data are described in Figure 2 and Table 1. All but three patients assessed moderate, rather severe, or very severe pain just now on the ODI scale. Irrespective of these ODI-pain levels, the (VAS, ODI) pairs are overlapping between the VAS-pain positions (4–96).

Figure 2

The paired distribution of pain assessments made by 101 patients on the Visual Analogue Scale (VAS) and the ODI pain intensity scale. The ODI scale categories: I have no (0), very mild (1), moderate (2), rather severe (3), very severe (4), and completely unbearable (5) pain just now.

One patient assessed very mild ODI-pain and marked VAS-pain position 36. Figure 2 shows that 20 patients have marked lower VAS-pain than 36, but higher ODI-pain than very mild. These pairs will get the VAS-ranks (2–21). Another two patients assessed VAS-pain 36 and moderate ODI-pain. They will get the tied VAS-rank 23, which is the mean of the ranks 22–24. In the proportion of different (VAS, ODI)-pairs among all possible combinations of pairs, the measure of disorder, D, was 18.7%, and the measure of order consistency, MA, was 0.63. The (VAS, EQ-5D) pairs of pain data were overlapping, the measure of disorder, D, was 0.146, and MA = 0.71, Table 1.

All overlapping different pairs of these (VAS, VDS) pain data indicate that continuous VAS data are not comparable to data from VDS. The increased median VAS levels for higher VDS categories, and the levels of disorder explain the positive Spearman rank-order correlation coefficients, r _s, with wide 95% confidence intervals, CI(r _s). These measures of relationships were: (VAS, ODI): r _s: 0.45, 95% CI(r _s): (0.27–0.59), and (VAS, EQ-5D): r _s: 0.35, 95% CI(r _s): (0.17–0.51), respectively.

3.2 Paired (equidistant VAS, VDS) pain data

The paired (VAS, VDS)-distribution is the basis for creating equidistant VAS pain scores. This means, for example, that the observed (VAS, ODI)-overlap in Figure 2 will have a decisive impact on the paired (equidistant VAS,ODI)-interchangeability.

The pairs of (equidistant VAS, ODI)-data are grouped in (6 × 6)-cells, which will affect the number of tied and of disordered pairs, and hence of the measures of comparability. For example, the single pair: (36, very mild pain), shown in Figure 2, is now tied to the group of 18 pairs within the VAS-pain interval (34–50) corresponding to moderate pain, and the single pair (100, unbearable pain) is tied to the VAS unbearable pain interval (84–100), Figure 3.

Figure 3

The 6 × 6 frequency distribution of the paired data from the equidistant VAS pain intervals and the ODI pain scale categories.

The statistical description of the six equidistant VAS pain intervals of the (equidistant VAS, ODI) pain dataset is median: (51–67), Q₁: (34–50), and Q₃: (68–83) corresponding to rather severe pain, moderate pain, and very severe pain, respectively.

By this paired (equidistant VAS, ODI)-categorization, Figure 3, the percentage agreement, PA is 38%. The 39 pairs in the lower right region of the agreement-diagonal represent patients with higher (equidistant VAS) than ODI pain categories, and the opposite holds for the 24 pairs in the upper left region. The frequency distributions of the marginals differ, indicating systematic disagreement, bias, between the comparing datasets, mainly explained by a bias in concentration, RC, 0.40. Hence, it is 40% units more likely that ODI rather than equidistant VAS pain data will be concentrated in central categories more frequently than the opposite, (Figure 3 and Table 2). Correspondingly, the PA of the (equidistant VAS, EQ-5D) pairs of data was 49%, and the 51% disagreeing pairs are explained by systematic disagreements in position, RP, and in concentration, RC, Table 2. These observed significant systematic disagreements between the paired (equidistant VAS, VDS) pain data restrict the possibility of attaining unbiased scale-interchangeability.

3.3 Paired comparisons of categorized VAS intervals unbiased to VDS data

A total interchangeability between categorized VAS and VDS data requires identical marginal distributions, which means rank-transformable pairs of data. Hence, an unbiased categorized VAS will be uniquely adapted for the specific paired scale and the dataset only.

By pairing off the single sets of frequency distributions from the VAS and the ODI data, the rank-transformable pattern of agreement, RTPA of the VAS-ODI comparison was constructed, Figure S1 (Supplementary material). By this pairing procedure, the discretized VAS intervals will have the same frequency distributions as the corresponding ODI categories. The lack of systematic disagreement, bias, is confirmed by the identical marginal distributions.

The (6 × 6)-paired distribution of data from the discretized VAS intervals that are unbiased the ODI categories is shown in Figure 4. The single pair: (36, very mild pain), shown in Figure 2, is now tied to the group of 36 pairs within the VAS-pain interval (7–51) that corresponds to moderate pain. The percentage agreement, PA is 54%, Figure 4 and Table 3.

Figure 4

The 6 × 6 paired (VAS,ODI) frequency distribution of the discretized VAS pain intervals that are unbiased ODI pain scale categories. The agreement diagonal is marked.

Based on this paired (unbiased VAS, ODI) pain data, the median VAS pain is (52–78) corresponding to rather severe pain, and Q₁: (7–51), Q₃: (52–78), corresponding to rather severe pain and moderate pain, respectively.

The VAS-intervals that are interchangeable to each of the unbiased pain categories of the ODI and the EQ-5D, respectively, are shown in Table 3. The unbiased (2 × 2) distribution of the dichotomized VAS and the EQ-5D categories’ moderate and severe pain agreed completely, PA = 100%.

The (unbiased VAS, ODI) pairs of pain data reduced the proportion of different disordered pairs to D: 12% (Table 3) from the 19% disordered (continuous VAS, ODI)-pairs (Figure 2 and Table 1), and the 15% disordered (equidistant VAS, ODI-pairs (Table 2).

We have shown, by the pairs of (VAS,ODI) pain data, Figure 2, that a consequence of equidistant categorization of VAS data, Figure 3, is the increase in tied observations and hence a decrease from 19 to 15% in the proportion of disordered groups of data. Correspondingly, for the pairs of (VAS, EQ-5D) pain data, there is a decrease from 15 to 10% disordered pairs.

We have also shown that the most powerful paired (VAS,VDS) comparison was obtained by the pairs of (unbiased VAS, VDS) pain data. The percentage agreement of the paired comparisons of categorized VAS data that are unbiased to a specified VDS and to the dataset ranged from 54 to 100%, and the interchangeability, MA ranged from 0.77 to 1.

It must be stressed that unbiased VAS categorization is only valid for the specific comparison of discrete scale and the current dataset. The presence of overlapping (continuous VAS, VDS) data determines the (equidistant VAS, VDS) comparisons. From all the comparison tests performed, it is concluded that results from continuous VAS-data are exclusive and should not be transformed, or categorized, to be compared with data from other rating scales.

4.1 Measurement properties of qualitative data

Irrespective of the type of scale and numerical coding, the data collected from subjective assessments represent only an ordering, and can never reach the linear properties of being quantitative with univocal magnitude and standardized difference [18,19,20,42,43,44]. Consequently, calculations of sum scores of multi-item assessments, and of differences between before and after treatment scores are not permissible or valid. Also, calculations of the mean score and standard deviation are inappropriate analytical operations, even though this fact is often ignored. Statistical methods based on the ranks of the ordered categorical scores must be used [21,22,23,24,25,26,27,28,42,44].

In our study we have used rank-based statistical methods taking into account the rank-invariant properties of the data from the patient’s subjective experience of pain. Therefore, the paired VAS-VDS comparisons regarding interchangeability and order consistency will provide reliable results [13,14,15,34,35,37]. The fact that only non-parametric rank-based statistical methods are appropriate to use for data from scale assessments for meaningful and firm conclusions are known [18,19,20,21,22,23,24,25,26,27,28,42,44,48,49,50]. Despite this, the increasing use of incorrect parametric methods seems to be a well-established, acceptable norm deviation or tradition [21,22,23,41,42,43,46,47,50,51,52,53,54]. Interestingly, studies with results that are based on well-motivated appropriate non-parametric statistical methods have been questioned and regarded as wrong in review studies [47,53,54]. There is also a risk of publication delay, or even rejection of manuscripts that have used well-motivated appropriate non-parametric, statistical methods. However, as stated in COSMIN guidelines, results and conclusions that are based on wrong methods of design and statistics will always be wrong and useless [55].

4.2 Different operational definitions of perceived pain

It could be a scientific challenge to compare data from assessments on different pain scales when only categorical codes are presented in a data list, because of hidden variations in the operational definitions of the same variable. In this study, the scale category “moderate” in the data list was short for: the ODI item: “moderate pain just now and the EQ-5D item: “moderate pain or complaints” [7]. These different definitions of perceived pain underline the importance of descriptions of the operational definitions in the results.

Our study also demonstrated that a position on the VAS was not related to a certain verbal descriptive pain level. A mark on the VAS is not verbally interpretable and must not be regarded as a distance, or a percentage, with metric properties [13,14,15,37,42,43,44,48]. We have shown that individuals with the same VDS pain score could assess different VAS pain assessments.

4.3 Inter-scale comparisons: The lack of comparability

In the paired (VAS,VDS)-comparisons, the large overlap of the VAS assessments on the discrete verbal categories indicates that marks on the line are individual and cannot be used for group descriptions of subjective feelings. Corresponding individual variability in the marks on the VAS for each ordered scale category has been found in other studies [13,14,37,40,45]. Methodological studies have shown that the use of NRS or VAS increases the proportion of disordered pairs as compared with the use of VDS or the Graphic Rating Scale which is a VAS with ordered categories indicated below the line [13,34,42,43,44].

Inter-scale comparisons are commonly evaluated by calculating the correlation coefficient [2,17,47,56], which is one of the most misused statistical measures as stated by Altman et al. [21,57,58]. In a systematic review of 54 papers of pain scale comparisons, the conclusions in all but 11 papers were based on the correlation coefficient [2]. However, interchangeability of different scales for the same variable refers to measures of agreement and not to association [33,34,35]. A systematic shift in scores towards higher categories by one of the datasets will result in a strong association, i.e., high correlation, but a low level of inter-scale agreement. Monotonic agreement, MA, is the measure of the extent to which pairs of assessments agree in rank ordering, but not necessarily to the same category [13,14,15,33,34,35].

In a systematic review regarding the validity and reliability of pain measures, the authors concluded that the use of VAS was advantageous over the other scales. In all but one of the 34 papers reviewed, the correlation coefficient was used for the scale comparisons. In only one of the papers reviewed, the authors have used the rank-based measure of ordered consistency, MA, between the VAS and the VDS pain data, which resulted in a completely opposite conclusion than that of the other 33 papers [13]. The conclusions of this single study that were based on correct, well-motivated, rank-based statistical methods were questioned by the authors [47].

Interchangeability of rating scales requires agreement in the rank-ordering of the paired data of the comparing scales [13,14,15,20,26,33,34,35,37]. As evident by the paired distribution of the (VAS, ODI) pain scales, Figure 2, the positive correlation coefficient, 0.45, is not a measure of the comparability of the pairs of pain data. The correlation coefficient indicates that (r _s ²: 0.45²) = 20% of the variability could be attributed to the relationship between the pairs of data. Corresponding proportions for the (VAS, EQ) paired pain distributions was 12%. Similar levels of inter-scale variability could be estimated in a profound consensus study regarding different pain scales [55].

Transformations and other statistical manipulations to meet the requirements for using parametric statistical methods and/or for creating comparative scores are common in single studies and in systematic reviews [2,46,47,52,53,54]. Consequently, most conclusions were based on statistical methods that require normally distributed quantitative data for obtaining valid results. Based on manipulations and parametric statistical methods, the conclusions were that all types of scales (NRS, VDS, and VAS) worked quite well, even though the NRS seemed preferred over the VAS in most studies [17,47,52,53,54]. Conflicting results were found and questioned regarding three papers that have used appropriate rank-based non-parametric methods instead of psychometric analysis [47,53,54].

Altman has pointed out the ethical implications of the choice of inappropriate statistical methods [21]. The results of statistical treatment and evaluations of the data are valid only if the conditions on the data and the research questions for their use are met. Furthermore, recommendations, guidelines, and applications of inappropriate, often parametric, statistical methods for describing and evaluating subjective assessments of all kinds not only risk jeopardizing the scientific quality of future studies but also the quality of evidence-based care decisions and patient safety. It is also ethically important when assessing evidence for care interventions [21,22,23,24,26,27,28,48,49,55].

4.4 Limitations

The patients included in this study had different diagnoses as an indication for the upcoming spinal surgery. However, all of them were affected by long-term and stable pain from the lumbar region. This could be considered as limitation of the study but is counteracted by the paired study design which means that each patient is his/her own control. Contrary to studies using group comparisons of data from different pain scale assessments, our paired design allows for a wider range of diagnoses and of assessed pain levels ranging from no (0) to max (100) pain. Comparing pain data from VAS and from different ordered categorical scales means that data from different operational definitions of “pain just now” are compared. We believe that the rated “pain now,” “pain today,” and “pain just now” the evening before surgery reflect the same pain experience among these patients, since their clinical pain was regarded as long-term and stable.

The different variables of interest were VAS: low back pain, ODI: pain intensity, and EQ-5D: pain and complaints. However, it is always a challenge to have exact knowledge of what individuals include in their rating of subjective variables such as pain. Based on the pre-surgery rating of the patient’s painful situation, characterized by long-term pain originating from the lumbar region, we assume that they did rate the same pain experience despite the different pain expressions.