Reliability of three linguistically and culturally validated pain assessment tools for sedated ICU patients by ICU nurses in Finland

1 Introduction

Patient pain assessment and management are required for quality care in intensive care units (ICUs). Studies have shown that pain assessment and management in ICU patients are insufficient [1], [2]. For example, in Gélinas’ [3] study, up to 74% of ICU patients suffered moderate or severe pain, and up to 77% recalled experiencing pain [3]. In addition to illness, a patient’s pain can be caused by events such as endotracheal suctioning, chest tube removal, wound drain removal and arterial line insertion, which are some of the most painful procedures in intensive care [4]. Previous research has shown that systematic pain assessment decreases pain, the duration of mechanical ventilation and nosocomial infections [5], [6]. However, pain assessment in the ICU remains a challenge, especially when patients are sedated and not able to communicate.

Few potential pain assessment tools are available for sedated intensive care patients [7]. The Behavioral Pain Scale (BPS) [8], the Critical-Care Pain Observation Tool (CPOT) [9] and the Nonverbal Adult Pain Assessment Scale (NVPS) [10] have shown promising psychometric qualities in systematic reviews [11], [12], [13], [14]. Fair to good agreement (0.74 and 0.75) has been found when evaluating the reliability of the BPS and the CPOT. Both instruments were able to discriminate between a painful and a non-painful procedure [15]. The reliability varied when the BPS and the NVPS were compared in situations with medical patients (the BPS=0.60, and the NVPS=0.65) and surgical patients (the BPS=0.77, and the NVPS=0.80). The validity of both tools was adequate [16]. In validation of the Swedish version of the CPOT also demonstrated good inter-rater reliability (on average 0.84) as well as adequate discriminant validity [17]. Validation of the Dutch version of the CPOT showed fair to moderate inter-rater reliability. In that study, the internal consistency was 0.56 (Cronbach α) during rest and when turning [18], and validity of the Swedish version of the CPOT showed that internal consistency was from 0.31 to 0.81 (Cronbach α) [17].

Deep sedation affected the pain assessment, so those scores are lower than without sedation [8]. Systematic pain assessment offers possibilities to ensure patients receive the right amount of sedatives and pain medication [5], [19]. With a systematic pain assessment, we also can diminish the amount of sedation; hence, the procedural pain can be cured adequately [7].

However, in 2017, no valid pain assessment tools were in systematic use in the Finnish ICUs. It is important to use well-established, properly translated and culturally adapted tools for pain care. There is a need for feasible, valid and reliable pain assessment tools for sedated ICU patients in Finland; therefore, we translated and culturally adapted the three above-mentioned tools [20].

The objective of this feasibility study was to test the reliability of the three Finnish translations of the pain assessment tools for sedated intensive care patients.

2 Materials and methods

2.4 Procedure for data collection

Each patient was at rest for 20 min before they were touched. Patients received their normal medications throughout the procedure. After the 20-min rest time, the patients were prepared for endotracheal suctioning. The nurse touched the patient, called him or her by name and advised what she was going to do. The nurse also prepared the equipment. Three patients each received an extra bolus of pain medication prior to the procedure. The nurses determined the dosage of the bolus based on their evaluation of each patient’s pain. It was not possible to control the amount additional pain medication administered during the study due to ethical reasons.

The nurse performed the suctioning. After the procedure, the patient rested for 20 min. The nurse prepared the patient for creaming of the feet in the same manner as for the suctioning and then performed the creaming. The session ended with the patient being untouched for another 20 min. Each video recording lasted, on average, just over 1 h. The design is seen in Fig. 1.

Fig. 1:

Study design and data collection. BPS, the Behavioral Pain Scale; CPOT, the Critical-Care Pain Observation Tool; NVPS, the Nonverbal Adult Pain Assessment Scale; NRS, the Numeric Rating Scale; RASS, the Richmond Agitation Scale.

Eight experts in pain care and ICU nursing care assessed the patients’ pain by viewing the video recordings. The pain was assessed using four instruments: the BPS, the CPOT and the NVPS, and the NRS, which was estimated by the nurse as a control instrument. Each expert assessed the patients’ pain at five measurement points (Fig. 1). After 6 months, the video recordings were evaluated with the same protocol by the same experts for testing the test-retest reliability.

3 Results

Four participants were male and two were female. Their diagnoses were pneumonia (4), acute respiratory insufficiency (1) and epiglottitis (1). The patients’ mean age was 69 (from 40 to 83). Their sedation levels varied from −3 to −5. Propofol 120–180 mg/h was given for sedation, and oxycodone hydrochloride 1–3 mg/h was given for pain. The sedation medication and pain medication were administered, on average, 7 min 30 s (from 0 to 20 min) before the ETS. One patient did not receive additional medication. The propofol bolus averaged 40 mg (two patients were not administered a propofol bolus), and the oxycodone hydrochloride bolus averaged 2.3 mg (three patients did not receive an oxycodone hydrochloride bolus). Systematic pain and agitation guidelines were not in use. The nurses followed the doctor’s orders when administering the medications. Prior to the treatment, the nurses determined the dosage of each bolus. Nine hundred and sixty observations were evaluated. Mean pain scores and standard deviations are described in Table 1. The MAP and HR, which remained fairly stable throughout the study, also are illustrated in Table 1.

Internal consistency with Cronbach’s alpha coefficient varied greatly with all instruments. The lowest values were recorded in those measurement points where the pain scores were 0. Highest scores were achieved after the endotracheal suctioning at rest: for the BPS, the score was then 0.86, the CPOT was 0.96, and the NVPS was 0.90. Table 2 shows the results of the internal consistency measures for the study.

The inter-rater reliability with the Shrout-Fleiss ICC tests showed the best results after the painful procedure and during the creaming. The scores were slightly lower for the BPS compared to the CPOT and the NVPS. At the baseline measurement point, the eight evaluators gave a score of 0 for pain in 80% of the evaluations with any of the instruments. Therefore, the reliability scores are close to 0. See Table 2 for detailed results of the inter-rater reliability scores.

The test-retest results are illustrated in Figs. 2–4. The BPS Bland-Altman plots are illustrated in Fig. 2, the CPOT Bland-Altman plots are in Fig. 3, and the NVPS Bland-Altman plots are in Fig. 4. The Bland-Altman plots show that all instruments gave similar results. The retest values were always within one point of the first measurement and had a 95% confidence interval. The figures show less variability between the test and the retest measurements for the NVPS at time points 2 and 4 compared to those for the CPOT.

Fig. 2:

BPS Bland-Altman plots in time 1–5.

Fig. 3:

CPOT Bland-Altman plots in time 1–5.

Fig. 4:

NVPS Bland-Altman plots in time 1–5.

4 Discussion

The participants were fairly deeply sedated, and pain medication was administered continuously in an infusion. Some patients also were given boluses before the procedure. This seems to have affected the study results – during the painful procedure, the participants were not expressing pain, but when the effects of the bolus were diminishing, the patients showed expressions of pain even 20 min after the procedure. This could be seen in all results. The pain medication oxycodone hydrochloride was given in a varied scheme ranging from 0 to 20 min before the procedure. The effect of this medication was seen a few minutes after administration [27]. In some cases, the medication did not have an effect on the patient’s pain. Nurses did not necessarily evaluate the connection between the extra pain medication bolus and the procedure. This indicates a serious lack in systematic pain assessment and pain care. A more systematic approach to pain care would help intensive care nurses to recognize pain symptoms and signs. A systematic approach will secure correct doses of pain medication and sedation [5], [7], [19], [28]. In our study, the patients’ MAPs and HRs did not react greatly to pain.

Internal consistency varied with all instruments. However, they were slightly lower than in previous studies that used instruments in the original language, English or French [e.g. 9], [10], [29]. In our study, several patients had a pain score of 0 at several measurement points. This affected the results and lowered the ICCs. Similar results were previously reported when the Dutch version of the CPOT was tested [18]. Perhaps this is due to the small sample size and the type of patients participating in the study. In addition, we utilized eight evaluators instead of two, which is usually the case in similar studies [e.g. 30, 31, 32]. In real life, there are often more than two pain assessors for the same patient; hence, our study resembled actual practice. Also, the ICC was lower in the Dutch study, which used more than two evaluators [18]. Furthermore, deep sedation diminishes a patient’s pain expressions. Payen et al. [8] had low scores with the BPS with deeply sedated patients. In Wøien et al.’s [28] study, nurses estimated the pain of deeply sedated patients using the NRS, and the results were in line with Payen’s [8] study and with ours.

Even though the inter-rater reliability calculations gave low interclass correlations at measurement points with low pain scores, it does not mean that the reliability is low. This is due to how the calculations were performed. The differences between raters were measured in relation to the total variation, and if the differences in pain scores between the participants are smaller, then the variation between the raters is greater compared to the variation when differences in pain scores are bigger. Based on the inter-rater reliability scores, the CPOT differentiates mild pain less accurately than the NVPS.

All the instruments reacted to painful situations, an effect that has been seen in other studies [e.g. 29]. Payen et al. [8] made similar findings in deeply sedated patients while Topolevic-Vranic et al. [33] showed that the CPOT and the NVPS scores increased during a painful situation.

The test-retest reliability proved to be similar and good for all tools. All tool scores were within one point difference between the two measurement times with a 95% confidence interval. Similar results have been reported in two Chinese studies where the test-retest reliability of the Chinese version of the CPOT ranged from 0.81 to 0.93 [31], and test-retest reliability of the Chinese version of the BPS ranged from 0.50 to 0.84 [32].

Pain scores measured with the NRS were higher at all measurement points than with other tools. Similar results were reported previously [34].

Our results support the previous studies, and the Finnish translations of the BPS and the CPOT can be utilized as pain assessment tools in Finnish ICUs [29]. Accurate pain assessment supports ICU pain care and leads to an increase in care outcomes [6], [7].

This study had some limitations. Our sample size was small. However, the inclusion and exclusion criteria for the participants were strict, and the sample was homogeneous. In addition, we were able to use the same eight experts for both measurement time points, which increased the trustworthiness of the results. The deep sedation of the patients was challenging and made the assessment slightly difficult, especially when the participant was at rest. Some low pain scores may be due to the fairly deep sedation of the participants. The participants also were given intravenous pain medication. It would have been preferable to evaluate the pain of the participants in real-life situations. For example, in some cases it was difficult to evaluate the muscle tension since the evaluator was not able to touch the patient and assess the tension. However, our design made it possible to use the test-retest design. This was rarely used in previous studies.

5 Conclusions

To our knowledge, this was the first time all three behavioral pain assessment tools were evaluated in the same study in a language other than English or French. Although the study had its limitations, we concluded that the three tools had good internal consistency, but the results were better for the CPOT and the NVPS tools compared to the BPS. The inter-rater reliability was best for the NVPS, but it was good for the other two as well. The test-retest reliability was strongest for the CPOT, but the other instruments were not far behind. All three tools proved to be reliable for further testing in clinical use.

6 Implications

There is a need for reliable pain assessment tools for clinical use in Finnish ICUs. Our study design made it possible to test and retest three pain assessment tools for sedated patients in ICUs. All three instruments (the CPOT, the NVPS and the BPS) proved to be reliable with sedated ICU patients. Further clinical testing with larger samples is recommended.