Development of a binary classifier model from extended facial codes toward video-based pain recognition in cancer patients

Introduction

According to the International Association for the Study of Pain (IASP), chronic pain refers to persistent or recurring pain that lasts for an extended period, typically lasting for three months or more. It is a complex multidimensional experience that can have a significant impact on an individual’s physical, emotional, and social well-being. This condition can severely compromise the patient’s quality of life, often limiting the ability to work, sleep, and affecting social interactions with friends and family [1]. Cancer pain is a common problem among cancer patients, and its prevalence can vary depending on the stage and type of cancer, as well as the individual patient’s pain tolerance and pain management strategies. The World Health Organization (WHO) reported that approximately 30–50 % of patients with cancer experience moderate to severe pain, and 70–90 % of individuals with advanced cancer experience pain [2].

A comprehensive approach to pain treatment usually involves a combination of pharmacological and non-pharmacological interventions and should be tailored to the individual needs of each patient. Careful pain assessment, therefore, is of paramount importance to guide the decision-making process pertaining to therapy [3]. Nevertheless, clinicians define treatment based on subjective pain scales and continuously refine drug dosages based on the patient’s responses [4]. On the other hand, pain is a complex interplay of sensory and emotional components that is often challenging to verbally articulate [5, 6]. Furthermore, although self-report quantitative methods such as the Numeric Rating Scale (NRS) and Visual Analog Scale (VAS) are widely regarded as the standard for pain assessment [7], they are prone to reporting bias. This bias stems from the inherent nature of self-reporting and can be influenced by various psychosocial factors, including patients’ tendencies to catastrophize or underreport their pain [8].

Notably, in cancer patients, pain perception and expression are influenced by a wide range of variables, including psychosocial issues and distress, as well as the type and progression of the primary tumor [9], and pain pathophysiology [10]. Moreover, the individual’s pain tolerance, communication abilities [11], coping strategies, and emotional state [12] further complicate the clinical evaluation. In this complex scenario, the limits of subjective pain assessment emerge and consequently, it is important to develop models for the study of pain focused on automatic and objective responses.

Automatic pain assessment (APA) relies on the use of objective measures to assess pain intensity and other pain-related factors [13]. These methods aim to provide a more objective measure of pain, as opposed to relying on subjective pain scales. Strategies for APA may be particularly useful in situations in which obtaining reliable self-report data is difficult, such as in children with cognitive disabilities [9], as well as patients of all ages who face communication challenges [14], including individuals with dementia or those who are non-verbal or intubated [15]. These techniques are also particularly useful for research purposes, enabling the study of pain experiences across diverse populations and settings [16].

There are different approaches to APA, including the use of physiological indicators, such as heart rate or electrodermal activity, and the application of brain imaging techniques, such as functional magnetic resonance imaging (fMRI) or electroencephalography (EEG), to measure pain-related brain activity [17]. Behavioral pain assessment is another group of APA methods. It involves observing and recording an individual’s behavior due to pain experience and analyzing the data collected to identify patterns and trends. Behavioral-based approaches encompass facial expressions, qualitative (e.g., content analysis) and quantitative (e.g., computational linguistics) methods for linguistic analyses [18], and non-verbal physical indicators of pain such as body movements and gestures [19]. Current methodologies tend to incorporate multimodal techniques that integrate observable behaviors with neurophysiological elements [20].

Facial expressions are a crucial aspect of non-verbal communication, especially in conveying pain to others. Therefore, extensive research has focused on studying facial expressions as a pain behavior to enable automatic and objective pain assessment [21, 22]. Moreover, the facial expression of pain shows consistency across ages, genders, cognitive states (e.g., noncommunicative patients), and different types of pain and may correlate with self-report of pain [23]. Facial expressions that indicate pain can be classified into voluntary and involuntary physical manifestations. The former is intentionally triggered by the cortex and can be consciously controlled in real time. However, these expressions tend to be less smooth and exhibit more variations in movement. Involuntary facial expressions, on the other hand, are unconsciously triggered by subcortical processes and are recognized by their symmetrical, synchronized, reflex-like movements of facial muscles, which give them a seamless appearance.

Action Units (AUs) are a set of specific facial muscle movements or activations that are used to describe and analyze facial expressions. The concept of AUs was developed, in 1978, by psychologists Paul Ekman and Wallace V. Friesen as part of their Facial Action Coding System (FACS), which is a comprehensive tool for describing and analyzing human facial expressions [24]. There are 46 individual AUs that are defined and categorized by the FACS. Each AU represents a specific movement or combination of movements of the facial muscles, such as raising the eyebrows, wrinkling the nose, or pursing the lips. The manual analysis of the process is often exceedingly intricate. Nevertheless, with the advent of artificial intelligence (AI) techniques, there has been a shift toward computer-mediated automatic detection of pain-related behaviors and various AI-based image-processing strategies have been implemented [25]. Consequently, the utilization of AI methodologies can offer a significant and valuable application in the field of APA, particularly when conducting assessments based on facial behaviors.

Although the growing interest in studying physiological signals to investigate APA in cancer patients [13, 20], there is insufficient research on pain behavior associated with APA in this population. On these premises, we investigated facial pain behaviors (i.e., AUs) related to APA and developed a binary classifier to distinguish between “pain” and “no-pain” using videos collected from cancer patients.

Methods

The study was performed in accordance with the Declaration of Helsinki and the patients provided written informed consent. The Medical Ethics Committee of the Istituto Nazionale Tumori, Fondazione Pascale (Napoli, Italy) approved the research (Pascale study, Pain ASsessment in CAncer Patients by Machine Learning). Protocol code 41/20 Oss (26 November 2020). The study is part of a multimodal APA project encompassing computer vision and natural language processing approaches, as well as physiological signals and clinical investigations.

Action Units selection and analysis

For AUs analysis, OpenFace 2.0 was adopted. It is a free and open-source toolkit for facial recognition and facial landmark detection. The tool uses Deep Neural Networks (DNN) to perform facial recognition and facial landmark detection, and it can be used to build applications for tasks such as facial recognition, facial tracking, and facial expression analysis [28, 29].

The Neural Network classifier used the intensities (r) of 17 AUs that were specifically chosen. As stated in the official OpenFace documentation [28], these intensities are measured on a five-point scale with an additional point, where a score of 1 indicates minimal presence, 5 indicates maximum presence, and 0 indicates no presence at all.

According to Prkachin et al. [30], we considered the AUs related to brow lowering (AU4), orbital tightening (AU6 and AU7), nose wrinkler (AU9), upper lip raiser (AU10), and eye closure (AU43) and representing the “core” actions of pain expression. Such a core set of six AUs has been extended with 11 AUs considered in Littlewort’s study on spontaneous pain expressions [31]. They included inner brow raiser (AU01), outer brow raiser (AU02), upper lid raiser (AU05), lip corner puller (AU12), dimpler (AU14), lip corner depressor (AU15), chin raiser (AU17), lip stretcher (AU20), lip tightener (AU23), lips part (AU25) and jaw drop (AU26) (Figure 1).

Figure 1:

The action units (AUs) employed. The “core” set of six pain expressions (AU4, AU6, AU7, AU9, AU10, and AU43) [28] (green boxes) was extended with 11 selected AUs (light blue boxes), including inner brow raiser (AU1), outer brow raiser (AU2), upper lid raiser (AU5), lip corner puller (AU12), dimpler (AU14), lip corner depressor (AU15), chin raiser (AU17), lip stretcher (AU20), lip tightener (AU23), lips part (AU25), and jaw drop (AU26). Created with BioRender.com (accessed on 25 April 2023).

For each image, OpenFace extracts the considered AUs. It takes the input video and extracts AUs frame by frame. Then, the collected data were grouped, according to the associated class, and split into train and test sets, respectively 80 and 20 % of extracted data. Subsequently, 20 % of the train set was reserved for the validation set.

Neural Network classifier

This developed Neural Network classifier is based on the Shallow Network architecture. This structure usually consists of an understandable combination of two layers and has been demonstrated to be as effective as DNN on small-scale datasets [32]. The dataset used for model training and validation, which includes parameters and AUs extracted using the OpenFace toolkit, is available at reference [33].

The first layer of the classifier takes as input the 17 intensity values related to the considered AUs extracted by OpenFace, for each image, with a ReLU/sigmoid as an activation function. The last layer (output) utilizes the sigmoid activation function to generate an output of a classification label, specifically indicating pain (1) or no-pain (0) (Figure 2). The proposed network uses a binary cross-entropy loss function, the standard Adam optimizer (from the Keras library) was employed to minimize the validation loss.

Figure 2:

The binary classifier model. The classifier is made up of two dense layers. The first layer consists of 17 nodes associated with the facial Action Units (AUs) extracted by OpenFace for each image. The final layer (i.e., output layer), uses a sigmoid activation function to output a classification label of either “pain” (1) or “no-pain” (0), based on the outputs from the neurons in the previous layer. Patient consent was acquired for the study (ClinicalTrials.gov Identifier: NCT04726228) and scientific divulgation of personal identifiable information.

Video annotation

For video annotation, we used Eudico Linguistic Annotator (ELAN), version 6.4 [34]. It is a software tool for creating, editing, and analyzing multimedia data like video, audio, and text. Time-aligned annotations enable the assessment of the temporal relationship between events.

In order to perform continuous estimation, the entire patient video, along with its corresponding frame prediction values of 0 (no-pain) or 1 (pain), was imported into ELAN (Figure 3).

Figure 3:

Frame prediction values of “pain” ranging from 0 to 1 (red line). The annotation tool for audio and video recordings ELAN 6.4 was implemented. Patient consent was acquired for the study (ClinicalTrials.gov Identifier: NCT04726228) and scientific divulgation of personal identifiable information.

Results

The binary classifier obtained an accuracy of 0.9448 after about 400 training epochs. In addition to the achieved accuracy, the AUROC value of ∼0.98 (Figure 4).

Figure 4:

The area under the receiver operating characteristic (ROC) curve (AUC) of the considered model.

The measure of the error (binary cross-entropy) on the validation set is 0.1581. Precision and recall measures are 0.9528 and 0.968, respectively. The performance of the classifier is shown in Table 2.

Discussion

In the field of APA, the study of facial expressions is of paramount importance. We used two datasets to develop a binary classifier capable of distinguishing between the categories “pain=1” and “non-pain=0”. The complete patient video, together with its associated frame prediction values (0 and 1), was loaded into ELAN. This process allowed for the analysis of the video in greater detail, as the frame prediction values provided a quantitative measure of the pain levels present in each individual frame of the video. This enabled a more accurate and precise estimation of the overall pain levels experienced by the patient, as well as the specific moments and duration of pain throughout the video. To our knowledge, this is the first study conducted for the APA using AUs in a setting of cancer patients. The systems, originally trained using publicly available datasets with an expanded set of AUs to better capture pain expressions, have ultimately been tested on actual cancer pain patients in real-world scenarios.

Compared to other classifiers based on the Shallow network architecture [34], the proposed model employed meaningful features rather than extracting them through a feature processing layer, which is supposed to be trained along with the classification layer. The model’s validation revealed a high level of accuracy (∼94 %). This is not surprising, as error rates tend to decrease on smaller datasets when using shallow networks [37].

Other experiments were conducted with the aim of extracting pain characteristics from videos through the analysis of AUs. For example, Lucey et al. [38] used the UNBC-McMaster Shoulder Pain database and implemented an active appearance model for automatically detecting the frames in which a patient is in pain. They were able to extract visual features and derived 3D parameters. The features were utilized to classify individual AUs through a linear support vector machine (SVM). To merge the SVMs’ outputs for the AUs, linear logistic regression was employed, allowing for a supervised calibration of the scores into log-likelihood scores. The same authors also worked on AUs and performed a frame-by-frame analysis. A combination of a convolutional neural network (CNN) and recurrent neural network (RNN) was utilized to analyze the spatial characteristics and dynamic changes in video frames. The CNN was responsible for learning spatial features from each frame, while the RNN captured the temporal dynamics between consecutive frames. Following this analysis, a regression model was employed to estimate pain intensity scores for each video [39].

The FACS is a comprehensive system for describing all observable facial movements resulting from the activation of individual muscles. Despite FACS, by its nature, does not provide descriptors specific to emotions, the system is widely used in the field of psychology as a means of studying and exploring different aspects related to emotions as well as other human states [40]. Nevertheless, the specifics of the codes (e.g., number and types of AUS, namely “pain relevant AUs”) and methods (e.g., automated vs. semi-automated systems) used for pain research have yet to be determined [41]. Interestingly, Skiendziel et al. [42] utilized FaceReader 7, an automated emotion and AUs coding system, to analyze a dataset of standardized facial expressions of six basic emotions. In addition to the clinical setting, another peculiarity of our study is the use of an extended combination of AUs. The original “core” of six pain expressions [28] was expanded to obtain a higher accuracy of the data (i.e., a higher number of nodes of the input layer of the classifier). Additionally, the capability to analyze a video frame-by-frame can enable the detection of pain fluctuations. This aspect is crucial in cancer pain assessment and management, for example for addressing specific cancer pain phenomena such as breakthrough cancer pain [43].

Research in the field of APA for the management of cancer pain must address several issues. Probably, a key aspect is multimodal data collection. In this regard, the combination of data obtained from the analysis of language and text, as well as physiological data, will allow for the development of more accurate predictive models for the diagnosis of various components of pain in cancer patients [44]. This is because pain is a multidimensional experience, and it is influenced by not only physiological factors but also psychological and emotional elements [45]. By integrating these different types of data, more targeted and effective treatment plans could be designed [46].

Conclusions

Despite APA methods are not a substitute for subjective pain assessments and should be used in conjunction with other methods to provide a more comprehensive understanding of an individual’s pain experience, the study of facial behavior using automatic systems offers important perspectives. The proposed Shallow Network classifier combined with the augmented set of AUs demonstrated to be a fast and accurate method to identify pain based on facial expressions. More research is needed to improve the process and especially to integrate this data with multi-parameter analyses such as speech analysis, text analysis, and those obtained from physiological parameters.