Inferring pain from avatars

A comprehensive understanding of pain requires not only articulating its proximate mechanisms but to appreciate how adjustments to its presence regulate ourselves and the world around us through the behavioural changes that it drives. Such changes are the basis upon which most knowledge about pain mechanisms in humans and animals is based. As fundamentally social animals, many of the behaviour changes induced by pain in humans are communicative [1]. Some are communicative indirectly. For example, rubbing an injured forearm, while primarily aimed at downregulating discomfort, can lead an observer to infer something about the sufferer’s experience. Other pain-related behaviours are communicative “by design.” In other words, they appear to have evolved largely, if not primarily, as a means of transmitting something about the sufferer’s experience to others [2]. Among such behaviours are certain characteristic changes in facial expression that have been the subject of considerable scrutiny ever since methodology that enabled their objective empirical description became available [3]. Analysis of such behaviours enables us to describe the link between pain and its proximal causes on one hand and its behavioural expression on the other [4], [5] and, increasingly, to draw inferences about sufferers’ pain that would otherwise be difficult, if not impossible to make. For example, in previous work by the authors of the article by Meister, Horn-Hofmann, Kunz, Krumhuber and Lautenbacher ([6], this volume), by making careful measurements of the facial expressions of patients with dementia, Kunz, Scharmann, Hemmeter, Schepelmann and Lautenbacher [7] were able to demonstrate enhanced pain reactivity in patients who could no longer characterize their pain verbally as a result of dementia. While important, study of the links among bodily injury, pain experience, and pain expression nevertheless provides an incomplete characterization of the pain nexus because it ends at the expression and does not deal with its social consequences. A robust field of inquiry has emerged in recent years that focuses on those social consequences [8].

In fact, the article by Meister et al. [7] contributes both to our understanding of the properties of pain expression and of its social consequences. Early work on pain expression showed that pain arising from many different experimental manipulations or clinical conditions is accompanied by a limited number of changes in facial expression. The most common changes include contractions of the orbicularis oculi muscle, which narrows the eye aperture and raises the cheeks; contraction of the corrugator muscle, lowering and drawing together the eyebrows, contraction of the levator muscle, which raises the upper lip and, sometimes wrinkles the nose, and closure of the eyes [9], [10]. Empirical studies of pain expression using the gold-standard for dissecting facial movements, the Facial Action Coding System [11] (FACS) have occasionally reported other movements to be pain-related, including, curiously, contraction of zygomaticus major, the principal muscle involved in the smile; however, such findings have been inconsistent and occasional. The consistency with which the aforementioned actions have been found to be associated with pain have led to occasional conclusions that they comprise a unitary “pain expression.” However, later work by Kunz and Lautenbacher [12], using cluster analysis, has called this notion into question by identifying several distinct configurations of facial actions that occur during pain, in addition to a “stoic” expression, which accompanies pain among a significant number of people.

All of this work has been based on the FACS method, which is detailed and fine-grained, applied to recordings of people in various painful conditions. But to show that the behavioural features associated with the identified configurations are indeed socially meaningful, it is necessary to show that they are perceived coherently as communicating pain by others, using a conventional judgement study methodology [13]. To do so poses problems for several reasons, among which are that features of the behaviour to be shown other than the topographical configurations at issue–for example, the timing of the actions or the co-occurrence of other movements—could affect or even confound judgements. Technical advances in the form of avatars hold promise to overcome this problem because they allow the construction of realistic models of facial expression in which these other features can be controlled precisely. Moreover, they make it possible to manipulate other features of the face, such as its apparent gender, age, skin colour, etc., so that understanding can be built of their role in social perception of the pain sufferer.

Avatars were programmed using software that generates realistic-looking facial expressions corresponding to FACS Action Units. The apparent gender and age of the avatars were varied systematically. Observers viewed the programmed videos, rating the degree of pain displayed and their confidence in their judgements. Analyses revealed that three of the five clusters were judged to show pain of moderate intensity. The ones that did not included a “stoic” cluster and a cluster involving raised brows.

The judgement study methodology thus helps to sharpen our understanding of the empirical configurations identified through cluster analysis by showing that not all of them are perceived reliably as communicating pain. Further, and interestingly, the findings appear to emphasize the central importance of the action of orbicularis oculi, which is a component of all the configurations perceived as painful. Although the authors largely focus on what the findings may imply in terms of training people and computer vision systems to identify pain in others, the demonstration of these three configurations opens questions about their underlying determinants that further research could profitably pursue.

The other major substantive finding was that female avatars were rated as displaying substantially more pain than avatars who appeared to be male. The difference was greatest within the cluster defined exclusively by furrowed brows and narrowed eyes. Although, as the authors point out, this is a not-unexpected finding in light of previous studies, the use of the avatar methodology, which makes it possible to present precisely the same actions on faces modified to look male or female provides a degree of experimental control heretofore unavailable. Thus, the article by Meister et al. provides a kind of “proof of concept” that can in principle be applied to other socially-relevant physiognomic and behavioural characteristics.

Increasingly in recent years, we have become aware of the downside of the explosive developments that have occurred with information technology. Nevertheless, some of the capabilities it has given us have allowed for advances that it is hard to imagine having taken place in other ways. Indeed, as other recent work involving the use of avatars and other complex features enabled by information technology has shown [14], it is becoming increasingly possible to model real-life social interactions in the clinical setting and, in so doing, both deepen our understanding of the social nexus of pain and to identify ways of improving clinical practice. The paper by Meister et al. adds to this emergent field. As a proof-of-concept, it is likely to stimulate numerous novel lines of research.