Computational research and the case for taking humor seriously

1.1 Groundwork for computational research

The April Fools’ Day Workshop conceptualized computational research on humor as a field that would naturally engage with several others.

HCI gave research practical motives. In keeping with the case for taking humor seriously, it provided evidence against the assumption that introducing humor into serious tasks could only deflect attention, and impair performance (Morkes et al. 1999). Raskin (2002) summarized the applications that were foreseen, using a list due to Stock, with two additions of his own:

1. business world applications (such as advertising, e-commerce, etc …);

2. general computer-mediated communication and human-computer interaction;

3. increase the friendliness of natural language interfaces;

4. edutainment and autonomous agents systems;

5. customer acceptance enhancement; and

6. humor detection.

Linguistics provided a highly developed theoretical analysis of patterns that generate amusement, derived from the General Theory of Verbal Humour of Attardo and Raskin (1991), and represented in the workshop by Attardo (2002) and Raskin (2002). Broadly, the patterns involve partial parallels between two semantic structures. The starting point for a joke is a reasonably natural way of putting most of the elements on one side into correspondence with elements on the other. However, the correspondence invites us to pair up the remaining elements on either side: and that pairing is strikingly anomalous. The combination of match and anomaly is what prompts amusement. Predictions from that framework were supported by empirical research (Ruch et al. 1993). Conference papers reflected work which had already used related ideas to generate riddles, the Jape system (Binsted and Ritchie 1997). Related ideas were applied to amusing visual stimuli (Giora 2002).

Empirical contributions highlighted the significance of context. Ruch (2002) stressed that different kinds of humor characterize different kinds of personality, and are received differently by them. Ben Ze’ev (2002) considered the different roles that humor plays in different media, and the enhanced role it plays in online communication.

Research on emotion also featured, though less systematically. The view that amusement is a form of pleasure was occasionally explicit (Giora 2002), but often implicit. Nijholt (2002) proposed a more theoretical conception: humor as a form of appraisal, involving patterns that are anomalous, but not harmful. He also noted its role in expressing emotion acceptably: “if [artificial agents] are going to show emotion, we surely hope that they would show a little humour too.”

The range of resources that were considered underlines the scale of the challenge. Looking at the result, again broadly, it seems fair divide it into two. Particular directions have developed strongly. The achievements there serve as landmarks. Elsewhere, the significant achievement is clarifying targets that are important, but challenging. The next two sections look at those in turn. That provides at least a broad basis for assessing how attempts to match these particular human abilities contribute to understanding ourselves.

2.1 Comedians

‘Comedian’ is a natural term for someone whose goal – at a given time – is to make people amused. Most people do not adopt the role very often: humor usually arises in the course of pursuing other kinds of goal (or abandoning them). In contrast, it has been very prominent in computational research.

The field’s early successes were in that vein. For example, the Jape system (Ritchie 2003) produced punning riddles such as:

What is the difference between a pretty glove and a silent cat?

One is a cute mitten, the other is a mute kitten.

The evaluation was revealing: Jape’s best outputs were compared with jokes in children’s books, and children found them about equally amusing (Binsted et al. 1997).

Varied formal patterns were explored. The HAHAcronym generator (Stock and Strapparava 2002) generated amusing misinterpretations of acronyms by choosing words with associations diametrically opposed to the originals: so, MIT becomes “Mythical Institute of Theology.” In similar vein, Valitutti (2009) explored creative variation of familiar expressions: so “Tomorrow is Another Day” became “Tomorrow is Another Bay.” Tinholt and Nijholt (2007) made a significant innovation: the system interacted with humans, and took up opportunities to make formulaic jokes that arose from what they said. Its formula detected the referential ambiguity in a sentence like “The cops arrested the demonstrators because they were violent,” and highlighted the anomalous reading: “What? The cops were violent?”

Several of those systems have a second level. The first generates texts in a form that can be funny: the second evaluates its outputs, using various criteria. HAHAcronym checks for contrast in meaning between the original phrase and its alternative; Hempelmann’s (2003) punning system checks how close the punning sound is to the original; and rules involving the familiarity, concreteness and imageability of the descriptions were used to select outputs from Jape that were likely to amuse raters. It seems possible that humans do something similar.

Jape’s output was compared with material from children’s joke books: matching comic performance is a natural follow-on. Efforts in that direction are summarized by Nijholt (2018). They are strikingly unlike earlier comedic work in some respects, strikingly like it in others.

Nijholt notes a striking dissimilarity: the performance systems make limited use of techniques for generating jokes. Scripts are generally provided by humans; the system’s task is delivery. The striking similarity is that there are staple patterns to guide delivery as well as content.

Manzai comedy provides one set of staples, involving exaggerated behaviors, unexpected actions, interactions involving imitation or misunderstanding, and some kinds of verbal humor. Examples involve interactions between a robot and humans (Kishi et al. 2014), and between two robots (Umetani et al. 2017). Another robot comedian used four ‘performative gestures’ derived from the ‘stand-up’ style: an opening welcome gesture; a reprise (“I said hello”); pointing to the audience; and applause elicitation at the end of the performance (Katevas et al. 2015). Underpinning all of those are various levels of interaction with the audience: looking at individuals, detecting amusement, using their prompts to choose among available topics.

The state of the art in this area is a revealing marker. We might naively assume that since a General Theory of Verbal Humour was articulated several decades ago, computational research would by now have translated it into scintillating humorists. The actual state of the art underlines Stock’s assessment: “deep modeling of humor … is not something for the near future.”

2.2 Laughter

Laughter is a natural step beyond textual humor. It is the obvious sign of amusement, and effective performance comedy depends on detecting it. Since it is infectious, it is also a means of inducing amusement. But as with comedy, exploring the subject led well beyond the obvious.

Basic parts of the work could, in principle, have been done for pure scientific curiosity; but in practice, computational aims drove it. On one side of that is machine perception: finding ways to derive useful descriptions from sensors. That depends on data, and so computational needs prompted the creation of laughter databases (see Dupont et al. 2016). On the other side, there is the problem of working out when and how an agent should laugh.

Recognizing laughter sounds simple, but it turned out to be intricate (Kantharaju et al. 2019). A first stage is separating standard speech from pure laughter, voiced and unvoiced; and speech overlaid with laughter. Out of a range of tools, visual as well as acoustic, the most effective there was a battery of acoustic features used in other emotion detection contexts, eGeMAPS. For the next step, identifying the emotion carried, only voiced laughter and laughter in speech gave useful indications. They came from combining eGeMAPS features with visual information. That combination also distinguished between spontaneous and acted laughter.

Moving to another level again, humans judge that laughs differ in intensity. That judgment turns out to depend on multiple acoustic features, including voicing; but also duration, the existence of harmonic structure, and different features related to pitch (Rychlowska et al. 2019). Intensity then features in another level (Curran et al. 2018). High-intensity laughs appear to have an intrinsic meaning; but low-intensity laughs acquire meaning from the context (so their perceived meaning will change if they are inserted into a new context).

The inverse problem is to establish how and when to generate laughter. Recent studies point to three natural ideas. Mancini et al. (2017) describe an agent that laughs in response to humor in external stimuli (pieces of music). Their question is how the laughter affects the perception of a human who is also listening. The agent in Niewiadomski et al. (2013) uses a contrasting strategy, qualified imitation: it laughs in response to its human companion’s laughs, somewhat (but not exactly) similarly. Piot et al. (2014) take a third approach: their agent’s laughter is governed by a learning algorithm, which takes account of situational variables and a human partner’s responses. It seems reasonable to assume that all three strategies capture something significant: combining them is another level of complexity.

Laughter illustrates the general theme particularly sharply. It is an aspect of life that is prone to be regarded as crude, even animalistic. But trying to match what we do reveals that it is bewilderingly complex. It only appears simple because the subtleties are handled outside conscious awareness.

2.3 Irony

Irony came to notice as a feature of online texting. Work there (e.g. Valitutti and Veale 2015) showed how simple devices – scare quotes and dramatic adjectives – could induce a ‘valence shift’ in attitudes to an underlying message (e.g. The vegetables are mixed in “healthful” salads). The topic attracted considerable interest in human-robot interaction. It has several attractions there. It is a form that can be integrated into communication, rather than standing alone, as a joke does. It also relates to a particular problem, which is orienting a human to a robot’s mix of abilities and limitations. A ‘valence shift’ that makes a joke of failures applies neatly to that problem.

A dramatic demonstration was provided by two robots, which responded to failures in contrasting ways (Mirnig et al. 2016). Both were humorous, but in one it took the form of schadenfreude, mocking the other’s failures, where the other used irony to mock its own. The second was rated more likeable. That is a useful counter to the assumption in early research that humor per se is the key to evoking positive reactions. Clearly, the kind of humor matters too.

‘Irony man’ (Ritschel et al. 2019) used prior theory to develop the approach. The basic tool is ideational reversal, where “the intended meaning arises as a result of negation of a chosen element of the literally expressed meaning or the pragmatic import of the entire utterance.” Instead of ‘scare quotes’, the key element is marked by signals in various modalities – exaggerated or understated language, prosody, gaze, facial expression, and gesture.

A further feature is that the robot uses irony in the context of a discussion with a person and chooses when to use it. A simple principle governs the choice. Irony is used to signal agreement with an emotive statement made by the user, by repeating the statement with an inversion that is signaled as ironic. For instance, the system responds to “I hate raining” by saying “Super! I utterly love raining” – with suitable markers on ‘love ,’ The simplicity is valuable, because it allows the kind of learning that has already been mentioned in connection with laughter, aimed at recognizing contexts where the tactic is likely to work.

An unusually broad evaluation indicates that the basic form does work. It compared versions of the robot with and without irony. The ironic version was perceived as more natural, and more pleasant on a measure covering a range of subjective evaluations – more stylish, presentable, attractive, appealing, and likeable.

2.4 Outcomes

Irony research illustrates a last major area of progress: clarifying what systems achieve. It is useful to draw a rough line between two generations of work.

In the first generation, the issue was global: whether computational humor could have positive effects. Varied approaches converged on a positive conclusion. Babu et al. (2006) studied responses to a virtual receptionist, Marve. Users could choose how, and how much, they interacted with him. Listening to his jokes was much the most popular option, accounting for 50% of interaction time. Dybala et al. (2009) compared two versions of a conversational system, identical except that one (Pundalin) told a pun every third conversational turn. Pundalin received predominantly positive affective responses, the humorless system predominantly negative. Similar evaluations continue: e.g., Mohammadi et al. (2019) compared agents that play dice with users, one which interacts humorously, the other competitively. The humorous agent was judged more animated, likeable, and empathic, and more able to experience a range of emotions.

Computational humor certainly can have positive effects. But, notoriously, humor out of place can go badly wrong. The second generation aimed to disentangle appropriate and inappropriate contexts.

Khooshabeh et al. (2011) considered humorous and humorless versions of an agent designed to persuade users. Different users responded differently. When users did not like the humor, it made them less likely to be persuaded. Karpinska-Krakowiak et al. (2014, 2018) highlighted the growing use of pranks in online advertising. Those effects too are unstable: they put off people who identify with the victims of the pranks. With schadenfreude humor (Mirnig et al. 2017), the reactions divided along gender lines. Torre et al. (2019) considered a subtler division, using a game where an avatar suggested solutions to a problem, accompanied by a smile. A smile signaled only visually made its suggestions more likely to be accepted, but adding smiling voice made them less so. A degree of neutrality increases trust.

AI offers an obvious response to the finding that both positive and negative outcomes are possible. It is to use machine learning techniques to ensure that humor is properly placed. Examples of that approach have been mentioned, involving laughter and particularly irony (Weber et al. 2018). However, that exposes another level of challenge: how to understand exactly which outcomes are desirable, or how they are signaled; or along which dimensions situations, and users, need to be distinguished. Those point back to the questions about the context of humor that were raised in the April Fool’s Day workshop.

3.1 Pervasive and emergent humor

Emotion research highlights an issue that extends to humor. The word ‘emotion’ tends to call to mind distinctive states where emotion interrupts rational thought: ‘emergent emotions.’ Those contrast with another distinctive state, being unemotional. However, most of everyday life is neither. The norm is ‘pervasive emotion’, which colors the way people think, and act, and interact, without suppressing reason (Cowie 2010).

Pervasive humor is an implicit challenge reflected in several strands of computational research. Irony is a form of humor that can pervade interaction (unlike constant set piece jokes). Acceptable pervasiveness depends on taking up opportunities that interaction and context offer, either to laugh (Piot et al. 2014) or to make jokes (Wong and Gonzalez 2016). Linking humor to personality recognizes one of the major inputs from psychology at the April Fool’s Day workshop: a pervasive style of humor is one of the things that distinguishes a person (Ruch 2002).

All of those represent progress into the ground between humorless systems and comedians. The limits of progress make a very enlightening point. The middle ground is not a routine compromise between the extremes. Mastering the relevant blends and contingencies is extraordinarily difficult.

3.2 Data

AI in general began with a hypothetico-deductive model: machines derived the implications of principles formulated by humans. That wave – ‘good old-fashioned AI’ – was superseded by techniques based on learning (Haugeland 1989). The general pattern extends to computational humor: hypothetico-deductive approaches were prominent in the April Fool’s Day Workshop, and learning has come increasingly to the fore.

Learning techniques invite a shift from symbolic representations of meaning to probabilistic networks, and that has quite subtle consequences (Hempelmann and Petrenko 2015). More straightforwardly, learning depends on access to large quantities of reliable data. Acquiring it is a much bigger part of the enterprise than people outside the field commonly recognized, not least because the raw material must be labelled to reflect people’s responses to it.

There are substantial bodies of data in some subareas. Online sources have been used to build databases of jokes and funny dialogues, spanning different languages. For example, the HAHA challenge compares systems’ ability to learn from a database of 30,000 annotated Spanish tweets (Chiruzzo et al. 2019); and Blinov et al. (2019) describe a Russian dataset including more than 300,000 short texts. For laughter, the ILHAIRE project left a legacy of annotated databases, spanning different contexts and cultures (Dupont et al. 2016). That effort continues (Ginzburg and Pelachaud 2018).

Other areas use varied forms of data. Radev et al. (2015) considered cartoons and their captions using over 400 cartoons and more than 2 million proposed captions, acquired from New Yorker competitions. Oliveira et al. (2016) generated amusing picture/text combinations using a stock of preselected images and subjects drawn from newspaper headlines. Systems that learn by interaction, mentioned earlier, effectively use another form of data: each item pairs an effort at humor with an evaluation of listeners’ responses.

The issue is an illuminating one. Despite the effort involved, available resources only partially address specific subtasks. The degrees of freedom involved in modelling pervasive humor would be of a different order of magnitude. It is hard to imagine the amount of data that would have to be collected, or how it could be labelled. Progress in that direction matters for understanding ourselves, though. Learning involves finding solutions to the problems that a database defines. Restricted databases may well allow solutions quite unlike those that let people find humor in hugely diverse situations. Building databases that direct learning towards human-like solutions is a non-trivial part of the challenge.

3.3 Extending modalities

Theory addressed visual jokes long ago (e.g. Giora 2002), but early computational work was entirely verbal. Extension to other modalities is an ongoing challenge.

Humor generation provides examples that have already been mentioned. In Manzai comedy, posture and gesture function like jokes. In irony, visual and prosodic markers enable verbal humor. The modalities also work together in picture/text combinations. A relatively recent point is that verbal and non-verbal signals are often interchangeable rather than additive (Mirnig et al. 2017). Humor in video games uses multiple modalities, up to and including visual slapstick (Hessler 2019).

Multimodal perception is much harder. It is used to detect a human’s amusement, or laughter. Recognizing what is amusing is a different matter. The study of New Yorker cartoons is illuminating. The cartoons only entered the analysis by way of verbal descriptions identifying the objects in them (which were provided by the researchers). That is far short of implementing theories of humor in cartoons, like Giora’s; or registering the slapstick, or funny incongruity, in video clips.

Again, that relates to pervasive humor. One of the everyday benefits of a sense of humor is the ability to see the funny side of a situation that could be distressing. Matching that would need major advances in perception.

The issue is not new. Nijholt’s paper in the April Fool’s Day workshop considered humor as an appraisal. Its business is detecting that a situation violates expectations embedded in plans, or standard images of things: but it is not threatening. The associated action tendency could plausibly involve suppressing responses to the apparent threat, and giving positive signals. That portrays humor as critiquing our representations and projections. It implies a kind of double perception, of reality and of the way we represent it. Implementing that is a long-term project.

3.4 Applications

Applicability was crucial for the April Fool’s Day conference, and it is intimately linked to the case for taking humor seriously. Applications have materialized, though not quite as anticipated.

Stock’s list did not mention gaming, but it is an application that became established early on (Dormann and Biddle 2009; Grönroos 2013). Applications in education, which did feature, followed a related path: humor is a regular element of games designed to teach (e.g., Pezzullo et al. 2017). The usefulness of the technology has to be assessed case by case (Foldager et al. 2016). Another application area involves ‘artificial companions’, particularly to supplement human support in long term care environments. In a forum on the subject, Romano (2007) began her contribution: “A truly good friend is the one that makes you laugh.” It has been seen as a role where agents could use techniques derived from stand-up comedy (e.g. Kishi et al. 2014).

However, the outstanding applications reflect Stock’s concern to “increase the friendliness of natural language interfaces.” Chatbots have become an established form of interface (Grudin and Jacques 2019), and humor is an established tool. Significantly, it is a feature of the major IT companies’ chatbots, Cortana, Siri, and Alexa.

The major chatbots cast a revealing light on the state of the art. On one side, their use of humor puts it beyond doubt that the technology has applications. On the other hand, it highlights the limits of the technology. The outstanding examples rest on the same theme. Humor is part of larger wholes; and assuming that it can be detached from the wholes, except in special circumstances, is misunderstanding it.

In Luger and Sellen’s memorable description, using a chatbot can be “like having a really bad PA” (2016). They found that systems could not live up to the expectations they raised, resulting in disillusionment and curtailed interaction. Humor played a part in that. It encouraged playful interactions early on. A detailed study by Liao et al. (2018) noted a similar pattern of initial playfulness, which humor promoted. However, the initial interactions did not simply generate positive affect. Users understood them as a sign of a larger whole, involving a range of competences: and reacted negatively when they realized that the system did not have those qualities.

Alongside the work on software agents, there is emerging work on humor in interactions with physical robots (including work on irony that was mentioned earlier). It raises an intriguing question: whether different embodiments require different humors.

Practical experience also underlines the importance of another whole: that is, personality. The humor in the major companies’ chatbots is not wholly computational. The systems that deal directly with human users accumulate large amounts of data and learn from it. But they are backed up by human teams, who monitor and develop response patterns. The process, described by Elgan (2016), is “a lot like novel writing. There᾽s no story or plot, but there is one character, which has a gender, an age, an educational level, a particular sense of humor and more. That human-created character or personality has to remain consistent across queries and contexts.”

The linkage between personality and humor is no new discovery: Ruch (2002) underlined it at the April Fool’s Day workshop. What has emerged is how challenging it is to ensure that a sense of humor is internally coherent and coheres with a wider personality. For that, the best-resourced teams still rely on creative human intuition.