Analyzing the semiotic nature of GIFs: visual nominalization and visual telicity

3.1 Multimodal polyphony and interaction

Kress and van Leeuwen (2006) note that “[w]hen there is a disjunction between the context of production and the context of reception, the producer is not physically present, and the viewer is alone with the image and cannot reciprocate – an illuminating exception is the case of the ‘defacement’ of billboard advertisements, when graffiti artists ‘respond’ to the initial ‘turn’ or statement of the image” (pp. 114–115).

In the realm of GIFs, such situations are not “exceptions” but are in fact, pervasive. GIFs are responses to the initial turn of the source film or video. It is a whole semiotic mode whose existence necessitates reciprocation to some degree. Compared to the still image and the film, the GIF is in a position of having more possible complex relations among interactive participants and between interactive and represented participants. In the act of creating the GIF, the GIF creator responds to and quotes the source. Many of film’s visual resources percolate to the GIF form: the gaze, the type of shot, the camera angle, the composition, the sequence of shots, and the gesture of the represented participants all generate interactive meaning. The GIF may contain an additional level of meaning from the content added by the GIF creator. This can include, and is not limited to, alterations to sequence and speed; cropping and resizing; changes in focus through partial freezing or zooming; alterations in color and saturation; superimposition of additional images or splicing from multiple sources of media; and superimposition of text.

Gürsimsek (2016) identifies GIF as a form of vernacular expression, as opposed to institutional or official expression, and GIF creators as “transmedia audience members who do extreme close-reading of narratives, performances, and settings in order to isolate specific moments that are GIF material” (p. 347). The extreme close reading, in selection, is passing judgment on what is shareable, what is valuable beyond its context, what can serve as a possible sign, and what can undergo resignifying.

The act of curating is itself significant. At the extreme, the selection of which part of the source film to extract can itself create a change to a “default” meaning interpretation that may differ from the original contextual meaning. Even when such transformations are less obvious, curating signals motivation.

The layers of this semiotic artifact can be understood through the lens of Bakhtin’s (1981) concept of “double-voiced discourse” which originally referred to literature, but seems applicable here as well: “It serves two speakers at the same time and expresses simultaneously two different intentions: the direct intention of the character who is speaking, and the refracted intention of the author” (p. 324). The represented participants, the visible characters in the film source, refract the intention of the director or the editor of the original film; these voices undergo additional refraction upon GIF creation. “Double-voice discourse” can serve as a model for understanding the multi-voiced layers of multimodal communication: modifications and additions by the GIF creator may serve to amplify, challenge, or repurpose the meanings of the film source. The modifications and additions by the GIF creator can serve as signposts for contextualization, or specification of the otherwise polysemous visual sign, providing a frame for interpretation, when the GIF gains a third level of voice upon deployment. Indeed, as a vast quantity of GIFs on the Internet remain anonymous creations, the GIF creator’s voice can also be thought of as a disembodied voice for which the user may try on, upon selecting and using the GIF. Such selection can hinge upon identifying and aligning with the GIF creator’s judgment of curating and commentary.

Additions by the GIF creator typically occupy the same “canvas” or visual space as the source, such that even when differing meanings can be discerned, the holistic GIF presents a complete message. These changes affect the overall visual nominalization meaning and the degree of how participants are backgrounded or kept salient, which then shapes or limits how a GIF may be used in a bigger multimodal discourse. Polyphonous artefacts hold potential for subversiveness, playfulness, and punning, such that, the same time span from the same source film content can result in, from the modifications by different GIF creators, very different end products in the way they express meaning.

When the structure of the processes as well as the identity of the actors and participants in the original is preserved, and additional content serves to reinforce the meaning, emphasize or draw attention to part of the meaning, the interaction can be called refinement. This manifests in image-image interaction as pictorial elaboration through ways such as the partial freezing of the areas in the frame deemed unimportant; zooming in; and pictorial annotations such as circles and arrows drawing the focus on a part of the image. In the image-text interaction, refinement can occur as semiotic repair, where words are used as accurate subtitles to replace the lost audio, or as resemiotization, where captioned words describe the action in the GIF.

When new identities or roles are encouraged for some or all of the main participants, multimodal metaphors, analogies, and personification can be created. This interaction can be called semantic reframing. Semantic reframing takes inspiration from Fillmore’s semantic frames (cf. Fillmore 1982). The process is visually unchanged, but conceptually changed or reevaluated, due to the changes in participant identity.

Semantic reframing manifests in the image–image modality interaction as substitutional superimposition, where the GIF creator places an image over the represented participant; and in image–text interaction as role labeling, where words are superimposed on or are in close proximity to the represented participant. A trait of reframing is that the added image or text is typically dynamic to ensure a consistent relative location with respect to the original represented participant. Pictorially, the original represented participant’s head is commonly the location covered up, as the face is associated with identity. If the original represented participant is an inanimate object, the central area of the object or the area undergoing action may be selected.

Figure 3 is an instance of role labeling, where the words supplied by the GIF creator interact with the source footage to create a verbo-pictorial metaphor. It shows the sequence of three dolphins leaping out of the ocean one after the other, with Saturday, Sunday, and Monday in bold capitalized white text directly superimposed on the bodies of the dolphins, moving along with them and disappearing when the dolphins vanish out of view in the water. As the “SUNDAY” dolphin is mid-leap, the “MONDAY” dolphin emerges right below it, crashing into it and butting it off-course. We can interpret the metaphor as: days of the week are dolphins. Inanimate abstractions of days are given corporeal form. The sequential leaps become a visualization of the passing of time, and the ill-timed leap of the “MONDAY” dolphin colliding with “SUNDAY” expresses the disruption of the weekend by the new weekday.

Figure 3:

Voice interactions creating metaphor within GIFs. GIF source (left): https://giphy.com/gifs/monday-fail-weekend-3KVTDgHuQSQM. GIF source (right): https://giphy.com/gifs/weekends-ApJaIsTFQtJOU.

Such labeling can result in metaphors with richness and subtlety. Consider a similar GIF that shows through interaction a metaphor for days of the week. The sequence is of a young man doing various gymnastic flips, while white letters hover around him: for the duration of one flip, the word is “Saturday…” the next flip is marked “Sunday…” and the last moment, a direct drop onto the floor, is labeled “Monday…” The words change location to be close to the young man, which serves to bring him into focus, counterbalancing the impersonal social distance brought by the long shot of the sequence. That the words are always hovering close to his head, like in a thought bubble’s position, rather than superimposed on a central location of his body, suggests that the man is not the one transformed, but that the segments of his experiences are transformed. The metaphor appears to be: the experiencing of the days is the performance of the gymnastic sequence. What is labeled is invisible; it is a segment of action. The ellipsis in the text, indicating incompletion, connects to the ongoing sequence of each gymnastic maneuver, mapping the success of the day to the success of the movements, in terms of the height achieved and the smoothness of the performance.

Both metaphors express a negative view about Mondays following the weekend, but in the dolphin GIF, we see subjective perception as objective observable phenomena, and in the gymnastics GIF, we see mental perception as actual bodily experience. Each GIF exploits the traits of their respective source materials to add different nuances and complexity, for example, in their treatment of agency and causality. In the dolphin GIF, wildlife in their natural habitat act without human interference; there is an absence of the experience of days represented in the frame, and there is just the perspective of the passive GIF viewer, which metaphorically suggests a lack of agency: flawed Mondays are a phenomenon dictated by the forces of nature and are not in human control. The ill-timed collision by the Monday dolphin is not the fault of the preceding dolphins but simply due to its own caprice. In the gymnastics GIF, the depicted man is taken to be the experience of days, and expends effort to launch himself into the routine, thus allowing the metaphor to retain a sense of agency: the flawed Monday is of man’s own making. Flips deplete his energy, and each flip builds momentum, which challenges his balance and control. The metaphor retains causal connections, showing that the failure of Monday is to an extent the result of a cumulative effect of the previous days.

The visual nominalization and visual telicity of the GIF facilitate a play of the visible and invisible, packaged and compressed into the brevity of the GIF that may otherwise take many words to fully describe. The surprise of Monday upon the first viewing evens out to predictability, and when the incidental becomes the routine, the humor in unexpectedness ripens to humor in obstinacy. The Monday dolphin’s mistimed leap, originally appearing accidental, may start to seem scheduled and premeditated. The gymnast is forever with an imperfect last flip, as if such a situation is always inevitable.

The interaction of voices enhances our perception of the metaphor as opportunistically clever. The viewer recognizes the metaphor created as an opportunistic commentary. The discrepancy between the two voices is key for enhancing humor. No matter whether the source voice is a casual video or a cinematic production, it is answered by a voice that is consciously public, anonymous, and vernacular, and transformed into something malleable for other kinds of texts. When such connections are made, specificity and abstraction collide.

Arguably, the nature of the process can most visibly be disrupted through means like extreme speed adjustment and frame order reversal. Such drastic disruption can be spotted as unnatural, and the interaction of voices can generate two concurrent meanings: the meanings of the adjustment, and the meaning of the original. For example, a sequence depicting an individual entering a room can be reversed and transformed into a sped-up curious retreat. Kress and van Leeuwen (2006) identify three kinds of circumstances in images: location, means and accompaniment (p. 77). Video, through their sequential frames, can express some more kinds, such as purpose and cause. GIFs can alter circumstances, such as splicing source material with additional content to imply new causality relations. Misrepair of audio in purposefully incorrect subtitles and opportunistic misresemiotization of the action may successfully obscure or contradict the process, although not entirely change it. We recognize that the source has been “tampered” with. These interactions of voices can create irony, satire, and memes.

Memes are widely circulated social artifacts in the form of images, videos, and discourses. Some GIFs become memes, while some GIFs are created in the model of an existing meme, being a variation of a meme. GIFs belonging to the same meme undergo similarities of modification. The meme, “Deal With It”, often involves these three words captioned, along with the participant putting on sunglasses, or sunglasses descending to rest on the participant’s face. Footage of people putting on sunglasses in typical circumstances unrelated to the meme can become subject to this meme transformation. This owl GIF (see Figure 4) is an instance where the meme message created by the GIF creator disrupts the original source, which is from a documentary film. The close shot of the documentary spotlights the marvel of the owl’s head rotation, but the GIF creator seizes the fluid motion for its atelic presentation potential. The editing of the glasses to appear on alternate rotations, with no hands and no clear indication of how the glasses ended up upon the owl’s face as its face is rotated away from the viewer, is a visual equivalence to the sleight-of-hand of glasses descending from above. The source footage becomes secondary and subservient to the meme in the interpretation of meaning. When the meme is a constant, details of the participant that may be backgrounded then have a possibility to be considered more relevant. When multiple variations of a meme GIF exist, the distinguishing details among them gain significance. There is humor in wrestling the content and subjects of copyright into the frame of the uncopyrighted meme culture, in the reduction of public figures and fictional characters to shorthand for subtleties to be added to the same meme message.

Figure 4:

Meme GIF. GIF source: https://giphy.com/gifs/cat-love-u6z0UkJQnnWzC.

The voice of the original artefact has undergone changes through the GIF creator’s additions. Going back to Kress and van Leeuwen’s (2006) example of a graffiti artist defacing a billboard, where it is an individual adding onto a more public or authoritative content sanctioned and created by teams, the polyphony of a GIF can be more dexterous. It can either serve in a similarly graffiti-like manner, a renegade anonymous “defacement” of a content of a formal, public content. It can also serve to elevate informal, personal recordings to become abstract, not just a documentary of some moments but potentially new signs for public circulation. The opportunistic nature of the viewer reciprocation to the producer appeals to the secondary viewer, the end user who ultimately selects the GIF to become a part of a bigger multimodal discourse.

In theory, there can be instances where there may be ambiguity as to whether some image or text element has been added or is part of the original. What is relevant here is what the culture of polyphony leads the end user of the GIF to believe in. Not everyone is a graffiti artist. However, everyone can experience vicarious exhilaration of a graffiti artist, a visual responding to the turn of various public and private images and videos, by tacitly aligning with the voice of the GIF creator upon selecting the GIF for use. Because the GIF is used not merely to exhibit its literal meaning, but can itself have communicative functions in use, so the voices of the GIF become transformed and embedded in the final voice of the user.

3.2 Transcontexualization

Visual nominalization and visual telicity mediate how context contributes to meaning in a GIF. Context is the situational and cultural background knowledge relevant to happenings and discourses, and it is dynamic, consisting of potential meanings derived from shared subject matters, cultural conventions, and participants’ experiences. In the case of the GIF, some of the relevant context may include its placement in relation to the whole sequence or structure of its source video or film; the source video or film’s genre; and status of the source video or film in culture. The meanings of the GIF are richly dependent on its cumulative layers and on the ways users highlight and access parts of its layers, and context is never fully removed, but is in fact a resource that transforms. The GIF can be said to undergo transcontextualization, a blending of contexts such that the old original context becomes a reference point for understanding the new contexts. Transcontextualization is in some ways an extension of the meaning organization of visual nominalization. As depicted participants are backgrounded and the process of the action or gesture foregrounded, the identity of the participants becomes fluid as participants can become stand-ins for the communicator. However, the identity is not completely erased, and instead may be understood as resources qualifying the understanding of the action. For example, children and small animals can become qualifications of innocence, harmlessness, or playfulness; certain occupations can stand in for authoritativeness; well-known public figures and characters may have associations that can stand out as attributes. Stereotypes and cultural generalizations enable the transformation of identities into more subjective qualities. Similarly, the context of the original GIF undergoes a transformation for additions of meaning. The context of the original GIF refers to the immediate situation of it in the moment of the narrative, as well as the nature of the source work as a whole, its genre and type. Qualities of a genre can become meaningful qualifiers. Fictional genres may give a sense of “life as play-acting,” while documentary news genres may convey seriousness or mock-seriousness. The disparity between the original context and the new context the GIF can be used to provide a tension that powers the humorous possibilities of the GIF. Since the identity of the depicted participant is naturally associated with the genre and the context, transcontextualization builds upon the abstract transformative powers of visual nominalization.

There is, indeed, a genre of transcontextualized GIFs where users create a multimodal discourse by supplying the context with written language, a practice so popular it has engendered abbreviations such as MRW (my reaction when) and MFW (my face when).

However, transcontextualization of GIFs cannot be discussed without mentioning the special instance of one-to-one messaging, where the GIF’s new context is not provided by the GIF user, but by the conversation partner. Gif repositories such as GIPHY, Gfycat, and Google Tenor not only have successful integration into many main social media platforms, but also have keyboard apps with phone messaging systems that allows standardized availability, wide distribution, and repeated use, and instant use. There is minimal effort on the users’ part, as there is no need to create the GIF from scratch. The user instead can evaluate the nature of the discourse context to determine what kind of reaction may be suitable, and type keywords in the repositories’ search function to browse GIFs. The process can become a comparison of contexts and a gauging of how GIFs’ source contexts can be used as a resource to indicate the user’s attitude toward the current context. Transcontextualization is also what enables some GIFs to become memes, as each context of use, if visible to many, can become a further reference, such that the history of contexts can even eclipse the original context.

In the previous section, we addressed the interactions of meaning between the content of the source and the content added by the GIF creator. The source content and the GIF-creator-added content share the same semiotic space; they are layered; they are superimposed. When the GIF user deploys a GIF in a multimodal discourse, the GIF’s relation to new content is now juxtaposed; it is serial, instead of sharing the same semiotic space in the way of a digital palimpsest. Furthermore, technology constraints the format and layout, such that certain relations (an image’s display size, its relative orientation with respect to writing) that may have significant meaning in traditional contexts (print advertisements, books) now simply are a product of platform design. For example, the fact that GIFs cannot be inserted in line with text in the same way that emoji can be, limits its function as a replacement for a below-the-phrase unit of written language, and instead shifts it towards functioning as a complete speech act or a complete “phrase.”

Although GIFs become somewhat decontextualized by virtue of their brevity and derivation, there is a market for partially determined new context. Just as there is always a demand for news and facts related to current events of large scope or influence, so there is also a market for pre-formed opinions and stances on such topics for people to quickly identify and align to. Convenience of expression can be appealing in the way of the shortcuts of partially assembled or frozen meals. As new situations can be understood by finding connections to old and familiar content and constructs, so previous footage or GIFs from past media and events are brought in, as if from the Internet’s collective memory, to situate new events and topics. Decontextualization may be only in the literal sense, for a GIF can never be fully erased of the nuances of the contexts of its origin and the contexts of patterns of past use, a multimodal palimpsest of the digital age. Recontextualization is never a true replacement of context. Transcontextualization expresses the incompleteness of the two processes of decontextualization and recontextualization.

3.3 Dual deixis

One effect of GIFs’ transcontextualization is dual deixis, which is facilitated by GIFs’ visual nominalization and visual telicity. GIFs can be multimodal in nature, and many may include written text alongside the image. The words can be from captions originally created in the film or video source, or added by the GIF creator as commentary or to provide intersemiotic repair to account for lost audio.

A significant amount of GIFs in repositories contain words falling in deictic categories. Deixis is the system in language where referents of particular words are dependent on the situation and positioning of the speaker. This system contains egocentric deixis of person, place and time, discourse deixis of text and composition, and social deixis (c.f. Levinson 1983; Lyons 1977). In addressing the deixis of literary texts, scholars of cognitive poetics note in Deictic Shift Theory (see Duchan et al. 1995) that texts can cue readers to create deictic shifts in distinguishing perceptual deictic centers (Stockwell 2020). It is not a far cry to note that the same can occur in visual discourse, i.e., films and videos. The viewer of the derived GIF understands subtitles and captions to relate to the deictic center of the depicted participant in the GIF. Subtitles containing “I” will have its referent as the depicted participant, “you” to an addressee that is often absent from the GIF but whose identity is accessible only through knowledge of the source film or video.

The author argues that GIFs in use allow a double shift of deictic center regarding egocentric and discourse deixis – a dual deixis. That is, there is a deictic shift into the world of the GIF, but also a reshift back to the GIF user as a deictic center, motivated by visual nominalization and visual telicity. Without this double shift, GIFs would only function as quotes. The disruption of the narrative linear sequence and the abstraction of visual nominalization facilitate this deictic jump for the GIF user’s egocentric particulars to be relevant in determining meaning of the GIF in discourse. This goes hand in hand with the embodied enactment in GIFs without words, where the depicted participant is understood as representing the GIF user in enacting gestures and facial expressions. An instance of multimodal deixis in GIF use is the transformation of spatial deixis into discourse deixis – a deictic metaphor of sorts. Written language assumes spatial orientation of top–down to follow time progression, and in multimodal texts, a GIF with gestural deixis of pointing upwards can be deployed as discourse deixis to refer to an immediately preceding point.

In the situation where multiple GIFs in a repository may serve to express a particular response or gesture, the selection of a GIF can be motivated by visual details in GIF functioning as social deixis. Social deixis is when “the different relative social positions of utterer and receiver are marked” (Stockwell 2020, p. 359). Just as “accent and dialect invoke schematic social stereotypes about those speech groups” (Stockwell 2020, p. 362), so visual traits of the depicted person or character also invoke schematic social stereotypes. This can include stereotypes regarding race, gender, age, and occupation. In cases where the depicted person is recognizable as a public figure or a widely-known fictionalized character, specific associated traits become used as social deixis as well. This aspect of GIFs in use symbolically indexes discourse community schema.