The Functional Grammar of Dance applied to ELAN annotation: meaning beyond the naked eye

1 Introduction

The Functional Grammar of Dance (FGD) (Maiorani 2021a) is a door. It was created to open the possibility of studying, appreciating, and understanding the way dancers communicate with their audiences to people who do not necessarily have expertise or a background in dance studies or practice. It was designed to be used with a non-specialist terminology and developed through data-collection sessions of live performances by professional dancers in real rehearsal studios. In this article, we will introduce its model and its fundamental principles to show how this allowed for an expansion of Systemic Functional Linguistics (SFL) theory into the domain of linguistic ontology through the provision of an effective interface for the interpretation of movement-based communication provided by the FGD as an ontological framework. In this respect, by demonstrating how the application of the annotation system implementing the FGD in ELAN operationalises this interface, we will also show how movement-based communication is enabled as well as restricted by the body structures combinations that the materiality of the human body in interaction with the surrounding space allows us to perform. ELAN (https://archive.mpi.nl/tla/elan), developed by the Max Planck Institute for Psycholinguistics, has been widely used for annotating audio and video materials (e.g. Cowan 2017; Drummond and Wildfeuer 2020; Mondada 2014). The annotation system for dance sequences using ELAN was developed through the research carried out within The Kinesemiotic Body project, an international research collaboration funded by the AHRC (Arts and Humanities Research Council) in the UK and the equivalent institution in Germany, the DFG (Deutsche Forschungsgemeinschaft, ‘German Research Foundation’), which involved a team of researchers based at Loughborough University, UK, and a team of researchers based at the University of Bremen, Germany.

Finally, we will propose examples of annotation with three case studies, through which we demonstrate how we annotate dance sequences using ELAN software with a specifically developed controlled vocabulary. Through the case studies, we will also demonstrate particular features that emerge which would not be normally captured by the naked eye. We will conclude with a report on some interesting results obtained so far through the annotation work and with indications for future research.

2 SFL, the FGD, and Kinesemiotics

The FGD (Maiorani 2017, 2021a) draws on a theoretical framework where the principles of communication put forward by Halliday’s Functional Grammar for verbal language (Halliday and Matthiessen 2014) are integrated with notions of Multimodal Discourse Analysis (Bateman 2019; Bateman et al. 2017; Kress 2010; Kress and van Leeuwen 2020; O’Toole 2010) to address the analysis for movement-based communication and capture its flow in a systematic way. The FGD model has been designed to be used by scholars from different disciplines to research and analyse forms of communication based on the interaction between body and space, and unlike the traditional dance notation systems it does not require any previous knowledge of dance terminology or a special training. However, being based on the principles of SFL in its structure, the FGD is initially easier to use for scholars who share a basic knowledge of this theoretical framework.

The FGD was created and developed to provide theoretical and practical support for understanding how movement-based performances generate and communicate meaning, and it is flexible enough in its structure to allow for its adaptation not only to various styles of dance but also to different types of performance that involve movement-based communication, thus fostering interdisciplinarity. It is therefore not intended to be an alternative to traditional notation systems (i.e. Labanotation or Benesh notation), which focus on physical movement and its qualities (see respectively Guest [2005] and Causley [1967]), nor is it intended to replace video-recording of performances: all data that is collected through the use of the FGD has the dancer/performer as its centre and is not dependent on the point of view of a camera and its operator.

Why then create the FGD? Because it is a model that unlike other systems of annotation, analysis, or recording created mostly for dance and in general for movement-based performances, focuses on the communicative aspect of these performances: it analyses how meaning is created during a performance (it can be applied both to live performances and video-recordings or streaming), and it captures dancers’ individual role interpretations through dance discourse in fieri, and from their own point of view. Traditional notation systems are movement analysis systems rather than dance analysis systems whereas a ‘dance’ already implies the interpretation and communication of meaning through movement (see Adshead-Lansdale 1994: 16). Moreover, dance notation systems do not take into consideration the semiotic role of space and how it interacts with dance movement in a performance setting (Brandão 2017; Munjee 2015).

Like the Hallidayan Functional Grammar for verbal language, the FGD is based on the realisation of three types of meanings that account for the three main metafunctions of communication that all semiotic systems are supposed to fulfil: Experiential, Interpersonal, and Textual. Therefore, when communicating with movement, a choreography will engage performers in three communicative tasks simultaneously: respectively, representing a happening or an event, establishing and entertaining relationships, and producing coherent sequences of movements that allow them to do so. A choreography will also be designed and performed within a physical as well as socio-cultural context, which is what differentiates it from dance training as simple physical exercise. Following the principles of SFL, the context of a performance will comprise three dimensions: Field, the topic of the event or happening; Tenor, the type of relationships that are created and entertained; and Mode, the way the coherence of the movement-based communication is ensured.

These three dimensions of the context of a movement-based performance activate respectively Experiential, Interpersonal, and Textual meanings that will be realised by choreographic affordances, which are structures available for each choreographer to create a piece of dance.

Choreographic affordances, however, do not create meaning by themselves but in interaction with the performance space, which is characterised by meaningful areas populated by significant items. When dancers turn and extend in various ways their articulators (head, torso, arms, hands, legs, and feet) towards these meaningful portions of space, they create meaning by projections. A projection is “the interactive connection between body parts and space that generates movement-based communication” (Maiorani 2021a: 28). Projections connect extended body parts to surrounding people or objects, creating a meaningful visual interaction; in this way, projections allow us to visualise basic meanings that viewers of a dance performance will interpret within the context of the performance itself. For example, a dancer extending their arm towards a castle painted on a stage backdrop while turning their head towards another dancer will be addressing the other dancer and communicating to them the intention to go to the castle.

The notion of projection is phenomenological, and it draws attention to the ontology of dance discourse, which is limited by the materiality of the human body but also capable of potentially infinite semantic productivity because of the potentially infinite number of possible configurations of the contextual space with which the body interacts. The notion of projection partially draws on Peirce’s phenomenological description of signs in that projections are created ex-novo each time a dance takes place through choreographed movement, and they generate a very dynamic semiotic process, but they cannot be taken as symbols in themselves. Projections do not constitute in themselves what in Peircean terms would be the representamen. As visualisations, they cannot be simply translated into specific words, phrases or sentences, which would be a useless and meaningless enterprise (see for example Kjeldsen 2015: 119); projections provide visual cues. In the semiotic relationship that connects objects, interpretants, and representamen (see Peircean chains in Durst-Andersen and Bentsen 2021), projections function as connections between significant spots, objects, or people and dancers, and this connection works at the same time both as movement-based visual object and as representamen. The interactive power of projections also resides in the fact that they ‘pull’ the audience to contextualising basic, underspecified meanings and consequently provide them with more specificity, thus activating contextualisation as a discourse mechanism (Bateman et al. 2019). Whereas dance steps are carried out in physical space, which is what happens, for example, during a dance class when dancers carry out movement combinations to train their bodies in a specific technique, a performance choreography is created to be performed within a specific contextual space that is marked by meaningful areas where significant items (object, people, props) will determine contextual interpretation by the audience. The FGD effectively provides an interface to grammaticalize movement-based communication in context, thus allowing for its interpretation.

In the FGD, choreographic affordances realise meanings within the basic unit of analysis, the Move, which is “the smallest structural unit of motivated movement that marks the enactment of projections by separating them through the necessary flow of body parts” (Maiorani 2021a: 34). The Move incorporates the idea of movement flow across space: a move is made by performing physical movement across the physical space and by the interaction of the different dancer’s body parts with the contextual performance space, an interaction that is marked by projections. By marking sets of projections that realise all types of meanings exemplified in the FGD model at the starting and at the arrival point of a Move, it is possible to define how movement develops across space and its semiotic dynamics. Within the Move, projections are connected through the displacement across space itself in a specific direction (MD), and the analysis of a Move cannot therefore be misunderstood for the analysis of static positions. Moves are marked by necessary boundaries between perceived moments in time and positions in space that are essential for the perception of continuity (see Kuhn et al. 2021). A relevant concept to the Move is Elaborations. When Elaboration occurs, a Move is marked by a starting set of projections and an arrival set of projections but the latter, after a pause, is immediately followed by another set of projections that involves a change in position of body articulators but no movement across space.

There are two types of projections that are realised simultaneously in the FGD: Narrative Projections, which express action, interaction, and emotional change; and Interactive Projections, that determine the difference between a dancer’s interacting with people and/or items on stage or with the audience. Narrative Projections can be intensified by narrathletic enhancers (Maiorani 2021a: 33), movements that accompany the narration with physical virtuosismos that are mainly meant to showcase the dancers’ technical capabilities. Narrative and Interactive projections realise Experiential and Interpersonal meanings respectively.

Projections might also be distinguished in terms of orientation towards the inner/personal world of the character interpreted by the dancer or vice versa: reflective projections will be directed by the dancer towards their own body, indicating a focus on the narrative towards the character’s personal sphere; deflective projections, will instead indicate a focus towards interaction with other characters on stage (see Maiorani et al. 2022).

Projections can also be modalised by focus or amplification: these two modal values depend on how many body articulators respectively project in the same direction or in different directions. The analysis of modalisation is relevant when it becomes a conspicuous choreographic choice, as when focus of projections from different articulators all in the same direction or their expansion in different directions are regularly repeated or alternate regularly and repeatedly as interactive markers of Moves and MBS.

Textual meanings are realised by Choreographic Units of different sizes depending on the number of Moves they contain. The local semantics of Moves are therefore developed into discursive strategies within sequences. Figure 1 shows the FGD model of analysis.

Figure 1:

The Functional Grammar of Dance model (Maiorani 2021a: 30).

The structural unit above the Move is the Minimal Ballet Sequence (hereafter MBS), which comprises two consecutive moves. The MBS is the smallest unit that can define a trajectory, and it also defines the syntactic relationship existing between Moves. Depending on whether the trajectory of an MBS does or does not maintain the same direction between the two consecutive Moves within it, the syntactic relationship will be defined respectively as continuous or varied. This distinction allows us to identify more extended dance patterns and provides us with a more consistent dance discourse structure.

The case studies we will discuss in this article are placed within a long tradition of studies that approach dance as a language (see for example Blacking 1983; Hanna 1979). Adshead-Lansdale (1981) and Foster (1986) already called for a linguistics-based approach to the study of dance and the analysis of choreography that would facilitate the description of meaning created through movement for students and non-practitioners, a trend that led to Guest’s reconsideration of Labanotation itself (2005). Williams (1999) approached dancers’ bodies as socio-semiotic constructions, preceding the distinctively practice-based studies in kinesthetic empathy which focused on the audience’s empathic response to movement (Opacic et al. 2009; Reason and Reynolds 2010). More recently, scholars started to promote the idea of dance as a system of signs in need of a grammar (Bannerman 2010, 2014; Keevallik 2018; Brooks and Meglin 2015).

However, while the research covered by these studies tried to understand whether any universals could be traced among different styles, and whether these could be recognised as forming a basic, overarching semiotic system, our work already posits that dance and movement-based communication in performance can be studied according to basic principles of communication that inform the FGD and allow for its adaptation to the analysis of different styles of choreographed movement. It is important to remember, though, that even though the FGD draws on some principles of Systemic Functional Linguistic theory generated for verbal language analysis, the aim is not that of translating dance into verbal language. Our analysis is finalised at studying what the viewer is offered to experience during a performance in terms of movement-based communication, indeed, the movement-based discourse on which an interpretation can be elaborated. Our approach works with the ontology of movement-based communication and the materialities of the human body in connection with different types of performance space. When we consider dance as a mode (Maiorani 2021a) using the FGD, its materiality is the human body and the contextual space in which the body moves.

The work carried out so far using the FGD is at the core of a new interdisciplinary area of research called Kinesemiotics, which is aimed at the development of multimodal theory on movement-based communication. Our research in Kinesemiotics also looks at developing practical applications in various domains, including software and digital imaging, data archival, immersive visualisations, tools and supports for enhanced teaching interactive activities, heritage preservation, etc. Kinesemiotics is being developed through an international collaborative project called The Kinesemiotic Body,^[1] funded jointly by the AHRC in UK and the DFG in Germany and involving a team based at Loughborough University in the UK, and one based at the University of Bremen in Germany,^[2] as well as a collaboration with professional artists from the English National Ballet.

3 A multilevel annotation framework drawn from the FGD for the analysis of dance sequences and the ELAN annotation scheme

We have developed the annotation framework by closely drawing upon the FGD. We have included the description of different levels, including MBSs, Moves, and Elaborations. For Moves and Elaborations, we annotate the physical movement, the projection structure that articulators build in relation to direction and orientation, narrative projections, interactive projections, modal values generated by the articulators collectively focusing towards one direction or not, narrathletic enhancers when present. Figure 2 outlines the annotation framework we have developed. We then transfer the annotation framework detailed in Figure 2 into a specific ELAN template which includes tier hierarchies and various levels of tier dependencies (see Figure 3).

Figure 2:

An FGD-derived annotation framework for the analysis of dance sequences.

Figure 3:

Transfer of the annotation framework into ELAN’s annotation tiers.

In the ELAN annotation template, we create different tiers to annotate the respective units and coded aspects. The MBSs tier is the first and highest description level on which the other tiers on Moves and Elaborations are dependent. The Move tier holds the second highest description level, while the Elaborations tier is the third. Dependent on the Move tier, various tiers relating to the physical movement, projection structure, narrative projections, interactive projections, modal values, and narrathletic enhancers are created. Analogously, the Elaboration tier also has similar dependent tiers.

The dance sequence is segmented in accordance with the three description levels, aligning with the duration of each MBS, Move, and Elaboration. Tiers that are dependent on Moves follow the segments created on the Move tier. Similarly, tiers that are dependent on Elaborations follow the segments created on the Elaborations tier. For the segmentation of MBSs, there is no gap between two segments, which means that the endpoint of one MBS aligns with the starting point of the subsequent MBS. The same holds true for the Move tier and its dependent tiers except the narratheletic enhancers and modal values tiers as they are generally only carried out in some and not all the Moves. The segments on the Elaboration tier and its dependent tiers are, however, non-adjacent, given that Elaborations generally occur periodically (see Figure 4, a list of legenda is provided at the end of the article).

Figure 4:

The segmentation follows the three description levels, i.e. MBSs, Moves and Elaborations and the segments on the first two levels are adjacent (example sequence from Swan Lake).

The annotations on the tiers of MBSs, Moves, and Elaborations include the count of respective units. Except for these three tiers, each of the lower tiers is linked to a controlled vocabulary which encompasses all annotation specifics, drawn from the options available in the FGD and the annotation framework. For instance, tiers dedicated to annotating the physical movement of hands are linked to a controlled vocabulary featuring eight options (see Figure 5). The options included in the controlled vocabulary can then be generated to a dropdown list in ELAN from which an annotator can select. The detailed and extensive controlled vocabularies we have complied can be applied to annotating dance or, more generally, movement sequences of different styles, while systematising annotation labels and ensuring consistency in the annotation.

Figure 5:

The creation of entries in controlled vocabularies (left) and the corresponding dropdown list generated in ELAN (right).

While the annotation framework and template presented in Figures 2 and 3 aim to encompass all the units and features found in our annotated sequences, not all the tiers depicted in Figure 3 are utilised in the annotation of every dance sequence. For instance, if no Elaboration is found in a sequence, the Elaboration tier and its dependent tiers become unnecessary, and the suffix “(M)_” in dependent tier names for the “Move” tier would be excluded. This adaptability of the annotation template allows customisation for different dance sequences. In Sections 4, 5, and 6, we will present three case studies illustrating how we modify the template and the insights gained from our annotations, made possible only by the annotation framework and scheme that are similar to the ones developed in the present study.

4 Case study one: how the FGD annotation can unveil hidden discursive patterns

The solo sequence we annotated as a starting point of a corpus collection was taken from the classical ballet Raymonda, originally choreographed by Marius Petipa on Alexander Glazunov’s music score and premiered at the Mariinsky Theatre in Saint Petersburg by the Imperial Ballet in 1898. However, the version we recorded and annotated is taken from the new version premiered by the English National Ballet in 2022 with the revised choreography and concept by Tamara Rojo. We recorded the data at the English National Ballet premises in London, the Mulrayan Centre, with first soloist Junor Souza. The data collection was carried out as part of the research activities of The Kinesemiotic Body research project. In the new English National Ballet version, Raymonda is a nurse who works among the injured soldiers during the Crimean war. She is engaged to a soldier named John but develops feelings also for his friend Abdur, who is an officer in the Ottoman Army. What we recorded in the rehearsal studio is a sequence from Abdur’s solo that he dances in the presence of Raymonda when they are left alone.

The materiality at our disposal was constituted by the movement data we collected live from the sensors applied to the body of the dancer moving in the reconstructed performance space; the data was then turned into a Python script that digitally translated what is captured structurally and semantically by the FGD. It was the annotation we carried out with ELAN using the FGD controlled vocabulary that allowed us to acquire the necessary information for investigating the discourse patterns that we then turned into interpretative hypotheses.

The solo does not show any elaborations, thus the elaborations tier and its dependent tiers from the annotation template outlined in Section 3 are absent from the annotation. The solo we annotated lasts for 51 s, and our annotation results show that it is made of 26 Moves and 13 MBSs. The solo has a very dramatic style that highlights the athleticism of the dancer and takes place when Abdur declares his love to Raymonda, who is already engaged to his friend John. What we found out with our annotation is that there are features to the choreography of this piece that may not be immediately noticed by the naked eye but that may impact on the interpretation of the solo as much as the dancer’s passionate interpretation and the movement flow, which is usually the aspect on which the soloist focuses for his interpretation and the performance feature that immediately captures the attention of the viewer.

As a matter of fact, our annotation highlighted the role of a significant number of mirrored repetition patterns at the level of Moves that are not detected, instead, at the higher level of MBS, a choreographic device that impacts on the movement-based discourse enacted by the character of Abdur. An example of this type of mirrored pattern is provided by the Moves in MBSs 8 and 9 (see Figures 6 and 7) which are quite central to the whole sequence.

Figure 6:

The arrival points of Moves 15, 16, 17, and 18 (data from Raymonda).

Figure 7:

MBSs 8 and 9 and their constituent Moves, i.e. Moves 15, 16, 17, and 18 (data from Raymonda).

This is a defining moment in the development of the story plot of the ballet, and Abdur is the only one dancing in front of the only other character present, Raymonda, on a scene devoid of people where the only noticeable elements are a canopy that stretches over most of the stage and the seat in the left front corner (from Abdur’s perspective on stage) where Raymonda is seated. The two MBSs are made by Moves 15, 16, 17, and 18 (henceforth indicated by an M) and they realise varied trajectories as Move direction changes from left front corner in Move 15 to right back corner in Move 16, and then from right back corner in Move 17 to left front corner again in Move 18. However, the changes in direction already suggest a mirrored pattern (see Figures 6 and 7). If we look at the annotation, starting from the Structure level that describes how the different articulators are positioned with respect to Move direction in the starting and arrival sets, torso and head realise a mirrored pattern by repeating the same movement following first the Move direction (M15) and then opposing it (M16) in MBS8, and then moving from opposing Move direction (M17) to following it again (M18) in MBS9. Legs also realise several movement repetitions following reciprocal mirrored patterns: the right one starting from a vertically perpendicular movement to the Move direction then following it, then following it again to get back to the vertically perpendicular movement; the left one starting from a movement opposite to Move direction to then change to a vertically perpendicular one, then starting again from the vertically perpendicular one to get back to the movement opposite to Move direction. Arms also move in a very repetitive way: the right one always performing movement vertically perpendicular to the Move direction; the left one first following the Move direction then moving to a perpendicular position, then starting from a perpendicular position to return to following the Move direction. The whole body, then repeats several structures in relation to the Moves directions several times but every articulator, except the right arm, enacts repetition following a mirrored pattern. These patterns are then reflected by all types of movements performed by the articulators and all types of Projections (see Figure 7), with very interesting narrative and interactive discursive effects. In terms of Narrative Projections, the focus of the head addressing the left front corner is reinforced by the torso engaging with the same corner, thus establishing a strong visual connection between these articulators and Raymonda, who during Abdur’s solo remains seated in that corner. Whether Abdur moves away or towards Raymonda, his constant attention is towards her. Meanwhile his legs design paths that mark the locations the dancer takes on the ground while alternatively approaching and moving away from Raymonda, and while the right arm projects in a mirrored pattern towards the ground, then to the corner opposite Raymonda then back, the left arm follows a similar mirrored pattern of projections towards Raymonda, then towards the background and finally towards Raymonda again. Discursively, all the Narrative Projections realised by all articulators indicate that Abdur, who is in love with Raymonda, is being pushed and pulled in opposite ways, interfering in the engaged couple formed by Raymonda and John, who is his friend, but receiving nonetheless Raymonda’s attention. All Interactive projections are directed towards Raymonda or elements of the stage set that populate the contrasting directions between which Abdur is moving; the only exception is in M17, when the right arm projects sporadically towards the audience.

The annotation provides some really interesting insights into the choreography discursive strategy: while the naked eye can immediately capture that the dancer keeps on moving in similar ways on the same spatial plane (the same diagonal going from the left front corner where Raymonda is seated to the right back corner of the stage), the mirrored patterns captured by the annotation actually reveal how the choreography reproduces the emotional and situational dilemma in which the character is involved through the way the body articulators project towards meaningful portions of the performance space and in relation to Move directions. If we look at the whole solo, the annotation also reveals that there is more repetition in the varied MBSs than in the continuous ones, which in turn suggests a specific rhetorical use of varied MBSs: whenever Abdur changes direction in the trajectory designed by an MBS, he performs more repeated movements with a consequent repeated set of Narrative and Interactive projections. Therefore, variety at the discursive level of MBSs is matched by repetition at the semantic level of Moves.

This initial example of the application of the FGD in ELAN demonstrates how this type of annotation can offer an insightful visualisation of movement-based discursive patterns, and how this can potentially be the starting point for a much wider range of empirical investigations of movement-based communication strategies.

5 Case study two: the adaptability of FGD annotation

The analysis of Lamentation, an iconic modern solo by Martha Graham premiered at Maxine Elliott’s Theater in New York in 1930, with Martha Graham as performer, is the first example of a serious challenge to our analytical method. This solo piece created on a music score by Zoltàn Kodàly, proposes an innovative choreography in a minimal setting where the dancer’s body moves through the constraints of a costume made of a unique, tubular, elastic piece of cloth (see Figure 8). The costume only allows the viewer to see the face, hands and feet of the dancer, who never leaves the bench on which she is sitting for most of the duration of the performance. For this solo, we analysed the video recording of the memorable performance by Peggy Lyman in 1976.^[3]

Figure 8:

Images from Lamentation (The images are from Peggy Lyman’s performance of Lamentation in 1976. To comply with copyright regulations, the media frames have been blurred).

This is then a case study where we demonstrate the applicability of our methodology to a recorded performance made through camera movement that is nonetheless based on the point of view of the dancer.

When we started working on it, we were initially faced with an ontological dilemma: during the whole piece, the dancer never moves across space, which is a fundamental condition for the constitution, recognition and segmentation of the Move, the FGD basic unit of analysis. A reconceptualization of the Move was therefore necessary for the adaptation of our analytical method to this choreography and similar pieces of movement-based performance that we may come across. Maintaining a recognisable unit of analysis marked by sets of projections was a necessary step, one that would allow us to keep on incorporating the flow of movement into our annotation as well as segmenting the piece in a coherent, objective way. We therefore focused on the torso as this is the only articulator to which all other articulators are attached either directly (head, arms, and legs) or indirectly (hands through arms and feet through legs). The solution we found was marking the starting and arrival sets of projections whenever the dancer’s torso changed direction (Maiorani and Liu 2022). The dimension of direction is a fundamental component of the FGD model and the way we map dance discourse through it, not only for the determination of trajectory type in MBSs but also because of the relationship that orientation of articulators has with it: as a matter of fact, orientation depends on Move direction. For example, if a Move direction is towards the right front corner of the stage, a dancer with their torso and/or head projecting towards the left back corner will be perceived as ‘going backwards’. In our adapted annotation, the moment the dancer’s torso starts changing direction marks the starting set of projections of a new segment; the arrival point of a projection is segmented on the moment when torso direction starts to change again: like the segmentation created in the original template, the arrival set of projections of a Move corresponds to the starting set of the following one.

As a consequence of the adaptation of the Move unit, we had to make changes in the controlled vocabulary we had created for implementing the FGD in ELAN; we therefore used the label ‘connecting to’ instead of labels like ‘going to’ or coming from’, a solution that allowed us to describe better the particular dynamically static nature of the movements performed in this choreography that does not move across space, as well as the interaction of articulators with the meaningful portions of the performance space. This label also reflects the nature of our Move segmentation: for Lamentation, we recognised Moves according to the ‘intention’ of moving across space in a specific direction, which is a solution that is partially informed by Cohn’s theorisation of the cognitive visual processes that are enacted by human beings when reading comics (Cohn 2013, 2020, 2021). The fact that here we draw on Cohn’s work on comics instead of work on film and the moving image already carried out by experts in multimodality (Bateman 2007, 2022; Bateman and Schmidt 2012; Kress 2004; Tseng et al. 2021; Wildfeuer 2014, 2018) is justified by the fact that segmentation in our dance discourse analysis is based on projection structures framed within a continuous flow of dance, and these framed projection structures – the starting and arrival point of Moves and MBS – are marked against the flow of movement through which they are connected exactly as comic frames mark specific moments in the narrative flow of a comic strip or book or graphic novel. According to Cohn, comic readers are required to create dynamic connections during the reading process, which provide the narrative flow between the different cartoon bubbles. Besides, these connections are created by the comic reader thanks to the trigger of structural visual cues. We therefore took the change in torso orientation as a visual cue for segmenting the different Moves in Lamentation. We also expanded the controlled vocabulary with labels to describe hand positions we had not encountered before in classical ballet choreography: for example, as Lamentation experiments considerably with the interaction between body and costume, we created labels like ‘handling’ or ‘tight’ for the hands that manipulate the cloth in specific moments of the solo as this manipulation is a necessary part of the choreography itself, and it is not a mimicking or spontaneous gesture enacted by the dancer on her own initiative. Another label we added to the controlled vocabulary defines a choreographic item that, once more, we had not typically encountered when analysing classical ballet: usually, when a dancer is about to start a classical solo, they take a fixed position that defines the starting set of the first Move. This is a condition that can be observed in most solos for both male and female characters in most classical choreographies within the traditional ballet repertoire. Paradoxically, Lamentation, which is a piece where the dancer does not make any move across space, breaks this tradition and starts with a position where the dancer keeps the torso in the same place but repeats the same shaking movement of her head while facing the audience. This is quite a dramatic choreographic choice that lasts for a relatively long interval right at the beginning of the piece: we labelled this as a ‘dynamic set of starting projections’, a label that we envisage could be used for any such choreographic choice across dance styles. All these adaptations enriched our understanding of the complex materiality of dance as a semiotic mode.

Apart from adjusting the notion of Move and the entries in the controlled vocabularies, other aspects we have adjusted from the annotation template (see Section 3) include the exclusion of (1) Elaborations tier and its dependent tiers as no elaborations are found from the sequence, and (2) narrathletic enhancers and modal values tiers as they are not carried out in the piece. Based on our annotation, Lamentation, which lasts for 3 min and 20 s, is made of 74 Moves and 37 MBSs.

The whole piece is danced on an empty, dark background; on stage, there is only the bench on which the anonymous character is sitting. No props or other people or items mark any meaningful portion of space and the character herself is almost completely covered by an elastic tubular costume, therefore the character assumes an abstract, universal value, and so do the directions towards which her articulators project (see Table 1).

Both Narrative and Interactive projections will therefore target areas with abstract values when directed away from the dancer’s body (deflective projections); the only character they can target is the dancer herself when being directed reflectively towards the dancer’s body (reflective projections). In this particular case study, we also started including quantitative data, as the number of possible repetitions enacted in such a static choreography and such an empty stage would help us recognise discursive patterns in terms of intensity of repeated projections; in particular, we were interested in seeing whether any specific abstract areas of space would be especially targeted, thus reinforcing the perception of the symbolic value the viewer would attach to them.

All projections are mediated by the elastic cloth of the costume that entraps the dancer like a shroud and produces visual shapes for the audience; the actual full silhouette of the dancer’s body remains hidden for the whole piece. The first choreographic feature that our annotation reveals about Lamentation is that the functions of Narrative projections are constantly and regularly distributed among specific groups of articulators; from Move 1 to Move 24, arms and hands project reflectively towards the dancer’s body while holding tight to the costume, thus focusing the narrative on the figure of the woman who is grieving on the bench. Then the narrative realised by the projections of these articulators gradually changes as projections start to target alternatively ground and top areas of the stage while stretching the costume more dramatically, which enhances the visualisation of the projections themselves. When achieving Move 40, arms and hands start projecting also towards right and left side areas, thus widening the space of the narrative carried out by the upper body projections and suggesting a cross-shaped pattern. This process starts regressing at Move 50: at this point, the left arm starts projecting reflectively towards the dancer again, joined by the right arm in Move 55. However, another change happens in the final section of the solo, when from Move 56 arms and hands start projecting alternatively towards top and bottom, and left and right, thus drawing a cross-shaped pattern more hectically until Move 64, when they recoil again towards the dancer’s body before the final, dramatic focus on the right arm projecting first towards the top and finally towards the bottom of the performance space, like a final attempt at reaching the sky above before retreating on the ground, with all the possible meaning associations to these areas of space one can imagine (see Figure 8 for relevant images from the solo). Legs and feet, instead, do not design any such pattern: the Narrative projections realised by these lower body articulators are almost exclusively directed towards the ground, thus constantly locating the character in a very little portion of the performance space. The only exceptions are small lifts of one leg and foot in Moves 15, 17, 40, 41, 49, 50, which mark even more the dramatic projections realised by the upper body articulators. The quantitative data shown in Tables 1 and 2 corroborate the results of the analysis performed qualitatively through the annotation as they confirm occurrence and distribution of vertically oriented and horizontally oriented Narrative projections realised by arms and hands of the dancer during the performance and the constant Narrative projections towards the ground realised by the lower body articulators.

On the other hand, torso and head Narrative projections carry out very similar projections and vary much more: they target alternatively right and left areas of the performance space as well as front, top, and ground. The quantitative data reveals a majority of alternations between front, top, and ground areas, which reinforces the focus on the character and her isolation on stage and enhances the verticality of the cross-shaped pattern designed by the upper body articulators. The quantitative data also highlights that the only interlocutor of the character is the audience, with which the torso constantly engages. Moreover, it also reveals a discursive strategy of the choreography that might not be immediately visible to the naked eye: the torso is the articulator that projects towards the audience most times, whereas the head only does it once enhancing once more the isolation of the character even when performing in front of the audience (see Tables 1 and 2).

Interactive projections are also distributed among the different articulators in significant patterns that characterise specific waves of engagement throughout the solo. Projections directed towards the audience are mostly concentrated in Moves forming the first two thirds of the piece and they become less frequent in the last third. They are also mostly realised by the torso, which becomes the central part of the cross-shaped pattern and focuses on the audience. Notably, the torso realises Interactive projections towards the audience at the beginning and at the end of the piece. Another interesting feature is that the only time the head realises Interactive projections towards the audience is within vertically oriented MBSs, therefore addressing the viewers when the dancer is designing the cross-shaped pattern through the Narrative projections of her upper articulators. The quantitative data reveals that most of the Interactive projections realised by all articulators target the empty areas of the stage, which is in line with the narrative of isolation carried out by Narrative projections (see Table 1). The patterns realised by all projections at Move level also show that the cross-shaped pattern repeatedly realised by the dancer is substantially bi-dimensional, spanning across top, bottom, right and left areas of the performance space: the only exploration of depth realised by the choreography is obtained through the visual effects achieved through the interaction between the dancer’s body and the costume.

Another consequence of our choice to adapt the unit of Move and segment the piece for annotation according to changes in torso direction is that all MBSs in this solo realise a varied trajectory. This, however, does not exclude the possibility of variation in discursive patterns. On the contrary: thanks to the inclusion of quantitative data in our analysis, we managed to capture discursive variation based on the alternation between horizontally and vertically oriented MBSs. As already suggested when analysing projection patterns in Moves, the discursive patterns realised in Lamentation follow the alternation of vertically and horizontally oriented MBSs, where the vertical or horizontal orientation depends on the number of projections realised in specific directions. This alternation happens in cycles: in vertically oriented cycles, most of the torso changes of direction in Moves happen between top and bottom areas of the performance space; in horizontally oriented cycles, most of the torso changes of direction in Moves happen between right and left areas. Our annotation also revealed changes in the way cycles perform rhythmically: as the piece approaches the finale, each cycle is formed by fewer and fewer Moves, thus becoming more hectic. This cyclical alternation suggests that the cross-shaped pattern that characterises Moves is also repeated at the discursive level of MBSs, thus amplifying the repetition of a powerful symbol across cultures that is not necessarily captured by the naked eye of the viewer. Once again, our method of analysis allowed for a new, insightful reading of this iconic choreography based on objectively and systematically retrieved empirical data.

6 Case study three: the phenomenon of Elaboration in FGD annotation

The annotation of several pieces from the ballet Swan Lake is our most recent case study. The choreographies we annotated were performed by ENB (English National Ballet) dance students in their final year of specialisation, and they were taken from the ENB version adapted for young audiences from the original created by Marius Petipa in the 19^th century. This specific work revealed not only features that are specifically character-related but also the function of choreographic elaboration, which is a phenomenon that the FGD picks up when the arrival set of projections in a move is actually made by two consecutive sets of projections (see Maiorani 2021b), i.e. Elaborations. In the coda from Act II performed by the character of Odette (a princess turned into a swan and kept captive by an evil magician king), Elaboration occurs repeatedly. This is a moment in the ballet when Odette has just met a prince when in her human form, and he has asked her to go to his birthday party at the castle where he will swear eternal love to her in front of the court, thus breaking the spell that keeps her captive. The coda is danced on the enchanted lake shore, in an alcove where Odette and her fellow swan girls are hiding.

Most tiers outlined in the annotation template are used in the annotation of Swan Lake, except the tiers on the modal values of legs and feet in moves and the modal values of the limbs in Elaborations as they do not characterise the piece. Our annotation results show that the whole coda lasts for 47 s, and it comprises 48 Moves and 24 MBSs. Elaborations happen increasingly towards the last section of the piece, specifically in Moves 11, 14, 15, 16, then in Moves from 24 to 36, and finally in moves 38 and 39.

The first group of Elaborations happens within a sequence of alternate MBS types: MBS6 (varied), MBS 7 (Continuous), and MBS 8 (varied), when the dancer shifts directions between the right and left front corners, and at the end of Moves that present some interesting narrative and interactive patterns. In Move 11, the dancer interpreting Odette starts from projecting with her right arm and hand towards the right front corner, following the Move direction; her head and torso project towards the same direction while her left arm, hand, leg and foot project towards the opposite way and her right leg supports the move on the ground. Narratively, she does project forwards, towards one side of the alcove where some of the other swan girls are and away from the lake in the background, but she is also pulled back by the projections of her left arm, hand, leg and foot which point at the lake in the background. Interactively, she only engages with the other swan girls. The Narrative and Interactive projections she realises at the end of this Move in the Elaboration set mark a change as her legs are both projecting towards the ground, her right arm is posing towards the right side whereas her left arm is projecting towards it, her head is addressing the top area of the stage and her torso is engaging the audience in the front. Narratively there is therefore still a push towards the right front corner but now both legs are stopped on the ground while the sky and the audience are now involved in the action and interaction. In Moves 12 and 13, Odette repeats the same projections she realised in the first arrival set of Move 11 but following a mirrored pattern towards the left front corner. At the end of Move 14, after repeating the same mirrored projections, she elaborates on a second arrival set that mirrors the Elaboration arrival set of Move 11; the alternation of these mirrored patterns is repeated in Moves 15 and 16, which also indicates an increase in narrative and interactive rhythm. The character is therefore repeatedly indicating her intention to move away from the lake and to the swan girls surrounding her and beyond, as well as her being pulled backwards towards it. She also projects towards the sky, where she would fly as a swan, and only partially engages the audience just with her torso. Elaboration here has the function of enhancing the narrative pattern and of involving the audience and the sky in the interaction every time Odette changes direction.

In Move 24, when the most substantial series of Elaborations occurs, Odette starts a series of Moves towards the back of the stage, in a sequence of Continuous MBSs until MBS20, when she changes direction at the end of Move 39. From Move 24 to Move 39, Narrative projections of the legs and feet are directed towards the ground whereas the arms project either towards the front corners, thus in opposite directions with respect to the direction of the Moves, or towards the right and left sides respectively or the top (or ‘sky’). The head and torso are projecting alternatively towards front corners and towards the audience in the front. The narrative patterns designed here show a clear divide between the lower part of the character’s body, which is entirely projected towards the ground, and the upper part of it, which escapes in waves towards the sky: this is very much in tune with the nature of the character herself, torn between her earthly nature as a human being and her spell-bound transformation into a bird. The audience is much more engaged in this part of the narrative, both narratively and interactively, not just by the number of Narrative and Interactive projections towards them but also by their variety as this time also arms and hands and the head are involved. This involvement of the audience is reinforced by the Elaborations that occur after jumps on the spot and are used mostly to double the number of Narrative and Interactive projections realised in the first arrival set of each Move.

The last two Elaborations occur between MBS20 and MBS21, just before the choreography takes its final trajectories to the end of the piece. In Move 38, Odette is still moving backwards towards the background, performing the same narrative and interactive patterns we have seen in Moves 24 to 36; the Elaboration at the end of this Move, involves the audience slightly more with Interactive projections of arms, hands and torso directed at them. In Move 39, instead, the whole focus of the first arrival set of projections is on marking the location of the dancer before she starts her final trajectory away from the lake: her right arm and hand are posing and extended towards the right front corner, opposite the move direction as she is going backwards on a diagonal while being pulled towards the direction she is about to take in the final steps; her left arm and hand are posing towards the top; right leg and foot are connected to the ground while left leg and foot bend reflectively towards them; the head is addressing the left front corner while the torso is engaging it. All interaction is directed towards the swan girls on stage. In the Elaboration of Move 39, the interactive patterns are still focused on the stage but now both legs and feet are planted on the ground and both arms and hands are posing, but the right arm and hand are extending towards the left front corner, while the head is addressing the opposite corner to the right, in line with the posing left arm and hand that are pulled towards the back left corner. Even this final and more static Elaboration expresses the condition of the character who is being pulled and pushed between the lake and her condition as captive swan girl and the way out, towards the castle, where the spell could be broken, and where she could return to her normal human condition.

In all sets of Elaborations, then, the main function is to enhance and underline the narrative and interactive meanings realised by the Moves, thus marking important features of the action and interaction realised by the character’s dance discourse. Moreover, the annotation also allowed us to observe how the specific dancer with whom we worked managed balance in different ways between the two legs: this is an extra-discursive feature that the FGD annotation can capture thanks to the tiers focusing on physical movement, and one that has potential for developing effective impact when monitoring not only dancers’ technique development but also their progress when recovering from injuries or when under the effect of physiotherapy or enhanced training as the model captures the reciprocal movement structures realised by all articulators in all sets of projections.

7 Conclusions

This article shows how the FGD framework for analysing choreographed movement can be implemented in ELAN annotation and its potential for flexibility and adaptation. The three case studies we presented demonstrate the effectiveness of this method of analysis for the annotation and the analysis of dance discourse both for live-captured dance performances and video recorded ones. Besides expanding the reach and applicability of Systemic Functional Linguistic principles to the domain of movement-based communication, our study shows how the annotation of ballet sequences provides an effective interface for the grammaticalization of movement-based communication that is flexible and adaptable and potentially applicable to several different forms of movement-based performance, including choreographed sports and digital interactions realised by game avatars.

The case studies specifically revealed the variety of insights that the analysis carried out through FGD annotation in ELAN can provide, especially in relation to discursive strategies that impact on discourse perception of movement-based communication, even if not immediately captured by the naked eye.

Our examples demonstrated how our method can capture narrative and interactive discursive patterns that characterise a choreography: this type of information can be used to investigate rhetorical trends in dance history across styles as well as to inform the learning and teaching activities centred on old and new choreographies and on chronological changes in strategies for engaging audiences.

We also demonstrated how our analysis can use a combination of qualitative and quantitative data to delve into discursive patterns that do not rely on physical movement across space. In this case, we also demonstrated the solidity and adaptability of the basic notions and theoretical foundations of our methodology, which strengthens even more the role of the FGD and SFL within the areas of social-semiotic investigation and ontological research. A software that can automatically read FGD annotations in ELAN and visualise projections is currently being developed by the Kinesemiotics research group at Loughborough University,^[4] and always in collaboration with the University of Bremen.

Finally, we demonstrated how our methodology can be applied to the study of character representation: this type of analysis can be expanded to investigate stereotypical representations of gender and ethnicity based on narrative and interactive movement patterns even outside the domain of classical ballet, thanks to the flexibility and adaptability of the analytical model and the creation of non-specialist controlled vocabularies.

Last but not least, the FGD annotation involves picking up on physical movement patterns and habits that can be investigated for monitoring and coaching dancers at all stages of their careers.

Lastly, as the FGD was originally created to improve the accessibility of dance discourse for non-specialists, we believe that the fact that this article is not published for the exclusive use of dance experts is evidence enough of its effectiveness in this respect.

Abbreviations: Legenda of participants and space with values

A

agent

AU

audience

BG

stage background

FR

stage front (audience)

GR

ground

LBC

left back corner

LFC

left front corner

LS

stage left side

POS

participant/s on stage (characters and items)

RBC

right back corner

RFC

right front corner

RS

stage right side

TP

top