Spatial constructions of time: exploring Co-speech gestures in lectures on programming

3.3.1 Gestural analysis

This study is mainly exploratory and qualitative and employs an inductive approach. Therefore, any co-speech gesture along each of the three axes (gestures along the lateral, vertical, and sagittal axes, respectively) was included in the study. Furthermore, less distinct yet reoccurring gestures were included as data for the study. Examples of such gestures include the teachers bursting out with their hands (henceforth palm-up gestures), gestures depicting containerlike entities (metaphorical gestures), and gestures illustrating concrete objects or actions (iconic gestures). All gestures coded for are described and exemplified in Table 1. Palm-up gestures are distinguished by speakers rotating their forearms with their palms turning upwards (e.g., Cooperrider et al. 2018). Both metaphorical gestures (such as the conduit gestures) and iconic gestures are placed in McNeill’s taxonomy of gestures (McNeill 1992). However, to distinguish between, for example, a reference to an object or a (computing) process, we have chosen to keep them disassociated. Moreover, gestures that were performed hidden behind the computer screen or a student, as well as deictic gestures, such as when the teacher pointed at the whiteboard, the computer screen, or one of the students, were not included in the data. Unclear or small gestures, so-called beats (McNeill 1992), for example, gestures with a low amplitude that did not allow us to decide the direction of a given gesture, were not coded for. In all, the coding of the classroom observations resulted in 329 gestures.

Table 1:

The codes of gestures, with verbal descriptions and illustrative examples from the data.

Gesture type	Verbal description	Illustrative example
Gesture along the lateral axis	A gesture along a lateral axis is performed from right to left (with respect to the speaker’s body) or in the opposite direction. It could also be performed from the body’s centre and outwards, or vice versa. Each gesture’s height (e.g., waist, chest, shoulders, head, or above head) is noted
Gesture along the vertical axis	A gesture along the vertical axis is performed top-down or bottom-up, either centred or to the right or left, with respect to the centre line of the speaker’s body. The height could vary from a few centimetres to the total reach of the speaker’s arm
Gesture along the sagittal axis	A gesture along a sagittal axis is performed front to back or in the opposite direction (with respect to the position of the speaker’s body). Each gesture’s height (e.g., waist, chest, shoulders, head, or above head) is noted
Palms-up gesture	The teacher bursts his hand, either one or both, from the body and out. Palm-up gestures are distinguished from gestures along the three axes in that they have a less distinct demarcation of the direction and no distinct start and end point	1. 2. 3.
Container gesture	For example, when the speaker holds one or two hands as a cup or holds the hands as a confined area that can contain an object
Iconic gesture	For example, when the speaker moves his fingers as if typing on a computer keyboard, moves his arm as if he is throwing a ball through the air, or holds his index finger in the air as if he is writing something where the finger symbolises a pen while talking about these specific acts

3.3.2 Linguistic analysis

When all gestures had been coded, the accompanying words and phrases were analysed linguistically to identify those that code time concepts. Three linguistic constructions were identified: time adverbials, verb tense, and adverbs denoting order. We used linguistic constructions to establish whether the identified directional gestures involve time or not.

In the linguistic analysis, time adverbials such as då (En. then), före (En. before), idag (En. today), sedan (En. later), and med en gång (En. right away), and the adjective asynkron (En. asynchronous) were considered to code time.

Furthermore, because verb tenses provide information on when a given action occurs in time, they were similarly considered to provide information on time (Teleman et al. 1999).

Adverbs such as först (En. first) and sist (En. the last time), and longer phrases such as hade hänt i tur och ordning (En. had happened in order) and har kört program uppifrån och ner (En. had run the program from top to bottom) were considered to code order and hence time as well. As stated in earlier work (Stolpe and Larsson 2023), the word “run” is used with a specific meaning in the programming context to describe a process that has to be carried out in a certain order.

In this way, the gestural and linguistic analyses have guided our systematic exploration of time concepts and the spatial organisation of such concepts as two Swedish-speaking programming teachers construe them.

4.2.1 Teacher 1

Teacher 1 primarily gestures along the vertical axis when using linguistic constructions of time. We identified seventeen gestures along the vertical axis. Of those seventeen identified gestures, sixteen were accompanied by a verb from which we learn that some action has been taken or will occur. Seven of the gestures were accompanied by time adverbials (e.g., sen, nu, and samtidigt, En. then, now, and simultaneously). One of the gestures was related to the order in which something is done (i.e., kört ett program uppifrån och ner, En. run a program from the top down) (Table 3, Memo 18). Table 3 shows an episode from Teacher 1’s teaching and the co-speech gestures. As the teacher says Hittills har vi kört program uppifrån och ner (En. Hitherto, we have run programs from the top to the bottom), he gestures from top to bottom along a vertical axis. A program on the screen is ordered from top to bottom, meaning code lines are read top-down. This gesture organises code lines in a spatial and temporal order. Thus, this gesture could be interpreted as both iconic, symbolising the code lines on a screen, and as temporal since it constructs a temporal order for the program to run. As seen in Memo 18, he gestures from top to bottom (Table 3). We thus suggest that his gestures function as a vertical and imaginary ‘timeline’ on which he can locate earlier times higher up than those that follow. This interpretation aligns with when Teacher 1 talks about what the students have done so far, and what they will continue to do later. For instance, when telling the students that they had activated lines in different places (referring to programming syntax available on their screen, thus an iconic gesture) (Table 3, Memo 20, 55), he finishes by saying, “och sen har ni gått tillbaka och fortsatt va” (En. ”and then you have gone back and continued, right?) (Table 3, Memo 21). During the episode, the teacher holds his left hand still as if it is a reference point for the location on the computer screen from which the students once started (Table 3, Memo 20). He then moves his right hand up towards the ‘reference point’ while saying och sen har ni gått tillbaka och fortsatt, En. and then you have returned and continued (Table 3, Memo 21), indicating that the students have returned up to where they started which in turn indicates that earlier times are up. Thus, in this specific context, temporality is suggested as a movement in time and a physical movement in space. The teacher’s gesture also indicates a movement in the computer program; the vertical movement by his right hand from the bottom up to the starting point is indicated by the left hand. Space and time are intertwined, and since spatiality is conceptualised in terms of code lines, time is given a vertical dimension.

Table 3:

An excerpt from Teacher 1’s teaching.

Memo	Verbal utterance by the teacher (Sw/En)	Gesture	Direction
18	Hittills har vi kört program uppifrån och ner	He holds a pen in his left hand and moves the pen from the height of his eyes to the height of his chest	Vertical
	Hitherto, we have run programs from the top to the bottom
	Ni har kallat på funktioner som har gjort att
	You have called on functions which have made that
19	ni har hoppat i koden	He holds his left hand still, in line with his shoulder. His right hand moves in a bow from the height of his eyes and down to the height of his waist. During the movement, he opens his hand, palm directed downwards	Vertical
	you have jumped in the code
20	och aktiverat rader	His right hand moves up and down in small movements at the level of his chest while he holds his left hand at the height of his shoulder	Vertical
	and activated lines
55	på olika platser	His right hand moves up and down in larger movements at the level of his hip while he holds his left hand still at the level of his shoulder	Vertical
55	in different places
21	och sen har ni gått tillbaka och fortsatt va?	His right hand moves from the bottom up in a bow back to his left hand. His right hand grabs around his left hand	Vertical
21	and then you have gone back and continued, right?

Teacher 1 also uses gestures along the lateral axis in reference to time and order. A total of 23 such gestures were identified, 20 of which include a verb that provides information on tense (e.g., ska starta, En. will start). Eight gestures were used together with time adverbials (e.g., för några år sedan, En. a few years ago), and one was used together with expressions indicating order (i.e., i tur och ordning, En. in order). He typically gestures from left to right, but occasionally in the other direction to indicate where some previous action or something that still applies is located – i.e., in all the gestures, the teacher spatialises earlier times to the left and later times to the right.

In one episode, Teacher 1 uses an analogy between cleaning one’s room and how to use a function^[1] (which is here used to illustrate that the behaviour of a program depends on what data has been put in by an end-user or received from a sub-routine or a database). At this stage, the teacher asks the students whether they have received any instructions for cleaning their rooms prior to cleaning their rooms (“fanns det någon inställning för hur man skulle städa sitt rum?”), for example, if they were supposed to “do so sloppily, carefully, quickly?” (“Ska man städa det hafsigt? Noga? Snabbt?”) ”or before grandma arrives” (” Innan mormor kommer?”). Just as he utters the word innan (En. before), he moves his head, followed by his upper body, from the centre to his left. He holds this position briefly and then returns to his previous position. We suggest that this may indicate a correspondence between a linguistic construction relating to time and a gesture along a lateral axis where innan (before) is located to his left.

Teacher 1’s sagittal gestures are not as closely connected with linguistic constructions of time. Six out of nineteen of these gestures are not accompanied by a phrase that includes a verb, and two are used together with time adverbials (i.e., with före and innan, En. before). While gesturing along this axis, the teacher discusses other real-world situations in which something happens or will happen. He also uses forward gestures in reference to the future (e.g., when something bör ske (En. should happen). Related to this result, some of his gestures suggest that programming implies moving forward into the space ahead of the programmer as if he is moving forward into a computer screen. The teacher holds his right hand close to his body when saying till vad vi gör i programmering och kod är att (En. to what we do in programming and code is to). However, when he finishes the sentence with och kod är att (En. and code is that), he moves his left hand forward in a sagittal gesture to indicate that the code should perform something in the future. One of his sagittal gestures goes in the other direction, where he references his experience with the phrase jag kan ta lite ur egen erfarenhet (En. I can take some from my own experience). Experience is something related to the past, hence having a past temporal connotation.

Teacher 1’s palm-up gestures were particularly present when turning to the class in front of him and asking them to consider the steps of the processes he teaches or a hypothetical scenario in the future or the past. For example, phrases such as Kan ni tänka er? (En. Can you imagine?), hade det varit? (En. would it have been?). Twenty-four palm-up gestures were identified, and nineteen verbs were mentioned together with the palm-up gestures. Of the nineteen verbs, thirteen are in the present tense and pertain to present or future time. Moreover, Teacher 1 uses two verbs which refer to times other than the now – two verbs in the future tense (e.g., vi ska städa, En. we will clean), three in the past (e.g., ni har varit med om, En. you have experienced) and one in the past perfect tense. Only three of his palm-up gestures are used together with time adverbials (e.g., då, En. then; tills, En. until, and så länge, En. as long as), and one with words or phrases for order (först när, En. not until). One of these gestures is used with the adjective asynchronous in reference to a function.

Teacher 1’s metaphorical and iconic gestures are not combined with any words or phrases for time apart from verbs that primarily signal present or future tense. Instead, the eighteen identified container gestures were accompanied by single words such as data, information, retrieve, and fetch and used in association with discussing how to use functions and how to retrieve data, making them conduit gestures. His five iconic gestures were primarily used with reference to an analogy between the form and direction of a curly bracket ‘{‘ (i.e., a character that is used to mark where a function starts and ends) and the wings of a seagull between his hands (i.e., a connection between a programming concept, and a verbal expression accompanied by a gesture).

4.2.2 Teacher 2

The majority of Teacher 2’s identified gestures were performed along the sagittal axis; eighteen out of nineteen are accompanied by verbs, and eight out of nineteen are by time adverbials. Nine of the verbs were in the present tense, and six were in the future tense. All but two gestures along the sagittal axis also involve forward bodily motion. The sagittal gestures are paired with language about what the teacher does or what he and the students do while programming (som ni gör nu, En. like you’re doing now) or will do (sen ska vi guida dem, En. then we will guide them). Related to this, five of the adverbs are instances of sen (En. then), but nu (En. now) and idag (En. today) are also used. The two remaining gestures are accompanied by the phrases då ska vi fånga upp det (En. then we will catch it) and som vi tar in (En. that we take in) – i.e., phrases that involve the motion of objects towards the teacher’s body. Also, these two gestures are iconic. The third exception is a gesture used with verbally asking the students to be more communicative. That is, focusing on the student’s future behaviour toward him. These gestures could also be interpreted as conduit gestures in which something is caught or taken. However, it could also be a combination of temporality and conduit gestures.

Teacher 2 used 44 gestures that were coded as palm-up gestures. Like Teacher 1, his palm-up gestures were used, turning to the class when asking them to consider a hypothetical action or how to perform a step in the programming procedure. Furthermore, his palm-up gestures were closely connected with time. Moreover, they are accompanied by language that includes more verbs (41 out of 44) and time adverbials (13 out of 44) than the gestures coded as performed along the three axes. Twenty-two verbs are primarily used in the present tense, and thirteen are used in the future tense. Twelve of the verbs in the present tense include a time adverbal (i.e., då; En. then; sen, En. then; and när, En. when), which either locates the action in the past or the future, while one of them locates the action in the now (i.e., nu, En. now). Five of the verbs referring to the steps of the programming process or something hypothetical include the conjunction om (En. if), which is used together with subclauses that express conditions. Unlike Teacher 1, Teacher 2 does not combine palm-up gestures with language representing order.

Teacher 2’s gestures along the lateral axis are used in reference to time but also to locations on an imaginary computer screen located in front of him (e.g., den här mappen, En. this folder), or to places elsewhere in the world (e.g., in flera länder, En. many countries). In such instances, the gesturing is interpreted as gazing at said locations while facing the students. In contrast to Teacher 2’s sagittal and palm-up gestures, his gestures along the lateral axis are not as closely connected with time (28 of the 38 identified gestures are accompanied by words in different tenses, and time adverbials accompany nine). Generally, Teacher 2’s gestures from left to right are paired with language about what he and the students do, did (och sen skrev jag, En. and then I wrote) or will do (lägg den, En. put it), whilst the gestures are paired with language about what he did in the past, (e.g., då var det enkelt att gå tillbaka, En. then it was easy to go back) are performed the other way around – an indication that time is spatialised from left to right.

Teacher 2 also performs inward gestures along the sagittal axis, using both his hands toward a location in front of him or outward gestures with both hands from this position. In one episode, he makes such a motion while saying (försöker casta, En. are trying to cast), and in another one, he makes such an outward gesture while saying (och så försvann då det där, En. and then that disappeared). Just like Teacher 1’s gestures along the vertical axis appear influenced by the fact that he stands in front of a computer screen looking at programming syntax. Here, for teacher 2, our interpretation is that events do not move in time from left to right but in relation to an imaginary screen. Another example is when he keeps his gesture still while talking about yesterday’s coding in front of the screen (igår satt jag och kodade, En. yesterday I sat coding). By keeping the gesture for a while, this activity becomes a point on a spatial timeline for as long as it lasts.

Teacher 2 uses twenty gestures along the vertical axis, less than half as many as the lateral ones. Eleven gestures along the vertical axis include a verb, and four are time adverbials. Only those that include a time adverbial are related to time, and none is related to order. In one of the episodes that pertain to time, he makes a light downward gesture with his left hand while saying (tio sekunder, En. ten seconds) – a gesture that represents a limited amount of time. The rest of his gestures along the vertical axis are associated with time concepts that concern locations upwards or downwards, representing future and past times. In two such episodes, Teacher 2 makes upward gestures in relation to future results. In another episode, he makes a downward gesture in relation to what should be done later (i.e.,  sen anropar du den, En. then you call it).

In addition, Teacher 2 uses 50 container gestures, either involving opening and closing one hand or holding both hands in front of him as if holding something between them. Twenty of these containerlike areas are accompanied by verbless phrases (e.g., bra för dig, En. good for you, två olika datatyper, En. two different data types), and only five include time adverbials. Container gestures are the only type of gestures that he uses while talking about the order of something (e.g., of doubles and strings). Teacher 2 only uses seven iconic gestures, none of which are used in reference to time. Instead, the iconic gestures are used primarily in reference to real-world things such as full stops, buttons, and credit card numbers.