A mixed-methods investigation of third and sixth graders’ academic sentence knowledge

3.2.1 How readers navigate cohesion

Related to these conceptions of cohesion as more semantic than syntactic (e.g. Halliday and Hasan 1976) is the research that explores how readers connect sentences and ideas across a text. Research has revealed that there are students for whom understanding or using cohesive ties to make inferences is difficult (Cox et al. 1990). One factor that influences readers’ ability to comprehend cohesive ties in a text is readers’ background knowledge, which is defined as the sum of what an individual or group knows (Lee 2007), and can include many different types of knowledge such as topic knowledge, linguistic knowledge, cultural knowledge, and experiential knowledge. An additional factor that influences readers’ understanding of cohesive ties is their skill as readers (e.g. Fraser et al. 2021; Yuill and Oakhill 1988). For example, highly skilled readers are more likely to connect pronouns to the attributing word it represents than less skilled readers. The amount and levels of cohesion in a text also influences how readers navigate relationships between ideas in texts (Gasparinatou and Grigoriadou 2013; Hall et al. 2015; Ozuru et al. 2009). Texts can be highly cohesive, meaning there are many cohesive ties cuing readers to connect ideas between and within sentences and across the text. Or, texts can be low cohesive, meaning the reader has to work harder to infer how ideas are connected. Dahl and colleagues (2021) found that students’ produced more inferences when reading texts that were high in global cohesion. Relating to the interaction between levels of cohesion and readers’ knowledge, Dahl and colleagues (2021) found that readers made more inferences from a low cohesive text when they had topic knowledge related to the text. This research suggests that attending to the degree of cohesion in texts matters for students, given that text cohesion can mediate differences in relevant topic knowledge for readers (e.g. Reed and Kershaw-Herrera 2016).

3.2.2 Two types of cohesive ties: connectives and anaphoric references

Exploring students’ conceptions about semantic functions of different types of cohesive ties, specifically connectives and anaphoric references, was a goal of the current study. Connectives, also called conjunctions (Halliday and Hasan 1976), are words and phrases that show relationships between two ideas within or across sentences and paragraphs. For example, connectives can show how one idea elaborates on another (e.g. specifically), how one thing happened at the same time as another (e.g. meanwhile), and how one thing contrasts with another (e.g. despite). There are different ways to categorize connectives, but the most common categories are additive, temporal, causal, and adversative (Halliday and Hasan 1976).

Anaphora, or the use of anaphoric references, is commonly used to support cohesion in a text. An anaphoric reference is a word or group of words that are used in place of another word or group of words within the same sentence or in a different sentence. Morphologically, “ana” is from the root meaning “back” and “phora” is from the root meaning “carry”. Anaphora literally carries meaning in the text, back and forth, from one idea to its referent. Anaphora can be a type of grammatical cohesion, in which pronouns are used to refer to a previous word, or a type of lexical cohesion, in which a synonym is used to refer to a previous word. Some scholars reserve the term anaphora just for pronoun replacements (e.g. Moats 2020), while others articulate different types of anaphora (e.g. concrete vs. Conceptual) (Uccelli et al. 2015).

There is some evidence of relationships between knowledge of connectives and anaphora with reading and writing. First, there is correlational evidence that knowledge of connectives relates to reading and writing (Crosson and Lesaux 2013; Duggleby et al. 2016; Taylor et al. 2019). Some early studies on the impacts of sentence-combining instruction showed some positive effects on reading comprehension and writing (Hillocks 1986; Kanellas et al. 1998; Saddler and Graham 2005; Wilkinson and Patty 1993), suggesting that supporting students with connectives instruction and practice may be beneficial. However, notably absent from these studies are multilingual learners who navigate academic language with different language strengths than monolingual learners.

4.2.1 Qualitative participants

Three third grade students from Ms. Mason’s class, and three sixth grade students from Ms. Libler’s class were selected to participate in metalinguistic interviews (e.g. Myhill 2016) related to academic sentences (all names are pseudonyms). These interviews asked students to reflect on purposes and functions of specific cohesive ties in academic sentences. All six students were native English speakers and ranged in reading performance on standardized assessments of reading comprehension (i.e. i-Ready and MAP). Teachers selected students based on teachers’ learning goals for the professional inquiry collaboration (e.g. selecting students with a variety of reading abilities in order to better understand how students at different reading levels navigated cohesion).

4.2.2 Quantitative participants

The quantitative component of the investigation was based on a convenience sample of teachers to extend our inquiry to a larger group of students at Carson Charter. This sample included the students from three third grade classrooms (n = 72), including Ms. Mason’s class, and students from all of Ms. Libler’s social studies classes (n = 88). Inclusion criteria for teachers included those who were interested in and able to administer an experimenter-designed Maze task, designed to measure knowledge of cohesive ties, to their students. No students in the aforementioned classrooms were excluded, although several were absent on the day the Maze tasks were administered and there was no time available for make-up testing.

4.3.1 Qualitative data sources

For research questions (RQ) 1, data sources included audio recordings and transcripts from metalinguistic interviews with students, as well as researcher memos from each cycle of interviews. Interviews with the three students from Ms. Mason’s and Ms. Libler’s classes occurred four times, every two to three weeks, from February to May of the 2021–2022 school year.

4.3.2 Quantitative data sources

Following the interview cycles, we supported teachers in learning more about their entire classes of students, including both their knowledge of cohesive ties and relationships between knowledge of cohesion and performance on measures of reading comprehension. The two quantitative data sources were experimenter-developed Maze tasks and reading scores from the i-Ready and MAP standardized assessments.

Maze task for knowledge of cohesive ties. We designed two Maze tasks (see Appendix A) which targeted comprehension of cohesive ties for third and sixth grade students. To design the Maze tasks, we first selected passages from popular websites with grade-level specific passages for students (e.g. NewsELA, Scholastic, Readworks). Because Maze tasks are typically completed independently and silently by students, our collaborating teachers provided feedback on the passages selected for the Maze tasks. Teachers shared their impressions of which passages would be accessible for most of their students, thereby allowing us to informally ensure that performance on the Maze items would not be overly impacted by independent, silent reading ability.

To create the items for the Maze task, our approach contrasted with a typical Maze measure which involves the removal of every seventh word from a text. Instead, we removed words that served as cohesive devices. Items were developed to target knowledge of connective words (e.g. however, despite) and anaphoric references (e.g., pronouns or other references to earlier ideas or people). For each item, students had three words or phrases to choose from. The third grade Maze task had a total of eight items, with four each targeting connectives and anaphoric references. The sixth grade Maze task had a total of 16 items, with 11 targeting connectives and five targeting anaphoric references. Our teacher collaborators received training on the purpose and procedures of the Maze tasks and administered the Maze tasks with a standardized script and time parameters. An example from the sixth grade Maze tasks follows, with the first item targeting knowledge of connectives and the second targeting anaphora knowledge:

Most of us are likely familiar with traditional printers, which use ink to create words and images on paper. ______ (Similarly, Despite that, However), 3-D printers work differently. ______ (It uses, They use, We use) materials such as plastic, metal, or concrete to create solid three-dimensional objects – meaning they have length, width, and height.

Performance on reading comprehension measures: iReady and MAP. The second data source for the quantitative component of the study was reading comprehension performance, based on i-Ready and MAP reading comprehension scores. i-Ready Diagnostic is a computer-adaptive assessment of reading sub-skills (e.g. phonological awareness, phonics, comprehension in both informational text and literature) and growth in grade-level reading goals for grades K-12 (Curriculum Associates 2022). i-Ready Diagnostic provides normative and criterion-referenced data on student reading performance and strongly correlates with state summative assessments of English Language Arts (Curriculum Associates 2022). Reliability estimates ranged from 0.91 to 0.97 (Swain et al. 2020). i-Ready data were available for both third and sixth graders.

Measures of Academic Progress (MAP) (Northwest Evaluation Association 2008) is a computer-adaptive, untimed test of reading comprehension available for students in Grades 2–12 to assist teachers in determining students’ reading ability. MAP is a norm-based assessment for measuring comprehension growth utilizing a Rasch Unit score, or RIT score, to determine students’ reading comprehension ability. Marginal reliability for third grade is reported at 0.96 for third graders (Northwest Evaluation Association 2019). MAP data was available for third graders only.

Both the standardized reading measures were administered in the spring of the school year by school personnel as part of the school’s assessment plan, and scores were made available to the research team. Importantly, these standardized reading measures operate from an assumption that there is such a thing as “normal reading” (Huettig and Ferreira 2022: 727) that can be measured in a reliable way, which belies the many types of and purposes for reading.

4.4.1 Qualitative analyses

For qualitative data analysis, we adopted a theoretical thematic analysis stance (Braun and Clarke 2006: 84) drawing on our research questions and relevant literature to guide our analysis. Throughout data collection, we had already familiarized ourselves with the data through the creation of interview summaries. After data collection, we proceeded with initial coding (Saldaña 2016). We looked through the interview summaries, identifying ideas and patterns that aligned with our research questions; this process evolved into our creation of more formal codes that we used to code the entire data set of interview summaries (see Appendix B for coding scheme). Codes were then grouped together according to potential themes, and short narratives were written to further define each theme and ensure that all grouped data aligned with this potential theme. For example, “metalinguistic awareness” was frequently used to code both third and sixth grade students’ explicit discussion or manipulation of cohesive ties. This broad code was broken down into three revised codes: metalinguistic awareness of cohesive ties and their functions, elaboration on relationships and ideas signaled by cohesion, and students’ manipulation of cohesive ties (see Appendix B for definitions and examples of each code). Together, these codes constituted the theme, “Students leverage their developing metalinguistic awareness to comprehend and convey cohesion in disciplinary texts”.

4.4.2 Quantitative analyses

We analyzed data separately for the third and sixth graders. We performed item analysis to assess how well individual items on the Maze Tasks worked for evaluating student knowledge of connective words and anaphoric references. Correlation analysis was performed to determine the association between Maze task total score and reading proficiency. An alpha of 0.05 was used as the cutoff for significance. Finally, to examine the features of the texts for the Maze tasks, the Coh-Metrix Text Easability Assessor was used to evaluate five features of the text (i.e. narrativity, syntactic simplicity, word concreteness, referential cohesion, and deep cohesion).

5.1.1 Case 1 themes

Theme 1: Third graders demonstrated developing metalinguistic awareness of cohesion via explanations of cohesive ties. Based on the taught and cold sentences, students typically demonstrated clear metalinguistic awareness of cohesion. This metalinguistic awareness was evidenced by their ability to explain the functions of cohesive ties in an explicit, decontextualized way. In other words, they used their metalanguage, or their language about language, to explain what specific cohesive ties were connecting. For example, Tamara explained that the word then meant that “something else was going to come”, demonstrating that she understood the meaning of this word and the relationship it showed between the ideas in two sentences. Some students’ responses demonstrated metalinguistic awareness that was still developing, which was evidenced by their self-corrections of their explanations. At times, they would rethink their responses and then offer different explanations or manipulations of cohesive ties. However, across multiple questions, all three students demonstrated metalinguistic awareness of cohesion in their explanations and manipulations of cohesive ties (see Table 1).

Theme 2: Third graders demonstrated developing epilinguistic knowledge of cohesion. Students also demonstrated their epilinguistic knowledge of cohesion throughout the interviews. Epilinguistic awareness is that tacit, implicit, unconscious use of language. With respect to cohesion, epilinguistic awareness would allow for overall comprehension of sentences without being able to explain the functions of the cohesive ties in an explicit or decontextualized way. When students were unable to explicitly explain the functions of cohesive ties, they were still able to make judgments about what “sounded right” or what “didn’t sound right” throughout the interviews. For example, Ethan was able to paraphrase a sentence with the word while in it, but he could not explain the specific meaning of while. And, he had similar patterns of responses to questions that asked about both connectives and anaphoric references (see Table 1).

Theme 3: Third graders’ developing knowledge of cohesion is influenced by text-based, student-based, and instruction-based factors. Multiple factors appeared to contribute to students’ metalinguistic and epilinguistic knowledge of cohesion. Students typically relied on the ideas in the sentences as they explained or manipulated cohesive ties. In addition, they also drew on their knowledge about the topic and relevant experiences to explain the relationships that were represented by cohesive ties. For example, when explaining the function of the word “but” in the academic sentences, both Denise and Ethan gave examples that related to their own lives, such as “I like pizza but not with peppers”. With the sentence, “As you read the book, your imagination goes wild”, students were asked about the meaning of the word As, and all three students talked through the metaphor of your imagination going “wild” as they explained what the word As meant. Finally, teachers played a clear role in students’ knowledge of cohesion, both their developing epilinguistic and metalinguistic knowledge, as evidenced by the students’ explanations of teacher’s explicit instruction and scaffolding routines (see Table 1).

5.1.2 Case 2 themes

Theme 1: Sixth graders leverage their developing metalinguistic awareness to comprehend and explain cohesion in disciplinary texts. Students demonstrated accurate and explicit knowledge of cohesive devices, specifically connectives and anaphora, throughout the interviews. For example, Caroline used an analogy of a U-turn to describe the meaning of the word “although’. Most of the time, students were able to correctly identify the function of connectives and anaphoric references. Further, this metalinguistic awareness of connectives and anaphora enabled students to more deeply engage with and elaborate on relationships between ideas in disciplinary texts (both “taught” and “cold”) and writing tasks. Additionally, when students demonstrated strong metalinguistic awareness of cohesive devices, they were more likely to manipulate cohesion through accurate substitution (e.g. replacing a connective with another connective that functioned similarly and maintained correct syntax) and paraphrasing sentences that utilized cohesive devices (see Table 2).

Theme 2: Sixth graders draw on epilinguistic awareness to comprehend challenging uses of cohesion, compensating for developing metalinguistic awareness. Throughout the interviews, all three students encountered cohesion that they found to be challenging or unfamiliar and attempted to make sense of these cohesive devices through various strategies, relying on both epilinguistic and metalinguistic awareness. For example, when unsure of the function of a connective, students discussed the different connective types to try to determine which one it might be, approximating its function through discussing the ideas and relationships in the text. Additionally, when students were unsure of an anaphoric reference, they often reread the excerpt to try to determine what idea was being referenced. However, when metalinguistic awareness was underdeveloped, students relied heavily on their epilinguistic awareness to make sense of cohesion. For example, Emily grappled with connective functions and the term “conjunction” (see Table 2) – sometimes inaccurately – demonstrating developing knowledge of both cohesion and academic vocabulary in general. Overall, though, students were aware that metalinguistic awareness (e.g. explicitly knowing the function of a connective) would support them in their comprehension and attempted to leverage it where possible to make sense of the text.

Theme 3: Sixth graders develop metalinguistic awareness through explicit instruction in cohesion, specifically in the context of disciplinary texts. With prompting during the interviews and through spontaneous discussion about Ms. Libler’s practice, students attributed their metalinguistic awareness of cohesion to her instruction. Students identified explicit instruction (e.g. teaching definitions of connectives) and repeated scaffolding (e.g. collaborative discussion) of connectives and anaphora as essential to their understanding of cohesion. Most often, students discussed explicit instruction and scaffolding of cohesion as taking place in the context of discipline-specific texts and tasks. When asked what scaffolding would be most helpful for their learning of cohesion, students emphasized the importance of repeated exposures to connectives and anaphora within disciplinary texts as well as repeated opportunities to discuss challenging or unfamiliar uses of cohesion.

5.2.1 Item analysis

For the eight items on the third grade Maze Task, item difficulties ranged from 0.75 to 0.97. Seven of the eight items had item difficulties greater than 0.80. These results revealed that these items were too easy for these third graders. Point-biserial correlations between the response to each item and the total Maze score ranged from 0.20 to 0.69. Using 0.15 as a cutoff (Varma 2006), all the items performed well in terms of discriminating high-performing students from low-performing students.

For the 16 items on the sixth grade Maze Task, item difficulties ranged from 0.48 to 0.94 and seven of the 16 items had item difficulties greater than 0.80. These results revealed that some items were too easy for these sixth graders. Point-biserial correlations between the response to each item and the total Maze score ranged from 0.17 to 0.60. Using 0.15 as a cutoff (Varma 2006), all the items performed well in terms of discriminating high-performing students from low-performing students.

5.2.2 Correlation between maze task total score and performance on measures of reading comprehension

The mean and the standard deviation of the Maze task total score for the third graders were 7.24 and 1.18, respectively (Table 3). The mean and the standard deviation of i-Ready score for the third graders were 562.82 and 38.85, respectively. The mean and the standard deviation of MAP score for the third graders were 208.38 and 12.94, respectively. The Pearson’s r correlation between Maze task total score and i-Ready scale score was significant (r (64) = 0.38, p = 0.002). The Pearson’s r correlation between Maze task total score and MAP score was also significant (r (45) = 0.40, p = 0.005).

The mean and the standard deviation of the Maze task total score for the sixth graders were 11.73 and 2.52, respectively (Table 4). The mean and the standard deviation of i-Ready score for the sixth graders were 620.31 and 39.81, respectively. There was a significant correlation between Maze task total score and i-Ready score (r (82) = 0.68, p < 0.001).

Following these results, to further explore item difficulty and relationships with performance on measures of reading comprehension, we conducted a subsequent analysis on both passages using Coh-Metrix Text Easability Assessor, which evaluates five features of the text: narrativity, syntactic simplicity, word concreteness, referential (local, or lexical) cohesion, and deep (global) cohesion to determine what aspects of the text make it complex.

These language features would not have impacted the selection of the passages, since the passages were chosen in collaboration with the teachers. However, as a follow-up analysis, identifying these language features allowed us to understand how they may have influenced students’ responses and ability to understand the cohesive ties in the text. Table 5 presents these results and shows sharp contrasts between the third and sixth grade texts with respect to cohesion. Specifically, the third grade passage had much higher referential cohesion, which suggests more anaphoric references. In contrast, the sixth grade text was very low in referential cohesion, which likely required that students would have had to work harder to make inferences throughout the text.