A Pilot Study on Investigating Primary School Students’ Eye Movements While Solving Compare Word Problems

2.1 Comprehension of Mathematical Text Problem

A considerable part of the mathematical tasks in primary school is in the form of word problems. An important percentage of these problems is arithmetic word problems, which require performing some mathematical operations to obtain the solution. The mathematical operations are embedded in the text instead of presenting it in mathematical form. Thus, text comprehension skills are necessary to translate the text into mathematical operations.

The solution of a mathematical word problem is obtained by going through several stages using cognitive operations (Blum & Leiß, 2007). Understanding the real situation (the word problem), the situation model can be obtained. Simplifying and structuring the information from the situational model, the real model is created, which can be mathematizing to obtain the mathematical model. After performing the mathematical operations needed, a mathematical result is obtained, which has to be interpreted to obtain the real results. The real result is validated by the situation model, and then the result of the problem is formulated and explained.

Although difficulties could occur in all the cognitive operations presented in the above model, the construction of the situation model has a crucial role in successfully solving the problem (Stephany, 2021). Some of the solvers even skip this step and perform some operations using numbers and relational terms from the text, using the so-called direct approach (Hegarty et al., 1995). This approach could lead to incorrect solutions, for example, in the case of inconsistent language (IL) compare problem. To obtain a correct solution, the solver has to understand the context of the problem to identify the variables and the relation between them interpreting correctly the relational words, i.e., building a correct situational model.

Another crucial aspect with regard to word problem comprehension is related to the magnitude-based mental representation of the variables (Múñez et al., 2013; Orrantia & Múñez, 2013). This means that during problem-solving, the magnitude of each variable should be mentally represented, which model also reflects the relation between these variables. For a successful solution, building a situational model and magnitude-based representational model could be decisive.

2.2 Consistency Effect in Compare Word Problems

A type of arithmetical problem is the compare problem in which a static numerical relation is given between the values of two variables (Hegarty et al., 1992). Research shows that compare problems are more difficult for students than other arithmetic problems (Boonen & Jolles, 2015; Riley et al., 1983).

Compare problems can be formulated using consistent language (CL) or inconsistent language (IL). In the case of CL problems, the unknown variable is the subject of the second sentence, and the relational term is consistent with the arithmetic operation, which has to be used to find the solution (Lewis & Mayer, 1987). In the case of IL problems, the unknown variable is the object of the second sentence, and the relational term is not consistent with the arithmetical operation (Lewis & Mayer, 1987). In Table 1, examples of CL and IL problems are given. Most of the studies regarding compare problems conclude that students are less successful in solving IL than CL problems (Múñez et al., 2013; Orrantia & Múñez, 2013; Pape, 2003; Riley & Greeno, 1988), and they also need more time to solve IL problems (Múñez et al., 2013). This was labeled the consistency effect (Lewis & Mayer, 1987). The main difficulty in the case of IL problems is due to the fact that the relational term is not consistent with the mathematical operation which has to be performed; thus, students have to make a reversal of the relational term (Lewis & Mayer, 1987). Another difficulty is related to the presence of a pronominal reference in the case of IL problems (Hegarty et al., 1992); thus, students have to search for the referent of the pronoun. The third difficulty is related to the role of the unknown variable from the second sentence, which is the subject in CL problems and the object in IL problems (Lewis & Mayer, 1987). In contrast with previous studies, Boonen and Jolles (2015) report no consistent effect in their study of 7–8-year-olds.

The relational term in compare problems can be unmarked (i.e., “more”) and marked (i.e., “less”), see examples in Table 1. Students more often commit a reversal error in marked IL problems than in unmarked ones (Boonen et al., 2016; Lewis & Mayer, 1987). By analyzing brain activity with functional magnetic resonance imaging methods while solving compare problems, the results show that in the case of problems that require performing addition for the solution, IL was associated with stronger brain activation, but the consistency effect was inverse when the solution requires subtraction (Ng, Lung, & Chang, 2021).

3.1 Scope and Hypothesis

The scope of the research is to study how primary school pupils comprehend the text and solve compare word problems by analyzing ET data. As part of this question, this article has two main aims: (1) to compare fixation durations on different text elements in the case of IL and CL problems and (2) to compare fixation durations on different text elements by successful and unsuccessful solvers. In both cases, we had formulated a hypothesis that had not been studied before.

The first hypothesis is that the fixation duration on numbers is longer in the case of IL problems than in the case of CL problems. To solve a word problem, it is evident to use the numbers from the text. So, even if someone does not really understand the relations of variables, he/she tries to do some operation with the numbers (Hegarty et al., 1995). However, when dealing with an inconsistent task, the relational term must be inverted for the correct solution (Lewis & Mayer, 1987), and it can be assumed that this difficulty affects the fixation duration of the numbers as well. Although, as mentioned before, unsuccessful solvers spend more time fixating on numbers (Hegarty et al., 1995), we are interested in whether this also applies to successful solvers. Moreover, if the reversal of the operation in IL problems increases the working memory load, a second hypothesis can be formulated as the whole relational sentence is fixated for a longer time in IL problems compared to CL problems. In addition to numbers, in this sentence, there are other important data as well. A difficulty in understanding the text of the IL compare word problems is related to the presence of the pronominal reference in the relational sentence (Hegarty et al., 1995). To identify the referent of the pronoun, the solver has to search the statement sentence. Unsuccessful problem solvers could have difficulty identifying the variable name to which the pronoun refers. Thus, the third hypothesis of this study is that unsuccessful problem solvers have longer fixation duration on the pronominal reference word in the case of IL problems compared to successful problem solvers.

The method of choice to study these questions is ET, since this technology has become highly efficient in the past years. ET is the most effective way to measure and record eye movements on different text elements, word problems in this case. Furthermore, numerous previous studies (Daroczy et al., 2015; De Corte et al., 1990; Hegarty et al., 1992; van der Schoot et al., 2009) used this technology to analyze the solving process of arithmetic word problems.

Analyzing data collected by this technology goes beyond simply determining the locus and the time of different fixations; but it can also reflect the problem-solving strategy used by the pupil. Fixation time on a determined area of interest (AOI) can indicate different parts of the strategy used, for example, identifying the relevant information in the text. Moreover, by synthesizing these data, it can be concluded whether the pupil was able to construct the situation model or used the direct approach in the problem-solving process.

3.2 Participants

Forty-two 10–11-year-old pupils (4th graders, 26 girls, and 16 boys) from Romania participated in the study. The participants were selected from four different schools where the language of instruction is Hungarian. Parental consent was obtained, and participation was voluntary. The study utilized convenience sampling to select the classes, taking into consideration to include both urban and rural classes. Pupils were randomly chosen from these classes. Five pupils were excluded from the analysis due to vertical drift or poor reading ability, which would result in data distortion.

3.3 Material

In this study, two-step compare problems were used, which were written in Hungarian language. The problems contain three sentences. The first sentence is an assignment sentence giving the numerical value of the first variable. The second sentence is relational, giving the numerical relation between the values of the two variables. The third sentence is the question that asks for the value of some quantity of the second variable. This type of problem has been used in several studies (Hegarty et al., 1992, 1995; Lewis & Mayer, 1987; van der Schoot et al., 2009); therefore, the tasks were based on the same schemata, but on different familiar contexts. From the pupils’ point of view, it was necessary to avoid the feeling that they were given the same task again and to prevent overloading working memory by comparing consecutive tasks.

The difficulty of the problems corresponds to the cognitive level and the curriculum requirements. From a methodological standpoint, it would have been preferable to include more problems, but the attentional capacity of primary school children is limited compared to adult problem solvers. With respect to arithmetical operations, the solutions required addition/subtraction and multiplication with one- and two-digit numbers. The most difficult arithmetic operations required subtraction with carrying and multiplication of a two-digit number by a one-digit number.

In the test, four problems were included (Table 2), two of them CL and two of them IL problems. In the case of both CL and IL problems, in one of them, the relational term was unmarked, and in the other one, marked. The order of the items was the same for all participants. The difficulty of the problems increased gradually, while CL and IL problems were presented alternately.

3.4 Apparatus

To study the performance and text reading patterns of primary school pupils while solving arithmetic compare word problems, Tobii Pro Fusion hardware and Tobii Pro Lab Screen-Based Edition software were used to collect eye movement data. It samples real-time fixations at a 250 Hz sampling rate. The eye-tracker was connected to a laptop (with a 15.6-in., non-touch screen) and positioned beneath its screen. A separate room within the schools served as the location for data collection, which contributed to the comfortable and usual feeling and the ease of testing and measurement. We considered using a familiar environment to be more important than the advantages of the laboratory environment. During data collection, adequate lighting conditions and limitation of distractions were ensured.

3.5 Procedure

Instructions before testing consisted of first explaining to the child that four simple arithmetic word problems will appear on the screen. It was then told that position herself/himself comfortably on the chair at the beginning because it is important to stay mostly stable during working with the tasks and not cover his/her eyes. The distance between the stimuli (the laptop screen) and the participant was about 60 cm. Before starting the study, the experimenter calibrated the eye-tracker for the participants (five calibration and four validation points were used). Participants were instructed that word problems would appear on the screen, and their task was to read the problem from the screen and write down the arithmetical calculations and the solution of each task on the answer sheet. Participants were also told that there are no time constraints and they can work at their own pace. Moving from one problem to another was managed by the experimenter. Pupils were not allowed to touch the laptop.

3.6 Data Analysis

For the two types of items (CL and IL), different AOIs were established. Common areas were the following: numbers, relational terms, and sentences of the problems (sentence 1 – statement sentence, sentence 2 – relational sentence, and sentence 3 – question). For the CL problems, one additional AOI was defined: the subject in the relational statement. For the IL problems, the pronominal reference word in the relational sentence was defined as additional AOI. For the established AOIs, fixation duration and number of fixations were recorded, as well as solution time for every problem. In the data analysis, descriptive statistics (mean, standard deviation) and comparisons with non-parametrical tests (Friedman test, Conover’s post hoc test, Wilcoxon matched-pairs signed rank test, and Mann–Whitney’s test) were performed.

4.1 Fixation Duration for Items and Correctness of the Solution

By analyzing the answer sheets, all solutions are considered correct where the pupil wrote the arithmetic operation correctly, even if the final result was incorrect due to a minor calculation error.

Figure 1 shows how many of the problems were solved correctly. One-third of the pupils successfully solved all the problems, one-third of the pupils successfully solved three problems, one-third of the pupils successfully solved one or two problems, and there were three who failed to complete any problem correctly.

Figure 1

Number of successful and unsuccessful solutions per item.

In the following, when speaking about performance, pupils are divided into two groups based on their results: successful solvers and unsuccessful solvers. Successful solvers have solved at least three problems correctly, and unsuccessful solvers have solved 0, 1, or 2 problems correctly. Considering previous studies (Boonen et al., 2016; van der Schoot et al., 2009), grouping pupils by their arithmetical performance was necessary to identify whether there is any difference regarding their reading pattern, specifically with reference to the third hypothesis.

The success rate was different for each item (Figure 1). Item 3 proved to be the easiest, followed by Item 1 (these two items are CL problems). The fact that no warm-up task was offered prior to Item 1 may have negatively influenced the success rate of this problem. Items 2 and 4 proved to be the most difficult, as they are both IL problems.

The performance presented above and the solution time of each problem are not clearly related. The pupils spent the least time on Item 3 (Table 3), which proved to be the easiest problem. Although Item 1 had a high correct solution rate, participants spent the most time on average on this problem. A Friedman test was carried out to compare the total solution duration for the four items. A significant difference was found between the solution duration of the items, χ ² (3) = 15,800, p = 0.001. Based on Conover’s post hoc test, a significant difference was found between Items 1 and 3 (p = 0.001), Items 2 and 3 (p = 0.001), and Items 3 and 4 (p = 0.010). The solution duration for the two IL problems, Items 2 and 4, is similar (Table 3). A significant negative correlation between average solution duration and performance was found: unsuccessful problem solvers spend more time solving the problems (r = −0.439, p = 0.004).

4.2 Fixation Durations for the Sentences of the Problems

To compare fixation duration on different sentences of the problems, fixation durations were weighted by the number of characters, the comparisons were made on the average fixation duration on a character from each sentence.

Comparing the average fixation duration on sentences in the case of the consistent problems, the Friedman test shows significant differences, χ ² (2) = 41.333, p < 0.001. The decreasing order of average fixation durations on sentences is the following: Sentence 1 (M = 607.91, SD = 382.80), Sentence 2 (M = 524.62, SD = 316.10), and Sentence 3 (M = 398.00, SD = 261.15). Conovers’ post hoc test indicates significant differences between the average fixations on characters from Sentences 1 and 2 (p = 0.032), Sentences 1 and 3 (p < 0.001), and Sentences 2 and 3 (p < 0.001).

Comparing the average fixation duration on sentences in the case of the inconsistent problems, the Friedman test shows significant differences, χ ²(2) = 41.333, p < 0.001. The decreasing order of average fixation durations on sentences is the following: Sentence 2 (M = 706.60, SD = 437.12), Sentence 1 (M = 650.43, SD = 371.83), and Sentence 3 (M = 372.36, SD = 261.83). Conovers’ post hoc test indicates significant differences between the average fixations on characters from Sentences 1 and 3 (p < 0.001) and Sentences 2 and 3 (p < 0.001), but there is no significant difference between Sentences 1 and 2 (p = 0.278).

The Wilcoxon matched-pairs signed rank test indicates that there is a significant difference in the average fixation duration on Sentence 2 between consistent and inconsistent problems (Table 4).

Comparing the average fixation duration on characters from different sentences in the case of successful and unsuccessful pupils, the Mann–Whitney U test indicates that the unsuccessful pupils fixate more on each sentence both in the case of CL problems and IL problems (Table 5).

In the case of successful solvers, Friedman test indicates significant differences between average fixation duration on different sentences both for consistent problems (χ ² (2) = 9.500, p = 0.009) and inconsistent problems (χ ²(2) = 18.500, p < 0.001). The decreasing order of average fixation durations on sentences is Sentence 1, Sentence 2, and Sentence 3 in case of consistent problems, and Sentence 2, Sentence 1, and Sentence 3 in case of inconsistent problems (Table 4). In both cases, Conovers’ post hoc test indicates significant differences between the average fixations on characters from Sentences 1 and 3 (p = 0.008 for CL problems and p = 0.001 for IL problems) and Sentences 2 and 3 (p = 0.019 for CL problems and p < 0.001 for IL problems).

In the case of unsuccessful solvers, Friedman test indicates significant differences between average fixation durations on different sentences both for consistent problems (χ ² (2) = 34.154, p < 0.001) and for inconsistent problems (χ ² (2) = 29.154, p < 0.001). The decreasing order of average fixation durations on sentences is similar to in the case of successful solvers: Sentence 1, Sentence 2, and Sentence 3 in case of consistent problems and Sentence 2, Sentence 1, and Sentence 3 in case of inconsistent problems (Table 4). In both cases, Conovers’ post hoc test indicates significant differences between the average fixations on characters from Sentences 1 and 3 (p = 0.008 for CL problems and p = 0.001 for IL problems) and Sentences 2 and 3 (p = 0.019 for CL problems and p < 0.001 for IL problems). Additionally, in the case of CL problems, there is also a significant difference between average fixations on Sentences 1 and 2 (p = 0.016).

4.3 Fixation Duration on the Problems’ Data

In the following, the fixation durations of data are described. These are the relevant data from the problems that pupils must use to successfully solve the problem: three different numbers, relational words, reference words, and the names of the variables. The first number in the problems represents the value of the first variable, and the second number represents the difference between the first and second variables. The third number appears in the question, and this represents the quantity that must be taken from the second variable (i.e., the value of the second variable has to be multiplied by this number).

Total fixation durations on numbers from different items are presented in Table 6. The Friedman test indicates a significant difference between fixation durations on numbers from different items, χ ² (3) = 26.486, p < 0.001. Pupils fixated the longest on numbers from Item 4 and based on the fixation duration on numbers from Item 4 is significantly higher than for numbers from Item 1 (p = 0.025) and from Item 3 (p < 0.001). Pupils fixated at least on numbers from Item 3, and the fixation duration on numbers from Item 3 is significantly shorter than on numbers from Item 1 (p < 0.001) and Item 2 (p = 0.020).

Comparing consistent and inconsistent problems, the Wilcoxon signed-rank test indicates a significantly longer fixation duration (W = 227.000 and p = 0.004) on numbers from IL problems (M = 17783.93) than on numbers from CL problems (M = 12964.98). This does not apply to consistent problems. The fixation duration on numbers takes 14.83% of the total fixation duration in the case of CL problems, and 19.41% in the case of IL problems.

Comparing average fixation duration on numbers from different sentences in the case of successful and unsuccessful solvers with the Mann–Whitney U test, the results indicate that in the case of consistent problems, there is a significant difference for Number 1 and Number 2, unsuccessful solvers fixated significantly longer on these numbers (Table 7). There are no significant differences between fixation duration on numbers by successful and unsuccessful solvers in case of inconsistent problems.

Comparing fixation durations on different numbers in the case of unsuccessful solvers, the Friedman test indicates that there is no significant difference in the case of CL problems, χ ² (2) = 1.462, p = 0.482. In the case of IL problems, the Friedman test indicates that there are differences between fixation duration on different numbers, χ ² (2) = 13.000, p = 0.002. The numbers from the relational sentences were significantly shorter time fixated than numbers from the statement sentence (p = 0.002) or from the question (p = 0.004). Making the same comparisons in the case of the successful solvers, the Friedman test indicates that there is no significant difference in the case of CL problems, χ ² (2) = 3.875, p = 0.144. In the case of IL problems, the Friedman test indicates that there are differences between fixation duration on different numbers, χ ² (2) = 7.125, p = 0.028, but the difference is only between Number 1 and Number 2, successful solvers fixated significantly shorter time the number from the relational sentence than the number from the statement sentence (p = 0.038).

In addition to the numerical data, other important data stand out from the context, which is essential for a successful solution: the name of the second variable in the case of Items 1 and 3, the reference word (which refers to the value of the first variable) in the case of Items 2 and 4, and the relational term for each item.

As regards the relational terms from the problems, two relational terms appeared: more (in Items 1 and 2) and less (in Items 3 and 4). The Wilcoxon signed-rank test indicates that pupils fixated significantly more (W = 290.000, p = 0.043) on the relational terms of the IL problems (M = 3974.21) than the CL problems (M = 3193.52). In addition, the Wilcoxon signed-rank test indicates that pupils fixated significantly more on the relational terms of the marked problems (M = 4024.29) than the unmarked problems (M = 3143.45), W = 289.000, p = 0.042. Analyzing separately the fixation durations on the relational terms by unsuccessful and successful problem solvers, the Mann–Whitney U test indicates that the only significant difference is in the case of unsuccessful solvers; they have fixated significantly more on the relational terms of the IL problems (M = 3461.77) than the CL problems (M = 2304.19), W = 92.000, p = 0.033.

The pronominal reference from the IL problems was significantly longer fixated by unsuccessful solvers (M = 2217.38) than by successful solvers (M = 1756.70), as indicated by the Mann–Whitney test (W = 286.000, p = 0.045).