Limits of Metacognitive Prompts for Confidence Judgments in an Interactive Learning Environment

1.1 Metacognitive Accuracy

Monitoring can lead to more or less accurate judgments about cognition. Metacognitive accuracy is the degree of correspondence between confidence and performance (Jang, Lee, Kim, & Min, 2020). It is further distinguished into several aspects of metacognitive accuracy, the most common ones being relative accuracy (resolution) and absolute accuracy (calibration [Jang et al., 2020; Schraw, 2009]). Absolute accuracy is low if a student works on a set of tasks and expresses a high confidence, for example, between 80 and 100%, for these tasks but then scores low. However, for the same confidence levels and results, relative accuracy can be high if the results are better on the tasks that had a confidence rating of 100% than on the tasks with a confidence of 80%. Conversely, relative accuracy can be low while absolute accuracy is high. Therefore, these two aspects of metacognitive accuracy are distinct. In assessments, both measures should be used to complement each other (Schraw, 2009), and students ideally should be accurate in both dimensions (Schwartz & Efklides, 2012).

Generally, metacognitive accuracy is found to be far from perfect (Glenberg & Epstein, 1985). Overall, students tend to be overconfident in predictions of their performance (Dunning, 2011; Hacker, Bol, Horgan, & Rakow, 2000; Maki & Berry, 1984; Miller & Geraci, 2011). More specifically, Hacker et al. (2000) found that high-performing students were rather accurate in predicting and postdicting (i.e., prediction after task completion) their exam scores. The highest-performing group even showed some underconfidence. In contrast, the lower performing groups showed overconfidence which increased as performance decreased (see also Bol, Hacker, O’Shea, & Allen, 2005; Lingel, Lenhart, & Schneider, 2019; Maki & Berry, 1984).

This is in line with the Dunning–Kruger–Effect (Dunning, 2011), which stipulates that less skilled people are overconfident in their self-judgments because their lack of skill keeps them from correctly assessing what they do not know. Highly skilled people show a weak tendency to underestimate their abilities (Schwartz & Efklides, 2012). Their more comprehensive knowledge of a domain enables them to better assess which aspects they might not know yet. However, since research shows that even high-performing K-12 students can be overconfident (Lingel et al., 2019), support structures for metacognition could benefit K-12 students from all achievement levels.

1.2 Metacognitive Prompts

For students to use metacognition effectively, they need the ability to monitor and control (Nelson & Narens, 1990), and they also need to use these abilities frequently enough (Bannert & Mengelkamp, 2013). Instruction can help establish the ability, while reminders in the form of prompts can help increase the frequency of metacognitive activities (Moser, Zumbach, & Deibl, 2017). Metacognitive prompts can take on different forms. In some studies, students are prompted to verbalize their thoughts and decisions while learning (think-aloud method [Bannert & Mengelkamp, 2008]), or prompts can ask the students to take notes on how they want to plan their learning (Zumbach, Rammerstorfer, & Deibl, 2020). Prompts can also ask the students how confident they feel in their answers (Feyzi-Behnagh et al., 2014). Such a prompt for a confidence judgment requires the learner to use cues in order to assess their own performance (Koriat, 1997). With practice, students should get more fluent in recognizing cues for understanding or a lack thereof. The more often students recognize a lack of understanding, the more often they have the chance to regulate their learning, for example, by rereading the instructions, and thus to improve their performance.

It is reported that metacognitive prompts can be beneficial for learning because they can make students reflect on their learning more often (Sonnenberg & Bannert, 2015), and with practice, confidence judgments can become more accurate (Feyzi-Behnagh et al., 2014). More accurate judgments help the students to regulate their learning more effectively and thus perform better. However, performance does not improve when students are able to monitor their learning process but fail to take the step of regulating their learning behavior (Dunlosky et al., 2021). There is also evidence that metacognitive prompts evoke neither monitoring nor regulation of the learning process (Johnson, Azevedo, & D’Mello, 2011), and in several studies, students did not use the prompts as intended (Bannert & Mengelkamp, 2013; Lingel et al., 2019; Moser et al., 2017). Therefore, evidence on the usefulness of metacognitive prompts is mixed.

3.1 The Platform and the Course

The study was conducted in cooperation with the German learning platform PearUp,^[1] which was founded in 2017 and has now merged with the platform eduki.^[2] PearUp was a for-profit learning platform where teachers could create interactive material and design interactive courses. Inherent to the learning experience with PearUp was a gamified meta-narrative: Students started their own “start-up business,” and whenever they solved tasks, they collected “PearCoins,” which they could invest into their start-up. In terms of feedback, “PearCoins” were also a rough quantitative measure of how many tasks they had completed compared to their peers. There was a leaderboard of the three students scoring highest on the indicators of the start-up. As part of the treatment implementation, two dashboards were developed, showing the rate of correct first attempts and metacognitive accuracy. Because these features were implemented in the live product, it was not possible to conduct randomized A/B-Testing but instead the experimental data were collected after implementation, and data from before the implementation were used for control.

The course titled “Introduction to Computational Thinking” was designed by the content creators of PearUp. The average duration was stated as 2h. There were six units on: “Sequences and Algorithms,” “While-Loops,” “For-Loops,” “Comparing While-Loops and For-Loops,” “Conditions,” and “Nested For-Loops.” The course was designed as a fully online, self-paced course. The majority of the tasks were interactive, such as multiple-choice questions and coding tasks.

3.2 Design of the Prompts

As discussed earlier, in experiments, metacognition is often measured through verbalizations made by the learners after completing a set of tasks. However, online monitoring methods correlate more strongly with higher performance than offline measures (Ohtani & Hisasaka, 2018), so in this study, metacognitive prompts were presented online, with each multiple-choice question.

The prompts consisted of four buttons representing a 4-point confidence scale: “sure, rather sure, unsure, no clue.” The buttons took the place of the “Submit”-button, so that in order to submit the task, students had to click one of the four buttons. Figure 1 shows the multiple-choice question design with the metacognitive prompt.

Figure 1

Exemplary MC task with metacognitive prompts below, from the left: “sure,” “rather sure,” “unsure,” and “no clue” – experimental group (screenshot taken by the first author).

3.3 Sampling

Before the implementation of the prompts, 150 students had completed the course already, and their data were used for the control group. No personal data were gathered but teachers had to give a “class name” when they wanted to use the material for their groups. These class names were provided by PearUp, and most of them implied the students’ grade levels. Thus, it was inferred that the students in the control group were aged 11–18. The class names also implied that many students accessed this course as part of an extracurricular activity.

Data for the experimental group were gathered in several ways: One of the authors taught the course as part of classes and extracurriculars; 28 students completed the course this way. These were students from grades 7 to 9, so they were aged 12–16. The rest of the data stem from classes which were taught by other teachers. These teachers were recruited through PearUp and call for participation on social media; 23 students from a similar completed the course in this way. The teachers reported a range of ages 13–16. Together, this results in a sample size of 51 for the experimental group.

PearUp complies with the General Data Protection Regulation, and no personal data were gathered. The students were informed that the anonymous interaction data gathered by the platform would be used for research purposes, and they were given the option not to use the material. None of the students chose this option.

3.4 Data Processing and Analysis

PearUp provided the raw data as well as a Jupyter Notebook file with some preliminary pre-processing code written in Python. For further pre-processing, Python code was written in a Jupyter Notebook, using the pandas library. Performance was coded as correct (1.0) and wrong (0.0). Metacognitive judgments were coded as sure (1.0), rather sure (0.66), unsure (0.33), and no clue (0.0).

3.5 Measures for Metacognitive Accuracy

Absolute metacognitive accuracy measures how well students can judge their performance. Schraw (2009, p. 36) suggested the Absolute Accuracy Index:

Performance is scored as either 0 or 1. The learners give a confidence rating ranging between 0 and 1. The performance score is subtracted from the confidence score, resulting in a number between −1 and 1. This number is squared, so the result ranges from 0 to 1, with 0 indicating the highest accuracy because the deviation between confidence and performance is the lowest. Division through the number of items results in the mean absolute accuracy.

Relative metacognitive accuracy measures how confidence and performance correlate. Thus, one established measure of relative metacognitive accuracy is Pearson’s r (Schraw, 2009).

4.1 Descriptive Statistics

4.2 Testing the Hypotheses

5.1 Empirical Contributions

There is mixed evidence on the effects of metacognitive prompts on students’ performance. Metacognitive prompts were shown to lead to better performance in some cases (Renner & Renner, 2001; Veenman, Kok, & Blöte, 2005), though some found an effect only for transfer tasks (Bannert & Mengelkamp, 2013; Lin & Lehman, 1999). Along these lines, Stark and Krause (2009) found improved performance only for complex tasks, not simple ones. As the tasks used in the course for this study were less complex, the lack of performance improvement could be a further indication of this pattern.

There is less research on whether students improve metacognitive accuracy with repeated self-assessment. Here, the students did not improve their metacognitive accuracy, which is in line with studies that saw no improvement in calibration after several training sets of quizzes or practice tests (Bol et al., 2005; Bol & Hacker, 2001). In a study by Hacker et al. (2000) students made predictions and postdictions for three exams and throughout the course, there was instruction and emphasis on the benefits of self-assessment. Here, the high-performing students, but not the low-performing students, showed an increase in accuracy. The present study provides evidence that without additional instruction, repeated exposure to metacognitive prompts does not increase metacognitive accuracy.

5.2 Practical Contributions

When designing interactive learning environments, prompts like the ones used in this study are relatively easy to implement and could be used as part of the default course design.^[3] However, as a limit, it should be critically examined whether such prompts by themselves have a beneficial effect. As Schwonke (2015) pointed out, metacognitive processes could be ineffective or even detrimental if they use cognitive resources in a way that does not support the learning process. Schwonke suggested categorizing metacognitive load as a kind of working memory load. It is plausible to assume that mismatched or overly complex metacognitive prompts hinder the learning process.

And yet, metacognitive prompts – even in the simple form that was used in this study – might be beneficial if they are complemented with explicit instruction on metacognition and its role in learning (Kistner et al., 2010) and enough training time for students to engage in monitoring (Bannert & Mengelkamp, 2013). In order to avoid fatigue, these prompts could be used for a limited amount of course time with instruction in the beginning and formative reflection throughout the course. Furthermore, the learning environment used for this study had a gamified meta-narrative, but students did not receive game benefits if they judged themselves correctly. In a future iteration of the prompting feature, it could be tested whether game benefits for accuracy could provide an incentive for students to invest the required mental effort demanded by monitoring.

5.3 Limitations and Future Research

Above all, the lack of variation in confidence judgments indicates that metacognitive prompts were not used as intended, especially regarding, but not limited to, the students who only used the highest confidence judgment. This non-compliant use presents an issue concerning validity and impedes testing the hypotheses to some degree. Conclusions about the effects of metacognitive prompts can only be made to the degree that the selection of the confidence button reflects an actual metacognitive judgment made by the student.

This student behavior might have been caused by some choices in the sampling and research design. The sample of the prompted group was primarily gathered in contexts of extracurricular, voluntary activities where students might not have been motivated to invest the extra mental effort demanded by the prompts. On top of that, the gamified design of the learning platform might have contributed to this by evoking a playful sense of learning which does not ask for the increased mental effort required by metacognitive monitoring and control. However, as described earlier, participants in a study by Lingel et al. (2019), students of similar demography, showed similar overconfidence in a more formal setting. Still, future studies could investigate whether students react to such prompts differently in more formal learning contexts.