ChatGPT as a direct formative assessment tool in stoichiometry: correlation with achievement and insights into pre-service science teachers’ perceptions

2.1 Research design

This study uses a mixed-method research design, in particular an explanatory sequential design. ⁴³ The overall research process is illustrated in Figure 1. The aim was to investigate the use of ChatGPT as a tool for assessing formative understanding in a stoichiometry unit. The initial phase of this research involved collecting quantitative data and using a correlational research design. This phase aimed to examine the relationship between the ChatGPT assessment scores and final achievement scores (RO1), as well as the relationship between the ChatGPT assessment scores and the instructor assessment scores (RO2). These quantitative data were analyzed using Spearman’s rank correlation coefficient (r_s) to determine the magnitude and direction of the relationships between these variables.

Figure 1:

Explanatory sequential mixed-methods research design employed in the study.

The subsequent qualitative phase was designed to explain the quantitative findings, focusing specifically on the relationship between ChatGPT assessment scores and final achievement scores as identified in RO1. Semi-structured interviews served as the primary method for exploring pre-service science teachers’ perceptions of utilizing ChatGPT as a formative assessment tool (addressing RO3). The interview transcripts were subjected to thematic analysis to identify key themes reflecting students’ perceptions and experiences. ⁴⁴ Ultimately, these qualitative findings were integrated to provide in-depth explanations that enrich the interpretation and contextual understanding of the quantitative data.

2.2 Participants

The participants in this study were drawn from a cohort of 62 first-year pre-service science teachers enrolled in the Chemistry for Teachers 1 course within a faculty of education (equivalent to a school or college of education) at a university in Bangkok, Thailand. For clarity, these participants are hereafter referred to as students. The term pre-service science teachers will be retained where their professional context is relevant. The learning activities described in Section 2.3 were a part of the standard coursework for all enrolled students. Prior to the study commencing, all students received an information sheet and consent form outlining the research objectives, procedures, and their rights as participants. It was explicitly stated that participation was voluntary, that students could withdraw their consent for data use at any time without penalty, and that their decision would not affect course grades or standing (see Research Ethics and Informed Consent sections for details).

For the final data analysis, data were included only from students who met the following inclusion criteria: (1) remained enrolled in the course throughout the instructional period and (2) completed all 10 learning processes. A total of 38 students who met these criteria also provided signed voluntary informed consent for their data to be used in the study. Therefore, the complete dataset from these 38 students was used for the final analysis. Detailed demographics of the students are presented in Supplementary Information (Section S3).

For the qualitative phase, students from the main cohort of 38 were invited to voluntarily participate in semi-structured interviews. From those who provided prior consent and were available, students were purposefully selected using a maximum variation strategy to capture diverse perspectives in terms of achievement level and gender. The final number of interview participants was determined during data collection based on thematic saturation (see Section 2.5 for details).

2.3 Learning design and ChatGPT integration

The instructional method employed for the stoichiometry unit was a structured learning process grounded in the principles of formative assessment. ^{45 , 46} This approach emphasizes the continuous monitoring of learner understanding and the provision of individualized feedback to enhance learning throughout the instructional process. The stoichiometry unit’s instructional content was primarily based on the textbook chemistry, ⁴⁷ adapted to align with the Thai higher education curriculum context. This content was divided into 10 distinct subtopics. For each subtopic, the instructor established clear Lesson Learning Outcomes (LLOs) to define specific learning objectives and ensure that they are consistent with the course learning outcomes (a detailed description of LLOs is provided in Table S1, Supplementary Information). This five-step ChatGPT-enhanced formative assessment process is illustrated in Figure 2 (detailed descriptions of each step are provided in Section S2 of the Supplementary Information). For each of the 10 subtopics, the learning process began with Instruction & Concept Building, followed by Independent Practice and a Class-wide Review.

Figure 2:

ChatGPT-enhanced formative assessment learning process for stoichiometry unit.

The final two steps centered on the ChatGPT interaction. In the fourth step, ChatGPT Formative Assessment, students were first presented with a distinct set of formative conceptual understanding assessment questions. They then submitted their answers to these questions to ChatGPT using the following standardized prompt:

Assess what percentage I understand for each question, provide suggestions for each, and assess my overall understanding percentage for this topic, offering recommendations for learning improvement.

A key methodological choice was the deliberate exclusion of a specific scoring rubric from the prompt given to ChatGPT. This was to ensure that the AI’s assessment was an authentic assessment based on its internal models, rather than a test of its ability to follow prescribed criteria. This approach allowed for a genuine benchmark of its Large Language Model (LLM) capabilities against human assessment.

In the final step, Data-Informed Reflection & Reporting, students used the ChatGPT-generated feedback and scores to reflect on their understanding and were required to report their scores and submit screenshots of the entire interaction via a designated Google Form. This five-step learning process was repeated for each of the 10 subtopics over a four-week instructional period. It should be noted that this instructional design did not involve instructor-led reteaching of a subtopic. Instead, the formative adjustment was designed to occur at the student level, where students were expected to use the Gen AI feedback to self-regulate and adapt their understanding and learning strategies. Upon completion of all 10 learning processes, all participating students undertook the final achievement test.

2.4 Instruments

The instruments employed in this research included (1) a 30-item multiple-choice stoichiometry achievement test, (2) 10 sets of formative conceptual understanding assessment questions, and (3) a semi-structured interview protocol. The development, validation, and reliability details for each instrument are fully documented in the Supplementary Information (Sections S4–S6), which also includes the complete interview protocol and examples of the formative conceptual understanding questions.

2.5 Data collection

The quantitative data for RO1 and RO2 were gathered during the four-week instructional period. The ChatGPT assessment scores were collected via designated Google Forms after each of the 10 learning processes described in Section 2.3. Upon completion of the entire instructional period, the final achievement scores were collected from all participants.

To fulfill RO2, student responses to the formative conceptual understanding assessment questions (captured as screenshots of their interactions with ChatGPT and subsequently collected via Google Forms) were assessed by three chemistry instructors (the course instructor and two external chemistry instructors). Each expert independently assessed responses using an identical scoring rubric developed by the course instructor. To establish Inter-Rater Reliability (IRR), the Intraclass Correlation Coefficient (ICC) was calculated (two-way mixed effects, absolute agreement). The ICC for average measures (N = 3 raters) was 0.973 (95 % CI [0.951, 0.985], p < 0.001), indicating excellent inter-rater reliability. ⁴⁸ After high IRR was confirmed, the scores assigned by each instructor based on the rubric were converted into percentages. For each student, the percentage scores from the three experts were then averaged. This final averaged percentage value represented the instructor assessment score and was then employed in the correlation analysis against the ChatGPT assessment score to fulfill RO2.

Qualitative data were collected through semi-structured interviews. As described in Section 2.2, data collection and preliminary analysis were conducted concurrently in an iterative process. Interviewing continued until thematic saturation was reached, which was determined when no new significant themes emerged from the data after the 13th interview. ⁴⁹ The interviews were conducted face-to-face using a semi-structured protocol, with probing techniques employed to encourage open sharing. Each session lasted approximately 40–45 min, was audio-recorded with permission, and was subsequently transcribed verbatim for analysis. In total, 13 students participated in the interviews, comprising high-achieving (n = 4), medium-achieving (n = 5), and low-achieving students (n = 4) based on their midterm scores, with seven males and six females.

2.6 Data analysis

Quantitative data analysis began with testing the normality of the data distribution for the primary variables. These variables included the ChatGPT assessment scores, final achievement scores, and the instructor assessment scores. This assessment was conducted utilizing the Shapiro-Wilk test. The results indicated that the distributions for all three variables significantly deviated from normality (p < 0.05). Based on this finding of non-normality, Spearman’s rank correlation coefficient (Spearman’s rho) was chosen to analyze variables according to RO1 (relationships between ChatGPT assessment scores and final achievement scores) and RO2 (relationship between ChatGPT assessment scores and instructor assessment scores). All statistical analyses were performed using IBM SPSS Statistics (version 26.0; IBM Corp., Armonk, NY, USA).

Data from semi-structured interviews were analysed using thematic analysis, following the six-phase approach outlined by Braun and Clarke, ^{44 , 50} to explore pre-service science teachers’ perceptions (RO3). This method was chosen for its flexibility in systematically identifying themes from qualitative data. The process involved (1) data familiarization (including transcription and repeated reading), (2) generating initial codes across the dataset, (3) searching for potential themes, (4) reviewing and refining those themes, (5) defining and naming the final themes, and (6) producing the final analysis. The analytical process was rigorous and iterative, occurring concurrently with data collection as described in Section 2.5. After an initial set of interviews was transcribed, the three researchers independently generated initial codes. The team then convened to compare these early codes, discuss discrepancies, and develop a preliminary codebook. This codebook was applied to the initial transcripts and then progressively refined as more interviews were conducted and analysed. This iterative cycle of independent coding, team discussion, and codebook refinement continued throughout the data collection period, ensuring that the final themes were robust interpretations grounded in the participants’ collective experiences. Further details on data organization and the consensus-building procedure are provided in Supplementary Information (Section S7).

3.1 Quantitative results

Spearman’s rank correlation coefficients that explain the relationships between the main variables in the study are shown in Table 1. For the first objective, the correlation between the ChatGPT assessment scores and final achievement scores in the stoichiometry unit was analyzed via Spearman’s correlation. A strong positive correlation (r_s = 0.670, p < 0.001) was found, indicating that students with higher ChatGPT assessment scores – reflecting a greater formative understanding – tended to perform better on the final achievement test.

Regarding the second research objective, which explored the relationship between assessment outcomes from ChatGPT and those from human instructors, the analysis found that the ChatGPT assessment scores demonstrated a very strong, statistically significant positive correlation with the instructor assessment scores (r_s = 0.882, p < 0.001) (see Table 1). This result indicates a high degree of agreement between the formative assessment of understanding conducted through ChatGPT and the assessment conducted by human instructors.

3.2 Qualitative results

Thematic analysis was conducted on the interview data, exploring pre-service science teachers’ perceptions regarding the use of ChatGPT as a formative assessment tool within the stoichiometry unit. The primary aim of this analysis was to provide in-depth explanations for the observed correlation between the ChatGPT assessment scores and the final achievement scores. This analysis revealed four key themes, as follows:

4.1 Explaining the relationship between ChatGPT assessment and achievement through pre-service science teacher perceptions

The strong positive correlation between the ChatGPT assessment scores and final achievement scores (r_s = 0.670) indicates that students who achieved higher assessment scores from ChatGPT during instruction also tended to score higher on the final achievement test. With Spearman’s coefficient of determination (r_s² = 0.45), approximately 45 % of the variance in final achievement scores was associated with ChatGPT score variance, signifying its role as an important indicator, although other factors account for ∼55 %. This strong positive correlation suggests ChatGPT demonstrates considerable capability in assessing student understanding in ways that align with final achievement outcomes. This finding is consistent with recent research by Chang and Chien, ⁵¹ who also found a positive relationship between scores from a Gen AI-powered quiz platform and post-learning achievement in a programming course.

While correlation does not imply causation, qualitative data from thematic analysis of pre-service science teacher interviews offered in-depth explanations for potential mechanisms underlying this quantitative relationship (addressing RO3), as detailed below.

Firstly, the capability of ChatGPT to provide immediate diagnosis and feedback (Theme 1) was a significant factor. Students reported that the rapid identification of their conceptual misunderstandings (S08, S02) was highly beneficial. This enabled them to confront their errors and begin the process of conceptual correction before a misunderstanding could become more entrenched. This process, where immediate feedback directly fuels conceptual change, not only aligns with the core principles of effective formative assessment ⁴⁵ but is also consistent with existing research on Gen AI’s capacity to deliver timely, adaptive feedback that can effectively diagnose specific student weaknesses. ^{52 , 53 , 54}

Secondly, ChatGPT’s influence in stimulating key processes of Self-Regulated Learning (SRL), specifically self-review and the adjustment of learning strategies (Theme 3), provides another explanatory mechanism. Assessment results from ChatGPT prompted self-reflection and self-evaluation. When students received unsatisfactory scores, they tended to monitor their understanding, re-analyze areas of difficulty, and begin the process of correcting misconceptions to deepen comprehension (S06, S07, S10) – core SRL components. ^{55 , 56} Moreover, a significant influence involved fostering strategy adaptation; students utilized ChatGPT’s feedback and scores to inform study planning and enhance review effectiveness (S03, S08), often focusing practice on personal weaknesses. Such adjustments potentially facilitated a shift from rote memorization towards genuine conceptual understanding, aligning with research linking adaptive strategy use post-assessment to enhanced comprehension. ^{57 , 58} Therefore, ChatGPT appeared to act as a technological scaffold supporting multiple facets of SRL: ^{37 , 59} students used its feedback for self-monitoring and evaluation, leading to strategic planning and behavioral adaptation. This AI-supported SRL process confirms Gen AI’s potential in fostering crucial learning skills ^{59 , 60 , 61} and serves as a key mechanism linking formative assessment to final learning achievement.

Crucially, the significance of this pre-service science teacher cohort is central to the study’s findings, particularly regarding the affective and motivational dimensions of their SRL. ⁶² The feedback from ChatGPT functioned as a powerful emotional driver precisely because the process was not merely about academic performance but was deeply intertwined with their emerging professional identities. This is powerfully articulated by student S08, who viewed unsatisfactory performance not as a simple academic setback, but as a direct challenge to their future role as a teacher:

“…if I can’t solve these problems, but I’m going to become a teacher, what on earth will I teach my students in the future? So, because of that, I just have to try harder; I have to practice more.”

This quote reveals a unique motivational mechanism where their professional identity fueled a commitment to self-improvement. ^{63 , 64} For this specific cohort, therefore, the entire formative assessment loop was contextualized by their developing identity, providing a compelling explanation for the study’s findings that may not manifest in other student populations.

The final point contributing to this explanation relates to students’ perception of the connection between ChatGPT assessment scores and their final achievement scores (Theme 4). Most students personally perceived this relationship, often viewing ChatGPT’s assessment as a preliminary indicator of their understanding and potential exam performance. Their concrete examples of this consistency provide experiential validation, lending weight and meaning to the statistical correlation, demonstrating it as an observable and meaningful phenomenon for learners. This perceived connection may then motivate students to engage seriously with ChatGPT’s formative assessment and diligently utilize its feedback (Theme 3) due to the potential link to exam scores. This aligns with findings by Dahri et al. ⁶⁵ and Amin et al. ⁶⁶ on trust and confidence in Gen AI accuracy influencing feedback use, and with Uppal and Hajian ⁶⁷ on perceived AI utility correlating with better outcomes. However, students were also cognizant of other factors influencing their exam scores, including personal effort and preparation (S04, S11), test-taking conditions like stress (S09, S10), and potentially the inherent limitations of the AI assessment itself, such as perceived inaccuracies or discrepancies in its assessment methods (as discussed in Theme 2). This awareness corresponds with the coefficient of determination (r_s² = 0.45), which indicated that ∼45 % of the variance in final achievement scores was associated with ChatGPT score, confirming other personal, contextual and tool-related factors contribute ∼55 %. Thus, while Theme 4’s learner perception may not be the primary driver, it likely plays an important supporting role by providing experiential validation, adding meaning to the assessment, and reinforcing motivation.

Synthesizing these mechanisms reveals the cohesive logic behind the observed correlation. The immediate, diagnostic feedback from ChatGPT (Theme 1) acted as a catalyst for an active cycle of SRL (Theme 3), amplified by the students’ perception of the value of this process for their final performance (Theme 4). Students who fully engaged in this motivation-enhanced feedback loop were empowered to build a more robust conceptual understanding – rather than ChatGPT being the direct origin of that understanding. This deeper understanding is the common underlying factor, meaning high ChatGPT scores and strong final test results should be viewed as parallel outcomes of the same successful learning process. Therefore, the ChatGPT assessment score functions as a powerful indicator of final achievement, not because it directly causes improved performance, but because it effectively captures the artifact of a successful, self-aware, and self-regulated learning process.

4.2 Agreement between ChatGPT and instructor assessments

Analysis of the ChatGPT assessment scores and the instructor assessment scores revealed a very high positive correlation (r_s = 0.882), indicating a very high level of agreement. However, interpreting this correlation requires considering fundamental differences in assessment processes: human instructors used an explicitly defined scoring rubric, whereas ChatGPT leveraged its LLM capabilities and a standardized prompt, without this specific rubric. This high agreement is therefore particularly remarkable. A possible factor contributing to this level of agreement may be the nature of the formative assessment questions and expected answers. The questions, while primarily in short-answer or well-defined answer formats, were designed to assess conceptual accuracy and response completeness. These are dimensions that both sophisticated LLMs and human instructors may be capable of assessing with considerable alignment. ^{29 , 68} This implies ChatGPT can capture an overall assessment of understanding largely consistent with human assessment. This aligns with Fernández et al., ⁶⁹ who found a positive rubric-free ChatGPT-human correlation for university entrance exams, and Morjaria et al. ²⁹ for medical short answers. Therefore, this study contributes additional supporting evidence that ChatGPT, even without explicit rubrics, demonstrates considerable potential for performing human-consistent assessments. This outcome also resonates with the broader educational technology trend of reducing instructor workload. ^{42 , 70 , 71}

It is noteworthy that this correlation (r_s = 0.882) was considerably higher than the correlation with final achievement scores (r_s = 0.670). This difference is likely because the former comparison involved both AI and instructors assessing the exact same set of formative, open-ended conceptual questions, enabling a direct comparison of assessment capabilities. In contrast, the latter comparison involved different instruments – the formative questions versus a summative, multiple-choice achievement test – which measure broader cumulative knowledge influenced by additional factors beyond formative assessments.

While the level of agreement between the two assessment formats was very high (r_s = 0.882), qualitative interview data provided valuable perspectives on students’ perceptions of ChatGPT’s capabilities, highlighting perceived limitations and accuracy concerns (Theme 2). Specifically, students noted issues such as excessive strictness with calculation details (e.g., decimals, S05), perceived overestimation of understanding (S02), failure to identify errors like missing units (S10), or ChatGPT utilizing different problem-solving methods (S07). Given the very high overall correlation, the impact of these reported limitations on understanding assessment scores may be limited. Nevertheless, these perceived shortcomings clearly reflect areas where AI systems could be further developed to achieve flexibility and contextual understanding in specialized domains like chemistry, approaching that of human instructors. ^{72 , 73}