AI-based causal evaluation of teacher’s opening lessons: a multidimensional study

2.1 Research design and procedure

The research workflow implemented in this study is illustrated in Figure 1. The process began with collecting video data from 200 classroom lesson recordings (observational design), where teachers performed the lesson introduction (opening phase). These videos were then transcribed automatically using an automatic speech recognition (ASR) system, and the transcripts were verified through a structured cleaning procedure (e.g., spot-checking and manual correction on a defined subset) to reduce errors from classroom noise and overlapping speech. Once transcripts were available, an AI model (Gemini) was used to score the lesson introduction based on a standardized rubric focusing on three teacher-actionable, video-observable features: attention, initial interaction, and media use. Human raters scored a subset of videos using the same rubric to benchmark the AI output and to report agreement indices (e.g., ICC/kappa) for transparency. The resulting scores were analysed descriptively and then modelled using causal-inference methods to estimate the likely contribution of specific opening-phase features to the rubric-based opening-phase quality score (the outcome in this study). The primary aim was to evaluate how elements such as attention-building activities, the clarity of objectives, the use of media, and initial interactions influence opening-phase lesson quality, providing actionable evidence for teacher development.

Figure 1:

Reaserch process flow.

The research methodology employs a systematic approach to evaluate teacher performance using teacher-readable, evidence-based indicators supported by AI. The process begins with automatic transcription of classroom video data and subsequent analysis through a Gemini-powered scoring procedure. This model uses a standardized rubric to assess attention, initial interaction, and media use during the lesson introduction phase, which are widely recognised as core elements shaping students’ readiness to learn. To improve reproducibility, the AI scoring procedure should be reported with model/version and access date, prompt template, and generation settings (e.g., temperature), and the prompt can be provided as Supplementary Material. To ensure reliability, the AI assessments are benchmarked against human evaluations conducted by trained raters, and interrater reliability should be reported for the human scoring and for AI–human agreement. ²⁰ After validation, the scored data undergo analysis using descriptive statistics and causal-inference methods such as average treatment effect (ATE), heterogeneous treatment effect (HTE), and mediation analysis. For teacher interpretability, these analyses are used to answer practical questions (e.g., “Which opening practices are most likely linked to higher-quality lesson introductions, and for which teachers?”), rather than only reporting technical parameters. Visualisations of the findings provide actionable insights for educators to improve teaching quality in the lesson introduction phase. This AI-driven framework offers a scalable and objective approach for evaluating teaching practices and supporting professional development, with the study outcome defined as the rubric-based opening-phase quality score (not student test scores).

Research has shown that attention-building activities and teacher-student interactions are crucial for enhancing student engagement and learning outcomes. Strategies like interactive questioning and clear learning objectives are associated with improvements focus and material retention, with teachers’ questions directly influencing student participation and responses. ²¹ When combined with proactive teacher-student interactions, these strategies promote cognitive engagement and improve social dynamics, fostering deeper understanding and critical thinking. ²² Furthermore, the integration of artificial intelligence (AI) in educational research has introduced scalable and objective methods for analyzing complex data. AI models using causal inference techniques, such as propensity score matching (PSM), allow for more accurate evaluation of teaching strategies by controlling for confounding variables. ²³ These methods enable researchers to isolate the estimated effects of instructional strategies on the study outcome, under explicit assumptions. ²⁴ The application of AI in causal inference frameworks improves the reliability and precision of educational research, ultimately supporting more effective teaching practices and interventions. ²⁵

2.2 Data and instrumentation

The data instrumentation for this study establishes a structured framework to evaluate teaching performance during lesson introductions using 200 video recordings produced by participants in Indonesia’s Teacher Professional Education (PPG) program, including both in-service (PPG Dalam Jabatan) and pre-service (PPG Prajabatan) teacher candidates undergoing the professional teacher certification process. The recordings captured participants’ teaching practice sessions (e.g., microteaching and/or practicum-based teaching), and the analysis focused specifically on the lesson introduction segment. Participant background characteristics (e.g., teaching experience for in-service candidates and prior practicum exposure for pre-service candidates) and practicum/classroom context (e.g., placement setting and instructional conditions) were summarised to support representativeness and potential bias assessment.

The videos were transcribed using an automatic speech recognition (ASR) system, followed by systematic cleaning and coding using a predefined codebook to ensure data accuracy. To strengthen transparency, transcription quality was audited through manual verification of a stratified subset, with corrections applied where necessary; additionally, intercoder agreement was calculated for the cleaning/coding process to document procedural reliability. Scoring was conducted using Gemini AI based on a rubric comprising three key indicators: attention, initial interaction, and media use. Finally, in line with the program’s formative assessment cycle and participants’ consent, the scoring outputs were synthesised into individualised feedback to support improvement in subsequent teaching sessions. Attention measures how effectively a teacher captures student focus through greetings and engaging techniques, with high scores awarded for methods that are highly relevant and spark curiosity, while lower scores indicate minimal or routine efforts. ^{26 , 27} Initial interaction assesses the teacher’s ability to promote student participation through questioning or warm-up activities, ranging from dynamic and inclusive engagement to a complete absence of interaction. ^{28 , 29} Media use evaluates the relevance and creativity of visual or audio aids used during the lesson opening, with top scores given when media significantly enhances understanding and engagement, and lower scores when media is absent or ineffective. ^{30 , 31}

The integration of Gemini AI into scoring for educational assessments enhances consistency and objectivity in evaluating teaching behaviors, enabling reliable, replicable, and large-scale analyses while reducing human bias through nuanced, rubric-based evaluations. ³² Beyond automation, AI supports teachers in lesson planning, execution, and appraisal, ³³ fostering personalized learning and targeted professional development. Its effectiveness in video-based studies allows multi-dimensional assessment of pedagogical strategies, deepening insights into teaching quality. However, successful implementation depends on teacher trust, transparency, and willingness to adopt AI tools, influenced by prior experiences and perceived usability. ³⁴ Ethical integration, as emphasized by Lee et al. and Liu, is critical to ensure AI promotes equity, avoids bias, and supports reflective teaching practices. ³⁵ This synergy between AI and structured rubrics advances evidence-based evaluation frameworks that strengthen both teacher development and the quality of lesson introductions.

2.3 Descriptive analysis

Descriptive analysis of teacher performance during lesson opening is summarized in the Descriptive Statistics of Opening Lessons Scores, presented in Table 1. This table highlights three main indicators: attention, interaction, and media use. The data shows that most teachers achieved moderate to high scores across these indicators, with Interaction showing the highest mean score of 3.44 and median of 4, reflecting strong and consistent engagement between teachers and students. ³⁶ Otherwise, media use recorded the lowest mean score of 3.04 and the highest variability, indicating that media integration was inconsistent among teachers. ^{37 , 38} Negative skewness values across indicators, especially −1.36 for interaction, suggest that the majority of teachers perform well in fostering interaction during lesson opening, supported by kurtosis values that show high concentration of scores. ³⁹ An interquartile range of 1 across indicators indicates moderate score stability among teachers, while the percentages of teachers scoring 3 or above 90 % for interaction, 86.5 % for attention, and 76.5 % for media underscore general strengths in interpersonal skills, but reveal opportunities for improvement in media use. ^{40 , 41} Importantly, the “Percent ≥ 3” column applies to indicator scores (1–4 scale) and is not directly comparable to the total score; therefore, we report total score descriptively without using the same threshold to avoid misinterpretation. ⁴²

2.4 Causal analysis framework

This study utilised causal analysis to evaluate the impact of key instructional features – attention, initial interaction, and media use – on opening-phase quality scores using the T-Learner model and propensity score matching (PSM) to calculate average treatment effects (ATE). ^{43 , 44} In addition, heterogeneous treatment effects (HTE) were examined across quartiles of teacher performance to identify how different groups of teachers respond to instructional strategies. ^{45 , 46} Mediation analyses were conducted to investigate whether the effect of media use on opening-phase quality was mediated by teacher–student interactions, highlighting the role of engagement in improving lesson introduction practices. ^{47 , 48} Data processing used Gemini AI within Google Colab, while causal relationships were stringently analyzed using Causal AI, enabling an objective and measurable assessment framework to improve educational practices. To strengthen methodological transparency, we explicitly state the key assumptions (e.g., no unmeasured confounding/ignorability given observed covariates, positivity/overlap, and stable unit treatment value) and report diagnostics (e.g., covariate balance after matching, overlap checks, and sensitivity analysis) to support trust in the estimated effects.

3.1 Sensitivity of causal estimates

The Tornado Plot (Figure 2) visualizes the sensitivity of the average treatment effect (ATE) to unmeasured confounding. It illustrates how the estimated ATE changes in response to varying levels of assumed unobserved confounder bias, represented on the vertical axis. The plot displays two sets of bars: one set for the baseline (no-bias) ATE estimate (blue bars) and the other for the bias-adjusted ATE under each assumed bias level (orange bars). This comparison helps readers judge how robust the estimated effects are to plausible violations of the “no unmeasured confounding” assumption in an observational design, rather than implying that unmeasured confounding is fully eliminated.

Figure 2:

Tornado Plot – sensitivity of ATE to unobserved cofounding.

The Tornado Plot in Figure 2 illustrates the sensitivity of the average treatment effect (ATE) to varying levels of unmeasured confounding bias, ranging from 2 % to 25 %. At lower bias levels (2–5%), ATE estimates remain relatively stable, suggesting that the main conclusions would not change under small-to-moderate unmeasured confounding. ⁴⁹ Around the 10 % level, the plot indicates a practical “tipping point” where the magnitude of the ATE begins to change more noticeably, signalling that stronger unmeasured confounding could meaningfully alter the interpretation. ⁵⁰ At higher assumed bias (e.g., above 10 %), the ATE becomes increasingly sensitive, indicating that the causal estimates depend more strongly on the validity of the ignorability assumption and the adequacy of measured covariates. This underscores the importance of reporting sensitivity analysis in observational causal studies and considering robustness-oriented approaches (e.g., alternative estimators such as TMLE or doubly robust methods) as complementary checks. ⁵¹ Overall, Figure 2 supports the interpretation that the estimated ATE is reasonably robust to modest unobserved confounding, while also transparently acknowledging that substantial unmeasured confounding could distort the estimated effects, reinforcing the need for careful study design, covariate measurement, and diagnostic reporting.

3.2 Multidimensional causal effects

The multidimensional causal effects analysis using 2D causal effect landscapes, 3D partial dependence plots, and a ternary plot provides a comprehensive overview of how the combined levels of attention, media, and interaction relate to opening-phase quality (rubric-based opening-lesson score). The multidimensional analysis suggests that opening-phase quality is highest when attention, interaction, and media use are all strong and reasonably balanced, rather than when one element is high in isolation. The 2D landscapes indicate strong joint patterns for attention–interaction, the partial dependence plots show that the association of media use with higher scores is strongest when attention is also high, and the ternary plot illustrates that the most favourable region occurs near balanced combinations of the three factors. Together, these visualisations provide an interpretable, teacher-facing summary of how opening-phase practices may work in combination to support higher-quality lesson introductions.

The causal effect landscapes in Figure 3a, b, and c show that estimated effects on opening-phase quality (ATE-based patterns) are highest when key opening-phase practices – attention, interaction, and media use – are jointly strong. Figure 3a shows a strong co-occurrence/synergistic pattern between attention and interaction, which emphasises the importance of engaging students cognitively and socially from the start. ^{52 , 53} Figure 3b shows that media use is more strongly associated with higher opening-phase quality when paired with high attention, highlighting the complementary role of media in strengthening student focus during the lesson introduction. ^{54 , 55} Finally, Figure 3c indicates that the interaction–media combination yields the largest estimated improvement pattern, suggesting that interactive opening strategies may be most effective when supported by relevant media. ^{56 , 57} Collectively, these findings underscore the value of combining attention, interaction, and media use to optimise lesson introductions, while recognising that these are estimated associations under the study’s causal-inference assumptions.

Figure 3:

Causal effect landscape. (a) Attention versus interaction, (b) attention versus media, (c) interaction versus media.

The findings from Figure 4a, b, and c further highlight the combined relationships among attention, interaction, and media use with opening-phase quality, visualised via partial dependence patterns. Figure 4a shows that the highest predicted opening-phase quality occurs when both attention and media use are high, consistent with research suggesting that media can enhance learning environments by capturing students’ attention when combined with focused engagement. ^{58 , 59} Figure 4b emphasises the importance of teacher–student interaction combined with attention, which aligns with evidence that such interaction promotes cognitive and social engagement during instruction. ^{60 , 61} Figure 4c reinforces this by indicating that higher predicted scores are achieved when media use is combined with strong teacher–student interaction, consistent with findings that multimedia can enhance motivation and lesson clarity through dynamic interaction. ^{58 , 62} Overall, these results suggest that balanced integration of attention, interaction, and media use is important for strengthening the quality of lesson introductions, rather than treating any single element as sufficient on its own.

Figure 4:

Partial dependence plot. (a) Attention versus media, (b) interaction versus attention, (c) media versus interaction.

Figure 5’s ternary plot illustrates the combined influence of attention, interaction, and media use on opening-phase quality scores (opening-lesson rubric totals). The triangular graph shows that the highest scores occur when these three factors are simultaneously high and relatively balanced, with regions near high attention and interaction corresponding to better opening-phase quality. The plot also indicates that emphasising one factor excessively (e.g., media use without adequate attention or interaction) is less likely to maximise effectiveness. This highlights the practical importance of integrating all three elements coherently when designing lesson introductions.

Figure 5:

Ternary Plot.

Figure 5 highlights the importance of balancing attention, interaction, and media use in lesson introductions, showing that stronger outcomes occur when all three are well integrated. The analysis suggests that high levels of attention and interaction are consistently linked to higher opening-phase quality, with media use playing a supportive role that can strengthen their effects when applied purposefully. In practical terms, effective openings begin by capturing student focus and encouraging meaningful engagement, while media resources are used to reinforce clarity and interest in the learning objectives. Prior research by Greve et al. ⁵² and Mavilidi and Vazou ⁵⁶ similarly shows the strong synergy between attention and interaction in improving teaching effectiveness, while Ref. ⁶³ emphasize the supportive role of media in enhancing learning environments.

3.3 Heterogeneous effects by teacher performance

The analysis of heterogeneous effects by teacher performance, shown through the Raincloud and Interaction Plots, suggests that lower-performing teachers (Q1) show larger estimated gains in opening-phase quality compared to higher-performing teachers (Q4). This aligns with research indicating that targeted support – such as structured guidance and clear learning goals – can be particularly beneficial for less experienced or less confident educators. ^{64 , 65} Lower-performing teachers, often lacking a strong pedagogical foundation, are more receptive to professional development initiatives that offer clear expectations. Studies also highlight that a supportive school climate, strong principal-teacher relationships, and a conducive educational environment significantly enhance teacher performance. ⁶⁶ Overall, these patterns support the practical value of differentiated professional development that is matched to teachers’ current performance level, with the outcome here defined as the rubric-based opening-phase quality score (not student test achievement).

The Raincloud Plot in Figure 6 illustrates how the average treatment effect (ATE) varies across teacher performance quartiles, from lowest (Q1) to highest (Q4). It shows that lower-performing teachers (Q1) experience a wider range of estimated ATE values, with many benefiting more strongly from improved lesson introduction practices. In contrast, higher-performing teachers (Q4) show less variation and more consistent but smaller estimated gains, suggesting a potential ceiling effect because many already implement effective opening routines.

Figure 6:

Raincloud Plot.

The Raincloud Plot suggests that teachers in the lower quartiles (Q1 and Q2) experience greater positive effects from changes in lesson introduction practices, indicating they may benefit more from improvements in opening-phase techniques. Meanwhile, teachers in higher quartiles (Q3 and Q4) show more stable but smaller changes, suggesting that their baseline practices are already relatively strong. This pattern supports targeted support for Q1–Q2 teachers, while Q3–Q4 teachers may benefit more from refinement-focused feedback (e.g., higher-level questioning, efficiency, and consistency).

The Interaction Plot shows how learning objective clarity affects teachers differently across performance groups. Lower-performing teachers (Q1) show larger improvements in their opening-phase quality scores as learning objectives become clearer, indicating that explicit learning goals may be a high-leverage practice for teachers who are still developing opening routines. Higher-performing teachers (Q4) show smaller changes, likely because they already communicate objectives effectively and therefore gain less from additional increases in clarity.

The plot in Figure 7 shows that teachers in the middle quartiles (Q2 and Q3) experience moderate improvements from clearer learning objectives, indicating that the impact varies by performance level. While learning objectives appear beneficial across groups, their estimated effect is strongest for lower-performing teachers (Q1), highlighting the value of targeted support for teachers who need it most. The findings from the heterogeneous effects analysis further indicate that Q1 teachers show the most substantial improvement when learning objectives are clearly communicated, suggesting that objective clarity is a practical, teacher-actionable focus for professional development. ⁶⁷ In contrast, higher-performing teachers (Q3 and Q4) show little change, suggesting they already effectively plan and deliver lessons. ⁶⁸ This underscores the importance of targeted professional development focusing on clear learning goals for lower-performing teachers. Studies confirm that academic supervision and goal clarity are vital for improving teaching quality, especially for those in lower performance. ^{69 , 70} Accordingly, interventions that strengthen objective clarity for Q1 teachers are likely to improve the quality of lesson introductions (the outcome in this study), while broader claims about student learning should be treated as theoretical implications rather than directly measured effects in this dataset.

Figure 7:

Interaction Plot.

4.1 Evaluating the causal impact of key features

Evaluation of the estimated causal impact of key features in the lesson introduction highlighted that capturing students’ attention had the strongest estimated effect on opening-phase quality scores (rubric-based opening-lesson score), reflected by the highest average treatment effect (ATE). This supports educational theories emphasising that clarity of purpose at the beginning of a lesson can improve students’ mental structuring of the material, promoting engagement and better understanding. ^{71 , 72} Moreover, the combined pattern of interaction and media use suggests that interaction can be strengthened when supported by appropriate media, as illustrated in the ternary plot (Figure 5), indicating that media may function as a supportive resource that amplifies attention and interaction during the lesson introduction. ⁷³ The robustness checks in the sensitivity analysis (Figure 2) suggest that the main conclusions are reasonably stable under modest levels of unobserved bias, while also highlighting that stronger unmeasured confounding could change effect magnitudes – an expected limitation in observational causal inference. ⁷⁴ Collectively, these findings underscore the potential of AI-supported evaluation systems to move beyond superficial performance impressions by providing transparent, rubric-based evidence about which opening-phase practices are most strongly linked to higher-quality lesson introductions. ^{75 , 76}

4.2 Heterogeneity and personalization in teacher evaluation

The purpose of this analysis is to explain variation in impact across teacher groups using heterogeneous treatment effects (HTE). Figure 6 suggests that teachers with lower baseline scores experience the greatest improvement in opening-phase quality when learning objectives are communicated more clearly, highlighting the potential value of personalised evaluation and training systems tailored to teachers’ starting points. This approach challenges a “one-size-fits-all” model and supports coaching practices focused on high-leverage, teacher-actionable moves in the lesson introduction. The interaction plot in Figure 7 illustrates how lesson introduction strategies can be optimised based on teacher profiles, with objective clarity appearing particularly beneficial for lower-performing teachers. These findings emphasise the importance of incorporating HTE analysis into teacher quality improvement policies, as it helps identify who benefits most from particular supports. ⁷⁷ It also aligns with trends toward individualised professional development and feedback-oriented coaching, ^{78 , 79 , 80 , 81} supporting effective and equitable improvements in teaching practice. ⁸²

4.3 Strengths, weaknesses, and limitations of the multidimensional approach

One of the major strengths of the multidimensional approach used in this study is its ability to assess the integrated relationships among attention, teacher–student interaction, and media use in the lesson introduction phase. This approach supports a more nuanced interpretation of how these factors may work together to strengthen opening-phase quality, rather than treating each practice in isolation. The use of multiple visual representations, including the ternary plot (Figure 5), causal landscapes (Figure 3), and partial dependence plots (Figure 4a–c), provides complementary perspectives by highlighting both individual patterns and combined effects, offering richer interpretation than a single summary estimate. For teacher readers, these visualisations can function as practical “maps” that show which combinations of opening practices are most consistently associated with stronger lesson introductions. ^{69 , 70}

However, one limitation of this approach is the reliance on classroom video data and AI-supported scoring, which may not fully capture the complexity of in-class dynamics. Although human raters were involved to validate the AI scoring, some subtleties of teacher–student interaction (e.g., tone, classroom climate, and non-verbal cues) may be partially missed by automated pipelines. ^{78 , 79 , 80 , 81} Moreover, because the study is observational, the multidimensional analysis may not account for all potential confounders (e.g., class composition, school policies, topic difficulty, or prior teacher training), which can limit generalisability and requires careful interpretation of causal claims. ^{75 , 76} Despite these limitations, the approach provides a structured framework for evaluating lesson introductions by combining rubric-based scoring, causal-inference modelling, and interpretable visual summaries. Importantly, the study outcome is the rubric-based opening-phase quality score; therefore, references to “student learning outcomes” should be treated as theoretical implications supported by prior literature rather than as directly measured effects in this dataset. This work provides a foundation for future research that strengthens external validity (e.g., broader sampling and context variables), improves reproducibility (e.g., published prompts and scoring protocols), and combines AI-supported evaluation with teacher coaching cycles to support sustainable professional growth.