Teachers’ Perceptions of a Chatbot’s Role in School-based Professional Learning

2.1 School-based Activities with ICT-Enhanced Substantial Learning Environments

Substantial learning environments provide a wide array of rich mathematical activities not only for students but also for pre-service and in-service teachers, offering opportunities for exploration at an advanced level. This section aims to offer an overview of substantial learning environments and the activities they entail, both in general and with specific emphasis on ICT integration. Subsequently, it discusses the potential challenges teachers may encounter when engaging with substantial learning environments.

The concept of substantial learning environments, grounded in Wittmann’s (1984, 2021) design science approach to mathematics education, embraces a constructivist perspective. These environments, designed to accommodate diverse student needs, facilitate engagement in mathematical activities and subject-specific communication. Structured around a central theme, substantial learning environments consist of thoughtfully designed tasks that can be adapted to various classroom settings.

For TPL, these environments provide a broad spectrum of activities. Teachers can analyse the epistemological structure within these environments, reflect on didactic principles (such as goal-specific task potential or characteristics of the utilised ICT), tailor a substantial learning environment to a potential classroom scenario, and ultimately test it in practice. Additionally, teachers can partake in activities focused on analysing traditional tasks to explore how ICT can enhance their development (Hammer & Ufer, 2023; Nührenbörger et al., 2016; Wollring, 2004).

Certain activities have proven to be beneficial in the context of engaging with ICT. These include independent experimentation with ICT, experiencing competence, and collaborating with more experienced peers (Ottenbreit-Leftwich et al., 2018). These characteristics are relevant for individual school-based activities as well as for face-to-face teacher training sessions.

School-based try-outs play a crucial role in enhancing teacher professional competence. In general, these opportunities to learn (OTL) enhance TPL by providing chances for self-directed, problem-oriented work and fostering a sense of self-efficacy, while also integrating everyday practice (Barzel & Biehler, 2020; Beyer, 2022a,b). However, they have inherent challenges that need to be addressed (Lipowsky & Rzejak, 2015), such as the lack of immediate instructions or managing simultaneous tasks (Beyer, 2022b). In addition, Rich, Yadav, and Fessler (2024) highlight the diversity in teachers’ perceptions of teaching materials, prioritising distinct evaluation criteria, adaptations of substantial learning environments, and implementations in the classroom. It is important to note that not all approaches have demonstrated effectiveness in fully realising the potential of ICT-enhanced substantial learning environments.

2.2 Chatbots for Supporting Teachers School-based Professional Learning

Beyer (2022a,b) has explored both face-to-face and traditional e-learning approaches to supporting TPL activities. However, there has been a shortcoming in supporting situated learning, particularly in school-based activities. It is impractical for teacher training facilitators to directly accompany many participants during on-site preparation and teaching activities due to limited resources. Additionally, existing e-learning approaches lacked a strong link to classroom activities and placed too much emphasis on formalised contexts. As a result, the risk of isolated information that is not effectively utilised or accessed in a timely manner due to resource constraints is high.

This underlines the importance of a different perspective on the role of ICT in teacher education. Aubusson, Schuck, and Burden (2009) suggested that mobile learning with ICT is well-suited to the daily work of teachers. Mobile learning enables flexible learning opportunities at various times and places, affecting positively both formal and informal educational settings.

Due to technological advances, learning activities with chatbots have gained more attention in this context. Messenger apps are commonly used by a majority of individuals on a regular basis. As such, one has assumed that chatbots are promising for educational purposes as they do not require much effort to get used to (Hobert & Meyer von Wolff, 2019). Wollny et al. (2021) defined a chatbot as follows:

Chatbots are digital systems that can be interacted with entirely through natural language via text or voice interfaces. They are intended to automate conversations by simulating a human conversation partner and can be integrated into software, such as online platforms, digital assistants, or be interfaced through messaging services. (p. 2)

Based on findings from multiple systematic reviews spanning the period from 1999 to 2021, the primary application fields for educational chatbots were identified as learning languages and programming, with mathematics playing a significantly minor role (Hwang & Chang, 2023; Kuhail, Alturki, Alramlawi, & Alhejori, 2023; Wollny et al., 2021). Typical applications related to mathematics include micro-learning of mathematical content (e.g. Yin, Goh, Yang, & Xiaobin, 2021) and tutoring systems that teach, for example, natural deduction to undergraduates (e.g. Miwa, Terai, Kanzaki, & Nakaike, 2014).

Chiu, Xia, Zhou, Chai, and Cheng (2023) demonstrated that most similar applications predominantly target students in schools and universities. Their comprehensive review encompassed 92 studies, with only one emphasising in-service teacher training as a primary focus. An illustration of pre-service teacher education includes the application of responsive teaching in mathematics with teachable agents (Lee & Yeo, 2022). Overall, teacher education is found to play a minor role.

2.3 Technology Acceptance Model (TAM)

The concept of TA is highly relevant in the context described, as the adoption and voluntary, continuous use of ICT are crucial prerequisites for successful learning and teaching (Nistor, 2018). Nistor (2018) distinguished between the cognitive, emotional, and behavioural component of TA. The cognitive component involves making a usage decision based on a thorough cost–benefit analysis (Nistor, 2018). The emotional component stems from the individual’s feelings or emotions towards the object of interest and aims to either maintain or increase positively experienced emotions or reduce or avoid negatively experienced emotions (Nistor, 2018). The behavioural component is influenced by one’s previous behaviour towards the same object (Nistor, 2018). Numerous acceptance theories and models provided explicit frameworks for understanding and exploring these respective components. The primary focus of this paper is the cognitive acceptance component.

In this context, the TAM (Davis, 1989) holds significant importance due to its empirically proven ability to predict human behaviour regarding the adoption or rejection of ICT (Granić & Marangunić, 2019; Sánchez-Prieto, Olmos-Migueláñez, & García-Peñalvo, 2017; Scherer, Siddiq, & Tondeur, 2019). Figure 1 illustrates the original TAM, which posits a strong relationship between Intention to Use and Actual Use (as indicated on the right-hand side of the illustration). Intention to Use can be predicted by both Perceived Ease of Use and Perceived Usefulness, which are central predictors in the model (Davis, 1989; Sánchez-Prieto et al., 2017). Additionally, Perceived Ease of Use influences Perceived Usefulness, but not vice versa (as depicted in the centre of the illustration).

Figure 1

TAM based on Davis (1989). X ₁ to X _n serve as placeholders for different external factors influencing user motivation (own illustration).

These use-related beliefs are essential prerequisites for implementation, as they act as filters that shape individuals’ interpretation of new information and experiences. Teachers may exhibit hesitancy in adopting chatbots if they do not perceive them as useful or find them challenging to use. Studies on usage decisions, typically conducted at the classroom level, often explored the influence of pedagogical and epistemological beliefs held by teachers (Ottenbreit-Leftwich et al., 2018; Thurm & Barzel, 2020).

Within the scope of TAM research, it is noteworthy that, in addition to the original model and its core variables (depicted in the centre and right-hand side of the illustration in Figure 1), significant attention has been directed towards the study of external variables (located on the left-hand side of the illustration), such as self-efficacy, subjective norm, computer self-efficacy, enjoyment, and perceived content quality (Granić & Marangunić, 2019). Extensive empirical research has confirmed the impact and relevance of these external variables in their particular contexts (Scherer et al., 2019). This flexibility of application is another strength of the model, in addition to its parsimony (Granić & Marangunić, 2019).

For the education sector, Granić and Marangunić (2019) have highlighted that the focus of work on TAM between 2003 and 2018 has been on e-learning. These included learning management systems, video conferencing systems, and video platforms. In addition, mobile, technology-enhanced, learning contexts have been increasingly studied. In the context of mobile learning, further external variables have been identified, such as the quality of learning content, interactivity, user interface design, accessibility, and responsiveness. The results also showed that there is a correlation between previous experience using e-learning and current intention to use mobile learning.

However, the majority of these findings are based on samples of university students (83%). Only 6% of studies referred to teachers or lecturers, and then specifically to their role as teachers rather than to their role as learners (Granić & Marangunić, 2019). More recent research on TAM in (mathematics) teacher education has also focused heavily on pre-service teachers as future teachers, the influence of pedagogical and epistemological beliefs (e.g. Gurer & Akkaya, 2022), and the changes brought forth by the COVID19 pandemic (e.g. Wohlfart, Trumler, & Wagner, 2021). In addition, newer technologies have been investigated, such as the acceptability of learning analytics (e.g. Mavroudi, Papadakis, & Ioannou, 2021) or educational robotics (e.g. Alqahtani, Hall, Leventhal, & Argila, 2022). These more recent works also have in common that they confirmed TAM in its basic assumptions and flexibility.

Despite the considerable advancements in TAM research among (mathematics) teachers, there are notable gaps that require attention. Particularly, there is a lack of studies specifically investigating the utilisation of specific (mobile) technologies (Islamoglu, Kabakci Yurdakul, & Ursavas, 2021; Scherer et al., 2019). Consequently, the findings from such studies have had limited capacity to uncover the essential factors and reasons behind the acceptance or rejection of specific approaches to mobile learning (Islamoglu et al., 2021). Furthermore, there is a scarcity of research that examines TPL contexts (Sánchez-Prieto et al., 2017). Considering these limitations in the empirical foundation, it is crucial to approach the application of extended models with external variables cautiously, especially outside the contexts in which they have been validated (Chocarro, Cortiñas, & Marcos-Matás, 2023; Hansen-Casteel, 2020; Islamoglu et al., 2021; Sánchez-Prieto et al., 2017; Turner, Kitchenham, Brereton, Charters, & Budgen, 2010).

4.1 Design Research Process

In this study, a teacher training course serves as a crucial context for understanding how teachers navigate analogue and ICT-enhanced substantial learning environments, with a particular focus on the challenges and adaptations observed during school-based activities.

The course that focuses on addressing challenges in analogue geometry learning, specifically tasks like tiling a 6 × 10 rectangle with 12 pentominoes.^[1] The intricacies involve a high cognitive load due to numerous possible solutions and unmanageable sequences of actions. To alleviate these challenges, ICT, such as a pentomino app, has been suggested to aid problem reduction, provide step-by-step guidance, and enable progress monitoring alongside traditional materials.

The course structure involves four face-to-face sessions and three school-based phases, exploring both analogue and ICT-enhanced substantial learning environments. Teachers engage in activities that investigate the requirements and potential of these substantial learning environments (Section 2.1). They are encouraged to adapt and implement the pentomino substantial learning environment according to their classroom after two face-to-face sessions on teaching geometry with substantial learning environments (Figure 2) (Beyer, 2022b).

Figure 2

School-based activity between two face-to-face sessions (SLE – substantial learning environments) (own illustration).

The preliminary research aimed to identify key events and challenges in mathematics teachers’ practice during school-based activities as the basis for the chatbot development (Beyer, 2022a,b). The research revealed that teachers prioritise short-term goals related to materials and ICT, focusing on task completion over material choice or implementation analysis. Adaptations primarily aim to facilitate student task completion and engagement with ICT, enhancing their positive perception of the substantial learning environments.

The planning and preparation phase, recognised as a challenging aspect for teacher training facilitators employing traditional support methods, has been identified as a pivotal point in the development of the chatbot. The design process unfolded during design thinking workshops involving pre- and in-service teachers, teacher educators, teacher training facilitators, ICT professionals, and other researchers. These workshops were conducted under the supervision of an expert group, comprising an interdisciplinary team of researchers responsible for reviewing all design solutions. The subsequent section introduces the chatbot version used in the current study. For an in-depth exploration of the design process, refer to Beyer (2022b).

4.2 Design of the Chatbot

The description of the system characteristics is based on the six parameters of Adamopoulou and Moussiades (2020): knowledge domain, service provided, goals, input processing and response generation, human-aided, and development platform.

Justus is a web-based system that processes natural language input and aligns it with predefined intentions and contexts. It offers two approaches for utilising informative and task-based content: user-driven and chatbot-driven. The user-driven approach allows free interaction using natural language input, while the chatbot-driven approach follows a predefined sequence using conversational prompts (e.g. buttons with quick selection options). Response generation relies on a retrieval-based model that operates within a closed knowledge domain and provides pre-prepared responses that can be enriched with conversational details, such as the name or other entities provided by the user (see Table 1 for an example of the introductory dialogue). This tailored model supports teachers in planning while avoiding generative AI-related issues (e.g. so-called hallucinations). The chatbot does not have integrated human assistance, resulting in the teacher training facilitator being unable to provide help if the chatbot is incapable of answering the given question. The design recognises the importance of diverse media formats and reducing cognitive load for teachers during planning (Curum & Khedo, 2021).

The features of the chatbot presented in Table 1 demonstrate features that have the potential to alleviate the complexities associated with planning processes of the school-based activities. It provides mathematics teachers with a structured approach to facilitate the application of previously acquired knowledge in a practical manner and promote independent exploration. Justus provides three key context-related features (Introductory Dialogue, Planning Tool, and Recommended Educational Materials) and an overall QnA-feature. Table 1 depicts a linear use of the chatbot’s key features, although a non-linear one is also plausible.

In addition to the key features, the use of the chatbot can be complemented by using the QnA-feature to get answers to questions about planning and adapting substantial learning environments, principles of effective mathematics teaching, organising school-based activities, or teaching geometry. Upon the chatbot recognising a predetermined intent, users shall receive a pre-prepared response.

5.1 Data Collection

Prior to the commencement of data collection, the chatbot was tested by the participants. Following an introduction to its functional scope, the participants engaged in a scenario that aims to immerse participants in a potential planning situation by utilising the context and content described earlier (Nieveen & Folmer, 2013). As the participants possess general knowledge of substantial learning environments, including pentominoes, the scenario inputs prompted a focus on interacting with all of the chatbots’ features. Table 1 demonstrates the applied linear using process to get as much contact with the features as possible. This included welcoming the chatbot and going through the Introductory Dialogue, showing interest in the chatbot’s personal background, seeking assistance from the chatbot in planning a school-based activity (Planning Tool), requesting suggestions from the chatbot for a suitable substantial learning environment and exploring the provided materials (Recommended Educational Materials), and finally, completing the planning process and concluding the conversation. The chosen segment of the study emphasised the role of the chatbot during the preparation and planning phase, while other tools or resources were not considered in this specific research. Due to the pandemic, the interviewer connected with participants via video conference, observed the process, and remained available for any questions.

Subsequent to the test period, focused interviews were conducted with the participants. This form of interview focuses on a specific object of study – in this case, the chatbot – and seeks to capture respondents’ subjective views on the object of study as openly and non-directively as possible (Hopf, 2004). The core questions of the interview guide were derived from the object and centred around the chatbot’s characteristics. The following questions were included:

• You went through the Introductory Dialogue at the beginning. How did you perceive it?

• How did you perceive the Recommended Educational Materials feature?

• How did you perceive the Planning Tool?

• In which situations would you use the chatbot if you were faced with a school-based activity?

• What did you think of the chatbot’s communication style and personality when you used it?

Based on observations during the testing period and during the interviews, the core questions were not necessarily asked in order, and situational questions were added to support the interviewees’ self-exploration (Hopf, 2004). Examples of situational questions included:

• Would you like to be able to use the chatbot for your own teacher training course?

• This is a very positive assessment. Are there any points that are more negative or where you feel something is missing?

5.2 Participants

Participants were selected through contrastive sampling (Kergel, 2018), ensuring diverse characteristics in terms of age, teaching experience, and TPL experience. Participants, with a minimum of 12 months of teaching experience, taught mathematics classes from grades 1 to 6.^[2] The sample included eleven participants aged 27–42, with professional experience ranging from one to 15 years. Eight studied mathematics for primary school, two for secondary school I/II, and one was an out-of-field teacher. Participants had varied TPL experiences (Min = 0 h; Max = over 35 h) and varied teacher training facilitator experiences (Min = 0 h; Max = over 35 h) in the last 2 years, offering diverse perspectives on the chatbot evaluation. Recruitment sources included preliminary research participants, participants from nationally offered teacher training courses, and a corresponding pool of teacher training facilitators.

5.3 Data Analysis

The interviews, ranging from 20 to 60 min, were audio recorded, transcribed, and anonymised. The transcriptions underwent content-structuring qualitative content analysis (Kuckartz, 2019; Schreier, 2012). To assess the acceptance of the chatbot in a school-based activity by teachers and teacher training facilitators (RQ1), deductive categories were derived from the core elements of the TAM: Perceived Usefulness, Perceived Ease of Use, and Intention to Use.

For RQ2, inductive categories were derived. Due to the extensive range of coding categories found in the literature analysis, it was not feasible to use a deductive approach for forming categories of external variables. Moreover, caution should be exercised when applying external variables that were validated outside the present context (Section 2.3). Based on the analysis of the interview transcripts, the following categories were included: Teacher Heterogeneity, Perceived Quality, Chatbot Characteristics, and Mathematics (Teaching-) Related Attitudes and Beliefs. Further Development was included as a category to address the design component of the design research project (RQ3) (Table 2).

A consensus coding approach (Hemphill & Richards, 2018) was employed to ensure that all relevant passages in the material were identified and therefore all variables were determined. The respective versions of the coding frame were independently applied^[3] to the dataset by two researchers using MaxQDA software (Hemphill & Richards, 2018; Schreier, 2012). Through an iterative coding process, we refined the categories and descriptions. The research team resolved divergent findings through extensive coding discussions after each iteration.

All interviews were conducted in German and sample passages for this article were translated into English by the first author. To ensure the quality of the translation, the authors critically reviewed the translations together to guarantee that the meaning of the participants’ original statements was maintained.

6.1 User Motivation

6.2 External Variables

6.3 Further Development

This category summarises participants’ comments on the further development of the chatbot’s introductory dialogue, material display, and planning tool.

On a general level, there were suggestions for word changes in terms of teacher talk, reduction/splitting of content for a more streamlined layout, and more guidance on how to use the chatbot.

In particular, in the introductory dialogue, i.e. the creation of the profile, five participants (B3, B6, B9, B11, B7) would like to be able to save information about their mathematics classes. This would allow them to retrieve it more quickly when they needed it again. They also hoped that a future version of the chatbot would be able to use this information to personalise the adaptation instructions for the subtasks of the learning environment. In addition, as mentioned in Section 6.2.2, two participants suggested asking for general professional experience or specific self-reporting of competences related to the mathematical topic. This should personalise the advice.

In the context of material display, as already mentioned in Section 6.2.1, two teachers (B1, B11) would like the chatbot not only to present the subtasks but also to provide the solutions. In addition, two other teachers (B3, B4) would like the chatbot to give them prompts on how to use the resources and subtasks when presenting them, e.g. to encourage exploration and preparation:

[…] Maybe the chatbot can also point out: “‘I’m going to make you some suggestions of resources and you take a look at everything first” [B3; para. 23],

It would be important for the chatbot to say: “Pick up some pentomino pieces and try them out for yourself.” Then you are forced to touch it by yourself and not just play through it theoretically. [B4; para. 18]

Participants’ ideas for further development of the planning tool included the addition of a detailed lesson plan option, which they either already expected (B1, B3, B11) or felt would be helpful, especially for rarely taught mathematical topics, and to support out-of-field teachers (B7). Furthermore, an option to save input and restore it later (B2) as well as to get immediate feedback on inputs (B7) should be implemented. In addition, participants wish to have a brief summary of the chosen learning environment prior to the impulses of the planning tool (B8), and the resulting planning summaries should be presented as an interactive flowchart (B9, B10).