Analysis of online assignments designed by chemistry teachers based on their knowledge and self-regulation

1.1 Online learning and self-regulated learning

Traditional learning and teaching happen face-to-face in a classroom with students and their teacher. The transition into online learning occurred during the last decades along with the expanding use of the internet (Barak et al., 2016; Wei et al., 2022). Online learning passes the responsibility of learning regulation to the learners (Barnard-Brak et al., 2010; Hodges et al., 2020; Lynch & Dembo, 2004), but the integration of online assessment into online learning has become a challenge for teachers (Amasha et al., 2018; Eichler & Peeples, 2013). Yet, this challenge is accompanied by advantages, primarily the ability to provide personal feedback for the learners, which allows them to monitor their progress in learning (Dietrich et al., 2021; Dipietro, 2010; Vonderwell et al., 2007).

Self-regulation is needed for performing online assessment, which is a key aspect of students’ and teachers’ self-regulated learning (SRL). Teachers are required to plan and support such processes for themselves, as well as for their students (Kramarski & Heaysman, 2021; Kramarski & Michalsky, 2010). SRL is an active personal and constructivist process, in which the learner acquires learning skills, sets personal goals, chooses learning strategies, and monitors the efficiency of the paths taken (Zimmerman & Moylan, 2009). In this process, the learner participates in behavioral, metacognitive, and motivational ways.

Fostering of students’ SRL skills by teachers has been reviewed in the literature, relating to both explicit and implicit approaches (Cohen et al., 2021; de Bruijn-Smolders et al., 2016; Dignath & Büttner, 2008, 2018). The SRL skills in our study include time planning and management, goal setting, help-seeking, and reflection. The transformation of teachers from traditional teacher-centered, face-to-face practices into SRL mentors who promote their students’ SRL skills encompasses several domains of teachers’ knowledge types, discussed next.

1.2 Teachers’ knowledge types

In the process of professional development, teachers are expected to acquire large amounts of diverse knowledge. Shulman (1986) addressed different types of knowledge, primarily content knowledge (CK), pedagogical knowledge (PK), and pedagogical content knowledge (PCK). PCK was defined as an amalgam of content and pedagogy into a specific understanding of how domains and problems are organized, represented, and adapted to learners’ interests and capabilities and how they are aligned with teaching and learning (Shulman, 1987). Assessment knowledge (AK) has been proposed as a meaningful and unique type of teachers’ knowledge (Avargil et al., 2012; Dori & Avargil, 2015; Magnusson et al., 1999; Mertler, 2009; Tal et al., 2021). Studying the implementation of a new chemistry curriculum, researchers (Avargil et al., 2012; Mertler, 2009) found that AK is expressed in the teachers’ ability to develop high-level student-centered assignments. The expansion of technological knowledge, teachers are currently required to acquire, is expressed in their technological pedagogical and content knowledge (TPACK). TPACK was defined as the ability to include technology in a way best suited for teaching the subject matter at hand (Koehler et al., 2013; Voogt et al., 2013).

Methods for assessing teachers’ knowledge types in the literature can be divided into several approaches. The common approach is questionnaires that rely on teachers’ self-report and answers to close-ended questions regarding practice and perceptions (Willermark, 2018). Another approach is based on “paper and pencil” tests, which are tailored for a specific subject matter and usually focus on CK or PCK only (Gess-Newsome et al., 2019; Großschedl et al., 2019). A different approach is based on practice research and analysis of different representations of teachers’ practices, such as lesson videos, lesson reports, and student products (Canbazoglu et al., 2016; Harris et al., 2010). The advantage of the latter approach is that the teachers’ knowledge is attributed to their actual actions and performance rather than to their declarations.

Following this approach, analyzing teachers’ self-designed online assignments (Peretz et al., 2023) can reflect the depth and width of teachers’ knowledge in a broad range of skills and expose their perceptions. The underlying assumption here is that an assignment developed by the teacher can reveal their pedagogical, technological, and assessment competencies, as well as their perceptions, including self-efficacy and pedagogical considerations. Previous implementations of assignment analysis included probing TPACK of teachers (Koh, 2013; Oster-Levinz & Klieger, 2010) and probing “Assessment literacy” (Koh et al., 2018; Mertler, 2009), which is similar to AK.

TPACK is somewhat controversial, as it is described based on two approaches. The first one views TPACK as being comprised of three basic constructs: CK, PK, and TK. When these overlap, they give rise to PCK, TPK, and TCK, which, in turn, overlap to yield TPACK. Alternatively, TPACK is a construct (Brantley-Dias & Ertmer, 2013) that is integrative and transformative (Canbazoglu et al., 2016; Schmid et al.,2020). Rather than an accumulation of the three basic constructs (CK, PK, and TK) TPACK develops through teachers processing technological support and applying it into their PCK. Koehler et al. (2012) reviewed different TPACK measurement approaches to probe teachers’ beliefs, knowledge, skills, and self-efficacy. The current research assumes that teachers (especially experienced ones) come with an established PCK for their teaching, along with TK from their daily experience, upon which they develop their TPACK. Accordingly, we define TPACK as the average of their PCK and TK, in line with the transformative approach of Koehler et al. (2013).

A rubric for probing TPACK and AK (Herscovitz et al., 2023) served as a basis for this research and was expanded to assess the fostering of SRL skills as well. The goal of this research was to investigate chemistry teachers’ professional development (PD), focusing on the various levels of teachers’ knowledge types. To this end, we have analyzed the online assignments the teachers had designed, while they participated in a PD program and their reflections on the process they had gone through.

2.1 Research goal and questions

The research aimed at investigating the types of knowledge chemistry teachers leveraged in online assignments they had designed, which included guidelines to promote SRL. Since reflection is an important part of SRL, we also investigated the teachers’ reflections on developing and implementing the assignments in their classes. The research questions were: (1) What are the differences between various teachers’ knowledge types as reflected in the online assignments they had designed? (2) What are the characteristics of the teachers’ reflections on developing the online assignments to promote their students’SRL and the teachers’ declared intentions to integrate them into their classes?

2.2 Research participants and setting

The 20 high school chemistry teachers who participated in the research took part in two PD programs focused on fostering SRL. The chemistry teachers shared a common interest in broadening their knowledge and skills of teaching in online environments, assessment, and SRL. With this homogeneous content knowledge background, these teachers served also as a focus group. The research included two professional development (PD) programs: (1) a 10-h intervention as part of a professional learning community for leading chemistry teachers, and (2) a 30-h summer program. Both programs focused on five aspects:

1. Exposing the teachers to the significance of SRL and introducing the SRL cycle and phases: forethought, action, and reflection,

2. Experiencing SRL, i.e., teachers as self-regulated learners,

3. Presenting video clips as introductions^[1] to assist students in acquiring SRL skills,

4. Designing new online assignments or including SRL-promoting guidelines in assignments that teachers had previously used, and

5. Reflecting on the process teachers went through during the program, which included discussing and planning the class implementation of the SRL assignments, as well as teaching that promotes students’ SRL.

The 20 teachers varied in their experience: about half were leading chemistry teachers, and 40 % had less than 10 years of experience. As the term “experience” is sometimes referred to as “seniority” (Feldman, 1983), we use experience as a proxy to seniority when the experience spans at least a decade.

2.3 Research tools and data analysis

The research tools included online assignments or assignments with an online component that the teachers had developed, as well as teachers’ reflections. After the teachers had adapted the online assignments for SRL by including SRL fostering guidelines, the assignments were collected and analyzed. Analysis of the assignments was performed using a designated rubric, based on previous research (Herscovitz et al., 2023), referring to PCK, TK, and AK types of teachers’ knowledge. The TPACK score was calculated as the average of PCK, and TK component scores as expressed in the online assignment. In the rubric, we extract PCK from the online assignments developed by the chemistry teachers. The TK criteria relate to the technological attributes that promote PCK. Thus, both dimensions of the rubric probe TPACK as the overlap of PCK and TK, in line with Koehler et al. (2013). Averaging the two components normalizes the score, so it is on par with the AK and SRL scores. AK, especially that related to online assignments, was included in the analysis due to its importance. A unique type of teachers’ knowledge in this research was their ability to promote SRL (labeled SRL knowledge). Table 1 shows the categories and criteria of the rubric. The maximal rubric score for an assignment is 30 points, referring to an online assignment optimally suited for SRL. The teachers’ knowledge type and a specific criterion, chosen as an example from the rubric in Table 1, are presented next to an excerpt or a description found in the assignment, along with the reasoning for the ranking of that criterion. Some of the criteria can be demonstrated by the inclusion of a specific guideline or feature in the assignment. Other criteria may only be analyzed by a holistic view of the assignment, preventing the ability to match a criterion to an excerpt from the assignment.

Figure 1 presents a few snapshots and the analysis of specific attributes (criterion, score, and reasoning) from the assignment that is referred to in Table 2. Examples of the analysis method are shown in Table 2. A complete assignment is presented in Figure 2 (see Findings section) with ranking for each criterion of the rubric and its reasoning in Table 3.

Figure 1:

“It’s raining moles” – a google-form based assignment (participant 2010122) – a partial example of the analysis.

Figure 2:

The virtual “Chemical Escape Room” assignment (participant 2020122).

The participants were asked to conclude their learning process by reflecting on three aspects: (1) the strengths or advantages of implementing SRL-promoting assignments that they had developed, (2) the challenges or difficulties they anticipate in implementing the SRL-adapted assignment in their classroom, and (3) whether they intend to implement the assignments in their classrooms.

2.4 Ethics

This research was reviewed and approved by the Behavioral Sciences Research Ethics Committee of the Technion – Israel Institute of Technology (approval number 2020-092).

3.1 Teachers’ knowledge types as demonstrated in their self-designed online assignments

Following is a detailed example of the analysis of the assignment The virtual “Chemical Escape Room” of participant 2020122, inspired by the special EREA room designed by Avargil and colleagues (2021). The translated assignment is presented in Figure 2. Table 3 presents the rubric, including the knowledge type domains, criteria, scores, and the reasoning behind them. The total scores of the assignment for each knowledge type are: PCK is 8, TK is 7, and the average of these two – TPACK is 7.5. AK and SRL scores are 7 for each one of them.

The analysis, presented in Table 3, demonstrates that the assignment designed by participant 2020122, complies with most of the criteria in the rubric at a high level. Thus, testifying of high teachers’ knowledge levels in all the examined domains.

Table 4 presents the results of the analysis of 20 assignments. The level of knowledge type: PCK, TK, AK, and SRL were extracted using the rubric presented in Table 1, and the TPACK was calculated as the average of the PCK and TK scores.

Table 4 showcases a variety of chemical subjects and assignments, as well as different technological solutions for designing online learning assignments. The results of the analysis demonstrate high PCK levels of most teachers as well as TPACK. It can be seen clearly by observing the results for the different participants, that there is a variance of combinations for TPACK, SRL, and AK scores. Some teachers exhibit (through their online assignment analysis) a balanced knowledge profile, in which the levels of teachers’ knowledge types are relatively similar. Other teachers show differences in the knowledge type levels, with usually the AK and SRL being lower.

4.1 Teachers’ reflections

Content analysis revealed several categories for each aspect of the teachers’ reflections. Regarding strengths or advantages, most teachers pointed out the necessity to encourage students for independent learning and more responsibility for their learning. Some of the teachers found it advantageous that students’ responses to SRL guidelines could serve as feedback for them and allow improvements in their classroom teaching. Concerning the challenges, all the teachers mentioned their students’ difficulties to manage their learning and assume responsibility, especially when they must study an unfamiliar topic.

Most of the teachers stated that they would include guidelines to promote SRL in the assignments for their students. However, there were differences concerning the extent and mode of implementation, with some teachers stating they would integrate the SRL-promoting guidelines gradually. They intend to plan each online assignment to focus on guidelines for a specific SRL skill, such as time management and help-seeking. Other teachers noted that implementing assignments that promote SRL is suitable for lower high school classes, 9th and 10th grade, in which those skills are needed to allow students’ academic development. Other teachers thought that implementing SRL guidelines in students’ assignments in the 12th grade is favorable, and SRL could be integrated into the learning of the advanced topics of the curriculum, in which 21^st century learning skills are more pronounced.