The impact of a professional development MOOC on the teaching beliefs of University Science Laboratory Teachers

Teachers’ beliefs and intentions toward learning facilitation and knowledge transmission

The way teachers and lecturers teach is associated with their conception about good teaching and their discipline (Lindblom-Ylänne et al., 2006; Rienties et al., 2014). Research has shown that traditional lecturing doesn’t have a sufficient learning effect and often results in surface learning (Freeman et al., 2014; Lombardi & Shipley, 2021). Nevertheless, many teachers in higher education institutions (HEIs) still use traditional teaching methods. This is because their conceptions about what good teaching is are teacher-centred and connected to knowledge transmission. Teachers who have adopted a more student-centred approach to teaching are more likely to stimulate students to achieve deep learning (Gow & Kember, 1993; Postareff, Lindblom-Ylänne, & Nevgi, 2007; Prosser, Martin, Trigwell, Ramsden, & Lueckenhausen, 2005; Trigwell, Prosser, & Waterhouse, 1999; Uiboleht, Karm, & Postareff, 2018). Gow and Kember (1993) classified conceptions of teaching on a continuum from a totally teacher-centred, content-orientated conception to a totally student-centred and learning-orientated conception.

Changing teachers’ attitudes to a student-centred approach takes time. A study by Postareff et al. (2007) implied that it takes at least a one year long training process until significant effects emerge. Vilppu et al. (2019) however showed that a short online training course can affect interpretation of teaching and learning situations, especially for less experienced lecturers. Rienties et al. (2013) studied online professional development of university teachers. They argued that lecturers could benefit from online training programmes irrespective of whether they are more inclined towards student-centred or teacher-centred approaches. According to their results from using the teachers’ beliefs and intentions (TBI) instrument devised by Norton et al. (2005), Rienties et al. (2013) showed that lecturers often don’t have a strong orientation and that they can be classified as having both student and teacher-centred beliefs.

Teaching beliefs of chemistry lecturers have not often been examined. A recent small scale study by Popova et al. (2020) focussed on early career chemistry lecturers. Participants were found to have a range of beliefs. Some were teacher-centred, some were student-centred and some had transitional beliefs at the midpoint between these two. Content knowledge determines the pedagogical approach as well as the adoption of particular technologies (Mishra & Koehler, 2006). In a review of 118 course designs for transitional remedial education, Rienties et al. (2012) found that lecturers from 22 countries consistently aligned their content with their pedagogical approach. Norton et al. (2005) note that several authors have concluded that a change to a more student-centred approach can only be achieved through a professional development course if it involves addressing teachers’ underlying conceptions of teaching and learning.

Online course for university laboratory teachers

Many universities recognize nowadays that there is a need for professional development of lecturers in higher education and they organize professional development activities for their teaching staff. In some countries, lecturers can achieve a University Teaching and Learning Certificate or a similar qualification. Nevertheless, these programmes are often generic and don’t focus on subject specific STEM pedagogical aspects that have a consequence on how students learn (Walsh, 2017). Currently, the importance of professional development in STEM teaching is not recognized to the extent that it should be Winberg et al. (2019). There is also not enough attention given to teaching in university science laboratories.

The European Chemistry Thematic Network (ECTN, www.ectn.eu) has over 100 European HEI members and is committed to the improvement of university chemistry education in Europe. The focus of the ECTN Working Group on Lecturing Qualifications & Innovative Teaching Methods, which was established in 2015, is the improvement of teaching practice in chemistry laboratory classes in higher education and of teaching approaches of newly appointed higher education teaching staff. This Working Group has developed an open online course entitled “Teaching in University Science Laboratories (Developing Best Practice)”. Its purpose is to support relatively inexperienced university lecturers and their teaching assistants to cope with the complex teaching demands in laboratory classes and to connect them in fruitful discussions about this across university and country borders. The design of the course was established in a position paper (Brouwer et al., 2016). The approach used was to launch a pilot small private online course (SPOC) initially and to make modifications based on what was learned in order to develop a massive open online course (MOOC) as the next iteration.

The learning outcomes of the online course are that participants will be able to:

1. Identify the purposes of implementing laboratory classes in higher education.

2. Provide strategies on how to increase student engagement.

3. Develop effective questions to probe student understanding of laboratory practice.

4. Create a rubric for assessing a student performing a lab activity and subsequent reports.

Research questions

A series of four research sub-questions were formulated to allow all aspects of the development and implementation of this open online course to be evaluated;

1. To what extent was the chosen development process successful for developing a MOOC for continuing professional development of university laboratory teaching staff?

2. What are the differences in course completion between the SPOC and the MOOC phases, if any?

3. Did the open course change the beliefs of participants about good teaching and, if so, how?

4. Were there aspects of the online course that the participants found useful for their teaching practice and, if so, what were they?

Design and development of the online course

In order to reach as many lecturers in higher education who are involved in university laboratory courses as possible, the ECTN Working Group on Lecturing Qualifications & Innovative Teaching Methods decided to develop an open online course. The working group had no specific budget. The time and effort of its designers was provided on a voluntary basis and the University of Amsterdam enabled the working group to host the course on the MOOC platform Coursera. In several phases of the development of the online course, the team used voluntary input and feedback from ECTN network members across European universities. The approach used to design and develop the online course followed an adapted ADDIE model (Peterson, 2003) Figure 1 and had three ADDIE cycles. The steps within each cycle are described in Appendix A. The first cycle resulted in pilot 1, a SPOC 1 in 2017. The second ADDIE cycle resulted in SPOC 2 in 2018. This online course took 6 weeks (one week per module) and remained open for the participants to complete for another two weeks. The third cycle resulted in a MOOC 1 in 2019. The MOOC 1 took 6 weeks with an additional seventh week for completion of all assignments if needed. Once the initial 6 week iteration of MOOC 1 was complete, it remained available on the Coursera platform from the beginning of 2020 as MOOC 2. The MOOC 2 is operating in fully automated mode and participants could choose to begin the course whenever they prefer. We have analysed the data of all iterations. MOOC 2 data from March 2nd 2020 to February 14th 2021 are of special interest for several reasons. In March 2020, numerous restrictions to face to face interactions were introduced globally because of the COVID-19 pandemic. Also, the online course was not being promoted to the same extent through ECTN from stage MOOC 2 anymore and it was anticipated this could have an impact on the number of participants and the participant profile.

Figure 1:

MOOC creation process, adapted ADDIE model (Peterson, 2003).

Participants in the study

The participants in the study were enrolled on one of the four course iterations: SPOC 1, SPOC 2, MOOC 1 or MOOC 2. Table 1 presents the information collected on the number of participants and the responses obtained to questions focused on demographic information.

The female to male ratio favored women in all iterations of the online course in this study, but in MOOC iterations the ratio became more balanced (Table 1). The number of participants in the study who completed the evaluation survey in all four course iterations is 283: 23 in SPOC 1, 45 in SPOC 2, and 36 in MOOC 1 and 179 in MOOC 2. The number of participants who completed the TBI questionnaires in SPOC 1 was pre-test 33, post-test 14 and 11 completed both tests, for SPOC 2 pre-test 52, post-test 32 and both tests 21, and in MOOC 1: 84 pre-test, 42 post-test, and 27 completed both tests. 580 completed the TBI pre-test in MOOC 2, 85 post-test and 60 participants completed both TBI tests.

The participants in SPOC 1 and SPOC 2 came from 80 different countries. Further details are presented in the table in Appendix B. This information is sourced from the SPOC registration questionnaire and covers the period from Dec 2017 to Dec 2018.

The majority (54.6%) of participants in SPOC 1 and SPOC 2 came from European countries. The SPOCs also had participants from countries on other continents. In the table in Appendix C, the location of all participants up to July 2021 is presented based on Coursera data. It can be seen that the course reach after the SPOCs became even more global over time and that most participants (46.7%) now come from Asia.

Classification of teachers’ beliefs and intentions (TBI) profiles

Based on the scores in the TBI questionnaire instrument, the participants were classified in three different groups and the criteria used are shown below.

1. Teacher centred

   • TBI score in Teaching Beliefs: Learning Facilitation < Knowledge transmission and TBI score in Teaching intentions: Learning Facilitation < Knowledge transmission.

2. Student centred

   • A) Student centred 1

   • TBI score in Teaching Beliefs: (Learning Facilitation > Knowledge transmission) ≥ 0.3 and TBI score in Teaching intentions: (Learning Facilitation > Knowledge transmission) ≥ 0.3

   • B) Student centred 2

   • TBI score in Teaching Beliefs: (Learning Facilitation > Knowledge transmission) ≥ 0.3 and TBI score in Teaching intentions: (Learning Facilitation < Knowledge transmission) > 0 and < 0.3

   • or

   • TBI score Teaching Beliefs: (Learning Facilitation > Knowledge transmission) > 0 and <0.3 and TBI score in Teaching Intentions: (Learning Facilitation > Knowledge transmission) ≥ 0.3

3. Student- & Teacher centred

   TBI score in Teaching Beliefs: (Learning Facilitation > Knowledge transmission) < 0.3 and TBI score in Teaching intentions: (Learning Facilitation > Knowledge transmission)  < 0.3.

Completion rates and engagement in the course

Table 2 presents the completion numbers and rates for each iteration of the course. The final iteration, MOOC 2, had the lowest completion rate of 15%. However, as it was more flexible and participants could join at any stage over the timeframe analysed of almost 12 months, the completion number (266) was over 6 times greater than MOOC 1 (42). As explained in the footnote to Table 2, SPOC 1 and SPOC 2 included a cohort of blended learners and all of this group of participants completed the course (19 for SPOC 1 and 23 for SPOC 2). This method of embedding engagement with a SPOC into a larger CPD programme that includes face to face interaction is an approach that some other universities could consider (Jia et al., 2019). The large difference in completion rate could be explained by more intense involvement of the teaching staff (instructors) in the SPOC in comparison to the MOOC and thus a greater teaching presence. It is known that a balance of three presences: cognitive presence, teaching presence and social presence, is crucial for a successful online learning experience (Garrison, Anderson, & Archer, 2010; Nazir & Brouwer, 2019). Boltz et al. (2021) examined implementation of a MOOC to support school teachers and reflect that completion rates should be compared to those for other digital content such as podcasts instead of to formal online courses. They also report that a certificate of completion may not be the main motivation for some MOOC participants and their focus could be on learning something new or networking instead. For this reason, participants may not finish all elements of the course but may still have learned from it and achieved their goals.

Some participants expressed in the evaluation that they needed more time than was described in the instructions for the course. Based on these comments, the authors redesigned several parts of the course and added summaries of the reading material.

Evaluation of the participants’ experience

In the last course module, the participants were asked to complete a survey (Appendix D) with nine Likert scale questions, five open questions in which the participants could discuss their experience of the course and several demographic related questions. There was a response rate for the post-course evaluation survey of 77% of those who completed SPOC 1, 77% for SPOC 2, 86% for MOOC 1 and 67% for MOOC 2. From October 25 2019 to February 17 2021, 215 (of 2015) participants from 45 different countries and five unidentified participants completed the evaluation. The country that had most participants was India.

We were interested in the experiences of the participants with professional development. The answers to the Question “Which way do you usually develop your teaching competences? (you can choose more than one option)” provided by participants in MOOCs 1 and 2 are presented in Table 3. More than 34% had never before thought about developing teaching competences.

Responses to another question revealed that 33.2% of the participants in MOOC 1 and MOOC 2 had never before followed an online course and 44.4% had never completed one. However, 17.6% regularly participated in and completed online courses (Table 4).

The Likert scale section of the post-course evaluation allowed participants to evaluate the course quality using a range of criteria. Table 5 shows the results and each course iteration is presented separately. The values in each follow-up course after SPOC 1 show that the improvements made based on the evaluation of the previous iteration were beneficial.

For the question “What do you consider to have been the most useful aspects of the course?” (Appendix D), the answers can be classified in different categories, as shown in Figure 2. Participants mentioned topics such as rubrics, Johnstone’s triangle and cognitive overload. Several participants mentioned that the activities in the course were most useful, i.e. readings, discussion prompts and the assignments. Several mentioned that communication with other lecturers who were participating was most useful and others said the reflective activities were the most useful for them.

Figure 2:

Categories of participant responses to the question: “What do you consider to have been the most useful aspects of the course?”

Several examples selected from responses by MOOC 1 and MOOC 2 participants show these different perspectives and are presented in Figure 3.

Figure 3:

Representative responses of participants in MOOC 1 and MOOC 2 to the question: “What do you consider to have been the most useful aspects of the course?”

For the evaluation survey question, “What differences (if any) do you think participating in this course will make to your teaching practice?”, the answers showed different plans to improve their courses. In particular, respondents intend to improve the interactivity of their courses, improve their questioning techniques and introduce rubrics as an assessment instrument. Some representative responses are shown in Figure 4.

Figure 4:

Representative responses of participants in MOOC 1 and MOOC 2 to the question: “What differences (if any) do you think participating in this course will make to your teaching practice?”

The participants from Jagiellonian University who experienced a blended learning format instead of fully online provided feedback that they appreciated the blended format. Three face-to-face meetings made it possible to present the principles of the SPOC course, to deal with queries that arose during the implementation of the first three modules and to summarize and, at the last meeting, to correct any misconceptions. Taking part in the course allowed these participants to address stereotypes, such as the belief that the main goal of laboratory classes is to help learners understand the content presented during lectures. The new element for many of the participants was peer evaluation of their work with the help of rubrics. This raised doubts for them and clarification was needed (Brouwer et al., 2018). The observations from teaching this group in the blended format were very informative and were considered when making changes when SPOC 1 and 2 were being reviewed.

Evidence that the chosen course design was effective includes the relatively high completion rate, in particular in both SPOCs, together with very positive evaluations from the participants in all iterations of the online course which demonstrated that participants who completed the course felt that they achieved the learning outcomes. As shown in Table 5, a large majority agreed that, as a result of this course, they developed new ideas to help them to improve their teaching and that participation in the course increased their understanding of the multidimensional aspects of laboratory teaching and the challenges related to it. Figure 4 provides some specific examples of new ideas and approaches generated from the course. In their comments on the open question on what they consider the most useful aspects of the course, the participants often mentioned design elements, such as discussion prompts.

Teachers’ beliefs and intentions

By applying the profile classification system specified in the methods section to the responses to the TBI survey (Norton et al., 2005), it was possible to gain insights into what participants thought was most important in relation to teaching and learning. Figures 5 –8 show a pre and post comparison by survey category for the teachers’ beliefs of 11 of the participants who completed SPOC 1, 23 who completed SPOC 2, 27 who completed MOOC 1 and 60 who completed MOOC 2 respectively.

Figure 5:

Teachers’ beliefs of participants in SPOC 1 (Dec 2017–Jan 2018) who completed both the pre- and the post-course survey, n = 11.

Figure 6:

Teachers’ beliefs of participants in SPOC 2 (Nov 2018–Jan 2019) who completed both the pre- and the post-course survey, n = 23.

Figure 7:

Teachers’ beliefs of participants in MOOC 1 (October–December 2019) who completed both the pre- and the post-course survey, n = 27/26 (one post-test was not fully completed).

Figure 8:

Teachers’ beliefs of participants in MOOC 2 (March 2020–February 2021) who completed both the pre- and the post-course survey, n = 60.

For all iterations of the online course, an increase after the course was completed was observed in all of the teachers’ beliefs categories associated with Learning Facilitation (problem solving, interactive teaching, facilitative teaching, pastoral interest and motivating students). In SPOC 1 and the two MOOCs, an increase was also observed after the course was completed in the three categories associated with Knowledge Transmission (training for jobs, imparting information and knowledge of subject). However, this increase was to a lesser extent than for the Learning Facilitation category. For SPOC 2 participants, a decrease was observed for all Knowledge Transmission categories of questions while the increase for the Learning Facilitation dimension was more modest than for the other course iterations. The main outcome from this analysis is that Learning Facilitation beliefs increase for all participants after course completion and that Knowledge Transmission beliefs either decrease or do not increase to the same extent as the Learning Facilitation beliefs. This demonstrates that, on average, participants had a more student-centred focus after course completion.

Table 6 shows a comparison of the TBI pre-course profile classification for each short course cohort alongside data from another online teaching course which one of our team was involved in previously that included lecturers teaching a wide range of disciplines (Romero & Barberà, 2011).

Table 6 shows that the higher education lecturers who teach in the laboratory courses have less teacher-centred beliefs than found by Rienties et al. (2013) when they studied lecturers across a wide range of disciplines.

Table 7 shows the pre- and post-course TBI profile classifications for each course cohort and includes the participants’ teaching experience.

The results in Table 7 show a reduction in the number of participants categorised as teaching-centred for SPOC 1, SPOC 2 and MOOC 1. There is also an increase in the percentage of participants with a student-centred profile (the sum of categories SC1 and SC2) in SPOC 2, MOOC 1 and MOOC 2. The result observed in SPOC 1 showing a slight decrease in the percentage of participants with a student-centred profile is difficult to interpret and may not warrant too much focus. This is because the sample size is small and the participants included several experienced academics who provided specifically feedback to the course designers. These participants were not the intended target audience.

For MOOC 2, it was observed that, although the percentage of participants with a student-centred profile increased, the percentage with a teacher-centred profile also increased slightly from 18.3% (11 participants) to 20.0% (12 participants). Further analysis of the teaching beliefs scores for the 12 participants with a post-course teacher-centred profile showed that 2 of them showed increases in one or more Learning Facilitation categories (in the rest of the categories, there was no change). Four of them showed increases in two or more Learning Facilitation categories but showed a decrease in one other category and no change in the others. Therefore, six of the participants with a teacher-centred profile showed some aspects of being more student-centred after taking the MOOC 2. MOOC 2 was in operation during the COVID-19 pandemic and perhaps this may have had an impact on teachers’ beliefs of some participants. Also, because a MOOC doesn’t involve interaction with facilitators in the discussion board activities, there may have been some peer interaction taking place that reinforced teaching-centred beliefs and intentions. One disadvantage of MOOCs is the limitations to assessment of learning and the lack of opportunity for feedback to be provided from a tutor (Cabrera & Fernández-Ferrer, 2017; Kang & He, 2018; Wei, Saab, & Admiraal, 2021). Huisman et al. (2018) investigated the impact of peer-reviewer ability on performance of learners for a MOOC essay assessment. They observed a positive relationship.