The effectiveness of online learning activities in a blended EAP course: relationships between language proficiency, online engagement, and language performance

2.1 Online activities in blended English learning

In the context of blended English learning, past studies have suggested both generic and specific models. Wang et al.’s (2009) study summarised that a blended English course should comprise three key stages: 1) online preparation of basic knowledge, 2) face-to-face lecturing, and 3) online revision. Wang et al. (2009) recommended that the extent of participation, homework, and course exams be used to analyse and evaluate learner achievement. Nevertheless, this model remained at a procedural level and failed to provide specific insights into blended English course design, particularly regarding the details of the online learning activities. Aligned with these general procedures, Tao et al. (2024) provided more detailed approaches to their blended teaching mode, specifically regarding the online learning content. Prior to onsite classes, students were required to complete a series of self-study online tasks, including MOOCs, textbook-based tasks covering vocabulary, grammar, reading, listening, translation, and writing, and submit questions encountered during the online learning (Tao et al. 2024). One limitation of this blended model lies in its non-integration of speaking into the blended course. Another limitation is its lack of disclosure regarding how the online multimodal activities (e.g., videos, audio, texts) were organised and integrated with onsite learning activities. This study aims to address these theoretical gaps by proposing a blended EAP model that incorporates all four skills into the design of online activities and details how various online multimodal activities can be effectively integrated with each other and with onsite learning activities.

Although blended learning has been shown to improve English language performance across various contexts, its impact on specific language skills remains inconclusive. Hubackova and Ruzickova’s (2011) study revealed that the online activities were useful for improving grammar and vocabulary, which might allow opportunities for enhancing listening and reading competencies. Liu et al. (2020) demonstrated student improvements in listening and speaking through web-based autonomous learning. These discrepancies may arise from inconsistent designs of the online activities in the blended models. Regarding experimental studies comparing blended and traditional learning, Zhou (2018) found that blended learning significantly improved learners’ writing skills in the aspects of content relevance, logical structure, content sufficiency, and language expression. Tao et al.’s (2024) study showed that learners in the blended course outperformed those in traditional classrooms on CET6 (College English Test Band 6) in reading, listening, and writing. Similarly, Moradimokhles and Hwang (2022) found that students in blended learning scored higher on TOEFL reading, listening, and writing than those in traditional classrooms. However, neither Tao et al. (2024) nor Moradimokhles and Hwang (2022) included the speaking component in their blended course designs. Banditvilai (2016) and Nuri and Bostanci (2021) highlighted the positive effects of blended pedagogy on all four key language skills. Because these studies have examined the blended mode as a whole, little is known about whether its effectiveness is linked to specific online modalities or course designs. To address these gaps, this study aims to investigate the impact of the online learning activities on student performance across all four language skills and thereby evaluate the effectiveness of its design.

2.2 Online engagement and learner performance

Engagement is often explored in educational research as a key construct due to its potential contribution to academic success, higher attendance rate, and positive emotions (Fredricks 2015). Despite the variations in the definitions of the term, it is generally agreed that engagement comprises three core dimensions: cognitive, behavioural, and emotional engagement (Fredricks et al. 2016). First, cognitive engagement involves self-directed learning, employing deep learning strategies, and devoting efforts to comprehend complicated ideas; second, behavioural engagement is characterised by participation, attention, effort, and persistence; third, emotional engagement pertains to the degree of positive reactions toward school, teachers, or classmates, as well as the feeling of belonging and identification with the school or specific subjects (Fredricks et al. 2016).

The most commonly used method for assessing engagement is self-reported surveys (Fredricks et al. 2016; Lin 2018), especially when measuring emotional engagement. Despite their advantages in time- and cost-efficiency and ability to capture abstract concepts, self-report measures often exhibit substantial variability and rely heavily on researchers’ interpretations (Fredricks and McColskey 2012; Gobert et al. 2015; Greene 2015). An alternative approach to measuring online cognitive engagement involves regular formative assessments, such as weekly online quizzes. These quizzes were designed with varying cognitive levels of questions, incorporated automated feedback, and provided grades to facilitate learning (e.g., Hettiarachchi et al. 2015; Hughes et al. 2020). Woeste and Barham (2008) observed that students expressed high appreciation for the metacognitive benefits of weekly quizzes at later stages of their learning, noting their usefulness for understanding the content and serving as effective revision tools for exams. Additionally, behavioural engagement can be measured through various indicators, such as frequency and duration of online participation, as well as assignment completion. Specifically, Green et al. (2018) used metrics such as the number of videos and files viewed and hits on discussion forums as measures of behavioural engagement. Rubio et al. (2018) collected data on indicators including page views, the number of discussion posts, on-time assignment submissions, and active days. Similarly, Morris et al. (2005) examined students’ access log data, focusing on the frequency and duration of content page visits, the number of discussion posts read and replied to, and the time spent reading and responding to posts. Emotional engagement is excluded from this investigation due to its potential to introduce subjectivity into the measurement.

Online engagement in blended learning environments has been shown to directly contribute to improved student performance and course effectiveness. Green et al. (2018) found that higher levels of engagement with online materials predicted better grades in a blended medical course. Similarly, Rubio et al. (2018) reported a positive correlation between online engagement and final grades in a blended Spanish course, highlighting continuity as the strongest predictor of success. Furthermore, Panigrahi et al. (2022) demonstrated that online engagement positively influenced the perceived effectiveness of a blended Management Information Systems course, enhancing learners’ self-efficacy and the quality of interaction with the course. Similar findings were observed in Hu and Hui’s (2012) study of a blended Adobe Photoshop course. While these findings underscore the benefits of online engagement across various fields, there remains a lack of research on its impact within EAP blended courses, particularly in relation to learner performance.

Moreover, previous studies (e.g., Means et al. 2013; Panigrahi et al. 2022) have shown that student online engagement in blended learning can be effectively enhanced through interactive approaches. Wang (2008) proposed a generic model (see Figure 1) for integrating online activities into teaching and learning, which consists of three core elements: learner-content interaction, learner-socialiser interaction, and learner-interface interaction. Each of these elements has been identified as a significant predictor of academic performance or student satisfaction with online activities (e.g., Amoush and Mizher 2023; Joosten et al. 2019; Kintu et al. 2017). Additionally, Panigrahi et al. (2022) established a positive relationship between these interaction dimensions and the online behavioural, cognitive, and emotional engagement dimensions. This generic model serves as a valuable framework for understanding student willingness to engage with the online activities of a blended course.

Figure 1:

A generic model for integrating information and computer technology (Wang 2008).

2.3 Proficiency levels and online engagement

Although the effectiveness of blended English learning has been documented in existing studies, there is a paucity of studies examining its suitability for students with different English proficiency levels. Tao et al.’s (2024) study is one of the few that investigated this issue in an EFL context, concluding that blended learning primarily benefited students at the intermediate and lower proficiency levels, while those with medium-high and high proficiency levels benefited less from this approach. This was attributed to the blended course design, which mainly facilitated knowledge consolidation at the cognitive levels of memorisation and understanding, rather than promoting deep transfer and integration of language knowledge (Tao et al. 2024).

While Tao et al. (2024) highlight the limitations of blended learning for students with higher proficiency levels, some studies suggest that carefully designed online activities can benefit learners across ability levels. For instance, Green et al. (2018) demonstrated that online engagement in a blended medical course served as a mediator between students’ prior ability and final course grades, emphasising the importance of incorporating online engagement into blended course design. Unlike Tao et al.’s (2024) findings, Green et al. (2018) observed that online content benefited students across different ability levels, although students with stronger prior ability exhibited higher online engagement, leading to better final grades. Given that Green et al.’s (2018) study was conducted in a medical education context, further investigation is needed to determine how students with diverse English proficiency levels engage with online activities in a blended EAP learning environment.

3.1 Context

This study was conducted at a private transnational English-medium instruction (EMI) university in China. Stratified, credit-bearing English for Academic Purposes (EAP) courses were offered to all Year 1 students as a foundational course, preparing them for the Year 2 full EMI programmes. Students at different English proficiency levels were streamed into three EAP courses – Foundation, Intermediate, and Advanced – based on their Oxford Online Placement Test results. The Intermediate course was the largest, catering to approximately 3,500 Year 1 students at the pre-intermediate, intermediate, and upper-intermediate levels of the Common European Framework of Reference for Languages (CEFR). A systematic blended EAP learning method was adopted in this 26-week year-long course, with two 2-hour online lessons provided before three 2-hour onsite seminars every week. All five EAP lessons were compulsory and scheduled in the students’ timetables.

Each pre-seminar online lesson mainly consisted of two key elements: a 10 to 15-min video lesson recorded by EAP teachers using PowerPoint (PPT) slides, and a follow-up quiz checking student comprehension of the video and application of the taught skills. This means the online activities of the blended course provided learners with approximately 52 videos and 52 quizzes over the academic year. All quizzes were designed with closed-type questions (e.g., multiple-choice questions, matching, True/False) in order to provide timely online feedback to the students. To complete each online lesson, students were required to watch the video and pass the online quiz at 40 %. Meeting these two requirements would enable the students to be automatically recorded as complete for the lesson. Upon completion of each online lesson, students were usually required to prepare answers for a piece of homework, which would be checked during the onsite seminars. The content of the online lessons was closely linked to the learning objectives of the onsite seminars, forming an integral part of the curriculum and serving as preparation for the onsite ones. EAP teachers in this course constantly emphasised the importance of the online learning activities and occasionally displayed the completion report in class to encourage student engagement in the online lessons.

3.2 Participants

All participants were recruited from those who were enrolled in the Year 1 EAP-Intermediate course in the Academic Year 2022–2023. After obtaining ethical approval and informed consent, 131 participants were recruited using a convenience sampling strategy in the quantitative stage, and their permission to access their statistical data was gained via an online questionnaire. Using maximum variation sampling (Dörnyei 2007), a sub-cohort of 18 participants was recruited for the second stage of semi-structured interviews.

3.3 Data collection

This study employed the following instruments and methods to measure participants’ language proficiency, online engagement, and language performance, and to gather their perceptions:

1. Language proficiency instrument

   Oxford Online Placement Test: This test, conducted prior to the academic year, measured participants’ English proficiency levels and mapped their scores to the CEFR levels: A2 (pre-intermediate), B1 (intermediate), and B2 (upper-intermediate). According to the CEFR levels (Council of Europe 2024), A2 learners can handle basic needs and discuss familiar topics, B1 learners can engage in standard interactions and give simple explanations, and B2 learners can demonstrate fluency, comprehend complex texts, and provide detailed explanations.

2. Online engagement instruments

   Online Quiz Results: To complete each quiz, learners were allowed to attempt the quiz multiple times until they achieved a passing score of 40 %. Although the quizzes employed a closed-question format, they were designed to target both lower and higher-order cognitive levels, drawing on findings from previous studies (Hettiarachchi et al. 2015; Hughes et al. 2020). Specifically, the quizzes assessed learners’ understanding of the video lesson content, their ability to transfer and apply acquired knowledge into practice, and their capacity to evaluate and critique authentic good and bad samples. The inclusion of these cognitively demanding tasks encouraged learners to engage deeply with the content rather than relying on surface-level recall. Additionally, automated feedback for each question provided opportunities for learners to reflect on their responses, identify areas for improvement, and refine their understanding. This iterative process promoted the use of deep learning strategies, making the quiz results a more valid indicator of learners’ cognitive engagement in the online lessons.

   Online Lesson Completion Rates: This rate, which represents the percentage of lessons completed throughout the academic year, served as a key indicator of learners’ online behavioural engagement. To be marked as complete for each lesson, students were required to watch the video and pass the quiz, making the completion rate a measure that reflects both behaviours. Consistent with prior research on behavioural engagement (e.g., Fredricks et al. 2016; Green et al. 2018), a higher completion rate suggests that learners consistently engaged with the online content and demonstrated sustained effort, persistence and participation over time. As such, they provide valuable insights into students’ behavioural engagement with the online content.

3. Language performance instruments

   EAP Assessment Scores: Summative assessments in reading, listening, writing, and speaking were carefully designed to align with the CEFR standards, ensuring that Year 1 EAP students achieved an exit level equivalent to CEFR B2. The reading and listening tasks were calibrated to match the difficulty level of B2 standards, while the writing and speaking assessments employed CEFR-aligned marking rubrics. This alignment provided consistent and standardised evaluation criteria across all language skills.

4. Qualitative method

   Semi-Structured Interviews: Interviews were conducted in their first language (Chinese) with a subgroup of students drawn from the participants in the quantitative stage. See Appendix A for the interview protocol. These interviews provided qualitative insights into learners’ perceptions of the online learning activities of the blended course.

Using the above instruments and methods, the following data were collected:

1. Oxford Online Placement Test results prior to the commencement of the Academic Year 2022–2023 (n = 131);

2. End-of-year overall online quiz results, as well as the reading, listening, writing, and speaking component scores (n = 131);

3. End-of-year online lesson completion rates (n = 131);

4. End-of-year overall EAP scores, as well as the reading, listening, writing, and speaking component scores (n = 131);

5. Semi-structured interviews with a subgroup of participants (n = 18).

3.4 Data analysis

The quantitative data were analysed using IBM SPSS Statistics 27. Descriptive statistics were generated for all variables under investigation. Both simple and multiple linear regression analyses were performed on the dataset to answer RQs 1 and 2. Simple linear regression can provide a fundamental understanding of how each predictor variable individually correlates with the outcome variable. Multiple linear regression analysis, which has the capacity to explain the joint effect of a set of independent variables (de Vaus 2014; Field 2018; Tabachnick and Fidell 2014), was used to examine the relationship between online engagement and EAP language performance, specifically for RQ2.

The interview data were analysed and coded via Nvivo 14. Reflexive thematic analysis, following Braun and Clarke’s (2022) recommended steps, was conducted to categorise the interview data and identify the recurring patterns. To provide qualitative insights into the online lesson design, the main categories of the codes were labelled deductively based on these two key elements: online quizzes and asynchronous video lessons. The sub-themes were merged inductively based on the repetitive codes, revised and refined iteratively throughout the coding process, and then summarised and systematised for further analysis. The reliability of the qualitative data analysis was enhanced by the researchers’ continuous reflexivity and collaborative engagement, following recommendations by Braun and Clarke (2022).

4.1 Proficiency levels and online engagement

To answer RQ1, simple linear regressions were performed on the dataset. The findings showed that the placement test scores statistically significantly predicted the overall online quiz scores (p = 0.19), explaining 4.2 % of the variance in these scores (See Table 2). The overall quiz results increased by 0.11 for every point increase in the placement test scores. The effect size indicated by R² is small according to Ellis (2010) and Cohen’s (1988) suggested thresholds: small (R² = 0.02), medium (R² = 0.13), and large (R² = 0.26). Additionally, the placement test scores did not predict EAP learners’ online lesson completion rates (p = 0.615), implying that students’ English proficiency levels did not affect their online behavioural engagement.

4.2 Online engagement and language performance

To answer RQ2, both multiple and simple linear regression analyses were conducted. Prior to performing the multiple linear regression, the assumption of multicollinearity was checked by inspecting the correlation between predictors, as well as the variance inflation factor (VIF) and tolerance values. Pearson’s correlation between the two predictor variables (i.e., overall online quiz results and completion rates) was 0.17, indicating a small level of correlation (Cohen 1988; Ellis 2010). The VIF was 1.03, and the tolerance value was 0.97, meeting the recommended thresholds of VIF <10 and tolerance >0.2 (Field 2018; Tabachnick and Fidell 2014). Therefore, the assumption of no multicollinearity was satisfied.

The multiple regression analysis (see Table 3) revealed that the combination of online lesson quiz results and completion rates statistically significantly predicted the overall EAP final scores, explaining 14 % of the variance in the scores and indicating a medium effect size. These findings evidenced the positive impact of online cognitive and behavioural engagement on learners’ EAP performance.

To obtain a more comprehensive understanding of whether these two predictor variables correlated with the EAP component scores, further statistical analyses using simple linear regressions were conducted on the dataset. The analyses (see Table 4) demonstrated that the online reading, listening, and writing quiz results statistically significantly predicted EAP learners’ corresponding assessment scores. Higher scores on online reading, listening, and writing quizzes were positively associated with better performance on corresponding EAP assessments. However, the online speaking quiz results did not predict EAP speaking scores (p = 0.16), indicating the limited effectiveness of the online quizzes in improving learners’ speaking performance.

Furthermore, simple regression analyses (see Table 5) showed that online lesson completion rates were significant predictors of EAP learners’ writing and speaking assessment scores. In contrast, the completion rates did not predict their reading and listening scores. These findings suggest that online behavioural engagement is a significant indicator of EAP performance in productive skills – writing and speaking – but not in receptive skills – reading and listening. Given that the online speaking quizzes were found to be statistically insignificant for improving speaking performance, the effectiveness of completing the online lessons is more likely attributable to the speaking video lessons.

4.3 Learners’ perceptions of the effectiveness of online learning activities

To address RQ3, the interview data offered constructive feedback on the two key elements of the pre-seminar online activities: online quizzes and asynchronous video lessons. Representative excerpts, labelled with participants’ pseudonyms and final EAP scores, are presented in Tables 6 and 7.

5.1 Proficiency levels and online engagement

Simple linear regression findings showed that EAP learners’ placement test results did not predict their online lesson completion rates, though they positively predicted online quiz results to a small extent. This indicates the lack of correlation between EAP learners’ language proficiency levels and their online behavioural engagement. Although students with higher language proficiency levels demonstrated slightly higher cognitive engagement with the online content, it can be argued that students with lower proficiency levels should not be deprived of such learning opportunities, given the potential benefits of the online input. This finding aligns with Green et al.’s (2018) conclusion that online learning content can benefit students across different ability levels.

Nevertheless, as the participants of this study were all recruited from CEFR intermediate levels – pre-intermediate (A2), intermediate (B1), and upper-intermediate (B2), the generalisability of the findings is limited to this group of EAP learners. It remains uncertain whether the benefits of the online learning activities are similar for learners at CEFR low and advanced levels. According to Tao et al. (2024), with the implementation of the blended EFL learning mode, students at advanced language proficiency levels (with CET4 scores above 550) showed less improvement in language performance than those at intermediate or lower levels (with CET4 scores below 512). One reason for this finding could be because their investigated blended mode focused on addressing basic learning issues and had limited effects on deep knowledge processing and transfer. In contrast, the blended mode examined in the current study incorporated both basic (e.g., video content comprehension) and deep (e.g., analysing and evaluating writing and speaking samples) knowledge learning. Moreover, as the two language proficiency criteria – CEFR and CET – lack alignment, the findings of Tao et al.’s (2024) study are not directly applicable to the current research context. Therefore, the generalisability of the findings to CEFR learners at low and advanced levels remains to be verified in future research.

5.2 Online engagement and language performance

Multiple linear regression results confirmed a positive, medium-sized impact of online cognitive and behavioural engagement on EAP learners’ language performance, indicating the overall effectiveness of the pre-seminar online course design. This finding further supports Sadiq’s (2022) conclusion that online quizzes and asynchronous video lessons statistically significantly improved students’ English proficiency. In addition, simple linear regression results indicated that the corresponding online quizzes significantly contributed to EAP learners’ performance in reading, listening, and writing assessments, but not in speaking. The statistical insignificance of the speaking quizzes may be attributed to their closed-ended format, which did not require learners to produce spoken output, partly due to the challenges of providing timely teacher feedback. This design limited opportunities for speaking practice and constructive feedback on speaking performance, leading to ineffectiveness in enhancing final EAP speaking outcomes. This implies the need to employ other forms of online modalities, such as automated speaking performance evaluation tools, to improve the effectiveness of online speaking activities.

Interestingly, simple linear regression results revealed that online lesson completion rates positively contributed to EAP learners’ performance in writing and speaking assessments but not in reading and listening. This highlights the effectiveness of the speaking video lessons despite the limitations of the speaking quizzes. The findings reveal the varied effectiveness of the online multimodalities on learners’ language performance, warranting a more thoughtful design when mixing the modalities in the online learning activities. As Joosten et al. (2019) suggested, the leanness (e.g., texts) or richness (e.g., videos) of the media should be carefully considered and appropriately selected for the content being delivered, allowing adequate breadth and depth for learning. These findings might also explain why past studies (e.g., Liu et al. 2020; Tao et al. 2024; Zhou 2018) reached incongruous conclusions on the effectiveness of the blended mode, possibly due to the divergent online course designs.

5.3 Learners’ perceptions of the effectiveness of online learning activities

The interview findings shed light on the EAP learners’ perceptions of the effectiveness of the online activities. The findings echoed the three components of learner-content, learner-socialiser, and learner-interface interactions in Wang’s (2008) generic model of integrating online content. In terms of learner-content interaction, the interviewees positively commented on the usefulness and relevance of the writing and speaking video lessons, as well as the opportunities to use the online quizzes to practise their reading and listening skills. For both videos and quizzes, they suggested shorter lengths to potentially improve their willingness to engage with the online activities. As Joosten et al. (2019) noted, online content design was positively associated with students’ academic performance and their perceptions of knowledge acquired.

As for learner-socialiser interaction, students expressed a preference for teachers who provided detailed and engaging explanations in the asynchronous videos, rather than simply reading from PPTs. This interaction was perceived as a way for teachers to establish a stronger virtual presence, which helped simulate a sense of connection and guidance despite the lack of real-time communication. Moreover, learners favoured interactive videos, where the videos paused at key points and required them to answer instant questions via pop-up windows. These features were seen as opportunities for students to engage in active reflection and self-assessment, which helped replicate elements of learner-teacher interaction found in face-to-face communication. Additionally, learners expressed a desire for a more effective feedback mechanism in the online quizzes. For instance, providing automated feedback with detailed explanations for both correct and incorrect answers could support learners in understanding their performance, mirroring the clarifications typically offered in real-time teacher interactions. Although indirectly related to academic performance, Amoush and Mizher’s (2023) study revealed that student-instructor interaction was one of the strongest significant predictors of student satisfaction with online English courses.

Thirdly, the interviewees’ suggestions for making technological improvements to ease their challenges with completing online quizzes highlight the importance of enhancing learner-interface interaction. As Amoush and Mizher (2023) pointed out, interaction with technology was found to statistically significantly predict learners’ satisfaction with online English courses. In addition, Kintu et al. (2017) found that technology quality significantly influenced learners’ perceived ability to acquire knowledge and their intrinsic motivation to learn independently through online activities in blended learning. These findings emphasise the critical role of learner-interface interaction in designing effective online activities for blended courses.

Yet, beyond the considerations of learner-content, learner-socialiser, and learner-interface interactions, the importance of connecting these activities with onsite lessons and maintaining the teacher presence in traditional classrooms cannot be overlooked. Building on the fundamental principles of Wang’s (2008) model and based on the findings of this study, an effective blended EAP learning model (see Figure 2) is proposed below. This model has the potential to optimise the use of various online multimodalities, enhance EAP learners’ language performance, and ultimately achieve the desired learning outcomes.

Figure 2:

An effective blended EAP learning model.