Critiquing research published in SSCI and ESCI CALL journals between 2023 and 2024: a research design perspective

2.1.1 Selected journals

Selecting journals was the first step. To this end, the inclusion and exclusion criteria were established and applied (see Table 1).

Only journals that meet all the above criteria were selected. The search resulted in five journals (see Table 2). All of them have CALL in their titles except Language Learning & Technology (LL&T) which is counted in as a CALL journal in Choubsaz et al. (2024) and Mohsen et al. (2024).

2.1.2 Selected articles

After journal selection, the next step was to select articles published in the sample journals. These articles had to be analyzed manually piece by piece because no software is available to process data for the purpose of this study. To avoid an excessive workload, the scope of selection was limited within a two-year timeframe between 2023 and 2024. The inclusion and exclusion criteria were then established and applied (see Table 3).

The final sample consists of 338 articles (see Table 4). The articles included were then categorized using the three-level research framework widely adopted in the field of ODDE (Zawacki-Richter 2009; Zawacki-Richter and Anderson 2014). According to this framework, macro-level research covers aspects of education systems and theories, including access, equity and ethics; globalization; teaching/delivery systems and institutional partnership in developing cross-border programs; theories/theoretical research; research and knowledge transfer (including literature reviews). Meso-level research mainly refers to institution-level management, organization, and technology use/adoption (hence including technology/media review). Micro-level research focuses on learning and teaching practices, mainly with students and teachers as subjects of investigation. Table 4 displays the distribution of these three levels of research.

Included in the 338 articles selected are 282 articles reporting on empirical studies and 56 non-empirical research articles, namely theoretical/conceptual articles (macro-level category), literature reviews (macro-level category) and technology/media reviews (meso-level category).

4.1.1 Clear statements of context and purpose of study

Education is contextualized and influenced by contextual factors. Therefore, educational research should also be situated. Previous reviews tend to focus on the types of context in which CALL studies are situated (e.g., DeMolder et al. 2023; Klímová and Seraj 2023; Zhang et al. 2023). The current study differs from previous ones in that it centers on whether sufficient contextual information is provided which is key to the interpretation of the findings and to the replication of a study (Shortt et al. 2023). Researchers should have a clear idea of the context of study at the very beginning and state relevant details clearly in their publication. Clear description of the context of study is a strength in the empirical studies reviewed, with only 1.8 % less clear or descriptive.

Unlike Shortt et al. (2023) that identifies types of research questions (i.e. performance-, attitude and motivation-, or design-oriented) used in their samples, our study focuses on the framing to see whether they embed researcher bias, preconception or assumption which may mislead the research process, including data collection, analysis and interpretation. The purpose, research questions or hypotheses “provide critical information to readers about the direction of a research study…also raise questions that the research will answer through the data collection process” (Creswell 2012, p. 109), consequently defining “the most appropriate participants, source of data, and method of data analysis for the study” (Peng and Xiao 2022, p. 10). For example, Luo and Watts (2024) aim “to explore the nature of smartphone-assisted ELL…” (p. 614), hence requiring that the participants had smartphones and used them to learn English. However, the article does not provide this information. All the empirical research articles in our review state their purpose of study or research questions/hypotheses although 22 % of them are inadequate in that they are somewhat biased. For example, “How much does the use of Google Assistant significantly improve the adolescent EFL learners’ oral proficiency?” (Tai 2024, p. 1286) – this research question suggests that the use of Google Assistant has a positive effect on adolescent EFL learners’ oral proficiency. Nevertheless, this does not make sense in that the purpose of the study was to find out whether Google Assistant was conducive to adolescent learners’ EFL. With this preconception or assumption in mind, the researcher may not have been able to maintain neutrality in collecting data and interpreting the findings.

4.1.2 Increasing adoption of mixed-method approach

Another strength is the popularity of mixed-method approach studies representing nearly half of the empirical research samples, in comparison with purely quantitative or qualitative ones. Hubbard (2009) aptly observes that “although quantitative studies probably dominated in the early literature, qualitative and mixed-method studies are now common, especially in the area of computer mediated communication (CMC)” (p. 5). Our finding reinforces findings from other reviews (e.g., Lee 2023; Peterson 2023; Shadiev and Yu 2024; Shi and Aryadoust 2024). The proportions of different research approaches may vary a little bit in some reviews (e.g., Kim and Namkung 2024; Zhang and Sun 2023; Zheng et al. 2022), which may be due to the difference in the activities/tasks/interventions involved, the technologies used, or the specific topic reviewed, in other words, the subfields that they focus on. Our sample empirical studies are not focused on any particular subfield, hence more representative.

4.1.3 Substantial proportion of experimental studies

Study design is also a strength, with nearly half of the empirical studies (48.6 %) experimental in nature, far more than those in Loncar et al. (2023) (21 %), a review of technology-mediated feedback for L2 English writing literature and Shi and Aryadoust (2024) (17.6 %), a review of AI-based automated written feedback research. Increase in experimental studies may be an emerging feature of the CALL research landscape because, as pointed out above, our review is more comprehensive instead of focusing on a particular subfield. Of the experimental studies in our samples, nearly two-thirds (63.5 %) are quasi-experiments. Given the complexity of educational research, true experiments may not always be feasible, especially when taking logistical challenges and ethical consequences into consideration. Researchers are therefore justified in exercising certain degree of freedom or discretion in deciding how a project is conducted (van Drimmelen et al. 2024). In this sense, it is understandable that there are more quasi-experiments than true experiments.

4.1.4 Significant proportion of theoretically-informed studies

Unlike Shadiev and Yu (2024) that investigate the theory, hypothesis, model, or framework upon which their sample studies are based, our review focuses on whether a study is underpinned by any theory, hypothesis, model, or framework. Lack of theoretical underpinnings is a common feature of educational technology research in general (Bond et al. 2019; Bulfin et al. 2014; Prinsloo 2018). A review of three top educational technology journals Computers & Education; Learning, Media and Technology, and British Journal of Educational Technology shows that 40 % of their publications are atheoretical (Hew et al. 2019), consistent with the findings of a review of literature on personalization in educational technology between 1960 and 2015 (Bartolomé et al. 2018). A review of research on AI applications in higher education also reveals a weak connection of their sample studies to theoretical pedagogical perspectives possibly due to “the low presence of authors affiliated with Education Departments” (Zawacki-Richter et al. 2019, p. 22). In comparison, two-thirds of the empirical studies in our review (67.4 %) are underpinned by relevant theories possibly due to the higher percentage of authors affiliated to education-related departments (86.4 %). This is no doubt a positive trend. Nevertheless, more needs to be done given that one-third are still atheoretical, as in the case of Shadiev and Yu (2024), and especially in view of a longitudinal analysis of highly cited articles published between 1983 and 2019 in four CALL journals, according to which 18 % of the seminal studies “did not explicitly adopt any theory to frame their research” (Choubsaz et al. 2024, p. 49).

4.1.5 Frequent data triangulation and clear statements of data collection and analysis

The sources of data are twice the sample studies (575 vs. 282) because two-thirds of the empirical studies employ triangulation, which can definitely contribute to validity and reliability. In addition, only 6 % of the empirical research articles do not explain their data collection process clearly to enable readers to judge the rigorousness of data collection and facilitate replicability while about 20 % are not articulate enough about data analysis. Overall, data triangulation, collection and analysis can count as a strength. Having said that, like data collection, descriptions of data analysis should be as clear and specific as possible to enable readers to assess the trustworthiness of the findings and replicate the study if they so wish.

4.1.6 Acknowledgement of limitations

Finally, over ninety percent of the empirical research articles acknowledge the limitations of their studies. No doubt, this is a strength. No study is immune from limitations (Creswell 2012). Reflecting on the limitations of a study can help contextualize the interpretation of its findings and serve as directions of future research not only for the author(s) but also for other researchers. This process should not be taken as a procedural requirement of academic publication with perfunctory acknowledgement. A good case in point is Roy (2024) which, together with the easily identified limitations, points out that other possible and yet less easily spotted weaknesses “such as teacher quality, student motivation, or classroom environment may have influenced the observed differences between groups” and that “technical issues, such as occasional problems with the DST method, may impact the overall experience and results” (p. 15) (also see Kourtali and Borges 2024). It should be noted, though, that the limitations stated in some articles seem to be superficial, only mentioning those that readers can easily identify such as small sample size, short duration, or no triangulation.

4.2.1 Predominance of micro-level research

The current landscape is dominated by micro-level research typically exemplified by “This is what I did to my course or to my students”. Micro-level research constitutes the foundation of the field. However, learning and teaching is affected by factors not only at the micro level but also at the meso and macro levels, for example, socio-economic development, socio-cultural context, institutional infrastructure, cost and finance, quality evaluation and assurance policy and mechanism, educational resources as well as support for both faculty and students (Xiao 2023a). Research into these macro- and meso-level issues is essential to the dissemination or generalization of findings from micro-level research, hence the sustainable development of CALL. Meso and macro-level research represents only about one quarter of all the articles in our review, with the former focusing on technology trends and design, development, testing or validation and the latter falling within the areas of theoretical research, research and knowledge transfer (literature review). This lack of interest in meso and macro-level issues may be due to the high percentage (86.4 %) of authors with learning and teaching or education-related backgrounds, namely CALL practitioners. Researchers need to explore and understand the big picture of CALL rather than confine research to issues directly related to micro-level practice.

4.2.2 Scarcity of replication studies

Replication studies are “critical to the growth and credibility of our discipline” (McManus 2024, p. 1), refining findings, contributing to generalizability, and strengthening conclusions (Gass et al. 2021). It is “a sign of a field’s maturity” (Smith and Schulze 2013, p. i). However, only 6 % of our empirical samples replicate previous studies, a situation that warrants attention and needs redressing. This is an obvious weakness in that there is no knowing “if the results hold for a different population, in a different setting, or for a different modality” (Polio and Gass 1997, p. 502). A decade ago, CALICO Journal published a special issue on replication and evaluation in CALL, hoping that this would be “the beginning of many replication studies” to be published in this journal (Bikowski and Schulze 2015). Despite the editors’ promise that they will be “providing a venue for CALL scholars to disseminate their replication research” (Smith and Schulze 2013, p. ii) and the consensus on the imperative of replication research in CALL, this kind of research remains scarce (Tschichold 2023). Difficulties in conducting replication studies, in particular exact replication (Chun 2012; Foung and Kohnke 2023; Tschichold 2023) cannot justify the scarcity. More needs to be done.

4.2.3 Insensitivity to researcher biases

Another apparent weakness is that over 90 % of the empirical research articles do not acknowledge potential researcher biases and explain what measures the researchers have taken to prevent possible interference of these biases in the research process. The researcher’s neutrality in the research process, researcher-participant relationship, and interpretation of data and findings may suffer consequences because of these biases (Werth and Williams 2021). Researchers’ awareness of and actions to minimize own subjectivity cannot be overemphasized (Jung 2025). Negligence in this regard is likely to affect various stages of the research process, hence undermining the trustworthiness of the research outcomes. This seems to be a weakness of the research landscape of technology-enhanced education in general (Peng and Xiao 2022; Zhang et al. 2023).

4.2.4 Overuse of non-probabilistic, convenience sampling

Sampling is also found to be a weakness of the empirical studies reviewed in our study, echoing Ballance’s (2024) conclusion that “little attention is given to sampling” (p. 58). Only about one-fifth name their sampling strategies. Furthermore, of all the 282 empirical studies reviewed, only 2.8 % (n = 8) employ probabilistic sampling strategies, mostly random sampling while the overwhelming majority apply non-probabilistic sampling strategies, with three quarters of them convenience sampling. The dominance of non-probabilistic strategies, especially convenience sampling may undermine the reliability and validity of the findings (Ballance 2024; Vehovar et al. 2016). Regrettably, it is a phenomenon commonly found in other reviews (e.g. Shortt et al. 2023). While acknowledging the difficulties that may arise from other sampling strategies, in particular probabilistic sampling, we should not allow any particular sampling technique, especially non-probabilistic technique to prevail. Sampling matters to empirical studies because it affects whether “the effect of unknown sources of variance” can be neutralized (Ballance 2024, p. 64).

4.2.5 Shortage of longitudinal research

The shortage of longitudinal studies is a weakness, with only 20 % collecting their data more than one time. This also features in other reviews (e.g., Soyoof et al. 2023). As argued by Kim and Namkung (2024), “more longitudinal projects are warranted to understand learner perception dynamics … over time”(p. 17) in order to “provide more reliable and/or generalizable results” (Ballance 2024, p. 60), an argument supported by Xiao et al. (2024). “The effectiveness of an educational intervention needs to be tested over time” to overcome novelty effect (Xiao 2023a, p. 215). Reeves and Lin (2020) suggest a new direction of research which may be equally applicable to the field of CALL “whereby we develop robust, multi-year research agendas focused on important problems and innovative solutions, judge our worthiness for promotion and tenure on evidence of impact rather than simple article counts, closely collaborate with practitioners, and establish our field as preeminent in meeting global problems related to education” (p. 1999). As is the case with other weaknesses, while there are challenges with longitudinal empirical research, for example, time commitments, funding, and sample stability, among other things (Barkhuizen 2009; Jenkins et al. 2011), more longitudinal studies are needed to consolidate the knowledgebase of CALL.

4.2.6 Undue reliance on self-reported data and unverified and/or unavailable instruments

Another weakness relates to the undue reliance on self-reported data and unverified and/or unavailable instruments. Of the 19 sources identified, survey (questionnaire/scale/rubric and interview) and test are the most frequent sources, far outnumbering the other sources, similar to Kim and Namkung (2024) and Zheng et al. (2022). Questionnaire and interview are also found to be the most popular data sources in Shadiev and Yu (2024). On the other hand, although our sample empirical studies often use more than one instrument to collect data from survey, survey data is self-reported in nature and may not reflect the participants’ true feelings, ideas, or conditions, due to various factors such as social desirability bias. This is especially the case when only about 10 % of the instruments are both reviewed and piloted or statistically tested before they are applied in the current studies and 47.2 % are unavailable in their entirety. This situation may jeopardize the quality of the data obtained as well as research replicability. Instrument availability in its entirety can also enable readers to assess whether the instrument is fit for purpose or whether it is biased. For example, Yang et al. (2023) claims to use the Chinese version of the Foreign Language Classroom Anxiety Scale (FLCAS) by Horwitz et al. (1986) in their study but goes on to say “the revised FLCAS consisted of 28 items” (p. 1594). No Chinese version is available; no explanation is given as to what revisions had been made and why. Therefore, readers cannot possibly assess the suitability of this instrument. In contrast, some interview questions used in Hwang et al. (2024) may be misleading such as “Do you like to use Smart UEnglish to practice English? Why?” and “Which function of Smart UEnglish/UEnglish do you like most? Why?” (p. 1644). These two questions imply that the interviewees liked to use Smart UEnglish/UEnglish. The first question is not suitable especially for the interviewees from the control group in that the UEnglish they used was without smart mechanisms. Equally inappropriate for the control group interviewees are two other questions – “Do you want to keep using Smart UEnglish to practice English? Why?” and “Do you think using Smart UEnglish is helpful to your English? Why?” (Hwang et al. 2024, p. 1644). The instruments applied in both studies were used without passing through the stage of review and piloting.

Research instruments need to be carefully designed and rigorously reviewed and piloted before being used and readers should be able to access full contents conveniently. On the other hand, many of the less frequent sources may provide valuable data about pedagogical and psychological change in language learning, in particular those that reflect the participants’ actual performance or mental activities. In other words, more studies using an interpretative and naturalist approach to data should be encouraged (Zhang et al. 2023).

4.2.7 Insufficient attention to ethical issues

Given that 40 % of the empirical research articles do not include ethical statements, this also constitutes a weakness. Ethical issues arise “in all research designs involving human respondents owing to an intrinsic tension between the needs of the researcher to collect personal data on which to base generalizations and the rights of the participants to maintain their dignity and privacy”, in particular when there is a fiduciary relationship between the researcher(s) and the participants, for example, teacher-student relationship (Ferguson et al. 2004, p. 57). Therefore, ethical concerns, “unless properly addressed, may lead to resistance from participants and consequently data inaccuracy” (Peng and Xiao 2022, p. 12). On the other hand, although 60 % of our samples explain how ethical issues are addressed, the measures taken may not be adequate in all cases. For example, simply obtaining informed consent from participants “who are in dependent or restricted relationships with the researcher” is far from acceptable because there may be “a coercive element” in it (Ferguson et al. 2004, p. 58). This is especially the case with those participants who are minors and may be more vulnerable to pressure from the researchers or do not really know what they consent to. In the so-called intelligent age, informed consent often becomes “the pretext to justify the misuse of educational data” in particular when “people are forced to give their consent if they want to use” a smart app, software or device (Xiao et al. 2025, p. 6).

Equally insufficient are measures such as material or monetary rewards or giving extra score as a bonus. For example, in a study investigating the effects of individual versus collaborative processing of ChatGPT-generated assessment feedback, “to maintain students’ engagement and motivation in the project, the rated writing products were used as assignments for the grading of the course Intermediate English Writing, a compulsory course for all sophomore EFL students at the university. Outstanding performance in seeking, processing, and using ChatGPT-generated AF was awarded bonus marks” (Yan 2024, p. 5). This not only reflects the researcher’s stance/bias on the assumed positive impact of ChatGPT on student learning, exacerbating social desirability bias but is also ethically controversial because the non-participants were not fairly treated. An excellent example of how ethical concerns are addressed is Jensen (2024), a case study of an 11-year-old girl. In addition to registering the study with the relevant department of the home university and following the university’s ethical guidelines, the researcher sought informed consent from the girl’s parents and informed assent from the participant herself. Further, the participant and other people involved were anonymized. Another good example is Roy (2024) whose participants were 8th-grade students. In other words, measures to handle ethical issues should be specific and relevant to the participants of a study.

4.2.8 Over-favoring case study and survey research in non-experimental research

Case study (46.9 %) and survey research (29.7 %) account for nearly 80 % of non-experimental design studies, with the remaining nine designs combined representing less than a quarter (see Table 6). This is also a weakness. As argued in Peng and Xiao (2022), educational research may benefit from “designs such as action research, design-based research, narrative research, grounded theory research, phenomenological research” (p. 11). Given the longitudinal and situated nature of language learning, innovations are needed in CALL research design. A mixture of designs in a study may be a worthwhile direction for CALL research. For example, both narrative and observational research designs may be adopted in a study and so are case study and phenomenological research designs; experimental designs may be embedded into a design-based research study.

4.2.9 Use of ambiguous units of measurement

Finally, when it comes to duration of intervention/treatment, the greatest problem is not that a significant percentage of empirical studies do not indicate time span as in the case of Shi and Aryadoust (2024) which finds that 42.9 % of the studies on AI-based automated written feedback research fail to specify the duration of their intervention/treatment. There are only 8.7 % of studies in our review which do not detail how long the intervention/treatment lasted. The biggest problem with our samples is the use of units of measurement which may be context-sensitive but not further specified. For example, how long is a semester, course, module, class, session, and meeting? As pointed out by Peng and Xiao (2022), the lengths of these units may vary “a great deal across different countries or educational institutions, and even within an institution” (p. 7). Specification of the duration is essential to enhancing replicability of a study.