Opening the methodological black box of research synthesis in language education: where are we now and where are we heading?

1.1 Qualitative research synthesis and qualitative synthesis of research

In addition to meta-analysis, and its focus on the synthesis of quantitative data, the discipline of research synthesis has developed to incorporate alternative approaches to examining the totality of evidence in a given field. It is, therefore, important to note that research synthesis “transcends the narrower domain of meta-analysis and includes other quantitative as well as qualitative methods” (Norris and Ortega 2010, p. 461). In particular, as the field has developed we have begun to see types of research syntheses associated with the qualitative paradigm. We term these ‘qualitative research syntheses’ (research synthesis that employs qualitative methods to synthesise qualitative research evidence) and ‘qualitative synthesis of research’ (research synthesis that employs qualitative methods to synthesise research evidence derived from both qualitative and quantitative paradigms).

As with most types of syntheses, syntheses of qualitative evidence usually follow standard methodological stages: design research questions, identify keywords for literature search, conduct literature search, assess literature for eligibility using pre-specified inclusion and exclusion criteria, assess the quality of included research, extract data, synthesise data, and report the findings of that synthesis (Chong and Plonsky 2021). Qualitative syntheses differ from meta-analyses most substantially in the way that data are extracted and synthesised (Chong and Reinders 2021). Whereas meta-analyses are principally concerned with extracting summary data in the form of numbers, such as effect sizes, qualitative data extraction refers to extracting information in the form of words that are relevant to the aim of the synthesis. This process is similar to line-by-line coding or initial coding in primary studies that employ qualitative methods. Syntheses of the extracted data then involves the sense-making process using that information. The outcome of such a process is the development of higher-order themes based on the codes developed during data extraction. Data synthesis is reminiscent of focused and axial coding in qualitative studies that employ grounded theory (Charmaz 2006). In what follows, we outline three types of less common research synthesis in applied linguistics: scoping reviews, systematic literature reviews, and qualitative research syntheses.

Scoping reviews are a type of exploratory review that aims to survey the landscape of an emerging or existing field of research; sometimes, a scoping review is conducted to gauge the feasibility or necessity for conducting a larger-scale systematic review (Pham et al. 2014). Adhering to its exploratory nature, research questions of scoping reviews are generally broader; for example: ‘What is the extent and nature of research on X?’ (see Chong and Reinders (2022) for examples relating to conceptualisation, operationalisation, and evaluation in computer-assisted language learning (CALL) research). The search strategy of scoping reviews prioritises inclusivity because the primary objective of a scoping review is to map out the evidence base of a field of enquiry, assess the extent of the available evidence, organise it into groups, and highlight gaps. In this case, standards used to appraise studies to be included in scoping reviews are usually less stringent than those employed in a systematic review or a qualitative research synthesis.

Systematic literature reviews can have a substantive or methodological focus. Compared to those in scoping reviews, research questions of systematic literature reviews are usually more specific and more focussed. A systematic literature review is expansive in scope, aiming to provide a thorough state-of-the-art review on a topic. Reflecting the greater scope of this type of research, Language Teaching journal, for example, expects reviews that it publishes to have approximately 100 references and increases the word allowance to 20,000, considerably more than the typical word limit for reports of primary research (Language Teaching 2022). Despite having a broad focus, systematic literature reviews require a more rigorous evaluation of the quality of included studies. Typically, constituent studies will be evaluated for methodological rigour and susceptibility to bias, using tools like Cochrane Risk of Bias tool for experiments (Sterne et al. 2019), Quality in Qualitative Evaluation for qualitative designs (Spencer et al. 2003), the Newcastle Ottawa Scale for observational designs (Wells et al. 2021), the Mixed Methods Appraisal Tool for reviews that include a variety of designs (Hong et al. 2018), and AXIS for cross-sectional designs (Downes et al. 2016). This means that, in addition to reporting the overall findings of a body of research, systematic literature reviews provide an estimate of trustworthiness, both in terms of individual studies and of the literature as a whole.

Qualitative research synthesis differs from scoping and systematic literature reviews because it summarises only qualitative research evidence. Published qualitative research syntheses in applied linguistics focus on research that is wholly qualitative in nature or by extracting only qualitative elements of mixed methods research (e.g., Chen 2016; Chong and Reinders 2020; Çiftçi and Savaş 2018). Following the qualitative research paradigm, qualitative research synthesis is underpinned by an interpretivist epistemological view, aiming to capture the richness and diversity of beliefs and experiences of social phenomena such as language learning and teaching. To represent the complex relationship among various themes, synthesised qualitative evidence needs to be presented in a way that highlights connections. This can be done through a conceptual mind map that accompanies a written report of themes. The qualitative research synthesis by Çiftçi and Savaş (2018) on telecommunication provides an excellent example of such conceptual mind map (p. 291).

For a detailed analysis of these three (and other types of synthesis) particular to the field of applied linguistics, see Chong and Plonsky (this issue).

2.1 Further development of the field

Compiling this special issue has not only allowed us to take stock of where we are with research synthesis in applied linguistic, but it has given us the opportunity to consider related aspects of the evolution of the field, allowing us to consider where we want to go next. To conclude this editorial, we will highlight some of the areas where we see positive movement related to research synthesis and which we expect to see reflected in related scholarship: open science in applied linguistics, closing the research-practice gap, and the growing recognition of the methodological requirements of evidence synthesis.

2.2 Applied linguistics and open science

Open science, or open scholarship, refers to practices that aim to “make multilingual scientific knowledge openly available, accessible, and reusable for everyone” (UNESCO 2021, p. 6). In addition to the prospective and open publication of systematic review protocols on IDESR.org (see Chalmers, Brown and Koryakina, this issue), a variety of other open science practices are gaining traction in our field. These include publishing materials and data on IRIS-database.org, creating textual and visual summaries of publications for audiences outside of academia (OASIS-database.org and TESOLgraphics.com), and translating abstracts into languages other than English (Multilingual Repository for Abstracts in Applied Linguistics – multilingualrepository.org ). Recently, the “Open Applied Linguistics” Research Network, affiliated with the International Association of Applied Linguistics (AILA), was established. The conveners held an online symposium on open scholarship in applied linguistics (Meng et al. 2022), following which a statement on open scholarship in applied linguistics, endorsed by AILA, was released and (Meng and Chong 2022). This statement identified areas where resources for facilitating open science in applied linguistics are growing, and urged take up of these resources by researchers and practitioners. Open science, and in particular open data, is highly relevant to research synthesis because making raw data and research materials publicly available is crucial for synthesists to conduct high-quality reviews. It allows synthesists to examine the quality of the data collected and its analysis, thus helping to inform decisions of whether to include certain studies in reviews and facilitating more complete statistical syntheses. Additionally, open data enables the re-analysis of data by other synthesists. At present, however, the sharing of data does not appear to be common practice in applied linguistics research. For example, a meta-analysis conducted by Plonsky et al. (2015) reported that among 255 authors who were contacted to request the datasets generated by their research, the majority did not respond (n = 159). Of those that did, only 36 datasets were provided, of which only 25 were usable. If we strive to improve the quality of research synthesis in our field, in addition to rigorous methodology, synthesists need to be able to access usable raw data. Encouraging primary researchers to see the value in making their data publicly available via repositories like the IRIS database would be a positive first step.

2.3 Closing the research-practice gap

Researchers in our field have started to advocate the need for dialogues between researchers and practitioners. Sato and Loewen (2019) wisely urged researchers to “widen the door and increase the amount of dialogue” (p. 9). This call to bridge the research-practice chasm points to the ongoing problem that educational research evidence is rarely used by teachers to inform their practice (see e.g., Bond et al. 2019). The reasons are manyfold. Medgyes (2017), for instance, contended that there is distrust between researchers and practitioners. Isaacs and Chalmers (2023) note the lack of involvement by teachers in setting research agendas. That is, teachers are not routinely asked what kind of research they need to inform their practices, which risks researchers addressing questions that are not of practical importance to teachers. Another reason relates to teachers’ lack of access to language education journals. Although many teachers are interested in improving their practice on the basis of research evidence, they do not necessarily know where to locate credible research, and if they do, they are usually barred from doing so by journal paywalls (Sato and Loewen 2019). Even if they do have access, many report that it is a luxury for them to read academic texts given their heavy teaching and administrative workload (ibid).

In response to the above, Chong (2020, 2022) argued that research synthesis has a vital role to play in bridging the research-practice gap. As research syntheses summarise the whole body of evidence from primary studies on topics of interest to teachers, it is less time-consuming for teachers to read and provides the overview necessary to interpret the extent of our knowledge on a given topic, rather than being misled by selective citation of single studies in, for example, the popular press. Moreover, research syntheses are more substantial pieces that (usually) involve the work of a team of researchers, and they are often supported by research funding. As such, research syntheses are more likely to be open access, allowing teachers to access research syntheses without paying a subscription fee. Lastly, unlike primary studies, synthesised findings are often presented visually, in tables and diagrams, collating information in a more reader-friendly manner. One example is Hung et al.’s (2018) scoping review of research on digital game-based language learning where the authors summarised the various types of digital games documented in research in a table, including their definitions and relevant primary studies. From the perspective of language teachers who may be unfamiliar with the literature base, accessible summaries like this are extremely helpful because they provide a bird’s-eye view of the resources available to teachers and a summary of the research that has evaluated them.

With researchers’ growing interest in establishing connections with practitioners, we urge synthesists to include concrete and evidence-informed recommendations for teachers and policymakers. Preferably, such information can be presented in a way that encourages views from colleagues in the profession, such as using various visual representation of the synthesised data. To give an example, TESOLgraphics (Chong and Sato 2022) is an initiative that creates one-page, infographic summaries of research syntheses for English teachers. Research syntheses were summarised by teacher members of the project team, with the intention to incorporate teachers’ perspectives (e.g., what teachers view as important when reading a research synthesis; what practical insights teachers can get from reading the research synthesis) into the infographic.

2.4 Growing recognition of methodological requirements

Research synthesis has been increasingly recognised by journals in applied linguistics – not only for its inherent value, but for its differences with primary studies. One such difference is length. Research syntheses, which summarise multiple primary studies in a systematic and rigorous manner, usually result in pieces much longer than the 7,000 to 8,000 words typical of published research. Journals such as Language Learning and Language Teaching have long been at the forefront of publishing research syntheses in our field, with dedicated sections that allow pieces that exceed 10,000 words. What is exciting is that more journals have started to include dedicated sections for research syntheses, including recently, for example, Studies in Second Language Acquisition and Language Testing. A new journal, Research Methods in Applied Linguistics published by Elsevier, also includes research synthesis (“reviews”) as one of the article types, providing a more generous word limit of 10,000 words.