Data collection methods applied in studies in the journal Intercultural Pragmatics (2004–2020): a scientometric survey and mixed corpus study

2.1 On observing and extracting data

Traditionally, most sociolinguistic and pragmatic scholars would agree on naturally occurring discourse (NOD) – due to its untainted, i.e. non-elicited nature – being the non-plus-ultra kind of data desirable for empirical research (Bataller and Shively 2011; Bou-Franch and Lorenzo-Dus 2008; Félix-Brasdefer 2007; Golato 2017; Martínez-Flor 2006; Turnbull 2001; Sampietro et al. 2022). NOD represents the kind of data one will usually obtain through external, non-biasing observation of subjects and communicative scenarios (thus, “field method”; cf. Clark and Bangerter 2004). It is data that “has not been elicited by the researcher for the purpose of his or her research project but that occurs for communicative reasons outside of the research project for which it is used” (Jucker 2009: 1615).

Certain drawbacks have recurrently been raised when it comes to collecting data through observation – one of them being the “lack of control of speaker and context variables” (cf. Houck and Gass 1996: 47; also cf. Blum-Kulka et al. 1989b: 13). What is more, various factors need to be considered determining the validity (Golato 2017: 24; Yuan 2001) and, overall, the reliability of observed data (cf. McKay and Hornberger 2005, pp. 391–392; Leung et al. 2004: 242).

Apart from methods of observation, the extraction of certain data is a commonly applied field method. Extracted data points may be represented just as much by quantifiable, large sets of materials (available as electronic corpora) as they can be by individual texts that the researcher is zooming in on. Compared to observed data, which is, in principle, authentic as well as naturally occurring, extracted data may range between various categorizations of ‘naturalness’. As Schneider puts it,

corpus data do not all qualify as observational data. They are naturally occurring to the extent that their existence does not depend on a researcher. Yet there are significant differences between the data types included in machine-readable corpora, sometimes even in the same corpus. (2018: 50)

I therefore find the distinction between ‘observed’ and ‘extracted’ data rather useful, even though it is not generally made (cf. Kirner-Ludwig 2022).

2.2 On eliciting data through productive and comprehensive tasks

Data elicitation refers to the obtaining of data through applying systematically controlled settings, prompts, and variables (cf. Clark and Bangerter 2004). The present study distinguishes between non-experimental versus experimental settings based on the added complexities that come with experimental elicitation (see Section 4.2.3). This notional decision is in line with the trending research foci in (Intercultural) Pragmatics and particularly in acknowledgement of the field of Experimental Pragmatics (cf. Meibauer and Steinbach 2011; Noveck 2018: ch. 4; Noveck and Sperber 2004). Non-experimental research designs usually share their primary focus on data reflecting subjects’ actual language use (cf. 4.2.2), while experimental setups are saliently concerned with the aim of eliciting speaker intentions and shedding light on cognitive processing. They seek to elicit such data through manipulation of certain factors and by significantly frequently adhering to psycholinguistic methods (cf. 4.2.3).

I follow Schneider (2018) in my fluid categorizing of elicitation tasks according to a low, medium or high interaction/collaboration level required. More specifically, he proposes that elicitation tasks should be viewed on a “continuum […] decreasing [in] interactionality and, at the same time, increasing [with regard to] researcher control” (2018: 58). Respectively, low-interaction tasks are such that require a very restricted interaction level in order to elicit productive data. On the most basic level, there are intuitive tasks that will ask the participant to provide personal information, such as age, gender, occupation etc. In questionnaires, these tend to come as single-choice items. A more sophisticated instrument to elicit productive intuitive and introspective data is the diary or verbal-report method, which requires participants to note down relevant anecdotes or self-observational aspects (cf. e.g. Cohen 1996; Kasper 2008: 297; Schneider 2018: 73). Open-ended questionnaires, too, may be used to elicit rich introspective data, but will usually require significantly longer and more complex coding phases and analytical procedures.

The most salient low-interaction task (both on the meta-as well as the application level) is the so-called discourse-completion task (DCT), which, typically, requires participants to produce (written or oral) utterances that (in the participant’s view) appropriately complete or complement a prompt provided by the researcher. This prompt generally describes a specific socioculturally embedded situation in lockstep with the beginning of a (seemingly) authentic dialogue. As Mey puts it, “[t]his method basically consists in creating a (written) ‘role play’ situation” (2004: 39). The situational context provided in the prompt is deliberately construed so as to elicit the specific (pragmatic) aspect aimed for (often without the participant being aware of it so to avoid bias).^[2]

In general, DCTs have been used saliently within the context of research on speech acts (cf. Blum-Kulka et al. 1989a), formulaic language (cf. e.g. Kecskés 2000), and pragmatic knowledge (cf. Félix-Brasdefer and Hasler-Barker 2017), with colleagues acknowledging this instrument’s time efficiency, potential of cross- and interdisciplinary replicability as well as the high level of variable control (cf. Kasper and Roever 2005; Houck and Gass 1996). Yet, DCTs’ significant shortcomings are repeatedly highlighted in the literature, too, concerning, for instance, the unavoidable factor of artificiality entailed in presenting short written segments that are actually prompted and analyzed as if they were oral (cf. Martínez-Flor and Usó-Juan 2011: 53; cf. Golato 2017: 22; Turnbull 2001; Bou-Franch and Lorenzo-Dus 2008; Félix-Brasdefer 2010; Economidou-Kogetsidis 2013; Schauer and Adolphs 2006). At the end of the day, there seems to be a consensus that, while DCT responses may not adequately reflect natural speech, they do “accurately reflect the content expressed in natural speech” as well as “the values [and norms] of the native culture” (Beebe and Cummings 1995: 75; cf. Kasper 2008: 329). This is why DCTs remain a popular go-to data collection method in many subareas of Pragmatics research, including ICUP.

Medium-interaction tasks cover most kinds of elicited conversations, i.e. include all such tasks in which “researchers specify topics, interactional goals or discourse roles” (Kasper 2008: 287). Interviews have been found to be the most frequent subtype of elicited talk in this category (Schneider 2018: 62). They are usually conducted orally with the researcher eliciting responses through verbal (sometimes in combination with visual) prompts^[3] that start and structure the conversation to certain extents. Thus, interviews may be fully or in part narrative (i.e. unstructured, open-ended), semi-structured or structured, formal or informal, while the conversation between interviewee and interviewer (researcher) is usually audio- or video-recorded (provided the interviewee’s informed consent).

The category of high-interaction tasks pertains to collaborative learning activities of various kinds employed to elicit productive data. These may range from collaborative writing or translation assignments, to video-conferencing sessions, group discussions and peer feedback tasks. Also role-play tasks (RPTs) belong with this highly collaborative group. They are used relatively saliently in L2 contexts (cf. e.g. Abdoola et al. 2017; Ross and Kasper 2013; Taguchi and Kim 2018; Youn 2020), where they have occasionally even been found to “yield more realistic data than other data elicitation methods” (Golato 2017: 22; cf. also Félix-Brasdefer 2007; Kasper and Dahl 1991; Turnbull 2001). However, RPTs inherently pose significant logistical challenges and will not guarantee the quality of elicited data to be authentic and useable, which is the main reason for scholars refraining from this method.

Within any of these low-to-high interaction spheres, research designs may additionally feature experimental elements so as to elicit both productive and receptive data. In fact, pragmatic methods have been shifting significantly into experimental spheres in recent years (cf. Noveck and Sperber 2004: 8; Schlesewsky 2009).

2.3 On scientometric approaches in (Intercultural) Pragmatics

With the present study taking a scientometric approach to the trends in data collection methods, there is a small set of studies that deserve specific attention for having paved this way. In general, scientometric studies carving out trends and developments in Pragmatics are still relatively rare in our discipline, but have been presented by Bardovi-Harlig (2010), Culpeper and Gillings (2019), Hu and Fan (2011), Jucker and Staley (2017), Kecskés and Kirner-Ludwig (2020), and Nguyen (2019). Kecskés and Kirner-Ludwig (2020) include a scientometric case study in an introductory chapter to a volume on new directions in Pragmatics research, adhering to the Web of Science Corpus. Nguyen’s handbook chapter (2019) presents a survey of methods in L2 Pragmatics research based on Bardovi-Harlig’s data published in 2010 (i.e. overall a survey of 246 empirical studies in L2 pragmatics published from 1979 to 2017), and employs database meta-information from LLIBA, ERIC, and ProQuest. Jucker and Staley (2017)’s survey presents figures specifically concerned with preferred data types in research on politeness and impoliteness and uses journal materials from the Journal of Pragmatics. Culpeper and Gillings (2019), too, extracted a corpus of 200 papers published in the Journal of Pragmatics and carved out data trends in Pragmatics research over the course of 1999–2018. Hu and Fan (2011) surveyed journal data (dating from 2001 to 2005) with regard to research contents and methods in intercultural communication research with a focus on China.

3.1 Browsing electronic corpus data for data collection methods

All upcoming elaborations follow the pretense that only empirical data can be analyzed. Thus, non-collected data (i.e. what Bednarek refers to as “non-attested data”, cf. 2011) is here used as a notion to refer to non-empirical data and solely introspective or intuitive lines of argumentation as opposed to systematically obtained data via empirical methods. The latter include observed, extracted and elicited data collection procedures, out of which the first two inherently aim for naturally occurring data, while all three pertain to (quasi-)authentic data (cf. also Al-Surmi 2012: 673f.; Bednarek 2018, 2010; Dynel 2015; Rose 2001).

The present study adheres to electronic corpus data in order to provide an extensive diachronic, scientometric survey of methods and data collection approaches within ICUP since 2004. My research design is similar to Culpeper and Gillings’ (2019), Hu and Fan’s (2011), and Nguyen’s (2019) in terms of data surveyed and coding schemes developed, but does exceed their dataset size significantly. Out of the 200 papers that Culpeper and Gillings (2019) extracted from the Journal of Pragmatics, they only discussed the top 50 most-cited ones from four designated time periods. Nguyen’s (2019) corpus was comprised of 246 studies, out of which 105 were borrowed from Bardovi-Harlig’s (2010) dataset covering studies published between 1979 and 2008.

In comparison, the corpus that I have compiled for the present chapter is composed of the entirety of papers published by Intercultural Pragmatics since the journal’s launch in 2004 and until the end of 2020, altogether adding up to 358 papers in 17 volumes (excluding book reviews) and to a total of 3,385,001 words. This corpus will henceforth be referred to by the acronym CICUP (i.e. ‘Corpus of Intercultural Pragmatics’). Each paper in full (including titles, keywords if provided, and references) was downloaded, individually labelled as P1 – P358,^[4] and compiled into one searchable corpus that was then uploaded and quantitatively analyzed on Sketch Engine via concordance searches. This corpus approach was combined with the application of four sequential rounds of extensive manual coding protocols to each individual paper.

3.2 Coding protocol^[5]

While most of my coding foci also feature in the studies by Bardovi-Harlig (2010), Culpeper and Gillings (2019), and Nguyen (2019), I take a more in-depth and comprehensive approach to my significantly larger dataset. After all, my coding decisions took into account both meta- and object-data points, the former of which were not systematically or straightforwardly identifiable through word and concordance searches alone. Therefore, coding steps 1 and 2 included decision-making procedures based on manual and close-reading approaches.

In a first step, each one of the 358 papers in the dataset was categorized (upon close reading) according to whether the study was based on intuitive/introspective, observational or elicited data. Note again that papers that took a solely intuitive or introspective approach, i.e. “did not analyze actual language data but work with reflections on language” (Jucker 2009: 1615), were not included in any of the following deliberations, but only served me as a big-picture orientation point: out of 358 papers in total, I categorized 138 papers (i.e. 38.5%) as introspective/intuitive studies and excluded them (cf. Figure 1),^[6] which left 220 papers (i.e. 61.5%) for me to analyze in detail. In the following, these 220 papers will serve as my focus dataset (i.e. as 100% of those studies using empirical data).

Figure 1:

Distribution of data types and modes in CICUP.

In a second round of coding, the 220 papers were browsed for whether they contained sections explicitly dedicated to describing their methods (rather than their methodology). Finally, the browse function on Sketch Engine was used to doublecheck the date for specific features, such as ‘authentic data’, ‘participant observer’. These hits had to be investigated one-by-one within their context so as to make sure that they were actually referring to the main data in the respective study (rather than, e.g., to literature reviewed).

In a third go, the 220 papers were coded according to the collection mode of data, i.e. as to whether the data collected was written or spoken. As Figure 1 forestalls, the relation is more or less balanced, with a slightly higher preference for spoken data in CICUP.

Note that, while extracted data will necessarily be obtained in a written or transcribed form, these data may have been spoken to begin with – consider any spoken electronic corpus consisting of interviews, or group discussions. Special in-between cases are posed by such text types that are commonly (semi-)scripted in order to be performed in spoken mode (e.g. TV news reports, public speeches, certain kinds of telecinematic discourse). I therefore did acknowledge the original mode of data in my coding scheme.

In a final round, all papers were coded according to subcategories such as dataset size, languages and cultures addressed, and any specific elicitation tasks employed. Many, yet not all, of these variables could be uncovered by way of concordance and key word searches. All of those databased papers that did not offer methodological descriptions in the conventional register expectable had to be reviewed yet again at this stage. Note also that additional spot tests were conducted throughout, as not all concordance results could simply be taken at face value. Even with keywords such as ‘DCT’ or ‘experimental’ featuring in a paper, it still needed to be confirmed that these characteristics or specifications were in fact pertaining to methods actually adhered to rather than merely mentioned as approaches excluded by the authors.

4.1 Observed and extracted data in CICUP

Following the notional distinction between extracted and observed data as elaborated upon in 2.1, the distributions in CICUP are shown in Figures 2a and b. Note that the absolute sum of numbers in Figure 2a exceeds the number of studies in the focus dataset (i.e. 220), as a number of studies employ more than one data collection method and data type.

Figure 2a.

Absolute number of data types chosen.

As can be seen, the data preferences are equally divided between non-experimentally elicited data on the one hand and extracted data on the other. Particularly the latter reflects the fact that, with computer-aided research emerging, extracting authentic data can nowadays be done in immensely efficient ways. This includes, for instance, access to already-transcribed conversational data.^[7] Extracted data points may be represented by quantifiable, large sets of materials (electronic corpora, lexicographical resources, cf. e.g. P204) just as much as by individual texts or smaller datasets that the researcher is zooming in on. These may include newspaper articles (cf. e.g. P235, P236), multimodal data from websites (cf. P352), lifestyle weblogs and online fora (P303), or specific text types like electronic self-reviews (cf. P188). Email conversations are generally considered highly private and will thus not be shared widely by their composers, which is why such authentic data is particularly hard to come by unless the researcher was a conversant themselves (but cf. P216, P240). As for e.g. Facebook posts, the data extracted will often include a researcher falling back on their personal account contacts (cf. e.g. P217). As shown in Figure 2b, those 106 studies that use extracted data, are merely balanced with regard to using large corpus data (not self-compiled, 48%) versus self-compiled corpora (47%). 36 studies primarily focus on observed data with 10 of these obtaining their NOD via field notes.

Figure 2b.

Extracted data sources.

Despite of authentic data arguably being the non-plus-ultra kind of data desirable for empirical research (Bataller and Shively 2011; Bou-Franch and Lorenzo-Dus 2008; Félix-Brasdefer 2007; Martínez-Flor 2006; Turnbull 2001), only 51 papers in the CICUP focus dataset (i.e. 23.2%) actually emphasize their own use of authentic data (cf. P151, p. 668; P239; P285). As mentioned above (2.1), such data is not necessarily straightforward to come by and this is certainly reflected strongly in the data at hand (cf. again Figure 2a). The logistical complexities to data collection and the level of reliability of thus-obtained data are particularly precarious in such observational designs where the researcher keeps themselves out of the equation, i.e. the communicative scenario observed: In P156, for instance, in order to “ensure the naturalness of the data”, the researchers had

the participants record […] themselves without the presence of the researcher. Randomly selected participants were provided with cassette recorders and instructed to record their interactions (p. 99)

Participational or interactive observation designs are thus more and more emerging among field methods of choice. In CICUP, 7 out of 36 papers take an interactional-observational approach (cf. P95, P121, P160, P211, P215, P224, P319). Particularly elegant are such research designs that allow for the researcher to take on the role of participant observer whilst actually being a genuine part of the NOD scenario, as in meetings conventionally minuted or audio recorded anyway (cf. e.g. P211).

As Figure 3 shows, the salient standard design – diachronically speaking – has been the cross-sectional study with 9 rising to an average of 12 papers published per year since 2004. Longitudinal and case study designs are significantly rarer, yet also stable over time with an average of 1 and 2 papers published in Intercultural Pragmatics per year respectively.

Figure 3:

Research designs as applied in CICUP (2004–2020).

Note that Culpeper and Gillings (2019: 8) found case studies to be declining dramatically in Journal of Pragmatics papers between 2000 and 2008 and since then plateauing far beneath corpus studies. What we see in CICUP seems to be the continuance of this low-plateau trend of case studies in pragmatics research overall, which is, however, not entirely surprising given the fact that case studies will not conventionally offer strong implications of result representativeness.

4.2 Task-elicited data types in CICUP

The hypothesis that obtaining data by systematically controlling the settings and variables for data collection and by employing certain elicitation prompts to get a hold of the data of interest is widely preferred in ICUP research (cf. Clark and Bangerter 2004) finds affirmation in CICUP. As stated by P218, “[e]ven the authors who realize the deficiencies of elicitation tasks view their use as inevitable in contrastive studies” (p. 650). In CICUP, 64 studies (29.1% of the data-based papers) make use of questionnaires as direct lines to intuitive, meta- (e.g. P301) as well as task-elicited data. Subsections 4.2.1 and 4.2.2 outline the results in detail.

4.2.2 Production tasks in (semi-)designed and controlled settings

Given that the focus of task-based elicitation of data is on production overall, comprehension tasks are used very rarely in CICUP (cf. P313, P164; also cf. Takimoto 2009; but see 4.2.3). In line with Schneider’s categorization proposal of low- to high-interaction elicitation tasks (2018: 58), one might come to hypothesize that the higher the level of interactionality in a production task, the more likely it might be found worthwhile to ICUP research. This would lead to the assumption that e.g. role-plays and interviews would be featuring rather frequently in CICUP, whilst e.g. DCTs would not. As Figure 4 shows, this is, however, not exactly reflected in the data when it comes to the salient representatives of low, medium and high-interaction tasks, i.e. DCTs, interviews and role-plays respectively. While Martínez-Flor and Usó-Juan found the most widely used methods to elicit pragmatic data in Interlanguage Pragmatics to be role-plays and DCTs 10 years ago (2011: 51), CICUP studies turn out to have used interview data more than twice as often as role-plays and DCTs each (cf. Figure 4). However, one should keep in mind here that these proportions change if one allows for the possibility that the notions ‘DCT’ and ‘questionnaire’ may be used interchangeably by some researchers. Subsections 4.2.2.1 thru 4.2.2.3 offer detailed discussions of the results pertaining to low-, medium- and high-interaction tasks.

Figure 4:

Distribution of data elicitation tasks in CICUP.

4.2.2.1 Low-interaction tasks: written and oral DCTs

Amongst the various kinds of tasks created to elicit productive data and categorized here as ‘low-interaction’ are writing assignments (P168), read-aloud tasks for intonation research (P22), elicited narratives (P205), and, as the most salient group (both on the meta- as well as the application level), DCTs. In line with the respective state-of-the-art literature cited in 2.2, colleagues in ICUP making use of DCTs will usually appreciate their time-efficiency, potential of cross- and interdisciplinary replicability (P93), as well as the high level of variable control (cf. P140, p.317). Overall, DCTs are used by 31 papers in CICUP, with the vast majority (25 studies) administering them in written rather than in oral format.

CICUP also provides evidence of the fact that researchers have been trying to address much of the criticism that has been raised against DCTs (cf. e.g. Martínez-Flor and Usó-Juan 2011: 53). As a result, aimful enhancements of this data collection method have been proposed and put to action in a number of studies in the corpus, with e.g. P289 prioritizing a content-enriched description of the scenario (also cf. Billmyer and Varghese 2000), or P93 creating a DCT on the basis of large spoken (authentic) corpus data rather than self-created and potentially inauthentic examples and prompts. Most frequently, though, DCTs have been triangulated with other methods, e.g. observation (cf. P93), role-plays (cf. P239), or extracted data (cf. P237).

4.2.2.2 Medium-level interaction tasks: elicited dyadic conversations

Examples of elicited conversations that are not actual interviews in CICUP include various research designs in which dyadic conversations were specifically set up by the researchers (cf. P114, P144, P225, P242, P317). For instance, in P225, informants were instructed to make a table reservation at a restaurant via skype within the context of language proficiency assessment; P114 triangulates a set of interactive oral tasks, i.e. gap tasks, a dialogue reading task and an informal conversation.

Of the 220 papers working with collected data, almost a third (i.e. 61 papers, 27.7%) make use of interview data, which seems to align with e.g. Bardovi-Harlig and Salsbury’s (2004) reported impression that the richness of interview-elicited data that they collected longitudinally clearly outweighed the transcription workload by far.^[8] At the same time, the emergence of internet-mediated research has given rise to interviews also being conducted online e.g. via computer-mediated chat functions or teleconferencing platforms (cf. P238), which has rendered interviews a much more efficient method nowadays. Several papers in CICUP emphasize the added value in web-based communication particularly in intercultural scenarios (cf. e.g. P326).

The range of interview (sub)types is multifaceted, including what authors call, amongst others, ‘informal’ (P36, P197), ‘spontaneous’ (P108), ‘off-the-record’ (P131), ‘open-ended’ (P224, P95), ‘free’ (P209), and ‘(semi-) structured’ (P228, P260, P288, P332, P35, P39, P105, P127, P160). Specifically relating to L2 contexts are e.g. ‘language awareness interviews’ (P198) and ‘oral proficiency interviews’ (P7). The most frequent kind, however, are retrospective (post factum) interviews, usually conducted for affirming triangulation with other data types collected (P67, P83, P84, P93, P94, P105, P116 P 148, P159 P166, P194, P216, P222, P223, P224 P33). In rare cases, researchers take on the effort of incorporating interviews in a pre-data collection stage in order to proactively justify or strengthen their framework or terminology (cf. P332). In few cases, structured interviews are even used to collect background and demographic data from participants (cf. e.g. P260).

Finally, a particularly rich yet generally still under-used interview type has emerged with focus group interview designs.^[9] These “allow individuals to respond in their own words, using their own categorizations and perceived associations” (Stewart et al. 2007: 13), thus producing the type of interactional data that is “suitable for a detailed discursive analysis” (Goodman and Burke 2010: 328). Four CICUP papers employ this instrument (P121, P147, P152, P206), with e.g. P152 only providing one conversation trigger and thus leaving the conversants to themselves from there on up. Such designs clearly border on or even overlap with high-interaction tasks (cf. 4.2.2.3).

Note that yet another 14 papers (not counted into the 61 just discussed) have extracted various kinds of interview data from electronic corpora and other already existing datasets (e.g. political or live interviews, e.g. P79, P112). All of these observations suggest that data elicitation through interview techniques has been frequently applied by scholars in ICUP. At the same time, interview data alone is rarely considered a sufficient primary data source in and of itself. In fact, interviews often fulfil the function of confirming assumptions deducted from data collected and analyzed prior (cf. e.g. Moreland and Cowie 2005) and thus are usually triangulated with other complementing data, which is the case also in CICUP.

4.2.2.3 High-interaction and collaboration tasks

The range of collaborative, high-interaction learning activities as tasks used to elicit data in CICUP covers writing or translation assignments (e.g. P226, P257), video-conferencing sessions (P139), group discussions that may be set up as face-to-face (cf. e.g. P187, P222, P283) or via computer-mediation (P117, P118, P155, P226; P33), and peer assessment as well as peer feedback tasks (cf. P36, P326).

Further, 27 papers in CICUP employ RPTs (cf. e.g. P330, P260), whilst, however, acknowledging that method’s limitations and drawbacks already mentioned above (2.2; cf. P353, P216, P218, P239), One can certainly agree with P218 on the fact that

[t]he choice of situations to be put to a test in role plays […] should take into consideration that speakers of different cultural backgrounds may be sensitive to different aspects of the context related to multiparty interaction. (p. 671)

4.2.3 Data elicited under experimental conditions

Based on the CICUP data at hand, Experimental Pragmatics has hardly been informing ICUP research so far. Out of the total of 358 studies in CICUP, 15 do cite the relevant literature (e.g. Garrett and Harnish 2007; Meibauer 2012; Meibauer and Steinbach 2011; Noveck and Sperber 2004), but only one study, i.e. P306, does explicitly position itself as experimental.^[10] Reasons for this seemingly reluctant attitude are probably to be looked for in logistics mainly (sampling procedures, technical affordances, etc.). For instance, Experimental Pragmatics focuses not only on learners, but more so than other subfields on e.g. subjects with neuro-developmental disorders (Pijnacker et al. 2009) and language impairment (Katsos et al. 2011). Recruiting participants in general must therefore come with an unlimitedly transparent and ethically impeccable research design and recruitment procedures that ensure that participants will be able to freely give their informed consent to participate in the study. What is more, an experimental study will usually employ

a hypothesis testing procedure in which certain variables are manipulated while others are held constant. […] The manipulation is powerful and occurs in an experimentally controlled setting yet it is a common part of daily life and the study occurs “n the “rea” world” rather than in the laboratory. For these reasons, the data generated under these conditions should be highly representative or generalizable. (Turnbull 2001: 37f.)

As for technical equipment required for many experimental research designs, no single paper in CICUP draws upon data obtained through neuroimaging, event-related potentials, functional magnetic resonance imaging or electroencephalography.^[11] Only eye tracking is occasionally employed (cf. e.g. P101).

Still, 20 papers (including P306) do adhere to experimentally elicited data in their studies, even if they may not be explicitly claiming so (cf. Figure 2a above). Diachronically, experimental designs in CICUP reached their highest peak so far in 2008 (Figure 6 below).

While the 20 experimental studies in CICUP demonstrate quite a multifaceted scope of topics addressed (negation, context effects, intonation, speech acts, etc.) as well as a range of diversified experimental designs overall, most of them (i.e. 18) include a quantitative analysis of the obtained data, with 15 of these also employing descriptive or inferential statistics. Only two studies (P124, P266) zoom in on a solely qualitative interpretation of results. 10 studies apply mixed analytical approaches. One of these is P99, which investigates to what extent incongruent context in presented stories enhances the subjects’ use of negation. As visualized in Figure 5, the majority of the experimental papers targets aspects of comprehension and cognitive processing (13 papers), e.g. measuring processing times under self-paced reading conditions (cf. P100, P101, P243). P101 uses eye-tracking technology to do so.

Figure 5:

Distribution of experimental elicitation task approaches in CICUP.

In contrast, fewer designs in CICUP adhere to experimental production tasks (7 papers, cf. Figure 5), with only one of them (i.e. P342) explicitly eliciting both production as well as comprehension data. Yet, these 7 studies on a whole do provide an intriguing range of designed components to elicit comprehension data. For instance, P266 sets up an experimental, qualitative study to investigate speakers’ choices of overtness in discourse relations; P124 uses RPTs to elicit experimental data; P334 poses counter-expectational questions in order to elicit gestural data; and P242 manages to manipulate subjects into producing NOD, with the following conditions created:

At the end of the semester, two final exams for each level were intentionally scheduled to be administered on the same day. Teachers of those subjects informed students of the possibility of altering the timeslot of one of those exams, provided that half the group individually ask the administrator to defer or bring forward one of those exams to another day. […] The administrator was informed that his interactions with the learners were being recorded in order for the department to evaluate the progress of learners’ language. The learners, however, were not informed that their interactions with the administrator were recorded, making the data collected genuine. (p. 627)

While the absolute number of experimental designs in CICUP is admittedly relatively small, Figure 6 shows that experimental designs have been following a steady and constant trend in CICUP since 2004 after all.

Figure 6:

Trends of experimental and non-experimental data collection methods in CICUP (2004–2020).