Gender agreement among Russian learners of Spanish in an instructed versus naturalistic learning environment

2.1 Background

Grammatical gender is a common, typological feature, that is present in approximately twenty-five percent of world languages (Corbett 1991). Grammatical gender forms part of a larger noun class system that allows agreement between nouns and other linguistic structures such as adjectives, articles, pronouns, and/or verbs. According to Hockett (1958: 231), “[…] genders are classes of nouns reflected in the behavior of associated words.” Therefore, gender in languages is just one way of dividing nouns into classes and, according to some linguists, “grammatical gender” and “noun class” are synonymous terms.

Before proceeding to a discussion on the acquisition of morphological gender in Spanish in instructed and naturalistic environments, it is necessary to illustrate the morphological differences and similarities between the two languages under study, namely, Russian and Spanish.^[2] Although the two languages are considered to be typologically different due to their distinct language families, Russian and Spanish nonetheless share many similarities, one of which is gender marking and agreement. Additionally, the acquisition of gender tends to occur at similar developmental stages with gender acquisition occurring at the age of two in Russian monolingual children and at the age of three in typically developing Spanish monolingual children (see Hernández-Piña (1984) for Spanish gender acquisition and Schwartz et al. (2015) for developmental stages of Russian gender acquisition). To better facilitate an understanding of gender marking and agreement in the two languages under study, Sections 2.1.1 and 2.1.2 discuss grammatical gender in Russian and Spanish, respectively.

2.2 L2/L3 acquisition in naturalistic versus instructed environments

Though our study does not directly test for acquisition and learning differences, Krashen’s (1982) Monitor Model is relevant to our project when distinguishing between naturalistic and instructed modes of learning (Lightbown and Spada 2021). For Krashen, “the acquisition process is subconscious and ‘effortless’, given that the learner processes the language with minimal amount of cognitive or mental effort” (cited in Leow 2019: 481). Moreover, acquisition is treated as an implicit, informal, or natural form of learning. This is similar to Ellis et al. (2009) and Leow’s (2019) point, who stress that incidental learning is an absence of awareness with low depth of metalinguistic processing (Leow 2019: 481). The term “learning environment”, on the other hand, refers to explicit instruction where teachers provide explicit grammatical or instructional information in the target language. These definitions raise the issue of whether “native-like” proficiency can be attained in languages learned in late adulthood.

There has been an ongoing debate regarding whether late (adult) nonnative speakers are able to implicitly process language and reach native-like competence (Pliatsikas and Marinis 2013). There are many factors that can play a role in a learner’s acquisition of additional languages, including their working memory, proficiency, and the type of learning environment (e.g., instructed/classroom or naturalistic) to which the learner is exposed, among other factors. In a classroom-based environment, classes are usually held twice a week for a total of three to four hours per week; in more intensive programs, students engage in classroom study for multiple hours a day. Muñoz (2008) describes a classroom environment to be formal and highly structured with input limited to what the teacher provides and not much interaction between the learner and the target language outside of the classroom. Additionally, Loewen (2015), defines “traditional” or “classroom” instruction as focused heavily on explicit attention to language, rather than using language to communicate. One important aspect to consider is the language of instruction. This refers specifically to whether the target language is taught in the dominant language of the university or the target language itself. A naturalistic environment is unstructured, and the learner is exposed to virtually unlimited native-speaker input.

Researchers argue that the amount of input is crucial for processing and competence in a non-native language (e.g., Dussias 2003; Festman 2021; Flege and Liu 2001). Based mostly on semi-artificial language data, research shows that adult L2 learners may incidentally learn aspects of non-native morphosyntax while processing the target language for meaning and without being explicitly exposed to learning the language in a classroom setting (e.g., Grey et al. 2014; Rebuschat and Williams 2012; Williams and Kuribara 2008). In their study on L3 acquisition of word order and case marking in a semiartificial language, Grey et al. (2014) examined data from 36 undergraduate L1 English-L2 Spanish incidental learners who were exposed auditorily to L3 for two weeks. Twenty-one of the participants were advanced learners of Spanish with 9.7 semesters of prior exposure to Spanish, while the rest of the participants were beginner Spanish learners with 3.3 semesters of exposure in L2. Their results pointed to a significant task effect on the initial post-test such that participants performed better with the acceptability-grammaticality judgement task, compared to the picture naming task. The results on the delayed post-test showed significant improvements on the second task, where participants were able to discriminate with higher and consistent accuracy. Overall, these results align with other previous work (Rebuschat and Williams 2012; Williams and Kuribara 2008), which suggests that adults can learn grammatical aspects of language without explicit instruction in a short period of time without additional practice in the target language.

Other studies also suggest that learners with less than nine months of naturalistic exposure tend to produce errors that stem from erroneous L1 transfer or interference, while learners with more than five-years of residence in a country where the language is spoken no longer demonstrate these types of errors and are able to accurately resemble the speech of native speakers. In a study on highly proficient Greek learners of English, Pliatsikas and Marinis (2013) compared how L2 English learners in a classroom environment in Greece and L2 English learners in a naturalistic setting in the UK of nine years of exposure processed wh-dependencies in declarative sentences. Their results showed that the two groups, irrespective of type of exposure, performed at a similar rate of accuracy with comprehension questions (over 70 % accuracy in two groups). However, significant differences were found with processing sentences with intermediate gaps where the learners in the UK performed better than the other group. This suggests that linguistic immersion can lead to native-like processing and acquisition among highly proficient learners. Llanes and Muñoz (2013), conducted a study on Spanish-speaking children and adults who studied English at home and abroad in Ireland or England. The authors found that oral competence significantly increased for the study abroad groups, but at-home students performed better with writing tasks, suggesting that explicit instruction of grammatical structures was necessary for knowledge of the grammar, while the procedural and implicit knowledge that the study abroad groups received did not augment the knowledge of L2 grammar in writing. This is consistent with other studies which argue that oral fluency, especially at initial stages of learning, increases while studying abroad, whereas the knowledge of grammatical structures is facilitated more by the traditional classroom setting (see di Silvio et al. 2016; Du 2013; Mora and Valls-Ferrer 2012). In a study on the development of L2 fluency in Mandarin, Russian, and Spanish during study abroad programs in China, Russia, Peru and Chile, di Silvio et al. (2016) found that Mandarin and Spanish learners made significant gains in speech rate and mean sentence length. The pause rates also decreased among advanced learners, indicating speedier and more fluid speech.

Finally, when examining individuals learning a language in a traditional classroom versus those in an immersed environment it is important to tease apart whether the setting has an effect on how quickly an individual is able to perform a grammatical task and whether this rate of response is comparable to that of a native speaker. Klassen et al. (2023) conducted a study on Spanish gender agreement (determiner-noun concord) among intermediate English learners of Spanish prior and after a short-term study abroad program in Spain. Using a self-paced reading task and a picture selection task, the results showed that the participants overused the masculine article as a default form, specifically in the first task prior to immersive learning, corroborating previous studies (e.g., McCarthy 2008). After immersion, the number of errors decreased. As for the second task, L2 learners selected the images faster after the study abroad experience, but errors still persisted with gender marking. Again, similar to previous studies (di Silvio et al. 2016; Llanes and Muñoz 2013) examining adult language acquisition in the study abroad context, participants demonstrated more communicative and lexical gains, while losing sensitivity to grammatical errors.

Overall, studies on modes of acquisition have shown that at advanced stages, regardless of learning environment, learners are able to acquire the target language and resemble the speech of native speakers. At the beginner level, most studies observe fluency and lexical gains with participants who study the target language in the immersive setting, yet there is still debate regarding whether grammatical structures are easier to acquire in the instructed or naturalistic environment.

2.3 Acquisition of morphological gender

In the past two decades, much previous research has focused on the acquisition of L2 gender (see Montrul et al. 2008, 2014; Sabourin et al. 2006). Specifically, one of the main questions that researchers have been concerned with is whether typological similarity/proximity between two languages plays a role in L2 gender production. Sabourin et al. (2006) studied the assignment and agreement of grammatical gender in Dutch as an L2 among three groups of adult speakers: German, English, and Romance languages (Spanish, Italian, and French). German and Romance languages are groups of languages that exhibit a gender marking system. German, however, is typologically similar to Dutch due to similar ancestry, resulting in congruency between the two languages. Though Romance languages also exhibit a binary gender system, the systems are not congruent with Dutch. Seventy participants were recruited to partake in two experiments: a gender assignment task and a grammaticality judgement task. The results showed that all three groups had an accuracy score over 80 % on the gender assignment task. Out of the three groups, the German group performed the best, suggesting that this group directly transferred morphologically similar forms between their L1 and L2, known as surface transfer. The Romance group also performed well, specifically with higher frequency nouns (over 90 % accuracy). The English learners performed better with higher frequency nouns but overall performed the worst in comparison to the other two groups (83 %). Regarding the second task, similar results were found with the German group performing the best, specifically with higher frequency items, followed by the Romance group, suggesting that typological congruency plays a bigger role than the existence of abstract gender features. As for the middle frequency items, there was no significant difference between the two gender groups. This indicates that speakers of gendered languages have an advantage in the acquisition of grammatical gender over learners that have previous knowledge in languages that do not exhibit a gender marking system. In another study on L2 gender acquisition, Montrul et al. (2014) analyzed data of L2 Spanish intermediate-advanced learners and compared them to the heritage Spanish learners using an oral elicitation task.^[4] The L2 group was more accurate with masculine and canonically non-marked nouns, suggesting that the L2 group overgeneralized feminine nouns and by default assigned and produced a masculine form, corroborating the results of previous studies (e.g., Bruhn de Garavito and White 2002; Gamboa Rengifo 2012). Furthermore, more L2 errors were found with noun-determiner agreement than adjective-noun agreement.

When it comes to L3 acquisition, previous studies have demonstrated the advantage of learning an L3/Ln when the languages are typologically related and comprise overlapping properties (see Długosz 2023; Festman 2021; Jarvis 2015; Krenca et al. 2020; Puig-Mayenco et al. 2020; Rothman 2011, 2013, 2015; Rothman et al. 2019). Additionally, both Cenoz (2003) and later Festman (2021) in her review paper claim that bilingual and multilingual speakers are said to be equipped with a greater degree of metalinguistic awareness, which allows them to process language systems, structures, and rules quicker compared to monolingual speakers. This has been shown to be true with previous studies focusing on gender agreement. Specifically, Długosz (2023) studied gender agreement among two groups of intermediate and advanced participants: Polish native speakers of L2-English/- L3-Swedish and Polish native speakers of L2-English, L3-German, acquiring Swedish as a fourth language (L4). Though Polish is a gendered language, Swedish and German share similarity in gender marking, so it was expected that the L4 group would outperform the L3 group due to positive transfer. Using a speeded grammaticality judgement task, the results showed that the groups did not actually differ in terms of their judgement accuracy (roughly 81 % accuracy for both groups). Regarding task completion times, the L4 group processed noun phrases faster than the other group, but only at the intermediate level, which could be explained by a surface transfer of similar gender agreement marking that allows learners to automatize the gender agreement processes earlier. As for the sensitivity to ungrammatical structures, the learners of the two groups developed sensitivity to ungrammatically when they reached an advanced level of proficiency. As for acquisition of grammatical gender at the initial stages of L3, Brown (2020) investigated the topic using data from beginner L3 German learners with L1 English and L2 Spanish and the corresponding mirror group, L1 Spanish-L2 English-L3 German learners. The procedure consisted of an untimed grammaticality judgement task, in which 32 participants were presented with 68 sentences consisting of a singular noun in a nominative case and a definite article, which either matched or mismatched the gender of its corresponding noun. The results showed that the first group (L1 English-L2 Spanish-L3 German) significantly outperformed the other group (by 33 %), indicating that the transfer of gender occurred from L2 rather than L1, supporting the L2 Status Factor Hypothesis (Bardel and Falk 2007). Also, since both the L2 and the L3 were learned in an instructed setting, participants of that group received explicit metalinguistic knowledge of grammatical gender which facilitated their learning and equipped them with contexts of use. Regarding the matching-mismatching factor, participants were significantly less likely to correctly identify a grammatical gender error if the gender mismatch was “match” or “opposite”; in other words, both matching and mismatching nouns only resulted in 44–47 % accuracy, indicating that there was not much difference in terms of congruency. Sá-Leite et al. (2019), on the other hand, while performing a meta-analysis of a variety of language pairings among bilinguals, found that bilinguals were able to more quickly process nouns that were congruent in both their languages compared to those that were incongruent. The gender of incongruent nouns was often assigned based on their L1.

In a more recent study, directly relevant to our research, Tararova et al. (2023) investigated gender agreement among L1 English-L2 Spanish and L1 Russian/L1 Mandarin speakers who were proficient learners of English as L2 and were learning Spanish as their L3. All three learner groups (n = 45) were enrolled either in a beginner or advanced level Spanish course at a Canadian institution. In addition to these three learner groups, 15 native Spanish speakers were also included as a baseline for comparison. The participants completed two online (speeded) tasks, including a picture identification and a grammatically judgment task. The first task included 24 target items where participants saw a series of images and were asked to describe the picture pairings orally. In the second task, participants were asked to select the option that best answered the question. Four answers were provided to them with only one being the grammatically correct answer. Both tasks tested for gender form (canonical and non-canonical), gender class (masculine and feminine), and number^[5] (singular or plural), as well as task effect. The results showed that advanced learners of all three groups performed at or near ceiling, especially on Task 1. The Russian group outperformed the other two learner groups in both tasks converging on native-like performance. Regarding the beginner groups, again, the Russian group outperformed the other two learner groups, suggesting that L1 learners of gendered languages perform significantly better than the learners of non-gendered languages. In terms of gender form accuracy, participants were more accurate with canonical forms, which corroborates prior research (Foote 2015; Gamboa Rengifo 2012; Montrul et al. 2008). Regarding gender class, all three groups performed better with masculine target items than feminine forms. One explanation, provided by the authors, is based off of McCarthy’s (2008) hypothesis which states that nouns tend to be overgeneralized to the masculine form, as a general cognitive processing strategy. Even though Russian exhibits both masculine and feminine forms, the beginner group was more accurate with the masculine form (over 80 %) than with the feminine form (under 70 %). Given that the researchers did not analyze congruency between Russian and Spanish nouns, it is hard to interpret the results with much certainty. As for task effect, only the beginner groups showed more accuracy with the second task, suggesting that orthographic input during testing can yield higher accuracy (e.g., a written question with four answers available in Task 2 in comparison to a picture with only the determiner in Task 1). Overall, the study provides evidence that at advanced levels, participants of gendered languages (e.g., Russian) can resemble native-like speech, independent of the specific task and linguistic variables. At the beginner level, gender class, morphological form, as well as the nature of the task all play a role in L3 gender production.

3.1 Research questions and hypotheses

Based on the main objectives, the current study is guided by the following research questions:

1. How does context of acquisition (naturalistic vs. instructed) and proficiency level (beginner vs. non-beginner) impact grammatical gender accuracy and agreement in L3 Spanish?

2. How do noun type (canonical vs. non-canonical; masculine vs. feminine) and gender assignment congruency with Russian affect accuracy rates in each learner group (naturalistic and instructed)?

3. Is there a correlation between task completion time and accuracy scores across all four tasks, and, if so, is this correlation different for naturalistic and instructed learners across proficiency groups?

Based on our research questions and previous studies, we predict the following:

H1: Regarding learner type, we expect to find differences in accuracy scores between our naturalistic and instructed learners. Though the research is extremely limited on incidental learning, previous studies find that at the initial stages, immersion where the target language is spoken can be beneficial for fluency and potentially grammar acquisition due to increased and enhanced exposure and input (e.g., Pliatsikas and Marinis 2013). However, at later stages of acquisition, both groups will show similar behaviour and perform equally well.

H2: Regarding the linguistic variables, we predict that participants of both groups will be more accurate with canonical nouns compared to non-canonical forms (Montrul 2008; Sabourin et al. 2006; Tararova et al. 2023). Regarding masculine versus feminine forms, we expect participants to be more accurate with masculine forms than feminine, as shown in previous studies (e.g., Foote 2015; Gamboa Rengifo 2012; Tararova et al. 2023), since this will be more of a default form and, therefore, easier to retrieve. Regarding the effect of congruency, we expect to see more accuracy with nouns that are matching in gender between Russian and Spanish.

H3: Based on Sá-Leite et al. 2019, we predict that participants of all groups will be quicker and more accurate with non-marked forms (masculine, canonical, and congruent), than marked types (feminine, non-canonical, and incongruent). For task effect, we expect to find differences in task completion times with the oral translation task (Task 1) taking the longest and the grammaticality judgement task (Task 4) taking the least amount of time, specifically among our beginner learners. This is due to the orthographic cues that will be available in Task 4 and not in Task 1, since during the translation of a text, the participants will have to retrieve their knowledge of Spanish grammar and vocabulary in the absence of any cues.

3.2 Participants

A total of 64 adult participants were recruited.^[6] This included 15 native Spanish speakers which served as a baseline for comparison and 49 L1 Russian speakers who had English as their L2 and were learning Spanish as their L3.^[7] Of these, 24 learned Spanish in an instructed learning environment and 25 learned Spanish in a naturalistic environment while working/living in Mexico. Both L1 Russian groups were composed of speakers of differing levels of proficiency in Spanish and were subsequently placed into groups designated as either “beginner” or “non-beginner” (see Table 1 for a full breakdown of participant groups). The participants who learned Spanish in an instructed setting were students enrolled in a beginner-level Spanish course at a university in Ontario (beginner group) or alumni, graduate students or current upper-year students enrolled in Spanish courses across Ontario universities (non-beginner group). To ensure the beginner-level instructed learners would be able to complete and understand the tasks, we recruited individuals from this specific group after they had completed at least one semester (i.e., 4 months) of Spanish. The participants who had learned Spanish in a naturalistic setting included individuals who had been living in Mexico for at least one year, had not had any form of explicit or formal education in Spanish (e.g., through tutors or university classes), and were immersed in the language in their daily lives either through work, study, or family. Many of the naturalistic Russian participants had jobs in real estate, motivational careers, or were employed as personal trainers, among other occupations. Most used English in their work, but some did report using Spanish. See Table 2 for a more detailed breakdown on the average age, average age of acquisition of Spanish (AoA), and average length of residency (LoR) in an English-speaking country for the instructed learners and a Spanish-speaking country for the naturalistic learner group. To ensure correct placement of beginner and non-beginner participants, depending on their level of proficiency in Spanish, all Spanish learners completed a proficiency test which will be further described in the Study tasks and Procedures section.

Based on the participants’ self-reported data provided in the linguistic questionnaire and a brief conversation in Russian at the start of the session (see Section 4.3), all participants were verified as proficient in their native language in addition to English. In Ontario, French is mandatory until Grade 9. Therefore, participants were excluded if they were enrolled in French immersion or took it beyond the mandatory level. This was of particular importance for the instructed learning group, since they had completed schooling in Canada prior to enrolling in Spanish classes at the postsecondary level.

3.3 Study tasks

The present study included a total of four untimed tasks, namely, a translation, picture identification, oral description, and grammaticality forced choice (GFCT) tasks. Both experimental groups completed all four tasks, while the native control group did not complete Task 1 (Translation Task). All participants completed the study tasks online via the Zoom video conferencing platform. We used Zoom to conduct the study as it would allow us to reach a larger proportion of potential participants, especially for the naturalistic group who were living in Mexico at the time of the study. Additionally, the study was initiated during COVID-19, therefore, conducting the experimental procedure online was the easiest method for recruitment. All the target tokens for each of the tasks were analyzed and divided into three categories to examine accuracy, namely, morphological gender (masculine/feminine), canonical/non-canonical, and congruency of gender between Russian and Spanish. This categorization will be discussed further in the Data Analysis section. For the translation task (Task 1), participants were required to translate a passage from Russian into Spanish (see Example 1). Example 1 contains the first few sentences of the passage, but the entire passage consisted of 25 targets. Note that participants received a version of the passage with no underlining or identification of specific phrases to avoid thereby diverting their attention from the variables under study.

(Example 1) Soy una chica italiana y tengo una familia pequeña. Vivo con mi madre, mi papá y mi hermano. Mi papá es doctor y trabaja en un hospital nuevo .

I am an Italian girl and I have a small family. I live with my mother, my father, and my brother. My father is a doctor, and he works in a new hospital.

In Example 1, the underlined phrases represent the target phrases that were later analyzed for potential errors. The participants were not required to read the passage out loud in Russian, but rather provide their oral translation in Spanish sentence by sentence. The passage remained on the screen for the entirety of the task.

For the picture identification task (Task 2), participants were given a sentence containing a blank followed by a set of 3 images (see Figure 1, Example 2). Participants chose the image they believed was correct, with only 1 of the 3 options being grammatically correct. The task included 3 practice examples, followed by a total of 34 questions, 24 of which were the target items (12 masculine and 12 feminine nouns, divided into canonical/non-canonical and congruent/incongruent forms) and 10 distractor items, which included infinitive verb forms and did not test gender agreement.

Figure 1:

Example 2.

In Example 2, la ‘the’ refers to the feminine singular definite article in Spanish. The first option, falda ‘skirt’ is the only grammatically correct option since the noun is feminine and singular.

For the oral description task (Task 3), participants saw one of two scenarios: a number + an object or a colour + an object. They were then asked to combine the two images to create a short phrase (see Figure 2, Example 3). The task included 2 examples (with answers provided by the researcher) and 2 practice prompts (to be completed by the participant in preparation for the real trials), followed by a total of 34 prompts: 24 of these prompts were the target items (same division as Task 2) and 10 distractors which did not assess gender concord were also included. The target nouns used in this task were similar to that of Task 2 mentioned previously.

Figure 2:

Example 3.

In Example 3, rojo ‘red’ and barco ‘boat’ would grammatically produce el barco rojo ‘the red boat’ since boat is a masculine noun in Spanish and, therefore, the article and corresponding adjective must also correspond to the noun’s masculine form.

Finally, for the GFCT (Task 4), participants were given a question followed by four possible answers (see Figure 3, Example 4). Here, participants were instructed to choose one of the four options they believed was grammatically correct. The task included 3 practice items, followed by a total of 34 test questions. Twenty-four of these questions were the target items (same division as Task 2), in addition to 10 distractor items which were composed of questions that did not test for gender concord. The target nouns used in this task were similar to that of Task 2 mentioned previously.

Figure 3:

Example 4.

In Example 4, option 1 is the only grammatically correct option, since chocolates ‘chocolates’ is a masculine noun; therefore, the article los and the adjective amargos ‘bitter’ must agree with the masculine noun.

Though all four tasks focused on gender marking, they differed with regards to the amount of Spanish orthographic cues they provided. For instance, in Task 1, the participants only saw a paragraph in Russian, therefore, no orthographic cues were available in Spanish, in contrast to Task 4, where learners were explicitly provided with four different written options in Spanish. Besides orthographic cues, the nature of each task varied, each aiming to test a distinct feature to determine the role of task effect. More specifically, Task 1 assessed overall competence with gender assignment and agreement since participants had to produce sentences with no access to Spanish cues. Task 2 examined to what extent participants were able to correctly assign and form agreement between a provided article and noun (pictured in the images provided), while Task 3 determined whether participants were able to detect and assign the gender of an article, noun, and corresponding adjective based on the combination of noun and colours provided as images. Finally, Task 4 tested whether participants were able to choose the correct option based on gender agreement between an article, a noun, and an adjective. All the target items were controlled for their relative frequency and familiarity, as we used the most frequent nouns found in recently studied chapters of the beginner Spanish textbooks used in the Spanish language program at a post-secondary institution in Ontario. Additionally, we made sure to not include any targets in Spanish that belonged to the following categories: (i) nouns that when translated would pertain to the neuter category in Russian or (ii) any of the exceptions mentioned in Section 2.1.2 (Grammatical Gender in Spanish) such as la man o which ends in -o and would most often be masculine, but in this particular case is feminine. We also made sure to use vocabulary that beginner learners of Spanish would have access to through their textbook. For the naturalistic participants, we corroborated their knowledge of the vocabulary used in the target nouns through the proficiency test (Section 4.3) since they do not have access to the same textbooks the instructed learners did.

3.4 Procedure

Prior to the experimental session, all participants were vetted with a series of questions to ensure they met the eligibility criteria. They were then provided with a letter of information and a consent form to be signed. This was followed up by a Linguistic Background Questionnaire which required participants to answer questions regarding their gender identity, native language, language use, etc. This questionnaire was administered to gain insight on the participants’ language abilities and to further confirm their eligibility. Participants were then provided with a Zoom link to participate in the study. Sessions began with a short interview in the participant’s L1 to ensure their proficiency in the language. This was followed by a Language Proficiency Test in Spanish to determine the individual’s level of Spanish and place them into the appropriate proficiency group. This test was composed of two parts: (i) a fill-in-the-blank section where participants had to choose the best option to complete the sentence, and (ii) a cloze test where participants were required to choose the verb conjugation that best completed the story provided. The test was composed of easier questions that were appropriate for beginner learners, followed by questions increasing in difficulty, meant for more advanced participants. The native Spanish control group also completed this proficiency test to obtain a baseline of comparison for the experimental groups. Next, all the participants were administered the four tasks in the following and same order: Task 1–4. In the interest of confidentiality, all data were anonymized. At the end of the session, participants were compensated for their time spent in the study with a $20 (CAD) Amazon e-gift card. The study took place over one study session lasting approximately 45 min to an hour, where participants’ responses on Tasks 1–4 were recorded using the Zoom record function.

3.5 Data analysis

A total of 6,205 tokens were analyzed (i.e., 1,600 tokens for Task 1 and 1,535 tokens each for Tasks 2, 3, and 4). For Task 1, any productions that were given in a language other than Spanish (e.g., some participants produced English utterances) and productions that deviated too much from the target noun and did not permit for the analysis of gender concord (e.g., la universidad es muy popular ‘the university is very popular’ instead of “la universidad es muy famos a ” ‘the university is very famous’) were excluded from the final analysis. In the case of popular, items such as these could not be analyzed for gender concord since popular can be used to describe nouns that are masculine or feminine. The recordings for each participant were segmented per task using Audacity (version 3.3.3). The translation task was analyzed using the interview annotation program ELAN (version 6.7). The remainder of the tasks were carefully transcribed, and errors and accuracy scores were recorded in an Excel spreadsheet. For all four tasks, target data was analyzed and coded according to 3 categories: overall, canonical/noncanonical, masculine/feminine, and congruent/incongruent, and accuracy rates. Canonical targets included those ending in a vowel (e.g., un-a chic-a italian-a, a-f girl-f Italian-f, ‘an Italian girl’), while noncanonical targets were those ending in a consonant (e.g., la Universidad, the.f university-f). Masculine targets were mainly comprised of nouns/adjectives ending in an -o (e.g., el pel-o cort-o the.m hair-m short-m ‘the short hair’) but also included words that do not follow these rules such as those that are not canonically marked for gender (e.g., el jardín the.m garden-m). Feminine targets were mainly comprised of nouns ending in -a (e.g., la sill-a the.f chair-f) but also included some exceptions. Matching targets included those that had the same gender in both Russian and Spanish (e.g., el barc-o the.m boat-m), while mismatching targets included those that were one gender in Russian and the opposite gender in Spanish (e.g., el cuadern-o the.m notebook-m, masculine in Spanish, feminine in Russian). For further examples of target tokens, please refer to Section 3.3.

Task completion times were calculated for all tasks, considering task completion time to be the time elapsed between the participant’s voice onset for the first word in a task and the end of the participant’s response for the final stimulus in a task. This metric allows for uniformity of data analysis across all tasks, which encompass various response modalities.

Statistical analyses were performed in Jamovi (The Jamovi Project 2022), an open-source spreadsheet-based interface for using the R programming language. All statistical tests provided are two-tailed independent samples Student’s t-tests, with Welch’s t-test also provided when samples failed Levene’s test of equal variances. A confidence interval (CI) of 95 % was used. Please see Appendix A for a summary of all the variables analyzed in the study.

4.1 Overall accuracy scores across all four tasks

Figure 4 illustrates the results of overall accuracy across the four tasks^[8] among the four experimental groups (two beginner and two non-beginner) and the Spanish control group. As seen from Figure 4, the control group performed at ceiling for all linguistic variables. Regarding our experimental groups, both non-beginner naturalistic and instructed learners performed at ceiling and, overall, we do not see many differences between these two groups regarding gender production. Where the main differences lie is with the performance of the beginner groups. In general, the beginner naturalistic group performed better than the beginner instructed group, but the differences were minimal and there were few statistically significant findings across the four tasks employed. Regarding the linguistic variables analyzed, we find that all participants performed better with the masculine form and the canonical type, as predicted in Hypothesis 1. Interestingly, regarding congruency, we notice a slight increase in the accuracy of incongruent forms among the beginner groups. We now turn our discussion to Sections 4.2–4.6, where we present each task separately, while discussing the effect of linguistic factors on accuracy and task completion times with gender production.

Figure 4:

Overall rate of accuracy across all four tasks for four groups of Russian learners of Spanish.

4.2 Translation task

Figure 5 demonstrates how each group of Russian learners of Spanish performed on the translation task. No differences were found amongst the non-beginner groups for either method of learning (i.e., instructed, or naturalistic). However, differences were detected among the beginner instructed and naturalistic groups. Specifically, the beginner naturalistic group performed better than the instructed group with feminine and incongruent nouns between Russian and Spanish.

Figure 5:

Accuracy scores for Task 1 for four groups of Russian learners of Spanish.

In addition to analyzing overall trends and individual statistical analyses per variable for Task 1, we also ran an independent samples t-test to determine if there were any significant differences across proficiency groups and linguistic variables examined in the study. These results are presented in Table 3. Significant differences (marked by *) were not found for the non-beginner groups of either learning environment, but they were found for the beginner groups. Specifically, significant differences were observed for the production of feminine, canonical, and congruent nouns, with the naturalistic group outperforming the instructed group. Additionally, almost significant results were obtained for congruent nouns (p = 0.058).

4.3 Picture identification task

Figure 6 demonstrates the accuracy results of the five participant groups (divided by proficiency level and learning context) in the picture identification task according to the three linguistic variables (gender assignment, noun morphology, and cross-linguistic gender congruency). As predicted from Hypothesis 1, the masculine form was easier to produce than the feminine form: even at the non-beginner level, instructed learners performed better with the masculine gender class than the feminine one. Regarding differences between canonical and non-canonical forms, participants of all levels performed at ceiling while producing the canonical form.

Figure 6:

Accuracy scores for Task 2 for four groups of Russian learners of Spanish, compared to Spanish native speakers.

We also ran an independent samples t-test to determine significance levels for naturalistic-instructed differences of means according to our linguistic variables. As seen in Table 4, there was no significant difference when beginner groups are separated from the non-beginner groups. However, when we merged the proficiency levels for naturalistic and classroom learners, we notice significant effects on all three linguistic variables. Specifically, naturalistic learners performed significantly better with feminine, non-canonical and incongruent nouns than instructed learners.

4.4 Oral description task

Regarding the Oral Description Task, similar to the other two tasks, the masculine and canonical forms corresponded to higher accuracy rates. Regarding the effect of congruency, as shown in Figure 7, participants of all levels were more accurate with incongruent forms than congruent ones, which contradicts our initial hypothesis.

Figure 7:

Accuracy scores for Task 3 for four groups of Russian learners of Spanish, compared to Spanish native speakers.

However, when analyzing the independent t-test results, we see no significant effect between our two learner groups. See Appendix B.

4.5 Grammaticality forced choice task (GFCT)

The results for the last task, the Grammaticality Forced Choice Task (GFCT), are presented in Figure 8. Similar to the other three tasks described previously, both non-beginner groups performed at or near ceiling. This task also shows similarities to the other tasks in that the beginner naturalistic group performed better than the beginner instructed group with nouns that are feminine and those that are incongruent with Russian. However, in contrast to the results obtained in some of the other tasks, particularly Task 1, all groups had an overall score above 80 %. This could potentially indicate that having the orthographic representations of each option helps learners in selecting the correct sentence more often. This will be discussed further in the next section.

Figure 8:

Accuracy scores for Task 4 for four groups of Russian learners of Spanish, compared to Spanish native speakers.

Although we see these trends, we did not observe any significant differences when performing independent samples t-tests for Task 4 (see Appendix C).

4.6 Task completion times

Figures 9A–D demonstrate the task completion times for each of the four tasks for the four groups of Russian learners. Looking at Figure 9A, we see that the non-beginner groups of both learner types took a similar amount of time to complete Task 1 (the translation task), both groups being significantly faster than the instructed beginner learners (Table 5).

Figure 9A:

Task completion times for Task 1 for four groups of Russian learners of Spanish.

Figure 9B:

Task completion times for Task 2 for four groups of Russian learners of Spanish.

Figure 9C:

Task completion times for Task 3 for four groups of Russian learners of Spanish.

Figure 9D:

Task completion times for Task 4 for four groups of Russian learners of Spanish.

Graphs 9A–D also include their 95 % confidence interval. Regarding Task 2, we see an advantage in task completion time for both non-beginner learner groups as compared to the beginner-level participants, both naturalistic and instructed learners.

Regarding Task 3, the Oral Description Task, the only significant between-group difference was found between the non-beginner naturalistic learners and the instructed beginners. One of the possible explanations could be the nature of task itself, which will be further explained in the Discussion section. Regarding our non-beginner groups, similar to the previous tasks, we do not observe a significant difference, which suggests that both learning contexts equally prepare adult learners with knowledge of the Spanish gender system.

Concluding the results section is Figure 9D which illustrates the task completion times of each of the four groups for the linguistic variables analyzed in Task 4 (Grammaticality Forced Choice Task). Similar to Task 2, as demonstrated in Figure 9D, significant advantages are observed for the non-beginner naturalistic learners against both beginner groups, though there is no advantage between learner type for either beginners or non-beginners. This means that regardless of the learning environment, the participants were able to complete this specific task both relatively quickly and with a higher rate of accuracy as all groups performed above a score of 80 % overall, as seen in Section 4.5, indicating a potential effect of task type.

Linear correlation (r) (Figure 10) and a follow-up omnibus ANOVA test (Table 6) revealed a negative correlation between accuracy (calculated as accuracy across all four tasks) and task completion times (R ² = −0.491), indicating that individuals who responded faster in gender production tasks tended to be more accurate. Proficiency level was a nearly significant predictor of accuracy (p = 0.058), though this result is unsurprising given that accuracy and proficiency are, for most purposes, indistinguishable. Notably, learner type did not impact the relationship between task completion time and accuracy.

Figure 10:

Correlation of participant task completion times and accuracy scores.

5.1 Research question 1

RQ1: How does context of acquisition (naturalistic vs. instructed) and proficiency level (beginner vs. non-beginner) impact grammatical gender accuracy and agreement in L3 Spanish?

Regarding the first research question, we compared learner type (naturalistic vs. instructed) to investigate whether the context of acquisition affected accuracy of gender production and comprehension in Spanish. Non-beginner learners of two modes of acquisition performed almost at ceiling, which indicates that beyond the initial stages, learners of either environment are able to acquire grammatical gender. In this regard, we can assume that both learner groups have subconsciously learnt the syntactic phrasal agreement (Leow 2019).

As seen from the four tasks, the main differences were noticeable at initial stages, with naturalistic beginners performing better overall than instructed beginners, which contradicts previous studies (e.g., di Silvio et al. 2016; Du 2013; Llanes and Muñoz 2013; Mora and Valls-Ferrer 2012). One explanation for this could be the quality and quantity of input and exposure in the target-language environment for the naturalistic participants specifically. Though our beginner participants have been residing in Mexico anywhere from one to three years, during an informal interview with them, they confirmed that many are married to Mexican speakers and use Spanish and English on a daily basis in a variety of contexts and domains. Those who are not married, work in a Spanish-speaking environment, again increasing their amount of incidental exposure to Spanish over time, in comparison to our instructed participants. As previous research shows, those who are immersed in a naturalistic environment seem to demonstrate more fluency in their rate of speech compared to those who receive instruction in a classroom (e.g., di Silvio et al. 2016). We noticed this specifically in Task 1, where participants had to translate a passage from Russian to Spanish. This observation suggests that incidental learning at initial stages has certain benefits for language acquisition. Another explanation of naturalistic learner’s advantage can be linked to the years of exposure in the target country. In our study, our beginner participants have resided in Mexico at least a year, in comparison to other studies, where the exposure was less than nine months. For example, though Klassen et al. (2023) studied gender agreement among intermediate learners of Spanish, their data have shown that a short-term study abroad experience (25 days) did not lead to gains in grammar, but rather in lexicon and communicative competence. Similarly, Llanes and Muñoz’s (2013) study on Spanish children and adults learning English found that during two- to three-month stays in the UK, adults performed better orally and gained more lexical complexity rather than writing gains, compared to the other in-home/instructed adult group. Therefore, in the future, it will be interesting to compare participants with less than a year of naturalistic exposure to Spanish to those with more time to see at what point there is an advantage in grammatical accuracy.

5.2 Research question 2

RQ2: How do noun type (canonical vs. non-canonical; masculine vs. feminine) and gender assignment congruency with Russian affect accuracy rates in each learner group (naturalistic and instructed)?

Regarding the linguistic variables, as we predicted in our second hypothesis, our learner groups produced prototypical (masculine, canonical) and incongruent forms with greater accuracy, which supports previous studies (e.g., Foote 2015; Gamboa Rengifo 2012; Klassen et al. 2023; Montrul et al. 2008; Tararova et al. 2023). In our study, we also found that participants performed better with masculine forms than feminine, which corroborates other studies that indicate that the masculine form is used by default. We also observed the canonical form to be easier, again similar to previous studies, but it is interesting to point out that in Task 4, instructed participants performed better with non-canonical forms than canonical ones, which indicates that tasks with orthographic cues lead to better accuracy. Regarding the congruency effect, our results are different from Sá-Leite et al. (2019), who found that congruent forms would be easier to acquire, especially in initial stages of language acquisition. In our study, we did not see these results; moreover, in most of our tasks, we found that our participants performed better with non-congruent forms. One of the possible explanations for this finding could be vocabulary fluency and retrieval of more frequently used words. Though we controlled and included the vocabulary items used from our textbook, we did not test participants’ vocabulary familiarity. It is possible that they could have picked the forms (from Tasks 2 and 4) by guessing, rather than knowing noun gender. It is also possible that genetically- and typologically distant gender systems may not be as strongly integrated as closely related languages are. Paolieri et al. (2019) compared gender congruency effects between L1 Russian-L2 Spanish and L1 Italian-L2 Spanish bilinguals and found the presence of a gender congruency effect for both groups, but the magnitude of the effect in the Italian-Spanish bilinguals was greater; for example, Italian-Spanish bilinguals produced gender-congruent concrete noun phrases an average of 88 ms faster than Russian-Spanish bilinguals did in the same condition, with similar advantages for abstract nouns and for bare noun conditions. These findings indicate that typological similarity between languages might have precedence when considering a congruency effect. Since this study has not investigated any typologically similar languages, this observation merits future work.

5.3 Research question 3

RQ 3: Is there a correlation between task completion time and accuracy scores across all four tasks, and, if so, is this correlation different for naturalistic and instructed learners across proficiency groups?

We examined the accuracy scores of each group and their task completion times for the four tasks analyzed in addition to whether speed was correlated with accuracy level. Regardless of learning environment, we found that our non-beginner groups completed all tasks at a similar pace and had results near ceiling, comparable to those of our native speaker controls. Regarding the beginner learners, we found that both groups had similar task completion times for Tasks 2 and 3 and accuracy rates near or above 80 % overall. Some general observations that we made while participants were completing Task 1 specifically are that among the naturalistic group, beginners included, they tended to translate very quickly the passage given in Russian into Spanish and sounded more confident overall when providing their responses. However, as demonstrated in Section 5.2, the speed or rate of answering does not necessarily signify equal or heightened accuracy, as the instructed beginner group performed better than their naturalistic counterpart. The slightly more delayed responses from the beginner instructed group could, in part, be due to the nature of these two tasks. For both Tasks 1 and 3, learners were required to produce a response based on a passage in Russian (Task 1) or a combination of pictures with no words (Task 3). Learners were not given any orthographic input in Spanish as was the case with Tasks 2 and 4. This lack of orthographic cues in Tasks 1 and 3 required participants to retrieve the information from their own prior knowledge, thereby adding to the overall response rate for the beginner instructed learners. These results are in-line with Sá-Leite et al. (2019), who found that learners were quicker at identifying the gender of nouns that were high-frequency, canonically marked, and congruent between their L1 and L2. It is important to keep in mind, however, that the above study did not perform a comparative analysis between reaction times of classroom-based and naturalistic learners. Additionally, the participants in these studies were bilinguals and not trilinguals, as is the case with the current investigation. However, when we consider the total time that beginner instructed learners took compared to the naturalistic learners, we must also consider their relative accuracy rates. In Task 1, while beginner naturalistic learners were quicker, the beginner instructed learners had a higher rate of accuracy (i.e., 80 % overall accuracy for beginner instructed learners in comparison to 71 % for the naturalistic group). For Task 3, while naturalistic learners were also quicker, they only performed 2 % above the instructed group. Therefore, the takeaway here is that speed of responding does not necessarily yield a higher level of accuracy when it comes to gender production at the initial stages of learning, overall supporting Hypothesis 2.