The development and validation of a C-test and a pseudo-character test for online CFL assessment

2.1 Online foreign language assessment

Although remote teaching and testing have come under the spotlight in recent years as a result of the emergency response to the Covid-19 pandemic, online or computer-assisted testing in the foreign language classroom is not necessarily a new phenomenon. However, the mass migration of courses and tests to the online space in recent years, accelerated by Covid-19, warrants further research as the trend in at-distance and/or blended courses continues to rise.

Indeed, online tests can come in the form of international proficiency tests, university exams, or even in-class quizzes and assessments. With advances in technology in general, language teachers and instructors worldwide incorporate online assessment tools (such as Quizlet, Kahoot) to assist student learning, which are reported to be motivating and beneficial to both teachers and students in the foreign language classroom (e.g., Huyen, 2022). Certainly, such online tools are particularly beneficial when assessing a language that has limited opportunities to practise in a given country (Alsied & Pathan, 2013), while a study conducted by VanPatten et al. (2015) demonstrated no difference in scores between an in-class and online test in a particular communicative and proficiency-oriented language course. Overall, using online tools to assess foreign languages has proven to be convenient, reliable, and efficient (e.g., Wagner, 2020).

However, during the development of emergency online CFL assessments during the Covid-19 pandemic, serious issues relating to online assessment design emerged such as: an increase in opportunity for students to cheat (Daniels et al., 2021); the need to generate anti-OCR (Optical Character Recognition) documents for assessment (Wang & East, 2020); issues in testing handwriting (ibid.); and both students’ and teachers’ technology skills (Burns et al., 2020; Jin et al., 2021).

Indeed, one of the main issues in online CFL testing lies in the difficulty of testing handwriting and therefore character formation. Certainly, typing was seen to be used in online testing during the pandemic (e.g., Wang & East, 2020), however, as Guan et al. (2011) demonstrate, typing is not an appropriate substitute for assessing character formation. In addition, there is a paucity of research in the area of online CFL assessment methods, despite a huge increase of online CFL teaching and learning triggered by Covid-19 (e.g., Wang & East, 2020; Zhang, 2020). As a result, it is vital to go back to the basics of language assessment development to ensure that a successful online CFL assessment can be developed and validated.

2.2 Language assessment development

According to Zhang (2016), the process of developing a language test has three stages: (1) design and planning; (2) execution and administration; and (3) post-test analysis.

Understanding examinees is important for developing language assessments (Zhang, 2016), while Bachman (1990) prioritizes the test’s purpose in test design. To achieve the predetermined objectives, it is vital to select an appropriate exam type (e.g., proficiency tests, diagnostic tests, achievement tests) (Brown, 2004). The number of items and scoring procedure can also be significant factors since they influence the weighting of content (ibid.). Item writing and delivering the test are the main steps of test execution and administration. Alderson et al. (1995) note a good test writer should have specific experience in teaching and assessing students in the area being tested, while creativity is also essential as poorly constructed questions can damage the alignment of the assessment process (Tavakol & Dennick, 2011). Post-exam analysis techniques are then used to improve the quality and reliability of assessments (ibid.). Statistical analysis, item analysis, validity, and reliability are all important factors in evaluating a newly designed test, with the validity test being the most significant.

Although validity is generally viewed as a unified concept (Bachman, 1990; Messick, 1989), it is more practical to see validity as segmented into subcategories or steps that may easily be approached with manageable tools (Dobrić, 2018). The major and complementary types of evidence that need be gathered during the validation process include construct validity, content validity, and criterion-related validity (Bachman, 1990; Weir, 2005). In achieving criterion validity, an older, well-established test is typically used as a criterion, as it also serves as an indicator of the ability being tested (Bachman, 1990; Weir, 2005). The result of the comparison is usually expressed as a correlation coefficient, with higher coefficients demonstrating closely related and reliable tests (Alderson et al., 1995). Content validity is defined as “a conceptual or noun statistical validity based on a systematic analysis of the test content to determine whether it includes an adequate sample of the target domain to be measured” (Davies et al., 1999, p. 34). To establish content validity, experts are called upon to examine or rate the test tasks or items with a data collection instrument (Alderson et al., 1995; Feng et al., 2020). Conducting construct validation involves testing hypothesized relationships between test scores and abilities empirically (Bachman, 1990). Alderson et al. (1995) introduces five different approaches to achieve construct validity, including confirmatory factor analysis (CFA).

2.3 Language assessment development in the context of CFL

In the context of CFL, language assessment development requires deep consideration given the unique features of the language, particularly in the written form. The Chinese writing system is logographic and morpho-syllabic, whereby characters correspond to both syllables and morphemes (DeFrancis, 1989). This makes it difficult for L2 learners to determine whether new characters are formed based on pronunciation, meaning, or both (Osborne et al., 2018). Characters are also composed of radicals which can be further decomposed into strokes (Liu et al., 2007). Previous studies have shown that stroke patterns (Chen et al., 1996), and radicals (Anderson et al., 2013) both can function as units of character perception. For CFL students, it is imperative to be able to recognise Chinese characters as a whole as well as the identifiable units within, meaning that assessment design is vital in the CFL classroom.

The HSK 2.0 test, introduced in 2010, is China’s standardised language proficiency examination and is administered globally (McNaughton, 2005). According to Chinese Testing International (2022b), its main purpose is to assess Chinese language ability in various contexts, including daily, academic, and professional life. The HSK 3.0 has been introduced in 2021 with reforms in vocabulary distribution (three levels with nine bands) and the introduction of translation and handwriting on the test paper (China Education Center, 2023). While the participants of the current research are registered to a module aligned to HSK 2.0, with the introduction of handwriting exercises in the HSK 3.0 test paper, the need for the current research is again highlighted, particularly in reference to the online space.

In addition to nationally administered tests, several studies have focused on four language skills – reading, writing, speaking, and listening – and two types of language knowledge – vocabulary and grammar (Zhang, 2017). Given the unique features of Chinese, Chinese language teachers and researchers should pay more attention to developing new types of test questions that specifically target the unique features of Chinese. However, as Zhang (2017) notes, the majority of studies in developing Chinese testing are published in Chinese, making it difficult for the international community to keep abreast of developments in the field. Furthermore, Zhang (2017) also concludes that there is a paucity of literature on the assessment of reading Chinese – including the involvement of Pinyin on test papers – as well as in assessing character writing with computers. To address these limitations, we have identified that two test types – C-test and pseudo-character – are particularly promising.

While the cloze test has been demonstrated to be a valid instrument for measuring Chinese language proficiency (Feng et al., 2020) and the C-test, a variation of the cloze test, is predominantly used in English tests, the C-test is yet to be reported on in the context of teaching/learning CFL. Second, due to the fact that CFL students are generally not equipped with adequate amounts of characters (Zhou, 2007), as well as experiencing considerable difficulties identifying, writing, and using characters (Wu et al., 2017), educators should prioritise the development of tests that can assess character knowledge. The pseudo-word test may therefore have the potential to be modified as a pseudo-character test for the CFL context.

2.4 C-test

The cloze test was first introduced in the 1950s (Taylor, 1953) as a method of language testing (Oller, 1972, 1973). A considerable number of studies have demonstrated that the cloze test can not only serve as a standardised test to measure students’ language proficiency (Alderson, 1979; Feng et al., 2020; Oller, 1973) but also has a wide range of correlation coefficients with well-established tests such as TOEFL and HSK (Bachman, 1985; Feng et al., 2020). A number of variations of the cloze test have been developed over the past few decades (Zhang et al., 2005), one being the C-test developed by Raatz and Klein-Braley in the 1980s (Raatz & Klein-Braley, 1981).

While making the overall test shorter, the developers of the C-test “damage” words rather than delete them (Klein-Braley, 1997, p. 64). As an example of a C-test adapted from the DET, Figure 1 shows a paragraph that is presented to the test takers, with certain words missing letters. The test taker must be able to understand the context and spell the words correctly in order to pass this section.

Figure 1:

Sample of C-test adapted from Cardwell et al. (2022).

As a stable, economical, reliably scoring, and easy-to-write/administer test, the C-test has been widely used by test developers (Grotjahn, 2002; Harsch & Hartig, 2016). Compared with other reduced redundancy tests, the C-test comes top in terms of difficulty level, reliability, validity, and factorial validity (Klein-Braley, 1997). Additionally, given the nature of the C-test design, the answers are more definite, rather than being subjective as in cloze tests (ibid.). Similarly, the frequency of damaged words allows more classes of words to be tested (ibid.). Besides being highly correlated with listening and speaking tests, the C-test has also been shown to be used to measure students’ overall language ability (Eckes & Grotjahn, 2006). Therefore, as long as reasonable adaptations are made, the C-test may also be a useful tool for assessing Chinese learners holistically across phonetics, vocabulary, grammar, and discourse.

2.5 From pseudo-word test to pseudo-character test

The Yes-No Vocabulary Test (YNVT) format is widely used by language learners due to its efficiency, easy item development, and straightforward scoring system (Nation, 1990). However, one of the drawbacks of YNVT is students’ guessing behaviours (Pellicer-Sánchez & Schmitt, 2012). To address this issue, the pseudo-word test, developed by Zimmerman et al. (1977), is commonly used. In such sections, test takers are presented with a set of written real words mixed with pseudo words designed to appear like real words, and must choose the correct ones (Cardwell et al., 2022) (see Figure 2). In the context of English, the pseudo-word test can predict listening, reading, and writing skills (Milton et al., 2010).

Figure 2:

Sample of written YNVT pseudo-word test adapted from Cardwell et al. (2022).

In the context of CFL, pseudo-word tests have been adapted to pseudo-character tests. It is widely agreed that pseudo-characters serve as valuable stimuli in the exploration of the cognitive processes underlying character recognition of Chinese readers. For instance, the pseudo-characters can be used as testing items for monitoring the progress of Chinese learners’ orthographic awareness (Chung et al., 2012). Chang et al. (2022) developed a Chinese pseudo-character/non-character producing system (CPN) based on traditional Chinese characters and the results of their study demonstrate that highly proficient Chinese readers are better able to identify pseudo-characters. However, using pseudo-characters has also been subject to some criticisms. First, there is no consensus among researchers regarding the definition of pseudo-characters. Chen et al. (1996) defined pseudo-characters as the swapping of two real characters and the creation of two non-existent combinations of orthographic units that always display lexical radicals in their legitimate positions. Similarly, Sze et al. (2014) created pseudo-characters by swapping semantic components within each pair while Kuo et al. (2015) created pseudo-characters that resemble real Chinese characters in structure but do not exist in Chinese by combining a high-frequency semantic radical with one or more simple characters. In addition, Shi (2000) categorized students’ Chinese character writing errors as non-characters (非字), fake characters (假字), and characters with errors (别字), which can resemble pseudo-characters that are artificially created. However, numerous studies have shown that presenting errors to students in the long term can lead to anxiety, confusion, and reinforcement of errors (Khansir & Pakdel, 2018). Lastly, with a lack of tools for precisely creating pseudo-characters, it is a time-consuming task to create such (Chang et al., 2022).

While there has been some research into pseudo-characters in the context of CFL, very little academic research has been conducted on whether pseudo-characters could be used for achievement tests, how well they work, and the potential issues. This study thus aims to develop a simplified Chinese pseudo-character test for a Chinese beginner’s achievement test and address any issues therein.

The literature review has revealed some gaps in developing CFL assessment. First, the paucity of research in the area of Chinese language assessment development, particularly published in English, demonstrates the need for the present study. CFL teachers need valid and reliable testing instruments as well as essential knowledge on assessment development. In addition, C-test and pseudo-character tests designed for CFL students and teachers are lacking. Finally, innovative assessment methods that can also be applied to the online space should be explored in the CFL context. This study therefore aims to fill these gaps by addressing the following research questions:

3.1 Test type and scope

The current study develops an achievement test that contains two sections: (1) a C-test section and (2) a pseudo-character section for a 12-week (36 teaching hours) module. This module covers content from the absolute beginner level and caters to students with very little or no prior knowledge of Mandarin Chinese. The HSK Standard Course Textbook Volume 1 (Jiang, 2014) is the main material used. Upon successful completion, participants have acquired basic communicative competence and knowledge of pronunciation and tones, basic sentence structures, useful daily expressions, core vocabulary, and high-frequency-appearing characters broadly aligned to the HSK 1 test.

3.2 Item writing for C-test section

The C-test comprises 282 Chinese characters with 15 blanks (see Figure 3). All characters were selected from the HSK 1 textbook and supplementary materials used by the teachers during the module. Words that are not required to be recognised by participants, such as 西班牙 (Spain), are prompted with Pinyin.

Figure 3:

The C-test developed for the current study.

The researchers not only attempted to provide definite answers when writing C-test items, but also more detailed and eye-catching instructions, such as “②④⑤⑥⑧⑩⑪⑮: Please fill in the blank with ONE character ONLY”.

In addition, the current study adapted the concept of “damaged” items (家, 打, and 机). Specifically, one or two components of the chosen characters are deleted, and participants are asked to fill in the missing components. For example, participants are asked to complete the character 家 (family) in the following sentence: “我(③_____)有五口人 (there are five people in my family).” Furthermore, a similar approach is used in the testing of Chinese words whereby researchers “damaged” the words by deleting one character and asked the participants to fill in the missing characters. For example, the participants are required to provide the missing character “起” in the following sentence: “他昨天和他的朋友一(⑪____) 去了西班牙(Xībānyá) (He went to Spain with his friends yesterday).”

Most CFL teachers teach phonology when students first begin learning Chinese, but do not continue doing so afterward (Ye, 2003). Thus, incorporating a pinyin test into an achievement test for CFL beginners is essential. The initials “s” and “r”, as well as the finals “ui” and “e”, which have been shown by studies to be easily mistaken by students (Ran & Yu, 2019) are examined. Students are required to write the pinyin according to the characters “ 岁” and “ 热” indicated in brackets in question ① and question ⑭.

In fact, the content areas in the newly designed C-test correspond to a range of learner proficiencies from basic (e.g., phonology, character, morphology, and syntax) to more advanced (e.g., cohesion and grammar). The morpheme and word categories contained four items testing verbs, prepositions, adverbs, and concrete nouns, representing 26.66 % of the C-test. Regarding grammar, progressive action, complex sentences, and the particle “了” were included. Additionally, some of the questions required students to fill out the correct answers by understanding the contextual discourse logically. For example, in the sentence “我的爸爸很(⑨_____)吃水果”, only students who understand the meaning of the sentence that follows “今天早上,他吃了很多苹果 (This morning, he ate a lot of apples)” will be able to fill in the correct answer “爱 (love)” or “喜欢 (like)” in the previous sentence.

3.3 Item writing for pseudo-character section

As pseudo-character definitions and classifications have not yet reached a consensus, and as participants in this study are CFL beginners with limited understanding of Chinese characters, the pseudo-characters are thus defined in the current study as non-existent characters that are artificially constructed to look like real Chinese characters by changing orthographic units, such as strokes and radicals.

The following considerations led to the selection of the five characters in this study; 看 (to look), 做 (to do), 爱 (to love), 热 (hot), and 哪 (which) (see Figure 4). As this study focuses on developing an achievement test, the syllabus was strictly followed when developing test items. The characters were taken from the main textbook used by the participants, while the EBCL (European Benchmarking Chinese Language) Proposed List of A1 characters and the vocabulary list in New Practical Chinese Reader Volume 1 (Liu, 2015) were also consulted to ensure that the characters chosen are essential for CFL beginners, and their parts of speech are consistent.

Figure 4:

The pseudo-character test developed for the current study.

All Chinese characters selected are among the top Chinese characters in the Modern Chinese Character Frequency List, ranked as follows: 看 (76), 做 (246), 爱 (394), 热 (606), and 哪 (652). The construction of the characters – the radicals and strokes – was also taken into consideration. The number of strokes for most Chinese characters ranges from 6 to 13 (Anderson et al., 2013), with the characters selected in this study within this range at nine strokes (哪 and 看), 10 strokes (爱 and 热), and 11 strokes (做). Most modern Chinese characters are semantic–phonetic compound characters consisting of two major components (Anderson et al., 2013). As compound characters offer more possibilities for constructing pseudo-characters, all characters selected in this study are compound characters. Apart from combining two orthographic units that swap positions in Chinese characters to form pseudo characters (e.g., Chen et al., 1996), this study makes further attempts to produce pseudo characters, as shown in Table 1.

Table 1:

Creating pseudo-characters for the current study.

Action	Examples
	Original character	Result
1. Adding or deleting strokes	爱
2. Repositioning the position of orthographic units within a character	做
3. Utilising similar-shaped components to replace the originals	热
4. Utilising components with related meaning to replace the originals	哪

Finally, as “recognising a Chinese character generally means decoding both its pronunciation (phonology) and meaning (semantics)” (Zhang et al., 2022, p. 1), this study requires students to not only identify a real character from four options (three of which are pseudo-characters), but also to provide its Pinyin and English meaning.

3.4 Participants

A total of 53 participants took part in the study, 27 males and 26 females. Their average age was 19.39, with a range of 17–22. All participants were enrolled in a beginner’s Chinese module in an Irish university (where the researchers were teachers) and were therefore recruited on account of convenience. As this is an introductory-level course open to all students in the university, the majority of participants (62.26 %) were first-year students, while 15 participants (28.30 %) and five participants (9.43 %) were second- and third-year students respectively. In addition, the module was taken by 49.17 % of the participants as an elective, with the remainder taking it as a core or option module closely related to their academic programme.

3.5 Item weighting

In this study, the items are weighed according to their complexity and characteristics, and the overall score is calculated by adding up each questions’ marks. The pseudo-character section consists of five Chinese characters containing 15 items in total. For each character, the participants were tested on three separate items of form, phonology, and meaning, with an equal weighting of 0.5 points for each item. The 15 items on the C-test are more challenging than those in the pseudo-character section as they are designed to measure a student’s comprehensive understanding of Chinese. Consequently, the C-test is assigned higher marks; one point per item. Since most questions involve Chinese character writing, participants should not lose their entire mark for minor Chinese character errors – this would defeat the purpose of some questions. A detailed description of item weighting can be found in Table 2.

3.6 Data collection

In order to conduct the test systematically, the following steps were taken (Figure 5).

Figure 5:

The process of data collection.

Firstly, after ethical approval was received, all participants were informed that the results of the C-test, the pseudo-character section, and the criterion-referenced test would be used anonymously in the current research once the participants granted permission. Secondly, a C-test and a pseudo-character exercise were assigned 2 weeks prior to the test. This gave the participants an opportunity to practise the new exercises and to ask any questions. Although none of the participants expressed confusion regarding the new exercises, a few instructions were later rephrased to ensure clarity.

A sample HSK 1 test was carried out during regular class hours 1 week prior to the new test as a criterion-referenced test. Statistically, a criterion-referenced test only requires a certain number of samples (Zhang, 2016). A sample size of over 32 % can be considered a reasonable for criterion-referenced tests based on previous research (Feng et al., 2020). In the current study, 70 % of the participants (37 out of 53) took the sample HSK 1 test.

A week after the module was completed, the newly developed achievement test – which formed part of participants’ final grade – was administered. Detailed instructions were announced by the examiners as well as printed in English on the test paper. To minimise the risk of pseudo-character results being contaminated, this achievement test was divided into two testing phases. In the first phase, a pseudo-character section with 15 items was to be completed in 5 min. All participants were required to complete the items on a separate test paper and turn it in before proceeding to the second phase. In the second phase, 15 C-test items were included on the test paper.

3.7 Scoring

In terms of scoring, the two researchers who developed the new test also took on the role of raters. As a precaution against bias, both raters marked the tests with student identity completely concealed. To minimise the possibility of unfairness in test paper scoring due to individual differences between raters, one rater was assigned to grade the C-test and a second rater was solely responsible for grading the pseudo-character section. Following the first round of scoring, the test results were reviewed and re-calculated by two raters to ensure accuracy.

3.8 Methods of post-test analysis

This study aims to examine whether the newly developed pseudo-character test and C-test can accurately assess student achievement. To address this, it adopts mixed methods to assess descriptive statistics, difficulty, discrimination power, validity, and reliability. SPSS software (version 26) and the online platform SPSSAU (2021) are used for quantitative analysis, while a qualitative analysis is performed to examine the content validity of the new test.

4.1 Descriptive statistics

Table 3 shows the descriptive statistics for the new test. The mean score of 13.84 represents a fair level of achievement (61.45 %). In this study, the standard deviation (SD) of 5.5079 indicates that the test seems to be capable of distinguishing between students with different language levels, as the participants’ scores cover a relatively wide range.

However, there is a significant gap between high scores and low scores – the lowest score is 2.25 and the highest score is 22.5. Table 3 displays the individual descriptive statistics for the C-test and the pseudo-character section. The mean score of 5.4848 in the pseudo-character section is relatively high since the overall score is only 7.5. Correspondingly, due to the high scores achieved by most participants, the SD of the pseudo-character section is quite low, at 0.7071. In the C-test, the mean score is 8.3302 and the SD is 4.1855, which is more satisfactory as students learn at different rates and a dispersion around a mean is to be expected for an achievement test.

In this study, a normal distribution analysis was conducted as a prerequisite to further analysis. As shown in the Q–Q plot (Figure 6), the distribution of scores appears to be normal. However, to ensure accuracy, a Shapiro-Wilk test was also employed. This resulted in Sig.=0.074 (alpha level=0.05), confirming the null hypothesis that the test results of this study come from a normally distributed population.

Figure 6:

Q–Q plot of scores.

4.2 Item analysis

Item analysis examines student responses to individual test items in order to assess the quality of those items and of the test as a whole (Gajjar et al., 2014). Testers can use item analysis to determine which items should be adopted, revised, or discarded. Difficulty (henceforth p-value) and discrimination power (henceforth DP) are the two main factors assessed in the current study.

4.2.1 Difficulty (p-value)

The items on the C-test were scored on a scale of 0–1, while the pseudo-character marks ranged from zero to the full 0.5 marks. Therefore, p-value formulas (see Figure 7) for a constructed-response or performance item are employed in this study (Brookhart & Nitko, 2019; Zhang, 2016).

Figure 7:

Formula of difficulty index.

The range and recommendation of p-value used in this study are commonly used by scholars for educational domains (Johari et al., 2011) (see Table 4). As shown, the majority of the 21 items (70 %) are of acceptable difficulty, with a p-value of 30–80 %. Only one item in the C-test falls below 0.3, while eight items – one from the C-test and seven from the pseudo-character section – are above 0.80. The median difficulty level of the newly designed achievement test is 0.64, slightly more difficult than expected (ideally between 0.5 and 0.6 [Escudero et al., 2000]).

Table 4:

Level of difficulty (p-value).

Level of difficulty	p-Value range	Question number	N	%
Very difficult	<0.30	C15	1	3.33 %
Moderate	0.30–0.80	C1/C3/C4/C5/C6/C7/C8/C9/C10/	21	70.00 %
		C11/C12/C13/C14
		P2/P5/P6/P8/P11/P12/
		P14/P15
Very easy	>0.8	C2	8	26.67 %
		P1/P3/P4/P7/P9/P10/P13

1. ^aC refers to the items in the C-test section. ^bP refers to the items in the pseudo-character section.

4.3 Validity

4.3.3 Construct validity

According to Bachman (1982), if the test results match the model constructed when the questions were designed, this demonstrates that the test has some structural validity. Confirmatory factor analysis (CFA), widely regarded as a powerful tool for establishing evidence supporting construct validity of language tests, is adopted in this study. First, the model for the C-test is initially based on the categorisation of language competence, while the pseudo-character test model is based on the selected characters (see Table 2). In addition, according to previous research (Bachman, 1982; Fan et al., 2014), a higher-factor model incorporating one superordinate factor and a series of subordinate factors upon which a specified subgroup of items exist provide a satisfactory fit for language test data. As a result, higher-factor hypothetical models for the C-test and the pseudo-character section have been developed and are schematically illustrated in Figures 8 and 9.

Figure 8:

Hypothetical model for C-test.

Figure 9:

Hypothetical model for pseudo-character section.

To determine the models’ fit in the present study, two criteria are employed. Criteria one consists of selected global fit indices: (1) the comparative fit index (CFI); (2) the minimum discrepancy per degree of freedom (CMIN/DF); (3) the root mean square error of approximation (RMSEA); and (4) the standardised root mean square residual (SRMR). For CFI, values greater than 0.9 represent a good fit (Awang, 2012). Kline (1998) suggests an acceptable fit when CMIN/DF is <3. For RMSEA and SRMR, values of 0.05 or lower are indicative of a good and excellent fit (Burns & Patterson, 2000). A second criterion is the value of two psychometric property indicators, the composite reliability (CR) and the average variance extracted (AVE). If the minimum value of CR exceeds 0.60, AVE values above 0.40 are acceptable (Fornell & Larcker, 1981).

The online platform SPSSAU (2021) was employed to assess the fits of the models in this study. According to the Table 7, the C-test has a good overall model fit with χ ²/df=1.153, CFI=0.996, SRMR=0.073, and RMSEA=0.054. Pseudo-character test results were also satisfactory with χ ²/df=1.370, CFI=0.914, SRMR=0.090 and RMSEA=0.084.

Table 7:

Fit indices for the models.

	CMIN/DF (χ 2/df)	CFI	SRMR	RMSEA
C-test	1.153	0.996	0.073	0.054
Pseudo-character section	1.370	0.914	0.090	0.084
Reference values	<3	>0.9	<0.05	<0.05

Regarding the second criterion, satisfactory AVE values were obtained for the pseudo-character section; however, a slightly unsatisfactory CR value of 0.584 was obtained for the character “哪”. For the C-test, the results were more complicated. For the factors Phonology, Character & Component, and Morpheme & Word, the results were satisfactory. For the factor Discourse & Cohesion, although a fairly satisfactory value of 0.422 was obtained for AVE, a slightly unsatisfactory value of 0.587 was produced for CR. A critical finding was that the AVE value for the factor Grammar was 0.113 and the CR value was 0.294, suggesting that the four variables under the grammar factor were not as well converged (see Table 8).

Table 8:

AVE and CR results for the hypothetical models.

C-test section			Pseudo-character section
Factor	AVE	CR	Factor	AVE	CR
Phonology	0.494	0.659	Character 做	0.481	0.703
Character & Component	0.423	0.683	Character 爱	0.666	0.854
Morpheme & Word	0.558	0.834	Character 看	0.496	0.732
Grammar	0.113	0.294	Character 热	0.617	0.806
Discourse & Cohesion	0.422	0.587	Character 哪	0.458	0.584

1. Reference values: CR > 0.60, AVE > 0.40.

4.4 Reliability

Besides referring to the quality of the test scores, reliability also refers to their consistency. Internal consistency reliability, which focuses on the consistency of a test’s internal elements, is assessed in the current study. In theory, Cronbach’s Alpha can be applied when test items are weighted and vary from 0 to 1 (Jin & Qi, 2018). A higher Cronbach’s Alpha indicates greater relevance between items and greater internal consistency of reliability.

Table 9 reveals that the overall reliability of the new achievement test is good, with a Cronbach’s Alpha of 0.907. This indicates that the new achievement test is reliable to a degree of 90.7 %. Two of the subscales, the C-test and the pseudo-character section, have Cronbach’s Alpha coefficients of 0.875 and 0.840, respectively, which also indicates high levels of reliability.

5.1 The development and accuracy of the C-test and pseudo character test in the CFL classroom

As for the overall design of the achievement test, it differs significantly from the HSK 1 test in which the knowledge to be tested is randomly distributed over 40 questions that are not contextually relevant. In contrast, the newly developed achievement test is clearly composed of two main types of question, with the C-test assessing students’ overall competence in phonology, Chinese characters, vocabulary, grammar, and discourse, and the pseudo-character section focusing on beginners’ understanding of Chinese characters. In the redesigned C-test, the concept of “damage” has been incorporated and items are generated with damaged characters, words, and phrases. In addition, it tests Pinyin knowledge, which is rarely included in CFL assessments. The detailed instructions and broad range of content areas are among the innovations of the newly designed C-test, while all judges noted innovation to be one of the greatest strengths of the pseudo-character section. The attempts to select, define, and produce pseudo characters in this study can serve as inspiration to CFL teachers. Certainly, some drawbacks of these two question types were identified during the design process. For instance, for beginner students, learning Chinese characters is already a challenging task, and the pseudo-character section may further complicate the learning process and result in student anxiety. In addition, since the design of a programme to automatically generate pseudo-characters also requires the assistance of professionals, it is not currently possible to generate pseudo-characters in a time-saving manner using a single font.

Looking closely at the post-test analysis, a normal distribution with acceptable mean scores and SD provides evidence that the test can differentiate language levels to some extent. Regarding the item analysis, the overall difficulty of the test paper is slightly higher than expected. In particular, the pseudo-character section seems to be too easy, with seven questions classified as “very easy”. In contrast, the difficulty of C-test items is fairly good, with only one question ranked “very easy” and one “very difficult”. Approximately 80 % of the DP items are considered very good items, and none should be eliminated completely, since none have negative values. Satisfactory Cronbach’s Alpha values also indicate that the newly designed achievement test is highly consistent and reliable.

At the same time, the three major and complementary types of validity tests also reveal promising and interesting results. Firstly, having significant correlations with the sample HSK 1 test provides robust evidence for the criterion-related (concurrent) validity of the newly designed achievement test. Both the C-test and the pseudo-character section show significant correlations with the listening section and the reading section of the criterion. In other words, after further exploration, these two novel tests may have the potential to serve as a proficiency test for evaluating students’ overall language competence. With regard to construct validity, the CFA provides evidence that both the C-test and the pseudo-character section fit the hypothetical higher-order model fairly well, indicating that the newly developed achievement test reflects the original intentions of the test designers. Due to the fact that the four variables in the grammar factor in the C-test were not sufficiently converged, improvements can still be made.

The current paper has therefore addressed the research questions in both demonstrating how a C-test and pseudo-character test can be developed in the context of CFL, as well as identifying the accuracy of such in assessing students’ learning outcomes.

5.2 Implications for online testing

The digital-age learning environment shifts away from preparing language learners for traditional written tests to creating innovative, comprehensive assessments that reflect their progress and achievement. The newly developed C-test and pseudo-character test also have a high potential to be adopted in online assessments.

The artificially created pseudo-characters could be displayed as non-editable and non-Googleable images, thereby further reducing cheating and making OCR impossible. Also, although typing is not an appropriate method of assessing the composition of characters (Guan et al., 2011), the pseudo-character test mitigates this shortcoming by examining students’ knowledge of components and strokes. In addition, the pseudo-character test is not exclusively limited to formative assessments but can also be utilised as a summative assessment, a homework assignment, and an interactive online classroom activity or game. For example, by using a smartphone or tablet, students can identify the real characters among the pseudo-characters or even provide short answers detailing incorrect components or strokes in online polling applications, such as Kahoot!, Socrative, and Plickers. Pseudo-character tests also provide inspiration for technological specialists, for instance, programming techniques can be utilised to generate pseudo-characters automatically.

The C-test is also ideal for online testing with the potential to be developed as a computer-adaptive test (CAT). Another advantage of the C-test is the definite answers. With objective and exclusive answers, the C-test can be scored without much subjective judgement and can therefore be administered online with an auto-scoring system. Moreover, technologies such as machine learning and Natural Language Processing (NLP) are widely employed to automatically create cloze-test items as well as building large-scale cloze datasets (Correia et al., 2012; Xie et al., 2017). In everyday settings, the C-test can also be applied to online CFL classrooms. Using collaborative digital whiteboard platforms such as Padlet or Miro, students can complete C-test exercises in groups. The design concept of tile-matching video games, such as Tetris, can be employed in C-test exercises, whereby the sentence changes colour once all the blanks in the sentence have been correctly filled in by the students. Moreover, the C-test can also be designed as a simple matching-up or drop-down online practice.

Overall, the newly designed C-test and pseudo-character section have great potential and advantages to be adapted for the online space. The next step of the research will be to introduce them to online platforms, thus making them available for both the at-distance and face-to-face CFL classroom.