Development of a self-test for undergraduate chemistry students: how do students solve tasks on basic content knowledge?

3.1 Goal

For this study the following research questions will be investigated and answered:

1. How do the students rate the tasks?

2. How is the students’ content knowledge?

3. How do the students rate their own competences regarding the tasks?

3.2 Sample

The study was conducted in summer 2022 in the course “Organic Chemistry” for non-major chemistry students. These students study life sciences or nutritional science and are in their first year. The course consists of a lecture (3 h/week), a seminar (2 h/week), and a laboratory internship (one week). The testing of the tasks in preparation for the self-test has been done in the accompanying seminars with two groups of students (one Mondays and one Wednesdays). The tasks on reaction mechanisms were tested during the laboratory internship. In the seminars, depending on the group between 14 and 30 students (Mondays) or 12 and 35 students (Wednesdays) have actively taken part in the study. Because attendance of the seminar (and of this study) is voluntary, the students do not always attend regularly or do not stay till the end of the seminar (sometimes because of train or bus schedules). Therefore, during the semester the tasks have been solved at the beginning of the seminar instead of at the end, which led to better attendance in the evaluation. The students also switch between different groups (beside the two groups used for this study, other groups have been offered online). The students have been informed of the goal of the study and the use of the data. Ethical guidelines were followed. At this university, the approval of the Institutional Review Board is not required.

3.3 Design of the tasks

The tasks have been designed as multiple-choice or multiple response (Cook, 2023) questions because multiple-choice items are considered an effective form of educational assessment (Gierl et al., 2017). To choose the right answer option, the students must use their knowledge and problem-solving skills (Gierl et al., 2017). Butler (2018) lists several best practice tips for multiple-choice and multiple response items: they should not be too complex, they should engage specific cognitive processes, “none-of-the-above” and “all-of-the-above” as response options should be avoided, three plausible response options should be used, and the tests should be challenging but not too difficult. Therefore, the following design elements for the multiple-choice and multiple response items used in this study have been used:

1. The verbs telling the students what do to were printed in bold to support the students.

2. If more than one answer is correct, the plural form is underlined (this design element has been added during the semester, because of students’ mistakes while solving the first tasks).

3. The questions were formulated as clearly and easily understandable as possible.

4. The answers are as short as possible.

5. It should be avoided that all statements are correct.

6. The answers were arranged alphabetically regarding the key terms as for example electronegativity.

For each task, a questionnaire with closed and open items has been designed and used. All elements of the task are shown exemplary for the task on chemical bonding in Table 1. For this paper, the task was translated from German to English.

During the course, the questionnaires were evaluated after using the task at hand in the seminar. Then, as a formative assessment, the results were used for rephrasing the following tasks, especially if the tasks were very similar (for an example see Table 2). By using the original wording in task 3, the students did not understand that they had to tick only one box out of four; they did not identify the task as multiple-choice task, but often solved it as a multiple response task. The new task makes clear that only one box should be ticked, and that the description only fits for one answer.

3.4 The coding process

For evaluating and discussing the assessment of the results, the statistical software SPSS has been used for the closed items. The open items have been coded by both authors according to the work of Kuckartz (2016) using the method of qualitative content analysis. Both coders discussed and compared their assignment till 100 % inter-rater agreement was reached (Saldaña, 2013). For all codes and examples see Supplementary Material.

4.1 Research question 1: how do the students rate the tasks?

Although literature-based design features for the tasks have been used, the students were asked to rate the tasks. Especially the items “I have understood the task description” and “I knew, what I had to do” were of interest, although other reasons than the wording and design of the task can influence their rating, for example their reading competences or distractions during the reading and solving process. Therefore, the open item “for the task I have the following suggestions for improvement:” was included in the evaluation.

For the item “I have understood the task description” the arithmetic mean over all tasks was in the range of 3.66 till 3.88 (therefore between agree and strongly agree). For the item “I knew, what I had to do” the results are quite similar: M = 3.61–3.88 (for all results, see Table 3). Correlations for both items are statistically significant (r _S = 0.749**) which is not surprising.

For the further development of the tasks, the results of these items alone are not sufficient. However, the evaluation of the open item “for the task I have the following suggestions for improvement:” leads to the required information (for a summary see Figure 2, for all results see Supplementary Material).

Figure 2:

Summary of the codes for all categories.

Especially for the first category (Periodic Table of Elements), the students commented on the design of the task. As described above, those comments have been used for improving the design. Because comments on the design cannot be observed for the following categories it seems that students were more content with the design.

During the whole semester, students commented on the language of the tasks (also see Figure 2). Here an evaluation is more complex, because the reasons can be the formulations used in the task, but can also be of personal nature. Therefore, all statements coded with “language” have been evaluated anew. The statements were assigned to “task” or “personal” depending on the wording. For the first two categories (Periodic Table of Elements and Chemical bonding) the number of assignments to both codes are quite similar (10 for task and 9 for personal). For the last two categories more codes were assigned to task (7 versus 3). The changes in the formulations that have been made (for an example see Table 2) seem to result in a decreasing number of statements, especially for personal statements as for example “For me the third formulation was not clearly understandable”. Overall, the students’ ratings of the tasks support the intended use in the self-test; therefore, all rephrased and redesigned tasks will be used in the digital learning environment.

4.2 Research question 2: how is the students’ content knowledge?

For the development of the digital learning environment, information on the students’ content knowledge (knowledge of technical terms and concepts and their application) are of interest. It is shown in the literature that students need content knowledge and learning opportunities for their application (Carmel et al., 2015; Hermanns, 2021). If students have problems with certain technical terms, scaffolds can be used (Keller & Hermanns, 2023). For the digital learning environment, a glossary and learning games were planned. For all tasks of the self-test the items “I knew all technical terms”, “I knew the meaning of the technical terms” and “I can explain all technical terms” were included in the questionnaire. Table 4 shows for each item the highest und lowest ranking (all other rankings of technical terms were in between).

All lowest ratings are between disagree and agree; the highest between agree and strongly agree. This means that for all tasks some of the technical terms are either unknown to the students or their meaning is not clear. For identifying which terms are problematic, the questionnaires included the item: “For the following technical terms, I did not know what they mean”. For all tasks evaluated in the study, the students have written down technical terms (for an overview see Supplementary Material). They also often added the information that they had heard of the technical term but did not know exactly what it meant. However, students also added the information that they knew and understood all technical terms in the task at hand. Most terms were named twice, as for example valence bond, electrophile and nucleophile or resonance. The terms named most are period and bond-line structure (5×), heteroatom and Lewis base (7×), and ionization energy (9×). For the task where the students rated the technical terms best, they did not write down any unknown term. This can be seen as a hint for the fitting of their rating and naming of unknown technical terms. However, the digital learning environment should offer explanations for technical terms. As students are always heterogeneous groups, the explanations for the technical terms should not be visible for all students, because this would only unnecessarily enhance the cognitive load (Sweller, 1988) during the solving process. Therefore, the results of this evaluation are used for the development of a glossary and of suitable digital games. Students who need an explanation can use the glossary, and for learning the terms and their explanation the digital games.

For assessing the quality and the level of the tasks, the percentage of completely correct answers have been determined for all tasks. They vary between 0.0 and 88.2 %. For multiple-choice items (only one correct answer) the percentage varies between 11.5 and 81.0 %; for multiple response items (2–4 correct answers) between 0 and 71.4 %, and the other tasks (sequence or allocation) between 44.4 and 88.2 % (for all results see Supplementary Material). The students solved the multiple response tasks less successful than the multiple-choice tasks. Therefore, those tasks have been analyzed in more detail. First, both tasks with a success rate of zero percent will be discussed. For the task “octet rule” three out of four answers were correct. No student chose all these three answers. The answer “the octet rule is a special rule of the common noble gas rule” was chosen by few students. Because this knowledge is not necessary for using the octet rule correctly, the task has been changed accordingly. This answer has been removed and was replaced by a wrong answer. Therefore, in the newly designed task, only two correct answers should be chosen. The task “correct statements regarding acid-base reactions” has also been redesigned, because only two students chose the correct answer “reactions between Lewis bases and electron deficient compounds are acid base reactions”, this answer has been rephrased into “acid base reaction can also be described as reaction of a nucleophile with an electrophile”. All tasks of the self-test have been analyzed. For those tasks, where the students did not choose the correct answers, the answers have been rephrased or replaced if the answer was assessed as unnecessary.

However, the results show that the students’ prior knowledge shows gaps that can be closed by working with the learning environment. On the other hand, many answers the students have given are correct (very often, for multiple response tasks, they did not choose all correct answers). This is also important, because otherwise the students’ frustration with their own competences could be a hindrance for working with the learning environment. If there are too many gaps, their motivation most likely would decrease. The self-test seems to fit for the intended learning group which are students of the first two years of study. Therefore, the improved self-test will be evaluated as part of the complete learning environment by students in their first or second year in the winter term 2023–2024.

4.3 Research question 3: how do the students rate their own competences regarding the tasks?

As discussed for research question 1, the students rated the tasks. For this, a questionnaire with a four-item Likert scale (Allen & Seaman, 2007) has been designed and used (see Table 1). For comparing the item “I could solve the task without problems” with the students’ solving of the tasks, the students’ answers have been coded as completely correct or not correct. For the multiple-choice tasks, for 29.6–69.2 % of the tasks the students’ estimation was fitting. For the multiple response tasks, for 21.4–85.7 % of the tasks the students predicted their solving of the task correctly. For the tasks where the students had to put items in a sequence or allocate meanings to terms, the concordance was best (between 50.0 and 82.4 %). The most homogeneous results could be observed for the tasks the students solved during their laboratory internship. Here, the students’ concordance of their assessment with their own solving shows an arithmetic mean of 58.6 with a standard deviation of 9.3. One reason can be that during the laboratory internship only those students attend that are seriously studying or are still studying. During the second year, many students stop with their studies. It can be assumed that students with problems or who are not interested belong to the group that stops with their study and not to the group that takes part in the laboratory internship.

Overall, not all students estimate their own knowledge correctly. Depending on the task, between 14.3 and 78.6 % overestimate their knowledge as observed repeatedly in other studies and known as Dunning-Kruger effect (Kruger & Dunning, 1999). The students’ ratings of the technical terms in relation to their rating of the item “I could solve the task without problems” have also been evaluated by calculating Spearman correlations. For all three items regarding the technical terms, statistically significant correlations can be observed (r _S = 0.407**; 0.477** and 0.514**). The students’ overestimation of their knowledge is therefore consistent for those items that can be seen as relevant for the solving process. Therefore, the self-test includes feedback for all tasks. The students receive information about the results of the individual tasks and use this information for deciding which training materials to choose.