Don’t we know enough about models? Integrating a replication study into an introductory chemistry course in higher education

2.1 Epistemological considerations

The exploration of models and modeling in chemistry is inherently tied to epistemological considerations (Erduran, 2001; Klein, 2003). Simultaneously, within science education, there exists a tendency to view models as both direct representations of target systems and as subject-oriented processes for making sense of these target systems (Rost & Knuuttila, 2022). However, this conflation of perspectives is contradictory. On one hand, when a scientist attempts to depict an atom and argues for a structural relationship within the representational vehicle (e.g., a sketched circle with dots), there is no direct means of justifying that specific structure. On the other hand, if the scientist posits that a model retains value as long as it accurately represents the phenomenon under investigation (Stowe & Esselman, 2022), one could question how this approach generates subject-specific knowledge in the first place.

This article’s position aligns with the notion of models as epistemic artifacts (Knuuttila, 2011, 2021). This perspective alleviates the burden of striving for more or less precise representational endeavors or falling into the trap of regarding models solely as practical constructions. Instead, it allows for the construction of scientific knowledge.

2.2 Empirical considerations

In order to effectively mitigate drop-outs in chemistry courses, a thorough understanding of students’ comprehension of models and modeling, as well as its connection to their prior knowledge, is imperative (Chittleborough & Treagust, 2007). Merely assessing students’ understanding of models is far from adequate when it comes to teaching any science subject (Schwarz et al., 2022); however, it is a mandatory requirement (Constantinou et al., 2019). This necessity arises from two primary reasons. Firstly, Fleischer et al. (2019) demonstrate a direct influence of prior knowledge on students’ intentions to discontinue their studies. Secondly, the comprehension of the particle level precedes the acquisition of chemistry knowledge (Sumfleth & Nakoinz, 2019). Regrettably, the endeavor to integrate prior subject knowledge, and knowledge of models and modeling within the teaching process encounters various challenges, encompassing the cognitive demands placed on learners (Johnstone, 1993) and the epistemological perspectives of teachers (Oh & Oh, 2011). These challenges accumulate, contributing to the inherent difficulty of learning chemistry (Reid, 2021b). In conclusion, without gaining insights into how to engage with students’ preconceptions (Duit & Treagust, 2003), lecturers risk merely reproducing a superficial presentation of the philosophy of science during introductory chemistry lectures.

2.3 The SUMS inventory

In order to systematically comprehend students’ understanding of models, the utilization of an assessment tool becomes necessary for both researchers and teachers. Particularly in the context of university settings with large and diverse first-year cohorts, where individual assessment and acknowledgment of students’ perspectives are resource-intensive, a quantitative instrument would prove beneficial. The starting point for this endeavor is the SUMS inventory (Treagust et al., 2002), which has gained recognition as a questionnaire designed to quantitatively assess students’ comprehension of the properties and applications of models in science (Mathesius & Krell, 2019). Distinct from performance-oriented approaches such as the Framework for Modeling Competence (Constantinou et al., 2019; Upmeier zu Belzen et al., 2019), SUMS captures learners’ perspectives regarding models and their utilization by scientists. In this regard, Treagust et al. (2002) propose five latent factors to elucidate the understanding of models.

1. Models as multiple representations (MR) refers to the diverse forms that models can assume, illustrating how distinct model objects can convey a single target system. MR endeavors to encompass the range of representational variations associated with a particular target system. Example: “Many models represent different versions of the phenomenon.”

2. Models as exact replica (ER) refers to assessing the perceived proximity between a model object and its corresponding target system. ER entails establishing a structural connection between the model’s structure and that of the target system, thus determining the degree of structural resemblance. Example: “A model shows what the real thing does and what it looks like.”

3. Models as explanatory tools (ET) refers to the capacity of model objects to facilitate understanding of a target system by enabling access to non-visible objects and processes. ET serves to articulate learners’ comprehension through the utilization of a model as a representational means. Example: “Models help create a picture in your mind of the scientific happening.”

4. Uses of scientific models (USM) refers to the processes employed by scientists when utilizing models, specifically focusing on how models and modeling are implemented in scientific practice. USM encompasses the diverse ways in which models are employed within the scientific practices of various fields. Example: “Models are used to make and test predictions about a scientific event.”

5. Changing nature of models (CNM) refers to the capacity of model objects to be revised, indicating that models are not deterministic but rather susceptible to modification in response to new data or invalidated predictions. CNM encompasses information regarding the degree to which learners perceive models as either permanent or subject to change. Example: “A model can change if there are new findings.”

The SUMS inventory has recently garnered attention through replication studies conducted in the USA and Chile, resulting in mixed findings. Villablanca et al. (2020) present a successful replication of the original structure after translating the questionnaire into Spanish. They suggest further research to better comprehend the variance within the respective factors. However, their use of Cronbach’s α as a reliability measure has faced substantial criticism in the empirical literature over the past decade (Hayes & Coutts, 2020; Padilla, 2019; Taber, 2018; Yang & Green, 2011).

In contrast to the successful replication in Spanish, Lazenby and Becker (2021) conclude that there is insufficient evidence to support the use of the SUMS inventory in its original form as an assessment tool. They were unable to replicate the original factor structure and reported cross-loadings during their exploratory factor analysis. Despite making theoretically justified modifications to the scales, they were still required to reject their measurement models. These findings clearly indicate the limited transferability of the instrument, leading the authors to caution researchers and practitioners against using the SUMS instrument in contexts outside of its original development without additional evidence of validity and reliability.

Therefore, the main objective of this article is to fulfill this need by providing an accurate account of the translation process of the SUMS inventory, along with modifications that maintain sufficient fidelity to the original instrument. This approach ensures a transparent proposal and analysis of the psychometric structure. The research questions and proposed structure are presented as follows:

2.4 Research questions

1. Can the factor structure of the SUMS inventory, as originally proposed by Treagust et al. (2002), be replicated in a translated (German) version?

2. If the proposed factor structure holds, to what extent do the respective scales demonstrate the necessary reliability to serve as valid measurements for subsequent analyses?

2.5 Hypothetical psychometric structure

Four distinct models were agreed upon to integrate the aforementioned information into a testable empirical framework (Figures 1 –4). The first model replicates the structure proposed by Treagust et al. (2002), as the inconclusive findings from prior research resulted in retaining the assumption of five correlated factors. The second model was derived from theoretical considerations regarding the ER factor. The empirical challenges encountered thus far in relation to ER are theoretically justified, given the intention to conceptualize models beyond mere depictions. For example, “Many models are used to show how it depends on individual’s different ideas on what things look like or how they work.” (MR) should not correlate with “A model shows what the real thing does and what it looks like.” (ER). Hence, a proposal is put forth to test a structure comprising of only four correlated factors, namely MR, ET, USM, and CNM. The remaining two models represent expanded versions of the aforementioned structures, encompassing their respective five or four factors in addition to an overarching SUMS factor. By incorporating a higher-order factor model (Brunner et al., 2012), this approach signifies the assumption of a comprehensive factor of general model understanding. This overarching factor comprises subfactors that align with similar yet discernible directions, thereby reflecting Treagust et al. (2002) foundational assertion regarding students’ comprehension of models: “[They] have their own personal and unique understanding of the role of scientific models in science built up through their life experiences.” (p. 358)

Figure 1:

Reproduced psychometric assumption of SUMS by Treagust et al. (2002). Five correlated factors constitute views on models and modeling.

Figure 2:

Adapted psychometric assumption of SUMS by Treagust et al. (2002). Excluding ER leads to four correlated factors for views on models and modeling.

Figure 3:

Extended psychometric assumption of SUMS following Treagust et al. (2002). Five non-correlated factors are connected via an overarching general factor.

Figure 4:

Extended and adapted psychometric assumption of SUMS following Treagust et al. (2002). Excluding ER and introducing an overarching general factor constitute the theoretical assumption of views on models and modeling.

3.1 Context

This investigation contributes to a German university’s empirically grounded program aimed at facilitating meaningful comprehension of the submicroscopic level among first-semester students. The primary objective is to minimize the influence of prior knowledge on academic success in chemistry (Fleischer et al., 2019; Hailikari & Nevgi, 2010). Given the constraints of limited resources for individualized teaching in large cohorts, it becomes essential to establish efficient and generalizable measures that can be integrated into the learning environments.

With respect to chemistry-related lectures, the university has made the decision to review the content of their lectures based on empirical research in chemistry education. Consequently, the teaching staff aims to enhance their understanding of how students grasp the particle level in the context of scientific modeling, moving beyond the mere presentation of declarative knowledge and fostering a proactive learning experience.

3.2 Translation process

For the translation process, three individuals were involved, two of whom translated the items into German. One of these individuals was the first author of this paper, while the other was a native English speaker. Additionally, a German native speaker with an academic degree in English contributed to the translation process. Following this, the items were re-translated into German by the native English speaker, and deviations were carefully examined. Multiple rounds of discussion were conducted, which did not yield significant concerns regarding content validity or potential sources of misunderstanding. Throughout the translation process, precautions were taken to address common sources of validity constraints in quantitative tools. For instance, it was ensured that item texts containing conjunctions such as and were avoided, in accordance with recommendations by Haladyna et al. (2002). As a result, the item pool became more extensive than the original instrument, as items containing conjunctions were split into two separate items.

3.3 Target group & data collection

Following the transition of the questionnaire into an online format, data were collected from a total of 181 first-year students enrolled in three distinct study programs: physics (N = 60), dietetics (N = 79), and chemistry engineering (N = 42). The digital questionnaire was completed by the students using their personal devices during a lecture session, in the presence of the lecturer.

3.4 Data analysis

After extracting the data from the university’s database, the obtained Excel sheet was transferred to the statistical software R (R Core Team, 2020) for further analysis. An exploratory data analysis was initiated (Bryer & Speerschneider, 2016; Lüdecke et al., 2022; Revelle, 2019) with a specific focus on examining item correlations within the five hypothesized factors. The reporting of these correlations adheres to the guidelines outlined by Funder and Ozer (2019). Subsequently, confirmatory factor models were estimated and evaluated, which were then summarized in structural equation models (Hu & Bentler, 1999; Rosseel, 2012; Ziegler & Hagemann, 2015).