Utilizing Rasch analysis to establish the psychometric properties of a concept inventory on concepts important for developing proficiency in organic reaction mechanisms

Introduction

Organic chemistry has a reputation for many students of being a gatekeeper course: difficult, complex and with some material that students may perceive as irrelevant (Grove, Cooper, & Cox, 2012). An important topic in this course is mechanisms of reactions. Organic reaction mechanisms are represented using curved-arrows which depict the movement of electrons during the process of bond breaking and bond forming. This helps visualize the reaction as the reactants are converted into products. The difficulties students face with understanding organic reaction mechanisms and the alternate conceptions they develop have been a source of interest in chemical education research since the understanding of this fundamental topic is essential to mastering the course. Most of these studies have focused on redesigning the introductory organic chemistry curriculum and understanding how students use their organic chemistry knowledge to solve mechanistic problems (Bhattacharyya, 2013; Bhattacharyya & Bodner, 2005; Flynn & Ogilvie, 2015; Grove et al., 2012; Grove, Cooper, & Rush, 2012). These studies have focused on individual students’ perceptions but there have been no intervention tools developed to assess the difficulties that an entire organic chemistry class faces and this understanding would help instructors address these problem areas. The purpose of this study is to assess and validate an inventory on concepts important for developing proficiency in organic reaction mechanisms (RMCPI).

A crucial component of developing instruments is establishing psychometric properties which provide robust information about the instrument and its functionality. Establishing the validity and reliability of instruments like concept inventories, gives valuable information to the users regarding the quality of the items which is useful when using these instruments as diagnostic tools (Barbera, 2013). For instance, instructors will be able to use the RMCPI instrument to assess their students appropriately and determine if they are prepared to begin the study of organic reaction mechanisms.

A number of instruments developed in chemical education utilize Classical Test Theory (CTT) to establish validity and reliability of the data obtained with concept inventories (Brandriet & Bretz, 2014; Bretz & Murata Mayo, 2018; Chu, Treagust, Yeo, & Zadnik, 2012; Dick-Perez, Luxford, Windus, & Holme, 2016; Kamcharean & Wattanakasiwich, 2016; McClary & Bretz, 2012; Othman, Treagust, & Chandrasegaran, 2008; Prince, Vigeant, & Nottis, 2012; Sreenivasulu & Subramaniam, 2013). CTT is utilized regularly by researchers as the first step in establishing validity and reliability because it is based on relatively weak assumptions which are easily met by test data, simple mathematical techniques are used and moderate sample sizes are needed (Hambleton & Jones, 1993). The major drawback is item and person parameters are sample dependent. Item Response Theory (IRT) on the other hand is based on strong assumptions which are not easily met by the data but if the model fits the data then item and person parameters are sample independent (Hambleton & Jones, 1993). IRT models like the Rasch model, generally require larger sample sizes and this proves to be a drawback for researchers (Hambleton & Jones, 1993). Rasch analysis helps researchers determine the probability of an examinee answering an item which gives details on the test scores of an examinee population (Hambleton & Jones, 1993). This would be very useful information since the RMCPI instrument will be administered to organic chemistry students from different universities.

Literature review

Boone and Scantlebury (2006), used a science achievement test to describe the strengths of using the Rasch model as a psychometric tool and analysis technique. They state that Rasch techniques help researchers improve the quality of quantitative measurements by assisting them in monitoring scales and improving scales over time. This is especially useful for high-stakes assessment tests that are used in various areas of science. The strengths of using Rasch analysis to develop higher quality science education instruments has been reported and this helps science educators increase the rigor of instrument development and analysis (Boone, Townsend, & Staver, 2011).

In chemical education most of the instruments developed are relatively new (Arjoon, Xu, & Lewis, 2013). Therefore reporting the validity and reliability of these instruments is of great importance. Arjoon and co-workers (Arjoon et al., 2013) investigated the psychometric evidence reported in articles published in the Journal of Chemical Education between 2002 and 2011. It was found that researchers favor reporting test-retest reliability, internal consistency, and internal structure over response process validity. Thus there is a need for chemical educators to understand the importance of the quality of assessment tools used in research and to strive to give a detailed report of the quality of assessment tools which would assist future researchers.

Rasch analysis was used to assess the complexity of middle and high school students’ knowledge in various content areas of chemistry by predicting item difficulty (Bernholt & Parchmann, 2011). Rasch analysis was also used for investigating instrument functioning and item difficulty and person ability which helped validate an ordered multiple-choice test for assessing students’ understanding of matter (Hadenfeldt, Bernholt, Liu, Neumann, and Parchmann 2013). A similar study (Wei, Liu, Wang, & Wang, 2012) used partial-credit Rasch model to determine the validity and reliability of a computer-modeling based instrument to assess students’ understanding of matter. These studies demonstrate the limited use of the Rasch model in chemical education.

The Rasch model has been utilized sparingly while designing concept inventories. Rasch analysis was used on a thermochemistry concept inventory (TCI). It was reported that all the items showed good functioning, the TCI was well targeted to the ability of students and the TCI is a measure of overall student ability providing evidence of concurrent validity (Wren & Barbera, 2014).

The Rasch model however has been used to add to psychometric data on existing concept inventories. The Chemical Concepts Inventory (CCI) (Mulford & Robinson, 2002) has been the source of further investigations. A method based on Rasch modeling to measure learning gains in chemistry was introduced (Pentecost & Barbera, 2013). Gain analysis was compared using normalized learning gain calculations and Rasch modeling on data using the chemical concepts inventory (CCI) and it was found that Rasch modeling gave information on students’ learning of specific content by comparing item difficulty and student ability on the same scale. Additions were made to the existing psychometric data on the Chemical Concepts Inventory (CCI) by conducting CTT and Rasch analysis (Barbera, 2013). The unidimensionality, item fit and, reliability were reported using Rasch analysis.

In addition to Barbera’s (Barbera, 2013) work utilizing CTT and Rasch analysis to validate a concept inventory, these two models have been utilized together in chemical education research. CTT and Rasch analysis was employed to study the impact of the flipped classroom on students’ performance and retention rate where ACS general chemistry exam scores were used from pretest and posttest data and the items on the exam were validated (Ryan & Reid, 2016). CTT and Rasch analysis was also used to test the validity of the Chemical Representations Inventory (CRI) when administered to students and instructors and it was reported that the test functioned reasonably well for the intended purpose (Taskin, Bernholt, & Parchmann, 2015).

The presence of concept inventories in organic chemistry is lacking with the exception of ACID I (McClary & Bretz, 2012), which was used to test organic chemistry students’ alternate conceptions of acids and bases, and it was validated using only CTT. Hence the use of Rasch techniques to validate concept inventories in organic chemistry is absent. Rasch analysis was however used to explore the relationship between person ability and item difficulty on a questionnaire studying teaching assistants’ pedagogical content knowledge in ¹H NMR spectroscopy (Connor & Shultz, 2018). Rasch techniques were used to validate an instrument developed to measure graduate students’ pedagogical content knowledge of thin layer chromatography (Hale, Lutter, & Shultz, 2016). These studies demonstrate that there are examples of Rasch analysis being utilized in organic chemistry education but none have utilized Rasch analysis to validate concept inventories.

These studies prove the importance of using Rasch analysis techniques in order to determine the quality of assessment tools in chemical education. Rasch techniques are useful when designing concept inventories to not only establish the reliability and validity of the instrument but to also place item difficulty and person ability on the same scale which gives information about the test scores of different examinee populations (Barbera, 2013). Rasch analysis will help establish the quality of the RMCPI instrument when administered to students from different universities and give psychometric data on an assessment tool in the field of organic chemistry which lacks such diagnostic tools.

The present study

The focus of the present study is to make use of Rasch analysis techniques to establish the construct validity and reliability of the RMCPI instrument. With this focus in mind, the study is guided by the following research questions:

1. Does person separation reliability (estimate of how well one can differentiate persons on the measured variable) support the reliability of the RMCPI items?

2. Is the RMCPI data unidimensional?

3. What is the difficulty of the items for the RMCPI data?

4. How well do the RMCPI items fit to the Rasch model?

5. Does the item difficulty data obtained from Rasch analysis compare with the theoretical construct map?

Method

Results

Item difficulty and person ability

The Rasch model can be used to compare item difficulty and person ability by placing them on the same logit scale in the form of a vertical plot referred to as a Wright map. Figure 1 shows the Wright map for the RMCPI data.

Figure 1:

Wright map of RMCPI data showing logit values for person ability and item difficulty. Note: The dashed vertical line separates the person ability data (left side) from the item difficulty data (right side).

The Wright map shows that the range of logit measures is from a low of −3 to a high of +4. This shows that the data is slightly skewed with respect to person ability. The test is difficult for the students since there are five items (8, 9, 10, 21 and 23) that are above the ability of the students. Item difficulties range from about −2.8 logits for the easiest item to about 3.2 logits for the most difficult item. There are some items with a similar difficulty level like items 3, 5 and 15. The items 8, 9, 10, 21 and 23 are not targeted well and hence the difficulty of these items was estimated with high imprecision.

Item fit

Model fit is evaluated using two residual analysis, infit (weighted) and outfit (unweighted) statistics and this helps in identifying problematic items. Table 1 shows the infit and outfit statistics for the RMCPI data.

Table 1:

Fit statistics for the RMCPI data.

Item no.	Measure	SE	Infit MNSQ	Infit ZSTD	Outfit MNSQ	Outfit ZSTD
10	3.17	0.73	1.10	0.4	2.92	1.7
23	1.93	0.45	1.14	0.6	2.30	2.1
17	−1.16	0.36	1.27	1.5	1.48	1.9
14	0.74	0.34	1.14	1.0	1.34	1.4
4	−1.57	0.39	1.11	0.6	1.16	0.6
13	−0.79	0.34	1.14	1.0	1.15	0.9
8	2.15	0.49	1.09	0.4	1.14	0.4
21	1.93	0.45	.94	−0.1	1.08	0.3
9	1.56	0.41	1.07	0.4	1.04	0.2
11	0.86	0.35	1.05	0.4	1.01	0.1
16	−2.07	0.44	1.00	0.1	1.04	0.2
15	−1.03	0.35	0.96	−0.2	1.02	0.2
25	0.62	0.34	0.98	−0.1	1.01	0.1
24	−0.14	0.33	0.94	−0.5	1.00	0.0
19	0.51	0.34	0.98	−0.2	0.94	−0.3
22	−0.68	0.34	0.96	−0.2	0.98	0.0
12	0.51	0.34	0.96	−0.3	0.92	−0.4
7	0.40	0.33	0.94	−0.5	0.87	−0.7
20	−0.79	0.34	0.93	−0.5	0.84	−0.9
2	−1.89	0.42	0.90	−0.3	0.72	−0.7
18	0.86	0.35	0.90	−0.7	0.80	−0.8
3	−1.03	0.35	0.86	−0.9	0.82	−0.9
1	−1.16	0.36	0.85	−0.9	0.74	−1.2
5	−1.03	0.35	0.82	−1.2	0.73	−1.4
6	−1.89	0.42	0.78	−0.9	0.81	−0.4
M	0.00	0.39	0.99	0.0	1.11	0.1
SD	1.40	0.08	0.12	0.7	0.48	0.9

1. SE, Standard error; MNSQ, mean square value; ZSTD, standardized z-score.

The acceptable mean square (MNSQ) values for both infit and outfit statistics for a low stakes multiple-choice test is 0.70–1.30 (Bond & Fox, 2015). All the items have infit values which are within the acceptable range. Items 10, 23 and 17 have outfit values which are outside the acceptable range but since their infit values are well within the range, they are not considered to be problematic. The ZSTD data for both infit and outfit statistics for most items is within the acceptable range of −2.00 to +2.00 (Bond & Fox, 2015). This suggests that the items fit adequately to the dichotomous Rasch model.

Item fit can also be analyzed by a bubble chart. Figure 2 shows the bubble chart for the RMCPI data. It can be seen from the bubble chart that most of the items function as expected according to Rasch model as they are within the quality control line of −2 to +2 (Bond & Fox, 2015).

Figure 2:

Bubble chart for RMCPI items. Note: The bubble chart has item difficulty in logits on the y-axis and the standardized infit statistics on the x-axis. The bubbles represent the items.

Discussion

The PCA of the RMCPI data indicate that the instrument is unidimensional and this together with the fit statistics and the bubble chart indicate that the RMCPI data accurately fits the assumptions made by the dichotomous Rasch model. This suggests that there are no discrepancies between the observations and expectations for the RMCPI items. All items had good infit statistics which indicates good item functioning for students within the ability range of that item. These results, when compared to the theoretical construct map which is illustrated in Figure 3, establishes the construct validity of the RMCPI instrument. This suggests that the RMCPI instrument is measuring the unidimensional construct of students’ understanding of concepts important for developing proficiency in organic reaction mechanisms.

Figure 3:

Theoretical construct map for concepts related to RMCPI items. Note: The theoretical construct map shows the concepts related to RMCPI items ordered such that lower level concepts are fundamental concepts and higher level concepts are more advanced concepts.

The Wright map indicates that the test is difficult for the examinee population and the correlation between student ability and item difficulty is not adequate and the difficulty of items 8, 9, 10, 21 and 23 were estimated with high imprecision. Items 8, 9 and 10 are related to the concept of bonding which is a lower level concept that students learn in general chemistry. Additionally, a fundamental understanding of the concept of line drawings is also needed to answer item 10. These findings are consistent with previously reported results (Cooper, Grove, & Underwood, 2010) which indicate that students have difficulties with representational competence in chemistry. Items 21 and 23 are related to the concept of stability of intermediates and stereochemistry, respectively, which are higher order concepts that students traditionally find difficult in organic chemistry. One way to overcome this would be to increase the sample size in an attempt to see if these items are targeted well when more participants are recruited to represent a larger heterogeneous sample. Items 8, 9 and 10 may have to be modified to make them more explicit before collecting data on a larger sample. Items 21 and 23 may have to be eliminated because they are concepts that might not be introduced early on in the first semester of organic chemistry.

The development of a concept inventory is a multi-step, iterative process. Even after the alpha version of an inventory is finalized, the inventory may be subject to revisions based on data collected from additional institutions. Since the development of a concept inventory is an iterative process, further information regarding problematic items can be obtained by conducting additional interviews with students and by consulting the opinions of experts regarding the problematic items and the inventory as a whole. Rasch analysis can be used as one of the sources of information in this iterative process. The present manuscript is aimed at exhibiting the utility of Rasch analysis for obtaining information about items during the development phase of a concept inventory in chemical education where Rasch analysis is under-utilized.

The person separation reliability was not adequate and lower than the accepted value of 0.8 (Bond & Fox, 2015) and this indicates that there are problems with the consistency in the RMCPI items’ placement of students along the logit scale. This again could be due to the sample size and a larger sample size might help in overcoming this problem. This could additionally be attributed to the number of items in the inventory and having more items under each concept, might improve the person separation reliability. Additionally, interviews with students might also help give insight on whether students are actually using their conceptual knowledge to answer the questions (Wren & Barbera, 2013).

Limitations and future research directions

One of the limitations of the present study is the sample size and the generalizability of the sample. The sample is above the minimum sample size for dichotomous data which is 30. In order to increase the confidence interval from 95 % to 99 %, the minimum sample size needed would be 150 (Linacre, 1994). The other limitation of this study is that data was collected from only one university and therefore limited in its generalizability. Data will have to be collected from other universities to see if the items function well and if the results are reproducible (Barbera, 2013). A larger sample size would also be beneficial for measuring the item difficulty more precisely. Another limitation of this study is that the instrument contains only 25 items. More items could potentially help improve person separation reliability.

The RMCPI items will be modified based on the obtained results and more items will be added to the existing ones before collecting data on a larger sample. Classical test theory (CTT) will be utilized on this data to obtain the difficulty and discrimination and these results will be compared to the results obtained from Rasch analysis. The use of CTT and Rasch techniques will help add to the validity and reliability data for the RMCPI instrument. Rasch analysis helps assess properties beyond what CTT is capable of and this would contribute more to the psychometric properties of the instrument (Barbera, 2013). Distractor analysis will be used to test the quality of the distractors and remove under-utilized distractors.

Conclusion

Rasch analysis was utilized to establish the validity and reliability of the RMCPI instrument. The RMCPI items function well within the Rasch model and the unidimensionality of the instrument was established. This psychometric data is important while using the instrument because it establishes the fact that the instrument is measuring only the intended construct.