PQtutor, a quasi-intelligent tutoring system for quantitative problems in General Chemistry

Introduction

Learning to solve quantitative problems in college chemistry courses is a challenge for many students (Gulacar & Fynewever, 2010). Students have to integrate general problem-solving skills, chemical insight, unit algebra and error propagation into their solution, and learn how to tackle multi-step problems on paper. Learning happens mainly when students work out problems on their own and study for exams (Leinhardt, Cuadros, & Yaron, 2007). On the other hand, feedback is important in learning, and is most effective if it is timely (Crippen, Schraw, & Brooks, 2005). Some colleges provide drop-in help to support student’s independent study, offer supplemental instruction or recitations. In the absence of these, students who get frustrated or stuck while doing homework seek various forms of help. While there is a growing market for paid online supplements such as intelligent tutoring software (Wilson & Kennedy, 2017) or online human tutoring, not all students can afford these. There is also a growing amount of free help available on the internet’s question-and-answer sites, but these contain material that can be misleading, at an inappropriate level, or plain wrong. Overall, there is a lack of low-cost resources with high pedagogical value (Figure 1).

Striving to provide a free but valuable tutoring system to students, I decided to leverage two open resources, the OpenStax Chemistry textbook (Flowers et al., 2015) and the online calculator PQcalc developed in my previous work (Theis, 2015). The calculator PQcalc (short for Physical Quantities calculator) works through a browser, and input is plain text entered via the keyboard. Each input command starts with a name, describing the quantity one is defining or calculating, followed by a value or mathematical operations on quantities already defined. Figure 2 shows an example. To calculate the density ρ of a sample with a volume V of 54.3 mL and a mass m of 81.4 g, you would type commands into an input box and see the results after typing “return” or pressing “go”. Variable names and math are typeset in the output for better readability. For example, fractions are written with a fraction bar, and subscripts are used for chemical formulae and for variable names. The calculator takes care of the lower level mechanics such as number arithmetic, unit algebra and error propagation and catches errors such as trying to add a mass to a volume. The second open educational resource, the OpenStax chemistry textbook, contains material typically taught in General Chemistry I and II and provides hundreds of worked example problems as well as more than a thousand end-of-chapter problems. The online tutor PQtutor described here integrates selected problems from this textbook into the existing online calculator to create a free online homework system.

Figure 1:

Different ways of getting help with homework. Students seeking help while trying to solve quantitative problems have a choice of options differing in cost and in value. There are expensive options (various forms of tutoring) that have been shown to support learning, cheap options that are of questionable pedagogical value, and unethical services that sell homework solutions. However, there is a lack of homework help that has high pedagogical value while being affordable to all (corresponding to the upper left quadrant labelled “?” in the graph).

Figure 2:

User input (left) and output generated by PQcalc (right) for a one-step density calculation.

Objectives

The way PQtutor interacts with students is guided by three objectives. The first objective is to provide a tool that is easy to use and accessible, both while students are enrolled in a course and after. In an age where many students sell their textbooks after they complete a course and lose access to electronic resources just after learning how to use them, it is important to provide tools that remains available. To allow cheap development with limited resources, PQtutor is only quasi-intelligent in the sense that most of the knowledge lies in the model answers (Aleven, Mclaren, Sewall, & Koedinger, 2009). Also, to keep costs down, it does not maintain a model the student’s knowledge state (Desmarais & Baker, 2012). Despite this lack of artificial intelligence, it can assist learning nonetheless because students and instructors are intelligent (Baker, 2016).

The second objective of PQtutor is to preserve the merits of pencil-and-paper calculations (Smithrud & Pinhas, 2015) while providing closer connections to the chemistry context. On paper, students can summarize chemical information using formulas or equations, they can work out algebra and interpret the results of their calculations. Typically, however, the calculations are done with a hand-held calculator, removing the context (including the units) of the problem at hand. PQtutor allows students to practice quantitative problem-solving without losing sight of the chemistry context. Integrating the scientific context into the calculator is intended to bridge the divide between algorithmic and conceptual aspects of learning chemistry (Nakhleh & Mitchell, 1993; Cracolice, Deming, & Ehlert, 2008).

The third objective of PQtutor is to provide timely, useful, and friendly feedback to students. Asking for help triggers reminders of general problem-solving strategies first, followed by more specific hints that do not give away the answer (the numerical answer is in the textbook). The intent is to keep frustration and boredom at bay while encouraging students to think on their own (Baker, D’Mello, Rodrigo, & Graesser, 2010).

User interface

PQtutor works through a browser and has a user interface similar to the PQcalc calculator (Theis, 2015). Different parts of PQtutor are accessed through different links students receive from their instructor. The exercises currently available are all adapted from the OpenStax Chemistry textbook (Flowers et al., 2015). PQtutor displays worked examples from the textbook that demonstrate how to solve a certain type of problem (Figure 3 and link to PQtutor in Supplementary material S1). In the textbook, worked examples are paired with similar follow-up problems to which the numerical answer, but not the solution path is given. PQtutor displays these follow-up problems (or any other question) as well, and students enter commands into the input box to work out these problems (Figure 4 and link to live demo in Supplementary material S1). If students are unsure of how to proceed with a problem, or they just want feedback on what they already entered, they can request help from a virtual study group by just pressing go without any input. Before students submit their solution, they have to answer a multiple-choice question to reflect on the calculation and its meaning. A short video (Supplementary material S2) illustrates the mechanics of solving an exercise and submitting it to PQtutor (and the instructor).

Figure 3:

PQtutor showing a worked example. Links (underlined, blue) are to the chapter summary and to a related worked example.

Figure 4:

PQtutor posing a homework problem. The question is shown in maroon on the top. To work out the problem, students type commands into the input box.

Feedback and assessment

Tutors give feedback to students to scaffold their problem solving attempts. To achieve learning, the amount of information and the type of information given by the tutor has to be carefully chosen. Students should receive feedback, but not in a way that hinders self-regulated learning (Aleven, Roll, McLaren, & Koedinger, 2016; VanLehn, Siler, Murray, Yamauchi, & Baggett, 2003). In PQtutor, it is up to the student to seek help (by pressing go without entering any input) and to decide how much help to ask for. The feedback is structured around a problem solving model that guides students towards independent problem solving and self-regulated learning.

Virtual study group as pedagogical agents

Tutoring in PQtutor occurs through interaction with a virtual study group that acts as a learning companion (Chou, Chan, & Lin, 2003). The study group provides encouragement and specific help on the question at hand, in a sometimes playful but always polite manner (Wang et al., 2008). There are four characters in the virtual study group, Q, B, C, and L (Figure 5). Each one of them represents a different set of skills, strategies or tactics to tackle quantitative problems. Student Q has general quantitative problem-solving skills such as considering the relationship between knowns and unknowns, performing dimensional analysis, picking up on common patterns (e.g. two-state problems), realizing the need to solve for an unknown and some general troubleshooting strategies. Student B has a firm grasp of the big picture tools chemistry has to offer, such as the periodic table, symbolic notation (chemical formulae and equations), conventions for naming different types of quantities, and data tables available in the textbooks. Student C is the expert on the specific chapter, including the new concepts introduced, the common pitfalls associated with the chapter, and the chapter-specific tools the textbook offers. Student L, finally, likes to look up how others have worked out similar problems, primarily by referring to the worked example. Comments from the study group come in two flavors, corrective feedback or pumps for more information (Graesser, Lu, Jackson, Ventura, & Olney, 2004); corrective feedback points out problems with the work so far, while pumps are leading questions suggesting what the student might do next.

Figure 5:

Members of the virtual study group and their respective special power.

Pumps

Human tutors will prompt students to take the next step towards solving a problem, giving just the right amount of scaffolding so that students can figure out solutions by themselves. This requires substantial knowledge about the psychology of learning in general and about the discipline and the student specifically, none of which PQtutor possesses. Instead, PQtutor turns the model solution into a graph of dependencies (Figure 6), matches the student answer to the solution graph, and identifies reachable sub-goals. Then, characters in the virtual study group proceed to ask guiding questions about missing data from the question, missing data from other sources, relationships between knowns and unknowns, and sub-goals and goals (see video S3 and Supplementary material S1 for link to live demo). Pumps are given in order from general to specific. Students who take pride in figuring out a solution on their own will not receive help they did not ask for (i.e. as soon as one of the prompts gives them an idea on how to proceed, they can try on their own). Students who are trying to game the system and bottom-out the hints will receive plenty of hints, but the solution is never revealed directly (again, the numerical solution is available to students upfront). In fact, the bottom-out hint is a metacognitive one (Casselman & Atwood, 2017), suggesting to seek help from the instructor.

Figure 6:

Solution graph for the example shown in Figure 4.

Connecting an answer to the bigger picture

When students are ready to submit their solution to a homework question, they have to answer one final multiple-choice question before moving on. This final “What does the answer mean?” question was inspired by the last step in the COAST problem-solving recipe used in W.W. Norton’s chemistry textbooks (Gilbert, Kirss, Foster, Bretz, and Davies 2018). The purpose is to connect the numerical answer to the chemistry the student is learning in a given chapter, and to foster meaningful problem solving (Kennedy-Justice et al., 2000). The questions come in three flavors. Some ask whether the magnitude (or sign) of the result could have been anticipated from an estimate (Figure 7 top). Others ask for an interpretation of the calculated result (Figure 7 center). Finally, some go after common misconceptions (Figure 7 bottom). In all cases, the question asks the student to make a connection between the result of their calculation and concepts that they are familiar with or have just learned, and to evaluate their answer.

Figure 7:

Examples of different types of “What does the answer mean?” questions.

Textbook integration

A large portion of the effort behind PQtutor was to adopt the problems of the textbook in a format suitable for it, and writing “What does the answer mean” questions. A benefit of presenting the problems within PQtutor are links to dynamic textbook resources, including an electronic version of the text itself provided on the libretext site (Halpern & Larsen, 2017). To further support making connections between the problem and chemical knowledge, chemical data is available within PQtutor (e.g. molar masses, table of quantities and units) and via links to outside sources (e.g. periodic table, thermodynamic data). The process of adopting a wide range of problems and writing model answers led to ideas how to simplify and expand the ways students can document their problem-solving ideas in Pqtutor. Consequently, I incorporated these ideas into the user interface to improve it.

Adopting problems

It is fairly straightforward to adopt existing questions or author new ones, requiring minimal effort working within PQtutor. First, you provide a description of the problem in plain text. It will be formatted just like user input in PQtutor. Second, you formulate a solution that PQtutor understands, i.e. provide input for a correct solution. All logical intermediate steps should be shown to maximize feedback given to students. Third, you write a multiple-choice follow-up question, with the correct answer marked by an asterisk. For example, Figure 8 shows the text you would enter into a special question authoring form (see Supplementary material S1 for a link to the form) for the example 3.17b used throughout this paper. PQtutor has no other knowledge about this question, and generates the feedback and prompts from the model solution and from the logic of the underlying mathematics. It does have some general knowledge about dimensions, units, and commonly used symbols for certain quantities covering the concepts covered in General Chemistry I and II, so it makes sense to use the symbols for quantities according to IUPAC conventions (Nič, Jirát, Košata, Jenkins, & McNaught, 2009) and listed in a table available in PQtutor (Supplementary material S1).

Figure 8:

Instructor input to define a new homework question.

Pilot study using PQtutor

To explore how the tutoring system works in practice, I piloted PQtutor in a General Chemistry I course. In total, 33 students participated working on 8 different questions, and 178 answers were submitted. During the course, students received credit for participation. After the semester concluded, I replayed all the submissions and fine-tuned how the virtual study group reacts to input that matched the model solution, and to input that did not. Moreover, this first field test with students provided important pointers on how one might rigorously test the impact of PQtutor on the development of quantitative problem-solving skills in students of general chemistry. Finally, this anecdotally showed how students fared solving homework problems outside of class using PQtutor.

Using PQtutor in a course

Because PQtutor is an open educational resource, there is no registration or other time-consuming step to start working with it, and it can be used in a course as the sole online homework system or in combination with other tools without incurring additional cost to students. However, instructors should put some thought into how to introduce PQtutor to students, and how to integrate it into their course so it supports the learning objectives.

Discussion

The goal in creating the PQtutor software was to provide affordable tutoring software for students learning the quantitative aspects of chemistry. The user interface and the feedback are designed to allow students to integrate chemistry context and problem solving strategies while working on their homework. A pilot study in a General Chemistry I course suggested that PQtutor can play a valuable role in helping students to learn how to solve quantitative problems, but also that it has some limitations and room for further development.

Conclusions

The PQtutor software described here provides students with an accessible and affordable homework tutor that supports meaningful problem solving and gives timely feedback. It was designed to fill a need for high-quality supplements to existing high-quality open textbooks in chemistry. It focuses on quantitative problem solving, a set of skills that many students encounter as the main stumbling block to success in learning chemistry at the college level. When integrated into coursework in a thoughtful manner by an experienced instructor, it can play a role in increasing student success, helping to provide a solid foundation in quantitative problem solving as a preparation for careers that rely on generating and assessing data in a quantitative manner.

Supplementary information

S1: Links to live demonstrations of the software (attached below)

S2: Video of the user interface of PQtutor

S3: Video of how the virtual study group gives hints

S4: Video of how the virtual study group gives corrective feedback

Open source code available at github.com/ktheis/pqtutor. The online software is hosted at pqcalc.pythonanywhere.com.