A situated cognition model for clinical reasoning performance assessment: a narrative review

Assessee-case/patient interaction

The “case” element in a clinical reasoning performance assessment method typically refers to the stimulus format, the way a problem is presented to an assessee. Stimulus formats include written or computer-based vignettes, low or high-fidelity simulation, and standardized or real patients. A dramatic study by Nendaz et al. [15] demonstrated the significant impact that the stimulus format can have on clinical reasoning performance assessment. Assessees (n=91) that were randomized to receive only a chief complaint and had to ask questions to obtain the history and physical exam findings to determine the diagnosis had significantly lower diagnostic accuracy at all levels of experience (students, 10% versus 36%; residents, 47% versus 81%; and general internists, 59% versus 100%) as compared with assesses (n=64) who received a complete clinical vignette. All the assessees who only received the chief complaint obtained more data than that provided in the vignette but many failed to obtain the key information required to make the correct diagnosis. The authors speculated that assessees receiving the vignette more readily generated the correct hypothesis (i.e. the vignette triggered more relevant hypothesis generation).

Similarly, a systematic review of clinical reasoning performance showed that certain stimulus formats can alter physician behaviors by causing increased data collection [16]. When physician clinical reasoning performance on similar medical problems was assessed on paper cases versus real patient cases through chart audits, some researchers found a significant increase in data collection (e.g. lab testing) in paper case formats [16]. Several study design elements appeared important in leading to differences in clinical reasoning performance between the two stimulus formats. These included: (1) limited instructions about the nature of the task, (2) how closely the task on the paper cases mirrored real-life clinical reasoning tasks, (3) differences in the response formats (see Table 1 for definition) between the paper cases and the actual clinical documentation, and (4) differences in data analysis of the clinical reasoning performance of interest [16].

From a traditional information-processing perspective, the clinical reasoning performance assessment variation seen in these studies does not make sense. The performance should not be impacted by the stimulus format, given that the clinical content and knowledge required was the same in both formats. On the other hand, situated cognition theory provides a clear framework for understanding these performance and assessment differences because it accounts for contextual differences, such as changes in stimulus format. By comparing two different stimulus formats, both studies showed that assessee-case format interactions have the potential to significantly alter clinical reasoning performance assessment. The situated cognition perspective provided by these results raises the possibility that the traditional approach of vignette-based stimulus formats, which do not require whole task clinical reasoning performance, may overestimate assessee performance.

Assessee-environment interactions

Environmental factors can have a significant impact on assessee performance. For example, in emergency room settings, the multitude of patients, task-shifting requirements, and high stress levels can increase the chances of poor clinical reasoning performance and diagnostic error. In their study of assesse-environment interactions, Hughes and Young [17] demonstrated that contextual factors, such as hospital setting, influenced nurses’ clinical reasoning processes, highlighting the impact of environmental factors on clinical reasoning performance assessments. From an information processing view, these outcomes may seem surprising because health professionals aspire to “objective”, evidence-based clinical reasoning. The situated cognition view recognizes that nurses’ interactions with their hospital environment shape their clinical reasoning performance and its subsequent assessment. These types of cultural environmental interactions, which can occur in any inpatient or outpatient setting, are rarely considered in the assessment of clinical reasoning performance because these assessments often occur within single institutions.

Assessee-method interactions

Our systematic search [14] revealed a broad range of clinical reasoning assessment methods (Table 2). Assessment methods or tools are usually characterized by one specific response format (e.g. multiple-choice questions), but some allow for the use of a variety of response formats (e.g. a written note to assess diagnostic justification or learner performance assessed by a standardized patient checklist). The target goal of assessment and feasibility issues typically determine the method and response format selection. From a situated cognition view, increasing the quantity of contextual factors leads to more interactions with the assessee’s and rater’s thinking processes which may potentially lead to greater variation in performance assessment. We will highlight the impact of specific aspects of assessment methods on clinical reasoning performance assessment in subsequent sections.

Assessee-response format interactions: cueing effects

In medical education, cueing effects refer to the impact that the presentation of lists of answer choices has on assessee performance. McCarthy demonstrated that such lists increase both the number of selected answers (when allowed) and the correctness of the answer [18]. Three studies [18], [19], [20] demonstrated a significant increase in data collection in the patient management problem format due to the lists of options visible to examinees (i.e. “cueing”), as compared with chart audit, oral examination, or simulated patient exercises. In a study comparing data collection in standardized patient encounters versus patient management problems, differences in response format accounted for 36% of performance variation, while performance differences between students only accounted for 24% [19].

Schuwirth et al. [21] noted that cueing effects have their greatest impact on diagnostic questions and deconstructed the cueing effect on diagnostic accuracy using constructed response items (i.e. open-ended questions) followed by selected response items (i.e. —four- to eight-option multiple-choice questions) for the same case. Positive cueing (i.e. improved diagnostic accuracy) occurred in about 14% and negative cueing (i.e. worsened diagnostic accuracy) in about 6% of multiple-choice questions. Positive cueing occurred with harder items while negative cueing occurred with easier items. More advanced practitioners (e.g. physicians) were less impacted by cueing effects of multiple-choice questions than students.

These studies demonstrate the dramatic impact that assessment method can have on clinical reasoning performance assessment. The situated cognition perspective provides deeper insights into the nature of knowledge and cueing effects because it explains that knowledge is not a static entity that exists in an assessee’s mind. Rather, knowledge emerges from the interaction of the assessee with contextual factors. Certain contextual factors are more or less likely to lead to the emergence of knowledge that results in acceptable clinical reasoning performance (e.g. multiple-choice questions cue the mind to recognize the correct option in harder questions). Positive cueing in difficult questions may relate to diagnostic hypothesis generation, a clinical reasoning sub-task that multiple-choice questions allow assessees to bypass. Experts who have more knowledge appear to be less easily swayed by distractors than novices, although even they are prone to negative cueing effects on easier questions. This insight has important implications for how educators choose assessment methods to prove assessee competence.

Assessee-scoring activity: sequence of data collection

Scoring activity refers to the process by which an assessment method produces a judgment of performance. Typical scoring activities include marking multiple-choice questions correct or incorrect, directly observing clinical performance by global (i.e. gestalt) assessment, and assessing the quality of diagnostic justification provided in a written note. Clearly, such activities differ significantly from one another in terms of the quantity and quality of interactions occurring between assessees and contextual factors. LaRochelle et al. [22] created a unique measure of data collection focused on the coherence or logical sequencing of videotaped student history-taking in a standardized patient encounter. Two study authors determined an ideal approach for the sequence of critical historical questions and the sequence of history questions for a given global score was then processed into an average ordering matrix, and the probabilities for any key data factor following another were calculated. The probabilities were then added together to create a coherence score which was compared to faculty assessors’ global ratings for these encounters. For higher average ordering matrix numbers, higher coherence scores demonstrated moderate associations with global ratings and aided in discriminating among students in the highest global rating score category. For low scores on the average ordering matrix, a weak correlation between coherence and checklist scores existed (approximate r=0.1–0.3). For high scores on the average ordering matrix, moderate correlations were seen between coherence and checklist scores (approximate r=0.5).

The correlation of history-taking sequence with global assessment again provides support for the value of a situated cognition perspective for clinical reasoning performance assessment. LaRochelle et al. [22] went beyond standard knowledge/process checklists to study the dynamic interactions of assessees with patients in collecting history. This approach demonstrates the value of situated cognition in clinical reasoning performance assessment because it highlights how the study of interactions between contextual factors (i.e. the assessee and standardized patient) can increase the capability to discriminate between student performances. The use of a situated cognition perspective in this study led to a novel scoring activity which measured the sequence and relationship of questions asked during history-taking. Although not every institution can replicate the statistics of this approach, most could incorporate assessment items focused on the logical sequencing and organization of students’ histories into their objective structured clinical examinations.

Assessee-task effects: the intermediate effect

The tasks of clinical reasoning can consist of outcomes, specifically a diagnosis or a treatment, and/or processes that generically consist of data collection, problem representation, and hypothesis generation. For a more comprehensive list of potential clinical reasoning tasks, we recommend reviewing Table 3 [23], [24]. These tasks occur within a spectrum of authenticity with standardized written examinations representing the less authentic pole (i.e. in vitro assessment) and direct observation representing the more authentic pole (i.e. in vivo assessment). Clinical reasoning activities range from part tasks, such as a multiple-choice question that asks what historical question is most likely to aid in a patient’s diagnosis, to whole tasks, such as a directly observed admission of a complicated intensive care unit patient.

Inaccurate conclusions about clinical reasoning performance can be caused by assessee interactions with certain tasks related to clinical reasoning response formats. For example, advanced students and/or residents score higher than clinical practitioners in process-focused assessments [e.g. objective structured clinical examination (OSCE) standardized patient checklist] that reward examinees for comprehensive recall of patient’s clinical findings [25]. This “intermediate effect” likely relates to the fact that experienced clinicians make diagnoses through System 1 (e.g. pattern recognition) rather than a System 2 (e.g. analytic) approach. On the other hand, assessment tasks that measure more authentic clinical tasks, such as diagnostic accuracy, will lead to higher clinical reasoning performance scores for expert clinicians. The intermediate effect again demonstrates how contextual factor interactions, in this case the assessee with the task, artifactually lead to worsened clinical reasoning performance scores.

Assessee-rater effects

In clinical reasoning assessment methods, such as OSCEs, direct observations, and global rating scores, raters play a critical role. The exact extent of their impact on holistic clinical reasoning performance assessment, though uncertain, is undeniable. Nevertheless, significant evidence exists regarding the impact of rater variation on clinical reasoning sub-task performance, such as data collection. Rater effects may account for up to 34% of performance variance on mini-clinical examination exercises (CEXs) with an additional 7% and 4% of the variance associated with rater-case effects and rater-trainee interactions, respectively [26]. Both construct-relevant (i.e. content knowledge) and construct-irrelevant factors (i.e. verbal style and dress) can impact rater stringency [27]. In addition, as compared with independent raters, teachers scored their students higher on a script concordance test (72 vs. 76.3, p<0.01), the primary outcome, although relative ranking remained the same and the small absolute difference seems clinically insignificant [28].

Clearly, information-processing theory is critical to understanding a significant portion of rater-based variation seen in clinical reasoning performance assessment, because raters’ clinical and assessment knowledge and experience is the basis for their judgments. However, the situated cognition perspective provides insights into construct-irrelevant contextual factors, such as gender and dress, which do not relate to rater knowledge and experience. Furthermore, situated cognition theory illuminates potential modifications to assessment methods. For example, a separate rater assesses a transcript of an OSCE encounter blinded to assessee contextual factors, such as gender, to reduce the risk for rater bias. Kogan et al. [13] have created a conceptual model of rater cognition highlighting the influence of clinical competence, emotions, personality traits, frame of reference, clinical systems, and institutional culture, but more research is required to understand the impact of these on clinical reasoning performance assessment.

Assessee-case-environment interactions

Two articles highlight how case and environment interact with the assessee to impact clinical reasoning performance and its assessment. A study [29] of nurses’ ability to use vital sign cues to categorize patients as high or low risk for deterioration compared their accuracy using identical clinical scenarios presented either as written vignettes or high-fidelity simulation. The authors used judgment analysis, an approach which applies the Lens Model theory [30]. Briefly, the authors used logistic regression analysis to retrospectively determine the value of various clinical variables to predict the acute deterioration of real patients. The results of this regression analysis then became the gold standard by which study nurses’ predictions about the risk of the simulated patients deteriorating were judged. This type of representation of clinical reasoning is called “paramorphic” because it predicts the results of judgments even though clinicians do not actually perform such a detailed mental mathematical calculation (i.e. Bayesian reasoning). For both experienced and student nurses, nursing judgment accuracy, r_a, was statistically significantly lower for the high-fidelity simulation [mean r_a 0.502, standard deviation (SD) 0.15] as compared to paper cases (mean r_a 0.553, SD 0.14, p=0.007) [29].

This difference in judgment accuracy stemmed from less-accurate utilization of cues in the high-fidelity simulation cases as compared with paper cases, despite the presentation of identical information. The Lens Model or other knowledge-based theories such as information processing do not provide obvious insights into what caused less-accurate utilization of the high-fidelity simulation cues. On the other hand, situated cognition theory emphasizes how multiple factors may have impacted cognition negatively. The patient/case and tasks were clearly different given that the nurses in the high-fidelity simulation had to consciously recognize the signal (i.e. the key cues/data) among the significant noise of a simulated environment as opposed to reading the cues from a paper case. In addition, the motivation, emotions, and stress of high-fidelity simulation differ significantly from a low-fidelity environment. A situated cognition view could have helped the researchers consider additional hypotheses and develop a study design better equipped to explore the causes of the differences in the assessments of the nurses’ clinical reasoning performance. For example, a third arm of the study using standardized patients could have been developed which might have demonstrated improved performance by experienced nurses who might more readily detect key signals and clinical findings in an actor than a mannequin.

Similarly, a web-based simulated patient study demonstrated that second-year medical students from the class of 2010 (n=155) with unlimited time in a pass-fail examination requested half the number of history and physical examination inquiries (i.e. 100–150) and achieved 91% diagnostic accuracy (average time to exam completion 4 h, 13 m) as compared with students from the class of 2012 (n=175) taking a time-constrained (3-h time limit), stratified examination (i.e. honors, high pass, pass, fail grade) (i.e. average time to exam completion 2 h, 1 m) who requested 250–300 inquiries in a stratified grades examination but only achieved 31% diagnostic accuracy [31]. Despite being informed that efficiency (# of required inquiry items requested/total # of inquiry items requested) impacted scoring, students in the time-constrained situation asked far more questions than the non-time-constrained students.

This example again highlights the value of a situated cognition perspective in understanding the powerful impact of contextual factors on clinical reasoning performance assessment. Test format modifications from untimed to timed and pass/fail to stratified reduced the diagnostic accuracy rate to 31%. It is improbable that the entire class of students in the time-constrained test condition had 61% poorer diagnostic ability on identical knowledge content than their colleagues from a class 2 years earlier. Because of the stress produced by time-constraint and the stratified examination and its interaction with case format [32], students asked nearly twice as many inquiries of the simulated patient than their unstressed colleagues, even when they were explicitly told that such behavior would negatively impact their grade. This quasi-experimental study highlights the importance of using a situated cognition framework in developing clinical reasoning performance assessments because contextual factors can have a profound impact on assessment outcomes.

Assessee-case-environment-method interactions

Durning et al. [12] studied the impact of contextual factors on diagnostic performance of general internists on video vignettes that portrayed different chief complaints. Each vignette varied based on patient factors [i.e. diagnostic suggestion, an atypical presentation, English as a second language (ESL), and/or emotional volatility] or environmental factors (e.g. time constraints) on a constructed written response to a validated clinical reasoning post-encounter form [33] as well as a subsequent verbal think-aloud protocol. Analysis of covariance provided support for the mechanisms of cognitive load and time constraints negatively impacting post-encounter form scores. The authors concluded that cognitive load, wherein the interactions of the clinician with multiple elements within a task can increase demands on working memory and hence worsen performance, provides an explanatory model for their findings [12]. Likewise, a study of oral examinations, where performance scores emerging from the interactions of assessees, raters, method, and task, demonstrated that scoring leniency increased over the course of the day and on succeeding examination days [34].

In both these studies, researchers varied or analyzed non-traditional contextual factors to see how they impacted clinical reasoning performance assessment. A situated cognition framework makes such research questions obvious because clinical reasoning performance and its assessment performance are recognized as context dependent rather than independent.

Assessee-environment-method-task interactions

The time allowed to perform a clinical reasoning task clearly impacts the assessment of clinical reasoning performance. Time limitation to complete a post-encounter form accounted for 4–16% of the variation in assessments of clinical reasoning performance, with the largest variance seen in the task of providing supporting evidence for the diagnosis [12]. Although an information-processing view certainly recognizes time as a critical contextual factor in assessment, a situated cognition perspective generalizes this perspective to non-traditional contextual factors (e.g. institutional culture, patient’s poor health literacy) and encourages educators to consider assessment tasks as “contextual factor(s)”-sensitive or -insensitive and modify them as needed. For example, given that most clinicians practice in an “open-book” environment, a situated cognition view might lead to a greater focus on open-book assessments to determine assessee competence.

Assessee-method-rater interactions

Durning et al. [35] compared a post-encounter form (PEF) to standardized patient checklist, oral presentation scores, and scores on standardized written multiple-choice question examinations. In univariate analyses, they found low correlations of the PEF across different methods: total oral presentation r 0.17, total SP checklist r 0.20, and National Board of Medical Examiners (NBME) shelf exam 0.23. The causes of these low correlations are uncertain but potentially include the following: (1) rater effects given that a different rater scored each method, (2) attenuation effects due to measurement error, or (3) measurement of a different construct. The third seems likely to be a significant contributor given the different tasks required of the assessee in each method.

The situated cognition perspective absorbs these results readily. Different assessment methods are contextual factors that interact with learners and raters in unique ways, leading to variations in clinical reasoning performance assessment and decreased correlations across methods.

Understanding clinical reasoning assessment through contextual factor interactions

Applying the lens of situated cognition theory to the selected studies reveals how contextual factors can interact with each other in complex ways to alter clinical reasoning performance and its assessment. Medical educators or researchers who do not account for such interactions may misinterpret assessment outcomes (e.g. most students cannot perform diagnostic reasoning) [31]. Our situated cognition-based conceptual model (Figure 1) provides a simplified set of contextual factors for researchers to consider as they design research studies to assess the impact of contextual factors and for medical educators as they develop programs of clinical reasoning performance assessment. It complements other valuable theoretical frameworks relevant to clinical reasoning performance and its assessment (e.g. information processing, self-regulated learning, and motivation theories).

Potential applications of situated cognition theory and the model in clinical reasoning assessment

The conceptual model encourages educators to consider how contextual factors impact clinical reasoning assessment (Table 4). These factors can be modified to achieve various assessment goals. For example, efficiency in a diagnostic task is an important aspect of competence in residents, so patient complexity could vary with the duration of a clinical reasoning OSCE station to assess competence in a time-constrained environment. For example, residents could be provided an evaluation time of 15 min for diagnosis of a complex case with an atypical disease presentation versus 8 min for an uncomplicated, typical disease presentation.

An additional value of the model is conceptualizing rater variability as “context specificity for raters”. Assessment by raters is analogous to the diagnostic process in clinical reasoning; it is a categorization task (e.g. is the learner in the “excellent”, “good”, or “poor” performance category). Thus, it is not surprising that clinical reasoning performance assessment would demonstrate similar context specificity [36]. This model may be helpful in providing a framework for further explaining the black box of context specificity in clinical performance assessment [13]. We believe that medical educators familiar with the model will more easily recognize opportunities for extending clinical reasoning performance assessment beyond assessee and case factors only.

Practical implications

The phenomenon of context specificity and the complexity of clinical reasoning performance assessment have implications for medical educators who are responsible for passing/failing standards for assessees. Clinical reasoning performance variation due to context specificity requires that multiple assessments across various problems in diverse settings are essential to obtaining valid information. The conceptual model also supports the push for increased quantity and quality of workplace-based assessments that has been trumpeted by competency-based medical education proponents [37], because these assessments measure situated clinical reasoning performance in context-rich environments that provide insights into how assessee’s perform in real clinical encounters. High-volume workplace-based observations by multiple raters have the potential to overcome the lack of standardization and rater variability that plague workplace-based assessment because large sample sizes will likely lead to accurate overall assessments of clinical reasoning performance. Such an approach has been employed successfully by at least one institution [38].

Although the previously described psychometric approach has been used with some success in dealing with the challenges of workplace-based summative assessment, our conceptual model depicts the significant challenges facing raters given the contextual complexity and task-shifting that is required of them. An alternative approach to increasing the validity of clinical reasoning assessments is to emphasize deliberate practice [39] and mastery learning [40] for raters (Table 1).

Society entrusts health professions institutions to ensure the quality of their graduates. When framed this way, the deliberate development and maintenance of a professional direct observation “team” for every health care institution to improve the validity of workplace-based assessments seems reasonable and appropriate. One key criterion for selecting raters is that they have excellent clinical skills and reasoning ability as research has demonstrated that rater clinical skills play an important role in assessment variation [41]. Unfortunately, because many schools and training programs have difficulty obtaining adequate direct observations, they often accept any rater that is willing or is able to perform the task. This situation may change as coaching programs become more common in medication education [42].

Limitations

Although we systematically searched the literature, some articles may have been missed, particularly non-English language articles. We did collect additional articles through hand searching references from selected seminal papers to maximize the broadness of our search. Our last literature search occurred in 2016, so this review may be missing some newer relevant articles that shed additional light on the value of a situated cognition perspective in clinical reasoning performance assessment. Although we defined situated cognition theory for the reviewers in our group, their understanding of the term likely varied significantly and may have led to the misidentification or lack of recognition of articles that implicitly referenced situated cognition. Nevertheless, we found an adequate number of articles that provided valuable insights into the impact of contextual factors on clinical reasoning performance assessment. We chose a narrative review format because it allowed for a selection of articles that best exemplify the insights that situation cognition theory provides into clinical reasoning assessment performance. We did not feel that the narrative review format was a significant limitation given the purpose of trying to highlight the potential value of a situated cognition perspective for understanding variation in the assessment of clinical reasoning performance (i.e. a proof of conceptual framework). An article focused on teaching clinical reasoning recently used a similar approach [43].