Assessment of clinical reasoning: three evolutions of thought

Challenge 1: problem-solving as idiosyncratic and individual

The first challenge arose when expert panels were asked to provide scores for all the decision points in the simulation or paper case. These scores, which would identify whether a decision was good or poor, were used in the process of calculating a final clinical reasoning score. In so doing, it became clear that experts disagreed considerably about the value of certain decisions in the process. Indeed, and consistent with the family of theories discussed in this edition, these experts did not merely disagree on how many points to attribute to each decision, but also on the optimal pathway through the simulation [4].

From a cognitive psychological perspective, this “expert idiosyncrasy” is understandable. One of the more popular cognitive psychology theories that focuses on expertise development and problem-solving ability is script theory [6], [7]. Essentially, this theory claims that the development of clinical problem-solving expertise occurs through the creation of increasingly elaborate problem-solving (or illness) scripts. The process begins by collecting “islets” of isolated factual knowledge, which gradually connect to form semantic networks [8]. Gradually, the physician aggregates these semantic networks into illness scripts. These scripts enable the expert to quickly recognize and connect a patient problem with possible solutions, for example, “This is a typical XXX patient.” As physicians develop greater expertise, these illness scripts are enriched with contextual features, which extend beyond the content needed to arrive at a reasonable diagnostic and management plan. This allows the expert to recognize – almost effortlessly – the problem as a gestalt (or whole) [6]. From this theory, it is understandable that high-level illness scripts evolve as a result of an individual’s specific experiences in connection with his/her specific background knowledge. Because of this, it is, therefore, logical to assume that when an expert is asked to unpack this highly aggregated script, s/he will do so in a highly individual, idiosyncratic way. So, from an illness script theory perspective, understanding the development of clinical problem-solving expertise helps to explain why long, branched scenarios were an ineffective approach to the assessment of clinical reasoning.

Challenge 2: domain specificity

The second challenge–domain specificity – relates to the first. Successful clinical reasoning assumes that the physician has a good, relevant working knowledge as a prerequisite [9], [10], [11]. Herein lies the problem. One can be very knowledgeable in one domain and ignorant in another; hence, knowledge is “domain specific.” Because knowledge is both domain specific and a prerequisite for successful clinical reasoning, the process of clinical reasoning is also domain specific, leading to the phenomenon of “content specificity” [12]. Hence, the way in which a candidate reasons through one problem is a poor predictor of the way s/he reasons through another problem, even within the same domain. Studies examining this phenomenon identified correlations between cases of 0.1 and 0.2 [4]. The implication of domain specificity is that large numbers of cases are required to produce a sufficiently reliable score. Given the elaborate nature of such simulations, lengthy testing times rendered this impracticable and unfeasible [4].

Challenge 3: expertise is associated with efficiency

When measures were put in place to mitigate idiosyncrasy and content specificity concerns, it was found that practicing physicians were generally outperformed by final year medicine students and doctors in their first and second year post-graduation [13], [14]. From a cognitive psychological perspective, this is unsurprising. Expert performance is often associated with efficiency due to the formation of illness scripts. Consequently, experts generally require less information than non-experts and even intermediates to identify a correct solution [15], [16]. This finding seriously challenged the validity of PMPs because they reward thoroughness of information collection, rather than efficiency. In this way, experts were essentially punished for their expertise.

Clinical problem-solving developments: key feature approach and extended matching items

Apart from key learnings related to idiosyncrasy, domain and content specificity, another important lesson was derived from this: authenticity or fidelity is not the same as validity [4]. Of course, this was not a new insight as construct validity theory already indicated that what looks valid by observation does not necessarily need to have construct validity [17], [18]. This distinction is important because of its application to simulation, and therefore, any modern virtual reality attempt to assess clinical reasoning. If the problems surrounding the assessment of clinical reasoning are not about lack of authenticity, virtual reality simulations are not the solution.

In the late 1980s and early 1990s, in response to the three challenges described above, two approaches were developed to assess clinical reasoning: key feature approach and extended matching items. These approaches are similar in that they both use short cases and only require a small number of essential decisions to be made in relation to the case. In key feature assessment, short case presentations are coupled with a few questions aimed at essential decisions [19], [20]. Formats for these questions vary and may include open-ended and multiple-choice questions [21], [22], [23]. By keeping cases short and only asking a limited number of questions per case, a wide variety of problems can be explored per hour of testing time [24].

Extended matching items differ slightly in that they begin with the presentation of possible answers, together with a series of short cases (i.e. vignettes). The aim of this approach is to link one of the answers with each vignette [25]. For example, vignettes may be patient cases with a series of likely diagnoses from which to choose from. As with the key feature approach, extended matching uses short vignettes to enable the assessment of multiple problems per hour of testing time. These approaches were found to be successful in mitigating issues related to domain specificity, idiosyncrasy, and the intermediate effect [22], [25].

Although these two approaches were successful in addressing what they purported to achieve, they did not satisfy all components of the assessment of clinical reasoning. While both methods assess clinical decision making (with its focus on the final decision), they do not fully assess clinical reasoning (which focuses on the process and the decision). Although clinical decision making is generally seen as a pivotal component of clinical expertise, the challenge remained to develop assessment methods that would capture the thought processes behind the decision, to illuminate reasoning and error. An interesting development in this direction was a script concordance test [26], [27].

The script concordance test

In the script concordance test item, a very short vignette is presented in combination with a suggestion for a hypothesis (e.g. in a patient with symptoms A, B, and C, you are considering diagnosis X). An additional finding is presented, and the candidate is asked whether this finding makes the hypothesis more or less likely. The answer key and subsequent, weighted, scores are determined by the responses of a panel of experts [26], [27]. This test approach is accepted for its recognition of, and allowance for, candidates’ own idiosyncratic patterns or “scripts,” which is central to the illness script theory [28].

From an illness script theory perspective and our understanding of the idiosyncrasy of clinical reasoning, we may find it defensible to claim that for some, a certain finding makes the hypothesis more likely, and for others, less likely. This, however, is contentious. Some argue that this idiosyncrasy cannot be the case; in other words, a certain finding in a given context could result in either a hypothesis being more or less likely, but not both [29]. This paradox seems to be the result of the scoring approach used in the script concordance test. The stimulus (i.e. the short case and hypothesis) operates under the assumption that illness scripts are idiosyncratic, and there is an acceptable level of diversity. The scoring system operates under the assumption that individuals’ illness scripts must be concordant with the average of a group of experts. Highlighting this paradox is essential in our explanation of further developments about the assessment of clinical reasoning.

To further explore this paradox, it is helpful to make another distinction between diagnostic decision making and clinical reasoning. The literature suggests that diagnostic decision making is not the same as other aspects of decision making and reasoning and is likely to be prone to different types of errors [1]. Klein [30] argued that in order to better understand error in naturalistic decision making, a distinction between the quality of the decision-making process and the quality of the outcomes is important. If we make this distinction, diagnostic decision making is generally treated more as a “convergent” process involving a broad collection of information that is ultimately summarized into one (or two) best solutions. This is reflected in the diagnostic process, in which all available information (e.g. history, physical examination, laboratory results) is summarized into one diagnostic classification. For assessment, this process is then, logically, relatively straightforward, as the stimulus and a correct response can be pre-determined. In contrast to diagnostic decision making, clinical reasoning can often involve the concurrent identification of multiple, “good” solutions. This process is then likely to be better understood as a non-linear (often complex), divergent, and, to a certain extent, unpredictable process.

Clinical reasoning and situational awareness

Complex processes are not necessarily difficult; for example, history taking could be considered a complex process. Prior to a history taking, there is no way to predict what will be said at a certain point in time (e.g. 4 min and 15 s into the conversation). This is because the ensuing conversation is the result of myriads of interactions, both verbal and non-verbal, which are highlighted in the family of theories in this special edition. Yet, most clinicians typically manage this complexity without encountering any major problems. This example demonstrates that, although we might accept that there are multiple, concurrent, “good” solutions, it is also not a case of “anything goes” [31]. Rather, there are boundaries between acceptable and unacceptable actions and utterances [32], [33]. These boundaries may not be clear-cut or even pre-defined. Rather, they are fuzzy and can evolve or change during the course of the conversation. Consequently, it is important to utilize a range of strategies (i.e. actions and utterances) to flexibly deal with the interaction at any given point in time. Selecting and implementing an optimal strategy at every moment requires “situation awareness” – being mindful and responsive to what is effective and ineffective, and agilely moving between strategies. This (hopefully intuitive) understanding of how to navigate complexity is readily transferable to our understanding of clinical reasoning [34].

From this point of view, one could argue that clinical reasoning occurs as a narrative; either as an internal dialogue, or between the physician and others (e.g. the patient, colleagues, the examiner, additional staff). Consequently, the distinction between a good/poor clinical reasoner does not lie merely in the identification of the single best solution, but rather, the extent to which s/he can agilely utilize a repertoire of strategies and demonstrate situation awareness to convey his/her message. From a clinical reasoning assessment perspective, this means that the assessment must first establish whether the physician oversteps boundaries (e.g. utilizing incorrect or untrue pathophysiological explanations). Second, the physician’s capacity to flexibly explain the clinical situation from various viewpoints (and combine these perspectives) must be assessed (e.g. the ability to explain influenza from public health, infectious disease, immunological, and microbial perspectives, etc.). Finally, it is important to assess the physician’s ability to adapt his/her reasoning process in the moment, to connect with the other communication partner(s) present in the encounter (be it the patient, patient’s family, examiner, or colleague).

Clinical reasoning and situativity theory

The features outlined above, and particularly the situation-specific nature of clinical reasoning, has given rise to a final perspective on clinical reasoning; a social cognitive family of theories often referred to as “situativity theory” [35]. This family of theories suggests that clinical reasoning occurs in the moment and, therefore, should only be assessed in the “here-and-now.” For example, ecological psychology emphasizes that each situation provides “affordances” (i.e. opportunities for action) to the people – the actors – in the situation who, in turn, have effectivities (i.e. skills and abilities to act) [34], [36]. Affordances indicate what the situation enables and disallows the actor(s) to do. Effectivities relate to what the actor(s) is/are able and unable to do with the given situation. For example, the information that the patient is willing to share during history taking is an affordance that the clinician can use in the clinical reasoning process. Information that the patient does not share does not contribute to the affordances. The extent to which the clinician can, for example, record and remember patient information and make sense of that information in relation to his/her prior knowledge constitutes the effectivities. From a situativity perspective, good clinical reasoning is, therefore, the continual and purposeful alignment of effectivities and affordances. From this perspective, assessing clinical reasoning still requires direct human observation. This is because it involves constant evaluation of whether the physician’s utterances and actions remain within acceptable boundaries, are sufficiently varied, and adequately agile during the interaction.

The role of human judgment in assessment

Human judgment has long been discarded in assessment as being too subjective and unreliable. We have now, however, come to realize that human judgment is integral to the assessment of clinical reasoning and error, as for many other aspects of medical competence [37]. This is not to say that we have returned to the unstructured, often biased, bedside assessment of clinical reasoning of 60 years ago. On the contrary, we have learned much about the need for broad sampling [38], [39], [40] and the factors to address and avoid in direct observation- and workplace-based assessment [41], [42].

Indeed, literature is emerging, which helps to provide us with a better understanding of the role of expert human judgment in the assessment of clinical reasoning and error [37], [43], [44], [45], [46]. We have come to realize that clinical reasoning and error cannot be predefined and “objectively” measured but, rather, involve the creation of a shared subjectivity. Creating this shared subjectivity requires an understanding of narratives (or stories), and this is a focus of current research. Studies, which explore the nature of human decision making in assessment have adopted script theory from clinical decision making to view assessment as a “diagnostic” process [47], [48], [49]. Other studies explore the value of different perspectives on the same reasoning process. Traditional perspectives suggest that one view may be more correct than another, adopting the assumption that there is one single truth. From the perspective that there might be concurrent, multiple, good solutions, it is important to understand how different views contribute to our understanding of clinical reasoning as a multifaceted concept. Research by Gingerich et al. [37] recognizes this. Another line of research seeks to understand how narratives contribute to judgment and decision making – the narratives that assessors use and how assessors evaluate candidates’ narratives to judge the quality of their clinical reasoning [37], [43], [49]. Finally, another line of inquiry has explored circumstances surrounding “context specificity” – a situation in which a physician arrives at two different diagnoses after seeing two patients present with identical symptomatology. This is clearly a source of unwanted variance and error in health care. This research is seeking to unveil the unique affordances and effectivities that physicians use in a clinical situation, the findings of which will assist both our teaching and assessment of clinical reasoning [50], [51], [52].