Semantic competence and prototypical verbalizations are associated with higher OSCE and global medical degree scores: a multi-theory pilot study on year 6 medical student verbalizations

Pedro Grilo Diogo; Vítor Hugo Pereira; Frank Papa; Cees van der Vleuten; Steven J. Durning; Nuno Sousa

doi:10.1515/dx-2021-0048

Article

Semantic competence and prototypical verbalizations are associated with higher OSCE and global medical degree scores: a multi-theory pilot study on year 6 medical student verbalizations

Pedro Grilo Diogo , Vítor Hugo Pereira , Frank Papa , Cees van der Vleuten , Steven J. Durning and Nuno Sousa

Published/Copyright: March 15, 2023

Published by

Become an author with De Gruyter Brill

Submit Manuscript Author Information Explore this Subject

From the journal Diagnosis Volume 10 Issue 3

Abstract

Objectives

The organization of medical knowledge is reflected in language and can be studied from the viewpoints of semantics and prototype theory. The purpose of this study is to analyze student verbalizations during an Objective Structured Clinical Examination (OSCE) and correlate them with test scores and final medical degree (MD) scores. We hypothesize that students whose verbalizations are semantically richer and closer to the disease prototype will show better academic performance.

Methods

We conducted a single-center study during a year 6 (Y6) high-stakes OSCE where one probing intervention was included at the end of the exam to capture students’ reasoning about one of the clinical cases. Verbalizations were transcribed and coded. An assessment panel categorized verbalizations regarding their semantic value (Weak, Good, Strong). Semantic categories and prototypical elements were compared with OSCE, case-based exam and global MD scores.

Results

Students with Semantic ‘Strong’ verbalizations displayed higher OSCE, case-based exam and MD scores, while the use of prototypical elements was associated with higher OSCE and MD scores.

Conclusions

Semantic competence and verbalizations matching the disease prototype may identify students with better organization of medical knowledge. This work provides empirical groundwork for future research on language analysis to support assessment decisions.

Keywords: assessment; clinical reasoning; dual process theory; knowledge structures; OSCE; prototype theory; semantics

Corresponding author: Pedro Grilo Diogo, MD, MHPE, Escola de Medicina da Universidade do Minho, Campus de Gualtar, Braga, 4710-057, Portugal, E-mail: pgrilodiogo@gmail.com

Acknowledgments

The authors acknowledge the collaboration of all participating students and standardized patients, as well as the contribution made by the assessor panel.

Research funding: None declared.
Author contribution: Conceptualization: PGD; Methodology: PGD, VHP, NS; Formal analysis and investigation: PGD, VHP, FP; Writing (original draft preparation): PGD; Writing (review and editing): PGD, VHP, CVV, SJD, NS, FP; Supervision: CVV, NS. All authors have accepted responsibility for the entire content of this manuscript and approved its submission. The views expressed here are those of the authors and not necessarily those of the Department of Defense or other federal agencies.
Competing interests: Authors state no conflict of interest.
Informed consent: Informed consent was obtained from all individuals included in this study.
Ethical approval: This research complied with relevant national regulations and institutional policies and has been approved by the Ethics Committee of the School of Medicine – University of Minho in January 2017.

References

1. Elstein, AS, Shulman, LS, Sprafka, SA. Medical problem solving. Cambridge, MA: Harvard University Press; 1978.10.4159/harvard.9780674189089Search in Google Scholar

2. Schmidt, HG, Norman, GR, Boshuizen, HP. A cognitive perspective on medical expertise: theory and implication. Acad Med 1990;65:611–21. https://doi.org/10.1097/00001888-199010000-00001.Search in Google Scholar PubMed

3. Schmidt, HG, Rikers, RM. How expertise develops in medicine: knowledge encapsulation and illness script formation. Med Educ 2007;41:1133–9. https://doi.org/10.1111/j.1365-2923.2007.02915.x.Search in Google Scholar PubMed

4. Kalyuga, S. Rapid cognitive assessment of learners’ knowledge structures. Learn Instr 2006;16:1–11. https://doi.org/10.1016/j.learninstruc.2005.12.002.Search in Google Scholar

5. Bordage, G. Prototypes and semantic qualifiers: from past to present. Med Educ 2007;41:1117–21. https://doi.org/10.1111/j.1365-2923.2007.02919.x.Search in Google Scholar PubMed

6. Chi, MT, Feltovich, PJ, Glaser, R. Categorization and representation of physics problems by experts and novices. Cognit Sci 1981;5:121–52. https://doi.org/10.1207/s15516709cog0502_2.Search in Google Scholar

7. Ericsson, KA, Krampe, RT, Tesch-Römer, C. The role of deliberate practice in the acquisition of expert performance. Psychol Rev 1993;100:363–406. https://doi.org/10.1037/0033-295x.100.3.363.Search in Google Scholar

8. Pellegrino, J, Chudowsky, N, Glaser, R, editors. Knowing what students know: the science and design of educational assessment. Washington, DC: National Academies Press; 2001.Search in Google Scholar

9. Ericsson, KA. An expert-performance perspective of research on medical expertise: the study of clinical performance. Med Educ 2007;41:1124–30. https://doi.org/10.1111/j.1365-2923.2007.02946.x.Search in Google Scholar PubMed

10. Cruse, A. Meaning in language: an introduction to semantics and pragmatics. UK: Oxford University Press; 2011.Search in Google Scholar

11. Bordage, G, Lemieux, M. Semantic structures and diagnostic thinking of experts and novices. Acad Med 1991;66:S70–2. https://doi.org/10.1097/00001888-199109000-00045.Search in Google Scholar PubMed

12. Lemieux, M, Bordage, G. Propositional versus structural semantic analyses of medical diagnostic thinking. Cognit Sci 1992;16:185–204. https://doi.org/10.1207/s15516709cog1602_2.Search in Google Scholar

13. Bordage, G, Connell, KJ, Chang, RW, Gecht, MR, Sinacore, JM. Assessing the semantic content of clinical case presentations: studies of reliability and concurrent validity. Acad Med 1997;72:S37–9. https://doi.org/10.1097/00001888-199710000-00036.Search in Google Scholar

14. Baker, EA, Connell, KJ, Bordage, G, Sinacore, J. Can diagnostic semantic competence be assessed from the medical record? Acad Med 1999;74:S13–5. https://doi.org/10.1097/00001888-199910000-00026.Search in Google Scholar PubMed

15. Durning, SJ, Artino, A, Boulet, J, La Rochelle, J, Van Der Vleuten, C, Arze, B, et al.. The feasibility, reliability, and validity of a post-encounter form for evaluating clinical reasoning. Med Teach 2012;34:30–7. https://doi.org/10.3109/0142159x.2011.590557.Search in Google Scholar PubMed

16. Dory, V, Gagnon, R, Charlin, B, Vanpee, D, Leconte, S, Duyver, C, et al.. In brief: validity of case summaries in written examinations of clinical reasoning. Teach Learn Med 2016;28:1–10.10.1080/10401334.2016.1190730Search in Google Scholar PubMed

17. Smith, S, Kogan, JR, Berman, NB, Dell, MS, Brock, DM, Robins, LS. The development and preliminary validation of a rubric to assess medical students’ written summary statements in virtual patient cases. Acad Med 2016;91:94–100. https://doi.org/10.1097/acm.0000000000000800.Search in Google Scholar

18. Norman, G. Dual processing and diagnostic errors. Adv Health Sci Educ Theory Pract 2009;14:37–49. https://doi.org/10.1007/s10459-009-9179-x.Search in Google Scholar PubMed

19. Custers, EJ. Medical education and cognitive continuum theory: an alternative perspective on medical problem solving and clinical reasoning. Acad Med 2013;88:1074–80. https://doi.org/10.1097/acm.0b013e31829a3b10.Search in Google Scholar PubMed

20. Papa, FJ. A dual processing theory based approach to instruction and assessment of diagnostic competencies. Med Sci Educ. 2016;26:787–95. https://doi.org/10.1007/s40670-016-0326-8.Search in Google Scholar

21. Knuuti, J, Wijns, W, Saraste, A, Capodanno, D, Barbato, E, Funck-Brentano, C, et al.. ESC Guidelines for the diagnosis and management of chronic coronary syndromes: the task force for the diagnosis and management of chronic coronary syndromes of the European society of cardiology (ESC). Eur Heart J 2019;41:407–47.10.1093/eurheartj/ehz425Search in Google Scholar PubMed

22. Humphrey-Murto, S, Varpio, L, Gonsalves, C, Wood, TJ. Using consensus group methods such as Delphi and nominal group in medical education research. Med Teach 2017;39:14–9. https://doi.org/10.1080/0142159x.2017.1245856.Search in Google Scholar PubMed

23. LaRochelle, J, Durning, SJ, Boulet, JR, van der Vleuten, C, van Merrienboer, J, Donkers, J. Beyond standard checklist assessment: question sequence may impact student performance. Perspect Med Educ 2016;5:95–102. https://doi.org/10.1007/s40037-016-0265-5.Search in Google Scholar PubMed PubMed Central

24. Schuwirth, LW, van der Vleuten, CP. Programmatic assessment and Kane’s validity perspective. Med Educ 2012;46:38–48. https://doi.org/10.1111/j.1365-2923.2011.04098.x.Search in Google Scholar PubMed

25. Bissessur, SW, Geijteman, EC, Al-Dulaimy, M, Teunissen, PW, Richir, MC, Arnold, AE, et al.. Therapeutic reasoning: from hiatus to hypothetical model. J Eval Clin Pract 2009;15:985–9. https://doi.org/10.1111/j.1365-2753.2009.01136.x.Search in Google Scholar PubMed

Supplementary Material

This article contains supplementary material (https://doi.org/10.1515/dx-2021-0048).

Received: 2021-04-02

Accepted: 2023-02-20

Published Online: 2023-03-15

You are currently not able to access this content.

Supplementary Material

Articles in the same Issue

https://doi.org/10.1515/dx-2021-0048

Keywords for this article

assessment; clinical reasoning; dual process theory; knowledge structures; OSCE; prototype theory; semantics