Technical infrastructure for curriculum mapping in medical education: a narrative review

Included studies

Our search strategy identified 352 abstracts. We downloaded 21 studies for detailed inspection and excluded four of them due to their focus on repositories of reusable learning objects instead of curriculum mapping or description of a curriculum without specifying the technical details of the software. The manual search of references yielded six additional articles that were included in the final set of 23 studies. A tabular summary of the included studies is available in the Appendix. The studies contained descriptions of the use of 11 systems in curriculum mapping: ACLO-Web [14], CLUE [15], Electronic Thematic Map [8], Excel/Entrada [16], LOOOP [17], Medtrics [18], MERLIN [19], [20], [21], One45 [22], OPTIMED [23], [24], [25], [26], [27], Prudentia [28] and SOLE [22]. In four cases, the name of the system was not given [29], [30], [31], [32]. One study described a technological data exchange standard in curriculum management [5]. Two of the studies were reviews [33], [34].

Visualisations

In the first thematic category, we included scientific contributions that described experiences with visual data presentation in curriculum mapping. Topics from this category were frequently mentioned in the reviewed papers. The three common recommendations were to visualise curriculum maps as [1] graphs or networks, [2] business diagrams such as bar or pie charts or [3] as coloured tables or panels.

The application of graphs was explicitly discussed in two papers [24], [29], whereas several other studies used graph-based visualisations [23], [27], [30]. The nodes in the graphs represented learning units [29], medical disciplines [24], content descriptors from medical terminologies [23] or learning objectives [30]. The relations displayed as links between the nodes in the graphs visualised prerequisite subjects [29], similarities of textual descriptions of learning units in medical disciplines [24] and associations between keywords in learning unit metadata [23]. The weight of the edges connecting the nodes depicted the degree of similarity [24] or the attained learning objective success rate [30]. In the latter example, by graphically showing the presence or lack of educational success, the authors discussed reaching a constructive alignment of learning and assessment methods [30]. The topology of nodes in the curriculum maps was reported in one study to have been calculated using the Kamada–Kawai force-directed method [29]. By analysing the length and quality of the pathways of prerequisite relations, the authors attempted to discuss the coherence of the curriculum [29]. Subgraphs in the maps were detected using methods known from Social Network Analysis (e. g., the WalkTrap algorithm, the Louvain method) [24], [29]. The results were utilised to highlight areas of isolated knowledge communities in the maps that required better integration in the curriculum [23], [24], [29]. Visualisations of graphs were implemented using programming APIs such as Cytoscape, D3.js, iGraph (R), NetworkX (Python), NVD3 and yEd Graph Editor [23], [24], [].

Further types of visualisation included diagrams known from common business applications, such as bar, balloon, pie and doughnut charts. Fritze et al. provided a summary of 10 exploratory question types addressed by the MERlin web-application, which delivered insights into various aspects of curriculum maps such as presenting a longitudinal competency development profile or applied assessment methods [20]. The questions were answered by diagrams visualising a lot of information in one view by using several presentation dimensions as an arrangement of elements in the x-/y-axes, and varied size and colour of the shapes in the diagram. For a given sub-competency from the German NKLM learning objectives catalogue [35], the graphics showed a temporal breakdown of the covered learning objectives across semesters or departments, additionally indicating the covered level of competency (e. g. basic knowledge, applied knowledge, competence in practice) and frequency of occurrence in the curriculum [20]. Komenda et al. described the implementation of 25 analytical reports, including a visualisation of the use of Bloom’s taxonomy action verbs in learning objective descriptions across curricula [26].

Finally, several studies presented curriculum maps in tabular views and panels [8], [14], [16], [18], [31]. In two aligned columns of panels, Al-Eyd et al. [18] showed the learning objectives at the programme and course level. Jarvis-Selinger & Hubinette presented a medical undergraduate programme curriculum as a large spreadsheet table (called the Matrix) [16]: the columns represented individual medical systems, themes and clinical experiences; the rows represented courses, weeks and week topics; the cells indicated the focus of a specific week. Thanks to its chronological, vertically ordered rows and thematically diverse columns, the Matrix made it possible to verify whether the new curriculum had a developmental (increasingly complex), spiral (regularly revisited), integrated and competency-based style [16]. Presenting the Matrix as an Excel spreadsheet imposed a technical limitation as only a small amount of information could be presented at cell level. This issue was temporally solved by introducing Word documents called Virtual Course Books with itemised schedules for each week or session-level learning objective that were linked from the cells of the Matrix. However, the long-term plan was to introduce a more sophisticated software tool (Entrada). This and other similar experiences (e. g. [21], [31]) clearly show the limitations of using generic office software for curriculum mapping tasks.

Text-based descriptions and analytic functions

In this category, we covered methods of description, analysis and reporting of text-based curriculum data which encompass themes such as selection of the right description categories, terminologies and measures of similarity.

One of the biggest challenges that has to be faced in curriculum mapping is the laborious and tedious process of entering descriptions of curriculum elements [4]. This can be remediated in various ways that are described in the reviewed literature. One of them is to reduce the number of curricular aspects documented in the first instance. Although it might be tempting to describe all categories from the curriculum mapping framework proposed by Harden [1], some authors have recommended deliberately limiting the number of characterised aspects to make the data entry process manageable in a short time frame [20]. Fritze et al. reported that thanks to the reduction of the described curricular windows to four crucial ones (i. e., competency level, transparency, percentage of learning objectives covered and assessment format), it was possible to reduce the time needed for description of the whole medical curriculum to six months [20]. Likewise, the curriculum mapping initiative described by Al-Eyd et al. limited the number of ‘curriculum windows’ to four (i. e., learning expectations, learning event information, pedagogy and assessment) [18].

To enable flexible search functionality in curriculum maps, it was recommended to index descriptions with terms and concepts from medical thesauri and ontologies [14], [23]. This helped in handling the diversity of natural human language with its many synonyms, near synonyms, hypernyms and hyponyms. It was also advised to use standardised terminologies and coding systems to help achieve automated integration of curriculum mapping tools with the remaining IT components of universities’ technical infrastructure [23]. The added value was more precise and sensitive searches of curriculum descriptions, which contributed to more adaptive learning facilities and higher user satisfaction. The medical terminologies integrated in the described studies included MeSH [14], [17], [23], UMLS [28] and ICD10 [14], [17]. Balzer et al. [17] and Komenda et al. [26] incorporated in their systems a dictionary of action verbs from Bloom’s taxonomy [36]. Cottrell et al. reported using MedBiquitous controlled vocabulary to standardise instructional and assessment methods and types of learning resources; they also used USMLE Step 1 & 2 year-end profile reports to populate catalogues of tags to annotate curriculum descriptions [22]. Finally, Spreckelsen et al. implemented their software tool using the Semantic Media Wiki technology, which enabled distributed curriculum description, querying, and consistency validation using Semantic Web technologies (RDF/OWL/SPARQL) [14].

The last theme in this category related to numerical values which helped to summarise properties of curricula. Similarities between text descriptions of learning units in curricula were calculated using data- and text-mining techniques, including metrics such as cosine distance [24], normalised Pearson correlation coefficient [27] and Jaccard similarity coefficient [23]. The importance of nodes (learning units, medical disciplines) in curricula was expressed by centrality measures (closeness, betweenness, or eigenvector centrality) [24], [29]. Detection of similarities enabled automatic clustering of related parts of a given curriculum. For instance, Komenda et al. applied this to the automatic matching of virtual patients with relevant sections in the curriculum [27].

Outcome-based approach

This category is made up of functions that facilitate the outcome-based approach in curriculum mapping, e. g., outcome- and competency-based frameworks, hierarchies of curriculum building blocks, and technical standards for exchange and comparison of learning objectives and other elements of curricula descriptions.

Several competency-based frameworks and learning outcome catalogues were implemented in the reviewed software tools. The CLUE system was built around the competency frameworks of the General Medical Council, the Singapore Medical Council and the Accreditation Council for Graduate Medical Education [15]. Sharma et al. determined the alignment of a local curriculum with the British Association of Dermatologists undergraduate medical curriculum and the already mentioned General Medical Council standard [32]. Schneider et al. mapped the learning objectives in their curriculum against the Commission on Dental Accreditation (CODA) standard [8]. In the Prudentia system, cross-references were made to the Australian Medical Council Student Outcomes [28]. The LOOOP system visualised learning objectives according to the respective category of Anderson’s 24-square-table [36] and Reporter-Interpreter-Manager-Educator (RIME) roles [37]. In their visualisations, Vaitsis et al. used learning outcomes of the Swedish Higher Education Authority for undergraduate medical education [30]. The introduction of a national competency-based catalogue of learning objectives for undergraduate medical (NKLM) and dental education (NKLZ) in Germany in June 2015 [35] had a considerable impact that was visible in several reviewed papers. Systems such as MERlin or LOOOP were promoted as tools that help in achieving and maintaining the alignment of local curricula with the NKLM framework [19], [20], [21]. The NKLM list of competencies/sub-competencies and associated learning objectives is incorporated in these software tools and enabled rapid selection of the covered learning objectives (e. g., by ticking off checkboxes) in the described learning units [20], [21]. Filtering learning units by standardised learning objectives and competencies enabled rapid searches for gaps and redundancies in the curriculum. Using MERlin software, Behrends et al. searched the curriculum at Hannover Medical School for 52 medical informatics-related learning objectives from the NKLM catalogue to locate learning units in the curriculum which shared the same leaning objectives [19]. This was done to enable a more collaborative and integrated approach to teaching these competencies across different stages of the medical study programme.

However, not all countries had adopted national-wide learning objectives and competency catalogues. Additionally, some institutions from countries with formalised national competency frameworks were keen to maintain their internal competency rubrics. These users needed functionality to design spacious customary learning objective collections. For instance, teachers at Masaryk University used the OPTIMED system to write 7,000 local learning objectives [23]. The German ACLO-Web system encompassed 5,350 learning objectives in the Aachen Medical Model Curriculum [14], which illustrates how comprehensive a curriculum description might become. The built-in functionality to support authoring of learning objectives included the automatic detection and alignment of verbs in the new entries with lists of standardised action verbs from Bloom’s taxonomy (as available, e. g., in the LOOOP system [17]).

Standardisation of learning objectives in competency frameworks enabled wide-scale comparison and auditing of curricula. As the goal to promote the use of one single curriculum management system that covers the needs of all stakeholders turned out to be futile [5], the alternative idea was to equip the plethora of available software with standardised interfaces for data exchange. This was implemented by the MedBiquitous Curriculum Inventory standard [5]. This technical specification is based on an object-oriented model that consists of two classes of elements: base components (including learning events and expectations) and aggregating elements for curriculum organisation and structuring (academic levels, sequence blocks, integration blocks). The goal was to develop a technical infrastructure including curriculum inventory aggregators and analytic tools and reports around this standard to enable wide-scale auditing, quality improvement, and research in the area of medical curricula [5]. Other solutions for curriculum data interoperability included the use of Semantic Web technologies [14], a connection to electronic scheduling systems using the iCal format [17], PDF printouts and data collection forms [20], [22], and Microsoft Excel imports/exports [14].

Adaptability

In the last category we included themes which dealt with methods that enabled smooth access to curriculum data that was adjusted to the needs of a particular group of users.

It may be concluded from the reviewed literature that even though there are advantages to a common core database structure and shared development efforts, higher education institutions also require individual site-specific features [31]. One-fit-all solutions are likely to meet with resistance from prospective users, as has been demonstrated on the example of the CurrMIT system in North America [5], [18]. This observation was confirmed in Europe, as was reported in Fritze et al.’s study regarding four German medical schools’ shared effort to implement a software tool called MERlin [20]. The participating institutions wanted to add their own site-specific features, including local terminology, folksonomies, protected data-sets and free-text fields, which could be used to enter descriptions of characteristic elements of their curriculum (e. g., special teaching methods) [20]. An interesting feature for dynamic adjustment of the curriculum was implemented in the LOOOP system. This software tool analysed the demand to cover particular clinical learning objectives and estimated current ward teaching capacities. When an insufficient number or type of patients or teachers was available, the system dynamically suggested alternative wards where patients presenting the needed diagnoses were available [17].

As the perspectives of students, teachers and external accreditation auditors differ substantially, personalisation of functions based on the user’s role in the curriculum management process is highly recommendable. Balzer et al. developed a hierarchical system of online users’ roles with different rights to read or write in the designated sections in the LOOOP system [17]. In the MERlin curriculum mapping tool, management of differences in deployments was enabled by a three-tier hierarchy of users with one global administrator, several local administrators (responsible for on-site management of organisational structures), and local users (as mappers and curriculum developers) who authored and maintained the actual curriculum [20]. In the paper by Cottrell et al. [22], two user role models were presented. The first model was decentralised and involved course and clerkship directors who individually annotated their courses with tags from a controlled vocabulary (West Virginia University). The second tested model was centralised with 1–2 staff members working as points of contact to collect session plan templates and then enter them into the software system (Texas A&M). There was no clear preference as to which of these two models was better.

The described tools were implemented using web technologies. This enabled the use of systems from different client platforms and freed the user of the need to install any additional software. To further streamline the user experience, some of the developed tools underwent intensive usability testing [14] and were equipped with drop-down lists, check boxes and auto-completion fields to speed up curriculum mapping [14], [21]. The users could select flexible, exploratory views and an individual navigational path through the curriculum with personalised queries [14], [20], [27]. The writing process of descriptions was accelerated by the availability of prefilled session mapping templates [18], [22]. Finally, curriculum mapping skills were honed by organising dedicated courses and setting up telephone hotlines, frequently asked question sections on websites, and e-mail ticket systems for instant, tracked user support [17], [31].