A Safety-II approach to learn from practice variation in the diagnostic process in the emergency department: an action research study

Design

A single-center action research study with micro-experiments to identify best practices in the diagnostic process followed by the implementation and evaluation of practice changes [24]. This approach differs from traditional quality improvement methods (e.g. Plan-Do-Act-Study cycles) as these approaches start with predefined problems, while this approach is inductive as we started observing daily practice with the aim to identify practice variation.

The study consisted of a preparation phase followed by three subsequent observation cycles during which the diagnostic process of residents at the ED was directly observed to identify practice variation [25], 26]. This practice variation was discussed in co-creation sessions with stakeholders leading to the identification of best practices. Based on these identified best practices, practice changes were designed and implemented to improve the diagnostic process, and were evaluated in subsequent observation cycle(s).

Setting

The study was carried out at one of the largest EDs in the Netherlands, with approximately 45.000 presentations annually, from November 2021 until July 2022.

In the Netherlands, patients are often referred to the ED by general practitioners (GPs) as they act as gatekeepers to the wider Dutch healthcare system. GPs frequently make a preannouncement phone call to the consultant as part of the referral of a patient to the ED. Based on their medical urgency, patients arrive at the ED either by own transportation or an ambulance. At the ED, patients arriving by ambulance receive an ED room immediately. Patients arriving with their own transportation are placed in the waiting room, triaged according to their medical urgency, and moved to an ED room based on their medical urgency and room availability. At the ED, patients are primarily evaluated by nurses and residents, the latter subsequently discussing the case with their supervising consultant(s).

Participants

There were three types of participants: patients, residents, and stakeholders.

Materials

A standardised observation form was developed iteratively to objectively document information associated with the diagnostic process (see Supplemental Appendix 1). A first draft was developed by the study supervisors based on a model of the diagnostic process [1]. This version was revised based on the input of stakeholders in the kick-off co-creation session and two pilot observations, and subsequently used in the first observation cycle.

The observation form included an ‘other’ section for observers to report additional factors or circumstances that potentially influenced the diagnostic process. Every observed practice that potentially influenced the diagnostic process was considered a relevant finding. On the observation form, the exact time of all actions (e.g. supervision) was documented, thereby enabling the analysis of work-as-done using a chronological storyline. As contextual factors are known to influence the diagnostic process, the context in which actions took place in the diagnostic process was also documented on the form (e.g. face-to-face or by phone) [27].

Procedures

The study consisted of a preparation phase and three subsequent observation cycles. Each observation cycle consisted of the same six steps (see Figure 1). The preparation phase took two months in total. Each observation cycle took 8 weeks (two months).

Figure 1:

The six steps of each observation cycle.

Observation cycle 1

The first observation cycle revealed practice variation in how the preannouncement of the referral was handled. In one out of 10 patients, information on the preannouncement was documented in the EHR, which included presenting complaints, the referring physician, the medical history and suggested additional diagnostics. In the other nine cases, the information of the preannouncement was transferred by a phone call to the resident, with a limited amount of the aforementioned information (see Table 2). During the co-creation session, stakeholders identified the documentation of information about the preannouncement in the EHR as a best-practice, as it could prevent the loss of data and save time in the diagnostic process. To apply this in clinical practice, a standardised EHR template for the preannouncement was identified as a feasible practice change. The stakeholders identified seven items to be documented in this template, named the ‘Successful Seven’ (see Supplemental Appendix 2). This template was embedded in the EHR and shared with all involved personnel.

Furthermore, practice variation was seen in the use of digital information resources with some residents not using digital information resources related to the diagnosis or treatment while others used resources varying from international subscription databases (UpToDate) to national/local guidelines, external guidelines and Google searches. During the co-creation session, stakeholders reached consensus that a document with relevant digital information resources, supported by consultants, could facilitate the diagnostic process. A document with relevant digital information resources was created through input from six consultants from different subspecialties and subsequently shared with all involved personnel.

Observation cycle 2

Practice variation was observed in the extensiveness of the information documented in the EHR template with some consultants e.g. documenting intended diagnostics. Discussing this practice variation, residents and nurses stated that the diagnostic process went particularly smooth when intended diagnostics were documented in the ‘Successful Seven’ as they were more aware what additional samples needed to be taken upon ED arrival of the patient. Consensus was reached to imbed intended diagnostics in the ‘Successful Seven’ template in the EHR. All involved personnel was informed about the change in the EHR template.

Besides that, practice variation was seen in the method and frequency/timing of supervision of residents (see Table 2). In the co-creation session, the physical presence of Emergency Medicine consultants on the ED was identified as a best-practice, and consensus was reached that face-to-face supervision by Internal Medicine consultants was feasible and can be beneficial to the diagnostic process. All Internal Medicine consultants and residents were informed about this practice change in ED supervision.

Furthermore, variation was observed in the timing of obtaining blood samples in patients going through triage with blood samples being taken in triage in 2/7 patients (see Figure 2 and Table 2). During the co-creation session, stakeholders reached consensus that obtaining blood samples in triage could improve the ED workflow. The involved ED personnel was informed about the blood sampling during triage.

Figure 2:

Work-as-done showing practice variation in the timing of taking of blood and urine samples in the seven patients going through triage. In cases 3 and 7, the blood samples were taken in triage. PE, physical examination.

Practice variation was observed in the number of interactions within the diagnostic team: varying from 3 to 22 interactions by residents with other professionals. Discussing this practice variation, it was unclear for stakeholders whether variation in the communication within the diagnostic team was desired or undesired variation as it could be there is more communication in complex cases and variation is therefore desired. Therefore, in-depth observation of communication was postponed to a subsequent project.

Observation cycle 3

The observations showed that the Internal Medicine consultants documented information about the preannouncement in the EHR in 75 % of the patients referred during office hours, all supervision during office hours was carried out face-to-face (100 %), and blood samples were taken in triage in 67 % (6/9) of the patients going through triage. In the co-creation session, the ED nurses and residents agreed that adding the intended diagnostics to the ‘Successful Seven’ was beneficial to the diagnostic process. Stakeholders reached consensus that face-to-face supervision during office hours and blood sampling in triage were feasible.

Practice variation was observed in the timing of serum lipase testing (see Table 2). During the co-creation session, the stakeholders agreed that adding lipase to the standard ED laboratory package for patients presenting with undifferentiated complaints could save time in the diagnostic process. Serum lipase testing was therefore imbedded within the standard laboratory package at the ED for patients presenting with undifferentiated symptoms.

Furthermore, practice variation was observed in the use of referral letters from the GP or ambulance. Discussing this practice variation, the stakeholders reached consensus that the use of referral letters was a best practice, but that implementing a practice change was unnecessary as they were already used in the majority of cases (11/15).

Besides that, speak-up behaviour from a resident towards a consultant was observed. Regarding this, the stakeholders reached consensus that it was promising behaviour, but that implementing a practice change would require action behind the scope of this project.

Statement of principle findings

This study showed how a Safety-II approach can be applied to identify potential improvements in the diagnostic process. This was achieved by observing and reflecting on practice variation, followed by designing and implementing practice changes in the diagnostic process based on identified best practices. The observed practice variations and identified best practices showed that combining action research with direct observations is a suitable method to apply Safety-II in the diagnostic process in a complex and adaptive ED environment, and can lead to well-received practice changes.

In this study, we were able to learn from all 40 observed diagnostic processes instead of only learning from the five patients with misdiagnosis. The observations of the diagnostic process led to identification of eight practice variations, which were extensively discussed with stakeholders in co-creation sessions, and led to the identification of best practices. After reaching consensus on the potential impact and feasibility of these best practices, practice changes were co-designed with the stakeholder group for implementation in daily practice. This led to the implementation of five practice changes in the diagnostic process at the ED. Evaluation of the practice changes during subsequent co-creation session showed that their uptake varied from 67 to 100 %, and overall good stakeholder satisfaction.

Strengths and limitations

A particular strength of this study is that the practice changes were made without the need to identify diagnostic errors (which has shown to be challenging) [18]. Yet, registering practice variation has shown to be an excellent basis for improving the diagnostic process. Furthermore, our action research design with direct observations can potentially be used in other clinical settings. Another strength is the involvement of patients and medical professionals directly involved in the diagnostic process during co-creation sessions. This led to the identification of best practices and development of practice changes in the diagnostic process, which were well-received as they were tailored to the everyday ED work based on the expertise of the stakeholders.

A limitation of this study is that the implemented practice changes are relevant for the ED of the participating hospital, and may not be generalisable to other hospitals. However, the use of the Safety-II approach, in an action research design, is generalisable and could elicit practice variations elsewhere. Another limitation, inherent to the qualitative study design, is that a certain level of subjectivity of the study supervisors and the stakeholder group plays a role in the identification of practice variation, and development of practice changes. Lastly, our study was not designed to formally evaluate on a large scale whether the practice changes improved the diagnostic process. However, informal small-scale evaluations of the practice changes were performed during co-creation sessions based on the observations and the experiences of the stakeholders with the practice changes, and changes were made based on their practical use. Longer term and more comprehensive analysis of the impact of our approach and the implemented practice changes are needed for formal evaluation.

Interpretation within the context of the wider literature

To our knowledge, no other studies have used a Safety-II approach or similar approach to the diagnostic process. Based on our study findings, the Safety-II approach can unravel the complexity and variability in the diagnostic process, what is consistent with the expectations of the use of this approach in current Safety-II literature [15], 19], 21], 30]. This study adds to a scarce amount of studies bringing Safety-II concepts into medical practice, and of Safety-II studies leading to interventions improving quality of care [19], 20]. This study could catalyse additional work on using the Safety-II approach to study the diagnostic process.

Furthermore, this Safety-II study combines repetitive cycles of action research with direct observations. Most current Safety-II studies used FRAM to apply the Safety-II approach to improve processes in healthcare [31], [32], [33]. However, these studies mainly focussed on small demarcated complex processes (e.g. preoperative anticoagulation management), and the use of FRAM seems less suitable for complex and less demarcated processes (e.g. diagnostic process). Our action research design, with active stakeholder involvement in constructive co-creation sessions, led to better overall consensus on the development and implementation of practice changes, and facilitated their adoption in daily practice. Active involvement of stakeholders has shown similar results in checklist studies [34], 35].

In this study, we identified practice variation in the diagnostic process using the Safety-II approach, which resulted in the introduction of actionable practice changes in the diagnostic workflow protocols. Thereby, the implemented practice changes resulted in reducing the previously observed practice variability, what is more aligned with Safety-I thinking. However, through our iterative observations and co-creation sessions, we found that some variations reflected desirable adaptability to specific clinical circumstances and therefore were not considered suitable for standardisation. In other cases, practices initially considered promising for standardisation were later found to be context-dependent and ultimately retained as acceptable or even beneficial variations. This process illustrates that we did not seek to eliminate all variability, but rather to distinguish between variations that enhance adaptability and those that undermine safety and efficiency. Importantly, many of the identified best practices were derived from examples of care that went particularly well, which might not have been captured through a Safety-I lens alone. In recent literature, Safety-I and Safety-II are seen as distinct yet complementary to each other, and integrating them makes it possible to improve patient safety by learning from both things that go right and things that go wrong [30], 36], 37]. This is also supported by our study.

During the course of our study, we experienced a culture shift among the stakeholders to focus more on things that went well and identification of best practices in daily practice. This is in line with recent studies that concluded that looking at what goes well in everyday work (Safety-II approach) is less threatening for medical professionals compared to analysis of incidents (Safety-I approach), and will thereby contribute to a positive culture [21], 38], 39]. The ‘Successful Seven’ template for the preannouncement documentation in the EHR was adopted by seven other specialties at the ED that used similar templates adjusted to their needs. This shows how the positive approach contributed to a learning culture at the ED.

Implications for policy, practice and research

While our study shows the potential of the Safety-II approach, also illustrated by its adoption in recent grant programs by the Agency for Healthcare Research and Quality (AHRQ), Safety-II has sparked considerable debate in safety sciences. Critics argue that Safety-II merely reframes established quality improvement practices, and focuses too heavily on the human operator, ignoring the relevance of the broader sociotechnical system [40].