Study design and ethical considerations related to using direct observation to evaluate physician behavior: reflections after a recent study

Simulation vs. observation of real-time patient care activities

Over the last several decades, researchers have used both simulation and observation to audit physician behavior [8], [9], [10]. Simulation offers the advantage of a controlled environment with time and space to reflect on results immediately after a patient-physician encounter. However, while some experts in simulation have explored ways to impose distractions and time pressures on research subjects [11], [12], it may not be possible to replicate the complex conditions of a busy emergency department. An alternative to simulation is to observe patient care activities as they unfold, which is a technique that experts in human factors advocate [8], [13], [14]. A third option is to use unannounced standardized patients, which may be an appropriate middle ground for clinical settings where patients have pre-arranged appointments [15], [16]. While there is precedent for using unannounced standardized patients in the emergency department [17], the American College of Emergency Physicians has a policy statement that opposes the use of “fictitious patients” because of unintended negative effects such as delays in care for patients with critical conditions [18]. To determine the best course of action, our research team held discussions with ED administrators. For this particular study, we collectively decided to proceed with direct observation rather than simulation or unannounced standardized patients after considering the aforementioned factors.

Mode of observation

To obtain data describing physician actions, we considered the use of audio-recordings, video-recordings, real-time observation, or a combination of those modalities to generate data to describe physician actions during encounters with patients. Audio alone would be inadequate for our purposes, since we wanted to obtain data describing physicians’ observable actions related to the physical examination. Video might be sufficient, but we worried that patients would be less likely to participate, especially if they were seeking evaluation of conditions they considered embarrassing. Additionally, comprehensive evaluation with video would require two or more angles, which would be costly to set up and potentially burdensome to troubleshoot in the event it malfunctioned. Finally, real-time observation would be labor-intensive, but it also had the advantage that it would operate within the norms and expectations of a teaching facility, since it was typical for medical students or other observers to shadow physicians in our emergency departments. Given these factors and our specific clinical environment, we settled on the use of real-time observation to obtain data describing the physical examination.

Identity of observers

After deciding to utilize real-time observation to obtain physical examination data, we contemplated using several different types of observers. The most resource-intensive option would be to use physicians. Besides being costly, this option might induce the physicians under observation to alter their behavior [19]. Medical students were a second option, and using them seemed attractive because they would already have some clinical experience, including introductory training about how to perform physical examinations. However, we knew that we would require several dedicated individuals with flexible schedules to complete our goal of observing 240 encounters at two different institutions. For this reason, we decided to recruit undergraduate students to shoulder most of the burden.

Recruitment and training of observers

Undergraduate students, particularly those planning to pursue careers related to health, have a reputation for being highly dedicated, competitive individuals [20]. Thus, we designed a multi-stage recruitment and training process leveraging competition, which ultimately led to the recruitment of a final cohort of observers who had demonstrated outstanding performance at every stage. The first stage had already been completed for us. A self-selected undergraduate student organization, known as the Emergency Medicine Research Associates Program, had already recruited a group of about 100 students to participate in various emergency medicine research projects (Inclusion in the group required submission of transcripts, letters of recommendation, and two rounds of interviews) [21]. For that group, we held an educational session where we described and demonstrated physical examination maneuvers. We then presented our study idea and asked for volunteers to attend future training sessions. Thereafter, we held two more 2-h sessions in which 25 students practiced with a physical examination checklist, participated in a live examination, and completed a final, video-based examination. All participating observers scored higher than 97% on the video-based examination. Ultimately, we selected the ten undergraduate observers with the best scores to be members of our observation team.

When we commenced data collection for the study itself, both physician authors participated for the first 20 cases and compared our results with those of the trained undergraduate students. The initial interrater reliability among the group was promising (>97%), and we discussed disagreements immediately following the encounters. This led us to make some specific decisions about standardization of data collection process (e. g., Does an extremity examination require exposure of the entire extremity?), and we presented these recommendations to the team of observers to enhance the reliability of the entire team’s work. Throughout the study, we continued to use multiple observers for each patient-physician encounter when it was feasible so that we could generate robust interrater reliability scores. Overall, two observers were present for 53 of the encounters. We audited their agreement scores periodically throughout the study period, and this led us to believe that our observational data were reliable. Details describing inter-observer agreement are available in the original research article and its accompanying supplement [1].

To deceive or not to deceive

The final and perhaps most significant design consideration we faced was whether to deceive the study participants about the research goal. We feared that, if we described our true intent, they would alter their behavior and invalidate our results. Thus, we opted to deceive them: We told recruited physician study participants that we would be performing a time-and-motion study to understand how they performed histories and physical examinations. (In other words, we intentionally neglected to mention that we would be reviewing documentation.) In accordance with federal rules and regulations about protecting research subjects, we obtained specific institutional review board approval for the act of deception [22], [23]. Consequently, we were required to report the true intent of the study to participants after data were collected and give them a chance to withdraw participation.

Practically speaking, deceiving the physician participants meant that we had to maintain secrecy from everyone at the institution except core members of the research team. This proved challenging, because data collection occurred over the course of nearly 2 years. During this period, several participating physicians asked what we were “really studying”, so it became clear that some were suspicious of our intent.

Protections for research subjects

During the study’s initial conceptualization, we had engaged experts in study design to ask how to best protect the participating physicians. The protections we chose included typical research subject protections, such as keeping the identities of participants anonymous and destroying data, as well as more atypical ones, such as keeping the name of the institutions unpublished and masking the exact interval of data collection. Furthermore, we obtained a Certificate of Confidentiality from the federal government to prevent our research team from being required to disclose our results in the event of any legal action. (Note that the National Institutes of Health automatically awards Certificates of Confidentiality for any work that they fund. However, our study was funded internally, so we had to complete a brief application to receive the certificate.) [24]

Guidance regarding the acceptability of deception

The ethics of deception in research have been debated for decades, and some stakeholders believe deception is never ethical [25]. However, most published guidance seems to suggest that deception is allowable under certain conditions. For example, the Belmont Report recommended that “research involving incomplete disclosure” should have “an adequate plan for debriefing subjects, when appropriate” [26]. Currently, deception in research is discouraged by societies representing groups of professionals including psychologists and sociologists. However, these groups still allow for deception in low-risk settings as long as deception is disclosed to participants and participants can withdraw participation after the disclosure [27], [28]. Importantly, ethicists maintain that disclosure in a debriefing does not eliminate the “wrong” of deception. They encourage researchers to explore other options besides deception, and they recommend that researchers electing to deceive research subjects need to justify this decision and explain why alternatives are inferior [22], [29], [30], [31], [32].

Our experience with disclosure of deception

We first disclosed the study’s true intent and our results to physician research subjects during a conference call. At first, the research subjects were quiet, perhaps because they were considering what the implications of the results were (i. e., Am I a bad doctor? Are we all bad doctors? Is the system the problem?). A few spoke up and expressed concern that the results would harm the residency program’s reputation. We validated their concerns and then offered our opinion that the results could influence national policy toward adopting new rules and regulations that did not incentivize expanding documentation. A few participants seemed to like that idea. We closed the call by offering the participants further opportunities to discuss the results on a one-to-one basis and explicitly presented the option that individuals could withdraw their data if they wished.

A few weeks after the phone call, we presented our results in departmental grand rounds, which stimulated both support and opposition from participants and non-participants. Over the ensuing several weeks, we spoke with participants in one-on-one settings. Several individuals expressed dismay that the research team betrayed them. Others expressed fear that publication of the results would lead to legal consequences for them or the institutions where the work was performed. Ultimately, three out of our 12 physician participants decided to have their data withdrawn prior to publication because of such fears. While we were disappointed with this result, we respected (and still respect) the decisions made by those individuals. After all, the initial consent process was indeed misleading, and they ultimately perceived that risk to their professional reputations would exist for several years after publication of the study.

Reflections on downstream implications of deceiving research subjects

One author (CTB) is relatively junior within his organization. As such, he received comments from a few mentors during the study planning phase that discouraged him from completing the work because the ethics were controversial. Two such comments were: “This study is career suicide for you, since the results will be unpopular for your institution.” and “How will you ever recruit physicians to participate in future work?” Our research group thought carefully about whether the work was important to advancing the practice of medicine and whether the junior author’s career would realistically experience adverse effects. In the end, we decided that our motivation to present this work was strong because of its potential impact, and we believed that negative career consequences were unlikely to be major–though it is indeed possible that recruitment of physicians for future work may indeed prove challenging because we deceived our physician colleagues on this occasion.