What Skills Really Improve After a Flipped Educational Intervention to Train Medical Students and Residents to Break Bad News ?

Breaking bad news (BBN) is necessary in medical practice and requires training. The purpose of this study is to evaluate the efficacy and mainly explore the components involved in medical students’ and residents’ performance after a flipped educational intervention to train them to break bad news. A randomized controlled before-after study was conducted with 43 medical students and residents in the intervention group and 41 in the control group. The intervention combined an online multimedia program (DocCom) with a two-hour workshop. BBN performance was assessed at two clinical stations using Objective Structured Clinical Examination and analyzed using a mixed between-within subject analysis of variance. A factor analysis was conducted to analyze the performance by checklist components. The intervention group improved its overall performance in BBN over time (p = 0.000; Eta = 0.38) and when compared to the control group (p = 0.01; Eta = 0.12). The factor analysis revealed two main components: Factor 1—“giving bad news and responding with empathy”—and Factor 2—“using general communication skills”. Performance analysis by these components revealed that the improvement occurred mainly in Factor 1 (over time, p = 0.000; Eta = 0.48, group x time, p = 0.000; Eta = 0.38). The intervention combining DocCom Module 33 and a workshop had a moderate effect on the improvement of medical students’ and medical residents’ BBN overall performance in standardized encounters. This improvement was mainly related to communication skills for giving bad news and responding with empathy, in which the intervention effect was large over time and between groups.


Introduction
Breaking bad news (BBN) is necessary in medical practice and has an important effect on both patients and physicians (Baile et al., 2000).The use of relationship-centered communication skills (CS), which includes addressing patients' knowledge, perspectives, concerns, feelings and demands, as well as acknowledging and responding to their emotions with empathy and compassion, help patients cope better with bad news (Barnett et al., 2007;Fallowfield & Jenkins, 2004;Ranjan et al., 2015).However, when BBN is inappropriate, patients and physicians may feel dissatisfaction and discomfort, and physicians may even face malpractice lawsuits (Liu et al., 2015;Langewitz, 2017;Collins et al., 2018).This process requires training, and frameworks have been developed in order to guide physicians and other health professionals in BBN (Baile et al., 2000;Narayanan et al., 2010;Quill et al., 2014;Rat et al., 2018).BBN training programs usually associate theory with other teaching strategies, such as small group discussions, role-plays, simulations with standardized patients, and/or direct observation (Brown et al., 2009;Choudhary & Gupta, 2015).Less frequently, programs have used online modules, demonstrative videos and other multimedia resources (Novak et al., 2002;Cook et al., 2011).Feedback on trainees' performance is strongly recommended because it enhances their awareness of skills that are already used effectively and those that could be better developed or corrected (Kiluk et al., 2012).
A systematic review of the effectiveness of interventions involving BBN training for end-of-life care when compared to usual teaching, showed that among the 20 selected studies, only 8 had interventions that improved trainees' end-of-life communication scores rated during standardized patient encounters and the evidence quality was very low (H- O Chung et al., 2016).A meta-analysis of the effectiveness of interventions for BBN training in the oncology field found low to moderate improvement in BBN skills when performance was evaluated by simulated patients (Barth & Lannen, 2011).According to the authors, this low to moderate effect on the performance could be explained by a "ceiling effect" since participants had a prior high level of communication competence in BBN.They recommend that primary studies be encouraged to provide results for less experienced professionals.Furthermore, they suggest that studies present both the "overall communication competency score and some key dimensions of communication behavior", so that we can better understand which dimensions during the BBN process most benefit from some intervention.
In order to try to fill this knowledge gap, we developed a study to evaluate the efficacy and explore the components involved in medical students' and residents' performance after a flipped educational intervention to train them to break bad news.Our hypothesis is that the intervention will be effective and that within the BBN process its effect will be greater in the dimension related specifically to communications skills to BBN.

Study Design and Participants
We conducted a controlled before-after study between 2015 and 2016.The research project was approved by the Ethics Committee of the Federal University of Santa Catarina (UFSC), number 33292114.60000.0121.
We invited all UFSC medical students who were in medical clinical rotations during clerkship (n = 100), internal medical residents (n = 36) and family and community medical residents (n = 24).We calculated the sample size by considering a 95% chance of detecting a 5% increase in the BBN performance of the control group (CG) and a 35% increase in that of the intervention group (IG) as a measure of the primary outcome.The result was a sample size of 80 participants, with 40 in each group (Sealed Envelope, 2012).
Ninety one of the 160 eligible medical students (MS) and medical residents (MR) invited agreed to participate [34 MS (37.4%) and 57 MR (62.6%)].We randomly assigned participants to either the IG or the CG using the random number generator of the OpenEpi program, stratified by categories (MS and MR).The IG included 46 participants (50.5%), while the CG had 45 (49.5%), who were assigned to a waiting list.

The Flipped Educational Intervention
The intervention design was based on the teaching strategy used at Drexel University College of Medicine to teach BBN.Participants were asked to watch DocCom Module 33 "Delivering Bad News" online and send a response to one of the researchers for two DocCom reflection questions: "What have been your reactions or your family's reactions when you have heard bad news in the past?"; "In thinking about the communication of bad news, what makes it go well; what makes it go poorly?From whose standpoint are you answering this question?"(Quill et al., 2014).DocCom is an online resource for teaching and learning healthcare communication skills, featuring 42 multimedia modules.Each module contains evidence-based theory with an in-depth explanation of the relevant communication skills, together with videos demonstrating these skills and an annotated video demonstrating and commenting on those communication skills being applied.
Fifteen days after watching the DocCom video, a two-hour-workshop facilitated by one of the researchers was held with groups of 6 to 12 participants.Initially, based on the responses to the reflection questions, the facilitator encouraged participants to share their thoughts and personal experiences regarding BBN with the whole group.It was followed by a synthesis of what they learned from the module and a recap of the six steps for BBN.This process lasted 45 minutes.Afterwards, for one hour, the participants practiced the skills they had learned in three role-play scenarios based on clinical cases, alternating the roles of patient, physician and observer in each scenario.Feedback was given at the end of each role-play session.The last 15 minutes of the workshop were used for debriefing.

Outcome Measurement
All subjects participated in an Objective Structured Clinical Examination (OSCE) fifteen days before the intervention (OSCE1) and fifteen days after the intervention (OSCE2) in the communication skills laboratory.This setting has a system to capture video, allowing future analysis.
Each OSCE had two clinical stations.One station was the communication of HIV positive test result to a patient with chronic diarrhea and weight loss.The other was the communication of pulmonary cancer to a patient with recurrent pneumonia.OSCE1 and OSCE2 involved these same clinical cases, which were performed by different standardized patients (SP).The SP were volunteers who were invited to take part in the project.The SP received a script with detailed information about their roles and how to play them and were briefed on what was expected from their performance.After rehearsing their roles, the SP were evaluated in a role-play scenery with one of the researchers.Those SP who showed satisfactory performance participated in the OSCE stations.
Participants' performance in the OSCE was assessed using a 14-item checklist based on the six steps for BBN (DocCom Module 33) and initial patient reception (DocCom Module 6, Table 1).The OSCEs were video-recorded and two researchers, blinded to each participant's group and to the OSCE (before or after the intervention), watched the videos and rated the each OSCE checklist item.They rated zero when an item was not accomplished and one when the item was accomplished.To maintain uniformity in list completion, before rating the OSCEs, the raters discussed participants' behaviors expected for each item.Next, they watched 10 videos, compared the rates given, and discussed the differences for calibration purposes.Then, all the videos were individually rated by the two researchers.When there was a difference between the rates assigned, a third person, also duly blinded, decided the final rating.
The CG, in the period between the OSCEs followed with their curricular activities as usual, learning to break bad news through the observation of their preceptors.After OSCE2, feedback was offered to all the participants who were willing to receive it.For ethical reasons and educational purposes, we offered the intervention to the CG at the final research.

Data Analysis
The OSCEs' final scores were calculated as the means of the two station ratings.In order to homogenize the variances, we conducted the analysis using standard scores (Z) based on the raw scores: where X is the individual raw score, M sample is the medium sample considered, and SD sample is the standard deviation within the sample.Then, in order to facilitate the presentation of the results, we transformed the Z scores into standardized T-scores (T) through the formula (Adeyemi, 2011):

T = (10 x Z) + 50
The OSCE interrater reliability was calculated using the kappa coefficient (κ) and the checklist internal consistency was evaluated using Cronbach's alpha coefficient.
We analyzed baseline differences between gender and category (MS versus MR) using Pearson's chi-squared test.We analyzed OSCE performances differences between categories in the first OSCE ("baseline performance") using independent samples t-test and OSCE performances before and after the intervention using mixed between-within subjects analysis of variance.The effect size was established using partial eta-squared, considering 0.01 a small effect, 0.06 a moderate effect, and 0.14 a large effect (Cohen, 1998).
In order to better understand if the performance was similar in all the OSCE checklist items, we conducted an exploratory factor analysis of the main components with orthogonal rotation (Varimax).We considered significant an alpha level of 0.05.

Participants' Profile
Eithy-four participants (n = 84) completed both OSCE, before-after intervention and were included in the date analysis.Four MR in the CG failed to participate in the study; one MS and two MR in the IG also declined to take part in the intervention.Thus, the final sample comprised 84 participants, who completed all the stages of the research process.The participants' mean age was 26.3 years (SD = 3.6); 52 were women (59.8%) and 35 were men (40.2%).There was no baseline difference between the number of IG (n = 43) and CG (n = 41) participants regarding gender (p = 0.67), age (= 0.77), or participant category (p = 0.74).

OSCE Checklist Factor Analysis
The checklist reliability was 0.73, and the interrater reliability was 0.83.
The Factor Analysis revealed four components with eigenvalues exceeding 1, explaining 15.5%, 14.2%, 12% and 11.2% of the variance, respectively.An inspection of the scree plot revealed a clear break after the second component.Therefore, we decided to retain two components (two factors) for further investigation.
Factor 1 included 9 items of the six steps for BBN presented in DocCom Module 33 related to CS sharing illness information, acknowledging and responding to emotions, and for partnership (items 4, 5, 6, 7, 8, 9, 10, 11, and 13).This factor was named "giving bad news and responding with empathy".Factor 2 included three items from the first stage of consultation presented in DocCom Module 6 related to CS for patient reception (items 1, 2, and 3), and one item of the six steps for BBN, which is the establishment and agreement on plans for next steps (item 12).This factor was named "using general communication skills".This factorial structure explained 37.6% of the total variance of the checklist scores.Item 14 showed no variance, and was excluded (Table 2).Establishes and agrees on plans for the next steps 0.5 Note.F: Factor.

Participants' Performance Within and Between Groups
There were no baseline differences between groups (IG and CG) or categories (MS and MR) in OSCE1 performance (Table 3).The intervention had a large effect on IG overall performance improvement over time and a moderate effect on IG overall performance improvement when compared to the CG.No improvement was found on CG overall performance (Table 4).There was no significant difference between men's and women's performance over time [mean T-score = 51.9,SD = 7.8 and mean T-score = 53.7,SD = 7.8, respectively, t (82) = 1, p = 0.3].
The analysis of the participants' performance by the two components of factorial analysis revealed that the MS' and MR' performance improvement in the IG was mainly related to CS for sharing illness information, acknowledging and responding to emotions, and partnership, in which the effect was large over time and between groups (Table 5).

Discussion
The flipped educational intervention using the DocCom Module 33 and a two-hour workshop with role-play had a large effect on the CS with regard to giving bad news and responding with empathy in both medical students and residents.However, no effect was observed with regard to general communication skills, resulting in a moderate effect on the overall performance improvement.Alexander et al. (2006) studied medical residents' performance after a 16-hour intervention to break bad news, based on three predefined checklist domains: "delivering bad news", "responding to emotional cues," and "using general communication."The intervention had a moderate effect on the overall performance and there was improvement in the first two domains.These results are similar to ours.However, as we explored the checklist components after OSCE, we could verify that both domains loaded in one factor.
Our results are also similar to the findings of Gorniewicz et al. (2017), who studied the effect of an intervention using a BBN training module that incorporated patients' story preferences.They found that participants improved at their highest levels of CS with regard to "attention to patient responses after breaking bad news" and "communication related to patient emotions."However, no effect was found on other checklist items.According to these authors, one possible explanation could be that patient reception and therapeutic planning are more frequently taught and practiced as a part of the medical curriculum at the undergraduate level.Yedidia et al. (2003) teaching CS integrated in three medical schools, evaluated students' performance in OSCEs pre-post-intervention with predefined five dimensions in their checklist, one of them was to BBN.The authors found significance improvements in the intervention group in this dimension.However, they did not reveal the effect size of the results and neither did they conduct a factorial analysis of the checklist.Instead, the authors evaluated the five dimensions of CS separately, which allowed a better understanding of the main skills related to BBN.Using the factorial analysis, we intend to demonstrate the specific skills of BBN and analyze the effect size difference of the components of the checklist.Apart from that, our checklist was reliable, and Cronbach's alpha coefficient was similar to the results of Yedidia et al., which was 0.7.
This study had some limitations.It was conducted in only one institution and these results cannot be extrapolated to other medical education institutions.Also, we include the first 91 people who were willing to participate instead of a random selection thereof may have introduced an initial selection bias.
Our intervention was effective but somewhat difficult to work because it required three distinct days to the participants take part of the intervention in addition to the hour to watch the module on line.The risk of dropping participants during this time was high, but only seven participants declined to take part in the study after their initial formal agreement.
More studies are needed to evaluate the long-term impact of the intervention and investigate IG's performance with real patients.In addition, it is necessary to analyze the effectiveness of the intervention by dimensions; therefore, we suggest that factor analysis should always be conducted to better understand the effect of interventions and the main points to teach the BBN communication.

Conclusion
The intervention combining DocCom Module 33 and a two-hour workshop had a moderate effect on MS' and MR' BBN overall performance.The factorial analysis allowed us to identify that the improvement was mainly related to CS with regard to giving bad news and responding with empathy, in which the intervention effect was large over time and between groups.

Table 1 .
Objective structured clinical examination checklist for the evaluation of medical students' and residents' performance in breaking bad news

Table 2 .
Factor analysis of the objective structured clinical examination checklist for breaking bad news

Table 3 .
Medical students' and residents' baseline overall performance in the objective structured clinical examinations before (OSCE1) an intervention

Table 4 .
Medical students' and residents' overall performance in the objective structured clinical examinations before (OSCE1) and after (OSCE2) an intervention combining an online multimedia program (DocCom) with a two-hour workshop to train them to break bad news Mixed between-within subjects analysis of variance (over time = within groups; group x time = between groups); SD: Standard Deviation; ES: Effect size; MS: Medical Students; MR: Medical Residents; IG: Intervention Group; CG: Control Group.

Table 5 .
Medical students' and residents' performance in the objective structured clinical examinations by checklist component, before (OSCE1) and after (OSCE2) an intervention combining DocCom Module 33 and a workshop to train them to break bad news Daetwyler et al. (2010)also used DocCom Module 33 in their study.They supposed with the results in a pilot test that by adding educational interventions (WebOSCE to DocCom) the effect in participant's performance (medical students in clerkship) tends to be larger comparing with only watching the module.But they did not explore the checklist components.