A mixed-methods program evaluation of the Alda Healthcare ... - BMC Medical Education
Project aims & design
Grounded in implementation science and program evaluation [30], the present project will evaluate the implementation and effectiveness of the AHE, a medical improv workshop designed to improve communication skills among healthcare professionals. Specifically, using mixed methods, we will conduct an Effectiveness-Implementation Hybrid Design project [31] that includes [1] an outcome evaluation to determine the impact of this workshop on communication skills among healthcare professionals (effectiveness), and [2] a process evaluation to explore interprofessional communication and its effects on AHE implementation fidelity (implementation).
Our outcome evaluation will assess the impact of the AHE workshop on short- and long-term improvement in several relevant outcomes, including interprofessional communication, stress, and professional fulfillment. We will use an embedded experimental explanatory sequential mixed methods design for implementation and evaluation [32, 33]. The evaluation will use participatory action research as a guiding framework and will include both qualitative methods (e.g., dimensional analysis employing individual interviews) and quantitative methods (e.g., quasi-experimental longitudinal [34,35,36]). In the first phase (see Fig. 1), we will deliver the AHE workshops and conduct a quantitative prospective outcome evaluation to assess the intervention effects using a quasi-experimental approach with an intervention group and surveillance of a control group (wait-list).
Evaluation design
Next, our mixed-methods process evaluation will examine programmatic and individual barriers or facilitators to effective implementation of the AHE medical improv communication workshop. This evaluation will consist of qualitative interviews and quantitative analyses to explore organizational-, departmental-, and individual-level factors, related to interprofessional communication and its effects on delivery, fidelity, and effectiveness of the intervention. We will use our quantitative findings to inform the selection of AHE Informants (key AHE Stakeholders and AHE Project Participants) for qualitative in-depth interviews focused on interprofessional communication and the AHE workshop experience. In addition, we will conduct repeated brief interviews with clinical co-facilitators after AHE workshop sessions. Qualitative interviews will be aligned with dimensional analysis method [37, 38].
Alda healthcare experience intervention
The AHE workshops will target approximately 550 healthcare professionals and nurses—both practicing professionals and trainees. Physicians, resident physicians and fellows, and practicing nurses will be drawn from five Stony Brook University Hospital Departments (Surgery, Medicine, Obstetrics/Gynecology, Family Medicine, and Pathology) whose leaders have indicated their support for this project. Participation in the AHE workshop will be mandatory and free; participants will also receive continuing education credits (e.g., Continuing Medical Education, Continuing Nursing Education) for their attendance. All personnel trained in the AHE workshop will be eligible to participate in the program evaluation.
As suggested in the literature [14], each workshop is co-facilitated by two leaders, an improv-trained facilitator and a clinically trained facilitator. Using a train-the trainer approach, clinical professionals will be nominated by leadership from each participating department to receive specialized training by the Alda Center team to enable them to co-facilitate the workshops and be ambassadors of change in their department. Approximately 18 clinical co-facilitators will be trained by going through 2 3-hour group sessions followed by 1-hour (maximum) 1:1 coaching. The clinical co-facilitator training includes experiencing the AHE workshop as a participant (novel to the AHE). Participation in the evaluation component, in addition to the training, will be voluntary.
The Alda Center for Communicating ScienceⓇ has assigned three improvisation facilitators to this project. All three have extensive experience designing and facilitating experiential learning using the Alda MethodⓇ, and have taught prior versions of this curriculum. In addition to facilitating the workshops, one facilitator will serve as the designated liaison to the project team, lead curriculum designer and lead clinical co-facilitator trainer.
Based on the fundamentals of medical improv and experiential learning, the AHE program includes a two-hour in-person workshop comprised of a series of improvisational exercises that focus on specific communication skills (recognizing the give and take of relational communication, listening to collaborate, reflecting values that are shown through passion, thinking creatively and generativity about circumstances) followed by a discussion about practical application in healthcare [14,15,16]. The exercises are immersive and participatory - participants work in pairs or groups to complete each exercise, and then engage in a full group discussion guided by open-ended questions from the facilitators. At key points in the workshop, participants are given written prompts to reflect. Follow-up one-hour virtual workshops will be conducted 1–3 months after the workshop in groups of up to 30 aimed to refresh the principles of improvisation to the workshop participants and introduce to a new tool to help them strategize a process to start challenging interprofessional conversations with colleagues.
Based on pilot data, to ensure maximum participation, groups will include 7–10 healthcare professionals, an improv facilitator, and a clinical co-facilitator. There will be approximately 68 separate AHE in-person workshops with a total of n = 100 healthcare professionals trained by the end of 2022, n = 350 more in 2023, and n = 100 more in the first quarter of 2024.
Participants
Three types of participants will be involved in the present evaluation: AHE participants and stakeholders, clinical co-facilitators, and improv facilitators (see Table 1). AHE participants who choose to participate in the evaluation will be asked to join the project and complete online questionnaires. AHE Project Participants will receive $5 gift cards for each completed questionnaire. We expect 85% of the healthcare professionals that take part in the AHE workshop to agree to become AHE Project Participants for a baseline of N = 468 (n = 234 in each group). In addition, 20–40 AHE Project Participants and AHE Stakeholders will be chosen to be AHE Informants and take part in qualitative interviews. Of the AHE Informants, 5–10 will be chosen as Selected Informants and participate in a subsequent focus group. AHE Informants will be purposefully sampled based on individual characteristics and informed by the quantitative findings. AHE Informants will be chosen to reflect different professions, career levels, departments, racial and ethnic backgrounds, and gender identities. They will receive a $15 gift card for their time. As part of their role, all clinical co-facilitators will be interviewed following each AHE workshop they lead. With their agreement, these interviews will be used in real-time to improve the facilitation process as well as be used for evaluation purposes. Each co-facilitator will be eligible to receive a $3000 stipend for their roles (e.g., training, co-facilitating 4–6 AHE workshops, interviews). Improv facilitators will complete a brief online questionnaire following each AHE workshop session they lead. Their hourly pay is commensurate with experience.
The project was reviewed by the Stony Brook University IRB (IRB2022–00231) and approved as a project that is not considered human subject research based on Common Rule determination. Nonetheless, prior to completing the baseline survey, AHE workshop participants were presented with a brief description of the evaluation project, their involvement, confidentiality, and compensation. Participants who were interviewed were presented similar information about their involvement in the evaluation verbally as well.
Procedures- quantitative
For the AHE Project Participants, evaluation procedures include online surveys collected during key time points for the AHE (see Fig. 1). AHE participants who are interested, will be asked to become AHE Project Participants and complete four online surveys over the course of 2 years (see Table 2) that will be administered through Qualtrics. Surveys will take approximately 3–5 minutes to complete and will repeatedly assess constructs related to teamwork, burnout, stress, well-being, and communication. The following previously validated measures will be included in the surveys:
Communication skills
Communication will be assessed at all time points using the eight item Perceived Interpersonal Communications Skills Scale (PICSS) that was previously developed by the Alan Alda Center for Communicating Science [28]. Each item assesses a different skill related to flexible empathic communication (e.g., "Beginning a conversation with an open mind"). Participants are asked to rate their skill level for each communication item on a Likert-type scale ranging from 1= "not skilled at all" to 5 = "very skilled". This scale has shown validity in previous pilot studies (Cronbach's alpha is between 0.89 and 0.91; 28). The overall score is represented as the mean of responses. Higher scores indicate higher levels of communication skill.
Cohesion
Closeness with the healthcare team will be measured at time points 1, 3, and 5 using the Inclusion of Other in the Self Scale (IOS; 39). On a single item, respondents choose one of seven pairs of circles, labeled "Self" and "Other", with differing amounts of overlap (1 = "no overlap" to 7 = "most overlap"). Higher scores indicate greater self-perceived cohesion between participants and others in their healthcare team.
Teamwork and response to errors
Teamwork and response to errors will be measured at time points 1, 3, and 5 using the Surveys on Patient Safety Culture questionnaire (SOPS; 40). Two subscales of the SOPS will be included in the current evaluation: three items measuring Teamwork (e.g., "In this unit, we work together as an effective team") and four items measuring Response to Errors ("When an event is reported in this unit, it feels like the person is being written up, not the problem"). Participants are asked to respond to each statement on a Likert-type scale ranging from 1 = "strongly disagree" to 5 = "strongly agree". The Cronbach's alpha of the Teamwork subscale is 0.76. The Cronbach's alpha of the Response to Errors subscale is 0.83. Higher scores indicate greater teamwork and effective responses to errors.
Uncertainty tolerance
The Uncertainty Tolerance Scale (UTS; 41,42) will be used to assess uncertainty tolerance at time points 1 and 5. The UTS includes two subscales: five items measuring Uncertainty Tolerance (Cronbach's alpha is 0.70) and three items measuring Need for Predictability (Cronbach's alpha is 0.73; 28). Respondents indicate the degree to which they agree or disagree with each of eight statements (e.g., "I like change and excitement"), on a scale ranging from 1 = "strongly disagree" to 6 = "strongly agree". Higher scores indicate a greater need for predictability of a greater intolerance of uncertainty.
Stress
The Short Perceived Stress Scale (PSS-4; 43) will be used to assess self-reported stress at time 1, 3, and 5. Respondents will be asked to indicate how often each of four stated situations had occurred within the past month (e.g., In the last month, how often have you felt that you were unable to control the important things in your life?) on a Likert-type scale ranging from 0 = "never" to 4 = "very often". Cronbach's alpha for the PSS-4 is typically between 0.60 and 0.82 [44]. Higher scores indicate greater perceived stress.
Work exhaustion and professional fulfillment
Work exhaustion and professional fulfillment will be measured at time 1, 3, and 5 using the Stanford Professional Fulfillment Index (PFI; 45). We will use two subscales included in the PFI. The first subscale is Burnout which includes four items related to work exhaustion (e.g., "Physically exhausted at work"). The second subscale is Professional Fulfillment which includes five items related to satisfaction with work (e.g., "I feel happy at work"). Participants respond to each statement on a Likert-type scale ranging from 0= "not at all true" to 4 = "completely true" or 0= "not at all" to 4 = "extremely." The Cronbach's alpha of the Burnout subscale is 0.92 and the Cronbach's alpha of the Professional Fulfillment subscale is 0.91 [45]. Higher scores on the Burnout subscale indicate greater work exhaustion. Higher scores on the Professional Fulfillment subscale indicate greater fulfillment.
Well-being
The PROMIS Meaning and Purpose self-report scale [46] will be used to measure well-being at time points 1, 3, and 5. Responses to each of four items (e.g., "My life has meaning") are rated on a likert-type scale ranging from 1 = "strongly disagree" to 5 = "strongly agree", or 1= "not at all" to 5 = "very much"). The Cronbach's alpha for this scale is 0.90 [46]. Higher scores indicate greater wellbeing.
Excitement about AHE workshop
The extent to which AHE Project Participants are excited about participating in the workshop will be measured at time point 1 using a Visual Analogue Scale. Participants are asked to indicate "Using the scale, please indicate how excited you are about participating in the Alda Healthcare Experience communication workshop?" on a sliding scale from 0 = not excited at all to 100 = Extremely excited.
Engagement in workshop
The extent to which AHE Project Participants were actively engaged in the workshop will be measured at time 2 for Wave 1 and time 4 for Wave 2 using a Visual Analogue Scale. Participants are asked to indicate "How engaged you were in today's workshop" on a sliding scale from 0 = not engaged to 100 = fully engaged.
Background data
AHE Project participants' sociodemographic background (age, gender identification, race and ethnicity, degree/ credentials, and career level) will be assessed at time point 1. In addition, participants will be asked whether they have any prior experience with improv, and asked to indicate what kind of experience they have.
Employee engagement
Department-level employee engagement will be indexed by absenteeism (e.g., works days missed) and retention (e.g., turnover rates) every 6 months, starting with the 6 months before T1 and ending 6 months after the T5.
Debrief Improv facilitators surveys
Improv facilitators will complete a brief (2 minutes) online survey at the end of each AHE workshop session to document the workshop environment (e.g., distractibility, disruptions, perceived cohesion, initiation of questions, willingness to engage). Several outcomes (i.e., group size, disruptions, perceived cohesion, and engagement) will be collected, using original measures designed for this project.
Procedures- qualitative
All qualitative data collection and management procedures will be carried out using institutionally secured and password protected video-conferencing and data storage software by a team of three to five project team members trained and supervised by the project qualitative lead co-investigator. Qualitative interviews will be conducted virtually by a project team member trained in qualitative data collection. Interviewers will use semi-structured interview and discussion guides to ensure data elicitation on topics central to the aims of the project, which will be iteratively refined in alignment with emerging concurrent analysis and theoretical sampling. Once completed and audio-recorded, each interview will be transcribed using an institutionally approved transcription software service, and at least two project team members will independently align audio-recordings with their corresponding transcripts to ensure verbatim accuracy and remove identifiable information.
For the qualitative process evaluation of the AHE, we will use theoretical sampling to select 20–40 individuals for in-depth individual interviews, from the pool of AHE Project Participants and AHE Stakeholders that agree to being contacted to participate in interviews. Should we face recruitment challenges resulting in fewer than expected interviews, theoretical sampling will enable sufficient depth of data for analysis through application of it during interviewing by "steering questions in the direction of emergent theorizing," as well as after data collection is complete by theoretically sampling "within the data" [32 , p. 8]. An initial portion of the AHE Informants will be selected from the Wave 1 department to allow a critical mass for analysis (between T1-T3), then additional interviews will be conducted with Wave 2 AHE Project Participants and AHE Stakeholders (between T4-T5) to ensure sample heterogeneity in terms of clinical department, as well as overall data saturation. Each interview will last approximately 60 minutes and interview questions will focus specifically on participants' experiences of interprofessional communication, burnout, and the AHE workshop.
For the qualitative stakeholder participatory validation (after T5), we will sample 5–10 Selected Informants from the AHE Informants to participate in a focus group interview. The focus group will last approximately 60 minutes and will aim to establish participatory validation of the theoretical findings generated from individual interviews as well as any proposed modifications to the AHE rooted in these findings or the quantitative outcome evaluation.
For the clinical co-facilitator analysis (between T1-T5), each clinical co-facilitator will participate in brief semi-structured interviews after every AHE workshop they facilitate. These individual interviews will last approximately 10–30 minutes and are designed to meet multiple project-related goals. First, the interviews will function as a practical opportunity to learn of implementation issues requiring troubleshooting prior to subsequent AHE workshop sessions, across all clinical and improv co-facilitators. Second, the interviews are guided by the underlying goal of understanding the longitudinal experiences of clinical co-facilitators involved in delivering improv-based communication workshops.
Analysis plan- quantitative
Power analysis for the quantitative methods is based on our ability to conduct Structural Equation Modeling (SEM) among all participants and to detect differences between Wave 1 and Wave 2. For SEM we will need a final sample of n = 200, therefore, having a baseline sample of n = 234 AHE project participants (85% of all AHE participants in each Wave) in each Wave will ensure a sufficient final sample size of n = 210 in each wave (10% attrition over time). Having n = 234 in each Wave will also give us enough power to compare the two groups after controlling for individual and program level factors using techniques such as Hierarchical Linear Modeling.
Quantitative outcome analyses will examine inter-individual changes over time among AHE participants as well as departmental-level differences between intervention and control groups using independent and dependent t-tests. We will also examine sequential processes in outcomes using SEM. Our quantitative process evaluation will examine moderators for AHE impacts using PROCESS macros. Effects of individual (e.g., age, staff position), program (e.g., participants' engagement, facilitator-rated group dynamics, time of day), and organizational (e.g., department, team structures) level factors on program outputs will be tested using nested techniques such as Hierarchical Linear Modeling. To reduce bias associated with attrition from the study over time, we will analyze missingness patterns and handle any missing data accordingly (e.g., pairwise deletion, multiple imputation).
Analysis plan- qualitative
Inductive analysis of all qualitative data will be guided by dimensional analysis, an interactionist methodology [37, 38]. Each interview transcript will be coded over multiple passes by members of the project team trained in qualitative data analysis. First pass coding will involve a Gestalt read of the data and analytic memo writing. In subsequent passes, the data will be analyzed by generating open, axial, and theoretical codes. Constant comparison technique [47], dimensionalization [37, 38], and theoretical sampling [32] will guide each phase of iterative analysis, facilitated by regular dialogic engagement sessions to ensure research team coherence in analytic procedures.
Inductive analysis of the data collected for the qualitative process evaluation of the AHE will help identify theoretical relationships between key dimensions of healthcare team communication, safety issues, and attitudes towards communication training including workshop groups environment. For the qualitative stakeholder participatory validation, analysis of the data will seek to uncover dimensions relevant for program implementation – utility, feasibility, acceptability, and scalability. Analysis of clinical-co-facilitator interviews, will aim to identify theoretical relationships among concepts central to improv communication training among interprofessional healthcare workers.
Comments
Post a Comment