Health education accreditation standards continue to evolve with respect to interprofessional education (IPE) [1] through the inclusion of required experiences in medical [2] and other professional school curricula. Due to a number of identified barriers [3,4], the expectation for interprofessional practice (IPP) to manage patients with complex health conditions and social needs is limited by educational models that do not foster the team-based collaborative skills necessary to meet the inherent challenges. Even well-designed academic efforts to embed IPE in health professions programs often lack essential team-based IPP experiences. Despite these limitations, the merit of and need to advance IPE and IPP are reflected by ongoing initiatives to promote an array of methods [5] and to emphasize the need for faculty development and research to develop transferable training models that include patient-related outcomes [6]. As IPP training models evolve, it is essential to select assessment methods or tools suitable for discerning the educational impact of our interventions, although guidance is currently limited [7].

This study aimed to describe our experiences with the design and delivery of interprofessional learning experiences to didactic (n=865) and clinical (n=76) student cohorts at our comprehensive academic health center. We compared 2 validated survey instruments using a randomized design before and after the interprofessional learning experiences. Our specific objectives included assessing students’ perceptions of their interprofessional experiences to determine the impact of both modalities and sharing the details of our educational interventions for replication and refinement. The hypothesis was that brief longitudinal clinical experiences in IPP (the clinical cohort) would produce larger changes in student attitudes than didactic small-group interactive IPE experiences (the didactic cohort).

This research received an exemption approval from the campus Institutional Review Board for the Protection of Human Subjects and met the criteria for a waiver of informed consent (IRB #8434).

A randomized parallel cohort study of 2 validated survey instruments was deployed prior to and after didactic interactive small-group (All Professions Day [APD] 1 and 2) and clinical cohort (Empowering Patients through Interprofessional Collaboration, EPIC) learning experiences to measure changes in students’ perceptions according to their learning modality and assigned assessment instrument. The descriptions in the text follow the STROBE statement (https://strobe-statement.org/).

The participants were students enrolled in the Colleges of Allied Health, Dentistry, Medicine, Nursing, Pharmacy, Public Health, and the School of Social Work at the University of Oklahoma in the southern United States during the 2017–18 academic year. For the APD cohort, a total of 865 students from 25 degree programs at 7 colleges participated on 1 of 108 teams of 8 to 10 students. Each APD team was facilitated by 1 of 51 trained faculty members. All interprofessional student programming was intentionally based on the Interprofessional Education Collaborative Competencies (IPEC) [8,9]. The faculty engaged the APD cohort in a mixture of lecture, active learning, and team-building activities to introduce the 4 IPEC competencies and to achieve the learning objectives between the fall and spring semesters (Table 1). Parallel to the APD cohort, a separate cohort of 76 advanced career students representing 14 professional programs was assigned to complete an IPP clinical experience (EPIC) at a local charitable outpatient clinic after receiving 2 separate preparatory sessions.

The orientation (the first preparatory session) was 4 hours in length and included a team-building activity, an introduction to complex patient care in the context of social determinants of health, and a team discussion to develop a clinical strategy to coordinate/deliver care. The second preparatory session (4 hours in length) provided an orientation to the outpatient clinic, a structured clinical simulation using standardized patients to test and refine the teams’ strategy to coordinate care delivery when attending their clinical sessions, and an orientation to the web-based electronic medical record. After the 2 preparatory sessions, the student teams were scheduled to provide care at a charitable clinic each week for 4 hours on Thursday evening. Each of the 8 teams hosted a clinical session once monthly, with 2 teams per night. They repeated the sequence until all teams had completed 4 clinical sessions, approximately once monthly over 4 months during the fall and spring semesters. Two initial patients with complex medical and social needs were scheduled per team on the first clinic night, with 2 to 3 patients scheduled as a mixture of new and/or follow-up patients on subsequent clinic nights. A faculty facilitator drawn from any discipline and an attending medical provider were scheduled per team, with additional faculty members scheduled according to regulatory supervision requirements for the participating disciplines. The personnel of the charitable clinic were compensated for managing duties including scheduling and rooming patients, and a pharmacist was also compensated for fulfilling prescription orders during the evening clinics. Access to the web-based electronic medical record was granted to allow students and faculty to document their clinical encounters. Patient care was coordinated and delivered by the interprofessional student team members, according to their IPP team strategy, which was modified as required based on the needs of the patient. The clinical encounter time per patient was structured with approximately 45 minutes for patient assessment, 15 minutes for presentation/discussion with the attending medical provider/faculty member(s), and 15 minutes for the final interaction with the patient prior to scheduling follow-up actions. The student teams worked with patients to set goals at each encounter and modified the plan longitudinally as needed on subsequent visits. Final documentation in the electronic medical record was co-signed by the attending medical provider. A debriefing session of approximately 15 minutes was guided by the designated team non-medical provider faculty facilitator and concluded the first 3 of the 4 clinic sessions to prompt reflective learning and planned adjustments to care delivery in subsequent clinical sessions. The fourth and final clinic session was concluded by a global reflection on the interprofessional learning experience with an open discussion of 8 questions and response summaries by the faculty facilitators.

Two validated survey instruments were used, with permission of the authors, to assess changes in student perceptions. Students were requested to complete the instrument prior to the first planned event and following the last planned event, according to their assigned cohort. The Interprofessional Attitude Scale (IPAS) is a 27-item instrument that assesses the 5 subdomains of teamwork, roles, and responsibilities (9 questions), patient centeredness (5 questions), interprofessional biases (3 questions), diversity and ethics (4 questions), and community centeredness (6 questions) [10]. The Likert scale was scored as follows: 1, strongly disagree; 2, disagree; 3, somewhat disagree; 4, neither agree nor disagree; 5, somewhat agree; 6, agree; and 7, strongly agree. According to the survey instructions, 1 question was reverse scored. The total score and sub-domain averages were calculated.

The Collaborative Healthcare Interdisciplinary Relationship Planning (CHIRP) instrument, included in Supplement 1, is a 14-item survey that assesses a single domain, attitudes toward interdisciplinary teamwork [11]. Both surveys were created separately and administered via the online survey tool Qualtrics (https://www.qualtrics.com/, Provo, UT, USA). The IPAS survey had total of 49 questions that included 4 academic questions, the 27 validated IPAS questions, and 18 additional experimental IPAS questions that are undergoing validation (Supplement 2). The CHIRP survey had 18 questions (14 validated survey items and 4 academic questions). A 5-point scale of agreement was used and scored as follows: 1, I do not agree at all; 2, I somewhat agree; 3, I fairly much agree; 4, I very much agree; 5, I completely agree. The average was calculated as the overall score.

The baseline IPAS and CHIRP surveys were sent to 429 and 436 students, respectively. For each survey, 18% of respondents were from allied health professions, 12% from dentistry, 28% from medicine, 16% from nursing, 11% from pharmacy, 4% from public health, and 10% from social work. The overall baseline response rate was 58.7% for the IPAS and 62.4% for the CHIRP survey. After accounting for dropout, the post-2nd APD IPAS survey was sent to 348 students, of whom 54.0% responded. Similarly, the post-2nd APD CHIRP survey was sent to 353 students and 61.8% responded. Post-EPIC IPAS surveys were sent to 36 students, with a response rate of 97.2%. Lastly, post-EPIC CHIRP surveys were sent to 37 students and the response rate was 105.4%; we believe this error was due to some respondents misinterpreting the EPIC participation question as indicating their interest in future participation instead of actual participation during the study period.

Students were randomly assigned to complete 1 of the 2 survey instruments in a manner that ensured an even distribution of disciplines across both survey instruments. Survey responses were anonymous in accordance with IRB approval. Due to the anonymity of the responses, baseline and post-intervention data could not be linked to individual participants; therefore, traditional longitudinal analysis techniques could not be employed. The results were pooled and presented as overall baseline, post-2nd APD, and post-EPIC clinic results. Descriptive statistics were utilized to summarize the response rates and survey results. The response rates are reported as frequency (percent), while the survey results are presented as mean (standard deviation), minimum, median, and maximum values (Datasets 1, 2).

The impact of non-responders was of primary concern, and in accordance with published best practice guidelines [12] we sought to reach at least a 50% response rate. Additionally, responses were anonymous to address the potential impact of social desirability bias.

An a priori sample size calculation was not performed due to the large (n=865), campus-wide IPE effort to which our study was attached.

The Pittman nonparametric test using permutations of linear ranks was used to determine any differences in the distribution of scores. Due to the overall large sample size, exact permutation tests were not computationally possible; therefore, Monte Carlo estimates of the test’s P-value were used. Each test was set to run for 50,000,000 permuted samples, which resulted in 99% confidence intervals (CIs) to the ten-thousandths around the P-value (e.g., estimated P-value=0.0673; 99% CI, 0.0672–0.0674).

Group comparisons were made for the CHIRP overall score, IPAS total score, and each IPAS subdomain score. For each of these scores, 3 comparisons were made simultaneously: (1) overall baseline versus post-2nd APD; (2) overall baseline versus post-EPIC clinic; and (3) post-2nd APD versus post-EPIC clinic. To account for the simultaneous comparisons, alpha was adjusted to 0.0167 using the Bonferroni method. All tests were 1-tailed with the hypotheses that overall baseline score < post-2nd APD score < post-EPIC clinic score. All analyses were conducted using SAS software ver. 9.4 (SAS Institute Inc., Cary, NC, USA).

Our findings suggest that a series of 4 interprofessional ambulatory clinic session experiences following 2 preparatory sessions could significantly improve students’ perceptions as assessed by the IPAS instrument. The results were strongly influenced by improved responses in the IPAS teamwork, roles, and responsibilities subdomain. Purely didactic active learning activities can provide foundational principles for IPP, but application in the context of actual patient care is expected to provide more measurable changes in student perceptions, which the IPAS results did reveal in the aforementioned subdomain. Since the clinic design provides only 4 sessions of 3–4 hours per team, this subdomain may be most sensitive to changes in response to time-limited IPP. To our knowledge, this is the first report using the validated IPAS instrument to assess all health disciplines following delivery of actual patient care as an interprofessional team among students enrolled in medicine, nursing, pharmacy, physician assistant, public health, and allied health programs [10].

Our results are consistent with a report that used IPAS prior to and following a 1-month psychiatry clerkship involving students from medicine, physician assistant, pharmacy, and social work programs [13]. Following the psychiatry clerkship, a statistically significant increase (P=0.036) was seen only in the subdomain of teamwork, roles, and responsibilities for the 59 students completing the survey. This IPAS subdomain contains 9 of the 27 survey items and includes several that frame shared learning experiences in terms of individual and team benefits. In contrast to the IPAS results, the CHIRP instrument did not show a significant overall difference in either interprofessional intervention group. The CHIRP survey was selected as a second instrument based on its previous use in multiple methods of learning including lecture instruction, interactive videos with audience response, group role-play exercises, and group high-fidelity simulations among medical and nursing students [14]. However, the CHIRP instrument was originally validated only among medical and nursing students [11], and to our knowledge it has not been deployed at a comprehensive academic health center involving all student disciplines, nor involving students providing direct patient care, so it is unclear whether these variables impacted the ability of the instrument to discern any differences in our diverse study population.

The limitations of our results should be acknowledged, including the lack of a control group, anonymous survey completion limiting paired data analysis, differing general experience levels between the didactic and clinical cohorts, discrepancies in the length of the survey (CHIRP versus IPAS), and involvement of students from a single institution. The finding that the post-EPIC response rate slightly exceeded the expected response rate for the CHIRP survey also suggests a small number of students did not categorize their participation in the interprofessional clinical experience accurately, having only participated in the didactic/interactive APD experiences. Such occurrences would most likely decrease the post-EPIC scores, but were infrequent enough that the effect on the final CHIRP scores would be minimal.

A strength of the results for the EPIC clinical experience is that measurements were made across 8 separate interprofessional team groups consisting of 8–10 students per team, which may have facilitated homogeneity of the responses to the teamwork, roles, and responsibilities subdomain of the IPAS after only 4 ambulatory care patient clinic sessions. With respect to the CHIRP survey being significantly shorter than the IPAS survey, the potential exists for longer surveys to require extended concentration. However, only the first 31 questions of the IPAS were inclusive of the demographic and validated items, which compares more favorably to the 18 analogous items of the CHIRP. We suggest that the primary risk with the additional 18 pilot questions for the IPAS is for students to abandon completion of the survey, which may have occurred as shown by a lower percent completion for the IPAS. However, we only analyzed completed IPAS surveys, so we believe that those who persisted would be able to register meaningful responses. Although the EPIC cohort did have a generally higher experience level than the didactic cohort, we believe that the degree of change in student responses pre- and post-cohort would be more dependent on the nature of the interprofessional learning experience than on pre-existing general or interprofessional experience.

Overall, we believe our efforts represent broad inclusion of all professions at a comprehensive academic health center, as well as social work, and survey completion rates that are generally representative. The results of our educational interventions support a modification in student attitudes/perceptions consistent with level 2a of Kirkpatrick’s hierarchy (KH), which has been used to evaluate the effectiveness of training programs [15,16]; ultimately, the impact of interprofessional learning experiences should progress to higher levels of KH, including modification of knowledge/skills, behavior change, and improvement of patient outcomes. Future considerations for our curriculum design and assessment include ensuring APD participation by all early-career learners in our collective college curricula and exploring additional interprofessional patient care assessment instruments/methods to move beyond the modification of student perceptions to capture observable elements of student IPP competency.

Since the post-EPIC clinic team subdomain survey score was significantly higher than the overall baseline and post-didactic cohort scores, our hypothesis that brief longitudinal ambulatory clinic experiences in IPP would produce larger changes in student perceptions than didactic small group interactive IPE experiences was accurate. The IPAS instrument may be useful for detecting modifications of student attitudes in the teamwork, roles, and responsibilities domain in response actual IPP experiences, and it has now been studied in team-based ambulatory care involving learners from a broad range of health disciplines.