In this mixed methods study, de-identified records of 137 DPT graduates of the University of Cincinnati were examined in this retrospective cohort study to determine if predictive quantitative and qualitative indicators of clinical professional behavior problems could be identified. Records of graduates of the classes of 2009-2014 revealed that 21 graduates had at least one major clinical intervention or two minor clinical interventions requiring an onsite visits and/or more than two in-depth phone communications and follow-up involving the Director of Clinical Education (DCE). These students were defined as cases (students of interest, SIs). In each of these 21 cases, all SIs had professional behavior issues and some additionally had clinical skills deficits. Considering Fig. 1, there is an illustration of the frequency of the professional behavior issues of these students in clinic. The remaining 116 student records served as controls (non-students of interest, NSIs).

With respect to the quantitative aspect of the study, the background demographic status (age, gender, marital status, and responsibility for dependent children in the home), academic factors (undergraduate major, overall Grade Point Average (GPA), required pre-requisite science GPA, anatomy, physics, chemistry, and biology GPA’s, and Graduate Record Examination scores), pre-admission LOR’s and number of observation hours were studied. Additionally, rankings of professional behavior characteristics from LOR’s and the professional identification of the recommender (Physical Therapist (PT) or Non-PT) were collected. Once collected, investigation of the data was done by descriptive statistics to ascertain if demographic and academic differences existed. Wilcoxon Rank Sum Test was used to know if there were statistical differences with LORs between SIs and NSIs. All analyses were performed using R version 3.2.3 (2015-12-10) (R Foundation for Statistical Computing Platform). Statistical significance was determined by P-values < 0.05.

With respect to the qualitative aspect of the study, Strauss and Corbin’s grounded theory [9] was used to investigate 30 LORs. These LORs originated from randomly selected 5 SIs and 5 NSIs, all of which were de-identified. Themes emerged using open and then axial coding. The researchers selectively coded the data to create a qualitative data table, which consisted of SIs and NSIs characteristics. After constant comparison, the coders reduced the data table to determine significant and non-significant characteristics. Data were then reduced again to allow for comparison to determine if the LORs could predict SIs. Individual characteristics were defined as significant for the group of SIs if ≥ 4 or ≤ 1 students possessed the characteristic. This same process was used to identify significant characteristics for NSIs. However, if the characteristic was deemed significant for both the NSIs and SIs then the characteristic was deemed non-predictive. Only the characteristics that were deemed significant and were unique to the SIs were considered to be predictive. Prior to initiating the qualitative aspect of the study, the researchers sought to insure dependability which translates to reliability for quantitative studies. One trial LOR was reviewed and coded by both researchers simultaneously for coding training and confirming expectations and methodology. Both researchers independently open and axial coded all LORs. Analysis of 10% of the qualitative data set resulted in 87% coding agreement between researchers. With negotiation, the researchers came to 100% agreement.

Quantitatively, demographic differences were observed between SIs vs. NSIs in age at the start of the program (µ= 25.76 SIs, µ= 23.67 NSIs), gender (SIs= 87% female, 13% male; NSIs = 76% female, 24% male), the number of children at the start of the program (SIs 4.76%, NSIs 5.17%) and children during the program (SIs 9.5%,NSIs 5.2%). Academic differences in biology GPA (µ= 3.46 SIs, µ= 3.39 NSIs), physics GPA (µ= 3.48 SIs, µ= 3.39 NSIs), and verbal GRE percentile scores (µ= 59.6 SIs, µ= 53.7 NSIs) were noted. Other differences between the groups were negligible. On average, SIs were two years older, had more observation hours, have higher biology GPA (+0.07), higher physics GPA (+0.09), higher verbal GRE scores (+5.91%) and were less likely to take the GRE more than once in comparison to NSIs. The Wilcoxon rank sum test (Mann-Whitney u-test) was utilized in analyzing the differences in numeric ratings on LORs between the SIs and NSIs. Analysis of each of 11 professional behavior characteristics rated on LOR’s by PT’s showed three characteristics to be significantly different between the SIs and the NSIs; critical thinking (P= 0.0029), organizational skills (P = 0.0363) and judgement (P = 0.0001). None of the other professional behavior characteristics rated by PT’s in LORs were significant. PT ratings on LORs of communication (P= 0.0731) and professionalism (P= 0.0789) indicate marginal findings and may deserve further attention. Additionally, none of the professional behavior characteristics rated by Non-PTs in LOR’s were found to be significant.

Qualitatively, in order to establish the dependability of the study, the researchers individually, hypothesized which of the 10 students were SIs. Upon comparison of the researcher’s independent hypotheses, each had selected the same five students as SIs. However, only 2/5 hypothesized SIs were correct (40% accuracy). Using the previously stated criteria for significance, the following characteristics were determined to be significant for SIs: communicates effectively (4/5) and cultural competency (4/5), as seen in Table 1. NSIs characteristics included: conflicting personality descriptor (4/5), commitment to learning (5/5), balance (4/5), teamwork skills (0/5), potential future success (5/5), compatible learning skills (0/5), effective leadership skills (4/5), and emotional intelligence (4/5) as seen in Table 2. To clarify, an example of a conflicting descriptor is as following: One applicant being described by a recommender as “introverted” in 1 LOR; while, another LOR describes the applicant as “extroverted.” Quantitatively, we found significant differences in PT ratings on pre-admission LORs between SIs and NSIs for the characteristics of critical thinking, organizational skills, and judgement. However, we found no significant differences in pre-admission ratings by PTs in other professional behavior characteristics on LORs between SIs and NSIs. No significant differences were found in LOR ratings by Non-PTs in all professional behavior characteristics.

Therefore, our quantitative findings suggest our PT program may benefit from using PT ratings in pre-admission letters in the critical thinking, organizational skills, and judgement characteristics as predictors of future professional behaviors in the educational clinical setting. Use of LORs provided by Non-PTs did not appear to offer any predictive value in estimating a potential student’s professional behaviors in clinical education. Although critical thinking, judgement, and organizational skills were found to be significant predictors from LORs, our DCE spends the majority of time dealing with issues related to communication, clinical skills, confidence, commitment to learning and interpersonal skills which were not found to be significant predictors from either PTs or Non-PTs. As Table 3 illustrated, despite having followed a trustworthy qualitative method to analyze the data, the data did not yield any new insights to indicate that LORs could be used to predict clinical performance as it relates to professional behaviors. Thus, LORs appear to be of limited value in predicting professional behavior performance in clinic.

Limitations for this study included the sample size which was based upon one DPT program with a sample size of 137 students. Another limitation to our study was the skewed data from LORs and a ceiling effect on the rating scale for professional behaviors, as evidenced by overall positive scores on LOR’s and very few low ratings on a 5 point scale. Future research should include multiple DPT programs for a larger sample size. Additionally, an investigation of pre-admission characteristics for the development of a prediction rule for specific professional behaviors that relate to employability skills must occur as our profession goes forward [10].