Ethical statement

This study was approved by the Institutional Review Board of Sacred Heart University (IRB approval no., 180130A). Informed consent was obtained from the participants.

Study design

The first phase of the study consisted of focus groups with DPT subject matter experts (SMEs). The second phase was a cross-sectional survey of SMEs from 1 university.

Materials and subjects

Phase I: focus groups

Participants: SMEs were identified by 1 of the authors and were invited to participate. Three DPT faculty members (all classroom instructors, 1 of whom also oversaw clinical rotations) participated in one group. Four recent graduates of the DPT program participated in a separate group by conference call.

Materials and procedures: A list of 34 possible POJA traits were identified from the department’s list of essential functions for physical therapists [15], as well as from previous healthcare-related studies [8,14]. Descriptions of items were obtained from the International Personality Item Pool website and were edited for clarity. Research in leader performance has increasingly shown the importance of studying both negative and positive factors that affect performance, as they can coexist in people. Negative factors can derail what would otherwise be positive performance [16,17,18]. Therefore, the stimulus list purposely included factors that could be detrimental to performance or lead to performance difficulties.

SMEs were given the stimulus list of POJA traits and definitions and they were asked to critique them for clarity, to condense and combine traits and definitions, and to add any new traits if appropriate. This resulted in a list of 22 traits and definitions, 15 of which could be associated with successful performance, and 7 which could be associated with performance difficulties.

Phase II: personality-oriented job analysis questionnaire

Participants: Eleven SMEs were recruited to participate in the study. Nine SMEs participated, including 6 DPT faculty members and 3 recent graduates who had passed the NPTE and were employed as physical therapists.

Materials and procedure: A questionnaire with 22 POJA traits and definitions was developed and is presented in the Appendix 1. The response scales for the personality factors were adapted from the knowledge, skills, abilities, and other characteristics sections of Levine’s combination job analysis method (C-JAM) as described by Brannick et al. [19]. The wording of the scales was modified to be appropriate for student admissions rather than job applicants. The SMEs rated traits on 4 scales: (1) Is this factor necessary for new DPT students (yes/no)?; (2) Is this factor practical to expect among DPT applicants (yes/no)?; (3) To what extent is trouble likely if ignored in admissions? (rated on a 5-point scale: 1= very little or none to 5= to an extremely great extent); and (4) To what extent do different levels of the factor distinguish the superior from average student (rated on 5-point scale, 1= very little or none to 5= to an extremely great extent).

For the 7 factors that were thought to be possibly associated with performance difficulties, the first 2 scales were reworded to “Is the factor detrimental for new first-year DPT students?” and “Is the factor common among DPT applicants?”

Previous POJA studies identified several response errors, including self-serving bias, implicit trait policies, social projection, and false consensus. A randomized trial of Aguinis et al. [20] showed that frameof-reference (FOR) training reduced bias, decreased the correlation between SMEs’ own personality traits and the POJA trait ratings, and decreased overall POJA ratings, compared to a group given standard instructions. A subsequent meta-analysis showed that FOR training improved the accuracy of ratings [21]. Therefore, specific instructions were provided to reduce response errors and to improve accuracy. SMEs were presented with the following instructions at the beginning of the questionnaire: “When considering the factors it is important to think about DPT students in general, not just yourself as you were as a student. For example, while you may have always kept your desk area very neat and as organized as possible, many students may not do this and they may still be successful.” The questionnaire was created in SurveyMonkey (http://www.surveymonkey.com) and a link was emailed to 11 SMEs. The raw data are available in Supplement 1.

Technical information

The C-JAM process involves computing the number of SMEs who answered “yes” to (1) whether the trait is necessary for new students and (2) whether the trait is practical to expect among applicants [10]. Additionally, mean scores were computed for (1) trouble being likely if a trait is ignored in admissions, and (2) whether the trait would distinguish superior from average students. If a majority of SMEs answer that the trait is necessary and practical to expect and the mean score for trouble being likely if a trait is ignored is greater than 1.5, then the trait can be incorporated into selection procedures. For the traits to be used in selection, if the mean score for its ability to distinguish superior from average students is greater than 1.5, then selection procedures including it should result in a ranked ordering of applicants. If the mean score for distinguishing superior from average students is equal to or lower than 1.5, a pass/fail selection procedure should be developed.

C-JAM also has decision rules for which traits can be used in training. If less than a majority of SMEs agree that the trait is necessary for new students and the mean score for its ability to distinguish superior from average students is greater than 1.5, it is recommended for training.

The present study included several methodological improvements. First, 5-point rating scales were used, rather than 3-point scales, to allow for more nuanced variation in ratings [22]. Second, SMEs answered 4 questions about each factor, and the structured decision-making process of C-JAM was used to decide which personality factors to include in selection and which to recommend for training. Third, FOR instructions for SMEs were included, which have been shown to decrease self-serving bias, implicit trait policies, social projection, and false consensus, and to improve accuracy [20,21].

The focus group participants, including 3 DPT faculty members and 4 recent graduates of the DPT program, identified 22 non-cognitive factors. Fifteen factors were thought to be possibly associated with successful performance, and 7 factors were thought to be possibly associated with performance difficulties.

Table 1 presents the questionnaire results for factors to be considered for selection. Following the previously outlined decision rules for C-JAM, the SME ratings resulted in 12 factors to be considered for selection (11 desirable factors and 1 undesirable factor). Table 2 presents the results for factors to be considered for training. The SME ratings also yielded 10 factors recommended for training (4 desirable factors and 6 undesirable factors).

The C-JAM process resulted in 12 factors to be included in selection procedure validation, in addition to the cognitive procedures currently in place. Several of these factors aligned with the findings of Roll et al. [13]. Specifically, measures of adaptability, interpersonal skills, and tolerance in the POJA were similar to the measures of adaptability, intuitiveness, and engagement presented by Roll et al. [13], respectively. However, the results of the C-JAM analysis identified several additional factors that could be included in the selection process, 8 of which were expected to contribute positively to performance, and 1 of which was expected to contribute to performance difficulties. Additionally, 4 of the top 6 factors identified by McLarnon et al. [14] (conscientiousness, calm/relaxed, tolerance, and responsibility) were also identified in the present study. Of note, however, is that McLarnon et al. [14] utilized a rating scale that ranged from disastrous to essential, and selected the factors that scored the highest. This effectively eliminated any undesirable factors.

The present study separated ratings for desirable and undesirable factors to result in a spectrum of factors. Interestingly, the majority of SMEs indicated that those 6 undesirable factors are not common among applicants. C-JAM rules suggest that they can be recommended for further training. However, since they are uncommon, further discussion with SMEs is warranted regarding the utility of training for such uncommon factors. The present study included several methodological strengths. First C-JAM is a formal, standardized job analysis process with clear decision rules, and the results can be used for both selection and training. Second, performing job analysis and utilizing SMEs in that process is recommended in the ‘uniform guidelines on employee selection procedures’ to ensure validity and improve legal defensibility [23]. Third, while most previous studies have focused on positive factors, the present study included several factors that may lead to performance difficulties. Fourth, the instructions to SMEs included FOR training, which has been shown to reduce bias and to improve accuracy in ratings [20,21]. Fifth, a subsequent validation study will identify scientific and legally defensible non-cognitive measures that could help improve the selection and retention of first-year students, performance in clinical placements, and first-time pass rates on the NPTE.

A limitation is that the DPT education program studied utilizes a problem-based learning curriculum. This curricular model is used by 1.7% of physical therapy education programs nationally [24]. Therefore the results may not generalize to other DPT education curricula, and would need to be verified in other programs.

In conclusion, the present study is the first step in the process of identifying and validating non-cognitive factors that may contribute to success and difficulties in a DPT education program. Previous studies of non-cognitive factors such as demographics or essays have produced conflicting results. Using SMEs, the present analysis identified a broad array of factors that go beyond those examined in previous studies, and which can be validated. The next step will be to conduct a concurrent validation study by administering personality measures of those factors to current DPT students, and correlating scores on the measures with course grades, clinical rotation performance, and NPTE scores.

Acknowledgments

Supplementary material

References

• 1. Utzman RR, Riddle DL, Jewell DV. Use of demographic and quantitative admissions data to predict performance on the national physical therapy examination. Phys Ther 2007;87:1181-1193. https://doi.org/10.2522/ptj.20060222 ArticlePubMedPDF
• 2. Utzman RR, Riddle DL, Jewell DV. Use of demographic and quantitative admissions data to predict academic difficulty among professional physical therapist students. Phys Ther 2007;87:1164-1180. https://doi.org/10.2522/ptj.20060221 ArticlePubMedPDF
• 3. Balogun JA, Karacoloff LA, Farina NT. Predictors of academic achievement in physical therapy. Phys Ther 1986;66:976-980. https://doi.org/10.1093/ptj/66.6.976 ArticlePubMedPDF
• 4. Balogun JA. Predictors of academic and clinical performance in a baccalaureate physical therapy program. Phys Ther 1988;68:238-242. https://doi.org/10.1093/ptj/68.2.238 ArticlePubMedPDF
• 5. Dockter M. An analysis of physical therapy preadmission factors on academic success and success on the national licensing examination. J Phys Ther Educ 2001;15:60-64. https://doi.org/10.1097/00001416-200101000-00010 Article
• 6. Gabard DL, Porzio R, Oxford T, Braun R. Admission interviews: questions of utility and cost in masters of physical therapy programs in the United States. Physiother Res Int 1997;2:135-149. https://doi.org/10.1002/pri.96 ArticlePubMed
• 7. Hayes SH, Fiebert IM, Carroll SR, Magill RN. Predictors of academic success in a physical therapy program: is there a difference between traditional and nontraditional students? J Phys Ther Educ 1997;11:10-16. https://doi org/10;org/10:org/10-org/10. https://doi.org/10.1097/00001416-199701000-00003 Article
• 8. Hollman JH, Rindflesch AB, Youdas JW, Krause DA, Hellyer NJ, Kinlaw D. Retrospective analysis of the behavioral interview and other preadmission variables to predict licensure examination outcomes in physical therapy. J Allied Health 2008;37:97-104. PubMed
• 9. Kappe R, van der Flier H. Predicting academic success in higher education: what’s more important than being smart? Eur J Psychol Educ 2012;27:605-619. https://doi.org/10.1007/s10212-011-0099-9 Article
• 10. Galleher C, Rundquist PJ, Barker DB, Chang WP. Determining cognitive and non-cognitive predictors of success on the National Physical Therapy Examination. Internet J Allied Health Sci Pract 2012;10:7.Article
• 11. Guffey JS, Farris JW, Aldridge R, Thomas T. An evaluation of the usefulness of noncognitive variables as predictors of scores on the national physical therapy licensing examination. J Allied Health 2002;31:78-86. PubMed
• 12. Beauvais AM, Stewart JG, DeNisco S, Beauvais JE. Factors related to academic success among nursing students: a descriptive correlational research study. Nurse Educ Today 2014;34:918-923. https://doi.org/10.1016/j.nedt.2013.12.005 ArticlePubMed
• 13. Roll M, Canham L, Salamh P, Covington K, Simon C, Cook C. A novel tool for evaluating non-cognitive traits of doctor of physical therapy learners in the United States. J Educ Eval Health Prof 2018;15:19. https://doi.org/10.3352/jeehp.2018.15.19 ArticlePubMedPMCPDF
• 14. McLarnon MJ, Rothstein MG, Goffin RD, Rieder MJ, Poole A, Krajewski HT, Powell DM, Jelley RB, Mestdagh T. How important is personality in the selection of medical school students? Personal Individ Differ 2017;104:442-447. https://doi.org/10.1016/j.paid.2016.09.006 Article
• 15. Essential functions of the physical therapy profession [Internet]. Fairfield (CT): Sacred Heart University’s Doctoral Program of Physical Therapy; [cited 2018 Dec 6]. Available from: https://www.sacredheart.edu/academics/collegeofhealthprofessions/academicprograms/physicaltherapy/doctorofphysicaltherapy/essentialfunctionsofthephysicaltherapyprofession/
• 16. Hogan R, Curphy GJ, Hogan J. What we know about leadership: effectiveness and personality. Am Psychol 1994;49:493-504. https://doi.org/10.1037/0003-066x.49.6.493 ArticlePubMedPDF
• 17. Hogan R, Hogan J. Assessing leadership: a view from the dark side. Int J Sel Assess 2001;9:40-51. https://doi.org/10.1111/1468-2389.00162 Article
• 18. Guenole N. Maladaptive personality at work: exploring the darkness. Ind Organ Psychol 2014;7:85-97. https://doi.org/10.1111/iops.12114 Article
• 19. Brannick MT, Levine EL, Morgeson FP. Job and work analysis: methods, research, and applications for human resource management. Los Angeles (CA): Sage Publications; 2007.
• 20. Aguinis H, Mazurkiewicz MD, Heggestad ED. Using web-based frameof-reference training to decrease biases in personality-based job analysis: an experimental field study. Pers Psychol 2009;62:405-438. https://doi.org/10.1111/j.1744-6570.2009.01144.x Article
• 21. Roch SG, Woehr DJ, Mishra V, Kieszczynska U. Rater training revisited: an updated meta-analytic review of frame-of-reference training. J Occup Organ Psychol 2012;85:370-395. https://doi.org/10.1111/j.2044-8325.2011.02045.x Article
• 22. Raymark PH, Schmit MJ, Guion RM. Identifying potentially useful personality constructs for employee selection. Pers Psychol 1997;50:723-736. https://doi.org/10.1111/j.1744-6570.1997.tb00712.x Article
• 23. Uniform guidelines on employee selection procedures [Internet]. Washington (DC): US Code of Federal Regulations; 1978 [cited 2018 Nov 21]. Available from: https://www.gpo.gov/fdsys/pkg/CFR-2011-title29-vol4/xml/CFR-2011-title29-vol4-part1607.xml
• 24. Commission on Accreditation in Physical Therapy Education. Aggregate program data: 2016-2017 physical therapist education programs fact sheets [Internet]. Alexandria (VA): Commission on Accreditation in Physical Therapy Education; 2017 [cited 2018 Dec 6]. Available from: http://www.capteonline.org/uploadedFiles/CAPTEorg/About_CAPTE/Resources/Aggregate_Program_Data/Archived_Aggregate_Program_Data/2016PTFactSheetFinal.pdf

Appendix

Figure & Data

References

Citations

Citations to this article as recorded by  

• A Systematic Review of Variables Used in Physical Therapist Education Program Admissions Part 2: Noncognitive Variables
  Andrea N. Bowens
  Journal of Physical Therapy Education.2024; 38(3): 192.     CrossRef
• Personal characteristic differences among Doctor of Physical Therapy students with unique sociodemographic factors
  Kelly Reynolds, Maggie Horn, Karen Huhn, Steven Z. George
  BMC Medical Education.2024;[Epub]     CrossRef
• The relationship of non-cognitive factors to academic and clinical performance in graduate rehabilitation science students in the United States: a systematic review
  Kelly Reynolds, Caroline Bazemore, Cannon Hanebuth, Steph Hendren, Maggie Horn
  Journal of Educational Evaluation for Health Professions.2021; 18: 31.     CrossRef