This is part of a blog series, Bringing Clarity to the SSR. See Part 1.
Self-Study Report, Appendix 13 F
The new Self-Study Report (SSR) Appendix 13 F (also from the 5th Edition standards) lists several guidelines spotlighting necessary data analysis:
“Data analysis related to PANCE outcomes is to include, but not limited to, correlation of PANCE outcomes and the following:
Admissions criteria as a predictor of success
Course and instructor evaluations by students
Program instructional objectives, learning outcomes, and breadth and depth of the curriculum
Student summative evaluation results
Student progress criteria and attrition data
Feedback from students who are unsuccessful passing PANCE
Preceptor and graduate feedback”
Understanding these standards is vital to carrying them out. In order to gain increased clarity of the requirements, we’re breaking down each of these items and looking at how to comply with them. In Part 1, we reviewed Admissions Criteria as a Predictor of Success and Course Outcomes. Here in Part 2, we will review Student Summative Evaluation Results.
The summative exam requirements are now outlined in Standard B4.03:
“The program must conduct and document a summative evaluation of each student within the final four months of the program to verify that each student meets the program competencies required to enter clinical practice, including: a) clinical and technical skills, b) clinical reasoning and problem-solving abilities, c) interpersonal skills, d) medical knowledge, and e) professional behaviors.”
Student Summative Evaluation Results
Many aspects of the summative requirements do not directly relate to PANCE outcomes, so how can a PA Program determine the correlation between the two? The key is in finding what data can be correlated between the PA program’s summative exam and the PANCE exam.
For the purposes of this specific post, let’s focus on the medical knowledge component of the program’s summative exam. Here is an outlined approach that has proven successful on multiple occasions:
(Note that the suggestions listed below are based on a statistician’s insights into the data.)
Determine the correlational strength between the students’ raw scores on the knowledge component of the summative exam and the students’ PANCE scores
If possible, examine the relationship between the task’s components of the summative exam and the students’ performances on the individual task areas on the PANCE
Stratify the results of the PANCE scores, then compare with the summative scores. For example, look for a relationship between students who either failed the PANCE or achieved very low scores and their performance on the summative exam
Retrospectively gather several years of data (if available), compiling the students’ PANCE scores and the raw scores on the summative exam. Applying regression can actually generate predicted scores based upon your database. Note that this is strengthened when you compile a larger database
Based upon the results of the regression, see if it is possible to prospectively determine students that are or can become at risk. Determine whether or not coaching can provided through an academic improvement plan to work with such students during the last 3-4 months of the program
Personally, I’ve used this process to develop summative exams that are statistically valid and have a strong correlation with students’ PANCE outcomes. See examples of previous research I’ve performed in regard to the correlation between summative exams and PANCE here.
The summative exam should also have a direct connection to the curriculum. It is essential is map questions to specific content areas; this exam should represent a valid culmination of the PA Program’s curriculum. Be sure to identify the distribution of questions as it relates to organ systems. Further, tag the questions according to task area. Over time, you will be able to identify trends representing lower performance in these areas.
Consider this an organic process intending to improve your PA Program. Dissecting the summative exam and its relationship to the curriculum provides an opportunity to implement changes to the curriculum based on the results.
I personally have applied the same methodology to PACKRAT and the EORE to develop a risk profile. Students who are performing in the lower quartile can be identified as potentially at-risk for not passing the PANCE. Implementing this early alert system can assist the students and provide the support they need to ensure success. Certain students may need additional instruction regarding test-taking methods and/or strategic study plans. Over time, this coaching and support can make a measurable difference in students’ performances.