Comprehensive Data Services

Massey & Martin, LLC proudly offers comprehensive data analysis services for all elements of Appendix 14 of the 5thEdition ARC-PA Self-Study Report. We offer these services in two tiers; see below for a description of each. Pricing is available upon request.

Tier 1 Services

This service converts the raw data into the required templates in graphical or tabular format so PA program committees can interpret and appropriately assign action plans. This service is a cost-effective means by which programs can ensure the assessment process can incorporate all data components within the decision-making process. Overall, it saves PA programs from needing to hire extra staff or faculty members to perform these services.

 

This service includes a monthly phone conference with a consultant from Massey & Martin, LLC to address questions regarding the data analytics program. This tier does not include additional consultant oversight of programmatic assessment processes. (For programs in need of data interpretation, assessment and research services, incorporation of a comprehensive assessment process, and assistance with the SSR, we recommend Tier 2.)

 

This service does not include additional time with the faculty or provide mentoring on how to use the data. The costs outlined in this document are for delivery of data only.

Tier 2 Services

This service includes all elements of Tier 1 services, as well as comprehensive assistance with program assessment, faculty mentoring, data interpretation, and development of program committee structure. In addition, this service includes assistance and oversight with the development of the SSR prior to an ARC-PA on-site visit. 

 

This service includes a weekly phone conference with the faculty with a consultant from Massey & Martin, LLC to address questions regarding the data analytics program. This tier is ideal for PA programs in need of data interpretation, assessment and research services, incorporation of a comprehensive assessment process, and assistance with the SSR.

 

This service includes additional time with the faculty and provides mentoring on how to use the data.

Overview

Below is a summary of the data requirements for the 5th Edition Standards SSR Appendix 14, including descriptions of required program data to be transmitted to Massey & Martin, LLC for analysis. The data will be appropriately analyzed by data managers and statisticians, with the final product ready to be included in the SSR. 

 

Unless otherwise indicated, the SSR requires three years of trend data for all requirements.

Appendix 14B

This Appendix requires the program to submit data and analysis to assess the sufficiency and effectiveness of administrative aspects of the program and institutional resources, including:

  • Institutional resources in support of the program

  • Financial resources

  • Human resources

  • Physical resources

  • Technology resources and support

  • Safety and security, and student services (defined in the glossary as: academic advising, tutoring, career services, financial aid, student health, computing and library resources, and access)

  • Support for faculty development, clinical site development, curriculum design and course selection, program assessment, and program diversity and inclusion

  • Program administrative aspects

  • Policies/procedures

  • Admissions processes and outcomes

 

Surveys administered

  1. Sufficiency and effectiveness of administrative aspects faculty survey

  2. Sufficiency and effectiveness of administrative aspects faculty thematic analysis

  3. Sufficiency and effectiveness of administrative aspects student survey

  4. Sufficiency and effectiveness of administrative aspects student thematic analysis

 

Description of data

Massey & Martin, LLC will administer surveys directly or the institution will administer surveys to the faculty staff and students. To provide the program with tabulated and graphical data required by ARC-PA the program will need to provide the following:

  1. Final spreadsheet with responses to faculty evaluation of institutional and program sufficiency and effectiveness

  2. Final spreadsheet with comments related to faculty evaluation of institutional and program sufficiency and effectiveness

  3. Final spreadsheet with responses to student evaluation of institutional and program sufficiency and effectiveness

  4. Final spreadsheet with student comments related to institutional and program sufficiency and effectiveness

Appendix 14C: Effectiveness of the Didactic Curriculum

Data required

  • Student evaluation of didactic courses and instructors

  • The number of final didactic course grades at C or below

  • Student attrition and remediation* in didactic courses

  • How many students have repeated didactic courses in the three most recent graduating classes, as well as the classes currently enrolled?

 

Surveys administered

  1. Student evaluation of didactic courses

  2. Student evaluation of didactic courses thematic analysis

  3. Student evaluation of faculty performance 

  4. Student evaluation of faculty performance thematic analysis

 

Description of data

Massey & Martin, LLC will require the following data delivered from the program

  1. Student evaluation of didactic courses spring semester

  2. Student evaluation of didactic courses summer semester 

  3. Student evaluation of the didactic courses fall semester

  4. Student evaluation of faculty evaluations spring semester

  5. Student evaluations of faculty evaluations summer semester

  6. Student evaluations of faculty evaluations fall semester

 

Data analyzed

  1. Regression analysis involving remediation and performance on PANCE

  2. Regression analysis involving number of C grades and PANCE

 

Description of data

Massey & Martin, LLC will require the following data from the program

  1. A spreadsheet with notations indicating the number of times that students experience remediation throughout the program. This needs to be connected to a specific course. Excel spreadsheets is probably the most efficient. This analysis will focus on the didactic year. Please indicate the final percentage performance in each academic course throughout the academic year.

  2. A spreadsheet with notations indicating the number of times that students achieved a grade of C or below. This is best provided in an excel spreadsheet with the student’s final performance in each academic course throughout the program measured in percentages. In addition, student performance in formative and summative exams including PANCE needs to be included. If the data is organized so each student’s performance can be tracked longitudinally there does not need to be any identifier such as name. The use of a numerical indicator is best used.

Appendix 14D

Data required

  • Student evaluation of clinical rotations and preceptors

  • The number of final clinical rotation grades of C or below

  • Student attrition and remediation in clinical rotations

 

Appendix 14D: Student Evaluation of Clinical Rotations and Preceptors

  • Rotations listed by type (medical discipline)

  • Data included for all clinical rotations 

  • Preceptor aggregated by medical specialty/discipline

  • If reported, individual preceptors, not identified by name but with an anonymous identifier

  • Quantitative data reported in aggregate and displayed in tables or graphs that directly support the analysis. (Do not provide raw data) 

  • Qualitative data themes used in the analysis reported and summarized in the narrative or displayed in an appended document. (Do not provide raw data)

  • Data presented in a way that allows comparison among rotation evaluations and allows appreciation of trends over time

 

Appendix 14D: The Number of Final Clinical Rotation Grades of C or Below

  • How many students have repeated rotations in the three most recent graduating classes, as well as the classes currently enrolled? 

  • For how many students in the three most recent graduating classes, as well as the classes currently enrolled, has the program identified deficiencies in knowledge and skills during rotations, such that the correction of these deficiencies (remediation) was necessary for the student to continue in the program (not including those who repeated a rotation)?

 

Surveys administered

  1. Student evaluation of clinical rotations and preceptors-all clerkships

  2. Student evaluation of clinical rotation and preceptors - thematic analysis-all clerkships

 

Description of data

Massey & Martin, LLC will require the following data from each program each year

  1. Student evaluation of clinical rotations from all clerkships in the clinical year

  2. Student evaluations of all preceptors from all clerkships in the clinical year

  3. Student comments related to clinical rotations from all clerkships in the clinical year

  4. Student comments related to preceptors from all clerkships in the clinical year

 

Data analyzed

  1. Correlation and regression between EOR and PANCE

  2. Correlation and regression regarding performance in clinical year versus PANCE

 

Description of data

  1. A spreadsheet with notations indicating the number of times that students experienced remediation in the clinical phase of the program. For example, please indicate which clerkship or EORE the student had to retake. Include the grade from the first administration and the second. The use of Excel spreadsheets is probably the most efficient. This also includes the clinical year indicating when students had to retake EORE and SUMMATIVE/OSCE.

  2. A spreadsheet with notations indicating the number of times that students achieved a grade of c or below. This is best provided in an excel spreadsheet with the student’s final performance in each academic course throughout the program measured in percentages. In addition, student performance in formative and summative exams including PANCE needs to be included. If the data is organized so each student’s performance can be tracked longitudinally there does not need to be any identifier such as name. The use of a numerical indicator is best used.

Appendix 14E: Summative Evaluation Performance

Data required

  • Summative evaluation performance

  • Exiting student/graduate feedback 

  • Faculty evaluation of the curriculum to assess its ability to prepare students to achieve program defined competencies

 

Appendix 14E: Summative Evaluation Performance

Requires regression analysis on summative evaluation versus PANCE

  • Provide a tabular or graphic display of data collected by the program on the summative evaluation outcomes for the three most recent graduating classes

    • The data is to be used to support the narrative on analysis of the program’s ability to prepare graduates to achieve its defined competencies

 

Data analyzed

  1. Comprehensive analysis of summative evaluation including predictive formula

 

Description of data

Massey & Martin, LLC will require the following data:

  1. Each student’s summative exam performance. This includes all aspect of the summative exam such as written cognitive performance, OSCE performance and professionalism evaluation

  2. Each student’s PANCE performance. The program needs to include in identifier so they can determine the student’s identity in the final analysis.

  3. Each student’s PACKRAT and EORE performance. As stated above this data needs to be connected so the data can be analyzed as a package for each student.

 

Appendix 14E: Exiting Student/Graduate Feedback

Surveys administered

  1. Student exit survey

  2. Student exit survey thematic analysis

 

Description of data

Massey & Martin, LLC will require the following data:

  1. Spreadsheet with the results of the students exit survey

  2. Spreadsheet with the comments from the students exit survey

Appendix 14F: PANCE Performance

Data required

All data must be tagged for identification so this can be analyzed through regression:

  1. Admissions criteria as a predictor of success

    • Include admissions data from the past three graduating classes; this must be identified so the other academic variables can be longitudinally analyzed statistically

  2. Course outcomes

    • Include performance and percentage for all academic variables in the didactic year and clinical year

  3. Course and instructor evaluations by students 

  4. Program instructional objectives, learning outcomes, and breadth and depth of the curriculum

  5. Student summative evaluation results

    • Include the last three years of any comprehensive exams and summative exams; this data must be tagged to prerequisite data

  6. Remediation practices and results

    • Include the number of times students participated in remediation; this needs to be quantitative numbers by student; this data must be tagged to all other data, so each student/graduate is analyzed separately

  7. Student progress criteria and attrition data

    • Any students who decelerated or those who were withdrawn or dismissed; this data needs to be analyzed against those that were successful

  8. Preceptor and graduate feedback

    • Employer feedback is not required but may be helpful if available to the program

    • All preceptor evaluation of student data from all disciplines; this data must be tagged to all the aforementioned data

    • All graduate data if an exit survey is not conducted 

 

Surveys administered

  1. Preceptor evaluation of student for all 8 rotations

  2. Preceptor evaluation of student thematic analysis of all eight rotations 

 

Description of data

Massey & Martin, LLC will require the following data from the program:

  1. Spreadsheet with the results of preceptor evaluation of student for all clerkships in the program

  2. Spreadsheet with the comments from preceptor evaluation of students for all clerkships in the program

 

Data analyzed

  1. Comprehensive program statistical report

  2. Three year analysis of organ and task area performance

Description of data

Massey & Martin, LLC will require the following data from the program:

  1. Spreadsheet with student performance starting with prerequisite variables and all academic classes measured in final percentages. The spreadsheet also needs to include all clinical year performance such as final clerkship grades in percentage, formative and summative exams (PACKRAT/EORE/SUMMATIVE) and the PANCE 

  2. The program report from NCCPA that indicates the program’s performance for all organ systems compared to the national average

Appendix 14G: Sufficiency and Effectiveness of Principal Faculty and Instructional Faculty

Surveys administered

  1. Sufficiency and effectiveness of principal faculty and instructional faculty (faculty survey)

  2. Sufficiency and effectiveness of principal faculty and instructional faculty (faculty thematic analysis)

  3. Sufficiency and effectiveness of principal faculty and instructional faculty (student survey)

  4. Sufficiency and effectiveness of principal faculty instructional faculty (thematic analysis student)

  5. Sufficiency and effectiveness of program director

  6. Sufficiency and effectiveness of program director (thematic analysis)

 

Description of data

Massey & Martin, LLC will require the following data from the programs:

  1. Final spreadsheet with the results of sufficiency and effectiveness of principal faculty and instructional faculty-faculty survey

  2. Final spreadsheet with the comments from the sufficiency and effectiveness of principal faculty and instructional faculty-faculty survey

  3. Final spreadsheet from the results of sufficiency and effectiveness of principal faculty and instructional faculty - student survey 

  4. Final spreadsheet with the comments from the results of sufficiency and effectiveness of principal faculty and instructional faculty-student survey

  5. Final spreadsheet with the results of the sufficiency and effectiveness of program director

  6. Final spreadsheet with the comments from the sufficiency and effectiveness of program director 

Appendix 14H: Success in Meeting the Program’s Goals

  • Provide a tabular or graphic display of data collected by the program for each of its goals for the past three academic years

    • If already provided in another Appendix, reference that data in the analysis narrative

  • When creating your data display(s), note: 

    • Quantitative data must be reported in aggregate and displayed in tables or graphs that directly support the analysis. (Do not provide raw data)   

    • Qualitative data themes used in the analysis must be reported and summarized in the narrative or displayed in an appended document. (Do not provide raw data)

    • Data should be presented in a way that allows comparison across years and appreciation of trends over time

    • Reference other appendices of the SSR as needed

 

Data analyzed

  1. Assembly and tabulation of all outcome data related to goals 

 

Description of data

Massey & Martin, LLC will require the following data from the program:

  1. Spreadsheet with raw data related to all program goals

  2. Spreadsheet with raw data from surveys related to program goals

  3. Spreadsheet from admissions data related to program goals 

  4. Spreadsheets with program outcomes related to program goals

Elements of Comprehensive Statistical Report (Provided annually)

Each year, the University of PA studies is required to complete a comprehensive annual statistical report comprised of the elements described below, providing valuable information relevant to components within Appendix 14.

Example

1. Correlation between Performance Variables and PANCE Scores

2. Regression

  1. Stepwise Regression:

    • All Predictor Variables

  2. Stepwise Regression:

    • Phase I Predictor Variables

  3. Stepwise Regression:

    • Phase II Predictor Variables

  4. Regression:

    • Individual Predictor Variables

  5. Each academic course as a predictor of PANCE score

  6. PACKRAT 1:

    • Total Score as a Predictor of PANCE Score

  7. Each individual EORE as a Predictor of PANCE Score

  8. PACKRAT 2:

    • Total Score as a Predictor of PANCE Score

  9. Summative Exam Score as a Predictor of PANCE Score

  10. Remediation and C Grades as a predictor of PANCE scores

  11. Difference between PANCE score of candidates who received at least one remediation and the candidates who did not receive any remediation

  12. Difference between PANCE score of candidates who received at least one C grade and the candidates who did not receive any C grade

 

Description of data

The comprehensive statistical report requires a spreadsheet of student data starting with the prerequisite phase all the way through graduation. This needs to be organized and presented to show metric data such as the final percentage performance in a course. All comprehensive exams such as PACKRAT must be displayed in raw score only. Summative exams also need to be a raw score which is the total number achieved/number possible. The students numerical PANCE score must accompany the spreadsheet. Descriptors do not need to be used in terms of student ID. Numbers can be used as long as they follow the data across the spreadsheet.

 

The data also must contain the number of remediations and the number of C grades throughout the span of the program.

© 2021 by Massey & Martin, LLC

  • LinkedIn - White Circle

Tel: 954-551-0087

|           United States