Defining Outcomes and Creating Valid Measurement Tools in Athletic Training Education

1. Leamor Kahanov EdD, ATC & Lindsey E. Eberman PhD, ATC Defining Outcomes and Creating Valid Measurement Tools in Athletic Training Education

2. Outcome Measures • Presentation will focus on educational outcome measures • Other outcome measures in athletic training/medicine critical in the practice and delivery of health care: • Clinical Outcomes/Evidence Based Medicine • Patient Centered Outcomes/Patient Satisfaction Great Lakes Athletic Trainers Association

3. The Assessment Loop (Closing) 1. Goals, questions 4. Use (Closing) 2. Gathering evidence 3. Interpretation Great Lakes Athletic Trainers Association

4. Uses of Educational Outcomes • To validate current program outcomes • Basis for program modification or remediation Great Lakes Athletic Trainers Association

5. Types Outcome Models • 8-10 Educational Outcome Models (depending on literature source use) • Program Outcome Model: Uses set of related variables • OUTCOMES • New knowledge • Increased skill • Values/attitudes • Modified behavior • Improvements • OUTPUTS • Courses instructed • Educational Materials • ACIs • ACTIVITIES • Education • Counseling • Clinical Experiences • Service • INPUT • Staff/Faculty • Money • Equipment • Facilities

6. Creating and Validating Outcome Measures • Create appropriate questions to assess desired outcome • Assess the instrument constructed of questions • Validate the instrument to ensure accurate measurement Great Lakes Athletic Trainers Association

7. Where to Start Great Lakes Athletic Trainers Association

8. How to ask the questions and evaluate Great Lakes Athletic Trainers Association

9. How to Make a Valid Question • Questions need to be discrete and focused • Questions need to assess only one variable • Avoid Compound questions: “The instructors in the athletic training program provide advising and are personable.” Great Lakes Athletic Trainers Association

10. Creating “GOOD” Questions • Questions should avoid qualifiers “The athletic training program has a good laboratory.” “The athletic training program has enough laboratory space to conduct learning activities.” Great Lakes Athletic Trainers Association

11. Questions on Input “The mission of the Athletic Training Department is to provide excellence in professional preparation and patient healthcare. The department supports student growth and the development of productive citizens through mentorship in the quest for new knowledge and research, community engagement, clinical experience, service learning and progressive comprehensive health care.” Great Lakes Athletic Trainers Association

12. Constructing Input Questions • Mission Statement • The ATEP provides professional preparation for comprehensive health care. • The ATEP develops students through mentorship. • The ATEP develops students through exposure to new knowledge and research. • The ATEP develops students through community engagement. • The ATEP develops students through clinical experiences. Great Lakes Athletic Trainers Association

13. Questions on Output • Program Objectives “Cognitive and psychomotor skill mastery, clinical proficiency and demonstration of sound clinical decision making in the application of clinical evaluation and diagnosis. “ • Student evaluation • My education has taught me how to make sound clinical decision to evaluate and diagnose patients. • Employer evaluation • My employee makes sound clinical decisions in the evaluation and diagnosis of patients. Great Lakes Athletic Trainers Association

14. Questions on Activities • The Orthopedic Evaluation Lab course allowed enough time to practice skills. • The Clinical Site provided individualized attention. • The BOC practice examination prepared me to take the BOC exam. Great Lakes Athletic Trainers Association

15. To repeat or not to repeat? That is the question • Consistency and aggregate information • Ask consistent questions across evaluations • Activities will inherently ask different questions Exit Alumni Employer • I will be able to make sound clinical decisions in the evaluation and diagnosis of patients. • I make sound clinical decisions in the evaluation and diagnosis of patients. • My employee makes sound clinical decisions in the evaluation and diagnosis of patients. Great Lakes Athletic Trainers Association

16. The Assessment Loop (Closing) 1. Goals, questions 4. Use (Closing) 2. Gathering evidence 3. Interpretation

17. WARNING THIS IS NOT RESEARCH JUST GOOD ASSESSMENT Great Lakes Athletic Trainers Association

18. Instrument Validation • Validity • The extent to which a test measures what it is intended to measure (McDowell & Newell, 1987) • Factor Analysis • Are we asking the right questions? • Are we asking variable questions? • May require statistical expertise which should be available through University resources

19. Standards of Behavior for ACIs • Legal and Ethical Behavior • Communication skills • Interpersonal relationships • Instructional skills • Supervisory and administrative skills • Evaluation of performance • Clinical skills and knowledge

20. On-line ACI Evaluations

21. What’s next? FACTOR ANALYSIS Exploratory Confirmatory Confirm where your items are “loading” Structural Equation Modeling • Identify where your items are “loading” • Principle Component Analysis

22. Output

23. Interpretation

24. Interpretation

25. Reassessment and Revision • Items “loading” on too few/many factors • 34% of variance assumed by one factor • “Rapport” • 66% of variance assumed by nine other factors • May suggest that a positive/negative review will depend on these factors • Feedback from users • Means of administration • Usability

26. Modification • “I feel like my (A) CI works to protect me against experiences where I might be working independently, instead of being supervised directly.” • Only item to load on Factor # 9 • “Direct supervision” • Independence may be confused with indirect supervision • 2 items? • Autonomy • Direct supervision

27. The Assessment Loop (Closing) 1. Goals, questions 4. Use (Closing) 2. Gathering evidence 3. Interpretation

28. When to Assess and Revise Programs Program modification should occur at the end of an evaluation cycle 12-18 months

29. Thank you Leamor Kahanov EdD, ATC & Lindsey E. Eberman PhD, ATC Defining Outcomes and Creating Valid Measurement Tools in Athletic Training Education