An Introduction to Understanding the OTA Fieldwork Performance Evaluation (FWPE)

1. An Introduction to Understanding theOTA Fieldwork Performance Evaluation (FWPE) Karen Atler, MS, OTR Colorado State University, Fort Collins, CO Roberta Wimmer, OTR/L Pacific University, Forest Grove, OR  2003 The American Occupational Therapy Association, Inc. www.AOTA.org

2. Task Force Members • Carole Dennis, PhD, OTR • Ithaca College, New York • Carole Hays, MA, OTR • Springfield Hospital Center, Maryland • Becky Robler, MEd, OTR • Pueblo Community College, Colorado • Karen Atler, MS, OTR, Co-Chairperson • Colorado State University • Roberta Wimmer, OTR, Co-Chairperson • Pacific University, Oregon

3. Objectives for Today: • Describe the… • Entry-level practice competencies for OT and OTA students. • Purpose, format, content, and scoring of the companion evaluation forms. • New concepts and terminology used in the FWPE from the OT Practice Framework. • Begin to score items on the FWPE.

4. Task Force’s Charge • Revise/develop evaluation tools to measure assistant and professional Level II fieldwork student performance. • Expectations: • Conduct review of literature across disciplines. • Synthesize feedback on current AOTA FWE/OT forms. • Incorporate 1997 NBCOT Practice Analysis results. • Address identified desired characteristics.

5. Desired Characteristics • Companion documents for assistant and professional level that… • Measure entry-level competence. • Focus on occupation-based practice. • Reflect current and future practice. • Can be used in a variety of settings. • Provide feedback to students. • Can be easily used in a timely manner.

6. NBCOT Practice Analysis 1997 What OTs & OTAs DO • Determining needs/priorities for interventions. • Identifying/designing interventions. • Implementing interventions. • Reporting/evaluating intervention effectiveness. • Providing OT services for populations. • Managing delivery of OT services. • Advancing effectiveness of the OT profession.

7. What OTs & OTAs NEED TO KNOW Human development and performance. Principles/strategies in the identification/evaluation of strengths and needs. Principles/strategies in intervention/treatment planning. Principles/strategies in intervention. Nature of occupation and occupational performance. Service management. Responsibilities as a professional. NBCOT Practice Analysis 1997

8. Standards of Practice for OT • Identifies minimum standards. • Identifies key performance areas for the OT and OTA: • Professional standing and responsibility • Referral • Screening • Evaluation • Intervention plan • Intervention • Transition services • Discontinuation

9. ACOTE:Minimum Standards and Outcomes for OTA • Be a generalist. • Achieve entry-level competence. • Work under the supervision of andincooperation with the OT. • Articulate, apply, and justify interventions related to occupation. • Keep current with best practice. • Uphold the ethics, values, and attitudes of the profession.

10. Goal of Level II Fieldwork Education for the OTA Student • Develop competent, entry-level generalists. • Include an in-depth experience in delivering OT services. • Be designed to promote reasoning, enable ethical practice,anddevelop professionalism.

11. The Process • Began with OTA evaluation. • Reviewed by experienced panel. • Submitted to COE. • Made revisions. • Completed pilot studies (2 OTA, 1 OT).

12. Design and Analysis of Pilot Studies: The Rasch Measurement Model less able Student Ability more able

13. Rating Scale Usage in Pilot Studies II and III Rating Scale Descriptors % Usage OT % Usage OTA 1 = Unsatisfactory 0 0 2 = Needs Improvement 10 13 3 = Meets Standards 56 53 4 = Exceeds Standards 34 29 Results of Pilot Studies • Good representation in pilot samples. • Students and educators preferred new form. • Good scale and response validity. • Inaccurate use of scale.

14. FWPEs for OT and OTA Students • Companion documents • Terminology • Content layout • Purpose • Design • Rating scale • Scoring system

15. FWPEs OT and OTACompanion Documents • Focus • The OT process • The clinical reasoning process • Roles and responsibilities of the OT and OTA • Structure • Collaborative process—student and FW educator • Same layout • Same rating/scoring system

16. Terminology of the FWPEs • Reflects… • Standards of Practice for OT and ACOTE Education Standards • OT Practice Framework • The glossary

17. Content Layout of FWPEs • Summary Sheet • Overview/instructions • Organization of items • Space for comments—midterm and final • Performance Rating Summary Sheet

18. Content of OT and OTA Evaluations

19. Primary Purposes of the FWPEs • Measures entry-level competence: • Designed to differentiate the competent student from the incompetent student. • NOT designed to differentiate levels above entry-level competence.

20. Purpose (continued) • Provides student with accurate assessment of his/her competence for entry-level practice over time: • Growth occurs over time. • Midterm and final scores reflect this change. • Midterm scores: Satisfactory/unsatisfactory. • Final scores: Pass/no pass.

21. Purpose (continued) • Provides feedback to student. • Provides opportunity for student self-assessment.

22. Design of the FWPEs • The “doing” of the OT process is evaluated, not the individual tasks in isolation. • NOT all items are equal in level of difficulty (i.e., simple to complex). • Evaluation is supplemented with development of site-specific objectives.

23. Easier 2 - - Cultural competence; 25 - - Ethics; 1 - Interpersonal skills;24 - - Responds to Feedback;21 - Safety; 2 and 3 - - 1 - - - - Work Behaviors; 22 - - Therapeutic Use Self;16 - Written Communication;19 - - - 0 - Verbal Communication; 18 - Self-responsibility; 20 Implements intervntn; 14 - I - OT/OTA Roles 5DataGather - - - Selects Intervention;13 - Activity Analysis; 15 - - Evidence BasedPractic 6 - 1 - Plans Intervention; 12 Reports; 10 - - OT Philosophy; 4 Administer Assessmnts;8 - - Establishes Goals; 11 - Modifies Intrven Plan;17 - - Interprets Assessment;9 - - - 2 - Harder Rasch Ordering of Items OTA

24. 4 = Exceeds Standards Performance is highly skilled and self-initiated. This rating is rarely givenandwould represent the top 5% of all the students you have supervised. 3 = Meets Standards Performance is consistent withentry-levelpractice. This rating is infrequently given at midterm and is a strong rating at final. 2 = Needs Improvement Performance is progressing but still needs improvement for entry-level practice. This is a realistic rating of performance at midterm and some ratings of 2 may be reasonable at the final. 1 = Unsatisfactory Performance is below standards and requires development for entry-level practice. This rating is given when there is concern about performance. Rating Scale of FWPEs

25. Scoring System of FWPEs • Each item must be scored. • Ethics and safety items must be passed. • Each item rating recorded on Performance Rating Summary Sheet. • All items summed up at midterm and final. • Score compared to scales provided.

26. Midterm and Final Scores OTA Overall Midterm Score • Satisfactory: 54 & above • Unsatisfactory: 53 & below Overall Final Score • Pass: 70 & above • No Pass: 69 & below

27. Rating Performance Using the FWPEs Case Scenarios • Sandra—OTA Evaluation/Screening • David—OTA Intervention

28. Individualizing the FWPEs • Designed for additionalobjectives to be written to add clarification: • Site-specific objectives • NOT supervisor-specific • If an item is very clear and meets the RUMBA test, then there is no need to write another objective.

29. Objectives: An Example From FWPE for the OT Student • 16. Establishes accurate and appropriate plan • School—Develops behavioral-based, measurable OT goals during IEP process. • Acute Care—Overall intervention plan is achievable within client’s length of stay.

30. Summary • Evaluations designed to measure entry-level competence, NOT level of performance above competency. • OT practice examined as a generalist. • Evaluations reflect the OT process. • Performance develops over time.