An Introduction to Understanding the OT and OTA Fieldwork Performance Evaluations (FWPEs)

1. An Introduction to Understanding the OT and OTA Fieldwork Performance Evaluations (FWPEs) Karen Atler, MS, OTR Colorado State University, Fort Collins, CO Roberta Wimmer, OTR/L Pacific University, Forest Grove, OR  2003 The American Occupational Therapy Association, Inc. www.AOTA.org

2. Task Force Members • Carole Dennis, PhD, OTR • Ithaca College, New York • Carole Hays, MA, OTR • Springfield Hospital Center, Maryland • Becky Robler, MEd, OTR • Pueblo Community College, Colorado • Karen Atler, MS, OTR, Co-Chairperson • Colorado State University • Roberta Wimmer, OTR, Co-Chairperson • Pacific University, Oregon

3. Objectives • Describe the • Entry-level practice competencies for OT and OTA students • Purpose, format, content, and scoring of the companion evaluation forms • New concepts and terminology used in the FWPE from the OT Practice Framework • Begin to score items on the FWPE

4. Task Force’s Charge • Revise/develop evaluation tools to measure assistant and professional Level II fieldwork student performance. • Expectations • Conduct review of literature across disciplines • Synthesize feedback on current AOTA FWE/OT forms • Incorporate 1997 NBCOT Practice Analysis results • Address identified desired characteristics

5. Desired Characteristics • Companion documents for assistant and professional level that • Measure entry-level competence • Focus on occupation-based practice • Reflect current and future practice • Can be used in a variety of settings • Provide feedback to students • Can be easily used in a timely manner

6. What OTs & OTAs Do Determining needs/priorities for interventions Identifying/designing interventions Implementing interventions Reporting/evaluating intervention effectiveness Providing OT services for populations Managing delivery of OT services Advancing effectiveness ofthe OT profession What OTs & OTAs Need To Know Human development and performance Principles/strategies in the identification/evaluation of strengths and needs Principles/strategies in intervention/treatment planning Principles/strategies inintervention Nature of occupation and occupational performance Service management Responsibilities as a professional NBCOT Practice Analysis 1997

7. Standards of Practice for Occupational Therapy • Identifies minimum standards • Identifies key performance areas for the OT and OTA • Professional standing and responsibility • Referral • Screening • Evaluation • Intervention plan • Intervention • Transition services • Discontinuation

8. OT Be a generalist Achieve entry-level competence Articulate, apply, and justify occupation interventions Supervise and collaborate with the OTA Keep current with best practice Uphold the ethics, values, and attitudes of the profession Be an effective consumer of research and knowledge OTA Be a generalist Achieve entry-level competence Work under the supervision of andin cooperation with the OT Articulate, apply, and justify interventions related to occupation Keep current with best practice Uphold the ethics, values, and attitudes of the profession ACOTE: Minimum Standards and Outcomes

9. Goal of Level II Fieldwork Education for the OT and OTA Student • Develop competent, entry-level generalists • Include an in-depth experience in delivering occupational therapy services • Be designed to promote reasoning, enable ethical practice,anddevelop professionalism

10. The Process • Began with OTA evaluation • Reviewed by experienced panel • Submitted to COE • Made revisions • Completed pilot studies (2 OTA, 1 OT)

11. Design and Analysis of Pilot Studies: The Rasch Measurement Model less able Student Ability more able

12. Rating Scale Usage in Pilot Studies II and III Rating Scale Descriptors % Usage OT % Usage OTA 1 = Unsatisfactory 0 0 2 = Needs Improvement 10 13 3 = Meets Standards 56 53 4 = Exceeds Standards 34 29 Results of Pilot Studies • Good representation in pilot samples • Students and educators preferred new form • Good scale and response validity • Inaccurate use of scale

13. The FWPEs for OT and OTA Students • Companion documents • Terminology • Content layout • Purpose • Design • Rating scale • Scoring system

14. FWPEs OT and OTACompanion Documents • Focus • The occupational therapy process • The clinical reasoning process • Roles and responsibilities of the OT and OTA • Structure • Collaborative process—student and FW educator • Same layout • Same rating/scoring system

15. Terminology of the FWPEs • Reflects • Standards of Practice for Occupational Therapy and ACOTE Education Standards • Occupational Therapy Practice Framework • The glossary

16. Content Layout of FWPEs • Summary Sheet • Overview/instructions • Organization of items • Space for comments—midterm and final • Performance Rating Summary Sheet

17. OTA Fundamentals of practice (3) Basic tenets (3) Evaluation/screening (5) Intervention (6) Communication (2) Professional behavior (6) OT Fundamentals of practice (3) Basic tenets (4) Evaluation/screening (10) Intervention (9) Management of OT services (5) Communication (4) Professional behavior (7) Content of OT and OTA Evaluations

18. Primary Purposeof the FWPEs • Measures entry-level competence • Designed to differentiate the competent student from the incompetent student • Not designed to differentiate levels above entry-level competence

19. Purpose (continued) • Provides student with accurate assessment of his or her competence for entry-level practice over time • Growth occurs over time • Midterm and final scores reflect this change • Midterm scores: Satisfactory–unsatisfactory • Final scores: Pass–no pass

20. Purpose (continued) • Provides feedback to student • Provides opportunity for student self-assessment

21. Design of the FWPEs • The “doing” of the occupational therapy process is evaluated, not the individual tasks in isolation • NOT all items are equal in level of difficulty (i.e., simple to complex) • Evaluation is supplemented with development of site-specific objectives

22. - - 2- - - - - - - - - - 1- - - - - - - - - - 0- - - - - - - - - - -1- - - - - - - - - - -2- Interpersonal; 41 Respnds 2 fdbk 38 Diversity; 42 Ethics; 1 Collab c super;36 Workbeh 39, safety 2 Steps 3,Resp; 37 Timemng40; Legible 34 Client centered interv 22 Occupn based inter 23 Selects rel occu; 21 Produces work; 31 Clear documentation; 33 Org goals 30; verbl 32 Documents intrv; 26 Collaborates clnt; 7 Language reflects 35 Documents eval rslts 17 OTbeliefs 4 Obtain inf 12 Role of OT; 6 Rationale Tx 18, Administer asses 13 Artic ratinal eval; 8 Artic value occ 5 Modifies approach 24 Underst finances; 29 Collaborates OTA 28 Occ prof 10, Adjst ass 14 Est Plan 16, Assess ftr 11 Updates; 25, Evidence 19 Interprets eval reslts; 15 Selects rel asses methd; 9 Assigns resp OTA; 27 HARD RASCH Ordering of Items OT

23. Easier 2 - - Cultural competence; 25 - - Ethics; 1 - Interpersonal skills;24 - Responds to Feedback;21 - - Safety; 2 and 3 - - 1 - - - - Work Behaviors; 22 - - Therapeutic Use Self;16 - - Written Communication;19 - - 0 - Verbal Communication; 18 Self-responsibility; 20 - Implements intervntn; 14 - I - OT/OTA Roles 5DataGather - - - Selects Intervention;13 - Activity Analysis; 15 - - Evidence BasedPractic 6 - 1 - Plans Intervention; 12 - Reports; 10 - OT Philosophy; 4 - Administer Assessmnts;8 - Establishes Goals; 11 - Modifies Intrven Plan;17 - - Interprets Assessment;9 - - - 2 - Harder Rasch Ordering of Items OTA

24. 4 = Exceeds Standards Performance is highly skilled and self-initiated. This rating is rarely givenandwould represent the top 5% of all the students you have supervised. 3 = Meets Standards Performance is consistent withentry-levelpractice. This rating is infrequently given at midterm and is a strong rating at final. 2 = Needs Improvement Performance is progressing but still needs improvement for entry-level practice. This is a realistic rating of performance at midterm and some ratings of 2 may be reasonable at the final. 1 =Unsatisfactory Performance is below standards and requires development for entry-level practice. This rating is given when there is concern about performance. Rating Scale of FWPEs

25. Scoring System of FWPEs • Each item must be scored • Ethics and safety items must be passed • Each item rating recorded on Performance Rating Summary Sheet • All items summed up at midterm and final • Score compared to scales provided

26. OTA Overall Midterm Score Satisfactory: 54 & above Unsatisfactory: 53 & below Overall Final Score Pass: 70 & above No Pass: 69 & below OT Overall Midterm Score Satisfactory: 90 & above Unsatisfactory: 89 & below Overall Final Score Pass: 122 & above No Pass: 121 & below Midterm and Final Scoresof the FWPEs

27. Rating Performance Using the FWPEs Case Scenarios John—OT Evaluation Mary—OT Intervention Sandra—OTA Evaluation/Screening David—OTA Intervention

28. Individualizing the FWPEs • Designed for additionalobjectives to be written to add clarification • Site-specific objectives • NOT supervisor-specific • If an item is very clear and meets the RUMBA test, then there is no need to write another objective

29. Objectives: An Example From FWPE for the OT Student • 16. Establishes accurate and appropriate plan • School—Provides behavioral-based, measurable OT goals during IEP process • Acute Care—Overall intervention plan is achievable within client’s length of stay

30. Summary • Evaluations designed to measure entry-level competence, NOT level of performance above competency • OT practice examined as a generalist • Evaluations reflect the occupational therapy process • Performance develops over time