The Competency Assessment Toolkit for Professional Psychology

1. The Competency Assessment Toolkit for Professional Psychology Catherine L. Grus, Ph.D., Associate Executive Director for Professional Education and Training Education Directorate American Psychological Association

2. THANK YOU Many thanks to • Nadya Fouad, University of Wisconsin • Nadine Kaslow, Emory University • Benchmarks Workgroups • Assessment “Toolkit” Workgroup

3. Overview • Considerations in the assessment of competence • Development of the Toolkit • Putting the toolkit into practice

4. Competence … • “ … the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and community being served.” From Epstein and Hundert (2002)

5. Competence … • … is developmental depending on stage of professional functioning • … context dependent • … executed according to ethical principles, guidelines, standards, and values of the profession and • … requires public verification

6. Competencies … • Observable • Measurable • Containable • Practical • Derived by experts • And flexible • are elements of competence that are ….

7. Assessment of Competence - Considerations • Use of both “formative” and “summative” evaluation methods • Authentic evaluations • Real world tasks, meaningful activities • [High] Fidelity: Degree to which the assessment represents actual performance

8. Employ a range of perspectives (multi-informant) and methods • Challenge of evaluating attitudes and values • Psychometric issues • Feasibility/costs • Lifelong self-assessment

9. What methods are commonly used to assess students/ trainees? • In class examinations and course “grades” • Faculty/supervisor completes rating scale • Letters of recommendation • EPPP, state jurisprudence exams

10. Developing Faculty/Supervisor Competencies • Training in “difficult conversations” • How to “critique constructively” • Develop awareness of potential dual roles in evaluation and how that could influence results • Benefits of having other faculty evaluate students’ performance • Explicit obligations to make evaluations accurate • Grade inflation • Letters of recommendation to “market” vs. evaluate • Knowledge of a range of assessment tools, access to resources “toolkits”

11. Competency Assessment for “Toolkit” for Professional Psychology • Charge from APA Board of Educational Affairs: Develop a “Toolkit” for professional psychology • Purpose: Promote broader implementation of competence assessment and provide information about application of assessment methods to the assessment of competence • Coordinated with Benchmarks Work Group

12. Foundational Competencies: Professionalism Reflective practice Scientific knowledge and methods Relationships Individual and cultural diversity Ethical and legal standards and policy Interdisciplinary systems Functional Competencies: Assessment Intervention Consultation Research and evaluation Supervision Teaching Administration Advocacy Benchmarks Core Competencies

13. How are the Benchmarks Organized? • Core Foundational and Functional Competencies • Essential Component: what are the critical elements of/what knowledge/skills/attitudes that make up this competency? • Behavioral Anchor: what would it look like if you saw it (essential component)?

14. Components of Toolkit • Background and Introduction • Assessment Method Fact Sheets • Grid of Assessment Methods and Competencies/Essential Components Best used for

15. Fact Sheet: Portfolio Review Description A portfolio is a collection of products, gathered by the person being assessed … Application • Implementation • Deciding on form (e.g., web-based or hard copy) • Psychometric Properties • Reliability has not been well established due to the variable content included in a portfolio • Strengths • Provides a low cost assessment strategy • Challenges • Requires intense commitment of time and labor intensive for the assessors and the person being assessed • Future directions • Create and implement training modules for assessors and people being assessed in the portfolio assessment strategy

16. Grid of Possible Assessment Methods (Toolkit)

17. 360 evaluation Portfolio OSCE Structured Written & Oral Exams Case Presentation Simulation/Role Play Competence Evaluation Rating Form Self-Assessment Ratings of live or recorded performance Standardized Client Interview Client/Patient Process/Outcome Measure Consumer Satisfaction Survey End of Rotation Performance Review Assessment Methods in Toolkit

18. Putting it all Together … Benchmarks Toolkit Using the Toolkit and Benchmarks to Guide the Assessment of Student Learning Outcomes

19. Identify expected student learning outcomes using competency model (e.g., Benchmarks) Expected level of competence attainment and training plan developed with trainee Select assessment method(s) Student demonstrates problems with competencies acquisition Remediation plan developed Formative assessment Student demonstrates emerging competencies Training plan continues Summative evaluation to assess readiness for next level of training

20. Identify expected student learning outcomes using competency model (e.g., Benchmarks)

21. Program Outcome: Students will demonstrate the following intervention (functional) competencies prior to going on internship. Knowledge of scientific, theoretical, empirical and contextual bases of intervention, including theory, research, and practice Formulates and conceptualizes cases and plan interventions utilizing at least one consistent theoretical orientation Clinical skills Implements evidence-based interventions that take into account empirical support, clinical judgment, and client diversity (e.g., client characteristics, values, and context) Evaluate treatment progress and modify treatment planning as indicated, utilizing established outcome measures

22. Identify expected student learning outcomes using competency model (e.g., Benchmarks) Select assessment method(s)

23. Grid of Possible Assessment Methods (Toolkit)

24. Identify expected student learning outcomes using competency model (e.g., Benchmarks) Expected level of competence attainment and training plan developed with trainee Select assessment method(s)

26. Identify expected student learning outcomes using competency model (e.g., Benchmarks) Expected level of competence attainment and training plan developed with trainee Select assessment method(s) Student demonstrates problems with competencies acquisition Remediation plan developed Formative assessment Student demonstrates emerging competencies Training plan continues

27. Remediation Plan

28. Identify expected student learning outcomes using competency model (e.g., Benchmarks) Expected level of competence attainment and training plan developed with trainee Select assessment method(s) Student demonstrates problems with competencies acquisition Remediation plan developed Formative assessment Student demonstrates emerging competencies Training plan continues Summative evaluation to assess readiness for next level of training

29. For more information: http://www.apa.org/ed/graduate/competency.html Coming soon: Kaslow, N. J., Grus, C. L., Campbell, L. C., Fouad, N. A., Hatcher, R. L., & Rodolfa, E. R. (in press) Competency Assessment Toolkit for Professional Psychology. Training and Education in Professional Psychology.