Portfolio Evaluation: Assessing Competency

1. Portfolio Evaluation: Assessing Competency CCPTP conference: 2010 Joint Conference Michael J. Scheel, Ph.D. University of Nebraska-Lincoln

2. Portfolio: The Concept • “A purposeful collection of student work that tells the story of the student’s efforts, progress, or achievement in a given area(s)” (p. 36; Arter & Spandel, 1992). • Portfolio assessment is used for faculty tenure evaluations; for some license or diplomate evaluations • In counseling, beyond performance on a standardized test or completion of courses, case examples serve to demonstrate competence (Coleman, 1996). • “An artist’s portfolio shows not only what the artist knows but how he or she can put that knowledge into practice over a range of situations” (p. 219; Coleman, 1996).

3. Portfolio: The Advantages • Can demonstrate what one knows that reflects the complexity of the topic and the integration of skills with knowledge (Collins, 1992). • Integration of information collected over several contexts • Provides stimulus for discussion in which the student can present the meaning of an activity or a competency. • Can demonstrate increasing sophistication over time • Stimulates self-reflection because the learner is responsible for deciding the goal and the content of the assessment materials.

4. The portfolio process • Determine the goal • Identify what evidence will demonstrate acquistion of the goal • Submit a caption (competence statement) about the importance of the evidence • Prepare a statement (narrative) about how the portfolio as a whole reflects learning • Fosters a self-reflective process by the learner (Collins, 1992)

5. Examples of evidence/artifacts • Client rating of counselor effectiveness; supervisor evaluations • Self-evaluations; integrative narratives of a content area • Videotapes of therapy sessions that demonstrate an intervention approach • Case presentations; Ethical dilemmas presented through a case • A personal ethical decision-making model • Personal theory of change • One page essays on theory orientation, multicultural competence, use of assessment in practice etc. similar to an APPIC app. • Case presentations and write-ups • Outreach programs presented with powerpoints; presentations at prof. conferences; publications • Psych. assessments conducted; written reviews of assessment instruments; reviews of literature

6. UNL Copsych Portfolio • Only the counseling psychology program within the Dept. of Ed. Psych. uses portfolio evaluation with an oral examination; satisfies the Graduate College comprehensive exam requirement • Continuance refining process since 2004; Copsych program approved latest version in the Fall of 2009 as the only comprehensive exam method for our program

7. UNL Portfolio Procedure • meet with your academic advisor to discuss the portfolio procedure and plans for completing comprehensive examinations • develop a plan during Program of Study committee meeting in consultation with your doctoral committee regarding the portfolio procedure; doctoral committee members are invited to participate; all program core faculty are required to participate • work closely with your academic advisor to assemble portfolio; submit a completed version to your advisor; portfolio is comprised of a narrative summary for each competency area; written narratives are approx. 7 to 10 pp. for each area; the bulk of the portfolio is comprised of these narrative summaries; a limited number of accompanying documents/artifacts may be included as supplementary materials to enhance or further illustrate information contained in the narrative summaries; E.g., theoretical orientation essay; mc self-exploration paper, integrated assessment report.

8. UNL Portfolio Procedure (cont.) 4) Advisor distributes portfolio to each committee member and each committee member reviews 5) Meeting is held with the committee and student to discuss each area of the portfolio; Committee members bring questions and needs for clarification to the meeting after individual reviews of the portfolio • Committee arrives at a consensus decision of ‘pass’ or ‘no pass’ after the conclusion of the meeting; • If ‘no pass’ is assigned to a portion of the exam, the student will be given instructions concerning changes and additions to be required to bring the area or areas up to a passing level.

9. Contextual factors • The majority of coursework that fulfills the Counseling Psychology Core Curriculum should be completed at the time of Comprehensive Exam (most students complete portfolio exam in 5th or 6th semester). • Doctoral students applying for internship require notification from the Director of Training that they have completed their comprehensive exam at the time of their application. Thus, successful completion of comprehensive examinations (as well as successful proposal of dissertation) must be achieved by October 15th of the year in which the student is applying for pre-doctoral internship.

10. Portfolio Contents • Scoring rubrics are provided to assist students in considering what material to include in their doctoral portfolios. Specifically, students are expected to demonstrate competence in the following six areas: 1. Measurement, Assessment, and Psychological Testing 2. Counseling Theories and Practice 3. Career Development 4. Multicultural Issues 5. Ethical, Legal and Professional Issues 6. Area of Emphasis

11. Scoring Rubric

12. Counseling Theory and Practice • Portfolio: Examples of performance indicators 1) Knows constructs and major theories and systems of thought 2) Personal theory of change articulated 3) Counseling interventions 4) Evaluate effectiveness of treatment

14. Foundational competencies that line up with portfolio performance indicators Foundational Comp Benchmarks: • Scientific knowledge/methods: applies evidence in practice; compares and contrasts theoretical perspectives; scientific mindedness – articulates issues derived from lit. in supervision and case conf; reviews scholarly work related to clinical practice; willingness to present work for scrutiny by others • Reflective practice self-assessment: awareness of personal strengths; summarizes lit relevant to client care • Individual and cultural diversity: monitors and applies knowledge of self as a cultural being in assessment, treatment, and consultation • Interdisciplinary Systems: demonstrates ability to articulate the role that others provide in service to clients

15. Functional Competencies • Assessment, Diagnosis, and Case Conceptualization: identifies and conceptualizes symptoms through a developmental context; formulates treatment plans • Intervention: Knowledge of interventions • Intervention: planning • Intervention: implementation - applies evidence-based interventions that take into account empirical support, clinical judgment, and client diversity; clinical skills used in development of therapeutic relationships; uses clinical judgment effectively

16. Portfolio area: Multicultural Counseling • Examples of performance indicators • Integrated conceptualization of mc competencies • Awareness of personal values  other cultural groups • Knowledge and application of mc theories of psychology • Demonstrates mc skills • Demonstrated plan for continued mc competence

18. Foundational Competencies: Multicultural Counseling • Reflective practice self assessment: reflective practice self-identifies cultural identities; articulates attitudes, values, and beliefs toward diverse others • Relationships: interpersonal relationships: negotiates relationships with ind and groups that significantly differ from oneself • Individual Cultural Diversity: Self-awareness (e.g., knowledge of self as a cultural being; understands cultural identities; critically evaluates feedback and initiates consultation/supervision over diversity) Applied knowledge (e.g., knowledge of cultural diversity science, theory, and contextual issues; sensitivity to treatment applications with diverse others; understands individual cultural diversity and APA policies; conceptualizes diversity in appropriate and effective ways; considers diversity in assessment, diagnosis and case conceptualization; uses culturally sensitive instruments; seeks consultation; knowledge of multiple identities in case presentations and impact on these treatments)

19. Functional Competencies • Assessment, diagnosis and case conceptualization: assessment (e.g., aware and use culturally sensitive instruments); Integration (e.g., displays know. of assessment tools specific to a client pop. & specific site) • Intervention: Planning (e.g., contexually oriented including diversity); Implementation (e.g., applies evidence interventions while considering cultural diversity) • Supervision and teaching: Awareness of factors affecting quality (e.g., impact of diversity on prof. settings and supervision; awareness of APA and other policies as interpreted through cultural and other contexts)

20. Future directions for portfolio evaluation development • Further lining up of portfolio criteria with the competency benchmarks • A need for research to validate portfolio assessment as a method to be used to measure competence (e.g., concurrent validity studies) • Need to develop reliable methods of scoring and evaluation • Any scoring method must recognize and value the holistic and integrated nature of the performance (Collins, 1992). Resist the temptation to be reductionistic.

21. References • Baltimore, M. L., Hickson, J., George, J. D., Crutchfield, L. B. (1996). Portfolio assessment: A model for counselor education. Counselor Education and Supervision, 36(2), 113-121. • Coleman, H. L. K. (1996). Portfolio assessment of multicultural counseling competency. The Counseling Psychologist, 24(2), 216-229. • Coleman, H. L. K. (1997). Multicultural counseling competencies: Assessment, Education, and Training, and Supervision. In Pope-Davis, D. B. & Hardin, L. K. (Eds.) Multicultural aspects of counseling series, Vol. 7. pp. 43-59. Thousand Oaks, CA: Sage. •  Collins, A. (1992). Portfolios for science education: Issues in purpose, structure, and authenticity. Science Education, 76(4), 451-463. • Van derSchaaf, M. F., & Stokking, K. M. (2008). Developing and validating a design for teacher portfolio assessment. Assessment and Education in Higher Education, 33(3), 245-262.

22. Email address for UNL portfolio • Theodore.bartholomew@huskers.unl.edu • mscheel2@unl.edu For requests of copies of the portfolio