610 likes | 817 Views
New Ways to Evaluate Counselling Effectiveness. Bryan Hiebert Adjunct Professor Department of Educational Psychology & Leadership Studies University of Victoria Professor Emeritus in Education University of Calgary. What is counselling all about?. Counselling or Counselling Psychology
E N D
New Ways to Evaluate Counselling Effectiveness Bryan Hiebert Adjunct Professor Department of Educational Psychology & Leadership Studies University of Victoria Professor Emeritus in Education University of Calgary
What is counsellingall about? Counselling or Counselling Psychology Clinical Psychology Social Work etc. • Write down the words or phrases that come to your mind • 1 minute
Overview Background An alternate to RTCs A framework for demonstrating value Alternate evidence gathering tools Purpose An alternate way of looking at demonstrating the value of human services Intended to stimulate discussion
Evaluation is important • But counsellors do not evaluate their work with clients • 1993 study • 40% never evaluated their work with clients • 35% evaluated with client during interview • 2005 study • 33% did not answer • 56% reported data • Client flow, counsellor time use, etc.
How to address the problem We need an approach that is: • Comprehensive enough to include what is needed • Simple enough for people to use • Incorporates evaluation into standard practice • Your own practice becomes your data source for predicting client outcomes • Each client is a n = 1 experiment • Across time and across clients, counsellors acquire ability to make predictions linking interventions & outcomes Local Clinical Scientist (Professional Practitioner)
Evidence-Based Outcome-Focused Practice InputProcessOutcome Resources Actions Client change Need to link process with outcome
Evidence-Based Outcome-Focused Practice InputProcessOutcome • Indicators of client change • 1. Learning outcomes • Knowledge and skills linked to intervention • 2. Personal attribute outcomes • Changes in attitudes • Intrapersonal variables (self-esteem, motivation, independence) • 3. Impact outcomes • Impact of #1 & #2 on client’s life, e.g., employment status, enrolled in training • Societal, economic, relational impact
Outcomes of Counselling • Client learning outcomes • Knowledge • Skills • Impact on client’s life • Client presenting problem • Economic factors • Third party factors +PrecursorsPersonal Attributes
Precursors Intervene between learning outcomes & impact outcomes • Attitude • Motivation • Self-esteem • Stress • Internal locus of control • Belief that change is possible Changes counsellors mentioned That were being observed but not reported
Evidence-based Outcome-focused Practice InputProcessOutcome • Activities that link to outputs or deliverables • Generic interventions • Working alliance, microskills, etc. • Specific interventions • 1. Interventions used byservice providers • Skills used by service providers • Home practice completed by students • 2.Programs offered by school • 3. Involvement by 3rd parties 4. Quality of service indicators • Stakeholder satisfaction, including students
Evidence-based Outcome-focused Practice InputProcessOutcome • Resources available • 1. Staff • Number of staff, level of training, type of training • 2. Funding • Budget • 3.Service guidelines • Agency mandate 4. Facilities 5. Infrastructure 6. Community resources
Outcome Focused Evidence-Based Practice InputProcessOutcome Need to link process with outcome • What will I do? • What are the expected client changes? • What do I expect clients to learn? • What sorts of personal attributes do I want my clients to acquire? • What will be the impact on their lives? • How will I tell?
Outcome Focused Evidence-Based Practice • Client • Context • Needs • Goals • Resources available • Context: Structure of opportunity • Staff: Number of staff, level of training, type of training • Funding: Budget • Service guidelines • Facilities • Infrastructure • Community resources • Activities that link to outcomes or deliverables • Generic interventions • Working alliance, microskills, etc. • Specific interventions • Strategies linked to specific client problems(stress, grief, depression, career, etc.) • Client home practice • Other • Programs & Workshops • Facilitation guides • Intervention manuals • External Referral PROCESSES INPUTS OUTCOMES • Indicators of client (Learner) change • Learning outcomes • Changes in knowledge and skills linked to the program or intervention used • Progress Indicators End Result Indicators • Personal attribute outcomes • Changes in intrapersonal variables e.g., attitudes, self-esteem, motivation, etc. • Progress Indicators End Result Indicators • Impact Outcomes • Changes in the client’s life resulting from application of learning
Evidence gathering • A Framework for evaluation • Learning from the evaluation experts • Gathering evidence of our impact • Documenting what works and why • Developing evidence gathering tools • Need to use informal evidence Some examples
Assessment as Decision Making (vs. Judgement) Please use a two-step process • Would you say that your level of mastery of the attribute under considerations is • Then assign the appropriate rating • 0 = really quite poor • 1 = just about OK, but not quite • 2 = OK, but just barely • 4 = really very good • 3 = in between barely OK and really good unacceptable acceptable 0 1 2 3 4 0 4
Problem with skill self-assessment • Participants asked to rate their skill (or knowledge) before and after a program • Often, pre-workshop scores are high and post-workshop scores are lower • People find out as a result of the workshop that they knew less than they thought or had less skill than they thought • Based on the new awareness, post-scores are lower • People don’t know what they don’t know • How can we get around this problem?
Assessing Learning & Attribute Outcomes unacceptable acceptable 0 1 2 3 4 Post-Pre Assessment • We would like you to compare yourself now and before the workshop. Knowing what you know now, how would you rate yourself before the workshop, and how would you rate yourself now? • Please use a two-step process: • Decide whether the characteristic in question is acceptable or unacceptable, then • assign the appropriate rating 0 4
Applied Career Transitions Program(on-line program for unemployed university grads) For Module 1 • All together there were 10 (items) x 29 (participants) = 290 ratings • Pre: 144 Unacceptable Ratings – Post: 3 Unacceptable Ratings • Unacceptable Ratings decreased from 50% to 14% • Pre: 6 Exceptional Ratings – Post:130 Exceptional Ratings • Exceptional Ratings increased from 2 to 44% of the participants
Results: Impact outcomes Module 1 • 23 out of 29 had found a job • 10 of the jobs lined up well with career vision Module 2 • 4 out of 6 had found a job • 3 of the jobs lined up well with career vision
Post-Pre Assessment: Bridging Health Care with Self-Care • Knowledge of stress and stress control • Knowledge of how to manage personal change • Knowledge of nutrition and nutrition control - - - - - - - - - - - - - - • Level of stress • Level of nutrition (high=healthy) • Level of fitness • Confidence in ability to manage personal change
Post-Pre ResultsBridging Health Care with Self-CareStress & Stress Control Start of Program End of Program Vertical axis indicates % of participants
Post-Pre ResultsBridging Health Care with Self-CareManage Personal Change Start of Program End of Program Vertical axis indicates % of participants
Attribution for Change • To what extent would you say that any changes depicted above were a function of participating in our program, and to what extent were they a function of other factors (e.g., job, personal circumstances, etc.)
Participant Comments This program made me realize that change is possible for me to make, and that even a small effort can bring significant benefits. When I joined the program, I thought that only a superhuman effort would result in any positive benefits, and that thought itself was a barrier to making a behavioural change. Now I know that “I can do it” and that knowledge itself is half the battle This is a fantastic program. The biggest benefit for me is that I am starting to see myself as being a more active person, rather than a “couch potato”. I appreciated the focus of the program being on life change and wise choices, rather than on diet and weight loss. I
Tracking client goal attainment • Clients are different • Live in different contexts • have different goals • require different interventions • How can we consolidate the results?
Goal Attainment Scaling Much better than expected A bit better than expected About what I expected A bit worse than expected Much worse than expected ObservableIndicators +2 +1 0 -1 -2
Thought Listing Stimulus question • For Clients: • What are the main factors contributing to your presenting problem? • What needs to happen in order for you to achieve your goals? • What are the barriers you need to overcome? • For counsellors: • What is counselling all about? • What factors are responsible for client change? • Write down all the words or phrases that come to mind in response to this question.
Cognitive Mapping Task • Complete the thought listing task • Transfer each item to a “Post-it” sticker, 1 item per sticker. • Arrange the stickers on a large sheet of graph paper • Place related concepts close to each other • Draw connecting lines between stickers that represent related concepts • Draw a circle around any clusters of concepts • label each circle
Cognitive Mapping-Example What is counselling all about? Scoring scheme: http://cmap.ihmc.us
Relaxation Monitoring Sheet Feelings, body sensations, etc. Thoughts, perceptions, imagery, etc. .
Informal Evaluation of Headache, Pain, and Related Affective States
Self-Monitoring Headache 0 - No headache 1 - Low level, only enters awareness when you think about it 2 - Aware of headache most of the time, but it can be ignored at times 3 - Painful headache, but still able to continue job 4 - Severe headache, difficult to concentrate with demanding tasks 5 - Intense incapacitating headache
Headache Monitoring Grid Level 5 4 3 2 1 0 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 01 02 03 04 05 (time of day) Before treatment After treatment
Self-Monitoring Stress 0 - No stress 1 - Low level, only enters awareness when you think about it 2 - Aware of stress level most of the time,but still able to stay focused on job 3 - Very stressful, beginning to interfere with what you are doing 4 - Severe stress, creates major interference with what you are doing 5 - The most stress you ever feel
Self-Monitoring Confidence 0 -No confidence at all in your ability to handle the situation 1 -Low confidence, but you might get lucky and it will turn out OK 2 - Sort of confident, you think you might be able to handle it, but probably not 3 -Pretty confident, you think you have a chance of this working out OK 4 -Very confident, you’ll probably be able to pull this off 5 -Completely confident about what you are doing
Self-Monitoring Job Satisfaction 0 -No satisfaction at all with your job 1 -Somewhat satisfied, there are some things about your job that are OK 2 -Satisfied enough – as jobs go, this one is as good as any 3 -Mostly satisfied, occasionally there are things about your job that you actually like 4 -Very satisfied, but there are occasional things bother you 5 -Completely satisfied with your job, feeling a sense of self-fulfillment
Self-Monitoring Motivation 5 - Very motivated-it's the most important thing for me to do today 4 - Very motivated, but something might come up to interfere 3 - Quite a bit motivated, I think I will end up doing some of it 2 - I sort of care and I might get around to doing some work today 1 - If I run out of things to do, I'll try looking in some of my books 0 - I couldn't care less if I ever did my school work
Using Self-Talk to Control Anger With cognitively-challenged adults in residential settings Hiebert, B., & Malcolm, D. (1988). Cognitive strategies for mentally handicapped clients. In B. O'Byrne, (Ed.). Natcon-14 (pp. 289-298). Toronto, On: Ontario College Counsellor's Association. (Reprinted in Resources In Education, June 1989, ERIC Document Reproduction Service No. Ed 302800)
Interview Protocol for Assessment • I am going to show you 3 pictures and ask you some questions about each picture. • Look at this picture. • Tell me how many people are in the picture. • What are the people doing in the picture? • Point to the person that is being pushed/kicked/hit • If you were the person being pushed/kicked/hit, how would you feel? • If you were the person being pushed/kicked/hit, what would you feel? • If you were the person being pushed/kicked/hit, what would you think? • If you were the person being pushed/kicked/hit, what would you do?
Frequency of Anger Outbursts 10 Mentally Handicapped Residents
Monitoring Your Self-Talk Con Pro
Changing Your Self-Talk Con Pro
Self-monitoring Stress Subjective Units of Disturbance