Career Development: What’s our Proof?

1. Career Development: What’s our Proof? Lynne Bezanson &Sareena Hopkins Canadian Career Development FoundationIAEVG Québec City June 2014

2. Evidence-Based Research in Canada • National research agenda to better understand “what works” in career services • 3 calls for proposals since 2004 • Source of major international envy among Career Development Practitioners and Researchers

3. Why Evidence-Based Practice? • Ethics: Clients deserve interventions that have demonstrated success • Confidence: Practitioners should know that a given intervention will result in a predictable outcome • Security:Being able to provide evidence of success increases ability to obtain and keep funding for services and secure jobs for CDPs/Employment Counsellors

4. Why Evidence-Based Practice? • Outcome-focused intervention • Evidence of client change …contrast with… • Client flow • Ease of access • Practitioner time use It’s easy to measure variables that don’t address client change

5. Why Evidence-Based Practice? • Some successful programs are discontinued because there are no data indicating client change • Client flow client change • There is a need for • Evidence of client change • Evidence of impact of service on… • Client • Society • Economy

7. Session Objectives • Highlight recent research projects and what has been learned (what evidence do we have?), with a focus on the tools used and their impact • Explore how tools used in the research could be used to support evidence based practice • Identify the key characteristics/practices of the Evidence-Based Practitioner • Identify action steps to infuse Evidence-Based Practice into work with clients, into professional growth and into office culture

8. What evidence do you currently collect? • What is done with the evidence currently collected?

9. Evidence-Base: A Research Question • 3 recent Canadian research studies • Front-line research conducted with employment offices in Alberta, Manitoba, Québec and New Brunswick • Consistent and strong results

10. Questions to keep in mind • What did the research do? • What did we learn? • What was most surprising? • What are the implications for practice?

11. Research Project #1: The Impact of Labour Market Information on career decision making Research Questions If client needs are assessed and clients are given LMI consistent with their needs, • To what extent does assistance by a service provider enhance their effective use of LMI? OR • To what extent is independent self-help a sufficient process for clients to use LMI effectively?

12. Method • We prepared “guided” LMI packages (take-home booklets and Resource Centre binders) on: • Career Decision Making: • Know yourself • Know the Labour Market • Put it all Together • Job Search: • Check for “Fit” • Get Ready • Search for Work • Get a Job

13. Intervention All participants in the study: • Received a needs assessment interview & completed an initial, pre-program survey • Received an LMI package specific to their identified employability need • Were randomly assigned to either a self- directed intervention delivery method or an assisted self-directed intervention delivery method. • Were given an orientation to the Resource Centre which they could freely use on their own • Completed a pre and then and now post survey • Received a cash honorarium and certificate of participation

14. Intervention (cont) The self-directed group: • worked independently for 3 weeks, making use of the materials and the Resource Centre • returned in Week 4 for their exit interview

15. Intervention (cont) The assisted self-help clients received: • two additional AIS (Advice and Information) interviews (20-30 minutes) in weeks 1 and 3 focused on helping them understand, interpret and apply the LMI to their own situations and /or access additional LMI • returned in Week 4 for their exit interview

16. What did we Measure • General ability to use LMI • Knowledge • Clear vision of what I want in my career future • Knowledge of print and online resources • Skill • Have effective strategies for keeping myself motivated • Have a realistic action plan • Personal Attributes • Optimism about what lies ahead re meeting my career goals • Confidence in my ability to manage future career transitions

17. Differential Results-Total Score • For group as a whole: • significant increase in overall ability to use LMI • neither intervention was more conducive to one manner of delivery compared to the other • Both CDM and JS groups had significant increases across time • Change in CDM group was significantly larger than in JS group • Participants in the JS group had higher scores than participants in the CDM group, likely indicating that JS participants were more familiar with using LMI before the project began. • Participants receiving assistance demonstrated greater change across time than did those in the independent modeSimilar pattern for all subscales

18. Attribution for Change To what extent would you say that any changes in the ratings on the previous pages are a result of your participation in this research project, and to what extent were they a function of other factors in your life? Program

19. Follow up interviews (1 week and 4 months after intervention) • Week 1: 103 clients • 4 months: 65 clients • LMI overall relevant • Clients overall optimistic and confident • Clients continued to use LMI but lessoften

20. What did clients say? Themes: • I was surprised at how much information there was • Initially I thought I could not do this; but I found out I could • I got more focused on my goal • I got clearer about what I wanted to do • There was too much information and that made it difficult….very hard to do this alone (Those who were working –85-90% responses in these categories) • Did not help very much (only from those still unemployed at 4 months)

21. What did practitioners say? • I never really thought about clients as assisted or self-help—now it is built into all my assessments • Now I give more homework and I am more specific. The clients felt more focused and so did I • I used to be more maternal and I got more involved than I needed to be. Now I ask clients to be more responsible • I found the checklists inhibiting and awkward

22. What did we learn? • What was most surprising? • What are the implications for practice?

23. Applications to Service Delivery • Tailored LMI embedded in a learning process results in knowledge and skill acquisition as well as the capacity for self-management • LMI appropriate for a client’s specific need (opposed to general LMI) appears to support engagement and action • For many clients, a little (or no) professional support is enough • Structure and timelines appear to motivate action and a sense of progress • Giving clients hands-on tools appears to motivate more than money

24. Tools/ Supports for the evidence-based practitioner • Initial learning needs survey • Checklist for Employment Assessment Interview • Advice and Information Interview Checklist • LMI Resource Guide for Career Decision Making • LMI Resource Guide for Job Search

25. Research Project #2:Assessing the Impact of Career Resource and Supports Across the Employability Dimensions • This project builds substantially on the LMI Impact study • Self-Help Index and Labour Market Attachment Index

26. This Study • Also began with a needs assessment, but expanded to 4 needs: • Career decision making • Skills Enhancement • Work Search • Job Maintenance (and pre-employability) • Each client received a tailored resource package, but well beyond LMI to include coaching activities to help them to reflect and personalize.

27. This Study • Clients randomly assigned to one of two treatment conditions: • Practitioner assessed (in which they had their needs assessed, were oriented to the appropriate Guide and then worked independently with their Resource Guide for 4 weeks) • Practitioner assessed and supported (in which they also worked with their Career Consultant for 4 weeks). •  Examined: • Differential impact across treatment conditions (independent/supported) • Differential impact across labourmarket attachment & self-help ability (2 new variables)

28. Research Question If clients are given a comprehensive needs assessment to determine their employability need(s), what is the differential effect of independent and consultant-supported career resources on clients who are weakly attached to the labour market versus those who are strongly attached to the labour market?

29. Results • After just 4 weeks of intervention: • Tenfold increase in competencies rated in acceptable range • Percentage of employed clients rose from 27% to 45% (a 69% increase) • 81% of participants reported that their employment was a “good fit” with their career vision (a 200% increase) • 91% attributed the positive changes either partly (40%) or mostly (51%) to the intervention • 92% had a clear plan for next steps • 98% planned to use the Guides again

30. Results • Statistically and clinically significant positive impact of career interventions across ALL sub-scores and across ALL employability dimensions – compelling evidence of positive changes in clients as a result of career interventions • Although see more positive trends for supported group, differences are not statistically significant

31. Results • Career practitioners care very much about their clients, so much so that they were reluctant to ask clients who might need support to join the study— “in case” they were randomly assigned to the independent group • Therefore the sample was too homogeneous to test properly the Self-help and Labour Market attachment indices • Career practitioners care very much about their clients…BUT…in the next study, they CANNOT!

32. What did we learn • What was most surprising? • What are the implications for practice?

33. What did the study tell us? • Career Development Interventions Work! • Clients demonstrated substantial positive changes in skills, knowledge, personal attributes, employment and fit of employment. • Many clients can benefit significantly from self-help resources when they are matched to their need and they are “launched” – moreso than we anticipated! • This is especially noteworthy given the short intervention period of 4 weeks and the “real-life” setting in which the changes occurred.

34. The Tools and Guides are “Ready to Go” • Virtually no adjustments are needed for use in typical employment centres across Canada. Self-help guides provided after a thorough needs assessment and orientation are effective and can be used as a first line of intervention, saving valuable practitioner time for clients who really need it, or potentially opening a promising perspective on online career services.This finding has considerable practical significance.

35. Employability Dimensions Study: Tools • Employment Goal and Action Planner • Client Checklists/Tracking Sheets (all employability dimensions) • Comprehensive Coaching Guides (all employability dimensions)

36. Research Project #3:Common Indicators • What had we learned? • Where did we need to go next?

37. Resources available • Client needs assessment • Client employment history • Self-help index • Labour market attachment index • Activities that link to outcomes or deliverables • Generic interventions • Working alliance, • Specific interventions • LMI Booklets • Advice and Information Interviews • Employability Dimension Coaching Guides • Practitioner Coaching using the Guides • External Referral PROCESSES INPUTS • Client • Context • Life vision • Needs • Goals OUTCOMES • Learning outcomes • Ability to use LMI resources alone and with assistance • Ability to use Coaching Resources alone and with assistance • Having an employment goal/vision • Skills in employability dimension indicators • Personal attribute outcomes • Changes in intrapersonal variables e.g., attitudes, self-esteem, motivation, etc. • Impact Outcomes • Changes in the client’s life resulting from application of learning

38. Unanswered Questions • We have data/evidence of positive impacts from specific interventions for certain kinds of clients • We do not have data/evidence of positive impacts from employment services overall • We do not have a data gathering tool that can be used to gather common data across divergent employment service settings

39. Therefore the next step:Common Indicators Project Key research questions: • What common indicators are applicable across different contexts, different client groups, different agencies and different interventions? • What statements of service effectiveness can be made by tracking common indicators? and

40. IF numbers permit…the ultimate question: • What kinds of interventions in what contexts produce what kinds of results?

41. How we investigated • Literature Review • Focus Groups • Compare the two • Accept those in common and most robust

42. Focus Group Questions: • If clients have benefitted from career and employment services, they have experienced some changes in in their lives. They are not in the same position they were when they started. What indicates that clients or their situations have changed as a result of career and employment services? • How do you know? What are your clues? • What do you do to bring about change? What influences your ability to do this? • What about clients? What influences their ability to change? • What factors are beyond both of you to control?

43. Activities that link to outcomes or deliverables • Generic interventions • Working alliance, client engagement. • Specific interventions • Needs Assessment • Employment Counselling • Resources/Guides • Goal setting and Action Planning • Programs & Workshops • Progress Tracking • External Referral • Resources available • Availability of and access to community resources Availability of and access to employment opportunities and training • Practitioner Background and Experience • Client Employment History and Potential • Client Life Circumstances PROCESSES INPUTS • Client • Context • Life vision • Needs • Goals OUTCOMES • client change • Learning outcomes • Changes in knowledge and skills linked to the program or intervention used • Progress Indicators End Result Indicators • Personal attribute outcomes • Changes in intrapersonal variables e.g., self-confidence, self-esteem etc.. • Progress Indicators and Impact Outcomes • Changes in the client’s life resulting from application of learning

44. Practitioner/Research Partner Tasks • Recruit clients into the study (NO criteria); • Follow all research protocols • Use PRIME each time you see a client to assess and record PROGRESS/CHANGE in: • Goals and Action Plans • Employability Learning Needs/Changes • Personal Attribute Needs/Changes • Life Circumstance Needs/Changes • Working Alliance and Client Engagement

45. Practitioner/Research Partner Tasks • Work with each client for 6 weeks following Assessment interview and client entry into the research • At exit interview also gather data on…

46. Practitioner/Research Partner Tasks • Labour Market Outcomes: • Employment or Training Status • Fit of Employment or Training with skills, qualifications, vision • Adequacy of salary • Linked to future opportunity • Ongoing Progress Evidence: Actions Underway to move forward: • Have clients complete then and now questionnaire

47. Pre-Employability/Job Readiness

48. Life Circumstances

49. Personal Attributes Example

50. Self-Esteem Attribute