Investigating the Impact of LMI

Investigating the Impact of LMI Research Team Canadian Research Working Group in Evidence-Based Practice (CRWG), Canadian Career Development Foundation (CCDF) In partnership with New Brunswick Post-Secondary Education & Labour Saskatchewan Advanced Education, Employment & Immigration

Overview Provide background on the project Share results Hear the perspectives of the research partners Consider next research steps

Our Research: An Important Step Forward • Most LMI research focuses on usability of products • Readability, accuracy of information • Easy to access, amount of use • Most research is with students: very little with adults • little literature about the direct contribution of LMISavard & Michaud, The Impact of LMI on Career Decision-Making Process: Literature Review, FLMM, 2005 • Several questions remain unanswered • How do people use LMI? • What (if any) assistance would be helpful?

Research Questions If client needs are assessed and clients are given LMI consistent with their needs, • To what extent does assistance by a service provider enhance their effective use of LMI? + • To what extent is independent self-help a sufficient process for clients to use LMI effectively?

Our Approach We based our experimental process on the existing service delivery practices used by the participating agencies so the processes could be incorporated into daily practice if research results proved positive. We used real frontline counsellors with real “adult” clients in their customary settings – not counselling interns and students in an academic setting. We delivered interventions that isolated the effects of LMI on decision-making – separating LMI from other interventions such as career counselling or job search workshops that use LMI as part of a larger process.

Method • Intervention • Independent use • Assisted use • 4 weeks of intervention • AIS at weeks 1 & 3 • Review of current practices • Preparation of “guided” LMI packages • Career Decision Making: • Know yourself • Know the Labour Market • Put it all Together • Job Search: • Check for “Fit” • Get Ready • Search for Work • Get a Job

E M P L O Y A B I L I T Y D I M E N S I O N S CCDF, 2010

Evidence-based Outcome-focused Practice InputProcessOutcome Need to link process with outcome Framework developed by the Canadian Research Working Group on Evidence-Based Practice in Career Development

Assessment as Decision Making (vs. Judgement) Please use a two-step process • Would you say that your level of mastery of the attribute under considerations is • Then assign the appropriate rating • 0 = really quite poor • 1 = just about OK, but not quite • 2 = OK, but just barely • 4 = really very good • 3 = in between barely OK and really good unacceptable acceptable 0 1 2 3 4 0 4

Problem with skill self-assessment • Participants asked to rate their skill (or knowledge) before and after a program • Often, pre-workshop scores are high and post-workshop scores are lower • People find out as a result of the workshop that they knew less than they thought or had less skill than they thought • Based on the new awareness, post-scores are lower • Our study confirmed this to be true • People don’t know what they don’t know • How can we get around this problem?

unacceptable acceptable 0 1 2 3 4 Assessing Learning & Attribute Outcomes Post-Pre Assessment We would like you to compare yourself now and before the workshop. Knowing what you know now, how would you rate yourself before the workshop, and how would you rate yourself now? Please use a two-step process: • Decide whether the characteristicin question is acceptableor unacceptable, then • assign the appropriate rating 0 4

Sample of learning outcomes • On these 6 items (K = 1, 2, 3, 5; S = 4; PA = 6) • Pre: 402 Unacceptable Ratings – Post: 35 Unacceptable Ratings • Unacceptable Ratings decreased from 45% to 4% • Pre: 56 Exceptional Ratings – Post: 343 Exceptional Ratings • Exceptional Ratings increased from 6% to 38% of the participants • Pre: 80% of means were Not OK – Post: all means were more than min OK

Descriptive Results Looking at all 14 survey items Before the intervention: 45% of responses were NOT OK Afterthe intervention: 5% of responses were NOT OK. OR Beforethe intervention: 55% of responses were OK Afterthe intervention: 95% of responses were OK AND Beforethe intervention: 5% of responses were Exceptional Afterthe intervention: 39% of responses were Exceptional

Descriptive Results (Continued) • All Before Program mean scores were in the Not OK range, with 1 exception (2.05). • All After Program mean scores were greater than minimally OK (mean score 3 or greater) with 2 exceptions (2.19 & 2.96) • The amount of change was similar across all three dimensions of the survey • knowledge, skills, and personal attributes all demonstrated about the same amount of change.

Descriptive Results (Continued) • Of particular note are items that suggest increased ability to self-manage their careers, such as: • A clear understanding of what I need to do to move forward in my career. • A clear vision of what I want in my career future. • Knowledge of print and online resources that help me to research career/employment options. • The ability to access career resources that can help me implement my career vision. • Effective strategies for keeping myself motivated to achieve my career/employment goals. • A realistic action plan (or schedule) summarizing the main career/employment-related activities I want to pursue and the processes I am engaging in. • Confidence in your ability to manage future career transitions. • Confidence in my ability to research career, employment, and training options that are available

Differential Results-Total Score • For group as a whole: • significant increase in overall ability to use LMI • neither intervention was more conducive to one manner of delivery compared to the other • Both CDM and JS groups had significant increases across time • Change in CDM group was significantly larger than in JS group • Participants in the JS group had higher scores than participants in the CDM group, likely indicating that JS participants were more familiar with using LMI before the project began. • Participants receiving assistance demonstrated greater change across time than did those in the independent modeSimilar pattern for all subscales

Other Noteworthy Results • No significant differences for • Gender: Women and men responded equally well to all intervention-delivery combinations • Work history • Unemployment history • Significant age-by-time interaction effect • A differential effect for different ages groups across time. • change across time for all age groups was statistically significant • participants 25 years or younger and those ages 41-50 reported greater change than those 26-30, 36-40, and 51 years or older

Attribution for Change To what extent would you say that any changes in the ratings on the previous pages are a result of your participation in this research project, and to what extent were they a function of other factors in your life? Program

Impact Outcomes Employment status Before 13 working full time, 22 working part time 6 = good fit, 11 = OK fit, 18 = poor fit After 50 working 24 = good fit, 11 = OK fit, 15= poor fit 43% increase in employment 3-fold increase in “good fit” (17% vs. 48%)

Data from AIS Interview Checklist • Emphasis of service providers was on establishing a relationship, summarizing progress to date, and reviewing action plans for next steps • 95% of service providers reported doing this • one-third to one-half reported engaging in behaviours intended to help clients: • clarify their interpretation of the LMI they accessed • find other related LMI resources • apply the LMI to their own personal situations

Client LMI Use Checklist • Clients were very engaged in the intervention process • Clients accessed 1-20 different resources and accessed those resources 2-158 times • On average, clients accessed 10 different resources • Total number of access points was 26 times for CDM clients and 41 times for JS clients • The most common sources of LMI for CDM clients • Career Cruising & accessed by more than 85% of clients • National Job Bank, accessed by more than 85% of clients • provincial job banks, Job Futures, and the information binder were accessed by more than 70% of clients • The most common sources of LMI for JS clients • National Job Bank, provincial job banks, and Monster.ca, accessed by 60% of clients • Career Beacon, Career Cruising, or Career Owl, accessed by 50% of clients • Government of Canada LMI, the intervention binder, or otherresources in the Career Resource Centre, 40% of clients

LMI: General Summary of Results • All intervention-delivery combinations produced significant change • General ability to access and use LMI • Knowledge about how to use LMI • Skills for using LMI and taking action • Personal attributes, e.g., optimism, confidence, and by inference, motivation • Assisted use produced greater change across time than independent use • 80% of clients attribute change to the program and not other factors

Linking Intervention with outcome Inputs Resources available Staff time Resource Centres Self-selected on-line resources Needs determination protocol Tailored LMI packages Processes Activities of counsellors and clients Process in LMI Packages Protocol for assisted self-help Counsellor check lists Client checklists • Outcomes • Indicators of client change • Learning outcomes • Knowledge and skills about using LMI • Personal attribute outcomes • Changes in attitudes, confidence, optimism, etc. • Impact outcomes • employment status • Resources were created • Processes were followed • Outcomes were obtained

What have we learned?(From the Research Partners) • Structure and checklists are foreign at first • But help them to be more focused later • Service providers are willing research partners • Most said they would do it again if given the opportunity • Service providers are happy to follow procedures that result in meaningful evidence of client change

What have we learned?(From the clients) Tailored LMI embedded in a learning process results in knowledge and skill acquisition as well as the capacity for self-management LMI appropriate for a client’s specific need (opposed to general LMI) appears to support engagement and action For many clients, a little (or no) professional support is enough Structure and timelines appear to motivate action and create a sense of progress Giving clients hands-on tools appears to motivate more than money

What helped make the project work The leadership of the provinces was exceptionally strong (make sure managers are included) The service providers caught the spirit of what we were trying to do and went with it, even though it might not be an exact fit for them The front-end screening (service needs interview) was well done, and permitted a strong match between client need and intervention focus The tailored intervention packages were superb The attention to detail in data collection was very well done

Accountability Common Perspective Traditional focus was on: • Global outcomes that are influenced by many factors • Employment status • Enrolment in training • Easy to measure variables that don’t address client change • Client flow • Ease of access • Counsellor time use Guiding question was: Are the outcomes (results, effects, etc.) promised, achieved in practice?

Accountability Re-Framed Need to re-frame the focus in accountability … Are the indicators achievable, realistic, and relevant? What factors contribute to the outcomes? What interventions make the outcomes more likely to be achieved? How do we know that the outcomes are a result of the interventions? Central focus is on: creating a link between client change and factors responsible for client change

Investigating the Impact of LMI