1 / 21

Sharing Lessons on Learning Metrics

Sharing Lessons on Learning Metrics. LINGOs Annual Meeting Sept. 23-24, 2009 Steven Honeyman and Marie-Laure Curie Global Capacity Building Department. Let’s start with small group discussions. Briefly describe your learning metrics (overview only)

jakeem-adams
Download Presentation

Sharing Lessons on Learning Metrics

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Sharing Lessons on Learning Metrics LINGOs Annual Meeting Sept. 23-24, 2009 Steven Honeyman and Marie-Laure Curie Global Capacity Building Department

  2. Let’s start with small group discussions • Briefly describe your learning metrics (overview only) • Discuss advantages and disadvantages of different components • Summarize what the group thinks are some new or interesting ideas the whole group should know about LINGOs Annual Meeting - Sept. 23-24, 2009

  3. Kirkpatrick’s model Includes four levels/steps of outcome evaluation: • Level 1—Reaction – satisfaction survey • Level 2—Learning – tests • Level 3—Behavior – on-the job behavior change survey • Level 4—Results – business outcomes LINGOs Annual Meeting - Sept. 23-24, 2009

  4. Measuring Training Impact at PSI SMILE - Social Marketing Initiative for Learning and Excellence

  5. Introduction to the PSI Behavior Change Framework • Behavior change framework drives PSI marketing activities • Fundamental to social marketing • Applicable to all behaviors promoted by PSI • Based on behavior change theory • Determinants measured through TRaC (Tracking Results Continuously) survey LINGOs Annual Meeting - Sept. 23-24, 2009

  6. The overall framework IMPROVED HEALTH STATUS/QUALITY OF LIFE RISK REDUCING BEHAVIOR, PRODUCT OR SERVICE USE AT RISK POPULATIONS OPPORTUNITY ABILITY MOTIVATION Availability Brand Appeal Knowledge Attitudes Belief Outcome expectation Brand Attributes Quality of Care Social Support Intentions Locus of control Social Norm Self Efficacy Subjective norm Threat Willingness to pay POPULATION CHARACTERISTICS SOCIAL MARKETING INTERVENTION PRODUCT PRICE PLACE PROMOTION LINGOs Annual Meeting - Sept. 23-24, 2009

  7. Our sphere of influence What PSI does… IMPROVED HEALTH STATUS/QUALITY OF LIFE RISK REDUCING BEHAVIOR, PRODUCT OR SERVICE USE AT RISK POPULATIONS OPPORTUNITY ABILITY MOTIVATION Availability Brand Appeal Knowledge Attitudes Belief Outcome expectation Brand Attributes Quality of Care Social Support Intentions Locus of control Social Norm Self Efficacy Subjective norm Threat Willingness to pay POPULATION CHARACTERISTICS SOCIAL MARKETING INTERVENTION PRODUCT PRICE PLACE PROMOTION LINGOs Annual Meeting - Sept. 23-24, 2009

  8. What we try to influence… IMPROVED HEALTH STATUS/QUALITY OF LIFE RISK REDUCING BEHAVIOR, PRODUCT OR SERVICE USE AT RISK POPULATIONS OPPORTUNITY ABILITY MOTIVATION Availability Brand Appeal Knowledge Attitudes Belief Outcome expectation Brand Attributes Quality of Care Social Support Intentions Locus of control Social Norm Self Efficacy Subjective norm Threat Willingness to pay POPULATION CHARACTERISTICS SOCIAL MARKETING INTERVENTION PRODUCT PRICE PLACE PROMOTION LINGOs Annual Meeting - Sept. 23-24, 2009

  9. To change a behavior... IMPROVED HEALTH STATUS/QUALITY OF LIFE RISK REDUCING BEHAVIOR, PRODUCT OR SERVICE USE AT RISK POPULATIONS OPPORTUNITY ABILITY MOTIVATION Availability Brand Appeal Knowledge Attitudes Belief Outcome expectation Brand Attributes Quality of Care Social Support Intentions Locus of control Social Norm Self Efficacy Subjective norm Threat Willingness to pay POPULATION CHARACTERISTICS SOCIAL MARKETING INTERVENTION PRODUCT PRICE PLACE PROMOTION LINGOs Annual Meeting - Sept. 23-24, 2009

  10. That improves health IMPROVED HEALTH STATUS/QUALITY OF LIFE RISK REDUCING BEHAVIOR, PRODUCT OR SERVICE USE AT RISK POPULATIONS OPPORTUNITY ABILITY MOTIVATION Availability Brand Appeal Knowledge Attitudes Belief Outcome expectation Brand Attributes Quality of Care Social Support Intentions Locus of control Social Norm Self Efficacy Subjective norm Threat Willingness to pay POPULATION CHARACTERISTICS SOCIAL MARKETING INTERVENTION PRODUCT PRICE PLACE PROMOTION LINGOs Annual Meeting - Sept. 23-24, 2009

  11. OAM • Opportunity: institutional or structural factors that influence an individual’s chance to perform a promoted behavior. • Ability: an individual’s skills or proficiencies needed to perform a promoted behavior. • Motivation: an individual’s desire to perform a promoted behavior. LINGOs Annual Meeting - Sept. 23-24, 2009

  12. TRaC (Tracking Results Continuously)) • What is it? • Repetitive quantitative survey that measures behavior change of a specific behavior of a specific target group • Measure the effectiveness of project interventions and activities • Why do we use it? • To better design programs, to monitor program progress (ensure that the target population is adopting the promoted behavior and if needed, make adjustments) and evaluate how we did (target group and behavior) • Informs decision-maker where to invest more/fewer resources • Who does it? • All PSI programs • When does it occur? - Prior, during (sometimes often) and at the end of an intervention LINGOs Annual Meeting - Sept. 23-24, 2009

  13. The overall Capacity Building framework IMPROVED JOB PERFORMANCE AND OVERALL IMPACT OF PSI PROGRAMS USE OF SKILLS ACQUIRED AT THE TRAINING BY PSI STAFF OPPORTUNITY ABILITY MOTIVATION Availability Brand Appeal Knowledge Attitudes Belief Outcome expectation Brand Attributes Quality of Care Social Support Intentions Locus of control Social Norm Self Efficacy Subjective norm Threat Willingness to pay POPULATION CHARACTERISTICS CAPACITY BUILDING INTERVENTION LINGOs Annual Meeting - Sept. 23-24, 2009

  14. TRaC-T (Tracking Results Continuously for Training)) • What is it? • Short, quantitative survey • Measures application of training on-the-job • Why do we use it? • Programmatic check to ensure that the training had the intended impact of transferring needed skills/behaviors to staff and to identify gaps in training content/delivery • Informs CB decision-maker where to invest more/fewer resources • Who does it? • Global Capacity Building department (for virtual training as well) • When does it occur? • Approx. 6 to 8 weeks after the training ended LINGOs Annual Meeting - Sept. 23-24, 2009

  15. Kirkpatrick’s model Includes four levels/steps of outcome evaluation: • Level 1—Reaction – satisfaction survey • Level 2—Learning – tests • Level 3—Behavior – TRaC-T survey • Level 4—Results – business outcomes LINGOs Annual Meeting - Sept. 23-24, 2009

  16. TRaC-T survey (example) LINGOs Annual Meeting - Sept. 23-24, 2009

  17. LINGOs Annual Meeting - Sept. 23-24, 2009

  18. Some TRaC-T results TRaC-T Budget Review training - August 5 & 6 2009 LINGOs Annual Meeting - Sept. 23-24, 2009

  19. Some TRaC-T results LINGOs Annual Meeting - Sept. 23-24, 2009

  20. TRaC-T might have some limitations: • Self reported behavior, • Needs to be built into the training program, • Needs resources to develop and manage, • May not apply learning right away. But it has many potential benefits: • Demonstrates training effectiveness to senior managers, • Convinces supervisor of the value of a particular training, • Allows targeting of training resources to highly effective training, • Provides feedback to those leading training on whether they are being effective, and • Can be automatically set-up for online training ! LINGOs Annual Meeting - Sept. 23-24, 2009

  21. For more information: • www.psi.org/research/ • Marie-Laure Curie, Manager, GCB Dept., Mlcurie@psi.org • Steven Honeyman, Director, GCB Dept. Shoneyman@psiglobalcapacitybuilding.org LINGOs Annual Meeting - Sept. 23-24, 2009

More Related