1 / 12

EVALUATION RIGOR: IMPACT DATA

This article explores the need for a new measure of impact for Agricultural Extension programs, focusing on the complexity of behavior change and the antecedents of change. It proposes standardized change indicators as a way to assess the effectiveness of educational interventions in motivating behavioral change. Evaluation is essential for protecting the credibility of Extension programs and demonstrating their worth to society.

mwong
Download Presentation

EVALUATION RIGOR: IMPACT DATA

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. EVALUATION RIGOR: IMPACT DATA Lendel K. Narine, Ph.D. Evaluation Specialist

  2. UNSTANDARDIZED IMPACT • A common measure of Agricultural Extension impact used in many developing countries 100 years ago: AKAP • Awareness • Knowledge • Adoption • Productivity Change • Unlike extension systems around the world, CES is unique to the US • Last 100 years: Land grant goals shifted from technology transfer to human development • Extension now deals with all aspects of human resource development • Can AKAP measure the impact of CES?

  3. Rethinking Impact Does knowledge always lead to adoption? • Examples: • Does knowledge of opioid addiction prevent addiction? • Does knowledge of healthy eating practices lead to healthy eating?

  4. PLANNED CHANGE • After many failed Extension programs (i.e. knowledge gain with no impact) researchers agree on the complexity of change • Rogers (2003) Diffusion of Innovation theory served as a model for Extension in many countries • This theory captures the nuanced nature of change • What we know: Humans are inherently resistant to change • Why is this important? Extension programs are educational interventions aimed at motivating behavioral change • Evaluations should measure CHANGE caused by our educational interventions

  5. How? • What causes behavior change? • How can we measure change? • How can we attribute the cause of change to our program? • These questions are paramount to showing program impact • There’s no impact without change

  6. Reality Check: Logic Models without Logic To claim this We measure this

  7. Evaluation of Planned Change • All Extension programs are focused on change, whether we know it or not • A program is effective only when planned change occur: Our evaluation indicators should be factors of change • Every program is different because it targets different types of change: • An opioid program targets reduction in addiction – What social and psychological changes must happen before this reduction? • An after-school STEM program targets youth interest in STEM careers – How can motivate youth to enter STEM careers? • The standardized impact of Extension is Change, but different Antecedentsof change complicates standardization • Still, we can try!

  8. A Step Forward: Standardized Change Indicators: • Self-efficacy, includes knowledge (Bandura, 1989) • Knowledge, skills, attitudes, aspiration (TOP: Bennett & Rockwell, 2004) • Interest, motivation, expectation (Expectancy-Value: Atkinson, 1957) • Behavioral beliefs/perceptions, norms, barriers (TPB: Azjen, 2002) • Relative advantage, compatibility, complexity, observability, trialability (DOI: Rogers, 2003)

  9. Why • Value/Worth/Funding • Measure the antecedents of change to show an accurate reflection of our program’s worth to society • Future effort • We can assume adoption IF selected antecedents are present - No follow-up need for smaller programs • Publish • Publish your results in a reputable journal • Lead • Lead the Western Region

  10. Webinar: UMN Extension • Evaluation is vital because it protects the integrity of the program delivery organization • External stakeholders • Did not necessarily want to see the evaluations • Wanted to know Extension programs were evaluated and that we know our audiences well • Awareness of a solid evaluation protected credibility

  11. What’s Next? • One-on-one support: Talking Evaluation with Dr. L • Webinars: Monthly • Targeted Professional Development • Statewide Needs Assessment • Evaluation website for educational content • Newsletter/Publication

  12. Thank You!

More Related