Evaluation Based Program Development

1. Evaluation Based Program Development This material was developed in partnership with Rutgers's University and Lydia B. Blalock, Youth Development Specialist, New Brunswick, New Jersey

2. Why Evaluate? • I dunno • I think I knew why–once • They make me do it • Department chair • Funding organization • My mom • I don’t • Demonstrate success • Understand failure • Provide feedback for program improvement • Process • Outcome • Nip problems in the bud

3. Process Objectives If I: • Develop and presentseminar on EBPD • Develop and distribute appropriate handouts • Engage participants in group activities • Provide participants an opportunityto practice • Encouragequestions THEN

4. Outcome Objectives Participants will: • Program Development • Understand how to clearly define issue program will address. • Use program theory to plan activities. • Write clear outcome and process objectives. • Use EBPD Model to plan programs. • Program Evaluation • Develop skills to evaluate programs. • Select appropriate evaluation methods and designs. • Select criteria to evaluate objectives. • Use EBPD Model to plan evaluation.

5. BUT FIRST: Is it a Program or a program • Stakeholder leadership • Faculty leadership • Multi-disciplinary • Sustainable • Academic department • Student component • Research, teaching, outreach

6. The Steps of Evaluation-Based Program Development

7. Step 1: Define the Problem • Describe Current Situation • How does the world look today? • Why a Problem • Clearly articulate why a problem • Ask why until get to root of problem • Symptom vs. Cause • Identify Stakeholders • Population: Who is affected? How many? • Stakeholder Leadership: interest in solving problem? • Potential Collaborators: Already working on problem? • How: Evidence? Learn more? References & Facts • Learn all you can about the issue/problem –

8. Set Priorities • Factors that influence priorities • Difference between actual and target states • Degree of difficulty in addressing needs • Cost of implementing solutions • Political and other factors • Derive priorities from several data sources • Priorities without resources • Select Solution Strategies to Meet Needs • Consider alternative solutions for each need • Evaluate alternatives • Select solutions to implement • Scholarship Opportunities?

10. TALK IT OVER Fill Out Table 1 with your group.

11. Step 2: Develop Outcome Objectives • Describe Ideal Situation • How would world look w/o problem (perfect world)? • Define Target Audience/s • Who/what can we change (youth, families, policies, etc.)? • What do we know about culture/s of audience? • Identify Outcome Objectives • What changes can I influence in my target audience/s? • Identify Resources • What do we need? • What do we have?

12. Step 2. Develop Outcome Objectives Ï

13. Writing Measurable Objectives • Specific • Will increase knowledge vs. Will know 3 sources of Vitamin C • Time delineated • At the end of the program, in 6 weeks, etc. • Measurable • 75%, all, etc. • Example • Participants will increase knowledge of nutrition. • 75% of program participants will know 3 sources of Vitamin C 6 weeks after the program.

14. Table 2: Clarifying Your Projected Impact

15. Talk it Over Fill out Table 2 with your group. #4: Brainstorm Resources

16. Step 3. Plan Program Use “Program Theory” to develop process objectives! IF:THEN

17. Program Theory: The Key  to Unlocking  Change      

18. Output the Process of an Outcome …wha…? Outcome Objectives: the changes you will influence in your targetaudience (Learning, Action, Condition) Process Objectives: the events, programs, activities you will conduct, deliver, develop, train,provide, access, facilitate to reach your outcome objectives Program Outputs: your deliverables Outcome & Process Objective Activity

19. Talk it Over Fill out table 3 with your group

20. SO WHAT? Evaluation Friend to Groucho Marx: “Life is difficult!”  Marx to Friend: “Compared to what?”

21. Why We Don’t Evaluate • Too hard • Not qualified • Costs too much • Takes too much time • Don’t think about it “in time” • No one ever reads evaluation reports • Everyone “knows” it’s a good program

22. Step 4: Program Evaluation Select Evaluation Design Select Data Collection Method

23. One-Shot Data collected following program, Retrospective Pretest Data collected following program; participants recall behavior prior to and following program. One Group Pretest-Posttest - Data collected prior to and following program Time Series Data collected prior to, during, after program Evaluation Design 101 • Pretest-Posttest Control-Group • Data collected from 2 different groups prior toandfollowing program. • Treatment group: program • Control group: no program • Post-Only Control-Group • Data from 2 different groups collected following program only

24. Evaluation Design 101

25. Data Collection Methods • Face to Face Surveys • Respondent completes survey in presence of evaluator • Evaluator may record responses and is able to answer questions • Self Administered Questionnaires • Respondent given survey to complete on his/her own. • Respondent’s responsibility to return survey. • Personal Interviews • Respondent asked pre-determined questions. • Evaluator records responses, can ask respondent to clarify answers. • Focus Groups • Several respondents meet with evaluator. • Evaluator facilitates discussion around pre-determined questions. • Discussion recorded.

26. Learned Anything? Let’s Check! Surprise Exam!!!

27. Learning, Action & Condition Objectives

28. Think it Over Fill out Table 4 with your group. Learning Indicators Action Indicators Condition Indicators

29. Cultural Aspects of EBPD • What is culture? • “Behaviors and beliefs characteristic of a specific social, ethnic, or age group.” • We are a product of our culture • Culture influences (unconsciously) what we think of people from other cultures. • Recognize cultural biases towards participants and strive to eliminate them. • Be aware of how differences between cultures may impact EBPD • Neutralize cultural biases and prevent EBPD design flaws: • Know your audience (Step 2 – Develop Outcome Objects) • Develop cross-cultural efficacy!

30. Culturally Sensitive Design

31. Reporting & Scholarship

33. Process Objectives If I, as facilitator: • Develop and presentseminar on EBPD • Develop and distribute appropriate handouts • Engage participants in group activities • Provide participants an opportunityto practice • Encouragequestions THEN

34. Outcome Objectives Participants will: • Program Development • Understand how to clearly define issue program will address. • Use program theory to plan activities. • Write clear outcome and process objectives. • Use EBPD Model to plan programs. • Program Evaluation • Develop skills to evaluate programs. • Select appropriate evaluation methods and designs. • Select criteria to evaluate objectives. • Use EBPD Model to plan evaluation.

35. Condition Changes We’ll report stunning impacts from well planned programs And the folks in Pullman will love us the most And people will dump bucket loads of money into our programs And the world will be a better place!