Strategies for Evaluating Your Scholarship Program

1. Strategies for Evaluating Your Scholarship Program Irene F. Goodman, Ed.D., President, Goodman Research Group, Inc. Denise Callahan, Director of Scholarship Programs, The Ford Family Foundation NSPA Conference Connect Hour October 15, 2014

2. Goodman Research Group, Inc.

3. What brought you to this session? • Group discussion about the opportunities for and challenges of evaluating your scholarship program • What would you most like to know? • What is most useful to you? • How can evaluation help? • What you can start doing right away and what you can plan for in the long term Goodman Research Group, Inc.

4. Evaluation: An important and often neglected program component • Will compare list generated with three main questions that scholarship providers have often posed in the past: • What impact are we making? • What is the success rate of our participants? • What happens to our Scholars down the road? • Other questions generated by today’s group that can be addressed by evaluation Goodman Research Group, Inc.

5. Today’s Topics • Importance of evaluation • Considerations for evaluation • Needs assessment data are important • Goal setting crucial • Outcomes • Types of evaluation • Making links between activities and outcomes Goodman Research Group, Inc.

6. Evaluation is important because…. The trouble with not knowing what you want is that you don’t know what you’ve got when you get it.

7. Importance of Evaluation • Assists program staff in learning what is working and what isn’t • Decision making and performance management • Assists in obtaining funds • Provides accountability to board/funders/ other stakeholders Goodman Research Group, Inc.

8. Considerations for Evaluation • Is it worth evaluating? • Maturity, coherence of program • If so, what kind of evaluation should be done? • Formative testing of program component ideas? • Process evaluation or outcome evaluation or both? • Short term outcomes or long term outcomes? • Funding realities • Feasibility Goodman Research Group, Inc.

9. Different Types of Evaluation Goodman Research Group, Inc.

10. Goal Setting “Looks like the ghost of unarticulated goals has come back to haunt us.” Goodman Research Group, Inc.

11. Formative Evaluation No matter how far you have gone down the wrong road, turn back. Goodman Research Group, Inc.

12. Outcomes Evaluation The proof of the pudding is in the eating. Goodman Research Group, Inc.

13. Outcomes Questions to Ask • Ask yourself: What do I want to find out? What would success look like? • How will I know? (Indicators) • Produce a list and be specific • Get feedback and learn what else has been done • It’s okay that the questions you start with aren’t necessarily the questions you end with. Goodman Research Group, Inc.

14. Defining Impact • What impact do you wish to have? • What is realistic given your program activities? • Use “if/then” statements, e.g.: • “If we give $1,000 scholarships to 1,000 people then we expect X to happen.” • What is reasonable to expect in the short-term, in the mid-term, and in the long-term? Goodman Research Group, Inc.

15. Outcomes and Impact Goodman Research Group, Inc.

16. Logic Model INPUTS ACTIVITIES SHORT- TERM OUTCOMES IMPACT (LONG- TERM) OUTPUTS Goodman Research Group, Inc.

17. Group activity • Working on how to answer what you want to know Goodman Research Group, Inc.

18. Logic Model Elements and Structure Goodman Research Group, Inc.

19. From the Simple to the Complex • Tracking your participants – start with adding baseline questions to application • Retrospective survey of alumni • Following participants prospectively and longitudinally (baseline and annually) • More rigorous: comparing scholars with finalists Goodman Research Group, Inc.

20. Tracking is Extremely Important • Emails • Annual check-ins • Asking for preferred mode of communication • Birthday cards/postcards • Obtaining updated contact information • Using social media to reach out Goodman Research Group, Inc.

21. Retrospective Surveys of Alumni • Looking back to their experiences • Asking “retrospective pre” questions (When it has not been possible to get baseline data, these types of questions ask respondents to compare then to now.) Goodman Research Group, Inc.

22. Following participants prospectively and longitudinally • Starting when they have just started their scholarship (baseline) • Expectations • High school participation • Stress level and barriers • Comparing annual or so how their perspectives change • Can be a one-time snapshot or, if you have the luxury, annually Goodman Research Group, Inc.

23. More rigorous: Comparing scholars with finalists • This answers the counter-factual: How do scholars compare with similar students who did not receive your scholarship? • What might have happened/not happened if the students had not received the scholarship? • Must statistically control for group differences • Can also compare with state, regional, and national data Goodman Research Group, Inc.

24. Questions or comments? CONTACT US! Irene F. Goodman Goodman Research Group, Inc. www.grginc.com (617) 491-7033 goodman@grginc.com Denise Callahan The Ford Family Foundation www.tfff.org (541) 485-6211 dcallahan@tfff.org Goodman Research Group, Inc.