Evaluating Youth Financial Capability Projects: Lessons from YouthSave

1. YouthSave Helping Youth Build a Tangible Stake in Their Future Evaluating Youth Financial Capability Projects: Lessons from YouthSave Global Youth Economic Opportunities Conference September 12, 2012 Mat Despard, University of North Carolina at Chapel Hill Lissa Johnson, Washington University at St. Louis Gina Chowa, University of North Carolina at Chapel Hill

2. About YouthSave • Supported by The MasterCard Foundation • Investigating savings accounts as a tool for youth development & financial inclusion in Ghana, Colombia, Nepal, & Kenya • Tailored, sustainable products & financial capability programming

3. YouthSave Learning Agenda

4. Exercise 1: Matching Methods to Questions

5. Matching Methods to Questions • Good evaluation starts with asking the right questions • There are questions other than “Did it work?” • Evaluation questions can be answered using different research methods • Don’t do an impact evaluation before doing process evaluation to pilot test your project

6. Matching Methods to Questions • Consider both quantitative & qualitative (mixed methods) • Qualitative methods provide information on complex phenomena that cannot be captured using quantitative methods

7. Process Evaluation • Very important for assessing fidelity - whether your project is being implemented as intended • Use as an opportunity to “detect and correct” implementation problems, but also to discover bigger, underlying problems

8. Impact Evaluation • Consider use of a control (randomized assignment) or comparison group with a pre/post test design • Waitlist control group – good option when you wish not to deny assistance and you have resources to help only some youth • Assess outcomes based on dosage – level of exposure to the project

9. Savings Demand Assessment (SDA) Research question: Which youth, household, and saving product characteristics are associated with positive savings outcomes? Variables: Youth and household demographics Account information and savings

10. Exercise 2: Savings Demand Assessment

11. Youth Demographics and Program-Related Questions • Did the banks attract the client population? Included: V 2,3,6,7,12 Excluded: V 4,5,8,9,10 Variation on V 11 • What is account holder’s level of financial capability? Included V: 13 Excluded: V 12,14,15 • Are the youth earning any income (that could be deposited in account)? • Why did the youth open the account? (what are they saving for?) • How did the youth learn about the account? (important marketing question)

12. Savings and Savings Product-Related Questions • Are the youth making deposits? (How much, frequency, long term saving or withdrawals, saving trends) • Which youth are saving more (e.g. age, gender, schooling)? • Which household characteristics are associated with higher saving (e.g. HOH education, income, assets)? • Does access to deposit locations (school, branch near home) affect savings performance? • Does participation in financial education workshops or text messages affect savings performance?

13. Data Collection: Challenges and Strategies • Who completes SDA questionnaire to ensure accuracy (bank, youth, parent)? Strategy: Test instrument for understanding; train bank staff to assist clients • How can we ensure questions answered and time to answer them? Strategy: MIS, bank staff training, include questions in account opening • How can we reduce inconsistencies in demographic and transaction data? Strategy: MIS; review meaning of variables

14. Regulatory and Reporting Challenges • Proof of citizenship (or know Id) • Colombia - legislation requirements for youth’s participation: parents signature and legal document proving parenthood • Nepal - Approval of Aggregation Point Banking (and tracking) • Reporting - study will not match bank reporting

15. Exercise 3: Asking good questions about financial capability

16. Survey Questions • Shorter is better; use clear language • Do not assume youth will know what you mean by “save” and “budget” • Offer response choices that will give you useful information • Train interviewers – how to ask questions without biasing results

17. Survey Questions • Note if youth is unable to understand questions and/or is uncooperative • Include interviewer prompts in your instrument F1. How much money do you have that belongs to you? NOTE TO INTERVIEWER: Prompt respondent to think of all of the money they have, including pocket change, chop money, money she or he has have saved or set aside for safe-keeping, etc. ____________________________GHC

18. Thank You! Mat Despard despard@unc.edu Lissa Johnson ejohnson@gwbmail.wustl.edu Gina Chowa chowa@email.unc.edu