On the Methodology of Measuring the Effectiveness of Financial Education Programs

1. On the Methodology of Measuring the Effectiveness of Financial Education Programs Jeanne Hogarth Dan Gorin Casey Bell Consumer & Community Affairs Federal Reserve Board The analysis and conclusions set forth in this presentation represent the work of the author and do not indicate concurrence of the Federal Reserve Board, the Federal Reserve Banks, or their staff. Mention or display of a trademark, proprietary product, or firm in the presentation by the author does not constitute an endorsement or criticism by the Federal Reserve.

2. Critiques of Studies • No control group • No longitudinal studies • Sensitive to stages of behavior change • Multiple measures of outcomes • Behaviors, attitudes, satisfaction

3. Federal Reserve System efforts • Three major efforts at evaluation of financial education are in at various stages • Federal Reserve Board - Financial management course at Ft. Bliss • Philadelphia Reserve Bank (RB) – Homeownership Counseling study • Kansas City RB – Workplace Fin. Ed.

4. Federal Reserve System efforts • Philadelphia Reserve Bank longitudinal study of the effectiveness of homeownership counseling • $2 million study • 1,000 participants randomly assigned to two different levels of training • To be tracked for 5 years including tax returns and credit bureau records • Have enrolled over 600 families

5. Federal Reserve System efforts • Kansas City Reserve Bank evaluation of Workplace Counseling • Working with Wadell & Reed Financial Advisors • Access to large amount of proprietary data on employee responses to financial education efforts in the workplace

6. Financial Management Behaviors of Military Personnel Project • Addresses issues of control group and longitudinal nature of studies • Tracks changes in stages and behaviors • Includes multiple measures • Behaviors, attitudes, satisfaction

7. Project Timeline • 2002 – MOU with Defense • 2003 – identified AER as collaborator • 2004 – obtained funding and contractor • 2005 – developed survey and web site • 2006 – started paper surveys implemented 2nd round online surveys

8. Project Timeline • 2007 – started paper surveys for 2nd round • 2008 – on-post “umbrella week” paper survey • Some 2nd round surveys, some new baseline data

9. Lessons Learned • Be patient, but not too patient • Have back-up plan(s) • Communicate your priorities • Find an advocate • Follow up

10. Baseline Data • Ages 19-25 • 3/4ths are E1 or E2 ($1,350 - $1,500/mo) • 2/3rds have HS or GED • Race/ethnicity same as general population • Financial ed course vs. “basic training” • All have bank accounts

11. Baseline Data • Collected at time of course (financial education group) or “in-processing” at ACS orientation (control group) • Identify baseline differences between 2 groups • ~ 680 in control group, ~2625 in education group, ~95 time series obs.

12. Questions • Demographics • Pre-military history/experiences • Financial products • Current financial standing • Financial behaviors • Financial activities • Self-assessment – stress, skills, situation • Additional financial education or training

13. Good News

14. “Good” Financial Behaviors

15. Bad News

16. “Bad” Financial Behaviors

17. Formal Education & Behavior

18. Income & Behaviors

19. Early Exposure to Finances

20. Marital Status and Behaviors

21. Family Influences on Good Behaviors

22. What’s Next? • Comparisons • Education vs control • Reserve & National Guard vs others • “in-processing” control group vs umbrella week control group • 1-day education group vs 2-day education group • Time series • Changes over time