Beginning with the End in Mind: What We Know about Performance Measurement for VMF Programs

1. Beginning with the End in Mind: What We Know about Performance Measurement for VMF Programs If you have not already done so, please connect to the audio portion of this Webinar: 1. Dial 1-877-668-4490 2. Enter the access code 622 071 435 # 3. Enter your unique participant ID number displayed on your screen OR choose the option to let the program call you back!!!

2. Housekeeping • Phones are muted • Ask questions by sending a chat message to “All Participants” using the chat panel on the right side of your screen

3. Who’s on the Webinar 175 Attendees Representing Programs and Organizations from 44* States

4. Beginning with the End in Mind: What We Know about Performance Measurement for VMF Programs Outcomes You will: • Learn about Performance Measurement for Veteran and Military Family Programs (VMFP) • Discover methods to measure outputs and outcomes with available tools • Learn about VMFP data sources to help measure outputs and outcomes • Explore best practice examples in measuring outcomes from Equal Justice Works

5. Hello and Welcome from CNCS! • Koby Langley • Corporation for National and Community Service • Webinar hosts: Education Northwest

6. The Serve America Act’s Influence on CNCS VMF Performance Measures Employment *Service to VMF *Creating Service/ Volunteer Opportunities for VMF Employment Disaster Preparedness Youth Mentoring Access to Benefits Education Health Transportation Education and Certification Wellness and Other Support Services * Overarching activities of programs in SAA *Community Coordination Specific activities Mentioned in SAA

7. Session Presenters Dr. Chris Spera, CNCS Director of Research and Evaluation Kathryn Gravely, Program Manager, Federal Programs and Strategic Initiatives, Equal Justice Works Kerry O’Brien, Director of Federal Programs and Strategic Initiatives Equal Justice Works

8. Key National Service SAA Issues Areas by Program

10. Wellness/Other Supports and Services by Program (n=59)

11. Performance Measurement for Veteran and Military Family Programs • What is performance measurement? • A systematic process of tracking outputs and outcomes based on the goals of your program, as articulated in your program’s logic model • What is a logic model? • A logical chain of connectionsshowing what the program is to accomplish

12. Why is Performance Measurement important for CNCS? • Assess the individual and collective results of programs and continue to enhance program effectiveness • Expand CNCS’s ability to account for the combined contributions of all organizations making use of our resources

13. Why is Performance Measurement important for VMF Programs? • Assesses the individual and collective results of VMF programs by tracking outputs and outcomes related to improving • Military family strength and resilience • Quality of life for veterans, i.e.,: • Transition to civilian life • Education attainment • Job/vocational skills • Behavioral health

14. Performance Measurement for VMFP: Collecting Data on Outputs and Outcomes • Tools to measure outputs: • Intake forms, attendance and tracking sheets to keep count • Tools to measure outcomes: • Pre, mid-term and post program surveys designed to measure increases in specific areas over time, as a result of participation in the program, focus groups, interviews • Where to find instruments, tools and resources for performance measurement: • http://www.nationalservice.gov/resources/npm/home • search by focus area

15. Performance Measurement for Veteran and Military Family Programs • Important point: Performance measurement is NOT evaluation • “Unlike performance measures, evaluations estimate the impacts of programs by comparing the outcomes for individuals receiving a service or participating in a program to the outcomes for similar individuals not receiving a service or not participating in a program.” • Code of Federal Regulations for CNCS http://www.gpo.gov/fdsys/pkg/CFR-2010-title45-vol4/pdf/CFR-2010-title45-vol4-part2522-subpartE-subjectgroup-id1784.pdf

16. Performance Measurement for Veteran and Military Family Programs • Performance measures track outputs and outcomes based on the goals of your program’s logic model • A logic model shows the chain of connections illustrating plans to meet community needs; the theory of change Need: access to employment resources INPUTS OUTCOMES OUTPUTS Program investments Activities Participation Medium Long-term Short What we invest: People, time Service provided: # of in-take & referrals Service recipients: # of returning veterans receiving referrals What results/changes: Veterans placed in employment prep programs; then employed

17. Outputs for Veteran and Military Family Programs • Defining outputs • Amount of service provided (e.g., people served, products created, or programs developed) through the service intervention of your program • Common outputs for VMF programs • # of veterans or active Service members and families served • # of veterans or active Service members engaged in service provision through CNCS programs • Others?

18. Outputs for Veteran and Military Family Programs – Program Examples • Veterans Helping Veterans Now • # of veterans referred to service providers that can meet their needs • Metro Community College, Omaha, Nebraska • # of people who received outreach from VISTAs regarding support needed by veterans returning to college • CA Conservation Corps • # of veterans employed as crew members

19. Outcomes for Veteran and Military Family Programs • Defining outcomes • Reflect the changes in attitudes/beliefs, knowledge/skills, behavior, or conditions of individuals that occur in the organizations, communities, or environment as the result of the services provided • Common outcomes for VMF programs • Increase in educational attainment • Increase in vocational skills and credentials (e.g., professional licensure) • Increase in employment outcomes • Others?

20. Outcomes for Veteran and Military Family Programs– Program Examples • Veterans Helping Veterans Now • # of veterans receiving services as result of referral who then gain employment (for example) due to services received • Metro Community College, Omaha, Nebraska • # of veterans succeeding in college due to support of informed college staff and faculty • CA Conservation Corps • # of veterans who completed the program and gained key skills for future employment

21. Sources of Data Collection • Self-reported data through national service members and volunteers (e.g., # of veterans served) • Self-reported data from veterans through surveys (e.g., percentage of veterans served that obtain employment post-service) • Public use and administrative data sources (e.g., data from local VA organizations) • Observation – outside observers making ratings, counting behaviors

22. Self-Report Data from Service Members • Key Question: Who are you serving? • Self-reported data through national service members and volunteers • Example: # of veterans served • Example: # of military families served • Important to know penetration rate (i.e., percent of veterans served in relation to total size of veteran population in your community) • VA data source, called VETPop, has publicly available data source on veteran population by zip code

23. Self-Report Data from Beneficiaries • Key Question: What outcomes are you achieving? • Potential Outcomes: • Increase in educational attainment • Increase in vocational skills and credentials (e.g., professional licensure) • Increase in employment outcomes • Data Sources: • Self-report survey data from veterans/military • Public use data sources

24. Public Use Data Sources of Interest • Demographic Data on Population • Veterans Population (VETPoP) • DMDC (DOD) – demographics report • Public Use Survey Data • Health (Wellness) • Survey of Army Families (SAF) • Air Force Community Assessment Survey • National Survey of Veterans

25. Public Use Data Source of Interest • Public Use Survey Data (Cont.) • Health (Behavioral Health) • Survey of DoD Health Related Behaviors • MHAT Screenings • Sexual assault survey • Employment • VBA Vocational Rehab Survey • Homelessness • VA/DoL homeless counts

26. Administrative Data Sources of Interest • Administrative Data • Homelessness • DoL VETS data on HVRP data • Education • Data on GI bill usage (VA) • Vocational Rehabilitation data • Employment • Transition Assistance Program data • Local data sources from DVOPs, LVERs, etc.

27. Tools and Resources for Performance Measurement http://www.nationalservice.gov/resources/npm/home

28. Equal Justice Works Kathryn Gravely Program Manager Kerry O’Brien Director of Federal Programs

29. Equal Justice Works Overview • Mission • Equal Justice Works is a 501(c)(3) nonprofit organization dedicated to creating a just society by mobilizing the next generation of lawyers committed to equal justice  • AmeriCorps Program • 40 full-time lawyer members in 10 states. Most members serve for two years • 360 minimum-time law student members in 48 states. Most members serve 300 hours in the summer • Placement sites are legal aid organizations and courts working with low-income residents, including veterans

30. Equal Justice Works: Veterans Theory of Change Summary Need: Veterans have complex legal problems that keep them homeless and in poverty Intervention: Lawyers and law students, in conjunction with veterans service infrastructure, resolve legal problems Evidence: 2012 Evaluation, Studies show that those with lawyers fare better than those without Outcome: Lives of veterans are improved, increased income, access to health care, better housing

31. Equal Justice Works: Veterans Theory of Change • Need • Low-income and homeless veterans face complex legal problems related to disability benefits, fines, warrants, housing, employment, debt, and family issues • Lawyers are uniquely equipped to resolve such problems • Data Documenting Need • 2011 V.A. CHALENG report; Research articles on veterans & need for lawyers; HUD, HHA and Veterans Benefits Administration statistics; PTSD and TBI studies (e.g. 3 of top ten unmet needs of homeless vets are legal needs – CHALENG)

32. Equal Justice Works: Veterans Theory of Change • Intervention • Lawyers and law students will resolve the legal problems of low-income and homeless veterans, leading to improved lives: higher incomes, better health, safe and secure housing and family stability • Intended Outcome • Improve the lives of veterans • Evidence-basis for Intervention • Impact of 2010-2012 Veterans Pilot Project • Studies that lawyers make a difference

33. Equal Justice Works: Outputs vs. Outcomes • Overarching Goal • Through legal assistance, improve the lives of veterans • Outputs: Amount of Service Provided • Example: Assist 30 veterans with benefits claims • Outcomes: Reflect the Changes that Occur • Example: 20 veterans win benefit cases, providing access to health care and $360,000 in additional yearly income

34. Equal Justice Works: Outcomes

35. Equal Justice Works: Data Collection • What we could say in 2010 • 1,500 veterans served • Compelling stories about individual cases • What we could not say in 2010 • Outcomes of cases • Impact on veterans served • Specific information on veterans served (e.g., demographics, legal issues addressed, etc.) • Which sites were performing well/needed assistance to meet goals

36. Equal Justice Works: Data Collection • Today, we can say: • 1,500 veterans served • Members completed 226 thorough case evaluations • Filed 91 disability benefits petitions or appeals • Thus far have won 29 of these cases, securing $642,668 in back benefits and $430,933 in yearly benefits for their veteran clients • 114 benefits cases are still pending • Successfully able to remove a barrier to employment for 127 veterans served • Demographic information specific to veterans served (e.g., era of service, age, gender) • Number of veterans served for each particular legal issues, both on macro and micro levels

37. Equal Justice Works: Data Collection • Today, we can say: • Fines or other penalties waived or reduced for 78 veterans served, totaling $36,771 • Assisted 19 veterans in getting their criminal matters dismissed or expunged • Assisted 118 veterans in repairing their credit • Of the 1,416 veterans served, 740 were homeless or at risk of becoming homeless, and 739 were veterans with a disability • 77% were male and 23% were female—meaning that Fellows served a high number of female veterans, as 10% of veterans in the U.S. are female

38. Equal Justice Works: Data Collection • Today, we can say: • 20% of those receiving services served in the Vietnam era, and 23% served in Iraq and Afghanistan • The remaining veteran clients served in all other eras of service, including pre-Vietnam, the Gulf Wars and during peacetime • The majority of veterans served were either between 26 and 35 years of age or 55 and older (216 and 420 served, respectively)

39. Equal Justice Works: Data Collection How?

40. Equal Justice Works: Data Collection • 2011 • Conducted interviews and focus groups about outcomes • 2012 • Implemented new optional measures early 2012 • Brought supervisors together in person – October • Outside evaluation completed – December • 2013 • Data resulted in stronger application for funding – Feb • Implemented refined measures & made them mandatory in March 2013 • Report submitted May 1 substantially better

41. Equal Justice Works: Data Collection • Collection Format: • Was: Custom-built web application • Cost an hourly charge every time data needs changed • Was getting expensive • Output was Excel spreadsheet • Now: Google Drive - Forms • Program manager and coordinator designed, refined • Output is Excel spreadsheet • Drive.google.com – Create “Forms”

42. Equal Justice Works: Data Collection in Google Forms

43. Equal Justice Works: Great Stories • Equal Justice Works AmeriCorps Legal Fellow Rochelle Richardson • Homeless Persons Representation Project in Baltimore, MD • Many years experience as an employee of the VA • Reexamine the case of Mr. A, a homeless veteran with a service-connected mental disability whose claim for veteran's disability benefits had already been denied • Rochelle obtained records that had not previously been reviewed and petitioned to reopen the claim and submit further evidence • The petition was successful & the claim was granted in March 2013 • The client will receive a retroactive one-time payment of $50,000 and $3,000 a month in benefits

44. Equal Justice Works: Lessons Learned • Define intended outcomes early • Align reporting with what funders require • Have performance measures and data collection tools ready at the beginning of program year • Have a “steering committee” of on the ground experts to consult • Keep refining and improving measures to more accurately reflect program impact

45. Equal Justice Works: Next Steps • Figure out how to track and capture long-term outcomes • Example: Did increased income result in better housing, better health, better outcomes for children of veterans served? • Comparison Evaluation • Compare veteran outcomes in our program with national statistics or other programs • Big Vision • Randomized control trial

46. Performance Measurement for Veteran and Military Family Programs: Re-Cap • Performance Measurement: • Tracks outputs and outcomes based on the goals of your program to meet community needs • Outputs: • Amount of service provided (e.g., veterans) through the service intervention of your program • Outcomes: • Reflect the changes that occur as the result of the services provided (e.g., veterans employed)

47. Questions??? Dr. Chris Spera CNCS Kathryn Gravely Equal Justice Works Kerry O’Brien Equal Justice Works

48. To Continue this Discussion…and Others Veterans and Military Families Knowledge Network www.nationalserviceresources.org/veterans http://bit.ly/VetsNet

49. Join Us for the Next Webinar in this Series June 12, 2013 • Making a Difference: What We Know about Using an Evidence-based Approach • Register: http://tinyurl.com/c8gp8ll

50. Thank you! Please complete the post-webinar survey