Grassroots Grantmakers: Transforming Outcome Evaluation Presentation

1. Success Measures: Transforming Outcome Evaluation Presentation to Grassroots Grantmakers Maggie Grieve, Director, Success Measures Debby Visser, Director, Success Measures Investments and Partnerships Nancy Kopf, Senior Manager, Success Measures Evaluation Services Robert Krompak, Economic Development Specialist, Neighborhood Housing Services of Toledo September 13, 2011

3. Success Measures Overview Providing the community development field and philanthropy with practical, credible and accessible ways to collect and use outcome data for continuous evaluative learning, to demonstrate results and tell stories of change. • Participatory evaluation method • Indicators/data collection tools that measure outcomes • Training and technical assistance to build organizational capacity to conduct evaluation • Consulting to plan evaluation frameworks and support the entire evaluation cycle • Web-based data system

4. Shared Outcome Indicators/Data Tools • For a typical evaluation, users select 3 – 5 indicators/data collection tools from over 240 choices in 8categories: Affordable Housing Community Building and Organizing Community Stabilization Economic Development Financial Capability Green Affordable Housing and Community Race, Class and Community Value of Community Development Intermediary Services • Tools focus on valued levels of change Individual Community Organizational Broader community

5. Consulting and Training/TA Support • Assess evaluation interests, needs, capacity • Document program rationale/theory of change • Plan evaluation framework including definition of evaluation questions and measures • Develop data collection plans • Train organizations in data collection methods and/or coordinate data collection • Support implementation and data analysis/reporting • Facilitate organizational learning from data and integration of evaluation process into ongoing work

6. Success Measures Data System • Simple to use, sophisticated in design and scale • Web-based service provides: • Process Guidance • Success Measures Indicators and Data Collection Tools • Ability to send surveys electronically • Data Tabulation and Aggregation • Data Storage • Data Export to Excel • Options to add own indicators and data tools

11. Examples • Wells Fargo Regional Foundation Engaging grantees to explore grant & portfolio level outcomes • Habitat for Humanity International Building the capacity of staff and affiliate organizations to measure program and community change • NeighborWorks America Using a common framework to document community building and organizing strategies • Neighborhood Housing Services of Toledo Demonstrating neighborhood and resident change, leveraging investments

12. Wells Fargo Regional FoundationEvaluation of Neighborhood Revitalization Portfolio • Purpose: Document outcomes of neighborhood planning and revitalization grant programs; analyze across portfolio • Participatory Engagement: 50+ grantees use common framework and data collection tools to conduct resident quality of life surveys and document neighborhood change • Result: Foundation/grantee engaged in a learning process through use of common evaluation; longitudinal data on neighborhood change • Benefits: Improved grantee evaluation capacity; comprehensive, evolving understanding of outcomes of neighborhood change investments

14. Habitat for HumanityEvaluating Homeowner and Community Change • Purpose: Develop an outcome framework for primary Habitat programs; build staff and affiliate evaluation skills • Participatory Engagement: Involved Habitat leadership and affiliates in defining outcomes and selecting common measures at homeowner and community levels • Results: A pilot evaluation testing new evaluation framework; incorporation of this framework and evaluation method into new Neighborhood Revitalization Initiative • Expected Benefits: 10 HFHI staff trained to provide evaluation TA to affiliates; 100+ affiliates providing common data to demonstrate results of new Habitat approaches

15. Habitat for Humanity Evaluation Framework • Residents’ Perceptions of Habitat Homeownership and Living in the Neighborhood • Homeowner Benefits Survey • Homeowner Customer Satisfaction Survey • Resident Perception of the Neighborhood Survey • Resident Perception of the Neighborhood Focus Group/Interviews • Understanding Effectiveness of Affiliates’ Collaborations • Interviews/Surveys: Organizations’ Perceptions of Collaborative Effectiveness • Community Physical Conditions • Block Conditions Observations • Residential Property Observations • Vacant Property Observations • Local Economic Impact of New Construction/Renovation for Homeownership • Forecasts the economic impact of homeownership projects

16. NeighborWorks America A Common Evaluation for Community Building/Organizing • Purpose: Develop a common evaluation for community building and organizing programs that measures change at individual, group/organization and community levels. • Participatory Engagement: Framework developed by sample of organizations interested in measuring outcomes of community building efforts; built on existing Success Measures evaluation tools. Peer calls during planning and implementation. • Expected Results: 12 NeighborWorks organizations collecting baseline data in 2011. • Benefits: Baseline data on incremental development of individual, group and community change resulting from organizing and small project actions.

17. Community Building/Organizing Evaluation Framework • INDIVIDUALS: Measures perceptions, capabilities, and behaviors of residents, tenants, or other participants in community building and organizing activities. • GROUPS/ORGANIZATIONS: Measures functioning and effect of ongoing resident-led entities, such as neighborhood associations, tenant councils, and crime watches. • NEIGHBORHOODS/COMMUNITIES: Measures impact of community building and organizing on the neighborhood or community overall.

18. Evaluation Tools: Individuals • Individual Demographics Age, gender, own/rent, tenure in community, property resident or not • Resident/Tenant Engagement & Empowerment Participate in activities, serve as leader, comfortable voicing opinions, influence decisions, believe can bring about change • Leadership Development Follow up to resident leadership training: Leadership roles taken on, how apply skills learned, sought out as leader, changes attributed to training

20. Evaluation Tools: Groups/Organizations • Group Description Name, type of group, when established, purpose • Organizational Capacity Representative, accountable to community, effective leadership • Group Influence/Impact Successful in addressing issues, serve as catalyst, significant accomplishments

21. Evaluation Tools: Neighborhoods • Neighborhood/Community Description Name, boundaries, population size, # households Plus, select one or more of the following • Social Connectedness Social capital, neighborliness, sense of belonging • Capacity for Collective Action Advocacy, collaboration, ability to act as a group, voice is heard  • Quality of Life Safety, physical environment, public space, responsiveness of public services, available amenities, health and wellness, satisfaction, recommend neighborhood

23. NeighborWorks America A Common Evaluation for Community Stabilization • Purpose: Develop a common evaluation for community stabilization efforts that measures market health, physical conditions, community perceptions. • Participatory Engagement: Framework developed by nonprofit practitioners, researchers and funders. Staff and volunteers involved in data collection, analysis and use of results. Peer convening to maximize learning. • Result: 18 NeighborWorks organizations completed baseline data collection; repeating in 2012 or 2013. • Benefits: Baseline data on neighborhood conditions and vacancies directly used for strategy development. Community confidence data and neighborhood surveying also resident engagement tools.

24. Example: Neighborhood Housing Services of Toledo • Documented change from organizing, investments and other stabilization efforts in neighborhoods hard hit by foreclosures and other factors. • Evaluation in one neighborhood, led to expansion to 4 other areas. • Created ongoing partnership with large hospital to examine conditions in neighborhood surrounding medical center; led to new investments due to better understanding of safety and other issues.

25. WHAT ELEMENTS MAKE UP A SUCCESS MEASURES EVALUATION?

26. UTILIZING SUCCESS MEASURES AS A CATALYST FOR COMMUNITY CHANGE

27. Response to Housing Issues Identified by Legacy Area Residents

28. Response to Public Safety Issues Identified by Legacy Area Residents

29. Building Community Connections

30. www.nw.org/network/ps/successmeasures/ Maggie Grieve, Director mgrieve@nw.org (202) 220-2337 Debby Visser, Director, Investments and Partnerships dvisser@nw.org (202) 220-2348 Nancy Kopf, Senior Manager, Evaluation Services nkopf@nw.org (202) 220-2427 Bob Krompak, Economic Development Specialist, NHS of Toledo, Inc. bkrompak@nhstoledo.org (419)691-2900