Measuring effectiveness - one foundation’s experience

1. Measuring effectiveness - one foundation’s experience M. Christine DeVita President The Wallace Foundation April 11, 2005 56th Annual Conference The Council on Foundations

2. Evolution of The Wallace Foundation strategy • Created 15 years ago through bequest of DeWitt and Lila Wallace • Through 1999, made grants of nearly $1 billion to non-profits through 100+ different program • Much accomplished -- but thought we could do better in fostering long-term improvements in areas we were working • Board retreat in 1999 and agreement to address: • Root causes rather than symptoms • Seek improvements that outlasted grant dollars • Seek impact beyond our grant dollars • Contracted dozens of initiatives down to three: education leadership, arts participation, and out-of-school time learning • Clarified our approach • Support for innovation on the ground • Develop evidence about what works and does not • Share knowledge to inform the wider field

3. With new way of working, new challenge: measuring progress • If it’s about catalyzing positive change in major public systems… • Then to be effective, needed ongoing information about progress – for ourselves and our board • See how the pieces – innovation, knowledge development, communication - fit together • Because work is challenging and uncertain, ongoing feedback, revisions, mid-course corrections are crucial for success • To have useful strategy discussions with board – a focus beyond individual grants – needed evidence of results

4. Driving the measurement goal – focus on good governance and effectiveness • Good governance is the bedrock of effectiveness • Investment performance • Human resources • Expenses • Tougher challenge: What happens after money leaves the door • Robert Wood Johnson’s scorecard was a model • Decision: Aggregate progress of individual grantees within each focus area; produce report for annual board planning retreat in January

5. First try in 2003 – a focus on ‘what happened’ was a partial success • The idea: Compare multiple progress measures against long-term desired results • The how: Measured progress in three categories • Grantee work • Development of useful knowledge for the field • Achievement of national benefits • What we got: 23 pages of data on our focus area work – but little guidance on how to interpret • Board comment: ‘This is very useful for internal management, but we need to know what it all means’

6. Second try in 2004 – not only ‘what happened’ but ‘what does it mean?’ • Fine-tuned what we measured – added sustainability • Working the plan • Institutionalizing change • Benefits to people • Developing and spreading knowledge • Asked one or two questions in each category to assess progress • Answered each question with ‘core finding’

7. How do we make ‘core finding’ visible at a glance? • Considered three options • Traffic light approach: misleading • Scores of 1-5: too definitive • Eight-point progress gauges (little or none/modest/significant): much better • Permits an assessment • Allows comparisons • Suggests relative, rather than exact, determinations • Board comment: ‘A big improvement. The gauges are useful.’ Mixed/Modest Progress No/Unclear Progress Significant Progress

8. What it looks like in practice – working the plan • Are core groups of grantees working the plan? • Example Our work in 12 school districts to improve public education leadership • The question: “Is leadership development improving?” • Core finding: Increases in number and diversity of leaders getting leadership training and placed in high-need schools – but quality of training not systematically assessed • Evidence: Participation in training activities up almost 20 percent over previous year, with 9,500 in total trained. About 70 percent of those trained are female, 40 percent people of color • Gauge: Mixed or modest progress Mixed/Modest Progress No/Unclear Progress Significant Progress

9. What it looks like in practice – institutionalizing change • Are sites making institutional changes? • Example: Our work with arts organizations and state arts agencies to help them build participation • The question: “Are grantees changing priorities to reflect participation-building?” • Core finding: Non-Wallace spending on participation activities by our grantees declined slightly (2%). A minority of our state grantees adopted performance measures; many changed their strategic plans to focus on participation • Evidence: Among arts organizations, 47 percent spent more on participation; 53 percent spent less. Among state arts agencies, 77 percent changed their strategic plans; 40 percent substantially increased focus on participation • Gauge: Mixed or modest progress Mixed/Modest Progress No/Unclear Progress Significant Progress

10. What it looks like in practice – benefiting people • Are people benefiting from our work? • Example: Our work in arts participation • The question: “Are more people from diverse demographic backgrounds attending arts performances or exhibits at the organizations we fund?” • Core finding: Organizations that collect demographic data reported significant attendance gains among low-income and nonwhite participants from 2002 to 2003 • Evidence: Median increases of 15% for low-income attendees (13 organizations reporting), and 11% for non-white participants (14 organizations reporting) • Gauge: Significant progress Mixed/Modest Progress No/Unclear Progress Significant Progress

11. What it looks like in practice –producing useful knowledge • Are we producing and sharing useful knowledge? • Example: Our work in out-of-school time learning • The question: “Are we producing knowledge that addresses key issues relevant to improving quality?” • Core finding: Wallace-commissioned studies contribute to understanding demand and principles behind quality. No lessons yet on our work with cities to improve systems to support quality. • Evidence: Publication of All Work and No Play, a Public Agenda survey • Gauge: In between mixed and significant progress Mixed/Modest Progress No/Unclear Progress Significant Progress

12. Measures helped us determine action priorities • Finding: We don’t know quality of leader training programs • Action: Invest in assessment of quality • Finding: Arts organizations are slightly cutting spending on participation-building • Action: Invest in helping organizations build endowments or cash reserves for this purpose • Finding: Practitioners find Wallace-commissioned studies helpful, but modest circulation of them • Action: Invest in doing more to share knowledge

13. Lessons we’ve learned • We needed to get the questions right • ‘Absent a clear conception of effectiveness, foundations cannot assess (quantitatively or otherwise) whether or not they have achieved it, and risk adopting measures first and then adjusting their conceptions of success to fit the measures.’ • Foundation Effectiveness, Francie Ostrower, The Urban Institute • We needed to answer both ‘what happened?’ and ‘what did it mean?’ • Board members found gauges useful to answer the latter • Early indication: Helps board and staff get ‘big picture’ of how we are doing • Early indication: Helps us plan more effectively

14. Challenges we face • How easy or tough to grade? • How do we account for barriers to action? • How do we make the process efficient? • How do we incorporate it into ongoing strategy development? • How do we continue to engage board on effectiveness?

15. Measurement – a means to the end of greater effectiveness • Provide input to strategy development • The search for an appropriate measure starts with a clear description of what you want to achieve • Use measurement of results to guide decisions about allocating funds between different initiatives • Use measurement of results to guide decisions about what to invest within an initiative