Measuring Results That Matter: Evaluating CED Impacts

1. Measuring Results That Matter: Evaluating CED Impacts The Community Economic Development Technical Assistance Program (CEDTAP)

2. Presenters: David Driscoll Chair, CEDTAP Advisory Panel Edward Jackson Director Gail Zboch National Coordinator Ray Funk Prairies Regional Coordinator Colleen Kasting BC Regional Coordinator

4. Measuring Results That Matter: Evaluating CED Impacts Overview of session: 1:30-4:00 • CEDTAP at-a-glance • CED knowledge clusters • Results to date • Evaluating CED impacts: assessing the CEDTAP portfolio • Group discussion • Sharing preliminary results of impact surveys • Next steps

5. Goal, Objectives and Methods Goal • Demonstrate the effectiveness and legitimacy of community economic development as a strategy for economic and social change Objectives • Strengthen the capacity in CED of community-based organizations • Strengthen the capacity of the CED sector as a whole Methods • Grant-making to CEDOs to access technical assistance • Building of a pool of TA Providers • Grant-making to sector organizations • Knowledge production and dissemination • Participation in research and policy alliances and consortia

6. CEDTAP At-a-glance Donors: J.W. McConnell Family Fdn Bell Canada The Ontario Trillium Fdn WED, DEC-Québec Other Corporations 3CI Advisory Panel CEDTAP National Secretariat Technical Assistance Providers CED Organizations (CEDOs) CEDTAP Regional Coordinators

7. Program Phases, 1997-2008 Phase I pilot phase (1997-2000) $3M • Some 90 grants, 30 TA Providers, three national conferences • Funder: McConnell Foundation Phase II expansion phase (2001-2005) $7M • Some 330 grants, 800 TA Providers, three national conferences, policy roundtables • Funders: McConnell Foundation, Bell Canada, The Ontario Trillium Foundation, DEC Québec, WED-BC, WED-Prairies, RBC, Power Corp., Tides Canada Phase III transition phase (2006-2008) $3M+ • Grant-making, knowledge mobilization, corporate engagement, research alliances • Funders: Bell Canada, The Ontario Trillium Foundation, McConnell Foundation, WED-BC, WED-Prairies, DEC-Québec, SSHRC, Power Corp., RBC, others.

8. 20 CED Knowledge Cluster Areas • Agriculture & Fishing • Arts & Culture • Disability and Mental Health-Consumer Survivor Businesses • Community Land Trusts • Community Economic Renewal • Community Tourism • Employment Strategies • Environmental Management & Enterprises • E-Strategies • Financial Equity • Food Processing & Food Security • Health & Nutrition • Homelessness & Poverty • Individual Development Accounts • Industrial Re-conversion • Forestry • Sustainable Housing • Women Entrepreneurs • Youth Entrepreneurship

9. Community Strengthening Projects Categorized by CED Cluster (2001-2005)

10. Results: British Columbia, 2001-05 $564,000 granted to 49 projects Top CED clusters: • Community economic renewal • Employment strategies • Environmental management • Food processing/marketing Current activities: • Bell/McConnell • WED support for regional coordination • Partnership with Vancity Community Foundation

11. Potluck Café Society Inc. Potluck's Flash Frozen Gourmet Meals Project

12. Results: Prairies, 2001-05 Top CED clusters: • Agriculture & fisheries • Community economic renewal • Disability and mental health-consumer survivor businesses Current activities: • Bell / McConnell • WED support for regional coordination $496,500 granted to 43 projects

13. Stardale Women’s Group Inc. Foundation Stardale Artistic Co-operative

14. Results: Ontario, 2001-05 $762,000 granted to 76 projects • Top CED clusters: • Community economic renewal • Disability and mental health-consumer survivor businesses • Employment strategies • Women entrepreneurs • Current activities: • Bell/McConnell • The Ontario Trillium Foundation

15. Good Day Workshop Inc. Participants at the Good Day Workshop, refinishing old furniture

16. Results: Quebec, 2001-05 Top CED clusters: • Community economic renewal • Employment strategies • Environmental management • Youth entrepreneurship Current activities: • Bell / McConnell • DEC Québec $ 625,500 granted to 74 projects

17. L’atelier De Formation Socioprofessionnelle De La Petite-nation Des participant(e)s de l’Atelier FSPN

18. Results: Atlantic Region, 2001-05 $250,000 granted to 21 projects Top CED clusters: • Arts & culture • Community loan funds • Community tourism Current activities: • Bell / McConnell

19. Central Development Association Promotion of social and economic development in the Northern Peninsula Central Region of Newfoundland

20. Evaluating CED Impacts:Assessing the CEDTAP Portfolio Why evaluate CED impacts? • New policy actors (eg, new federal government) are unfamiliar with CED • Increased emphasis on accountability, value-for-money and results • Heightened competition for scarce public and private funds • Need to evaluate program effectiveness and impact • Opportunity to contribute to the body of knowledge on CED policy and practice • Opportunity to enable CED sector to lever bigger impacts in high growth contexts (winter Olympics in BC; energy and resources on prairies)

21. How to Evaluate CED Impacts? • Test against program results chain for community initiatives • Utilize best-practice methods and tools from the field as a whole and the portfolio, in particular • “Slice and dice” the portfolio by knowledge cluster and policy area • Be clear about units of analysis: individual/household, group or enterprise, program/fund, CEDO, community, region • Assess progress on qualitative and quantitative indicators • Survey CEDOs • Survey Technical Assistance Providers (TAPs) • Sample a small number of organizations and projects for detailed case study analysis (eg, 15)

22. Results Chains

23. Evaluation: “The 21st Cluster” Unit of analysisorganization/project Region Western Valley Development Authority, NS Evaluation Toolkit of Local Capacity Community CIEL, Community Vitality Index CEDO OISE/UT, Expanded Added Value Statement* PEACH, Evaluation Framework Program Saint John Human Development Council / Tamarack institute, Small Business Program Evaluation Framework Enterprise Social Capital Partners, Social Return on Investment* Households/ Eko Nomos/WEDC, Sustainable Livelihoods Individuals Approach* Flavour Budzzz, Wisconsin Quality of Life Index *Not CEDTAP project

24. Expanded Value Added Statement • Measures the economic and social value added to raw materials, products and services by non-profits and cooperatives’ use of labour and capital • A social accounting model created by Mook (see Quarter, Mook and Richmond, What Counts, 2003) • Estimates a comparative market value for non-monetized social contributions • Primary outputs refer to the value realized by the organization’s direct provision of services to advance its mission (eg. Revenues from fees for service, social or volunteer labour, services donated to the community) 

25. Expanded Value Added Statement (Cont) • Secondary outputs refer to value to an organization’s members or customers realized through indirect outputs (eg. Skills training, learning through membership meetings, board and committee governance meetings) • Tertiary outputs refer to value realized by those other than the organization’s members or customers (eg. Consultations with other organizations or distribution to stakeholders) • In practice, EVAS builds credibility through the use of conservative assumptions in calculating comparative market value of social contributions

26. Key Themes in Building a Long List of Potential Case Study Cedos • Urban youth/immigrant employment • Urban homelessness/poverty/redevelopment • Youth entrepreneurship • Alternative energy (urban) • Community loan funds (urban) • Women’s entrepreneurship (urban) • Disability groups (urban) • Mental health consumers/survivors (urban) • Restaurant/catering (urban) • Wood products (rural) • Community tourism (rural) • Community futures/development corporations (rural)

27. Exercise: Group Discussion At your table, discuss two questions: • How does your organization assess CED impacts? • What is the biggest challenge your organization faces in assessing CED impacts? • Please record your group’s answers. • You have 10 minutes for this exercise!

28. Findings in Process: Surveys of CEDTAP Grantees and TA Providers • Significant impacts of CEDTAP projects in which you have been involved? • Significant impacts of non-CEDTAP projects in which you have been involved? • Important indicators of: economic, social and environmental impact? • Key success factors that enable CED organizations to generate significant impacts? • Methods, tools and indicators for assessing impact that are most useful in your work? • Key barriers or obstacles that limit the impacts produced by CED organizations?

29. Generating Impacts: Top 10 Success Factors • Funding: start-up, core, promotional • Realistic and achievable plan • Input from stakeholders • Experienced and strong volunteer and board commitment • Commitment and participation of of management and staff • Time to plan and develop • Government, business and community support • Community partnerships • Public relations and marketing strategy • Ability to manage financial and social goals simultaneously

30. Economic Indicators • Number of jobs created • Assisting people with training or starting a business • Creation of social enterprises • Increase in wealth – individual and community • Increased spending on purchase of local supplies • Decreased expenses through increased productivity, sharing resources, eliminating wasteful practices • Change of attitude and behaviour in evaluating business from a financial measure to include the blended bottom line • Degree to which the venture can or willcontribute to the overall society

31. Social Indicators • Ability to support oneself and one’s family • Increase in self-esteem and self-image • Community ownership of initiative • Development of local leadership • Improvement in community quality of life issues: safety, housing, health, less hospitalization, recreation, transportation, childcare • Increase in the participation levels of the client group both as participants in the venture and recipients of the service • Legacy of work supported through long-term planning with government, business and not-for-profit sector

32. Notable Methods & Tools: • Formative and summative surveys • Good benchmark data  socio-economic indicators that measure lifestyle & healthy community improvements • Identify targets and projections coupled with goals and objectives, all in keeping with the mission • Pre and post evaluations using qualitative and quantitative measures • Working with an outside facilitator (arm’s-length)

33. Notable Methods & Tools (Cont): • Feedback from focus groups • Communicate often and openly with stakeholders • Sharing the learning with colleagues and community: ongoing email, coffees, lunches • “….Organizations have asked for a copy of our CEDTAP-funded strategic plan” • Tamarack teleconferencing and initiating our our teleconferencing for specific projects

34. Generating Impacts: Barriers/obstacles • Lack of enough resources: financial & human • Time spent on writing grants and looking for money • Lack of interest by mainstream social service agencies to take a CED approach • Lack of long-term vision and plan for sustainability • Lack of leadership • Reliance on volunteers

35. Barriers/obstacles (Cont) • Projects too short in length • Inability to engage community and stakeholders • Lack of understanding by government agencies and sponsors of the complexities of new industry development as well as “soft” changes that measure success • Success requires long-term commitment, for which some funders may not have patience

36. Measuring Results That Matter: Evaluating CED Impacts Next steps: • Additional analysis of survey data • Further analysis by clusters of the portfolio • Identification of case study organizations • Joint research with CEDOs, technical assistance providers, academics and funders • Dissemination of findings Thank you!