Evaluating Our Coalition: Are We Making an Impact?

1. Evaluating Our Coalition: Are We Making an Impact? Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013

2. Why a Coalition?

4. Why Evaluate? Evaluation can guide the change process Evaluation can monitor the health of your coalition Evaluation can “tell your story” Evaluation can “dig deeper” Evaluation can measure your success and answer the “so what” questions

5. Why Evaluate? Provide Accountability to Community, Funders & Stakeholders Quality Efficiency Effectiveness Effectiveness Effectiveness Effectiveness

6. Network Dialogue Informal linkages Information exchange Step 1: Defining YOUR Coalition Cooperative Group • Short term goals/tasks • Linkages may be formalized but still advisory

7. Partnership Share/merge resources Joint planning but individualized authority Frequent communication Common issue Collaborative • Long-term commitment • Formal agreements • Shared leadership and decision making • Shared community-wide indicators of success

8. SO WHAT!! The Impact of Your Coalition

9. Step 2: Meeting Evaluation Challenges • Lack of clarity about the expected outcomes from the coalition: What to evaluate • Coalitions take time to build • Coalitions are not static • Measurement: Attribution vs. Contribution (who gets “credit”)

10. Challenge #1: Lack of Clarity about Expected Outcomes

11. Some Solutions • Reflect back to your logic model: reasonable, feasible, focused outcomes • Clarity about what the coalition will bring to the community that individual programs cannot. • The evaluation must match the level of anticipated outcome

12. Level of Impact

13. Individuals Change in behaviors, attitudes, skills

14. Agencies/Organizations Change in behaviors, attitudes, knowledge and skills Change number and type services provided Change in policies, procedures Leadership development

15. Systems Change in behaviors, attitudes, knowledge and skills Increase in coordination/collaboration/cohesion within or across systems Change in service delivery Joint decision making

16. Community More resources/services for the community Change in legislation, regulation, policyHeightened community awareness/civic action Demonstrated change in social, economic, environmental conditions

17. Challenge #2: Coalitions Take Time to Build

18. Some Solutions • Clarify the “development phase” of your coalition and the key tasks that need to be accomplished • That will lead you to appropriate questions and important data to collect

19. Evaluating the right thing at the right time! “Developmental Evaluation”: Evaluating the Right Thing at the Right Time… • Planning and forming • Why do we need this collaboration? • Who should be involved? • Do we have a common vision? Formative Evaluation: --Readiness and capacity

20. Evaluating the Right Thing at the Right Time Developing • Are we on track? • Is our coalition meeting the needs of members? • Are we achieving early outcomes? Process Evaluation: -- Beginning to implement

21. Evaluating the right thing at the right time! Evaluating the RightThing at the RightTime Maturing and Sustaining • What have we accomplished? • What are the benefits for our coalition and for whom? • What is the value of our effort? Outcome/summative evaluation -- Accountability

22. Challenge #3: Service Systems and Communities Are Not Static

23. Solutions to enhance evaluation SomeSolutions • Revisit your logic model: • Are the new strategies are still in line with your vision • Does the coalition membership still include the “right” stakeholders

24. Challenge #4: Attribution vs. Contribution

25. How do we insure “self interest” of member agencies/organizations are met? • How do we distinguish outcomes of the coalition vs. outcomes by programs funded by the coalition vs. other contextual factors?

26. Solutions to enhance evaluation Some Solutions • Do a good job of collecting the right data • Reflects the coalition expected level of impact • Reflects the coalition developmental stage • Critical importance of process to outcome: continue to measure perception of members • Shared measurement systems

27. Not everything that counts can be counted!

28. Phases of Shared Measurement System STEP I Design STEP 2 Develop STEP 3 Deploy • Agree on system and relation to logic model • Review current state of knowledge/data • Agree on governance structure • Identify approach, metrics, confidentiality, etc. • Develop platform and tools • Test and refine platform/tools • Staff for data management and synthesis • Learning forums • Ongoing support • Review, refine, improve, ongoing evaluation of usability and impact Source: FSG Analysis, www.fsg.org 2011

29. The COMET® System

30. Cautions…. • Clarity among organizations about WHY you are collecting this data and how it will be used • Getting organizations to agree on a set of shared indicators that reflect their work • Silo nature and reporting requirements of funders • And………….

31. Evaluation Types are All Related…. Measure infrastructure, functioning and procedures of the coalition Measure indicators of extent of implementation Measure change and realization of vision

32. Tools of the Trade SURVEYS: member benefits, perceptions of stakeholders, informants, community INTERVIEWS/FOCUS GROUPS stakeholders, beneficiaries, community members OBSERVATIONS: site visits, meetings RECORDS, DOCUMENT REVIEW CASE STUDIES: policy analysis, beneficiaries, build a story of the coalition’s work and impact INDICATORS: statistical analysis for significance, trend data, change in targets, cost-benefits EVALUATION

33. Tools of the Trade Collect information from SEVERAL SOURCES (community residents at large, targeted clientele, elected officials) Collect information using SEVERAL METHODS (survey, focus group and site observations for perceptions) Consider OUTSIDE EVALUATORS (to add credibility, expertise, objectivity) Collect both QUANTITATIVE (the numbers) and QUALITATIVE(the story) data USE your findings EVALUATION

34. Successful Coalitions Pre-Conditions • Urgency for change • Adequate financial resources • Influential champions Conditions for having impact • A common agenda (vision) • Continuous communication • A shared measurement system with a “short list” of indicators • Mutually reinforcing activities (relating to vision • A centralized backbone structure Kania and Kramer, Collective Impact, 2011 www.ssireview.org

35. Contributions of a Literacy Coalition • Selection of a core set of community outcome indicators by the literacy coalition • Provide training and technical assistance in performance measurement • Assist community literacy programs to track and use their own outcomes • Provide support for community-wide outcome data collection • Provide support for analysis of the outcome information. Adapted by Apter & O’Connor from National Institute for Literacy, Guide to Performance Management for Community Literacy Coalitions, Washington, DC 20008

36. Contributions of Evaluation • Use the findings to help attract funding • Use the findings to identify and report community literacy program needs and literacy condition • Use the findings to help identify and disseminate successful (“best”) practices in your community and elsewhere • USE THE FINDINGS TO CELEBRATE GOOD LITERACY OUTCOMES Adapted by Apter & O’Connor from National Institute for Literacy, Guide to Performance Management for Community Literacy Coalitions, Washington, DC 20008

37. For more information… Literacy Powerline www.literacypowerline.com Apter & O’Connor Associates www.apteroconnor.com dianne@apteroconnor.com cynthia@apteroconnor.com

38. For more information… The Literacy Powerline Site www.literacypowerline.com or The Apter & O’Connor Associates Site www.apteroconnor.com or Dianne@apteroconnor.com Cynthia@apteroconnor.com