ECD in the 21 st Century: What’s The Best Way Forward?

1. ECD in the 21st Century: What’s The Best Way Forward? Karen Russon and Michele Tarsilla Evaluation Capacity Development Group (ECDG) AEA Presentation - October 18, 2013

2. ECDG organized an Int’l Workshop on ECD 17-21 October 2011 Today is a sort of anniversary .. Two years ago today, evaluation practitioners, commissioners and members of Standards organizations gathered to discuss the current situation and the future of ECD.

3. As a Result of the Workshop, Four Key ECD Principles were Identified Ownership Integration Usefulness Relevance

4. Today’s Questions • “What does ECD look like in practice?” • “How could ECD processes be enhanced and evaluated in the future?”

5. Today’s ECD Stories Are Global

6. A Global ECD was Conducted to Understand ECD better • Collection and Analysis of stories from multiple perspectives: actors, levels and contexts • Focus on: -ECD partnerships -Infrastructure changes -Organizational processes improvements

7. Key ECD Trends • Tri-partite ECD collaboration • Indigenous-to-indigenous ECD cooperation • Creative use of action learning/research to enhance ECD buy-in • ECD as a tool for social emancipation

8. A Few Selected ECD Stories

9. The Government of Papua New Guinea Actors/Location: Gov’t Agencies, Papua New Guinea Level: National, Provincial, Local Sector: Government Time period: 2003-present Documentation: Paper presented at AES 2013

10. The Government ofPapua New Guinea Lessons Learned • Developing a “fit-for-use M&E system” was conducive to enhancing coordination and harmonization among stakeholders in the public sector • The ECD participatory process eased the tensions among a variety of ECD stakeholders at different levels • The availability of better data translated into stronger district planning and decision-making.

11. The Papua New Guinea Assn. of Professional Evaluators(PNGAoPE) Actors/Location: Australian Evaluation Society, PNGAoPE, Coffey Int’l, Expat evaluators, AusAID, Gov’t of Papua New Guinea Level: Organizational Sector: Evaluation, multi-sectors Time period: 2006 – present Documentation: Guest blog posted on www.ecdg.net

12. The Papua New Guinea Assn. Of Professional Evaluators(PNGAoPE) Lessons Learned • The sustained support provided by international development partners is often key to ECD success but it is not sufficient. • The official government recognition of a national evaluation association was critical • The critical friend and support role played by expats could be effective during the development of a national evaluation association • The selection of topics to be discussed during monthly meetings was a sign of ownership and the first necessary step towards sustainability

13. Innovation Network Actors/Location: non-profit providing consulting services in research and evaluation to other non-profits and funders, three funders and their respective grantees; Washington, D.C. Level: Organizational Sector: women’s economic empowerment, social justice, corporate philanthropy Time Period: 2006-2011 Documentation: Personal communication with Innovation Network staff.

14. Innovation Network Lessons Learned: • Provision of “default” services is better, especially if within the scope of an existing evaluation assignment • ECD often consists of a Gradual Approach (shifting from funder’s interests to organizational needs and strategies) • It is important to respond to strategic grantees’ demand (that is, demand for critical evaluation services) and not to confused or disjointed requests; • The service provider should not be the one “running” the ECD program: leadership/management and internal staff or the funders themselves should play such role

15. DECI-2 Actors/Location: Funder, organizations, regional consultants Level: Organizational and Regional Sector: Information and Communication Technology (ICT) paired with UFE curriculum development Time Period: (DECI-1: 2009-2011; DECi-2:2012-2015) Documentation: E-mail Exchange with ECDG

16. DECI-2 Lessons Learned • Pairing ECD with capacity building in other technical sectors reinforces learning and use; • Participatory curriculum development is lengthy but ensures training uptake; • Two-level mentoring was effective: mentoring to partners and mentoring to regional mentors (“at their own pace” without a classic workshop approach on pre-established dates ); • Mostly virtual + 2 face-to-face mentoring sessions; • Timeliness of support enhanced and upfront investment to ensure organizational readiness; • Presented as a research project: joint learning- facilitator of learning process NOT external expert;

17. Habitat For Humanity - LAC • Actors/Location: HQ staff, regional managers, national technical specialists (planning and implementation officers) in Central and South America • Level: HQ, Regional and National • Sector: Housing Solutions, Community Development • Time Period: 2009-2012 • Documentation: Direct Exchange with LAC Regional Office Coordinator

18. Habitat For Humanity - LAC Lessons learned • The development of a qualitative instrument capturing outcome-related information could become the entry point for a learning and collaborative ECD effort across levels; • Getting managers and Implementers (not just researchers or evaluation specialists) involved in the development of a data collection tool made it possible to emphasize the need for timely and adequate resources as a prerequisite for better evidence; • Balance between technical rigor (19 staff and 10 experts) and face validity is critical for capacity development within and outside an organization (the HFH tool was also validated through focus groups with target populations in four countries) • Sustainability and scaling-up of the approach: this process will be replicated for the development of indicators on risk mitigation associated with HPC disaster response programs globally

19. PREVAL (Latin America) Actors/Location: UN (IFAD), regional capacity building platform, indigenous population groups, evaluation professionals in the Latin American region Level: national, regional, local Sector: Agriculture, Rural Community Development Time period: Participatory Research on the state of 15 projects M&E Systems (2004-2007) + Training on Innovative PME and Image-based M&E (2008-2010) Documentation: Reflection Paper submitted by the Program Coordinator

20. PREVAL (Latin America) Lessons Learned • ECD could be a remarkable tool for social emancipation and an effective enabler of transformational learning in marginalized communities • Joint Regional training aimed at both rural community leaders and evaluation specialists enhanced the ECD process • The use of several communication means (talking maps, thematic maps, photographs, info-graphs) and visits to indigenous communities enhanced inter-learning among the trainers’ workshop participants • The organization of peasant contests on who (within the community) had used the evaluation findings more effectively than others was a good entry point for transfer of evaluation knowledge and skills in the following months

21. Ukk'aa– Alaskan ECD • Actors/Location: ANZEA; indigenous research group from Massey University, NZ; Director & students at Interior-Aleutians Campus, Univ of Alaska Fairbanks; Evaluation Research Associates, Native Elders, Fulbright-scholar-in-residence; Ukk'aa - the Alaska Native group • Level: Individual - statewide; organizational • Sector: various • Time period: 2008 – present • Documentation: AJE article, email correspondence

22. Ukk'aa – Alaskan ECD Lessons Learned • Interactions and collaboration among indigenous groups promotes an authentic horizontal ECD process and collaborative learning • Buy-in by native Elders critical to project success • Critical friends are often key enablers of ECD processes: New Zealand Evaluation Association members (Maori) mentored Alaskan Native population on indigenous evaluation theory & practice • A research grant might bring ECD actors to the table for the first time and enable the development of an ECD plan

23. Civic Theater (Kalamazoo, MI) Actors/Location: Community theater, Kalamazoo, MI Level: Organizational Sector: Arts Time period: ongoing Documentation: Key informant interviews

24. Civic Theater (Kalamazoo, MI) Lessons Learned • The systemic integration of evaluation throughout an organization where all of management acts as evaluators is a remarkable ECD strategy; • Identifying and maximizing opportunities for conducting and using evaluation is a truly ECD-savvy strategy. Activities included: • -Seeking informal feedback as patrons exit theater lobby; • -Using evaluation data for marketing purposes; • -Establishing a mechanism where staff and supervisors could evaluate each other; • -Evaluating the quality of vendors and software;

25. Quick Observations on the Process The way a story was told was sometimes as illuminating as the content of the story. Two contrasting storytellers: •  New Zealand (Maori) and Alaskan (Native) began telling their story by introducing themselves, providing backgrounds and personal perspectives from participating in the ECD activity. • A different story of the creation of a new evaluation association began by describing political/economic background as one of deficit, needing to be developed and improved.

26. What’s the Best Way Forward? • Develop long-term collaborative relationships with funders as partners in the development process. • Don’t Develop Capacity, work with actors who are WILLING and MOTIVATED to develop it further • Work on at all three levels (individual, organizational and contextual) within your system • Step back and let ownership emerge: Home grown, not externally driven - imposed by funders, gov't or others "forcing" evaluation on an institution/gov't. • Need internal champions who are leaders/decision-makers and external champions who can act as critical friends/coaches/mentors.

27. What’s the Best Way Forward? • Give ECD a chance to take hold - long-term. Nonetheless, don't be averse to change: tactics/strategies, partnerships, actors, policies, budgeting, training methods, and infrastructural arrangements An evaluation system design should be appropriate to the needs of the institution (considering size, complexity, political environment, culture, etc.). Consider organizational design, job descriptions evaluation approaches that are unique to meet those needs. • Seeing the value of evaluation enhances its use/usefulness: providing evidence for policy-makers can dramatically affect resource Allocation • Acknowledge the power of peer-to-peer support: indigenous-to-indigenous

28. QUOTE “It takes time for an organisation to find its way into evaluation; and for evaluation to find its way into organisational functioning. This involves overcoming resistance, seeking best approaches, trial and error, staff training and exposure and eventual 'absorption' of the practices as part of ordinary functioning.” Community Development Resource Assn. (CDRA), South Africa

29. Thank You Very Much! Michele Tarsilla, Ph.D. E-mail: mitarsi@hotmail.com Karen Russon krusson@ecdg.net Please follow us on: www.ecdg.net