Qualitative M&E in the ‘Real World’

1. Qualitative M&E in the ‘Real World’ Carol J. Pierce Colfer Center for International Forestry Research & Cornell International Institute for Food, Agriculture and Development

2. What Follows: • Why do qualitative M&E? • Three Examples of M&E in action: • Quilcene, WA, educational research • Global comparative research on assessment of sustainable forest management • Global comparative action research with local communities • Some Dangers and Conclusions

3. Why Qualitative M&E? • Need for holistic understanding of a situation & how it has changed. • An unexpected finding emerged that hadn’t been measured initially. • An external, post-facto analysis is demanded, on issues not initially assessed. • The team has qualitative, not quantitative skills.

4. 1a: Qualitative Ethnographic M&E [Typically] involves • Participant observation • Long term residence in the research context • A holistic, inductive, open-ended orientation • Varying degrees of independence from project being evaluated

5. 1b: Rural Experimental Schools Project(National Institute of Education) • 10 US rural schools, given grants to experiment with their local schools • External, long term M&E, both qualitative (fieldwork) & quantitative (cross-site) ---------------------------------------------------------------- Field researchers lived in communities for ~3 years, documenting what happened and helping with cross-site studies – teasing out what went right, what went wrong.

6. 2a: Criteria & Indicators (C&I) A central idea has been that C&I can be used to monitor, assess, & even define a subject of interest. Hierarchy of Principles, Criteria, Indicators, Verifiers Ideal ones are SMART (Specific, Measurable, Achievable, Realistic, Timely) BUT some topics are difficult (impossible?) to quantify:

7. 2b: Governance Issues to Monitor(Landscape Mosaics) • Greater self-confidence among women & other marginalized groups • Improved knowledge of regulations among groups previously uninvolved • Involvement in enforcing sanctions, by a broader spectrum of stakeholders • Closer links between communities & outsiders (government officials, industry, projects, academics)

8. 2c: C&I for Sustainable Forest Management ++ (CIFOR) Aim was to develop widely agreed-upon C&I to define SFM, & for use in monitoring & assessing it (initially, in certification of timber). Series of 1-month, interdisciplinary, international field visits to compare & hone existing sets of C&I that would work in each country studied, using a series of filtering steps (described in CIFOR Toolbox No. 1).

9. 3a: Adaptive Collaborative Management A long term, learning-based approach involving (facilitated) community groups • identifying shared, future goals • Analyzing, planning, & implementing what is needed to reach those goals • Monitoring progress & revising plans accordingly • Linking productively with relevant external actors

10. 3b: Adaptive Collaborative Management (CIFOR) Facilitator/researchers worked with communities in 11+ countries (vertically, horizontally & iteratively) Teams assessed community progress in ways that worked in their contexts: • C&I that local people developed • repeated ‘reflection’ meetings • use of ethnographic observations

11. 3c: Adaptive Collaborative Management (CIFOR) – a tricky component • Ongoing M&E by community members – are we reaching our community/group goals? • M&E by researcher/facilitators---to what degree is ACM actually empowering, enriching people, &/or enhancing their well being or environments? • Result: A complicated life for researcher/facilitators; a qualitative, cross-site assessment, examining/comparing site experiences (7 dimension framework)

12. Some M&E Dangers • Getting too complex - e.g., Landscape Mosaic project’s 4 levels of monitoring • Producing something so holistic, ‘deep’ and long that no one will ever read it (cf. quantitative baseline surveys so long & complex that data never get entered, let alone analyzed). • [as with any method] Being swayed by your own ideological biases, or someone’s (donors’, employers’) desire for evidence of success

13. My Own Conclusions • Conducting qualitative M&E can be an uphill battle – donors, policymakers, & many researchers prefer quantitative assessments BUT • Qualitative M&E can often provide valuable insights, unexpected findings, not available with conventional quantitative approaches. IDEALLY: Combine the two!