UNWOMEN/IPEN Transformative Mixed Methods Evaluation: Day 3 Mixed Methods

1. UNWOMEN/IPEN Transformative Mixed Methods Evaluation:Day 3 Mixed Methods Prof. Donna M. Mertens Gallaudet University Almaty, Kazakhstan July 2011

2. Three days together Day 1: What is the role of the evaluator? Overview of evaluation (Terms of Reference) Transformative paradigm Diversity & social justice; privilege Cultural competence Day 2: Quantitative & qualitative approaches Day 3: Mixed methods approaches IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

3. Methods Options • Quantitative approaches such as experimental, quasi-experimental, causal-comparative, correlational, survey, and single-case designs • Qualitative approaches such as group processes (e.g., focus groups or some indigenous methods), case studies, ethnographic research, phenomenological research, and PAR • Gender analysis is a mixed-methods approach that provides a framework for transformative research and evaluation that has potential for transfer to other groups that experience discrimination. • Mixed methods are most likely to be the approach of choice because of the need to integrate community perspectives into the inquiry process, thus necessitating collection of qualitative data during the research or evaluation process. (Mertens, 2009, TRE, p. 165). IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

4. Mixed Methods Decisions • Compatibility: Match the purpose, focus, questions, and design. • Timing: Determine the temporal relationship between the quantitative and qualitative data collection. • Weight: Establish priority or emphasis of the qualitative and quantitative approaches in the study. • Mixing: Determine when quantitative and qualitative strategies/data will be mixed in the process of evaluation. IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

5. Quantitative Qualitative Qualitative Quantitative Concurrent Design Quantitative and Qualitative occur more or less simultaneously Sequential Design: Quantitative Followed by Qualitative OR Sequential Design: Qualitative Followed by Quantitative IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods Mertens, 2009, TRE, p. 167

6. Community Participation IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

7. Example: Making Visible Botswana youth: addressing power inequities in the fight against HIV/AIDS using a transformative lens IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

8. Transformative Mixed Methods Design Post tests: Quant Qual; Behavior & Policy Change; Transfer To other contexts IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

9. Transformative Design Components:Post-Colonial Critical Ethnography • A statement of your evaluation problem or questions (focus). • A description of your data-collection methods, including interviewing, journaling, and coding processes, and how these will be accomplished with the evaluator as a co-performer in the field or participant observer. • An explanation of your ethical methods and how the welfare of the participants will be put first by protecting their rights, interests, privacy, sensibility, and offering reports at key stages to them, including the final report. • A description of the participants in terms of population, geographic location, norms and rules, significant historical and cultural context, and expectations for key informants. • A time frame for entering the field, collecting the data, departing from the field, coding and analysis and completion of the written report, and/or public performance. • Use of a critical theoretical framework in the design, implementation, and dissemination of the study. Madison (2005) IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

10. PAR and Transformative Design 1. The group decides on the focus and questions for the evaluation. 2. Evaluators and participants observe, engage in action, observe and record. 3. Evaluators and participants immerse themselves in action and elaborate and deepen their understandings. 4. Group members reassemble and share their knowledge, using this iteration as an opportunity to revise their plans for the next cycle ofevaluation. 5. This cycle might be repeated between 6 and 10 times depending on the complexity of the evaluation context. (Heron & Reason, 2006). IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

11. Transformative MM and PAR Greenwood and Levine (2007) summarize: “Surveys, statistical analyses, interviews, focus groups, ethnographies, and life histories are all acceptable, if the reason for deploying them has been agreed upon by the AR collaborators and if they are used in a way that does not oppress the participants” (p. 6). IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

12. Questions for Thought: Design • What is your reaction to the idea that one group will receive an intervention and another group will not? • Under what circumstances would you accept that format as an ethical course of action? • What other alternatives are possible? (Mertens, 2009, TRE, p. 197) IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

13. Gender Analysis and MM IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

14. Outcomes, Outputs & Impacts Inputs—>Activities—>Outputs—>Outcomes—>Impacts • Outputsare the products, services, and capacities that result from the completion of activities • Outcomesare the intended or achieved short-term and medium-term effects of an intervention’s outputs, usually requiring the collective effort of partners. • Impactsare the long term effects or change to which the programme, through collective effort with partners, will contribute. GE/HR evaluation: how did the programme contribute to GE and HR?

15. INDICATORS 15 IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods • Indicators are the means by which you determine progress towards a result or whether an expected result has been achieved. • Developed for ALL levels of results – outputs, outcomes & impact • Indicators measure (quantitatively or qualitatively) the status of an expected result Source: UNIFEM RBM training Module 2-1

16. INDICATORS Module 2-1 16 IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods Quantitative: focus on numbers & counting (percentage of women and men in parliament, male and female wage rates, school enrolment for girls and boys) Qualitative: capture opinions, attitudes and feelings (nature of dissatisfaction, extent of increased awareness)

17. EXAMPLES Source: UNIFEM RBM Training. Module 2-1 17 IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods Quantitative - Can be directly counted & expressed as a number • % of … # of … Frequency of … Ratio of … Amount of … • Timeliness of… Qualitative - Involves perception (can be expressed quantitatively or as narrative) • Level of Satisfaction with … Knowledge of … • Ability to … Appropriateness of... Importance of … • Use and usefulness of …

18. GENDER SENSITIVE INDICATORS Source: A. Moser. 2007 Gender and Indicators Module 2-1 18 IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods • Measures gender-related changes in society over time • Includes • sex-disaggregated indicators that provide separate measures for women and men • Indicators that are gender-specific to either women or men

19. HUMAN RIGHTS INDICATORS Module 2-1 19 IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods Assess both state progress in guaranteeing human rights and individual programme adherence and promotion of rights. HR indicators should measure: • Improvements in the capacities of rights holders and duty bearers to realize rights • Improvements in the enjoyments of rights

20. GE & HR RESPONSIVE EVALUATION Follow-Up and Use: • Dissemination strategies should make evaluation findings accessible and barrier-free to women, including both RHs and DBs • Targeting women’s organizations/networks and knowledge networks • User-friendly language • Stakeholder workshops that include women and other groups subject to discrimination (RHs &DBs) • Management Response should be issued to ensure follow-up on key gender and human rights issues IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

21. EXAMPLE INDICATORS Outcome indicator: Evidence of changes in the capability of ministries of xxx to formulate and implement policies responsive to indigenous women Output indicator: Capacity assessments (focusing on individual competencies within ministries of xxx) indicating increase in relevant knowledge, skills and attitudes within key public institutions IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

22. Examples: GE/HR Indicators • proportion of target group by sex aware of the benefits of birth registration,  • employment to population ratio by age and sex,  • gross primary graduation ratio by sex,  • percentage of women in parliament.

23. “REALITY CHECK” • Expense and making use of available data; are there too many indicators? • Unrelenting pull to quantitative measures - need a balance of qualitative and quantitative data • Proxy indicators, where measurement takes an more indirect path • Indicators should be disaggregated, as much as possible, by sex, race, ethnicity, age, geographic area • Indicators do not exist in a vacuum and must be tied to a result. IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

24. EXERCISE: DEVELOP INDICATORS Develop twoindicators(one qualitative and one quantitative) for the course example. IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

25. Sample Relevance Evaluation ?’s • How well do the programme objectives target the identified rights and needs of male and female beneficiaries? • What rights does the programme advance under CEDAW, the Millennium Development Goals and other international development commitments?

26. Sample Evaluation Effectiveness ?’s • To what extent have the objectives been achieved, and do the intended and unintended benefits meet fairly the needs of disadvantaged women? • To what extent have the capacities of duty-bearers and rights-holders been strengthened?

27. Sample Evaluation Efficiency ?’s • Could the activities and outputs have been delivered with fewer resources to the target populations without reducing their quality and quantity?  • How has the programme maximized partnerships in the delivery of the programme? • Have UN Woman’s organisational structure, managerial support and coordination mechanisms effectively supported the delivery of the programme?

28. Sample Evaluation Impact ?’s • To what extent have efforts been successful in stopping harmful and discriminatory practices against women? • What is the evidence that the programme enabled the rights-holders to claim their rights more successfully and the duty-holders to perform their duties more efficiently?

29. Sample Evaluation Sustainability ?’s • Is the programme supported by national and local women’s organisations? • Do these organisations demonstrate leadership commitment and technical capacity to continue to work with the programme or advocate for change?

30. Additional GE/HR ?’s • Equality and non-discrimination: Did the programme benefits affect equally men and women? • Empowerment: Did the budget designate sufficient resources and level of effort to address the inclusion of disadvantaged or marginalised groups? • Accountability: Were monitoring data (disaggregated according to relevant criteria such as gender, age, ethnicity, location, and income) collected and used to adjust implementation? • Social transformation: Does the intervention’s theory of change include attention to GE and HR? • Participation and inclusion: Did the implementation make systematic and appropriate efforts to include women and men, and/or reach out to disadvantaged groups? UN Women 2011

31. GE/HR Evaluation Methods • Avoid bias: selection of data collection methods & in sampling methods • Potential biases: Gender & Power (sources able to contribute more easily because of privacy & confidentiality issues ), Class or distance (favoring the more accessible) • Plans for how to include marginalized groups • Mixed methods: Quan & Qual

32. Rigor applied to methods choices IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

33. Internal & External Validity In the positivist tradition, there are two important tests of knowledge claims: 1. Is the knowledge claim true in this situation? Are the changes observed in the dependent variable due to the effect of the independent variable? (Internal Validity) 2. Is the knowledge true in other situations? Generalizability? (External Validity) IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

34. Comparative Studies: Rigor • Causal relationship assumed? Competing explanations? • Comparable groups in causal comparative? • Third variable cause both predictor and criterion variables? • Sub groups analysis? • Correlational: ordering of variables? • Predictive studies: Other screening variables? Level of .8 or better? • Reliability and range of variables IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

35. Criteria that establish rigor in qualitative methods Credibility: just like validity in qualitative research – asks if there is a correspondence between the way the respondents actually perceive social constructs and the way the evaluators portray their viewpoints. IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

36. Criteria that establish rigor in qualitative methods (a) persistent observation (b) peer debriefing (c) progressive subjectivity (d) member checks (e) triangulation (f) transferability (g) dependability (h)confirmability (i) authenticity and fairness IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

37. TRANSFORMATIVE EVALUATION: Rigor • Emphasizes Human Rights and Social justice • Analyses asymmetric power relations • Advocates culturally competent relations between the evaluator and community members • Employs culturally appropriate mixed methods tied to social action • Applies feminist theory, critical race theory, postcolonial and indigenous theories Mertens (2009) Transformative Research and Evaluation. The Guilford Press. IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

38. Authenticity (Transformative criteria) • Fairness: The evaluator presents all value differences, views, and conflicts. • Ontological Authenticity: An individual’s or groups’ conscious experience of the world became more informed and sophisticated. • Catalytic Authenticity: Action was stimulated by the inquiry process. IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

39. Power and Privilege • How do we understand the dynamics of power when participatory methods are employed by the powerful? • Whose voices are raised and whose are heard? • How are these voices mediated as issues of representation become more complex with the use of participatory methods in larger-scale planning and consultation exercises? (Mertens, 2009, P. 85) • What if I am a member of the community? How does that prepare me to work in that community? What if I am not a member of a community? To what extent is it necessary to share salient characteristics of a community? • How does cultural competency come into the discussion of interactions in evaluation contexts? (Mertens & Wilson, in press) IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

40. Criteria for Rigor: Utilization • Dissemination • Use • Management Response • Engagement with Stakeholders IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

41. Dissemination Purposes • Transparency and accountability • Informing and improving an organisation’s work • Sharing good practices and ‘how to’ to advance women’s human rights • Sharing lessons with partners on the ground and building their capacity, and • Generating knowledge on how gender equality, women’s empowerment and human rights can be advanced

42. Dissemination • Language and presentation of the report: use graphs or pictures, and be written in a manner that is gender and culturally sensitive. • Translation: anticipate the need for translation when planning the evaluation so time and budget can be included) • Making the report public: Within the UN, there is requirement to make all evaluations public (UNEG Norms and Standards) and in UN Women, this is accomplished by having the Evaluation Unit post the evaluation report in the Evaluation Resource Center (which is publicly accessible). • Reaching target audiences in a user-friendly way: lower rates of literacy? The manager needs to be creative in using other dissemination channels, suited to different audiences.

43. Dissemination Strategies Brochures outlining key evaluation lessons and recommendations Annual reports Articles in technical or organisational newsletters News releases Press conferences Media appearances Public meetings, public debates or town halls Seminars, workshops, and informal group discussions Electronic media (e-mail, websites, blogs etc.) Meeting with community leaders, one on one.

44. Challenges to Use • Lack of consensus on recommendations & required action • Those who don’t like the results attack the process • Dissemination is minimal: no funds, interest or time • No follow-up process

45. Formal Management Response • presentation of the response, action or non action to evaluation recommendations and lessons learned and follow-up or tracking mechanisms. • engage with stakeholders to reflect on the evaluation process, findings, recommendations and lessons learned. • Involve reference group in developing the management response, with the manager.

46. Agenda for Action • How does the information you learned here apply to the work that you are/will be doing in evaluation? IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

47. Resources • Mertens, D. M. & Wilson, A. (in press). Program Evaluation. NY: Guilford. • Mertens, D. M. (2010). Research and evaluation in education and psychology: Integrating diversity with qual, quant and mixed methods. 3rd ed. Thousand Oaks, CA: Sage. • Mertens, D. M. (2009). Transformative research & evaluation. NY: Guilford • Mertens, D. M. & Ginsberg, P. (2009).(Eds.) Handbook of Social Research Ethics. Thousand Oaks, CA: Sage. IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods

48. Contact information • Donna M. Mertens, Gallaudet University • Donna.Mertens@Gallaudet.edu IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods