Chapter 5: Considering the Evaluation Approach 模块五：评价方法的考虑

1. Chapter 5: Considering the Evaluation Approach模块五：评价方法的考虑

2. Introduction to Evaluation Approaches评价方法简介 • An Evaluation Approach is a • “general way of looking at or conceptualizing evaluation, which often incorporates a philosophy and a set of values” (Duigen,2007) • 评价方法是一种: • 观察评价或将评价概念化的一种普遍方式，通常包括一种理念和一套价值观(Duigen,2007) 2

3. Approaches to Development Evaluation发展评价的方法 Some approaches to development evaluation have been used and tested for many years and continue to be valuable 有些发展评价方法已经过多年的使用和试验，并且仍旧很有价值 A variety of approaches and strategies have been developed to meet the changing requirements of development evaluation 开发了多种方法和策略来满足发展评价的不断变化的要求。

4. Evaluation Approaches评价方法

5. Prospective Evaluation前瞻性评价 • Evaluation in which a project is reviewed before it begins • 在项目开始之前对它进行评价的评价 • Attempts to: • 旨在： • assess the project’s readiness to move into the implementation phase • 评价项目是否已准备好进入实施阶段 • predict its cost • 预测它的成本 • analyze alternative proposals and projections • 分析其它提议与预测

6. Types of GAO Forward Looking Questions美国审计总署前瞻性问题类型 6

7. Activities for Prospective Evaluations前瞻性评价的活动 • Careful, skilled, textual analysis of the intervention • 对干预进行认真熟练的书面分析 • Review and synthesis of evaluation studies from similar interventions • 对类似干预的评价研究进行评议和综合提炼 • A prediction of likely success/failure, given a future context that is not too different from the past and suggestions on strengthening proposed program and policy is going forward. • 如果未来情形与过去差别不大，且计划和政策的改进建议在如期实行，对成败的预测

8. Evaluability Assessment可评价性评估 • A brief preliminary study to determine whether an evaluation would be useful and feasible • 对评价的简要的初步研究，以确定评价是否有用和可行 • Helps decide whether or not the intervention is sufficiently clear so that one can conduct an evaluation • 有助于决定干预活动是否足够明确以确保评价能够开展 • Helps refocus the goals, outcomes, and targets to be absolutely clear on what is to be achieved • 有助于重新确定目的、成果以及目标

9. Steps in Evaluability Assessment可评价性评价的步骤 • review materials that define and describe the intervention • 审查干预措施的界定和描述材料 • identify any modifications to the implemented intervention from what was originally planned • 确认已实施的干预在最初规划的基础上可以做出哪些修改 • interview intervention managers and staff about the goals and objectives • 探访干预管理人员和员工，了解干预的目标与目的 • interview stakeholders • 探访利益相关者 • develop an evaluation model • 创建一个评价模型 • identify sources of data • 确认数据来源 • identify people and organizations that can implement any possible recommendations from the evaluation • 确认可以实施评价建议的人员与组织

10. Advantages of Evaluability Assessment可评价性评估的优势 Can improve 可以提高: the ability to distinguish between program failure and evaluation failure 计划失败和评价失败的区分能力 accurate estimation of long term outcomes 对于长期成果的准确估计 stakeholder investment in the program 利益相关者在计划中的投资 program performance 计划的绩效 program development and evaluation skills of staff 计划发展和员工的评价技能 visibility and accountability for the program 计划的可视性和问责制度 administrative understanding of the program 对于计划的行政理解 policy choices 政策选择 continued support 持续支持

11. Challenges of Evaluability Assessment可评价性评估的挑战 Can be time consuming 可能会很耗时 If evaluation team does not work well together, can be costly 如果评价小组没能很好地合作的话，评价成本会很高

12. Goal-based Evaluation限定目标评价 • A goal-based (or objectives-based) evaluation限定目标评价 : • measures the extent to which a program or intervention has attained clear and specific objectives • 衡量计划或者干预活动达到明确特定目标的程度 • focuses on the stated outcomes (goals and objectives) • 注重明确的成果（目标） • is used by most development organization project evaluation systems • 被大多数的发展组织项目评价系统所采用

13. Criticism of Goal-based对限定目标评价的批评 • Concentrate on the economical and technical aspects instead of social and human aspects • 集中在经济和技术层面而非社会和人文层面 • Focus only on stated goals只关注明确说明的目标 • other goals may be implicit or discussed earlier then not reflected in the stated goals • 其它的目标可能是不明确的，或者是先前讨论过但是没有体现在明确说明的目标中 • Different from results-based 和结果导向的评价不同 • results-based looks for results whether or not they were stated as goals or objectives • 结果导向的评价注重结果，不论它是目的还是目标

14. Goal-Free Evaluations非限定目标评价 • The evaluator purposefully avoids becoming aware of the program goals • 评价者有意地避免知道计划的目标 • Predetermined goals are not permitted to narrow the focus of the evaluation study • 不允许有预先确定的目标来缩小评价研究的焦点 • Focuses on actual outcomes rather than intended program outcomes • 把重点放在实际成果上面，而不是计划的预期成果 • Goal-free evaluator has minimal contact with the program manager and staff • 非限定目标评价者最大限度地减少与计划的管理人员和员工的接触 • Increases the likelihood that unanticipated side effects will be noted • 增加了发现意外副作用的可能性

15. Multi-Site Evaluations多地点评价 • An evaluation of a set of interventions that share a common mission, strategy, and target population • 对一系列有着共同任务、战略和目标人口的干预进行评价 • Considers: • 考虑： • what is common to all the interventions • 所有干预的共同点是什么？ • what varies and why • 不同点是？为什么？ • differences in cultural, political, social, economic, and historical contexts • 文化、政治、社会、经济和历史背景的差异 • comparability of indicators across these different contexts • 在这些不同背景下指标的可比较性

16. Advantage of Multi-Site多地点评价的优势 • Typically a stronger design than an evaluation of a single intervention in a single location • 与对单一地点单一干预的评价相比，通常在设计上更强大 • Has a larger sample and more diverse set of intervention situations • 取样更大，包括更多不同的干预情况 • Provides stronger evidence of intervention effectiveness • 为干预的成效提供了更加有力的证据

17. Challenges of Multi-Site多地点评价面临的挑战 • Data collection must be as standardized as possible • 数据收集必须做到尽可能标准化 • Requires well-trained staff, access to all sites, and sufficient information ahead of time to design the data collection instruments • 需要训练有素的员工，可以到达所有地点，提前掌握足够的信息，来设计数据收集工具 • Data collection needs to be collected in order to understand differences within each intervention and their communities • 为了理解各个干预的不同及群体差异，需要进行数据收集

18. Cluster Evaluations群体评价 • Evaluation of a set of related activities, projects, and/or programs • 对一系列相关的活动、项目和/或计划进行评价 • Focus is on ascertaining lessons learned • 重点关注汲取到的教训 • Similar to multi-site evaluations but the intention is different • 类似于多地点评价，但意图不同 • Information reported only in aggregate • 只进行总体信息报告 (continued on next slide) （转下页）

19. Cluster Evaluations(cont.)群体评价（续） • Stakeholder participation is key • 利益相关者的参与是关键 • NOT concerned with generalizability or replicability, variation seen as positive • 不具有普遍性或可复制性，变化被视为积极的 • More likely to use qualitative approaches • 更可能使用定性方法

20. Social Assessment社会评价 • Looks at various structures, processes, and changes within a group or community • 关注一个团体或社团之中的结构、流程和变化 • Brings relevant social information into the decision-making process for program design, implementation, monitoring and evaluation • 将相关的社会信息纳入到计划的设计、实施、监测和评价的决策过程中 • Used to ensure that social impacts of development projects are taken into account • 用来确保发展项目的社会影响被纳入考虑范围之内

21. Social Assessment(cont.)社会评价 （续） • Involve stakeholders to assure that intended beneficiaries find project goals acceptable • 让利益相关者参与进来，以确保目标受益人觉得项目目标可接受 • Assess adverse impacts and determine how to mitigate • 评估不利影响，并决定减缓方法 • Assists in forming key outcome measures • 帮助形成关键的成果衡量方法

22. Common Questions during Social Assessment社会评估阶段的普遍问题 Who are the stakeholders? Are the objectives of the project consistent with their needs, interests, and capacities? 谁是利益相关者？ 项目目标和需求、利益以及能力是否一致？ What social and cultural factors affect the ability of stakeholders to participate or benefit from the operations proposed? 哪些社会和文化因素会影响利益相关者参与提议活动或者从中受益？ (continued on next slide)（下页续）

23. Common Questions (cont.)普遍问题（续） What is the impact of the project or program on the various stakeholders, particularly on women and vulnerable groups? What are the social risks that might affect the success of the project or program? 项目或计划，对各种利益相关者，特别是女性或者弱势团体，有什么影响？哪些社会风险可能会影响项目或计划的成功？ What institutional arrangements are needed for participation and project delivery? Are there adequate plans for building the capacity required for each? 为了参与和项目交付需要什么样的制度安排？有没有充足的方案来进行能力建设？

24. Tools and Approaches工具与方法 • Stakeholder analysis • 利益相关者分析 • Gender analysis • 性别分析 • Participatory rural appraisal • 参与式农村评价 • Observation, interviews, focus groups • 考察、访谈、专题组讨论 • Mapping, analysis of tasks, wealth ranking • 规划、任务分析、财富排名 • Workshops: objective-oriented project planning, team-up • 专题研讨会：以目标为导向的项目规划，团队协作

25. Environment and Social Assessment环境和社会评估 Environment and Social (E&S) addresses the impact of development on these issues 环境和社会评估探究的是发展对这些问题的影响 Development organizations are recognizing the role that local people can play in the design and implementation of interventions for the environment and natural resources 发展组织现在不断认识到本地人群在设计或实施环境和自然资源干预活动方面可发挥的作用 (continued on next slide) （下页续）

26. E&S Assessment (cont.)环境和社会评估（续） E&S assessment may be the sole purpose of the exercise or it may be embedded in the project evaluation 环境和社会评估可能是任务的唯一目的或者是已经包含在项目评价中了 Many interventions may have environmental impacts 许多干预活动可能会给环境带来影响 Most development organizations adhere to core E&S standards and must evaluate their implementation in projects and programs 大多数发展组织遵守环境社会核心标准，并且必须对项目和计划的实施情况做出评价

27. E&S Guidelines/ Standards/Strategies环境和社会指导方针/标准/战略 Used to help assess the impact of the intervention on the ESHS 用于评估干预对环境、社会、健康、安全的影响 Three main sources三个主要来源 Equator Principles 赤道原则 ISO 14031 Sustainable Development Strategies: A Resource Book 可持续发展战略：资料读物

28. Participatory Evaluation参与式评价 • Representatives of agencies and stakeholders (including beneficiaries) work together in designing, carrying out and interpreting an evaluation • 机构和利益相关者（包括受益者）的代表共同设计、实施并解释评价 • Breaks from the audit ideal of independence • 不具备审计独立性 • Breaks from scientific detachment • 脱离科学分离 • Responsibility for planning, implementing, evaluation, and reporting is shared with all stakeholders • 规划、实施、评价和汇报责任由所有利益相关者共同承担 • Partnership based on dialogue and negotiation • 合作伙伴关系建立在对话和协商的基础上

29. Participatory Basic Principles参与式评价的基本原则 • Evaluation involves participants skills in goal setting, establishing priorities, selecting questions, analyzing data, and making decision on the data • 评价涉及到参与者在制定目标、确立优先重点、选择问题、分析数据和对数据做出决定等领域的技能 • Participants own the evaluation — they make decisions and draw their own conclusions • 参与者拥有评价——他们做决定并得出他们自己的结论 • Participants ensure that the evaluation focuses on methods and results that they consider important • 参与者确保评价的重点放在他们认为重要的方法和结果上 (continued on next slide) （转下页）

30. Participatory Basic Principles(cont.)参与式评价的基本原则（续） • People work together and group unity is facilitated and promoted • 各方通力合作，团结意识得到增强 • All aspects of the evaluation are understandable and meaningful to participants • 对参与者来说，评价的所有方面都是可以理解并有意义的 • Self accountability is highly valued • 高度重视自我负责 • Facilitators act as resources for learning • 推动者充当学习的资源 • Participants are decision makers and evaluators • 参与者是决策者和评价者

31. Characteristics of Participatory Evaluation参与式评价的特征 • More meetings • 更多的会议 • Planning decisions are made by group • 集体做出规划决定 • Participants may: • 参与者可能： • be asked to keep diaries or journals • 需要坚持记日记或日志 • interview others or conduct focus groups • 会见其他人或进行专题组讨论 • conduct field workshops • 开展实地专题研讨 • write the report • 撰写报告

32. Participatory 参与式评价 participant focus and ownership 参与者为评价的焦点和所有者 focus on learning 强调学习 flexible design 设计灵活 rapid appraisal methods 采用快速的评价方法 outsiders are facilitators 外部人员是推动者 Traditional 传统评价 donor focus and ownership 援助者为评价的焦点和所有者 focus on accountability and judgment 强调责任与公正 predetermined design 预先决定的设计 formal methods 采用正式的方法 evaluators are experts 评价者是专家 Comparison of Participatory and Traditional参与式评价与传统评价对照

33. How to for participatory:如何进行参与式评价 • No single right way • 没有唯一的正确方法 • Commitment to the principles of participation and inclusion • 坚持参与和包含的原则 • those closest to the situation have valuable and necessary information • 那些与情况最接近的拥有宝贵和必要的信息 • Develop strategies to develop trust and honest communication • 制定战略来培养信任和诚实沟通 • information sharing and decision-making • 信息共享与决策 • create “even ground” • 创建“对等的立场”

34. Challenges of Participatory 参与式评价的挑战 • Concern that evaluation will not be objective • 担心评价不客观 • Those closest to the intervention may not be able to see what is actually happening if it is not what they expect • 如果不是他们所期望的，那些与干预最近的人可能无法看到实际发生的事情 • Participants may be fearful of raising negative views • 参与者可能害怕提出负面的观点 • Time consuming • 耗时 • Clarifying roles, responsibilities, and process • 澄清任务、职责和流程 • Skilled facilitation • 有技巧地推动 • Just-in-time training • 及时的培训

35. Benefits of Participatory参与式评价的好处 • Results are more likely to be used • 结果的利用程度更高 • Increased buy-in, less resistance • 买进增加，阻力减少 • Increased sustainability • 可持续性增加了 • Increased credibility of results • 结果的可靠性增强了 • More flexibility in approaches • 方法更加灵活 • Can be systematic way of learning from experience • 可以作为系统地从经验中学习的方法

36. Is Participatory Right for You?参与式评价是否适合你？ • Is there a need for: • 是否存在以下需求： • an independent outside judgment? • 独立的外部判断？ • Considerable technical information? • 大量技术信息？ • （maybe not) • （或许没有） • Will stakeholders want to participate? • 利益相关者是否希望参与？ • Is there sufficient agreement among the stakeholders so they can work together， trust each other, and view themselves as partners? • 利益相关者之间是否能达成一致意见，共同合作，互相信任，并且视彼此为合作伙伴？ • ( Maybe so) • (或许如此)

37. Outcome Mapping成果规划评价 • Focuses on one specific type of result: outcomes as behavioral change • 重点关注某个具体类型的成果：产生行为变化的成果 • A process to engage citizens in understanding their community • 一个增进公民对社区的了解的过程 • A method for collecting and plotting information on the distribution, access and use of resources within a community • 收集和规划社区内资源分配、利用与使用信息的方法 • A useful tool for participatory evaluation • 参与式评价的一个有用工具 • Focus is people and behavior change • 重点关注人与行为的变化

38. Boundary Partners边界合作伙伴 Individuals, groups, and organizations who interact with projects, program, and policy 和项目、计划和政策有互动的个人、团体及组织 Those who may have the most opportunities for influence 那些最有可能产生影响的团体组织 Outcome mapping assumes boundary partners control change 成果规划假定了边界合作伙伴可以控制变化

39. Outcome Mapping and other Approaches成果规划和其他方法 Outcome mapping does not attempt to replace the more traditional forms of evaluation 成果规划并非要取代其它传统的评价形式 Outcome mapping supplements other forms by focusing on behavioral change 成果规划通过关注行为变化与其他方式互为补充

40. Rapid Assessment快速评估 • Intended to do evaluations quickly while obtaining reasonably accurate and useful information • 旨在通过获得适度准确实用的信息，快速进行评价 • Uses a systematic strategy to obtain just essential information • 使用系统策略来获得最基本的信息 • focus is on practical issues “must know” rather than “nice to know” • 焦点为实际的必须了解的问题，而非最好能了解的问题 • Best when looking at processes and issues • 在考察流程和问题时最有效果 • When time and resource constraints or lack of baseline data • 最适合用于时间和资源有限或者缺少基线数据的时候

41. Rapid Assessment Approach快速评估方法 Observation of the intervention within its setting 在干预的背景下观察干预活动 Excellent listening skills needed 需要出色的倾听技巧 Use more than one source of information 使用不止一个信息来源 Can use the same data collection as others, only smaller scale 可以使用同样的数据收集，只是规模更小

42. Evaluation Synthesis综合评价 • A systematic way to: • 系统的方法： • summarize and judge previous studies • 总结并评判先前的研究 • to synthesize their results • 综合它们的成果 • Useful when many studies have already been done • 在已经进行了许多研究时非常有用 • Useful when you went to know “on average, does it work?” • 当你想知道“总体上是否有效时”非常有用

43. Steps in Evaluation Synthesis综合评价的步骤 • Locate all relevant studies • 找出所有相关的研究 • Establish criteria to determine the quality of the studies • 制定确定研究质量的标准 • Include only quality studies • 只包含高质量的研究 • Combine the results: chart the quality of each study and the key measures of impact • 将成果结合起来：用图表说明各个研究的质量以及影响的关键量度 • can be table or chart showing the number of studies with similar results • 可以是说明具有类似结果的研究的数量的表格或图表

44. Advantages and Challenges of Evaluation Synthesis综合评价的优势和挑战 • Advantages: 优势 • Uses available research • 使用现有研究成果 • Avoids original data collection • 避免了原始数据收集 • Is cost effective • 节省成本 • Challenges：挑战 • Locating all the relevant studies • 找到所有的相关研究 • Obtaining permission to use the data • 获得使用数据的许可 • Same group may have done several studies • 同一个团体可能进行过多份研究 • Developing a credible measure of quality • 制定一个可靠的质量衡量标准

45. Meta-evaluation广义评价 Meta-evaluation is viewed as expert review of one or more evaluations against professional quality standards 广义评价是指按照专业质量标准对一个或多个评价进行专家评议

46. Emerging Approaches新兴的方法 Utilization-focused evaluation 以利用为焦点的评价 Empowerment evaluation 赋权增能评价 Realist evaluation 现实主义评价 Inclusive evaluation 相容评价 Beneficiary evaluation 受益者评价 Horizontal evaluation 横向评价

47. Challenges Going Forward未来的挑战 MDGs major implications for development evaluation 千年发展目标对发展评价有重大意义 Shift from evaluating project goals and objectives to evaluating MDGs 从项目目标目的的评价转向对千年发展目标的评价 Should also shift from development organizations evaluating to developing countries receiving aid evaluating 评价的主体还应该从发展机构转向接受援助的发展中国家

48. Five Major Methodological Consequences五个主要方法论后果 • Alignment – all development interventions should be assessed on their impact on the MDGs as articulated in development programs owned by developing countries. • 一致性：应按发展中国家的发展计划所述，对所有发展干预活动对千年发展目标的影响进行评估 • Aggregation – the results of development interventions should be measured at the country level. • 聚合：发展干预活动的结果应该在国家层面来衡量 (continued on next slide)（下页续）

49. Five Major Methodological Consequences (cont.)五个主要方法论后果（续） • Accountability – given the global partnership for development embedded in the MDGs, the performance of all development actors should be evaluated in terms of their distinctive accountabilities and reciprocal obligations. • 问责：鉴于全球合作促进发展已被纳入千年发展目标，对所有发展参与者的绩效的评价应主要考量其各自的责任以及对等的义务 • Attribution – neither the evaluation community nor the economics profession have managed to convince a skeptical public that “aid works.” More robust evidence is needed to demonstrate a causal link between aid operations and country level economic performance. • 归因：评价圈或者经济学家们都未能使持怀疑态度的公众信服援助的成效。需要更多有力的证据来表明援助活动和国家层面的经济表现之间的因果联系。 (continued on next slide) （下页续）

50. Five Major Methodological Consequences (cont.)五个主要方法论后果（续） Asymmetry – The imbalance between the seven MDGs that address poor countries’ performance and the one (the eighth) that addresses the need to improve the enabling global policy environment for development should be redressed: that is, the MDGs themselves (and the processes that underlie their monitoring and their oversight) should be evaluated. 不对称性——八条千年发展目标中，有七条关注的是提高穷国绩效，第八条是改善发展的全球政策环境，这之间的不平衡性应加以矫正：也就是说，千年发展目标本身（以及其监测和监管流程）需要被评价