Chapter 11: Managing for Quality and Use 模块 11 ： 质量及使用导向的管理

1. Chapter 11: Managing for Quality and Use模块11：质量及使用导向的管理

2. Introduction 简介 • Managing the Design Matrix • 管理设计矩阵 • Managing an Evaluation • 管理评价 • Managing Effectively • 有效管理 • Assessing Quality of an Evaluation • 对评价质量进行评估 • Using Evaluation Results • 使用评价结果

3. Evaluation Design Matrix评价设计矩阵 • Key to success in evaluation is in planning • 评价成功的关键因素的规划 • Evaluation Design Matrix, links descriptive, normative, and impact evaluation questions • 评价设计矩阵联系描述性、规范性以及影响力评价问题 • Helps to systematically map out the evaluation • 协助系统地描绘出评价的全貌 • Focuses the attention on each of the major components • 注意力集中在每个重要的成分上

4. About the Design Matrix关于设计矩阵 • Takes time to fill in since you do not have all the information available at start • 不用着急填写，因为一开始你没有所有有用的信息 • It is an iterative process • 这是一个重复叠加的过程 • Can change the format of the design matrix to fit your needs • 可以改变设计矩阵的模式来适应你的需要

5. Guide for Using the Evaluation Design Matrix如何使用评价设计矩阵 • For each question, complete all columns • 完成每个问题的所有表栏 • One row for each question 一个问题一行 • if you have two main questions, the matrix should have two rows • 如果你有2个主要问题，矩阵应该有2行 • each row specifies how you plan to design your evaluation for each question • 每一行都具体说明你是如何计划设计每个问题的评价 • Comment column may help keep track of unresolved issues and/or concerns • 评论栏能帮助你同没有解决的问题或焦点保持跟进

6. Two Useful Tools两个有用的工具 • Evaluation Plans and Operations Checklist (Stufflebeam) • 评价计划以及操作检查单（Stufflebeam） • Checklist for Program Evaluation Planning (McNamara) • 项目评价计划的核查单（麦克那马拉） • You may want to adapt these to fit your needs • 你可以更改这些以适应你的需求

7. Reviewing Your Design检查你的设计 • Make sure all of the pieces connect and will give you the best chance of obtaining the data necessary to answer the evaluation questions • 保证所有部分之间都有连接，而且可以让你尽可能获得必要的数据来回答评价问题

8. Pre-testing事前测试 • Pre-testing is essential 事前测试很重要 • if survey, have several people use it in a real setting • 如果是调查，安排几个人在真实的场景中使用 • if focus group, conduct as if real • 如果是专题讨论组，就像真的一样来进行测试 • 进行测试De-brief respondents, find out what worked, what did not • 仔细检查回复，找出什么管用，什么不能 • ask how they would fix the questions • 问他们将如何回答问题 • Have experts review your plans • 让专家来检查你的计划 • Have cold reader review surveys for clarity, grammar, and typing • 就明晰性、语法以及拼写进行咨询

9. Front-End Planning从始至终的计划 • Pay attention to: • 注意： • timing and time management • 时间的设置和安排 • selection of actors and resources involved in the evaluation • 对要素的选择以及评价涉及到的资源 • role of program logic and program theory • 项目逻辑和项目理论的作用 • design of the study • 研究的设计 • question to what extent data that are needed can be made available • 询问能够得到多少必要数据

10. Timing of the Evaluation评价的时间安排 • For an evaluation that emphasizes use, time it so your findings are available when decisions are being made or actions being taken • 对于强调使用性的评价，对它进行时间安排，这样在你做出决定或者采取行动时，就可以有发现作参考了。

11. Questions to Help Timing有助于时间设置的问题 • What decision, if any, is the evaluation finding expected to influence? 评价的发现能影响什么样的决策？ • When will decisions be made? By whom? When, then, must the evaluations findings be presented to be timely and influential? • 决策什么时候做出？由谁做出？之后，评价的发现什么时候被展示出来才是及时以及有影响力的？ • What is the history and context of the decision-making process? • 决策过程的历史及背景如何？ • What other factors will affect the decision-making? • 其他什么因素能影响决策？ • How much influence do you expect the evaluation to have — realistically? • 实事求是来说，你希望评价具有什么样的影响力？ (continued on next slide) 续下页

12. Questions to Help Timing (cont.)有助于时间设置的问题（续） • To what extent has the outcome of the decision already been determined? • 决策的结果多大程度上已经被决定了？ • What data and findings are needed to support decision making? • 需要什么样的信息和发现来支持决策过程？ • What needs to be done to achieve that level of influence? • 达到这样的影响力需要做到什么？ • How will we know afterward if the evaluation was used as intended? • 我们之后如何知道是否评价的使用是同预期相同的？

13. The Evaluation Team评价团队 • Who 谁 • client (commissioner of the evaluation) • 客户（评价的委托人） • Stakeholders • 利益相关者 • other evaluators • 其他评价者 • Participants • 参与者 • How 如何 • Terms of Reference • 任务大纲

14. Value of Terms of Reference (TOR)任务大纲（TOR）的价值 • Terms of Reference (TOR) describe the overall evaluation and establish the initial agreements prior to the work plan • 任务大纲描述整个评价过程，并且建立工作计划开始之前的最初协议 • TOR process: • 任务大纲的过程： • ensures all stakeholders are included in the discussion and decision-making • 保证所有的利益相关者都被融入到了讨论以及决策过程 • establishes the basic guidelines so everyone understands expectations and context of the evaluation • 建立最基本的指导方针，这样每个人都知道评价的内容和预期

15. OECD, DAC Glossary DefinitionOECD, DAC 的词汇定义 • Terms of Reference (TOR) are a written documentation that present: • 任务大纲是一个书面文本，它展示了： • the purpose and scope of the evaluation • 评价的目的及范围 • the methods to be used • 使用的方法 • the standard against which performance is to be assessed or analyses are to be conducted • 评估表现或者进行分析的标准 • the resources and time allocated • 需要分配的时间和资源 • reporting requirements • 报告要求

16. TOR Typically Include:任务大纲包括： • Title: short and descriptive • 标题：简短的，描述性的 • Project or program description • 计划或项目描述 • Reasons for the evaluation and expectations • 评价的原因及期望 • Scope and focus of the evaluation: the issues to be addressed and questions to be answered • 评价的范围以及重点：需要谈到的和需要回答的问题 • Stakeholder involvement: who will be involved, defined responsibilities, and accountability process • 利益相关者的介入：谁会介入，定义责任以及负责过程 (continued on next slide)续下页

17. TOR Typically Include: (cont.)任务大纲通常包括： （续） • Evaluation process: what will be done • 评价过程：需要做什么 • Deliverables: typically an evaluation work plan, interim report, final report and presentations • 可提交的：通常是评价工作计划、中期报告、最终报告和展示 • Evaluator qualifications: education, experience, skills and abilities required • 评价者的资质：教育、经验、技能以及需要的能力 • Cost projection based on activities, time, number of people, professional fees, travel, and any other related costs • 基于活动、时间、人数、专业费用以及其他相关费用智商的成本预计

18. Guidelines for Writing TOR任务大纲的写作技巧 • State clearly the objectives of the evaluation • 说清楚评价的目的 • identify the stakeholders of the evaluation • 说明评价的利益相关者 • the products expected from the evaluation • 评价预期产生的产品 • how the products are to be used • 产品如何使用 • the specific issues to be addressed • 要谈到的具体问题 • the methodology • 方法论 • arrangements for the evaluation • 评价的准备 (continued on next slide)续下页

19. Guidelines for Writing TOR(cont.)任务大纲的写作技巧 • Do not simply state the objectives in technical or process terms. Be clear on how the evaluation is expected to help the organization • 不要仅仅用专业术语或过程来描述目的，清楚说明评价如何能帮助组织结构 • Focus on key issues to be addressed by the evaluation • 集中注意力于评价的主要问题 • Avoid too many issues. It is better to have an evaluation that examines a few issues in-depth rather than one that looks into a broad range of issues superficially • 避免太多问题。集中注意力集中的深入的检查几个问题比肤浅的涉及很多的问题要好

20. TOR Checklists任务大纲的检查单 • Evaluation Contracts Checklist (Stufflebeam) • 评价合同检查单（ Stufflebeam） • Checklist for Developing and Evaluating Evaluation Budgets (Horn) • 发展及评估评价预算的检查单（Horn） • Key Evaluation Checklist (Scriven) • 主要的评价检查单（Scriven）

21. Contracting Evaluations聘用他人进行评价 • If human resources are not available in your organization, you may need to hire one or more other people to assist • 如果你的机构人员不足，你需要雇佣其他人员进行协助 • Can be brought in for the entire study or only parts • 可以是全程参与，也可以是部分参与

22. Two Main Steps for Hiring Contractor雇佣合同工的两个步骤 • Developing a request for proposal (RFP) 完成一个征求建议书（RFP） • Use a selection panel to choose a contract evaluator 组成一个遴选委员会来挑选合同制评价者

23. Items for Contracting合同的条款 • Purposes of the evaluation 评价目的 • Background and context for the study 研究背景 • Key information requirements 主要的信息要求 • Evaluation objectives 评价目标 • Deliverables required 要求的提交内容 • Time frame 时间表 • Criteria for tender selection 细致选择的标准 • Contract details for the project manager 项目经理的合同细节 • Deadline for the proposal 议定书的截止日期 • Budget and other resources 预算以及其他资源

24. Selection Process for Contracting承包的选择过程 • Select a panel comprising people with: • 遴选委员会的成员须具备： • evaluation knowledge and experience • 评价知识和经验 • knowledge of the program area • 项目领域的知识 • knowledge of the culture • 文化方面的知识 • ownership of the findings and their uses • 成果及其使用的所有者 • Have the panel select the proposal using the criteria in the RFP • 委员会利用RFP的要求来选择议定书 • Keep a record of the selection process • 对筛选过程进行记录

25. Criteria for Selecting the Contractor选择承包方的标准 • What is the contractor’s record of accomplishment? • 承包方取得了什么样的成就？ • Has the RFP been adequately addressed? • RFP是否得到了充分的重视？ • Is there a detailed explanation of implementation? • 是否有对执行过程的详细解释？ • What is the communicating and reporting strategy? • 沟通策略和汇报策略是什么？ • Is there evidence of competencies? • 是否有能胜任的证据？ • What is the cost — is it specified in detail? • 成本是什么？是否有详细说明？

26. Responsibilities of Purchaser, Once Hired一旦被雇佣，采购方的责任 • Keeping goals and objectives clear • 保证目标明确 • Maintaining ownership of the study • 保持研究的所有权 • Monitoring the work and providing timely feedback • 监测工作，提供适时反馈 • Decision-making — in good time • 在合适时做出决策 • If changes are required to the contract, being open to negotiation with the contractor • 如果合同需要改动，应该同承包方公开谈论

27. Managing Roles and Responsibilities管理角色以及责任 • Clearly define and agree to who is responsible for what and when • 明确定义并就何人就什么以及何时负责达成一致 • Who needs roles and responsibilities defined: • 谁需要定义角色和责任： • evaluation managers • 评价经理 • Evaluators • 评价者 • Client • 客户 • providers of information (stakeholders) • 信息提供者（利益相关者） • consumers (impactees) • 消费者（受影响者）

28. Evaluation Manager Roles评价经理的角色 • Helping evaluators do their work 帮助评价者工作 • clarifying the TOR for the team • 帮助团队明确任务大纲 • answering questions • 回答问题 • checking on status of responsibilities • 检查责任的归属及状态 • asking if they need additional resources • 询问他们是否需要额外的资源 • helping team learn • 帮助团队学习 • facilitating team meetings • 主持团队会议 • Choosing staff 挑选成员 • Reviewing results and report 检查结果以及报告

29. Evaluation Manager Responsibilities项目经理的责任 • Detailed responsibilities in three areas: • 在三个领域的责任细则： • Preparation • 准备 • Implementation • 执行 • follow-up • 跟进

30. Evaluators Skills and Abilities评价者的技巧和能力 • Expertise in the specific subject matter • 在相关领域的专业能力 • Knowledge of key development issues especially those relating to the main goals or the ability to see the “big picture” • 重点发展领域的知识，尤其是那些跟主要目标以及需要见到全貌的能力有关的知识 • Familiarity with organization’s business and the way such business is conducted • 对组织业务以及组织业务开展方式的熟悉 • Evaluation skills in design, data collection, data analysis, and preparing reports • 设计中的评价技巧、数据收集、数据分析以及准备汇报 • Skills in the use of information technology • 信息技术中使用的技巧

31. Essential Competencies for Program Evaluators (ECPE)项目评价者的主要资质（ECPE） • See Table 12.1 in text • 参见 表12.1 • Includes these major areas: • 包括以下主要领域： • professional practice 专业经验 • systematic inquiry 系统咨询 • situational analysis 情景分析 • project management 项目管理 • interpersonal competence 人际技能

32. Team Leader团队领导 • If multiple evaluators are working on an evaluation, may have a team leader • 如果多个评价者在从事同一评价，需要一个团队领导 • Responsibilities • 责任： • supervise team members and manage day-to-day process • 监督团队成员，并管理日常事务 • act as mediator if there are dissenting voices within the evaluation team • 如果评价团队中出现不同意见，领导应能进行仲裁 • may write the report • 可以撰写报告 • make sure the team satisfies the TOR • 保证团队对于任务大纲是满意的

33. Client客户 • The person who officially requests the evaluation • 正式要求进行评价的人 • Best if this is the same person to whom the person reports • 最好与汇报人是同一个人

34. Managing Projects项目管理 • Managing time • 时间管理 • Managing scope • 范畴管理 • Managing people • 人员管理 • Managing money • 资金管理 • Managing resources • 资源管理

35. Project Management Process项目管理过程 Closing 结束 Controlling 控制 Executing 执行 Initiating 启动 Planning 计划 Skip to Managing People

36. Phase I: Initiating第一阶段：启动 • Projects begin by confirming there is a need for the project • 该项目的需求得到确认之时，即是该项目的启动之时 • Three Actions to be undertaken in this phase • 这个阶段需要进行三个行动 • Demonstrate project need and feasibility • 表明项目的需要和可行性 • Obtain project authorization • 获得项目授权 • Obtain authorization for the phase • 获得这个阶段的授权

37. Phase II: Planning第二阶段：计划 • Managers spend much of their time planning • 管理人员需要花很多时间来计划 • 13 actions in this phase • 这个阶段有13个行动 • Some steps are optional, depends on the size and scope of the project • 一些步骤是可选的，这取决于项目的大小以及范畴

38. Actions for Planning项目行动 • Describe the project scope • 描述项目范畴 • Define and sequence project activities • 定义次序以及项目活动 • Estimate durations for activities and resources required • 估计活动的持续时间以及需要的资源 • Develop a project schedule • 完成一个项目计划表 • Estimate costs • 估计成本 • Build a budget and spending plan • 完成预算以及预算使用计划 (continued on next slide)续下页

39. Actions for Planning(cont.)项目行动(续) • Create a formal quality plan • 制定一个正式的质量计划 • Create a formal project communication plan • 制定一个正式的项目沟通计划 • Organize and acquire staff • 组织及获取人员 • Identify risks and plan to respond • 确认风险以及应对计划 • Plan for and acquire outside resources • 计划以及获得外在的资源 • Organize the project plan • 组织项目计划 • Close out the project planning phase • 完成项目计划阶段

40. Phase III: Executing第三阶段：执行 • Manager makes sure the tasks are being completed • 管理人员保证任务完成 • One action to be undertaken: • 一个行动应该完成： • Execute project activities • 执行项目活动

41. Phase IV: Controlling第四阶段：控制 • Manager keeps track of people, activities, money, and scope • 经理对人员、活动、经费以及范畴保持沟通 • One action to be undertaken • 完成一个行动 • Control project activities • 控制项目活动

42. Phase V: Closing第五阶段：收尾 • Manager finalizes activities, archives information • 管理人员完成活动，对信息整理归档 • One action to be undertaken • 一个行动需要完成 • Close out project activities • 完成项目活动

43. Role of Evaluation Manager评价经理的角色 • Clearly describe the desired end project • 清晰的描述理想的完成项目 • Describe what it was that you liked about the relevant previous efforts • 描述你认为有关前期努力的优点在哪里 • Involve the evaluator(s) in the planning • 使评价人员参与到计划中 • Monitor the progress of the evaluation • 监测评价的过程 • Establish a timeline • 建立时间表 • Motivate the evaluator’s to produce their best • 激励评价人员发挥他们的最佳水平 • Avoid micro-managing the evaluator(s) • 避免对评价人员的管理过度 • Thank the evaluator(s) for their work • 感谢评价人员付出的努力

44. Managing People人员管理 • As a manager, you are managing people, NOT evaluations! • 作为经理，你管理的是人，不是评价！ • People are complicated. They are not machines. Their behavior will change from day to day. • 人是复杂的，他们不是机器，他们的行为会随时间改变 • The manager is responsible for everything that happens within the manager’s scope of authority. • 经理须对在其职权范围内所发生的一切负责

45. Working with Stakeholders同利益相关者合作 • Building trust • 建立信任 • Involving Directly Affected Stakeholders • 纳入直接受影响的利益相关者 • Seeking Feedback • 寻找反馈 • Involving the Voiceless • 纳入那些不发表意见的人 • Involving the Opposition • 纳入反对意见

46. Managing Tasks任务管理 • May be easier than managing people • 比人员管理容易 • Task map • 任务图 • everyone’s assignments • 每个人的任务 • start and completion dates • 开始以及完成时间 • Gantt chart • 甘特图表 • chart showing interrelationships of projects and schedules • 说明项目以及日程表之间关系的表格

47. Example of Task Map任务图的例子

48. Example of Gantt Chart甘特图表的例子

49. Scope of Work工作范畴 • Similar to TOR, they covers these tasks: • 同任务大纲类似，他们包括以下任务： • identify what is to be evaluated • 确认评价的对象 • provide a brief background on the intervention • 对于干预提供简短的背景 • identify existing data • 确认已有的信息 • state the purpose of the evaluation along with audience and use • 从评价目的、对象以及使用开始 • identify questions 确认问题 • specify the methods 确认方法 • discuss the composition of the team • 讨论团队的组成 • sets out the schedule and budget • 制订日程以及预算 Go ahead to Assessing Quality

50. A Good Evaluation:一个好的评价： • Meets stakeholder needs and requirements • 达到利益相关者的需要和要求 • Is relevant and realistic in scope • 范畴上相关且实际 • Uses appropriate methods • 使用合适的方法 • Produces reliable, accurate, and valid data • 产生可靠的、精确的以及有效的数据 • Includes appropriate and accurate analysis of data • 包括合适、精确的数据分析 • Presents impartial conclusions • 展示不偏不颇的结论 • Conveys results clearly — in oral or written form • 清楚地说明结果——以口头和书面形式 • Meets professional standards • 达到专业标准