Indicators, Evaluation and Performance Budgeting

评价与结果学习中心 Indicators, Evaluation and Performance Budgeting 指标、评价和绩效预算 Performance Budgeting Course 绩效预算课程 AFDC, Shanghai, October 2010 Prepared by Marc Robinson

This session 本节内容 • Key performance concepts关于绩效的几个重要概念 • Outcomes, outputs, activities/processes and inputs 结果、产出、活动/过程和投入 • Performance indicators绩效指标 • Evaluation评价

Performance Concepts 绩效相关的几个概念

Outcomes 结果 • Desired impacts of services:服务的预期影响 • Upon the individuals, social structures, or physical environment • 对个人、社会结构或自然环境的预期影响 • Examples:例子 • Improved health of patients 病人健康状况的改善 • Reduced crime 犯罪的减少 • Conservation of natural environment 对自然环境的保护 • Effectivenessan outcome concept 一个结果概念—有效性 High-level vsIntermediateoutcomes 高阶结果与中间结果 • Impactssometimes used for high level outcomes 影响有时是指高阶结果产出中间结果高阶结果

Outputs 产出 • Outputs = products产出=产品 • But in govt, most are services, not goods 但在政府部门，产出大多是服务，而非商品 • Services to external client or subject提供给外部客户或对象的服务 • Examples of outputs:有关产出的例子 • A school student who receives teaching受教育的学生 • A patients treated in a public hospital在公立医院接受治疗的病人 • Benefit payment administered管理的保险费 • Enforcement of legislation法律的强制执行 • Quantity, quality & efficiency measures数量、质量和效率衡量 • Outcomes & outputs = “results”结果+产出=成效

Inputs 投入 • Resources used to produce outputs用于产出的资源 • Human & organizational resources人力资源和组织资源 • Physical assets实物资产 • Materials and other inputs原料投入和其他投入 • Some purchased, some created internally有些是购买的，有些是内部生成的 • Workforce indicators员工指标 • Training conducted, staff turnover etc员工培训、员工流失率 • Asset indicators资产指标 • School classrooms per school-age child学校教室/学龄儿童

Activities and processes 活动和过程 • Activities produce outputs活动实现产出 • Support activities – human resources, IT... 支援活动—人力资源、IT活动 • Direct service activities – nursing直接服务活动—护理 • Processes: sequence of activities过程：活动的顺序 • Activity indicators 活动指标 • Job vacancies filled填补的空缺职位 • Pension applications processed处理的养老金申请 • Activities are not outputs活动不是产出 • Not capable alone of delivering outcome不能独自产生结果的活动 • Quantity, quality and efficiency measures数量、质量及效率的衡量

The Results Chain 结果链 High-level OUTCOMES 高阶结果 Intermediate OUTCOMES 中间结果 OUTPUTS 产出 ACTIVITIES/ PROCESSES 活动/过程 INPUTS 投入

Results Chain Example 结果链举例 OUTCOME Fewer people develop smoking-related diseases 结果因吸烟患病的人越来越少 Intermediate outcome Fewer people smoking 中间结果吸烟的人越来越少 Intermediate outcome Fewer people take up smoking 中间结果开始吸烟的人越来越少 Intermediate outcome People view smoking negatively 中间结果人们对吸烟持负面观点 Output (service) Awareness on health dangers of smoking 产出（服务）意识到吸烟对健康有危害

Ministry perspective 政府部门视角 • Outcomes and outputs delivered to external parties 提供给外部当事人的结果和产出 For individual ministries, this includes services to rest of government对单个政府部门而言，这包括对其余政府部门的服务 Not just to the community不是仅对本团体而言 • Civil service ministry example:例如公务员局 • Output: recruitment of civil servants产出：公务员的招募 • Outcomes: a better quality civil service, improved governance结果：更好的公共服务，更好的政府管理

Outcomes: some questions … 结果：一些问题…… What are the intended outcomes of: 预期的结果是什么 • Anti-pollution programs? 污染治理项目？ • School education? 学校教育？

Outputs: some questions … 产出：一些问题…… • Patient given anesthetic before operation手术前麻醉的病人 • Is an output provided?提供产出了吗？ • Roads and bridges道路及桥梁 • Are they outputs? If not, what is? 它们是产出吗？如果不是，那什么才是产出？ • Educational output: the best measure is: 教育产出：最好的衡量方法是： • Teaching hours? Numbers of students taught? 教学时间？受教育学生的数量？

Performance Information 绩效信息 • Performance Measures绩效测量方法 • Quantitative measures of performance定量的绩效衡量方法 • Performance indicators the same thing同样的绩效指标 • Evaluation评价 • Retrospective analysis of performance绩效的回顾性分析 • Interpretive, not merely measures解释，而不仅仅是测量方法 • Can use, for example, relevant theory例如，能使用相关定理 • Performance budgeting requires both绩效预算时,两者都需要

Performance Indicators 绩效指标

Measures vs. Objectives & Targets 衡量方法与目的和目标 • Performance measure:绩效测量方法 • Survival rate of lung cancer patients 2 year after diagnosis 肺癌确诊2年后病人的存活率 • Objective目的 • Reduce death and disability arising from lung cancer 降低肺癌引起的死亡和残疾率 • Target目标 • Increase the 2 year survival rate by at least one-third within the next decade 在未来十年内将确诊2年的肺癌病人的存活率提高至少三分之一

Criteria for good indicators 有效指标的标准 • Clearly linked to objectives 与目的有清晰的联系 • Comprehensible and unambiguous 容易理解而且非常清晰 • Reliable 可靠性 • Timeliness 及时性 • Perverse effects minimized 负面影响最小 • Cost effective 低成本

Perverse Effects 负面效果 • The more PMs matter, the more they: 绩效管理越重要，他们就 • May be manipulated 越有可能被操控 • May lead to perverse results越有可能导致负面结果 • Dumping, creaming, storming etc倾销、撇脂、震荡等 • Will explore throughout workshop 将通过专题研讨会探索 • Particularly in session on Targets尤其是在目标这个阶段 • Design should aim to minimize this 设计的目的应该是尽量减少负面效果 • But can never eliminate但不可能永远将其消除 • Consequence of inherent limitations of PMs是由绩效管理的内在缺陷导致的

Usable Performance Information 可用的绩效信息 • Too much PI can be a problem绩效信息太多可能会带来麻烦 • For budget decision-makers对预算决策者来说 • Limited, strategic information有限的战略信息 • Results indicators are primarily relevant for budget decisions结果指标主要与预算决策紧密相关 • Outcomes, outputs … 结果、产出…… • Activity, input indicators primarily of internal relevance 活动与投入指标具有内在关联性 • Measures presented in context具体情况中的测量方法 • Description of program performance对项目绩效的描述 • Drawing on evaluations etc对评价等方面的描述 • One page or so per program每个项目一页纸左右

Evaluation and Performance Budgeting 评价和绩效预算

Performance Budgeting & Evaluation 绩效预算与评价 • Role of evaluation under program budgeting: 项目预算下评估的作用 • Expenditure prioritization支出的优先顺序 • Which programs which should be cut back or eliminated? 哪些项目应该减支或淘汰？ • Pressure on agencies to perform对执行机构的压力 • By reviewing their budget requests in the light of their past performance in effectively using resources 根据机构过去实际使用资源的绩效来审查其预算请求 • Role in budget-linked performance targets system 预算相关的绩效目标体系的作用 • Assess how far the agency is responsible for any failure to meet target 对机构未实现目标而评估其应该负多少责任 • E.g. versus impact of external factors例如，有多少是外部因素影响而导致的

Relevant types of evaluation 评价的相关类型 • Outcome evaluations most relevant最相关的结果评价 • Ineffective programs are key candidates for cuts 没有产生预期效果的项目属于应该进行削减的首选 • Agencies with many ineffective programs should probably not be given more money 如果一个机构承担的项目很多没有产生预期效果，那么这个机构或许不应该得到更多的经费 • Process evaluations relevant to more limited degree 有限相关的过程评价 • Before cutting a program which is ineffective 在削减没有产生预期效果的项目之前 • Need to know whether could be effective if redesigned or better managed 需要知道重新设计项目或更好地加以管理是否能使项目有效 • But detail of process improvements not relevant for PB 但是对绩效预算来说，过程改善的细节是无关的 • In-depth process evaluations more relevant for internal management purposes对内部管理来说，深度过程评价更相关

Evaluations and Spending Review 评价和支出审查 • Spending Review (SR) 支出审查 • Systematic scrutiny of ongoing spending对正在现有支出进行系统的审查 • To identify cuts in low-priority & ineffective programs 确认哪些优先程度较低而且没有达到预期效果的项目需要削减 • And savings from efficiency enhancement 因效率提高而产生的结余 • Evaluation as key information input to SR 对作为信息关键而投入到支出审查中的评价 • Not the only input不是唯一的投入 • Performance indicators绩效指标 • Analysis of program priorities (not part of evaluation) 项目优先级分析（不属于评价的内容） • Selection of programs to be evaluated待评价项目的选择 • SR process often includes selection of programs for evaluation 支出审查过程常常包括评价项目的选择 • Programs suspected of being ineffective or low-priority 被认为效果小或优先级别较低的项目 • Annual budget-related evaluation program与年度预算相关的评价项目

Evaluation & the Budget 评价与预算 • Evaluations timing must fit in with budget preparation 评价的时机必须与预算的准备相一致 • Means evaluations usually need to be quick这就意味着评价通常要迅速 • E.g. commission at or before start of budget process 例如预算过程开始或之前的委托 • Results before final budget decisions taken 作出预算决定之前的结果 • Rapid evaluation is most relevant快速评价是最相关的 • Evaluation reports designed to tell budget decision-makers what they need to know 评价报告旨在告诉预算决策者他们所需要知道的 • How effective the program is项目的有效性如何 • Whether it is inherently ineffective 是否其内在因素决定了其无法达到预期效果 • Or could be effective with better implementation 或者是可通过更好实施才能有效

Relation of Evaluation to Budget Process 评价和预算过程间相关性 Performance Indicators 绩效指标 Budget-Linked Evaluations 预算相关的评价 Spending Review 支出审查 Budget Preparation Process 预算编制过程 Time 时间

Who conducts budget evaluations? 谁是预算评价的实施者？ • Purely internal evaluations not sufficient for budget纯粹的内部评价对预算来说是不够的 • Danger that these will be self-serving rather than objective 内部评价往往缺乏客观性，所以会导致为己牟利的危险 • Many are detailed process evaluations 许多都是详细的过程评价 • External evaluations needed外部评价是必需的 • Under MOF leadership在财政部的领导下 • Although ministry concerned may share direction尽管相关部委可能参与指导 • MOF will usually not carry out the evaluations itself 财政部通常不会自己进行评价 • Commission evaluations from experts由专家组成评估委员会 • Retired senior civil servants can be good evaluation leaders 退休高级公务人员将是称职的评估小组领导人 • Can’t necessarily expect ministry to “own” results 不一定指望政府部门“拥有”评价结果 • But that’s not the primary objective of these evaluations 但是这不是这些评价的首要目标

智利 Chile • System of evaluation linked to budget since 1997 从1997年开始建立与预算相关的评价体系 • Evaluations geared to budget preparation 面向预算编制的评价 • Started with desk (rapid) evaluations从桌面(快速)评价开始 • Later expanded to more detailed effectiveness evaluations 后来延伸到更详细的有效性评价 • Desk evaluations still dominate (13 out of 17 conducted in 2006) 快速评价仍旧占主导地位（2006年进行的17项评价中快速评价就有13项） • Evaluations delivered within 4-10 months历时4-10个月的评价 • Selection of programs for evaluation with budget in mind 根据预算仔细选择待评价的项目 • Proven impact on budget funding decisions 对预算分配决策的实际影响 • Finance ministry role财政部的职责 • Key role in selecting programs for review 在选择审查项目时具有关键作用 • Key role in selecting independent evaluators & managing evaluations 在选择独立评价人员和管理评价活动时具有关键作用

Evaluation and PB in Low Income Countries 低收入国家的评价和绩效预算 • Limited financial & skilled human resources 财力和技术人才的缺乏 • Need to be very selective in performance information 需要精心选择绩效信息 • Rapid evaluations should play dominant role 快速评价应该占据主要位置 • Because low cost 原因在于成本较低 • Availability of good performance indicators important 能否获得好的绩效指标是非常重要的 • Otherwise rapid evaluations have to rely entirely on evaluation of program logic否则快速评价必须完全依赖项目逻辑性的评价 • Evaluation capacity to be gradually built up 评价能力有待逐步建立

Conclusions 总结 • Results chain结果链 • Clear framework for PI绩效指标的清晰框架 • Choosing right performance measures选择正确的绩效衡量方法 • For budget decision makers对于预算决策者来说 • Relevance, reliability etc相关性、可靠性等 • Not too many不需要太多

Indicators, Evaluation and Performance Budgeting