Distinguishing Inputs, Outputs, Outcomes and Impacts

1. Distinguishing Inputs, Outputs, Outcomes and Impacts www.summerinstitute.eu

2. Important points to start with • Decision making • Policy analysis • Limited rationality • Incentives • Process and learning

3. Contents • Outputs & inputs • Outcomes • Indicators • Logic model • Budgeting & PI • Future

4. Contents • Outputs & inputs • Outcomes • Indicators • Logic model • Budgeting & PI • Future

5. Outputs - definition • Services or goods which are provided by ministries and other state organizations to external stakeholders / beneficiaries (directly or through other organizations) • External stakeholders include: general public, citizens, businesses, NGOs, media and other state bodies including the minister, other ministries and state organizations, and the parliament

6. Examples • Provision of prosthetic devices to disabled • Provision of schooling • Health centre services • Analysis and policy advice • Laws • Inspections • Training

7. Outputs should not be • Activities • Processes • Inter-mediate output • Capacity building initiatives

8. Example: output – schooling • Number of graduates from school • Activity - teaching • Process - recruitment of teachers • IMO – Number of enrolled chidlren • Capacity building – increasing professional qualification of teachers

9. Consumption or provision? • Economic theory view: the delivery of service takes place between producer and consumer, and therefore transaction is complete, when service is used (output consumed) • Theatre plays • Patients checked • Public administration view: for many outputs it does not mater whether they are consumed or not – the service should be there in case it is needed • Emergency services • Prison services

10. Output characteristics • Volume or quantity of provision • Timeliness in provision • Quality of provision • Satisfaction - surveys and evaluation • Comparison – to others or standard • Coverage, meting of demand and accessibility • Equity

11. Example: output – provision of prosthetic devices • Volume – number of disabled people served • Timeliness – service provided in specific time – for example, one week after request • Quality – satisfaction of users of devices • Coverage – % of people in need of devices served • Equity – people with access difficulties served at home or transported to the service point

12. Selectivity • Not all output characteristics can/should be used for all outputs – be selective • Focus on key outputs for external users • For all other outputs – have at least the volume

13. Aspects to consider • Control of provision • Other government interventions • Measurability • Attribution • Users • Incentives & gaming

14. Control of provision • Provision of a single output is controlled (directly or indirectly) by government organization • But not its effectiveness (achieving desired policy objective). Many government outputs make sense only when used together with other outputs • Road safety and reduction of fatal accidents depends ona group of outputs – quality of road, speed control, technical state of cars, use of seat belts.

15. Control of provision (contd.) Many organizations Policy Budgeting Delivery One or more organizations One organization

16. Other government interventions • To implement policies governments provide not only goods and services, but also: • regulation • capital investments • social and various other benefits & compensations • subsidies • If these are not outputs, how do they relate to them?

17. Other government interventions(contd.) • It can be helpful to think of three broad categories • goods and services = outputs • administered items • capital investment • mandatory benefits & compensations • subsidies and other payments • general regulation • But remember – most of administred items and general regulation require outputs

18. Example – road safety • In budgets, some countries separate outputs / investment / financing in separate programs while others do not (but separate using economic classification)

19. Measurability • Some outputs are harder to measure than others due to the nature of organizations providing them Measurement difficulty Embassies Army Residential care Ambiguity (output) Road police Job Counselling Training institutions Unemployment agencies Policy making ministries Routine (process)

20. Attribution • Outputs mean something if they can be attributed to some objective (outcome) – they contribute to achieving of objective, e.g. Change in outcome • Generalized attribution: re-training of unemployed contributing to reduction of unemployment • Specific: re-training of unemployed contributing to employability of the trained person during the next year after training

21. Difficulties in establishing attribution Clear attribution: • Fixed arm • Medical service • Broken arm Less clear attribution: • Survival • Services • Cancer • Earlier diagnosis? • Better treatment? • Healthier life style? • Beneficial effects of affluence?

22. Users • Different PI users and decision making stages might have different information needs (and therefore require different formats of presentation of PI) Parliament Government Ministry of Finance Minister Permanent Secretary Unit Manager Employee Output focus • Using of PI means analysis and learning rather than money and / or promotion

23. Principal – Agent Problem • Principals are “owners” and “bosses”, but they lack time and expertise to do everything • Therefore principals hire agents, who have time and expertise, but are self interested Modern Medieval Business Principal King Shareholder People Agent Lords CEO Bureaucrats Delegated authority Tax collection Profit making Service delivery

24. Principal – Agent Problem(contd.) • Agents have one strong advantage over principals - information • This creates problem called “gaming”, which is based on two factors: • There is information asymmetry between the principal and agent; and • There are wrong incentives for disclosing or hiding information

25. Gaming & incentives • Bevan and Hood: "reactive subversion such as 'hitting the target and missing the point' and / or reducing performance where targets do not apply" • Gaming = manipulation with information that might affect service delivery • Incentives = factors (financial or non-financial) that provide motives for a particular course of action, or count as a reason for preferring one choice to the alternatives • Targets are the most common causes for perverse incentives

26. Gaming & incentives (contd.) High Gaming potential Information asymmetry Low Loose Tight Relationship with decision-making

27. Impact of gaming Strategy A Strategy B Strategy C Strategy D Arrival on time at final destination “Saboteur” manipulation of service and data “Smart Liar” or manipulation of data “False Champion” or service manipulation “Honest Fool” or no manipulation TARGET

28. Impact of gaming (contd.) • Gaming can result in: • Cherry picking – handling of easy cases, patients etc. at the expense of difficult ones • Ratchet effect – slowing down today to be on time tomorrow • Distortions – improving areas where measurement is dome while neglecting other areas • Threshold effect – focusing on 2nd best and forgetting about the best and worst, as best will be achieved anyway while worst require too much of an effort

29. Dealing with gaming • Learning and data improvements • Avoiding strong negative incentives • Clarify use of PI in advance • “Loose” rather than “tight” connection to decisio making • Provide feedback on data relevance and quality • Reward those with good PI • Do not overload with data – use the key measures

30. Dealing with gaming(contd.) • Dealing with principal – agent problem at all levels • Use of “auditors” • National Audit Office • Internal audit • Encourage ownership at production level

31. Summarizing about outputs • Goods and services • Basic element in performance information system • Can be measured in different ways, but there are difficulties • Thee are three most common difficulties: • Measurability • Attribution to objectives (outcomes) • Scope for manipulation • Loose link between output data and decision making generally is a better incentive to avoid gaming

32. Inputs - definition • Inputs are resources used in the delivery of goods and services • Labour and other means of service provision (facilities, computers, teachers, books etc.) • Money is not an input. It s a cost of input

33. Inputs – output combination More outputs V ? Less inputs More inputs ? X Less outputs

34. Improving performance More outputs Production possibility frontier Less inputs More inputs Less outputs

35. Contents • Outputs & inputs • Outcomes • Indicators • Logic model • Budgeting & PI • Future

36. Outcomes - definition • The effects which government is trying to achieve for the public. They are often expressed as improvement in the living conditions of people or favourable changes that contribute to these changes • Outcomes are changes in the economic, physical, social and cultural environments which the state agency(ies) is trying to influence through provision of goods and services, general regulation and financing • Outcomes are those events, occurrences, or conditions that are the intended or unintended results of government actions (OECD)

37. Examples • reduction in maternal / child mortality • increase in export • decrease of unemployment • reduced crime levels • improved literacy rate of the population • reduction in fatalities from road accidents • land productivity improvement

38. Examples (contd.)

39. Examples (contd.)

40. Examples (contd.)

41. State Measures and Change LATVIA 2004 2005 change % Number of road accidents 48912 47353 -3.2 Number of injured 6416 5600 -12.7 Number of fatal injuries 442 516 -14.3 Number of registered cars 966242 -7.6 898145

42. State Measures and Change(contd.) • the same data can be called state measure or performance measure • it depends whether you attribute policy (outputs, regulation and finances) to it or not • if you do attribute – you are interested in change influenced by your policy (the question – whether different policy generates different result, e.g. marginal changes in result depending on inputs) • + - 0

43. State Measures and Change(contd.) LATVIA 2004 2005 change % Number of road accidents 48912 47353 -3.2 Number of injured 6416 5600 -12.7 Number of fatal injuries 442 516 -14.3 Number of registered cars 966242 7.6 898145 • outputs attribution of policy to change • financing • regulation

44. Different outcomes – results & impacts Impact or longer term effects (FO) Global objectives Results or direct & immediate effects (SO) Specific objectives Operational objectives Outputs Activities Inputs

45. Results & impacts defined • Results relate to the direct and immediate effect brought about by a policy or program. They provide information on changes to, for example, the behaviour, capacity or performance of direct beneficiaries. Such indicators can be of a physical (reduction in journey times, number of successful trainees, number of roads accidents, etc.) or financial (leverage of private sector resources, decrease in transportation cost) nature • Impactsrefer to the consequences of theprogram beyond the immediateeffects on its direct beneficiaries. Two concepts of impact can be defined. Specificimpacts are those effects occurring after a certain lapse of time but which are, nonetheless, directly linked to the action taken. Global impacts are longer-term effectsaffecting a wider population

46. Examples Tax collection Economy Justice Inputs Inputs Inputs inputs Tax audits Regulatory burden studies Prisoner rehabilitation outputs Reduced tax evasion Better regulation Re-integration into community results or SO Reduced shadow economy Enhanced competitiveness Reduced repeat crimes impacts or O

47. Results & impacts defined(contd.) • Results = short to medium term outcomes = specific outcomes • Impacts = medium to long term or final outcomes = overall outcomes • Sometimes immediate, intermediate and final outcomes

48. Single outcomes and indexes • Sometimes indexes are used to describe and compare performance • Outcomes from several areas and sometimes even sub-areas are merged into one index • These are of particular use for policy planners • For example: the Enabling Trade Index

49. Example: The Enabling Trade Index 2007/8 MARKET ACCESS T&C INFRASTR. BUSINESS ENVIRONM. BOARDER ADMIN. • Regulatory environment • Availability & quality of transport infrastructure • Efficiency of customs administration • Tariff & non-tariff barriers • Proclivity or openness to trade • Physical security • Availability & use of ICTs • Efficiency of import – export procedures • Transparency of boarder administration

50. Example: The Enabling Trade Index 2007/8 (contd.)