Results Based Management Workshop Setting context for Leadership Performance Measurement

1. Results Based Management WorkshopSetting context for Leadership Performance Measurement Ging Wong Andrew Graham

2. Objective • Set the Project in context of Results Based Management developments in Canada and elsewhere • Provide an overview of the linkages of results, accountability, performance measurement and risk • Suggest ways that the Project can uniquely support the effective transfer of relevant and useful elements of RBM

3. ACCOUNTABILITY RISK PERFORMANCE MEASUREMENT

4. Flexibility and Control • Tension in management thinking and practice between structure/accountability and initiative/flexibility. • Challenge is how to create a balance that recognizes public sector accountabilities that create very high levels of expectation with the need to get the job done, i.e. meet the objectives of the program.

5. Flexibility and Control “While flexibility is becoming essential, the public is not willing to forsake accountability to achieve it. Confronted with antiquated systems of governance, managers must exercise leadership, not to in an ‘anything goes’ manner. In short mangers face the challenge of developing an alternative form of accountability that allows for greater flexibility of action.” Feldman and Khademian

6. Flexibility and Control • Public Sector accountability is complex and varied • It is changing with the role of public servants changed, becoming more involved • Understanding accountability in the public sector means understanding that there is a continuum of accountabilities: from operational to policy to political

7. Flexibility and Control • Reality that there are significant ‘gear slippages’ in public sector accountability in which the day-to-day at times takes on a much greater significance than so-called higher level accountabilities for policy outcomes • In the public sector it is more likely that a failed program can run really well and a successful one can founder on details

8. Defining the Concept of Accountability…. The basic notion of accountability is that those acting on behalf of another person or group report back to the person or group or are responsible to them in some way. - Owen E. Hughes, Public Management and Administration

9. Defining the Concept of Accountability…. “Accountability is about evaluating performance, meeting legitimate standards, fulfilling legitimate commitments, and holding responsible those who fail to meet the standards. The right to judge government performance flows naturally from the role of citizen, as does the right to sanction those who fail to meet the standards.” Janice Gross Stein, The Cult of Efficiency

10. Defining the Concept of Accountability…. • From a management perspective, accountability runs the full gambit of activities, from the specific to the general • In the public sector, this means from detailed administration to accountability for program outcomes • At no point is this either easy or simple. • There are no fire walls between the detailed and the general.

11. The Accountability Continuum “ Public mangers do not have the luxury of separating out expectations for flexible leadership from demands for strict accountability in the form of structures or guarantees that check and limit management action. They must grapple with both.” - Feldman and Khadamian

12. The Accountability Continuum INTERNAL ACCOUNTABILITY EXTERNAL ACCOUNTABILITY COMPLIANCE OUTPUTS MANAGEMENT OUTCOMES OPERATIONS MANAGEMENT PROGRAM COEFFICENTS COEFFICIENTS DATA KNOWLEDGE INFORMATION

13. Sustained accountability through results based management • Key trend in governments in Canada • Federal government and provinces have adopted various methodologies

14. Based on simple model of organizational learning and adaptation

15. RBM leads to consideration of key elements of a management framework • Effective statement of goals and outcomes • Performance measurement systems

16. RBM leads to consideration of key elements of a management framework • The role of monitoring and control in execution

17. RBM leads to consideration of key elements of a management framework • The role of evaluation in determining outcomes and the validity of the of the outcome expectations • Means to develop organizational learning across complex systems in both implementing and adapting policies (case studies, etc)

18. desirable properties of performance measuresPerformance measures have a number of desirable properties. Most of these can be dispensed with fairly quickly to focus on three key problems:relevance and balance
Performance measures such as those embodied in the IDTs are undoubtedly relevant. There are, however, two problems of balance. First is that the measures may not span an agency’s portfolio. For example, the health targets in DFID’s PSA exclude health activities outside of the top ten recipients. And the targets as a whole do not apply to activities in middle-income countries. The second problem of balance is that, for reasons discussed below, it is better to have a set of indicators spanning the range of the log-frame from inputs to outcomes.measures known, understood and trusted
Performance measures used in development are usually clearly defined and well understood, at least by those who work directly with them. There has been some tension between top-down setting of targets and a more participatory approach. This has also been a factor behind the promotion of the Millennium Development Goals (MDGs) as rivals to the IDTs. Data quality can be more problematic. The IDTs include maternal mortality, for which data are notoriously unreliable. The measure was included in DFID’s first PSA, but when it came to reporting DFID told the Treasury that data were unavailable. Country coverage for most indicators is patchy. But the more serious problem is the timeliness of the data, discussed below.affected and attributable
Changes in performance measures should be affected by the activities of the organisation and the extent of the effect measurable (attributable). However, attribution becomes harder to establish as we move through the log-frame from inputs to outcomes. As we saw above, USAID has abandoned claiming that changes in economic outcomes can be attributed to its work.achievable
Targets should be achievable, but not too easily. Most analyses suggest that the IDTs will not be met, although there is regional variation in performance. But this is not the same as saying that they could not be met if aid and government resources were better focused toward meeting them.linked to existing management systems
Organisations will already have in place management information systems. The rise of results-based measures was in part a response to the fact that existing systems focused on inputs and internal activities, such as spending and staffing, rather than achieving outcomes. But new results-oriented systems should not be separate or parallel to these existing systems. There are two further problems here: aggregation (can the various measures be added up to summarise performance for regions, sectors and the agency as a whole?) and alignment (do the data tell us about impact on the outcomes of interest?)

19. three key problemsThe above discussion identifies three problem areas. These problems are less severe (though present) when using RBM at the project level, becoming more intense as we attempt to measure results at country level or for the whole portfolio.data availability
Data become available only with a time lag, this being particularly so for developmental outcome measures like infant mortality and income-poverty. DFID’s PSA covering 2001-2004 includes amongst its targets 'a decrease in the average under-5 mortality rate from 132 per 1,000 live births in 1997 to 103 on the basis of data available in 2004'. Implicit here is the recognition that data available in 2004 will probably be for 2001, namely the start of the period of the current PSA. How can results of current strategies be judged on the basis of results achieved before the strategy was put in place?attribution
It is very difficult to attribute changes in developmental outcomes to the activities of an individual agency. This problem is encountered where agencies work together, as in sector programmes or providing budget support, or for agency performance at the country level. One agency’s support to public sector reform in an African country (providing the funds to pay for retrenchment packages) had as the outcome indicator: maternal mortality. It is difficult to imagine what sensible use could have been made of maternal mortality data (even with no time lag) to judge the performance of this programme.integration with existing management information systems
All donor agencies have some sort of monitoring and evaluation system at the project and programme level which should provide a basis for both feedback at the project level and 'feed-up' to management. But how do these systems tackle the problems of aggregation and alignment