Evaluation fThe implication of Performance Monitoring and or Government and Government systems

1. Evaluation fThe implication of Performance Monitoring and or Government and Government systems The Presidency, Department of Performance Monitoring and Evaluation Dr Solomon T. Bhunu Chief Director-Business Systems & IT SolomonB@po.gov.za

2. Content of presentation 1. Background 2. Role of DPME 3.Outcomes Approach 4. Way-Forward

3. Background Services not integrated across agencies ( Direct Impact) • There are no effective systems for coordination, • Lack of incentives for coordination, • Competition between agencies e.g Local & District municipalities, • Lack of rigorous whole-of-government planning • Inappropriate delivery mechanisms in some cases ( Indirect Impact) • Lack of awareness of different models, • Weariness of trying different models, • Inadequate use of new technology,

4. ……background Inappropriate things are being done ( Direct impact) • Lack rigorous planning e.g. poor theory of change, • Lack of focus on key priorities/ drivers of change, • Lack of feedback on what is working or not. • Not using services from other stakeholders appropriately,

5. …….background Inadequate data to inform management ( Direct Impact) • Lack of culture of using data to improve performance, • Suitable data not available • Staff not performing effectively ( Indirect Impact-outcome 12) • Lack of technical skills in public service • Incentives for not taking risk does not improve implementation • Bad implementation of management system • Lack of accountability ( Indirect Impact)

6. ……background Initial M&E thoughts Post 1994 massive policy overhaul Gradual awareness of M&E systems M&E practices inconsistent, quality uneven • Ensure transparency and accountability • Promote service delivery improvement • Ensure compliance with statutory and other • requirements • Promote the emergence of a learning • culture in the public sector

7. ……background Cabinet approved an Implementation plan to develop an M&E system across the whole of government Cabinet Memo 2005 Principles Monitoring Evaluation Early warning Verification Data collection Analysis Reporting Based on standard, shared indicators - dashboard Implementation plan Composite system Extractive Built over time Roll-out

8. ……background Period of hiatus Assumption that Departmental M&E systems will allow for extractive M&E system • Did not take complexity of inter governmental system (sub national) into account • Ignored extensive monitoring work in Treasury on budget performance measures and accountability cycle • Uninformed position on institutional alignment & coordination • Need specific capacity in place (not recognised)

9. ..background PRESIDENCY Dev Ind PoA DPSA NATIONAL TREASURY NATIONAL SECTOR DEPT OTHER NATIONAL DEPT PREMIERS OFFICE PROVINCIAL TREASURY Human resource info Output & outcome info Financial info PPI info Policy outcome info PROVINCIAL DEPARTMENT PoA During hiatus ?planned other info • Systems initiated in response to specific needs • Hard working, but not smart working systems • Smart analysis to align information

10. …..background • Shift from Task Team to Coordinating Forum, chaired by Pres • Close cooperation Treasury • Members: Treasury, DPSA, StatsSA, DPLG(CoGTA), PALAMA, AG (Premier’s Office, DoE) 2007 New direction • Simultaneous creation of two aspects • Create policy platform that can provide direction to M&E environment • Produce real M&E products for national oversight

11. The Role of Department of Performance Monitoring and Evaluation (DPME) The mandate of the Department of Performance, Monitoring and Evaluation is (DPME) is derived from Section 85 (2) (c) which states that the President exercises the executive authority, together with the other members of the Cabinet, by coordinating the functions of state departments and administrations. 2. Performance monitoring of individual departments Carry out PM&E of individual departments for the President; this includes assisting the President with the management of performance agreements of individual Ministers MAJOR MANDATES • 1. Outcomes approach • To coordinate and facilitate the development and ongoing performance monitoring and evaluation (PM&E) of the prioritised whole-of-government outcomes • To ensure that interventions take place where necessary to unblock bottlenecks in implementation

12. ………….the Role of DPME 3. General building of capacity for M&E across spheres, and use of M&E (M&E of both policy and implementation) • Custodian of Government-Wide Monitoring Evaluation System (GWMES) • Build capacity of M&E units across government • Ensure that capacity of line functions to carry out M&E is developed across government • Build management culture of using M&E information to effect change • Use of all types of M&E information to identify and implement key service delivery interventions • Formulate service delivery interventions based on results of all types of M&E • Ensure that interventions are implemented

13. ……..the Role of DPME Other MANDATES • A. Participating in the development of an African M&E network • Share experiences with other African countries • Build the network • Advisory support to President, Deputy President and Minister (M&E related issues) • C. Adhoc support to President, Deputy President and Minister (M&E related Issues)

14. The outcomes approach(Results Based Management)

15. Monitoring Traditional monitoring RBM monitoring • RBM monitoring adds a new emphasis: monitoring the outcomes,i.e. the consequence of having delivered outputs Focuses on implementation monitoring: • Tracking inputs (resources, strategies), activities (what actually took place), and outputs (products or services produced) • Monitoring how well a programme, project or policy is being implemented • Assessing compliance with workplace and budget Intent: to inform the efficiency of implementation and to incorporate lessons learned into decision-making

16. Are things being done right? Are the right things being done? Evaluation Formative evaluations are conducted during the implementation of an initiative and focus on the efficiency thereof Summative evaluationsare conducted after the implementation of a developmental initiative and focus on the effectiveness thereof

17. Secret ….the Outcomes Approach Outcomes Outputs Inputs Activities I n p u t - o u t c o m e m o d e l Step1 Step2 Step 4 Step3 How much do we need to achieve the best mix of desired outcomes Which priority outputs should we measure to see if we are achieving each outcome What are the key outcomes that Government wants to achieve Where should the system focus in order to achieve the outputs M o n i t o r i n g and E v a l u a t i o n • Derived from Election Manifesto • MTSF Measure outputs to test whether we are making progress • Determine optimal allocation of inputs • Reallocate Inputs if necessary Measure Delivery of key activities

18. Outcomes-Logic Model Ultimate Outcomes Intermediate Outcomes Immediate Outcomes Outputs Responsibility for execution Activities Inputs Manage/ influence to achieve these Contribution towards outcomes Manage & budget for these

19. 1. Problem Analysis: a clear understanding of the problem assumptions about causes and their relationships to effects are explicit ensure that the problem is understood from the point of view of the needs and concerns of the intended beneficiaries analytical tools such as problem trees can assist with problem analysis 2. Theory of Change: a clear understanding of key levers of change where do we place our focus, what will have the biggest impact? ‘theory of change’ based on the best available knowledge about causes and effects state assumptions clearly stated so that we can use evidence from M&E to test it through experience and build reliable knowledge about what works in what circumstances ….the Outcomes Approach Four guiding principles

20. 4. Intervention Logic: chain of logic assumptions about what results must be achieved to achieve the outcome, how they will be achieved and what resources will be necessary. 5.Clear indicators, baselines & targets: clear basis for monitoring progress and evaluating results. indicators for all levels check progress along the whole chain of delivery. each indicator should have a clear baseline, and targets and timelines should be clearly defined indicators must be measurable ….the Outcomes Approach Four guiding principles (continued)

21. ….the Outcomes Approach The Government 12 Outcomes 1. Quality basic education 2. A long and healthy life for all South Africans 3. All people in South Africa are and feel safe 4. Decent employment through inclusive economic growth 5. Skilled and capable workforce to support an inclusive growth path 6. An efficient, competitive and responsive economic infrastructure network 7. Vibrant, equitable, sustainable rural communities contributing towards food security for all* 8. Sustainable human settlements and improved quality of household life 9. Responsive, accountable, effective and efficient Local Government system 10. Protect and enhance our environmental assets and natural resources 11. Create a better South Africa, a better Africa and a better world 12. An efficient, effective and development oriented public service and an empowered, fair and inclusive citizenship**

22. Development of high level outcomes, outputs, activities and metrics Develop and implement detailed inputs, outputs, activities, metrics and roles and responsibilities Ruling Party election Manifesto: 5 priority areas Establish Implementation Forum MTSF: 10 strategic priorities Negotiate detailed inputs, activities, metrics and roles and responsibilities Step 1 (Done) 12 strategic outcomes (based on consultation process) Step 3 (July / August 2010) Delivery Agreements between stakeholders • Performance Agreements with Minister(s) • Based on outcomes • High level outputs, indicators, targets and activities per outcome • Request to work together in Implementation Forum • to produce a Delivery Agreement per outcome Coordinate implementation Step 4 ongoing Monitor and evaluate Step 2 (Done) Feed back loop to annual revisions of Delivery Agreements

23. ….the Outcomes Approach Sector Dept Derived information system Transversal systems GWM&E System: Data terrain Conceptual clarity Evaluations Social, Economic & Demographic Statistics Registers and Admin data PERSAL BAS Programme Performance Information DPLG DWAF

24. …the Outcomes Approach GWM&E System: Policy platform Evaluations Evaluation policy GWM&E framework Guidelines Social, Economic & Demographic Statistics Registers and Admin data SASQAF Programme Performance Information Common definitions PPI framework

25. Policy platform established 2007 Pres: GWM&E framework – system goals, describes constituent components and institutional location, role of all civil servants, principles Guidelines for Premiers Offices Guideline on annual M&E plans “ policy vision to operational reality” Treasury: Programme Performance Information framework – role of M&E in plannning and budgeting, link to dept performance plan, AR StatsSA: SA Statistical Quality Assurance framework – evaluation of statistics used by dept Public Sector Training Agency curriculum - M&E training secure footing DPLG(CoGTA): Draft LG framework Common definitions document Data Forum project …the Outcomes Approach Policy Platform formalised Status quo

26. ….the Outcomes Approach Sector Dept PoA GWM&E : Products / methodologies Evaluations Social, Economic & Demographic Statistics Registers and Admin data Development Indicators Programme Performance Information

27. …the Outcomes Approach GWM&E : Products / methodologies Status quo Cross cutting M&E products • Development Indicators – annual 76 Indicators progress of society • PoA–priority programmes progress reported to cabinet bi-monthly • Public Mang Watch – 13x HR indicators, 2 Financial, 1 audit, weighted to provide composite index • Annual performance plans and service delivery targets - 500x indicators - monitored quarterly • IGR publication of non financial information • OPSC reports – Annual State of the Public Service Report • AG office – audits of performance information • ‘Functionality report’ • GIS platform (uncoordinated)

28. …the Outcomes Approach PoA (Program of Action) • Annual State of the Nation – priorities are announced • Web based system 300 users • Bi-monthly report to Cabinet • Established a culture of reporting

29. ….the Outcomes Approach

30. ….the Outcomes Approach

31. ….the Outcomes Approach

32. ….the Outcomes Approach Conceptualised POA Workflow PROVINCIAL NATIONAL PME Concurs & updates POA POA trigger PREMIER Depts, Forums, Clusters Health Education Human Settlements updates POA validation POA updates trigger Publish Concurs

33. Political leadership and championing of M&E Incentives for M&E Top down and bottom up convergence Monitoring before evaluation Information and data constraints Capacity building Ownership of M&E system by line ministries International Comparison ….the Outcomes Approach

34. Coordination on M&E design between core national departments Different conceptualisation of role of state Different paradigms of public sector reform Decentralised approach vs national direction on policy monitoring Sector experts vs overarching direction Naïve view that single decision point in government that will decide on all indicators Deep dialogue approach Fault lines ….the Outcomes Approach

35. Frameworks set out requirements and expectations … but workplace culture remains stagnant Instances of ‘malicious compliance’ Information sharing Negative reaction to incorrect data Govt buying data that should be available from sister departments Evaluation policy lag Focus on M&E – emphasis on management information Some evaluation, not institutionalised Fault Lines ….the Outcomes Approach

36. Complexity of sub national government Policy making separate from budgeting and implementation Guidance to Provincial Premier’s Offices Fault Lines ….the Outcomes Approach

37. Appropriate balance between design and evolution of system Original design not adequate Lessons Leant …the Outcomes Approach • Coordination essential • Not all users have same need • Necessary tension between line ministries specialist view and overarching strategic discussion • Create dialogue, discussion, coordination, good practice

38. Dual implementation strategy Good policy platform But… also some real products M&E is a management system not an IT system Change management Resistance inevitable Takes about 3 budget cycles to bed reform down Target appropriate people Art & Science Pragmatic response that speaks to political, developmental and government context Lessons Leant ….the Outcomes Approach

39. Way-Forward • PME will work with Various stakeholders in the Establishment of an M&E IT Framework • POA will further be developed into a robust M&E system that will be a “system ” of “systems”. • DPME through various structures e.g. Data forums, M&E networks etc, etc, will work closely with all partners in achieving success to M&E culture and improved Service delivery

40. Acknowledgements 1. SITA...........PoA technical Support 2. TAU (Technical Assistance Unit).........Results Based Management support 3. Dr Sean Philips......... Director General DPME 4. Ms Ronette Engela......Deputy Director General DPME

41. KeyalebogaKe a lebohaKea lebogaNgiyabongaNdiyabulelaNgiyathokozaNgiyabongaInkomuNdikhoulivhuhaDankie Thank you