University of Cape Town Department of Statistical Sciences

1. Monitoring and evaluating policy across the spheres and functions of government University of Cape Town Department of Statistical Sciences Funding was contributed by the Conflict and Governance Facility, a project of National Treasury, which is funded by the European Union under the European Programme for Reconstruction and Development

2. M&E • Monitoring: A continuing function that uses systematic collection of data on specified indicators to provide management/stakeholders with indications of the extent of progress and achievement of objectives • Evaluation: The systematic and objective assessment of an ongoing or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, efficiency, effectiveness, impact and sustainability. Adapted from OECD; http://www.oecd.org/dataoecd/29/21/2754804.pdf 17 Oct 2005

3. M&E • Motherhood and apple pie? Is there anything to discuss? • Does access to data lead to better information, lead to better decision making, lead to better performance?

4. Evidence of what? Monitoring involves selecting an indicator that can provide evidence of progress towards some objective (value judgements) Assumptions: causality and the links between actions and results. To provide evidence: • Articulate the framework within which monitoring will occur. • Clarify the logic that will be applied in making the claim for evidence (of progress).

5. M&E of Policy in South Africa: Alignment across the Spheres and Functions of Government Purpose: • To develop a critical appraisal of the current system of monitoring and evaluation established by government around the Programme of Action (POA) • To propose a set of indicators that can be used to appraise performance. • To investigate the alignment of M&E processes across national and provincial spheres of government.- • To investigate the existence of appropriate data sets that can be used to calibrate indicators.

6. Goal setting and prioritisation MCDA • Problem Structuring : defining the problem arena, including stakeholders • Specifying criteria (in M&E these could be referred to as objectives, goals or outcomes) • Development of alternative courses of action • Preference modelling • Assigning weights to the criteria

7. A strategic framework for M&E • Articulating Goals/ Values/ High level Objectives (or, put simply “what is the problem?”) • Developing a ‘theory of change’ on how interventions will tackle the problems within the problem context • Determining what activities will be undertaken to address the problems. • Outcomes: if all interventions are successful where will we be? The answer is the (medium to long-term) outcome. NB: Check that activities outcomes objectives/ goals.

8. Problem Ind Strategic M&E Inputs Theory of Change Ind Objectives/ Goals Activities: Programmes projects, etc Ind Ind Outputs Does outcome lead to goal being fulfilled? Outcome Ind

9. High level objectives for the Province A Home for ALL To enhance economic participation, shared growth and development To enhance spatial integration and provide sustainable service delivery To ensure the integrity of the physical environment To develop social and human capital To practice holistic governance and administratn

10. A set of desired outcomes for each objective

11. A Home for ALL To enhance economic participation, shared growth and development To enhance spatial integration and provide sustainable service delivery To ensure the integrity of the physical environment To develop social and human capital To practice holistic governance and administration Enhanced broad-based economic participation Enhanced provincial economic growth % black ownership of companies % real GDPR growth per capita p.a. Pro-poor, shared economic growth Skills profile which matches the needs of the economy Share of poorest quintile in provincial consumption (MDG). Adequate spatial integration and transport infrastructure to support a growing and integrated economy % of Science, Engineering and Technology graduates Significantly increased employment % of individual income spent on transport Unemployment rate

12. A Home for ALL To enhance economic participation, shared growth and development To enhance spatial integration and provide sustainable service delivery To ensure the integrity of the physical environment To develop social and human capital To practice holistic governance and administration High levels of social capital in communities Improved human capital and educational status Safe communities Basic services and adequate bulk infrastructure which leads to improved quality of life An improved health profile An effective social safety net Spatially integrated, sustainable human settlements An environment which promotes sustainable livelihoods A physical environment which enhances quality of life Sustainable access to safe water resources Sustainable access to safe energy resources Government operates in a holistic manner Constitutional rights are protected Government is effective, efficient and has good customer (citizen) relations There is political stability and peace There is democracy and accountability There is adherence to the rule of law Government is ethical and free of corruption Adequate spatial integration and transport infrastructure to support a growing and integrated economy Enhanced provincial economic growth Pro-poor, shared economic growth Enhanced broad-based economic participation Significantly increased employment Skills profile which matches the needs of the economy

13. National value tree

14. A society that truly cares & a better life for all (as in the Bill of Rights& Constitution) Reduce preventable causes of death-TB, diabetes, malnutrition, maternal deaths, HIV/AIDS, malaria (9) Complete tertiary infrastructure improvements (42) Recruit & retain health personnel Improve services in health facilities Ensure involvement of communities in local economic initiatives Comprehensive assistance to emerging farmers, expand agricultural support programs (37-39) Encourage labour intensive methods Deal with negative impact of casualisation Extend Expanded Public Works Program (28-29) Skills training Learnerships Education system geared for productive work Support micro-loan financing Reform support agencies Change to financial system and institutions Complete land restitution Continue land reform Provide efficient, safe transport (9,26, 52) Create a single & integrated economy that benefits all. Build a healthier nation Promote social cohesion (14) & a non-racist, non-sexist, democratic society Improve safety & security Help consolidate peace, security & development in southern Africa, Africa & globe & Position SA as an effective global player Broaden access to services & consti-tutional rights and provide good governance Reduce unemployment Continue economic growth (6, 14, 22) Alleviate poverty Strengthen democracy esp. Zimbabwe, DRC, Angola, also Congo,Swaziland also Tanzania, Kenya, Burundi, Rwanda, Ghana, Senegal, Ivory Coast, Sudan, Ethiopia, Somalia, Mahgreb) (60, 61) Facilitate economic activity within the second economy, Pro-mote involvement of marginalised in economic activity (27-39) Pursue multilateral global approach. Democratise UN, IMF, WB, etc. Improve relations with China (64) Facilitate creation of jobs, skills and work experience (27,29,32-36,43,45) Improve opportunities for self-employment & Promote sustainable community livelihoods (27) Provide comprehensive social security (10) and ensure people access them (6) Reduce Corruption (10) Improve provision of information (49) Economic cooperation in southern Africa Realise NEPAD, strengthen AU, SADC (62) Efficient logistics and infrastructure (59) Cooperation with industrialised nations Promote growth of small & micro-enterprises (30,31) Facilitate broad-based BEE Provide the skills re-quired by the economy (27,29,32-36,43,45) Improve household & community assets Build community, good citizenship, solidarity Reduce crime & m. death from crime Improve state capacity for growth & development (47) Better interaction between govt& people (10, 50, 71) Better inter-govt. cooperation Better access to govt information Strengthen democratic institutions incl. courts Improve services in government Create new jobs Ensure low interest & inflation rates Speed up rural development & urban renewal Increase public & private of investment (24-26) Invest in road, rail & air transport, telecommunications & energy Improve efficiency of justice system (9,55) Address social roots of criminality Reduce death from road accidents Integrate traditional leadership Better implementation, monitoring, evaluation Create a compassionate govt service (9) Cooperation with South Strengthen SA’s Early Warning Mechanism Speed up delivery of water, sanitation, phones, health, electricity, housing, education Improve local govt capacity (48) Multipurpose centres, e-govt, etc. (49) Provide free basic water & electricity Provide efficient, safe transport (9,26, 52) Allocate more resources to welfare grants for 3 yrs (44) Improve UIF Ensure people access grants All children with decent classrooms Spend in favour of students from poor households Expand school nutrition program. Introduce national health insurance Expand primary school nutrition progm (43) Expand primary health care (42) Comprehensive care & treatment of HIV & AIDS. Reduce teacher/pupil ratio Free health service to disabled Protect borders (illegal immigration, crime, terrorism, marine resources) More, better trained police Improve anti-corruption structures Strengthen prosecution & Scorpions, special courts Intervene in crime areas-enhance normal communal life (53) Reduce awaiting trial numbers Improve coordination among relevant agencies Provide efficient, safe road &rail transport (9, 26, 52) Build more subsidised housing, medium den-sity housing closer to work & provide serviced stands

15. A society that truly cares & a better life for all (as in the Bill of Rights& Constitution) Create a single & integrated economy that benefits all. Build a healthier nation Promote social cohesion & a non-racist, non-sexist, demo-cratic society Improve safety & security Help consolidate peace, security & development in southern Africa, Africa & globe & Position SA as an effective global player Broaden access to services & constitutional rights and provide good governance Reduce unemployment Continue economic growth Alleviate poverty

16. A society that truly cares & a better life for all (as in the Bill of Rights & Constitution) Create a single & integrated economy that benefits all. Reduce unemployment Create new jobs Facilitate creation of jobs, skills and work experience Promote growth of small & micro-enterprises Improve opportunities for self-employment & Promote sustainable community livelihoods Facilitate economic activity within the second economy, Pro-mote involvement of marginalised in economic activity Comprehensive assistance to emerging farmers, expand agricultural support programs Ensure involvement of communities in local economic initiatives Encourage labour intensive methods Deal with negative impact of casualisation Extend Expanded Public Works Program Education system geared for productive work Change to financial system and institutions Provide efficient, safe transport Reform support agencies Skills training Learnerships Support micro-loan financing Complete land restitution Continue land reform

17. A society that truly cares & a better life for all (as in the Bill of Rights& Constitution) Build a healthier nation Complete tertiary infra-structure improvements Improve services in health facilities Recruit & retain health personnel Reduce preventable causes of death-TB, diabetes, malnutrition, maternal deaths, HIV/AIDS, malaria Expand primary school nutrition program Comprehensive care & treatment of HIV & AIDS. Expand primary health care

18. A society that truly cares & a better life for all (as in the Bill of Rights& Constitution) Broaden access to services & constitutional rights and provide good governance Improve state capacity for growth & develop-ment Better interaction between govt & people Better inter-govt. cooperation Integrate traditional leadership Better implemen-tation, monitoring, evaluation Better access to govt information Improve provision of information Strengthen democratic institutions incl. courts Improve services in govt Create a compassio-nate govt service Reduce corruption Improve local govt capacity Multipurpose centres, e-govt, etc.

19. A society that truly cares & a better life for all (as in the Bill of Rights & Constitution) Create a single & integrated economy that benefits all. Reduce unemployment Alleviate poverty Improve opportunities for self-employment & Promote sustainable community livelihoods Improve household & community assets Speed up delivery of water, sanitation, phones, health, electricity, housing, education Comprehensive assistance to emerging farmers, expand agricultural support programs Provide free basic water & electricity Complete land restitution Continue land reform Links to GW M&E system Build more subsidised housing, medium density housing closer to work& provide serviced stands The following is the heading and definition for ALL of the indicators mentioned on this page: Housing: These indicators measure the marginality of human living conditions (e) Proportion of HH with access to secure tenure (owned or rented) (Target 10 Indicator 32) (a) Proportion of population with access to an improved water source (both urban and rural) (Target 10 Indicator 30) (b) Proportion of population with access to improved sanitation (Target 10 Indicator 31) (c) % of gdp spent on sanitation infrastructure (d) Proportion of households with access to electricity (a) Percentage of population housed in informal/formal dwellings. (b) Percentage of the population which is homeless.

20. A society that truly cares & a better life for all (as enshrined in the Bill of Rights, constitution) Reduce preventable causes of death-TB, diabetes, malnutrition, maternal deaths, HIV/AIDS, malaria (9) Complete tertiary infrastructure improvements (42) Recruit & retain health personnel Improve services in health facilities Create a single & integrated economy that benefits all. Build a healthier nation Reduce unemployment Continue economic growth (6, 14, 22) Alleviate poverty Facilitate economic activity within the second economy, Pro-mote involvement of marginalised in economic activity (27-39) Facilitate creation of jobs, skills and work experience (27,29,32-36,43,45) Improve opportunities for self-employment & Promote sustainable community livelihoods (27) Provide comprehensive social security (10) and ensure people access them (6) Promote growth of small & micro-enterprises (30,31) Facilitate broad-based BEE Provide the skills re-quired by the economy (27,29,32-36,43,45) Improve household & community assets Create new jobs Ensure low interest & inflation rates Speed up rural development & urban renewal Increase public & private of investment (24-26) Invest in road, rail & air transport, telecommunications & energy Speed up delivery of water, sanitation, phones, health, electricity, housing, education Provide free basic water & electricity Provide efficient, safe transport (9,26, 52) Ensure involvement of communities in local economic initiatives Comprehensive assistance to emerging farmers, expand agricultural support programs (37-39) Allocate more resources to welfare grants for 3 yrs (44) Skills training Learnerships Education system geared for productive work Complete land restitution Continue land reform Improve UIF Ensure people access grants Encourage labour intensive methods Extend Expanded Public Works Program (28-29) Change to financial system and institutions All children with decent classrooms Spend in favour of students from poor households Expand school nutrition program. Introduce national health insurance Expand primary school nutrition progm (43) Expand primary health care (42) Comprehensive care & treatment of HIV & AIDS. Deal with negative impact of casualisation Support micro-loan financing Reform support agencies Provide efficient, safe transport (9,26, 52) Reduce teacher/pupil ratio Free health service to disabled Build more subsidised housing, medium den-sity housing closer to work & provide serviced stands

21. Measurement issues • Simple vs composite indicators • Parsimonious models • Robust or sensitive to change • Local or global views • Error (measurement, sampling, statistical bias, interpretation) • Measuring (step 1) and valuing (step 2)

22. Some conclusions… • Pervasive lack of clarity on high level objectives… value trees might be a useful tool • Priority. What’s that? • Danger of over-engineering • Danger of separating M&E from strategic planning (activities may not be aligned with desired outcomes) • Tremendous benefits of awareness created by MTEF…. but • Lack of quantitative/ problem solving/ modeling skills

23. Strategic M&E There is a strong need for a common strategic planning process which is individually tailored by each department/ govt entity within the context of clear national, provincial, local objectives.