Monitoring and Evaluation: Frameworks

1. Monitoring and Evaluation:Frameworks

2. Learning Objectives At the end of the session, participants will be able to: • Identify and differentiate between conceptual frameworks, results and logical frameworks, and logic models • Design goals and objectives for specific intervention programs • Design frameworks for specific intervention programs • Discuss how frameworks are used for M&E planning

3. Module Outline • Introduction to frameworks • Conceptual Frameworks • Goals and objectives • Results and Logical Frameworks • Logic models • Exercises developing frameworks • Interface between frameworks and M&E planning • Work on group project

4. Why are Frameworks Useful? Designing frameworks assist developing: • Clearly understood program/project goals and measurable, long-term, short-term, and intermediate objectives • Clearly defined relationships between program/project inputs, processes, outputs, and outcomes, and between program/project activities and the external context (environmental factors) • Sound implementation and M&E plans

5. Types of Frameworks • Conceptual frameworks • Results frameworks • Logical frameworks • Logic models

6. Conceptual Frameworks(a.k.a., research or theoretical frameworks) Diagram that identifies and illustrates the relationships between all relevant systemic, organizational, individual, or other salient factors that may influence program/project operation and the successful achievement of program or project goals. M&E Purpose: • To show where program fits into wider context • To clarify assumptions about causal relationships • To show how program components will operate to influence outcomes • To guide identification of indicators • To guide impact analysis (causal pathways) Similar frameworks: • Proximate Determinants

7. Individual characteristics Healthstatus Healthypractices Serviceutilization Institutionalcapacity Programsustainability Program supply Technicalinputs Conceptual Frameworks

8. Conceptual Frameworks National Tuberculosis (TB) Program • External Factors • Demographic: • (age, gender) • Socio-economic:(education, occupation) • Environmental: • (war, weather) TB infection • Health Systems (DOTS) • Availability • Access • Utilization • Case Detection • TB Morbidity • Prevalence • Incidence • HIV co-infection • MDR-TB TBmortality • Program Factors • Political commitment • Donor involvement • National TB program • Co-morbidity • HIV • Malnutrition • Alcoholism • Diabetes • TB Knowledge • Stigma • Treatable • Symptoms Source: MEASURE Evaluation 2004 (draft)

9. Child malnutrition, death and disability Inadequate Disease dietary intake Insufficient access to food Inadequate maternal & child care practices Poor water/sanitation & inadequate health services Quantity & quality of actual resources - human, economic and organizational – and the way they are controlled Potential Resources: environment, technology, people Conceptual Frameworks Causes of malnutrition in society Outcomes Immediate causes Underlying causes at household/ family level Basic causes at societal level Source: UNICEF, State of the World’s Children, 1998

10. Activity • Get into groups of 4 or 5 • Develop conceptual framework for utilization of family planning/maternal and child health services • Identify factors that influence health service utilization • Arrange factors into a conceptual framework • After 30 minutes, a member of each group will share the framework with all participants

11. Goals and Objectives • Goal: a broad statement of a desired, long-term outcome of the program • Objectives: statements of desired, specific, realisticand measurable program results • SMART • Specific: identifies concrete events or actions that will take place • Measurable: quantifies the amount of resources, activity, or change to be expended and achieved • Appropriate: logically relates to the overall problem statement and desired effects of the program • Realistic: Provides a realistic dimension that can be achieved with the available resources and plans for implementation • Time-based: specifies a time within which the objective will be achieved Source: GAP 2003

12. Formulating Program Objectives

13. Goals and Objectives National AIDS Committee (NAC) 2005 Strategic Plan • Goal: Prevent the spread of HIV epidemic and minimise its impact on nation by 2009 • Objective 1: Reduction of HIV prevalence in nation • Objective 2:Improved health & quality of life of people infected & affected by HIV/AIDS • Objective 3:Strengthened capacity of NAC & stakeholders to respond to the HIV/AIDS epidemic at all levels through improved research, M&E and improved management & coordination

14. Child malnutrition, death and disability Inadequate Disease dietary intake Insufficient access to food Inadequate maternal & child care practices Poor water/sanitation & inadequate health services Quantity & quality of actual resources - human, economic and organizational – and the way they are controlled Potential Resources: environment, technology, people Goals and Objectives: Activity Outcomes Immediate causes Underlying causes at household/ family level Basic causes at societal level Source: UNICEF, State of the World’s Children, 1998

15. Results Frameworks Diagrams that identify steps or levels of results and illustrate the causal relationships linking all levels of a program’s objectives. Purposes: • Provides a clarified focus on the causal relationships that connect incremental achievement of results to the comprehensive program impact • Clarifies project/program mechanics and factors’ relationships that suggest ways and means of objectively measuring the achievement of desired ends Other terms used: • Strategic frameworks

16. Results Frameworks Source: David Marsh, 1999

17. Results Frameworks Tuberculosis (TB) Control Programs SO1: Increase tuberculosis case detection rate to 70% IR1: Increased availabilityof quality services IR2: Increased demand for quality services IR2.1: Customerknowledge of TB improved IR1.1: Services increased IR1.2: Practitioners’ skillsand knowledge increased IR2.2: Social support for TB practices increased IR1.3: Improved program management Source: MEASURE Evaluation, 2004 (draft)

18. Results Frameworks Donor/USAID Reproductive Health Program SO1: Increased Utilization of Family Planning/Reproductive Health Services IR1 Strengthened sustainability of FP/RH Program IR2 Expansion of high quality FP/RH services in the public and private sectors IR2.1 Increased availability of postpartum and postabortion FP services IR1.1 Improved policy environment for the provision of FP/RH services in the public and private sectors IR2.2 Increased accurate knowledgeof clients about modern methodsand FP services IR1.2 Strengthened NGO advocacy for FP program IR2.3 Improved job performance of health providers, trainers, and administrators Source: USAID/Turkey Performance Monitoring Plan, 1998-2001

19. Results Frameworks FIVE-YEAR GOAL: Foundation Established for Reducing Famine Vulnerability, Hunger and Poverty SO 14: Human capacity and social resiliency increased IR 14.1: Use of high impact health, family planning, and nutrition services, products, and practices increased IR 14.2: HIV/AIDS prevalence reduced and mitigation of the impact of HIV/AIDS increased IR 14.3: Use of quality primary education services enhanced IR 14.2.1: Reduced risk behavior IR 14.3.1: Community participation in the management and delivery of primary education services strengthened IR 14.1.1: Community support for high impact health interventions increased IR 14.2.2: Mother-to-child transmission of HIV reduced IR 14.1.2: Availability of key health services and products improved IR 14.3.2: Planning, mgmt and monitoring and evaluation for delivery of primary education services strengthened IR 14.2.3: Access to care and treatment for people living with HIV and AIDS increased IR 14.1.3: Quality of key health services improved IR 14.2.4: Care and support for orphans and vulnerable children expanded IR 14.3.3: Quality of primary education improved IR 14.1.4: Health sector resources and systems improved IR 14.3.4: Equitable primary education services strengthened IR 14.2.5: A more supportive environment for responding to HIV/AIDS Source: USAID/Ethiopia Integrated Strategic Plan, February, 2004 (revised draft)

20. Logical Frameworks Present a standardized summary of the project and its logic. Purposes: • Summarizes what the project intends to do and how • Summarizes key assumptions • Summarizes outputs and outcomes that will be monitored and evaluated Other terms used: • Logframe matrix

21. Logical Frameworks

22. Logical Frameworks Taskforce on Communicable Disease Controlin the Region X: Tuberculosis

23. Logical Frameworks Taskforce on Communicable Disease Controlin the Barents and Baltic Sea Regions: Tuberculosis

24. Logical Frameworks Taskforce on Communicable Disease Controlin the Barents and Baltic Sea Regions: Tuberculosis(only one output and related activities shown here)

25. Activity • Return to small groups from previous activity • Develop goals and objectives for a program aimed at increasing utilization of FP/MCH health services • Develop a results or logical framework for this program • In 45 minutes, a member of each group will share the goals, objectives and framework

26. Logic Models Diagrams that identify and illustrate the linear relationships flowing from program inputs, processes, outputs, and outcomes. Inputs or resources affect Processes or activities which produce immediate results or Outputs, ultimately leading to longer term or broader results, or Outcomes. Purposes: • Provides a streamlined interpretation of planned use of resources and desired ends • Clarifies project/program assumptions about linear relationships between key factors relevant to desired ends Other terms used: • M&E Frameworks, Logical Frameworks

27. Logic Models: Training INPUT PROCESS OUTPUT OUTCOME IMPACT Develop clinical training curriculum Conduct training events Practitioners trained in new clinical techniques Increase in clients served by (newly) trained providers Declining morbidity levels in target population

28. Logic Models Voluntary Counseling and Testing (VCT) Problem Statement:HIV infection rates continue to rise, underscoring the importance for people to know their serostatus, develop personalized risk-reduction strategies, and access care and treatment services. OUTPUTS INPUTS PROCESSES OUTCOMES IMPACTS Financial Resources Supervision and training for VCT personnel Clients (HIV+ and -) develop & adhere to personalized HIV risk-reduction Risk behaviors decreased HIV transmission rates decreased People know their HIV status Demand for Services Provide pre-test counseling Increase in care, prevention, and treatment services for HIV+, HIV-, and discordant couples Infrastructure HIV+ Clients develop & adhere to personalized HIV care, support, and treatment plans Provide HIV testing HIV incidence decreased HIV+ people are referred to appropriate services VCT MIS** Provide post-test counseling VCT protocols, guidelines, and training documents** Refer HIV+ clients to appropriate services (PMTCT, care and support, treatment) HIV morbidity & mortality decreased Counseling and Testing Personnel* Health outcomes of HIV + improve Client records are available Continuity of care is available HIV test kits Maintain patient records Program and services are improved Service reports are produced Referral system for prevention & Tx services** Complete reporting requirements

29. Results Framework Tuberculosis (TB) Control Programs SO1: Increase tuberculosis case detection rate to 70% IR1: Increased availabilityof quality services IR2: Increased demand for quality services IR2.1: Customerknowledge of TB improved IR1.1: Services increased IR1.2: Practitioners’ skillsand knowledge increased IR2.2: Social support for TB practices increased IR1.3: Improved program management Source: MEASURE Evaluation, 2004 (draft)

30. Logic Model Portion of model for tuberculosis control relating to increasing demand for quality services • IMPACT • Decreased TB infection, morbidity and mortality • INPUT • Human and financial resources to develop and print educational brochure • PROCESS • Distribute brochure to health facilities • Meet with physicians to promote distribution of brochure • OUTPUT • Brochure distributed to clients of facilities • OUTCOME • Increased customer knowledge of TB transmission and treatment • Increased demand for quality TB services

31. Activity • Return to small groups from previous activities • Develop logic models for one of the activities of your program to increase utilization of FP/MCH health services • After 20 minutes, a member of each group will share one of the logical frameworks

32. Frameworks for M&E Planning • Purposes: • clarifying assumptions, goals, and interrelationships between factors relevant to the project or program • defining objectives • selecting activities • defining levels of performance and desired results in terms of planned activities and realistic, objective impacts • Monitoring and evaluation plans incorporate: • program managers’ assumptions and objectives, in a given context • a schematic design displaying the directional linkages between key program elements and/or planned results, and other relevant factors

33. Summary of Frameworks

34. References • AusGuide. The Logical Framework Approach. http://www.ausaid.gov.au/ausguide/ausguidelines/1-1-1.cfm • Bertrand, Jane T., Magnani, Robert J, and Rutenberg, Naomi, 1996. Evaluating Family Planning Programs, with Adaptations for Reproductive Health, Chapel Hill, N.C.: The EVALUATION Project. • Global AIDS Program. (2003) Monitoring and Evaluation Capacity Building for Program Improvement Field Guide, Version 1. U.S. Centers for Disease Control and Prevention, Atlanta, GA. • Marsh, David. 1999. Results Frameworks & Performance Monitoring. A Refresher by David Marsh (ppt) http://www.childsurvival.com/tools/Marsh/sld001.htm • Tsui, Amy. 1998. Frameworks (ppt). Presented at the Summer Institute, University of North Carolina, Chapel Hill. • Tsui, Amy. 1999. Frameworks (ppt). Presented at the Summer Institute, University of North Carolina, Chapel Hill. • UNICEF. 1998. State of the World’s Children.

35. Group Projects • Form groups for your group projects • For your project, develop • Goals and objectives • Conceptual framework • One other framework: results, logical or logic model