Abu Raihan, MD, MPH Director of Program, Asia Email: abu.raihan@orbis

1. Symposium 6: Challenges in Monitoring Clinical Program Outcomes Developing an Institution-Wide M &E paradigm: The ORBIS experience Abu Raihan, MD, MPH Director of Program, Asia Email: abu.raihan@orbis.org IAPB 9th GA, Hyderabad, 17-20 September 2012

2. Outline • Definition and purpose of M&E • ORBIS Paradigm: Importance of clinical outcomes • Principles of choosing and using key outcomes • Multi-tier challenges & barriers to data collection • Conclusions

3. Monitoring • Systematic and routine collection of information from the on-going programs for three main purposes: • To have internal and external accountability of the resources used and the results obtained • To learn from experience to improve practice and take informed decisions • To empower partners to deliver quality care

4. Evaluation • Assessing systematically and objectively the value of program. • Evaluations help to draw conclusions about five main aspects of the interventions: • Relevance • Effectiveness • Efficiency • Impact • Sustainability

5. Key Elements in Designing the M&E Framework • Define relevant indicators and their measures; • Identify and set benchmarks/standards; • Determine data collection strategy; • Analyze data and prepare reports for dissemination and decision making.

6. Purpose of Institution-wide Approach of M&E • It is one of the important component of program planning, development and implementation • Standardization of data collection and information flow • Provides checks and balances over program planning and implementation • Strong M&E leads to transparency and good governance for an organization

7. ORBIS Paradigm: Importance of Measuring Clinical Outcomes • To measure excellence in training and practice for quality patient care • To provide evidence for advocacy to persuade policy makers, partners to promote good quality eye health • Demonstrate impact of good vision on individuals, families and on communities • Sustainability ensured when good outcomes attract patients

8. Principles for Choosing Key Outcomes • Outcomes must be appropriateandpractical to collect by partners • Outcomes reflect quality • Outcome data supports Continuous Quality Improvement process

9. Complexity of Clinical Outcomes • There is no one standard for all diseases • For certain complex procedures, outcomes are the result of expert consultations • Pediatric cataract, glaucoma • ROP • Retina • Current standards are based on international best practices - PPP • This is a good place to start, however, these outcomes may need to be adapted as we put them into practice.

10. Internal Challenges • Data collection: who what when how frequent • Cost to implement strategy: information system and other resources • Which software to manage data on the outcomes? • Willingness and ability of staff / partners to collect • Analysis, use, and dissemination of results- requires additional skills

11. External Challenges • Cultural issues, not valuing the purpose • Suspicion and fear • System and resources lacking • Privacy / legal issues in some countries to patient data with foreign NGOs • Partners feel alienated from M&E process. • Often they have no inputs in the process- “one-size-fits-all” requirements • One-way, upward accountability, very little feedback loop.

12. Barriers to Outcome Data • No systems to collect/collate data • No culture to record • Suspicion/fear of recording Sub-Saharan Africa Barriers to data availability India • Limited systems to collate • Suspicion/Fear of sharing+/- USA Lack of information • Good patient note • Published and publicly available outcome data

13. Practical Approach to Clinical Outcomes • Invest resources for • Staff, training, and retention • System for data collection and management • Develop capacity to analyze data and give partners feedback on the results: close the loop! • Build or strengthen capacity of partners toward clinical excellence in practice • Bringing Ophthalmologists on board is the key !

14. Areas for Debate • Should current standards be universal across all countries? • What are context specific appropriate standards? • Ho do we change the culture?

15. Conclusions • Challenges exist, though most can be addressed through appropriate investment in resources • Attitudes and perceptions are crucial but are often difficult to change • The shift to include measurement of clinical outcomes is critical to good quality eye care programs for elimination of blindness.

16. Thank you