CMAM conference London 17 th – 18 th October 2013

The MRP – Development of a comprehensive CMAM reporting tool using a set of standardised indicators CMAM conference London 17th – 18th October 2013

Presentation outline • The MRP development – background, implementation etc. • The analysis, results and advantages and challenges of using the MRP as information system

Background • SFP review highlighted the inconsistencies, inadequacies and bias associated with reporting of Supplementary Feeding Programmes (SFP) • This means…. • Programmes can change the calculation of performance by changing the denominator • False “over-performance” • OTP discharges included in new SFP admissions • Transfers to TFP or medical care excluded from the denominator • Potential for improvement of programme quality is not recognised and acted upon • Data is not comparable between programmes/countries

Background cont… • The ‘Minimum Reporting Package’ (MRP) was developed in response to this paper with the initial intention of: • supporting standardised reporting for emergency SFPs, in order to improve programme management decisions, • improve accountability • assist urgently needed learning in the effectiveness of this programme approach • Development occurred over a number of years through a consultative process amongst the global nutrition community • 2009 – SFP indicators and reporting categories defined by a steering committee of 12 international agencies (later piloted in 4 countries) • 2011/12 – OTP and SC reporting categories added following consultation

The MRP today • The MRP has developed into a concise and comprehensive management tool, providing a contextualised overview of the treatment components of CMAM • MRP comprises • a set of guidelines defining indicators and reporting categories (both basic and advanced) • an access based software package and software guidelines. • Some agencies use only the guidelines on indicators and reporting categories to complement or improve their own systems • The optional software package which allows rapid programme monitoring facilitates consistent reporting and reduces the reporting workload of field workers. • Currently used by 7 NGOs in 15 countries (4 more countries in pipeline)

Common misconceptions • Throughout this presentation, we hope to address some misconceptions around the MRP, the most common being: • The MRP is all about software • The MRP is not suitable for ministries of health as implementers of CMAM programmes • The primary goal of the MRP is the standardisation of indicators and improved reporting

MRP indicators – basic and advanced Standardised indicators can be used at a basic or more advanced level, depending on capacity for all programmes (both SFP and OTP)

Case study example – Pakistan • Pakistan has its own national reporting system – the NIS (nutrition information system), in place since 2009 • Several features of the MRP could potentially improve the NIS for enhanced utility, improved calculation of performance and to ensure Pakistan’s CMAM data base is comparable internationally • Save the Children is currently following a consultative process to work on harmonisation of systems through introducing standardised indicators: • A consultation meeting was held in September in Pakistan • Many of the recommendations made by SCUK following this meeting have been agreed to be integrated into the NIS • On-going process integrated with CMAM guidelines revision

Case study example – Yemen • The MRP software replaced excel sheet reporting which was prone to many errors and had limited use • “With the MRP we can see indicators directly when entering data – a major advantage over the old system” • “We can take actions if indicators do not reach Sphere in single feeding sites” • “Data in the MRP report format is shared with the MoPH/UNICEF” • “The MRP helps to improve the quality of the program. Before starting the MRP, the defaulter rate of the program was very high. With the MRP it has been easier to monitor the data and to take corrective actions.”

Data collection through the MRP • In light of the learning objective of the MRP data is collected regularly by a group of MRP partners and feeds into a central database. Analysis is on-going and leading to a larger analysis planned for early 2014. The aims of these analyses are: • To describe the characteristics of CMAM programmes • To describe and assess the effect of CMAM programmes on rehabilitating malnourished individuals • To compare programme performance and outcomes according to contextual factors, differences in protocols or approaches

Methodology • SFP and OTP data collected between January 2012 and July 2013 was analysed • The length of programme data differs but is generally above 3 months in order to be able to analyse programme results • Exclusion of data from analysis • Data not in MRP format (due to time constraints with this analysis, on-going analysis includes all formats feeding in data as long as MRP definitions are used). • Sites where numbers in charge at the end of one month did not match the opening number for the next month (difference >5 excluded) • All analysis is supported by the ERRB (emergency response and recovery branch) team at CDC

Results – SAM • OTP data was available from 3 NGOs, supporting 11 programmes in 8 countries (Burkina Faso, Chad, Ethiopia, Ivory Coast, India, Kenya, Somalia, Yemen). • After excluding data, a total of 14,995 admissions were included • 97.2% of admissions were classed as new admissions (only four programmes reported relapses or re-admissions). • MUAC was the most common admission criteria (78.6%) , but varied between countries and programmes

Results – SAM performance • Performance indicators showed • Overall recovery rates of 80% • Death rates of 1% • Defaulter rates of 13.5% • Transfer to TFP rates of 3.3% • Non-recovery rates of 2.2%. • 5/10 programmes reported recovery rates above 90%.

SAM • Some programmes do better than others

SAM Presentation of data allows real time identification of sites with problems

Results – MAM • SFP data was available from 4 NGOs, supporting 10 programmes in 7 countries (Burkina Faso, Chad, Ethiopia, Ivory Coast, India, Kenya, Somalia) • After data cleaning, a total of 23,584 admissions and 15,496 were included • The majority of admissions were new admissions, (only four programmes reported relapses or re-admissions). • As with SAM, MUAC was the most common admission criteria (81.3% of admissions)

Results MAM – performance • Performance indicators showed • Overall recovery rates of 86.9% • Death rates of 0.1% • Defaulter rates of 10.8% • Transfer rates of 0.9% • Non-response rates of 1.3%. • 6/10 programmes reported recovery rates above 90%.

Results - MAM • Some programmes perform better than others

Results – MAM • Presentation of data allows real time identification of sites with problems

Lessons learnt • The presentation of descriptive data in the standardised MRP format allows: • real time presentation of programme data • easy comparison of different programmes, protocols and organisations • easy access to information on programme background and characteristics • better reporting of defaulting, or any other discharge not recovered. The graphs and validation tools highlight problems (including very high and very low recovery rates) assisting management. • This allows programme managers to identify: • programme characteristics • the impact of specific events that may be affecting the quality and outcomes of the programmes, and identify sites in need of supervisory support.

Emerging trends from the descriptive data • OTPs are implemented widely and overall obtain good results but with wide variation • Despite similar protocols, the contexts and some characteristics of the programmes were very different. • Overall results seem positive for SFP and borderline for OTP (high defaulters) • 5/10 programmes for SAM and 6/10 programmes for MAM reported recovery rates above 90% • Poor programme performance or more accurate reporting and perhaps better management? • Further investigation is needed into the difference between SAM and MAM data to determine if programmes are better at recording SAM data, or if high recovery rates are a true reflection of performance

Data quality • In response to concerns about data accuracy, a quick analysis of actual versus reported data was conducted in one SC programme. • Defaulting rates reported as <1% actually looked closer to 30-40%. • In another programme, children discharged as recovered were found to not meet discharge criteria and should have been reported as non-recovered. • The same children were later reported as new admissions rather than re-admissions. • One constraint identified was the issue of all hard data being located in field sites making verification difficult • In response to this - The MRP is piloting a quality appraisal tool to help assess the quality of the data.

Limitations and challenges • Limited number of programmes and reporting duration • Limited SAM analysis • Only descriptive statistics are presented in this report, contextual data has not been included in this analysis • There is an important lack of data variation in terms of protocols and performance • Despite low numbers of defaulting overall, the lack of verification of defaulting in MAM programmes through home visits may be masking a higher mortality rate, particularly in programmes with high defaulting rates • Questions around data quality • Software often challenging in terms of joining files, bugs, etc

Next steps • Continued advocacy for the use of standardised indicators in all programmes addressing acute malnutrition • Development and roll out of a web based version of the software with off-line capability • Addition of new features in web-based version of software • Large scale analysis in early 2014 • Data quality audit to apply a level of confidence to how correct our data is • Ensuring the MRP is a flexible tool which may take different forms depending on the context whilst still generating comparable and unbiased reporting. • Apply results and experiences to improving the reporting indicators and categories

Conclusion • The MRP is a useful management tool which uses standardised indicators to improve the monitoring and reporting of the treatment components of CMAM • It provides a comprehensive package for standardised monitoring of CMAM treatment in both emergency and development contexts • It can act as a ready-made system in contexts where no other reporting system exists or elements of the package can be incorporated to strengthen existing systems • The MRP allows humanitarian agencies, donors and governments to better monitor and compare performance of programmes in different contexts as well as comparison of different CMAM approaches, and enhances the management of CMAM programmes.

CMAM conference London 17 th – 18 th October 2013