Use of Data for Quality and Program Improvement

1. Use of Data for Quality and Program Improvement Hugh Sturrock Aimee Leidich

2. OUTLINE • Introduction to data quality • Basics of data visualization • Introduction to pivot tables • Example data exercise • Real-world data exploration

3. Introduction to data quality

4. Why good data is important • National level • Informs policy • Assists in planning and assessing various interventions to make strategic decisions about the improvement of those interventions • Works towards meeting the overall national goal of reducing the burden of poor health • Provides evidence towards meeting targets • Provides the basis for M&E • Region/district level • Informs acquisition and distribution of resources • Provides evidence for construction and/or expansion of facilities • Explains human resource capabilities and challenges • Assists with more precise budgeting • Assists council authorities in planning interventions and monitoring those activities • Demonstrates trends in calculated indicators used to estimate future changes • Demonstrates trends in calculated indicators used to estimate future changes • Facility Level • Serves as basis for planning and developing Interventions • Allows providers to identify patients/clients in need of services and/or referrals • Improves efficiency through administrative organization • Inventories resources and determines which supplies and medicines are available and which need to be ordered when • Monitors and evaluates quality of care

5. What is data quality?

6. Key Terms • Data • Indicator • Quality Data • Quality Control • Data Quality Checks • Data Quality Assessment

7. Quality Data • Data that is reliable and accurately represents the measure it was intended to present and is valid for the use to which it is applied. Decision makers have confidence in and rely upon quality data.

8. Quality Control • Process of controlling the usage of data with known quality measurement for an application or a process.

9. Data Quality Assessment Procedure for determining whether or not a data set is suitable for its intended purpose.

10. Data Quality Checks • Procedures for verifying that forms, registers and databases are completely and correctly filled at each step of the reporting process. • Examples: • Spot-checks • Cross-verifications

11. Spot-checks of actual service delivery tools Perform spot checks to verify the complete and accurate documentation of delivery of services or commodities. Missing data Missing date Incorrect gender entry

12. Cross-check with other data-sources Cross-check the verified report totals with other data-sources (e.g. inventory records, laboratory reports, aggregated reports etc). Facility 1: Cases: 25 Facility 2: Cases: 20

13. Data Quality Guiding Principles • Accuracy • Reliability • Completeness • Precision • Timeliness • Integrity • Confidentiality

14. Accuracy • Also known as validity. Accurate data are considered correct when the data measure what they are intended to measure. Accurate data minimize error (e.g., recording or interviewer bias, transcription error, sampling error) to a point of being negligible.

15. Precision • Data have sufficient detail meaning they have all the parameters and details needed to produce the required information.

16. Completeness • All variables in either reporting or recording tools must be filled. It represents the complete list of eligible persons or units and not just a fraction of the list.

17. Timeliness • Data are up-to-date (current) and information is available on time. This implies all the reports produced are submitted to the next level within the recommended timeframe. Due May 7th

18. Reliability • The data generated by a program’s information system are based on protocols and procedures that do not change according to who is using them and when or how often they are used. The data are reliable because they are measured and collected consistently.

19. Integrity • Data have integrity when the system used to generate them are protected from deliberate bias or manipulation for political or personal reasons.

20. Confidentiality • Clients are assured that their data will be maintained according to national and/or international standards for data. This means that personal data are not disclosed inappropriately and that data in hard copy and electronic form are treated with appropriate levels of security (e.g. kept in locked cabinets and in password protected files).

21. Factors that contribute to poor data quality • Data entry errors • Inconsistent reporting forms • Missing data • Delayed reporting • Failure to report

22. Common Sources of Errors • Transposition • Copying • Coding • Routing • Consistency • Range • Gaps • Calculation

23. Transposition Error 12 Transposition error When two numbers are switched. Usually causedby typing mistakes. (e.g. 12 is entered as 21)

24. Copying Error Letter Number O 0 Entered as When a number or letter is copied as the wrong number or letter. (e.g. 0 entered as the letter O)

25. Coding Error Entered as 4 during interview Coded as 3 in the dataset When the wrong code is entered. (e.g. interview subject circled 1 = Yes, but the coder copied 2 (= No) during coding)

26. Routing Gender erroneously entered into the age category When a number is placed in thewrong field or in the wrong order (e.g. gender entered into the age category)

27. Consistency Mary erroneously entered as a male When two or more responses on the same questionnaire are contradictory (e.g. birth date and age; name and gender)

28. Range Weight erroneously entered as 600kg When a number lies outside the range of probable or possible values (e.g. Age = 151 yrs)

29. Gaps Unique ID is missing When data are not filled in

30. Calculation 340 = 110 + 230 Total males and females added erroneously When data is not calculated correctly. (e.g. 3+1 = 5)

31. Introduction to data usage and visualization

32. Why Do We Spend So Much Time and Energy Collecting All This Data ?! Strengthen M&E programs Improve program planning and resource allocation Gain efficiency and effectiveness Use evidence for decision making Strengthen capacity of staff Improve data quality

33. Data Is At The Center of M&E But…..only if we review, discuss, interpret, and use it regularly!

34. Use Data To Guide Resource Allocation • A program needs adequate resources and staff in order to achieve its goals. • Presenting high-quality program data can help program managers to advocate for additional resources. Our RDT data suggest we need faster allocation of RDTs to avoid stockouts Our malaria surveillance data suggest we need more trained nurses! Our malaria surveillance data suggest we need more vehicles!

35. Data Use for Decision Making • No one “gold standard” approach • Hybrid of approaches depending on the context • Dissemination in all appropriate forums • Motivate/incentivise efforts in data use • Reduce institutional and behavioural barriers to data use (e.g. accountability and performance measurement; attitudes)

36. BASICS OF Visually presenting data

37. Key Definitions • Results:Simple description/observations of your results (who, what, where, when, magnitude, trend). • Interpretation: Explanation of why your results may have occurred. • Conclusion: the key message of your results, implications and the “action-plan” that you recommend based on your results. • The “Take Away” Interpretation Conclusion Result Nine elephants damaged storefronts on Market St in San Francisco in 2010, one elephant damaged a store in 2013. The number of elephants on Market St in San Francisco has decreased since 2010 because a zookeeper has started laying a trail of peanuts to Ocean Beach Citizens should be sensitized to encourage elephants to play at the beach instead of on Market St

38. RESULTS

39. Presenting Data In Tables • Tables may be the only presentation format needed when the data are few, relationships are straightforward and when display of exact values is important. Source: PEPFAR Annual Progress Report, Namibia 2013

40. Bar Charts Are Useful to Show Simple Comparisons, Esp. Differences in Quantity.

41. Line Charts Are Good for Showing Change Over Time (Trend)

42. Bar and Line Charts Can Be Used Together to Show Trends Of Several Related Indicators

43. Maps show geographic relationships Est. no. HIV + per sq km

44. Figure title • Be sure to include: • What (the indicator) • HIV prevalence • % circumcised • % alive on ART • Who • pregnant women age 15-49 • adults males age 15-49 • pediatric ART patients • When • in 2012 • from 2008 to 2012 • Where • in Namibia • in Ohangwenaregion • at Engela Hospital Clinic

45. Where ? What ? Who ? When ? Source: Namibia MOHSS (2012) Annual Implementation Progress Report for the National Strategic Framework (NSF) 2011/12.

46. Presenting Data Tips (2) • All relevant information needed to interpret the table, figure, or map should be included so that the reader can understand without reference to text (i.e. in a report) • Clearly label your X and Y axes, format consistently (font, font size, style, position) • Use data series legends /labels • Make the scale appropriate for the findings you want to convey. • Reference the source of your data

47. Scale spans to 100% to display complete picture Clear chart title Y-axis label X-axis label X-axis label Data source reference Series legend Source: Namibia MOHSS (2012) Annual Implementation Progress Report for the National Strategic Framework (NSF) 2011/12.

48. Stratification of Data • What is stratification? • Dividing into subgroups • What are common levels of data stratification? • Year, age, sex, geographic region, facility • Why do we stratify? • Let’s look at stratification within the indicator: • % of patients alive on ART 12 months after initiation

49. What Do You Think About This Figure?

50. We Can Stratify By Time, e.g. Initiation Cohort…