Data-based Decision Making: Understanding, Assessing, and Using Data

Data-based Decision Making: Understanding, Assessing, and Using Data Slides for Module 10 Topic: Assessing Data Quality and Usefulness

Assessing Data Quality and Usefulness 1 • Assessing Data Quality and Usefulness • Triangulating Data from Multiple Sources • Data Reports and Formats

Training Objectives Following the training, participants will be able to: • Identify and describe 4 factors to consider in assessing the quality and usefulness of a study or data report • Describe the use of triangulation in comparing data from multiple sources and studies • Describe at least 5 characteristics of effective HIV data reports prepared for PC/PBs

Training Objectives (cont.) • Identify at least 4 ways to make data presentations understandable and useful for all PC/PB members • Describe at least 3 ways to make data charts understandable and useful for PC/PB members, including those who are not data experts • Describe sound practices for receiving and discussing data

Data Myths and Realities • Consumers don’t understand data • Many people are uncomfortable using data: more people are innumerate than illiterate • People often think they won’t understand HIV-related data and charts because they don’t use them every day • It takes a Ph.D. to understand RWHAP data • People can learn to understand and use data • Moderate training is sufficient to understand most RWHAP data if it is well presented • Data will give you the answer • Data can be spun to give you many different answers • Data vary greatly in quality and need to be assessed

Assessing Data Quality and Usefulness 2 All data are not “created equal” – so PC/PBs need to assess the quality of the data and reports they receive

Assessing Data Quality PC/PB member roles: • Review data from multiple sources • Ask questions about how data were gathered, tabulated, and analyzed • Compare and weigh data from different sources and studies • Decide how much confidence to place in the data • Give the greatest weight in decision making to the “best data” PC/PB Support Staff, Consultants, & Recipient Staff: • Provide/present data from various sources • Understand and share information on data quality and limitations

Critical Factors for Reviewing or “Weighing” Data • Number of respondents/Size of study • Representativeness/Sampling • Content/Questions • Quality Control

Numbers and Representativeness: Who Was Included in the Survey or Study? 1 • Numbers: Numbers of people or sample size – more weight to data from larger numbers of people • Representativeness: More confidence in data when the individuals sampled were chosen to represent the entire HIV population, subpopulation, or the targeted portion of the community – through: • Probability sampling • Purposive or representative sampling

Numbers and Representativeness: Who Was Included in the Survey or Study?2 • Probability sampling: Using a random sampling method where each member of the population has an equal probability of being included, so that findings can be assumed to reflect the entire population from which the sample was drawn • Purposive or representative sampling: Selecting people for the study so that they mirror the HIV population in your EMA or TGA or the subpopulation you are targeting

Content/Questions and Quality Control • Content/Questions: Look at whether the questions asked were clear and understandable, so that they were likely to generate reliable data that really measure what they were supposed to be measuring • Quality control: Look for evidence that the data were collected using appropriate methods and by appropriately trained individuals • Data collection process carefully managed/monitored and agreed-upon methods implemented • Data reviewed for completeness, nonduplication, and data entry where relevant

Questions to Ask in Assessing and Interpreting Surveys and Studies • Who was responsible for the study? • Were knowledgeable consumers and other PLWH involved in design? • Does the “tool” use good questions? Are they clear and understandable? Do they seem likely to generate reliable data that really measure what the study is supposed to be measuring? Was the tool pre-tested? • What was the sample size? Is it representative? • What evidence is there that the data were collected using appropriate methods and by trained individuals? • Was there “quality control” to be sure the stated data gathering and analysis process was followed?

Terms for Key Measures of Data Quality:Reliability and Validity • Reliability involves consistency and “repeatability” of findings – you would get the same results if: • You did the study a second time • You asked the same person the same question again • Validity involves the credibility or “believability” of your findings – they truly represent the phenomenon you are trying to measure • The tools used measured what they were supposed to measure (internal validity) • The results can be “generalized” or assumed to apply to people beyond the sample in the study (external validity) Source: “Introduction: Reliability and Validity,” UC Davis, http://psc.dss.ucdavis.edu/sommerb/sommerdemo/intro/validity.htm

Quick Questions A: Assessing Surveys and Studies Your PC/PB had problems with data reliability in its last PLWH survey in 2016 – a lot of questions seem to have been misunderstood, and findings were inconsistent with epi data and HIV care continuum data. The survey was done in a hurry, so that findings could be used in preparing your Integrated HIV Prevention and Care Plan. PC/PB staff and members have limited expertise with large-scale surveys, but you want to do a new survey next year. • What are some practical things you can do to improve data quality? • Who within the PC/PB should lead this effort?

Quick Questions B: Assessing Surveys and Studies Your EMA/TGA has several large subrecipients including a federally qualified health center (FQHC) and a clinic associated with a large teaching hospital. These subrecipients all receive research grants and seem to do a lot of internal assessment of client needs and service quality. You want to encourage them to share research findings with the PC/PB, but you also want to be able to review and assess the quality of their data, as you do for other data. • What might be a good process for inviting, receiving, and reviewing such studies? • What are some possible concerns? • Who within the PC/PB should lead this effort?

Common Causes of Unreliable Data (Low Reliability) • People don’t know or can’t remember the answer to the question • Example: People may not accurately recall their CD4 count so survey data on CD4 counts are often not reliable • Example: Asked if they have been “out of care for more than a year at any point since diagnosis,” many PLWH don’t remember • Too much information is requested • Example: A survey is so long or intake form questions are so complicated that people responding get tired and answer later questions with little thought

Common Causes of Data Validity Issues (Low Validity) • Poorly stated questions • Example: A survey asks PLWH “whether you have been diagnosed with HIV disease/not AIDS or with AIDS (stage 3 HIV)”; confusion around “AIDS” and “diagnosed” leads people who currently have a high CD4 count to say “HIV/not AIDS” – responses are not valid • The sample is small or unrepresentative • Example: A focus group of 9 Latina PLWH provides new information about this population, which the PC/PB wants to use to redesign a service model – but the group were all of similar age, all English speakers – and their needs are unlikely to represent the needs of all Latina PLWH in the EMA/TGA

Quick Questions C: Data Reliability and Validity Issues The PC/PB wants to better understand issues facing recently incarcerated PLWH, who appear to have low retention, medication adherence, and viral suppression rates. When you look closely at RWHAP HIV care continuum data, PLWH survey data, and focus group data, you find that each one used a different definition of “recently” and “incarcerated.” You like the focus group definition best (“continuously in jail or prison for at least 6 months during the past 3 years”), but it included only 9 people, so you aren’t sure what weight to give the findings. • What kinds of data problems are these? • What might you do to avoid them?

Triangulating Data from Multiple Sources A PC/PB can have additional confidence in findings found in more than one data set, and can use multiple methods to increase understanding of a topic or issue–that process is called triangulation

Triangulation of Data • The process of comparing data on the same topic from 2 or more sources or research studies to: • “Cross check” or “cross-validate” the data-- See whether the different sources report similar findings • Increase understanding of the topic • PC/PBs can have greater confidence in findings that are reported from several different studies or sources or obtained through different methods

Approaches to Triangulation • Can involve data from several types of needs assessment activities (such as PLWH survey, focus groups, provider survey) • Can also involve data from the recipient or other sources (such as epi data, RSR data on client utilization, HIV care continuum data) • Often involves comparing quantitative and qualitative data • Comparisons should involve critical review of methods used to obtain data, to decide how much confidence to place in the data from each source

Using Triangulation • Identify an important question or issue • Look at multiple data sources to see what information they provide on the question or issues • Compare data to determine similarities and differences • Explore possible reasons for differences • Assess data sources to determine which should receive the most “weight” • Where findings are similar across sources, especially the “better quality” sources, use those findings in decision making • Use findings to inform additional data gathering

Examples of Triangulation 1 Issue: Treatment of Chronic Health Problems Data Sources Compared: PLWH Survey, Key Informant Interviews and Monitoring Data • PLWH Survey: A very high percentage of PLWH from outlying counties in your new survey report serious health problems, especially diabetes and high blood pressure, that are not being adequately treated • Interview and Monitoring Data: Key informant interviews and monitoring data confirm that these PLWH often have no primary care physician and infectious disease specialists are seriously overburdened

Examples of Triangulation 2 Issue: Barriers to care/linguistic access Data Sources Compared: Focus Group and Provider Survey • Focus Group: A focus group of Latino immigrants reports that some providers have no Spanish-speaking clinical staff and are using receptionists as untrained interpreters • Provider Survey: Provider surveys confirm a lack of language capacity and the need for funding for language services

Sum Up: Assessing Data Quality and Usefulness • All data are not created equal – PC/PB members need to be able to ask appropriate questions and “weigh” data quality • PC/PBs need to ask for and receive data in clear and useful formats, including reports and presentations that include user-friendly charts and explanations • Triangulation helps in comparing various data sets and identifying similar findings • PLWH play a valuable role in assessing data due to their familiarity with the system of care

Data Reports and Formats For maximum value in planning and decision making, even the most reliable and accurate data need to be presented in clear, understandable formats

Importance of How Data are Presented • Written reports, oral presentation, and charts have huge influence on understanding and use of data • PC/PBs can become good users of data through: • Use of well designed, clear reports and presentations • Sound basic training on data sources and uses • Mini-trainings provided along with presentations • Allocation of the time necessary to discuss and clarify data • PC/PB & recipient commitment to ensuring that all members feel comfortable asking questions and know they will receive thoughtful, useful answers

Preparing Data Reports for PC/PBs A data report prepared for the PC/PB should: • Include narrative, tables, and charts • Provide information on data sources, sample size and methods, and limitations • Use plain language – designed for your audience • A plain language document looks good, is organized logically, and is understandable the first time you read it • Use only necessary technical terms – defined & explained • Include an Executive Summary • Provide comparisons to epi and other data and studies, local and national • Be reviewed in draft by the responsible committee

Preparing Oral Presentations Data presentations to the PC/PB should: • Use plain language • Define new or unfamiliar terms • Maximize use of charts with brief text to explain them • Avoid data overload • Include structured “mini-training” to maximize understanding of data presented • Be previewed by the responsible committee and Executive Committee – and revised as needed • Be available electronically before the meeting, and copied and distributed at the meeting

Preparing Data Charts Charts in reports or presentations to the PC/PB should: • Be clear and easy to read – color contrast, data labels, numbers large enough to read • Provide data in a logical order • Include brief text to highlight and explain content • Use graphics to highlight key data • Use consistent formats • Same type of chart used for same type of data throughout • Same colors used for a population or other variable • Specify total number and percent of responses/clients

Making Data User-Friendly: Example The next two slides illustrate how data can be made easier to understand quickly – they present the same data, but: • One uses a table, with data presented clearly, but in no obvious order • The other uses a bar chart, with data presented in order based on percent of people using each service

Sample Data Table: PLWH Survey Services Received During the Past Year (Percent by Service Category) [N=486]

Sample Data Bar Chart: Same PLWH Survey Data Services Received During the Past Year (Percent by Service Category) [N=486]

Quick Questions D: Different Ways of Presenting Data Review the previous two slides, which present the same survey data in two different formats: as a data table and as a data bar chart. • Which one do you find easier to understand quickly? Why? • How much of the difference involves the format (table or bar chart), and how much involves other factors? • To what extent does your answer to Question 1 involve personal preferences? To what extent does it involve the level of data training you have had?

Presenting and Discussing Data • Summarize methods, numbers, sampling, limitations as needed to assess data quality • Refine/focus what you present based on your audience and the expected use of the data • Highlight and explain key findings • If you “go beyond the data” for interpretation or conclusions, clearly state your assumptions • Highlight key issues or uncertainties • Allow immediate clarifying questions and discussion at several points and at the end

Discussing the Data The presenter and/or PC/PB leadership should: • Allocate meaningful time for discussion whenever data are presented • Encourage questions about data source and quality • Identify and highlight findings that would benefit from diverse input and interpretation • Be sure major points of discussion are summarized and included in minutes • Document and follow up on data questions and obtain/present answers

Optional Slides for Activities

Activity 10.5 Instructions • Work with a small group, choosing a facilitator, recorder, and reporter. • Be prepared to share your work with the full group.

Activity 10.5: Using Triangulation Assume you are a committee of the PC/PB that just carried out the mental health data triangulation and received the data described in the handout, and answer the following questions: • What do you see as the key findings? • How much weight would you give the information from each source? Why? • What, if any, additional data might you want to review or collect before deciding on possible action? • How might you eventually use these data in planning?

Activity 10.6: Assessing Data Charts and Tables • Work in a small group, choosing a facilitator, recorder, and reporter. • Review the situation and the data tables, charts, and/or text summaries assigned to your group, and answer the related questions. • Prepare your reporter to present the assigned materials and your review to the full group.

Situation A: Sample Formats for Epidemiologic Data The formats presented here all involve tables and charts used to present data that come from Epidemiologic Profiles – though they can of course be used for other types of data presentations

Format #1: Race/Ethnicity of People Living with HIV in the EMA by Jurisdiction, 2017 (Percent) N = 12,370

Format #2: Proportion of Living HIV Cases, by Race/ Ethnicity, Sex and Mode of Transmission, EMA, 2017 (N=7,243)

Format #3: Race/Ethnicity of People Living with HIV in the TGA, 2017 Race/Ethnicity of People Living with HV in the TGA, December 2017

Format #4: People Living with HIV Disease in the EMA in 2017 A total of 5,828 people are living with HIV disease • 47% Black • 38% White • 12% Hispanic/Latino • 60% 30-49 years old • 49% MSM • 28% White MSM Out of 10 people living with HIV disease, approximately: • 5 are Black • 4 are White • 1 is Hispanic/Latino • 8 are male • 6 are 30-49 years old • 5 are MSM • 3 are White MSM

Format #5: Trends in New Diagnoses Sample Epi Profile Chart: 5-Year Trends in Newly Diagnosed HIV Cases by Year and Gender Identity, Washington DC Source: Annual Epidemiology and Surveillance Report: Data through 2016, Washington, DC • Trend chart with data labels showing percentages of newly diagnosed PLWH by year and by gender identity, plus the total number of PLWH newly diagnosed each year. May seem more complicated but can be used each year.

Situation B: Sample Formats for Needs Assessment Data The text slides, charts, and tables presented here come from needs assessment data reports – based on PLWH and providers surveys, sometimes supplemented by focus groups.

Format #1: Mental Health & Psychosocial Services • Uses: Mental health services are psychological and psychiatric treatment and counseling services offered to individuals with a diagnosed mental illness, conducted by a licensed mental health professional; psychosocial services include support & counseling services such as support groups that may be provided by peers or other non-licensed personnel (support service) • Need/gap: 45% of consumers responding to the survey reported “depression or other mental health issues” in the past year; PLWH indicated a similar level of need for mental health services and “peer support/support groups”: • 63% of consumers surveyed reported a need for mental health services, 52% received such services over the prior year, and 11% of those who needed services did not receive them • 60% reported a need for peer support/support groups, 46% received them, and 14% of those needing them did not; concern that some may soon end • Satisfaction: 72% of mental health clients and 65% of peer support/ support group clients were “very satisfied”; satisfaction with both services was somewhat higher in the central city than in other parts of the TGA

Format #2: Sample Needs Assessment Data: PLWH Survey Co-Occurring Conditions Reported by PLWH, by Gender Identity (Percent) The most frequently reported co-occurring condition PLWH reported dealing with over the past year is depression and other mental health issues – reported more often by transgender and female PLWH than by men

Format #3 Type of Medical Insurance, 2017 (N=3,657)

Data-based Decision Making: Understanding, Assessing, and Using Data