330 likes | 869 Views
National EHDI Conference Atlanta, Georgia, March 3, 2005. Evaluating Complex Systems of Care A Logic Model Approach to EHDI Program Evaluation. Sherry Spence, MA Oregon Department of Human Services Health Services, Office of Family Health.
E N D
National EHDI Conference Atlanta, Georgia, March 3, 2005 Evaluating Complex Systems of CareA Logic Model Approach to EHDI Program Evaluation Sherry Spence, MA Oregon Department of Human Services Health Services, Office of Family Health Supported in part by funding from the Maternal and Child Health Bureau (Title V, Social Security Act), Health Resources and Services Administration, and by the Centers for Disease Control and Prevention, US Department of Health and Human Services Opinions stated do not represent those of CDC, HRSA, or USDHHS
Acknowledgements • EHDI Staff at CDC and HRSA • EHDI Data Committee • Oregon’s EHDI Team • Claudia Bingham • Al Ferro • David Laszlo • Maureen McNamer • Amy Rosenthal
Acknowledgements • FamilyNet Executive Sponsors • Katherine Bradley • Lorriane Duncan • Office of Family Health Epidemiologists and Research Analysts • FamilyNet Project Teams and . . .
Foreword • “If you torture data sufficiently, it will confess to almost anything.” • Fred Menger (1937- ), chemistry professor • From A Word A Day (AWAD) - wsmith@wordsmith.org
Foreword • Corollary: “Sometimes you can coerce data by threatening deletion.” • Arthur Epstein, professor of chemistry and physics (from daughter, Melissa, responding in AWADmail in response to the Menger quote)
Foreword • Conclusion: “There are lies, damn lies, and statistics.” • Mark Twain (1835-1910), author and humorist
What is our goal for today? • Disprove the Twain/Menger-Epstein hypothesis • Working hypothesis: • It is possible, without false testimony, torture, or coercion, to develop an evaluation plan using our program data.
What is our goal for today? • Define a logical framework for evaluating EHDI programs • Discuss whether our logic model approach avoids the Twain/ Menger-Epstein condition. • Figure out where our model fits in identifying newborn hearing screening as a best practice.
What are we currently evaluating? Logical framework for EHDI evaluation
Within 10 Days of Screening, Testing or Referral Hospitals Diagnostic Facilities Early Intervention Facilities Screening Facilities Demographics, screening, testing and referral information to DHS Monthly reports, program statistics back to facilities DHS DHS Response When: Child Is 6 weeks of age Parent Healthcare Provider Reminder letters / program contact only when child has not completed the desired EHDI process by the desired time. Participating facilities, HCP and LPH able to return updated contact information 6 weeks after “Refer” Parent Healthcare Provider Local Public Health 2 weeks after Dx Hearing loss if no EI referral Parent Healthcare Provider Local Public Health
What are we trying to prove? • Success in achieving EHDI goals • Early screening of everyone • Early identification of some • Early preventive or treatment intervention • Acquisition of language and communication skills
What else do we need to show? • Success in achieving EHDI goals • Success in achieving health care goals & optimizing resource use • Success in all children achieving their full health and development potential
Children interacting and ready to learn Language and communication skills Intervention to assure skill acquisition Early and complete identification and risk monitoring High level outcomes - population attributes Intermediate outcomes - client/patient attributes Outputs - program activities and interventions What are the components of an EHDI logic model?
All children ready to interact and learn Language and communication skills Intervention to assure skill acquisition Early and complete identification and risk monitoring 4-5 years after birth 12-36 months after birth 3-6 months after birth 1-3 months after birth What is the approximate timing of this monitoring?
Conduct research Conduct research Monitor program outcomes Monitor program protocols 4-5 years after birth? 12-36 months after birth? 3-6 months after birth 1-3 months after birth How can we use this logic model in program evaluation?
Will system integration help? • Earlier screening and identification • More complete screening and identification • More timely follow-up • Less loss to follow-up
Why do we think this is so? • About 40% of hearing loss occurs with other conditions or syndromes, resulting in additional disabilities (Gallaudet Research Institute, 2003) • Additional issues increase likelihood of other public health program involvement. • In Oregon, about 50% of women giving birth are enrolled in WIC and will enroll their babies • The WIC program keeps track of its highly mobile population.
What are our intermediate system outcomes? • COORDINATION • CONTINUITY • EFFICIENCY • TIMELINESS • ACCURACY & APPROPRIATENESS
How does this fit with the logic model? • Population success • Program success • Process success Program Logic Model System Logic Model
Conclusions • One purpose of a system of services is to assure the success of the individual programs and processes it involves. • One purpose of a data system is to facilitate and enhance the success of the programs that use it.
What do we have so far? • Solid framework for measuring program processes • Solid framework for measuring system processes
What’s the key to all this? • The logic model works best if the program intervention is an established, evidence-based best practice.
Gold standard Controlled studies by programs Expert testimony by professionals Repeated measures and testimony Immoral or unethical Good evidence from which we can generalize? Good evidence or bias? Eliminates bias and demonstrates generalizability What best practice evidence to we need?
What evidence do we have? • Prospective and retrospective studies by programs and researchers? • Expert testimony by speech, hearing, and language professionals? • Repeated measures of all kinds?
Thank you. Thanks to Amy Rosenthal, Oregon EHDI Coordinator for her participation and to Scott Grosse, PhD, NCBDDD/CDC, for his participation and for noting an error in slide 32.