1 / 33

Evaluating Complex Systems of Care A Logic Model Approach to EHDI Program Evaluation

National EHDI Conference Atlanta, Georgia, March 3, 2005. Evaluating Complex Systems of Care A Logic Model Approach to EHDI Program Evaluation. Sherry Spence, MA Oregon Department of Human Services Health Services, Office of Family Health.

conway
Download Presentation

Evaluating Complex Systems of Care A Logic Model Approach to EHDI Program Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. National EHDI Conference Atlanta, Georgia, March 3, 2005 Evaluating Complex Systems of CareA Logic Model Approach to EHDI Program Evaluation Sherry Spence, MA Oregon Department of Human Services Health Services, Office of Family Health Supported in part by funding from the Maternal and Child Health Bureau (Title V, Social Security Act), Health Resources and Services Administration, and by the Centers for Disease Control and Prevention, US Department of Health and Human Services Opinions stated do not represent those of CDC, HRSA, or USDHHS

  2. Acknowledgements • EHDI Staff at CDC and HRSA • EHDI Data Committee • Oregon’s EHDI Team • Claudia Bingham • Al Ferro • David Laszlo • Maureen McNamer • Amy Rosenthal

  3. Acknowledgements • FamilyNet Executive Sponsors • Katherine Bradley • Lorriane Duncan • Office of Family Health Epidemiologists and Research Analysts • FamilyNet Project Teams and . . .

  4. And Oregon’s Families

  5. Foreword • “If you torture data sufficiently, it will confess to almost anything.” • Fred Menger (1937- ), chemistry professor • From A Word A Day (AWAD) - wsmith@wordsmith.org

  6. Foreword • Corollary: “Sometimes you can coerce data by threatening deletion.” • Arthur Epstein, professor of chemistry and physics (from daughter, Melissa, responding in AWADmail in response to the Menger quote)

  7. Foreword • Conclusion: “There are lies, damn lies, and statistics.” • Mark Twain (1835-1910), author and humorist

  8. What is our goal for today? • Disprove the Twain/Menger-Epstein hypothesis • Working hypothesis: • It is possible, without false testimony, torture, or coercion, to develop an evaluation plan using our program data.

  9. What is our goal, really?

  10. What is our goal for today? • Define a logical framework for evaluating EHDI programs • Discuss whether our logic model approach avoids the Twain/ Menger-Epstein condition. • Figure out where our model fits in identifying newborn hearing screening as a best practice.

  11. What are we currently evaluating? Logical framework for EHDI evaluation

  12. Within 10 Days of Screening, Testing or Referral Hospitals Diagnostic Facilities Early Intervention Facilities Screening Facilities Demographics, screening, testing and referral information to DHS Monthly reports, program statistics back to facilities DHS DHS Response When: Child Is 6 weeks of age Parent Healthcare Provider Reminder letters / program contact only when child has not completed the desired EHDI process by the desired time. Participating facilities, HCP and LPH able to return updated contact information 6 weeks after “Refer” Parent Healthcare Provider Local Public Health 2 weeks after Dx Hearing loss if no EI referral Parent Healthcare Provider Local Public Health

  13. What are we evaluating?

  14. What are we trying to prove? • Success in achieving EHDI goals • Early screening of everyone • Early identification of some • Early preventive or treatment intervention • Acquisition of language and communication skills

  15. What else do we need to show? • Success in achieving EHDI goals • Success in achieving health care goals & optimizing resource use • Success in all children achieving their full health and development potential

  16. What are the components of an EHDI logic model?

  17. Children interacting and ready to learn Language and communication skills Intervention to assure skill acquisition Early and complete identification and risk monitoring High level outcomes - population attributes Intermediate outcomes - client/patient attributes Outputs - program activities and interventions What are the components of an EHDI logic model?

  18. What data do we have?

  19. All children ready to interact and learn Language and communication skills Intervention to assure skill acquisition Early and complete identification and risk monitoring 4-5 years after birth 12-36 months after birth 3-6 months after birth 1-3 months after birth What is the approximate timing of this monitoring?

  20. So we won’t know for 4-5 years if this was a good idea?

  21. Conduct research Conduct research Monitor program outcomes Monitor program protocols 4-5 years after birth? 12-36 months after birth? 3-6 months after birth 1-3 months after birth How can we use this logic model in program evaluation?

  22. Will system integration help? • Earlier screening and identification • More complete screening and identification • More timely follow-up • Less loss to follow-up

  23. Why do we think this is so? • About 40% of hearing loss occurs with other conditions or syndromes, resulting in additional disabilities (Gallaudet Research Institute, 2003) • Additional issues increase likelihood of other public health program involvement. • In Oregon, about 50% of women giving birth are enrolled in WIC and will enroll their babies • The WIC program keeps track of its highly mobile population.

  24. How does the logic model work for systems?

  25. What are our intermediate system outcomes? • COORDINATION • CONTINUITY • EFFICIENCY • TIMELINESS • ACCURACY & APPROPRIATENESS

  26. How does this fit with the logic model? • Population success • Program success • Process success Program Logic Model System Logic Model

  27. Conclusions • One purpose of a system of services is to assure the success of the individual programs and processes it involves. • One purpose of a data system is to facilitate and enhance the success of the programs that use it.

  28. What do we have so far? • Solid framework for measuring program processes • Solid framework for measuring system processes

  29. So what?

  30. What’s the key to all this? • The logic model works best if the program intervention is an established, evidence-based best practice.

  31. Gold standard Controlled studies by programs Expert testimony by professionals Repeated measures and testimony Immoral or unethical Good evidence from which we can generalize? Good evidence or bias? Eliminates bias and demonstrates generalizability What best practice evidence to we need?

  32. What evidence do we have? • Prospective and retrospective studies by programs and researchers? • Expert testimony by speech, hearing, and language professionals? • Repeated measures of all kinds?

  33. Thank you. Thanks to Amy Rosenthal, Oregon EHDI Coordinator for her participation and to Scott Grosse, PhD, NCBDDD/CDC, for his participation and for noting an error in slide 32.

More Related