1 / 40

Program Evaluation

Program Evaluation. Presentation by: Kathleen Tebb , PhD Assistant Professor of Pediatrics Division of Adolescent Medicine University of California, San Francisco 11.12.09. Introduction. Eval. Research Examples : Influence of formula gift packs on breastfeeding cessation

tim
Download Presentation

Program Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Program Evaluation Presentation by: KathleenTebb, PhD Assistant Professor of Pediatrics Division of Adolescent Medicine University of California, San Francisco 11.12.09

  2. Introduction Eval. Research Examples: • Influence of formula gift packs on breastfeeding cessation • Intervention to Improve Infant & Toddler Immunizations • Mentoring Program for Teen Mothers & Babies • Improving Chlamydia Screening

  3. Presentation Objectives • Review Research Program that Utilized 3 Different Eval. Components • Understand & Appreciate the Value of Different Types of Program Eval. • Emphasis on Value of Process Evaluation

  4. Utilizes Three Types of Evaluation: 1. Formative 2. Process 3. Outcome/Summative Evaluation of Intervention to Improve CT Screening In Teens

  5. A problem well stated is a problemhalf solved….

  6. Problem: CT the Silent Epidemic • CT is most common reportable bacterial infection • Highest among 15-25 yo females • Most infections (70-80%) are asymptomatic • Untreated CT can cause PID & sequelae • Cost US health system $3-4 bil /yr • Easy to test, easy to treat • Screening rates remain unacceptably low

  7. What to do? Goal: Increase CT Screening Rates Setting: KP, N CA • Large Patient Population • Large # of Clinics (randomization) • Data Infrastructure, but indv’l clinics similar to small group practices • Existing Relationship with key champion

  8. Formative Evaluation Step 1: Needs assessment: • Clinician/Staff Barriers • SA rates • CT screening rates

  9. Formative Evaluation Clinicians Findings: • Discomfort speaking to teens about sexual activity • Difficulty establishing confidentiality • Time constraints – competing priorities • Misperceptions about teen SA & CT screening

  10. Formative Evaluation Admin data: • Very poor overall CT screening rates • Site specific screening rates

  11. Formative Evaluation Step 2: Pilot Test Intervention • Friendly/receptive site • Close proximity • Work out major kinks (implementation/data)

  12. RTC: Methods • Step 3: Randomized Clinical Trial of 10 clinics • Intervention: 5 Clinics • Control: 5 Clinics

  13. Clinical Practice Improvement Model Engage Team Building Re-Design Clinical Practice Sustain the Gain

  14. Clinical Practice Improvement Model • Leadership • Best practices • Define gap • Raise Awareness Engage Team Building Re-Design Clinical Practice Sustain the Gain

  15. Clinical Practice Improvement Model • ACTeam • Skills & tools Engage Team Building Re-Design Clinical Practice Sustain the Gain

  16. Clinical Practice Improvement Model Engage • Customize • Define success & • measures for it Team Building Re-Design Clinical Practice Sustain the Gain

  17. Clinical Practice Improvement Model Engage • Monitor performance • Continuous improvement Team Building Re-Design Clinical Practice Sustain the Gain

  18. Rapid Cycle Changes • Establish ACTeam • Monthly Meeting • Set Goal • Identify barriers • Decide solution • Try it out • Assess • Repeat “cycle” % Change in STD Screening Rate S t a t u s Q u o Time in months

  19. Results of Outcome Eval.

  20. Average screening rates & 95% CI by time and group 0.6 Treated Control 0.4 Proportion screened 0.2 0.0 -2 to 0 0 to 3 3 to 6 6 to 9 Months post-intervention Shafer, Tebb, et al. JAMA. 2002

  21. 2. Process Evaluation • Process evaluation examines each component of the intervention implementation • How was intervention implemented? • Was it implemented as planned? • -- Resources used, activities, quality, etc…

  22. Process Eval: Clinic Flow Chart Cue Charts Vitals Provider Encounter Urines To Lab Follow-Up CT+ ID eligible teens (age/gender) Charts stamped with cue Obtain & Record Sex Hx SA teens give urine sample Cue MD with Lab Slip Confidential Contact # MD gives teen CT info. Confirm Confidential Contact # MA refrigerates urine MA enters info in log book Runner takes urines to lab Lab runs CT test RN contacts CT + teen: confid. # Teen comes to clinic for Rx RN enters Rx in STD log book

  23. Process Eval. Cont. • Admin Data – clinic records, log books, etc. • Regular mtgs with Providers/staff • SRA observations • Chart review

  24. Process Evaluation Components CT Study: Multiple Methods Direct Observation recorded in log books 1:1 Interviews & monthly team mtgs Anonymous surveys, staff, providers, teens Admin Data: -- clinic records -- chart reviews

  25. Chart Review Ex. • Central laboratory database – identify consecutive sample • Retrospective chart review – by independent clinician • Standardized data tracking form

  26. Does Identification Lead to Follow-up? Tracking Form Based on CDC Guidelines • Appropriate antibiotics • Counseling on safer sex • Partner notification and treatment • Lab tests for other STI’s • Re-test at 3 months & as needed

  27. Results: Treatment and Follow-up (N=122)

  28. Process Eval. Lesson Learned Revealed Important Quality Gaps • CT screening lead to Rx but… • Successful identification does not always lead to successful management • Also… only 1/3 teens WCV in given year Hwang L., Tebb K., et al. Archives of Pediatrics & Adolescent Med. 2005

  29. NEXT STEP CT Screening in UC Similar intervention approach Similar evaluation methods

  30. Process Evaluation cont. Determining Failure: Implementation vs. Theory Implementation failure: • Program is not implemented as planned Theory failure: • Program is implemented as planned • Intervention does not produce intermediate results, and/or desired outcome

  31. What about the teens? • Outcome eval. info about CT screening rate, but no info from patient perspective • Anonymous post UC visit survey (N=365) • Clinician Communication& Teen Acceptability

  32. What about the teens? • High acceptability • Sexual Health 84% • Urine CT test 80% • Acceptability significantly associated with: • Clinician explained confidentiality • Knows how to “talk to teens like me” • “Listened carefully as I explained my concerns” Miller K, Tebb K., et al. Archives of Pediatrics & Adolescent Med. 2007

  33. Program Evaluation Challenges • Dealing with the “politics” of a program • Having program design/policy change mid course • Balancing tensions between rigor and practicality (for decision-makers) • Multiple stakeholders: clients, clinicians, parents… • Obtaining $$ & support for strong designs

  34. Lessons Learned: Valuing the Process • Gave over-worked staff sense of importance, success & control over workplace • Flexible, one solution does not fit all • UC more challenging than WC, different settings, different results • Identification of specific component processes & resources support TRIP

  35. Lessons Learned cont. • Multiple evaluation components: • lead to a better intervention design; • informed the interpretation of results along the way • multiple sub-studies, publications, preliminary data, funding support

  36. Re-Aim (Glasgow, 2001) • Reach, Effectiveness, Adoption, Implementation & Maintenance • Approach to address translation of research into practice • Examines the robustness or consistency of results across patient, setting, and clinician subgroups, as well as costs • www.re-aim.org for more info & sample studies

More Related