Framework for Excellence

2. Framework for Excellence Measuring Results Which helps in: Refining Site Analysis Marketing Curriculum Design Needs Assessment Course Delivery and Development Further Measurement and Evaluation!

3. PRESENTERS Debbie Isenberg, MPH, CHES and Margaret Clawson, MPH Southeast AETC Intensive On-Site Training Evaluation: A Mixed Methods Approach (Slides 6-13) Jennifer Gray, RN, PhD Texas/Oklahoma AETC Women & HIV Symposium Qualitative Outcome Evaluation (Slides 14-21) Brad Boekeloo, PhD, Sc.M. University of Maryland Impact of ASSESS on Adolescent Condom Use (Slides 22-27)

4. Measurement and Evaluation Why evaluate? To determine if the training was successful in meeting aims (for participants and faculty) To decide how to change training content To improve the quality of training Why measure provider behavior change? To determine if training has the desired effect on participants and ultimately, on quality of care

5. Kirkpatrick�s Model (from Kirkpatrick, Donald L. Evaluating Training Programs (2nd edition) 1998)

6. Intensive On-Site Training Evaluation: A Mixed Methods Approach Debbie Isenberg, MPH, CHES Margaret Clawson, MPH Southeast AIDS Training and Education Center

7. Background Intensive On-Site Training (IOST) Involves training, consultation, technical assistance and information dissemination (Levels I-V) Targeted towards new Ryan White Title III and other rural health sites Central office-based clinical instructor spends a half day to a full day at the site

8. Study Overview Main research questions Process and Reaction (Reaction and Learning) What was the quality of the training? How well were learning objectives met? What are the trainees� intentions to change their clinical practice? Outcome (Learning and Behavior) How has the provider�s experience in the clinical training program impacted his/her ability (if at all) to provide HIV quality care to PLWH?

9. Study Protocol Phase One Post training CQI form completed by participants Phase Two Recruitment packets mailed 3 months after last IOST Research staff contact potential participants 1 week later for interview Phase Three Reminder letter for 2nd interview sent 9 months after initial interview (total 12 months post IOST) Research staff contact participants 1 week later for interview Recruitment packets mailed 3 months after last IOST Recruitment letter Written informed consent information sheet Demographics survey Interview questions Research staff contact potential respondents 1 week later confirm packet receipt answer questions obtain oral consent schedule an interview time Recruitment packets mailed 3 months after last IOST Recruitment letter Written informed consent information sheet Demographics survey Interview questions Research staff contact potential respondents 1 week later confirm packet receipt answer questions obtain oral consent schedule an interview time Recruitment packets mailed 3 months after last IOST Recruitment letter Written informed consent information sheet Demographics survey Interview questions Research staff contact potential respondents 1 week later confirm packet receipt answer questions obtain oral consent schedule an interview time Recruitment packets mailed 3 months after last IOST Recruitment letter Written informed consent information sheet Demographics survey Interview questions Research staff contact potential respondents 1 week later confirm packet receipt answer questions obtain oral consent schedule an interview time

10. Content: Phase Two and Three Written Demographic Assessment (PIF+) Semi-Structured Phone Interview (Tape recorded) Quantitative: participant asked to rate the effect of training in each specific training area Qualitative: participant asked to give concrete examples of how training has affected their skills in the clinical area If no effect reported, participants are asked for more explanation Topic Areas:Provider knowledge and ability Identification of risk behavior, signs and symptoms of early and advanced HIV disease Viral load, CD4 counts, and treatment implications Comorbidity of substance abuse and mental illness PEP and PHS Guidelines Topic Areas:Provider knowledge and ability Identification of risk behavior, signs and symptoms of early and advanced HIV disease Viral load, CD4 counts, and treatment implications Comorbidity of substance abuse and mental illness PEP and PHS Guidelines

11. IOST Results 17/25 clinicians have participated (68% participation rate) 2 MDs, 9 NPs, 5 RNs, 1 CNM From AL, NC, SC & GA Areas most frequently mentioned where change occurred: Client communication/education Labs Medication Identification of early signs and symptoms of infection Convenience sample Quote: "[the SEATEC trainer] showed us some pictures of some different things like oral candidiasis and the patient that we actually diagnosed with HIV came in and had thrush. And I actually saw that before the doctor did and you know I told him that something�s not right with the patient. So we did an in-house HIV test like the Oraquick and it was positive. So had she not shown me what it looked like, I would have never known what I was looking at when the patient was describing his discomfort� Convenience sample Quote: "[the SEATEC trainer] showed us some pictures of some different things like oral candidiasis and the patient that we actually diagnosed with HIV came in and had thrush. And I actually saw that before the doctor did and you know I told him that something�s not right with the patient. So we did an in-house HIV test like the Oraquick and it was positive. So had she not shown me what it looked like, I would have never known what I was looking at when the patient was describing his discomfort�

12. Strengths and Challenges

13. Lessons Learned Think about what motivates the training audience to participate in the study when deciding on study design Develop the protocol to lower respondent form and time burden Don�t be afraid to change the protocol midway in the study if not working Consider the resources that you have to collect and analyze the data in choosing a study design

14. Qualitative Outcome Evaluation If all you have is a rope, tie a knot in it and use it.

15. TX/OK AETC Women & HIV Symposium Environmental scans documented increasing number of infected women. Needs assessments indicated lack of knowledge about gender-specific aspects of care. Multidisciplinary planning committee developed conference objectives and selected speakers.

16. Evaluation Plan Send email to all registrants a month after symposium Ask simple open-ended questions that can be answered by members of any discipline

17. Decision to Use this Evaluation Method Wanted to assess for behavior change (Level III- Kirkpatrick�s model) Limited resources and time No existing tool found that met needs First symposium- wanted qualitative feedback

18. TX/OK AETC Women & HIV Symposium

19. Findings Impact on patients 13 had taught patients information learned at the symposium 3 had used info for referrals 3 system changes- i. e. assessment forms, clinical strategies

20. Findings Shared information with others 8 informally 1 structured 4 materials Most common topics 1) medications and adherence 2)HIV in general

21. Lessons What we did right Announced at symposium Only 2 questions to encourage response What we would change Confirm email addresses Provide incentive Next steps Use these responses to develop short tool

22. Impact of ASSESS on Adolescent Condom Use Bradley O. Boekeloo, Ph.D., Sc.M. University of Maryland

23. Methods Patient education materials included a: risk assessment audiotape, abstinence brochure, condom brochure, and risk reduction �pyramid�. Physician education materials included: video modeling physician-adolescent risk reduction discussion, binder of educational materials, and risk assessment template.

27. Conclusion Preparing adolescents to discuss sexual behavior with their physician and educating them about sexual risk reduction before their health exam, then prompting the provider to conduct follow-up assessment and counseling can increase adolescent condom use.

28. Contact Information SEAETC: Study Evaluator, Margaret Clawson 404-712-8448, mclawso@emory.edu Study Director, Debbie Isenberg 404-727-2931, disenbe@emory.edu Texas/Oklahoma AETC Jennifer Gray, RN, PhD, Project Evaluator 817-272-2776, jgray@uta.edu NMAETC, Delta AETC Brad Boekeloo, Ph.D. 301-405-8546, bb153@umail.umd.edu ASSESS materials available at www.socio.com AETC National Evaluation Center Janet Myers, PhD, MPH, Director 415-597-8168, jmyers@psg.ucsf.edu

29. Conference Call Evaluation Call 7: June 24, 2004 http://www.ihi.org/feedback/survey.asp?surveycode=AETCCall062404 Survey Code: AETCCall062404 Call 8: July 13, 2004 http://www.ihi.org/feedback/survey.asp?surveycode=AETCCall072704 Survey code: AETCCall071304 For assistance contact: Lorna Macdonald at lmacdonald@ihi.org