410 likes | 651 Views
Framework for Excellence. Measuring Results Which helps in:Refining Site AnalysisMarketingCurriculum DesignNeeds AssessmentCourse Delivery and DevelopmentFurther Measurement and Evaluation!. PresentersCheryl Hamill, RN, MS, ACRN
E N D
2. Framework for Excellence Measuring Results
Which helps in:
Refining Site Analysis
Marketing
Curriculum Design
Needs Assessment
Course Delivery and Development
Further Measurement and Evaluation!
3.
Presenters
Cheryl Hamill, RN, MS, ACRN & Nancy Showers, DSW
Delta Region AETC
HIVQual Results 2002-2003
Sample RW Title III Community Health Center in Mississippi
Mari Millery, PhD
NY/NJ AETC
Lessons from Assessing Knowledge & Practice Outcomes of Level III Trainings
Jennifer Gray, RN, PhD & Richard Vezina, MPH
TX/OK AETC, Women & HIV Symposium (JG)
Pacific AETC, Asilomar Faculty Development Conference (RV)
Debbie Isenberg, MPH, CHES & Margaret Clawson, MPH
Southeast AETC
Intensive On-Site Training Evaluation: A Mixed Methods Approach
Brad Boekeloo, PhD, ScM
NMAETC, Delta AETC
Analysis of HIV Patient-Provider Communication
4. Measurement and Evaluation Why evaluate?
To determine if the training was successful in meeting aims (for participants and faculty)
To decide how to change training content
To improve the quality of training
Why measure provider behavior change?
To determine if training has the desired effect on participants and ultimately, on quality of care
5. Kirkpatricks Model (from Kirkpatrick, Donald L. Evaluating Training Programs (2nd edition) 1998)
6. The HIVQUAL Project
Nancy Showers, DSW
Delta Region AETC
7. The HIVQUAL Project Capacitybuilding and organizational support for QI
Individualized on-site consultation services
Strengthen HIV-specific QI structure
Foster leadership support for quality
Guide performance measurement
Facilitate implementation of QI projects
Train HIV staff in QI methods
Performance measurement data with comparative reports
Partnership with HRSA to support quality management in Ryan White CARE Act community-based programs
8. HIVQUAL Participants - 2003
9. Annual PAP Test
10. Annual Syphilis Screen
11. Hepatitis C Status Known
12. Adherence Discussed
13. Viral Load Every 4 Months
14. MAC Prophylaxis (CD4<50)
15. Annual Dental Exam
16. Annual Mental Health Assessment
17. Delta AIDS Education and Training Center (DRAETC)Mississippi LPS - Training Summary ReportReporting period: July 1, 2002 - June 30, 2003for Targeted RW Title-Funded Community Health Centers Cheryl Hamill, MS, RN, ACRNInstructor of MedicineResource Center Directorhttp://hivcenter.library.umc.edu
19. Lessons from Assessing Knowledge and Practice Outcomes of Level III Trainings Mari Millery, PhD
20. Decided to focus more outcome evaluation efforts on Level III because it is the most intensive and a high priority modality; and participants can be asked to devote time to extra paperwork
Pre-test, post-test, and 3-month follow-up surveys
Measures:
Self-rating of comfort in performing clinical tasks
Case-based knowledge questions
23. Lessons Learned Can be done but getting follow-up surveys back is a challenge
Preliminary results are encouraging self-reported practice comfort and case-based knowledge questions appear to work as measures
Survey needs to be minimum length
Dropped knowledge questions in post-test because they were too soon after baseline post-test focuses on feedback on training
Nature of Level III varies: intensity/length, profession trained, topics covered, etc.
Developed special versions for nurses and HepC
40 surveys collected with revised instruments this year still working on getting all follow-up surveys back
24. Measuring Training Outcomes Through Qualitative Interviewing TX/OK AETC Women & HIV Symposium (JG) and Asilomar Faculty Development Conference (RV) Jennifer Gray, RN, PhD (JG) Richard Vezina, MPH (RV)
TX/OK AETC Pacific AETC
25. TX/OK AETC Women & HIV Symposium (JG) First time region-wide symposium
Multidisciplinary planning committee
Lack of knowledge about gender-specific care
Increased # of HIV infections among women in the region.
Symposium goal:
Improved care of HIV+ women Annual region-wide training conference
125 Participants, all PAETC faculty and program staff
Conference goals:
Improved skills and knowledge among faculty/trainers
Improved training outcomes throughout region as a result of staff development
26. Evaluation Plans JG
Email one month post to all registrants
Simple open-ended questions, for all disciplines
Identify how content was used with patients and shared with peers.
RV
Post-Post:
Form A: Self-assessment at end of Conference
Identify skills and content learned, areas in which to integrate new skills and content
Form B: 6 month Follow-Up
Individualized telephone interviews, reviewing Form A
Focus on how skills/content were applied; barriers
27. Why these evaluation methods? Able to assess at multiple levels (Kirkpatrick model):
Level 2 (Learning: improved knowledge) (RV)
Level 3 (Behavior: change in practices) (JG, RV)
Seeking specific content regarding conference (RV)
Limited resources and time (JG)
No existing tool found that met needs (JG)
28. Findings Major Themes: (RV)
Identified high need for continued skills training
Transferred new skills/information to coworkers and employees
Barrier to continued integration: Time constraints
Major Themes: (JG)
Impact on patients
13 had taught patients information learned at the symposium
3 had used info for referrals
3 system changes- i. e. assessment forms, clinical strategies
Shared information with others:
8 informally, 1 structured, 4 created materials
Most common topics: medication/adherence, HIV in general
29. Strengths & Challenges of Methods What went well:
Announced at end of symposium/conf. (JG, RV)
Brief instrument encouraged higher response (JG)
Longer instrument yielded rich responses (RV)
Whats Next:
Provide Incentives (JG, RV)
Change instrument
Shorter, easier instrument for higher response rate (RV)
longer instrument for greater depth (JG)
More effective confirmation of contact information (JG, RV)
30. Intensive On-site Training Evaluation: A Mixed Methods Approach Debbie Isenberg, MPH, CHES
Margaret Clawson, MPH
Southeast AETC
Intensive On-Site Training (IOST)
Involves training, consultation, technical assistance and information dissemination (Levels I-V)
Targeted towards new Ryan White Title III and other rural health sites
Central office-based clinical instructor spends a half day to a full day at the site
Intensive On-Site Training (IOST)
Involves training, consultation, technical assistance and information dissemination (Levels I-V)
Targeted towards new Ryan White Title III and other rural health sites
Central office-based clinical instructor spends a half day to a full day at the site
31. Study Overview Main research questions
Process and Impact (Reaction and Learning)
What was the quality of the training?
How well were learning objectives met?
What are the trainees intentions to change their clinical practice?
Outcome (Learning and Behavior)
How has the providers experience in the clinical training program impacted his/her ability (if at all) to provide HIV quality care to PLWH?
32. Study Protocol Phase One
Post training CQI form completed by participants
Phase Two
Recruitment packets mailed 3 months after last IOST
Research staff contact potential participants 1 week later for interview
Phase Three
Reminder letter for 2nd interview sent 9 months after initial interview (total 12 months post IOST)
Research staff contact participants 1 week later for interview Recruitment packets mailed 3 months after last IOST
Recruitment letter
Written informed consent information sheet
Demographics survey
Interview questions
Research staff contact potential respondents 1 week later
confirm packet receipt
answer questions
obtain oral consent
schedule an interview time
Recruitment packets mailed 3 months after last IOST
Recruitment letter
Written informed consent information sheet
Demographics survey
Interview questions
Research staff contact potential respondents 1 week later
confirm packet receipt
answer questions
obtain oral consent
schedule an interview time
Recruitment packets mailed 3 months after last IOST
Recruitment letter
Written informed consent information sheet
Demographics survey
Interview questions
Research staff contact potential respondents 1 week later
confirm packet receipt
answer questions
obtain oral consent
schedule an interview time
Recruitment packets mailed 3 months after last IOST
Recruitment letter
Written informed consent information sheet
Demographics survey
Interview questions
Research staff contact potential respondents 1 week later
confirm packet receipt
answer questions
obtain oral consent
schedule an interview time
33. Content: Phase Two and Three Written Demographic Assessment (PIF+)
Semi-Structured Phone Interview (Tape recorded)
Quantitative: participant asked to rate the effect of training in each specific training area
Qualitative: participant asked to give concrete examples of how training has affected their skills in the clinical area
If no effect reported, participants are asked for more explanation Topic Areas:Provider knowledge and ability
Identification of risk behavior, signs and symptoms of early and advanced HIV disease
Viral load, CD4 counts, and treatment implications
Comorbidity of substance abuse and mental illness
PEP and PHS Guidelines
Topic Areas:Provider knowledge and ability
Identification of risk behavior, signs and symptoms of early and advanced HIV disease
Viral load, CD4 counts, and treatment implications
Comorbidity of substance abuse and mental illness
PEP and PHS Guidelines
34. Strengths and Challenges
35. Lessons Learned Think about what motivates the training audience to participate in the study when deciding on study design
Develop the protocol to lower respondent form and time burden
Dont be afraid to change the protocol midway in the study if not working
Consider the resources that you have to collect and analyze the data in choosing a study design
36. Analysis of HIV Patient-Provider Communication Bradley O. Boekeloo, Ph.D., Sc.M.
University of Maryland
37. Methods Providers Randomized (n=8)
Brief cultural competency training vs. none
Audiotapes of HIV Visits (n=24)
3 patient visits tape recorded per physician.
Tapes transcribed.
Patient Exit Questionnaire (n=24)
Interviewer read patient questions and patient answered on an answer form.
40. Hypothesis Based on Exploratory Data and Next Steps Brief Intervention not enough for change
Patients may be more comfortable discussing medical therapy than personal risk behaviors
Try to determine whether different types of communication on audiotapes account for differences in patient comfort communicating with physician.
41. Presenter Contact Information NY/NJ AETC:
Mari Millery, PhD
212-305-0409
mm994@columbia.edu
Delta Region AETC:
- Cheryl Hamill, RN, MS, ACRN
601-984-5552
chamill@medicine.umsmed.edu
- Nancy Showers, DSW
732-603-9681
njshowers@aol.com
Southeast AETC:
- Margaret Clawson, MPH
404-712-8448 mclawso@emory.edu
- Debbie Isenberg, MPH, CHES
404-727-2931 disenbe@emory.edu
42. Conference Call Evaluation Call 8: July 27, 2004
http://www.ihi.org/feedback/survey.asp?surveycode=AETCCall072704
Survey Code: AETCCall072704
For assistance contact: Lorna Macdonald at lmacdonald@ihi.org