F/C AETC-Project ECHO™ Team

1. Florida/Caribbean AETC Project ECHO™ Program: Co-management of HIV/AIDS by Community Health Care Providers and SpecialistsJoanne J. Orrick, PharmD, AAHIVEAssociate DirectorFlorida/Caribbean AETCUniversity of South Florida Center for HIV Education and Research

2. F/C AETC-Project ECHO™ Team Sean McIntosh, AS Program Coordinator Adis Kreso, BS Statistical Data Analyst Danchelle Jones Program Support Maximo Lora, BA Marketing & Technical Support Christine Makar, BA Continuing Education Support Nathan McMullen Technical Support Theresa C. Skipper, MPH Logistical Management Jeffrey Beal, MD, AAHIVS Clinical Director & Principal Investigator Kimberly Molnar, MAcc Director, F/C AETC Saniyyah Mahmoudi, ARNP Faculty Facilitator Joanne Orrick, PharmD, AAHIVE Faculty Facilitator Todd Wills, MD Faculty Facilitator Hesborn Wao, PhD Evaluator

3. Outline • Implementation • Evaluation • Preliminary results • Challenges • Future Directions • Question/Answer

4. Implementation • F/C AETC site visit to University of New Mexico Project ECHO™ • Weekly implementation meetings with F/C AETC program staff • Technical assistance from University of New Mexico Project ECHO™ • Polycom vs. Adobe Connect platform • Provider interest gauged at Annual F/C ATEC conference and through email contact

5. Session Overview • Sessions scheduled twice per month • 15 minute didactic presentation by a F/C AETC faculty member • Remainder of time (1.5 hours scheduled per session) spent discussing patient cases submitted by providers throughout the region • F/C AETC faculty member facilitates discussion among participants and other F/C AETC faculty

6. Case Presentation FormInteractive Fillable PDF

7. Evaluation of the Program Purpose of the evaluation • Determine extent to which F/C AETC Project ECHO™ helps in developing capacity among health care professionals in rural and underserved areas of Florida, Puerto Rico, and the U.S. Virgin Islands to safely and effectively treat HIV/AIDS and to monitor outcomes of this treatment • Learn from our implementation experience so that we can better manage performance and improve program design Program logic model

8. Evaluation of the Program Type of evaluation • Needs evaluation: Determine care provider needs • Process evaluation: Assess program activities/processes • Outcome evaluation: Assess program effects/impact Time points • Participant registration: Prior to scheduled session • Immediate post-training: ≤1 week (All participants) • Follow-up evaluation: 4-6 months (Case presenters) Methodology: Mixed-methods approach • Quantitative: Survey (numeric items) • Qualitative: Survey (open-ended items)

9. Program Logic Model

10. Evaluation Data Collection Follow-up evaluation • Patient benefited? • Change in KSA • KSA learned used to manage other patients? • KSA learned shared with others at workplace? • Challenges faced while implementing suggestions Immediate post-training • Demographics • Objectives for attending • Kirkpatrick's levels: • Satisfaction/reaction • Intention to change Knowledge, Skills, & Attitudes (KSA) • Learning (change in KSA) • KSA transfer to workplace • Improved patient outcome • Suggestions for improvement Kirkpatrick, D. L. (1994). Evaluating training programs. San Francisco: Berrett-Koehler Publishers, Inc.

11. Evaluation of the Program Data analysis Aims • Determine HCP’s satisfaction with the program • Examine change in HCPs’ KSA pre and post-session • Determine extent KSA are transferable to workplace • Determine degree program impacts patient outcomes Procedure • Retrieve data from NEC system (Excel) and clean • Ensure data integrity: Evaluation/data management team separate from program implementation team • Present results using graphs, charts and text

12. Preliminary Results Participants (6 ECHO sessions: 11/2011 - 02/2012)

13. Preliminary Results

14. Preliminary Results Examples of objectives by category Increase knowledge (41% participants) • “Broaden my knowledge about HIV resistance testing” • “Discuss ideas for increasing medication compliance” • “Increase my knowledge on perinatal HIV” • “Learn from every case I hear about” Update knowledge (20% participants) • “Stay updated and learn from others' experiences” • “Obtain newest information in HIV care • “Educate & update on providing holistic care to HIV pts”

15. Preliminary Results Examples of objectives by category (cont.) Observe (13% participants) • “Observe Perinatal Project ECHO™, learn about new perinatal guidelines” • “Observe the new ECHO ™session offered by F/C AETC” • “Observe the TeleECHO clinic” Network (8% participants) • “Network with other HIV providers” • “Hear what others perinatal providers around the state are doing”

16. Preliminary Results Level 1: Participants’ satisfaction/reaction

17. Preliminary Results Level 1: Participants’ satisfaction/reaction cont

18. Preliminary Results Level 2a: Learning (Change in Knowledge)

19. Preliminary Results Level 2b: Intention to change in practice Following the F/C AETC-Project ECHO™ session, I intend to make the following changes: • “…consider role-play for pts struggling with disclosure” • “Speak more directly with pts about missed doses…” • “Provide local providers with updated guidelines for administration of AZT” • “Change regimen on the patient that was discussed” • “Network with persons with similar duties”

20. Preliminary Results Level 3: Transfer of KSA to workplace

21. Preliminary Results Level 3: Transfer of KSA to workplace

22. Preliminary Results What participants liked MOST: • “Ability to participate without travel” • “Being able to interface with a panel of experts…” • “Expert discussion, 2) Enough data on the patient to be able to follow the discussion, 3) Collegial nature of the discussion” • “Hearing from experts, discussion with logic behind it, the different opinions” • “The format: Didactic and case presentations”

23. Preliminary Results What participants liked LEAST: • “It would be nice to see all the people on the call” • “Would like more case presentations each session” • “Early on - the technical difficulties” • “Not enough time to ask questions about our own case” • “Could not see all participants” • “The difficulty in getting the webcam to operate” • “Technical difficulties”

24. Preliminary Results How F/C AETC-Project ECHO™ can be improved • “Have a constant banner/header with how to mute/unmute phone lines” • “Include more non-clinical/treatment educational resources and alternative ways non-clinical/ treatment health care providers can implement the material and utilize within the community & patients.” • “Have more information presented on cases” • “Decrease background noise, presentation of more than one case study”

25. Resolution of Challenges • Troubleshoot technical difficulties • Check list for each program • Increase participants who participate with webcams • Engage more participants in the discussion • Increase the number of F/C AETC faculty who serve as facilitators • Increase the number of participating clinics and providers

26. Future Directions • Curriculum outlined through June 2012 • Includes Pediatric/Adolescent specialty sessions • Develop curriculum and brochure detailing all programs for July 2012-June 2013 AETC grant year • Expand to include other sub-specialty sessions (e.g. Hepatitis and HIV co-infection) • Collaborate and share resources with other AETCs using the Project ECHO™ model