Resident Research Using a Logic Model to Design, Implement and Evaluate a Program

1. Resident Research Using a Logic Model to Design, Implement and Evaluate a Program Mary Jo Knobloch, MPH, CPH WREN Conference 2009

2. Association of American Medical Colleges Study • Primary healthcare professionals need more training in basic research skills and increased opportunities to become involved in research • Research education in the early years of medical training associated with continued interest in scholarly activity

3. Objectives • To demonstrate use of a logic model as a framework for program development, implementation and evaluation • To present preliminary results of 2008 and 2009 assessment of first year residents

4. Current Setting • Marshfield Clinic Residency Program • Each program different in research learning approach • 5 Dermatology • 11 General Surgery • 19 Internal Medicine • 8 Medicine/Pediatric • 11 Pediatric • 2 Pharmacy

5. Logic Model Framework • Basic Framework • A living document • Inputs • Outputs – Activities and Participation • Outcomes-Impact • Short • Medium • Long

6. UWEX Logic Model Site • http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

8. Short-term Outcome Measures Using Resident Research Database THE NUMBERS - • Number of resident projects • Completed resident projects • Abstracts, posters, manuscripts • Number of active research mentors • Diversity of project study designs • QI and public health or advocacy projects

9. Resident Research DatabaseTracking the Short-Term Outcomes

10. Medium and Long-term OutcomesUsing Assessment Tool • Increase in knowledge and skill level related to basic research methods and processes • Increase in knowledge related to identification of study design and statistical methods • Increase in skill level related to literature searches • Change in confidence level related to understanding and implementation of research

11. Medium-Long Term Outcomes • Continued interest in scholarly activity • Use of evidence to guide practice • Ongoing provision of multiple training opportunities

12. Results to Date • Projects (the numbers) • Assessment results (the learning)

13. Projects • Incidence of Eosinophilic Esophagitis in MESA • Relationship of Urine Cotinine Levels and Wound Implications in Patients Undergoing Reduction Mammoplasty • Effects of HMR Weight Loss Intervention Program on Cardiovascular Disease Risk Factors • Effect of Hyperglycemia on Outcomes in Acute Exacerbation of COPD

14. Projects • CNS Blastomycosis in Marshfield Clinic System: A Case Series • A Pilot Project to Promote Early Literacy • Cardiovascular Morbidity and Mortality among SLE Patients in a Rural Population-based Cohort – ICTR project • Severely Suppressed Bone Turnover and Atypical Skeletal Fragility • Maja Visekruna, MD, Medicine/Pediatrics • Faculty and Resident Perspectives on the Dermatology Integrative Competency Curriculum • Modeling Clearance of Warfarin

15. Projects • Implementation of a Resident Well-Being Survey • Hamman-Rich Syndrome (Idiopathic Acute Interstitial Pneumonitis) • Evaluating Clinical Compliance with IDSA Guidelines for Diagnosis and Management of Lyme Disease • Incidence and Description of Anaplasmosis • Rapunzel Syndrome in a 5 Year-Old (Case Report) • Risk Factors Related to Recurrence of Umbilical Hernia

16. Projects • Renal Cell Carcinoma – Incidence Rates in Wisconsin • A Retrospective Review of Patients Diagnosed with Stenosing Tenosynovitis (Trigger Finger and Thumb) • Comparison of Wounds Before and After Sharp Debridement Using 16S DNA Sequencing • Identification of Drug Therapy Problems through Medication Reconciliation in Heart Failure Patients • The Role of Pharmacist Evaluation in Activation of Adult Asthma Patients (ICTR project). • A Public Awareness Campaign about Insulin Dependent Diabetes

17. Projects • Natural History of Idiopathic Neonatal Hypertension and the Long Term End Organ Effects in Children with Neonatal Hypertension • Do Clinicians Document the AAP Recommended Clinical Information Used to Diagnose and Manage Otitis Media with Effusion? • Defining Services and Referral Patterns for Infants with Positional Plagiocephaly within the Marshfield Clinic System • Implementing a Pharmacy Continuing Professional Development (CPD) Project at Marshfield Clinic • Factors Associated with Successful Matching and Residency Matriculation by Non-traditional Applicants to Dermatology Residency Programs • Effects of Metabolic Syndrome on Disease State and Long-Term Outcomes in Patients with Colorectal Cancer

18. Projects • Are Lytic Bacteriophages against Staph Aureus Found in the Anterior Nares of Humans? • A Rare Presentation of Painless Aortic Dissection as Seizures • Analysis of Dermatology Resident Self-Reported Successful Learning Styles and Implications for Core Competency Curriculum Development • Caustic Ingestion in a 2-year old (Case Report) • Assessment of Medical Residents’ Knowledge of Drug Interactions after an Educational Intervention

19. Combined Results for 2008 and 2009 First-Year Residents(the learning) • Assessment of attitude, confidence level and knowledge • 33 residents

20. Attitude Questions • To be an intelligent reader of the literature, it is necessary to know something about statistics • 100% agree or strongly agree

21. Attitude Questions • I often use statistical information in forming opinions or making decisions • 51% disagree or strongly disagree • Given the chance, I would like to learn more about biostatistics • 75% agree or strongly agree

22. Level of Confidence Questions • Finding clinical practice guidelines for patient care questions • 57% say they have fair or little amount of confidence • Exploring evidence using Cochrane Database • 39% say little or no confidence

23. Confidence Questions • Identifying factors that influence a study’s statistical power • 78% say fair, little or no confidence

24. Confidence Questions • Interpreting P-value for a given result • 70% say fair, little or no confidence • 30% say a lot or complete confidence

25. Knowledge Questions • Which of the following is true regarding p-values? • 27% correct • 73% incorrect • Which of the following is NOT considered a vulnerable population? • 38% correct • 62% incorrect

26. Knowledge Questions • Which of the following is an example of a testable null hypothesis • 40% correct • 60% incorrect

27. Knowledge Questions • Which of the following is true with regard to the differences between efficacy and effectiveness • 33% correct • 66% incorrect • Which one of the following measures of central tendency is least affected by outliers? • 37% correct • 63% incorrect

28. Knowledge Questions • Which statement is not true about confidence intervals? • 36% correct • 63% incorrect • The purpose of a double-blind or double-masked study • 97% correct • 3% incorrect

29. Knowledge • These 2008/2009 first year residents had difficulty answering questions related to • Study design • Identifying discrete versus continuous variables • Differences between efficacy and effectiveness • Definition of bias • Differences between mean, median, mode • Sensitivity and specificity • Definition of prevalence

30. Conclusions • A program planning tool usually used for public health or intervention planning or grant planning can be used to set up and evaluate a resident research program • Tracking systems must be in place • Preliminary assessment of resident attitude and knowledge of basic epidemiology and biostatistics demonstrates the need for continued work in teaching research methods and skills

31. Questions?