Jill Sherman, MPH Wayne Warry, PhD Centre for Rural and Northern Health Research

1. Developing an Evaluation Framework for the LEGs Initiative: Key Findings Jill Sherman, MPH Wayne Warry, PhD Centre for Rural and Northern Health Research Janice Willett, MD Northern Ontario School of Medicine Northern Health Research Conference Timmins, June 4-6, 2015

2. Conflict of Interest Declaration: Nothing to Disclose Presenter: Jill E. Sherman Title of Presentation:Developing an Evaluation Framework for the LEGs Initiative – Key Findings I have no financial or personal relationships to disclose This project was supported by the Northern Ontario Academic Medicine Association (NOAMA) AHSC AFP Innovation Fund award.

3. Learning Objectives • Learn about the progress of the LEGs Initiative, from the perspectives of LEG members • Understand why and how an evaluation framework was developed.

4. Background: Alternate Funding Plans (AFPs) • International challenge: A declining supply of physicians who are willing and able to train medical students and residents • Ontario AFPs: Funded by the MOHLTC under negotiated agreements with the OMA and individual Academic Health Science Centres and their faculty. • AFPs: Supplement the income of physician preceptors, to: • Increase physicians’ willingness to supervise learners • Offset (some of the) opportunity costs of teaching (“clinical repair”) • Encourage the development of academic culture at AHSCs • Ensure delivery of high quality, evidence-based care • Strengthen medical education • Increase research and innovation

5. Local Education Groups (LEGs)Different from other AFPs Other Ontario AFPs • Centrally administered and implemented through existing academic departments • Include academic and clinical faculty • Include funds for “clinical repair” LEGs Initiative • Independent association (NOAMA) created to administer the AFP • Implemented through 40+ LEGs at distributed learning sites across Northern Ontario • All NOSM faculty are clinical faculty • No funds for clinical repair

6. About LEGs • Local Education Groups (LEGs) are self-organized groups of physician clinical faculty with group responsibility for academic deliverables and clinical teaching. • The LEG provides an organizational structure and support to clinical faculty working in distributed learning sites • Eligible for extra funds (administrative support, professional development funds, research development funds) • LEG membership is optional (at this time); 2/3 of AFP members currently belong to a LEG. • Group must submit a proposal, have an approved governance agreement, before operating as a LEG • 42 LEGs have been approved (2015); 23 at start of project (Fall 2013). • Diversity of LEGs • Size: Range of 5 to 100 physician members • Organizational type: Community (multispecialty), Hospital Department; Family Health Team; Pan-Northern • Governance: Each LEG determines how funds are allocated among members

7. Research Highlights • Method: Semi-structured interviews (January–April 2014) • 8 Physician Leads, 1 Administrator (2 declined) • Purposively selected from established LEGs • Mix of hospital departments, small/medium/large community LEGs • Results: Major themes identified Benefits of becoming a LEG Activities and Innovations Factors influencing LEG development Challenges and suggestions for improvement Goals and ideas of success

8. Perceived Benefits of Becoming a LEG • Improved organization & delivery of medical education • Better organized • Funding for administrative support • Development of a collective approach to medical education • Local ownership • Greater consistency & accountability in delivering curriculum • Increases in academic activity • Increases in learner placements • New modules/activities developed • Increases in locally delivered CEPD • Greater flexibility and fairness in use of funds • Encouragement & support for research and innovation • Enhanced status of clinical teaching in the community • Community benefits • Direct and indirect clinical recruitment • Special projects • Community acceptance of learners

9. “Now that this is more structured and organized, and now that the LEG handles a lot of what would otherwise become a headache or nuisance for individual faculty members, I don’t think they’re experiencing any difficulty in placing all the clerks, in addition to elective students and residents that come through the community.” (Medium Community LEG)

10. Some Challenges • First wave of LEGs (early adoptors) • Inventing the LEG from the ground up – no models, templates • Organizational and administrative challenges • LEG activity is “extra” – lower priority • Burn out of leaders • Constraints to increasing academic activity • Physician shortages / Lack of spare capacity for innovation • Need to build capacity for research, KTE • Inter-organizational challenges • Perception of frequent rule changes, lack of clarity on rules • NOSM systems had not adapted to LEGs

11. Towards an Evaluation Framework:Developing a Program Logic Model 5 Domains of LEG Activity • Medical Education & Program Development • Professional Development • Scholarship, research, and innovation • Recruitment, retention, and community engagement • Leg Administration

12. Recommendations for Evaluation Planning • Allow 3-5 years to prepare for evaluation • Too early to evaluate, need time to develop data collection and reporting systems • Include preconditions as explanatory indicators • Factors outside of LEG control affect LEG performance • Align indicators with other data requirements • Minimize administrative burden on LEGs • The evaluation framework must be flexible • Evaluation priorities of LEGs vary; data should be useful to the LEGs themselves • The evaluation framework must be dynamic • Goals will change over time • Qualitative evaluation will remain important

13. What’s next? • NOAMA Working Group is addressing recommendations • Barriers, challenges, and participant recommendations • Continuation of the Evaluation Project – • Will develop and a test small number of indicators • Focus on academic domains • Professional Development, • Scholarship, Research and Innovation • Collect baseline data

14. For more information... Jill Sherman, Research Associate Centre for Rural and Northern Health Research, Laurentian University Sudbury, ON P3E 2C6 Tel: 705-675-1151 x4359 Email: jsherman@laurentian.ca www.cranhr.ca Dorothy Wright, Executive Director NOAMA Balmoral Street Centre, Room 1007 955 Oliver Road Thunder Bay, ON P7B 5E1 Tel: 807-766-7470 Email: dwright@noama.ca www.noama.ca