AI and the Future of Continuous Health Learning & Improvement

1. NAM Digital Learning CollaborativeAI and the Future of Continuous Health Learning & Improvement Workgroup – Publication Introduction

2. DLC AI & Future of Continuous Health Learning and Improvement Workgroup • Original Charter: to explore the fields of AI and their applications in health and health care • strategies to enhance data integration to advance healthcare AI • practical challenges to AI model development and implementation • opportunities for accelerating progress

3. Initial Workgroup Membership • Jonathan Perlin, CMO HCA, DLC Co-Chair • Reed Tuckson, Tuckson Health Con., DLC Co-Chair • Wendy Nilsen, NSF • Joachim Roski, Booz Allen Hamilton • Howard Underwood, Anthem • Daniel Yang, Moore Foundation • Doug Badzik, Department of Defense) • Carlos Blanco, National Institute on Drug Abuse • Paul Bleicher, OptumLabs • Carla Brodley, Northeastern University • Tim Estes, Digital Reasoning • Daniel Fabbri, Vanderbilt University Medical Center • Kenneth R. Gersing, NIH • Michael Howell, Google • Brigham Hyde , Precision Health Intelligence • Javier Jimenez, Sanofi • Jennifer MacDonald, VA • Nigam H. Shah, Stanford) • David Sontag, MIT • Noel Southall, NIH • Shawn Wang, Anthem • Maryan Zirkle, PCORI • SonooThadaney, Stanford (Workgroup Co-chair) • Michael Matheny, Vanderbilt (Workgroup co-chair) • John Burch, JLB Associates • Wendy Chapman, University of Utah • Jonathan Chen, Stanford University • Len D’Avolio, Cyft • SharamEbadollahi, IBM Watson Health Group • HossienEstiri, Harvard Medical School • Steve Fihn, University of Washington • Jim Fackler, John Hopkins School of Medicine • Seth Hain, Epic • Brigham Hyde, Precision Health Intelligence • Edmund Jackson, HCA • Hongfang Liu, Mayo Clinic • Doug McNair, Cerner • EneidaMendonca, University of Wisconsin Madison • Sean Khozin, FDA • Matthew Quinn, HRSA • Robert E. Samuel, Aetna • Bob Tavares, Emmi Solutions • Howard Underwood, Anthem) • Daniel Yang, Moore Foundation

4. NAM Workgroup Publication Objectives & Scope • Develop a reference document for model developers, clinical implementers, clinical users, and regulatory and policy makers to: • understand strengths and limitations of AI/ML • promote use of these methods and technologies within the healthcare system • Highlight areas of future work needed in research, implementation science, and regulatory bodies to facilitate broader use of AI/ML in healthcare

5. NAM DLC AI Publication: Organization TOPIC Leads • NAM DLC Jonathan Perlin, Reed Tuckson • NAM Program Office Danielle Whicher, Mahnoor Ahmed • Publication Editors SonooThadaney, Michael Matheny • Chapter 1: Introduction SonooThadaney, Michael Matheny • Chapter 2: History of AI Edmund Jackson, Jim Fackler • Chapter 3: Promise/Opportunities for AI Joachim Roski, Wendy Chapman • Chapter 4: Pitfalls/Challenges for AI Eneida Mendonca, Jonathan Chen • Chapter 5: AI Development & Validation Hongfang Liu, Nigam Shah • Chapter 6: AI Deployment in Clinical Settings Steve Fihn, Andy Auerbach • Chapter 7: Regulatory & Policy Issues Doug McNair, Nicholson Price • Chapter 8: Conclusions & Key Needs SonooThadaney, Michael Matheny

6. AI: What Do We Mean? https://www.legaltechnology.com/latest-news/artificial-intelligence-in-law-the-state-of-play-in-2015/

7. “Health & Healthcare” Settings • Direct Encounter-Based Care • “Non-Traditional” Settings: CVS, Home • Population Health Management • “Back Office” Healthcare Administration • Patient/Consumer Facing Technologies

8. Target Audiences • Direct Care Providers • Patients and their Caregivers • Healthcare System Leadership & Admin • Data Scientists (Developers) • Clinical Informatics (Implementers) • Legislative & Regulatory Bodies • Third Party Payors

9. Chapter 2: History & Current State of AI • Discusses history of AI with examples from other industries • Summarize the growth, maturity, and adoption in healthcare as compared to other industries. • Target general audience

10. Chapter 3: Promise & Potential Impact of AI • Focus on the utility of AI for improving healthcare delivery • Discuss near-future opportunities and potential gains from the use of AI • Target General Audiences

11. Chapter 4: Potential Unintended Consequences of AI • Focus on the potential unintended consequences of AI on: • work processes • culture • equity / fairness • patient-provider relationship • workforce composition & skills • Target General Audiences

12. Chapter 5: AI Modeling Development & Validation • Most Technical Chapter • Topics • process for developing and validating models • choice of data, variables, model complexity • performance metrics, validation • Target Model Developers

13. Chapter 6: Deploying AI in Clinical Settings • Focus on implementing and maintaining AI within ‘production’ healthcare domains • Address issues of: • Software development • Integration into a Learning Healthcare System • Applications of Implementation Science • Model Maintenance & Surveillance over Time • Target Healthcare System Leaders & Implementers

14. Chapter 7: Regulatory & Policy Considerations • Summarize key legislative and regulatory considerations for the use of AI in health care • Identify strengths and weaknesses in current framework • Discuss legal liability concerns • Make recommendations to address gaps

15. Chapter 8: Conclusions & Key Needs • build on and summarize key & cross-cutting themes from previous chapters • Recommend key areas for: • Moving the field forward • Highlight over-arcing these from chapters

16. Publication Timeline • NAM Meeting 11/2017 • Publication Workgroup Kick-Off 02/2018 • Content Scope Established 05/2018 • Chapter Outlines Completed 07/2018 • Chapter Draft Versions 09-12/2018 • NAM Meeting 01/2019 • Publication Revisions 01-02/2019 • NAM/External Reviews 03/2019 • Tentative Release 04/2019

17. Mental Framework for This Meeting • Out of Scope: Discussion of Major Content Additions/Subtractions • In Scope: Changes to Framing / Addressing Imbalance / Voice of Chapters • In Scope: Focus on Recommendations • Identify and discuss modifications, additions, and subtractions as each chapter is discussed • Be mindful of a desired balance between stakeholder groups (patients, providers, administrators, regulatory bodies, etc.) • If you felt like the opportunity to discuss a point passed and major themes, please send it to us in an email, or write it down and give it to us during a break • mahmed@nas.edu, dwhicher@nas.edu

18. Thank You • NAM Leadership • Victor Zhau • Michael McGinnis • DLC Leadership • Jonathan Perlin • Reed Tuckson • NAM Staff Leads • Danielle Whicher • Mahnoor Ahmed • DLC Clinical AI Workgroup Members