Unintended Consequences of AI in Healthcare: Risks and Recommendations

1. Potential Unintended Consequences of AI Jonathan H. ChenAndrew Beam Suchi SariaEneida Mendonca NAM Digital Learning CollaborativeArtificial Intelligence and Machine Learning in Health Care Working Group

2. Hype Harms Humanity

3. Gartner Hype Cycle, 2017 Deep Learning Machine Learning Blockchain Expectations Time

4. Superhuman Human Fallacy No need for over-hype (or pessimism) Plenty of Real Opportunities Undesirable Variability Inefficient Admin Scarce Resource Use

5. Harms How could improper AI hurt patients and the health system?

6. Though there is much upside in the potential for the use of AI in medicine, like all technologies, implementation does not come without certain risks. In this section we outline some ways in which AI in healthcare may cause harm in unintended ways.

7. Poorly constructed models from observational data can harm patients

8. The implementation of electronic health records and other health information systems have provided scientists with rich longitudinal, multi-dimensional, and detailed records about individual’s health data. Healthcare data scientists must be careful to apply the right types of modeling approaches when informing decision-making.

9. Recommendation: Seek out robust evaluations of model performance, utility, vulnerabilities, and bias.

10. Be skeptical of marketing hype; seek out carefully conducted evaluations measuring model performance and practical utility. Models may be prone to unreliable behavior due to shifts in population, practice-patterns, or other characteristics that do not generalize as we move from the development to the deployment environment. It is important to carefully probe models for such vulnerabilities. Computers/algorithms are effective at learning statistical structure, patterns, organization, and rules in complex data sources, but they do not offer meaning, purpose, or a sense of justice/fairness. Algorithms trained on biased datasets can amplify bias. Deliberate effort to identify and correct biases can allow the same tools to mitigate biases in decision-making. Tools that enable test-time auditing for determining predictions that are unreliable can reduce the risk of incorrect decision-making. Machine learning approaches that measure robustness to shifts in data that are likely to occur as one moves from evaluation on a fixed dataset to a dynamic deployment environment are needed. Ensure data that are used for developing and improving AIs are collected with clear consent and do not accidentally foster the age of a “Digital HeLa” where patient data is used in ways they do not accept.

11. Cybersecurity Vulnerabilities due to AI Automation

12. History has repeatedly demonstrated that new technologies often have unintended consequences when bad-actors use the technology in unanticipated ways.

13. Recommendation: Develop AI systems with adversaries (bad-actors) in mind

14. There are roughly two broad classes of possible defenses: infrastructural and algorithm. Both of types of defenses highlight the need for interdisciplinary teams of computer scientists and healthcare workers at every stage of design and implementation of these systems.

15. Humanity How could AI Reshape Jobs and Roles in Health?

16. Future of Employment? https://www.visualcapitalist.com/visualizing-jobs-lost-automation/

17. Any clinician who can be replaced by a computer, should be. Automation/Industrialization: Net Gains, Unequal Pains

18. Recommendation: Education Reform Emphasize Strengths Humans • Physical Interaction • Communication • Empathy • Management • Interpretation • Reasoning Computers • Memory • Calculation • Information Retrieval • Prediction • Repetition Consistency • Scalable

19. Recommendation: Identify Synergy rather than Replacement Humans + Computers Outperform Either >25M Deficient Access to Medical Specialty Care Growing Shortage of Scarcest Healthcare Resource (Human Experts)

20. Recommendation: Use AI to Engage (rather than Stifle) Human Qualities • Increase Efficiencies to Recover Time for Human Activities (e.g., automate mundane tasks to ease relationship building) • Avoid Workflow Disruption, Data Entry, etc. (Chapter 6) • Reach Spaces that Humans are Not (e.g., home monitoring, telemedicine in underserved areas)

21. Recommendations Beware Hype, Plenty of Opportunities Robust Evaluations of Performance, Utility, Vulnerabilities, and Bias Adversaries (Bad-Actors) in Mind Education Reform - Emphasize Strengths Synergy Rather than Replacement Engage (don’t Stifle) Human Qualities