Dr. Rina Dutta & Dr. Sumithra Velupillai King’s College London & KTH, Stockholm

1. Detection of Suicidality in Adolescents with ASD: Developing an NLP Approach for Use in EHRs LeveragingInformatics to ImproveSurveillance of Disease and Events in Health Systems - S89 Dr. RinaDutta & Dr. Sumithra Velupillai King’s College London & KTH, Stockholm Twitter: #AMIA2017

2. Disclosure • We do not have any relevant relationships with commercial interests to disclose. AMIA 2017 | amia.org

3. Learning Objectives • After participating in this session the learner should be better able to: • Design and evaluate a natural language processing approach to identify suicidality-related information in mental health records for adolescents with autism spectrum disorders AMIA 2017 | amia.org

4. The Challenges of Suicide risk recognition NCI: >80% ‘low risk’

5. Motivation • Over 1 in 6 young people with Autism Spectrum Disorders will contemplate or attempt suicide during childhood • Electronic Health Records (EHRs) can be used to develop and test new models of suicide risk behaviour • Information about suicidality in EHRs is predominantly written as free-text • Absence or negation of suicidal behaviour is also documented AMIA 2017 | amia.org

6. Challenges inEHR suicide research • CRIS Events - 18.6 million documents • Documents containing the word suicid* - 783,000 • medical notes different from medical • texts and scholarly publications – • Some clinicians terse; others verbose • Incomplete sentences used • Below par grammatically • Location of statements within documents is information-bearing

7. Patient EHR example Patient EHR trajectory • … • ASD follow-up… • … • Reports anxiety in school… • … • Difficult to ascertainwhether suicidal or not. May havefleeting suicidal thoughts… • … • On presentation: patient denies suicidal plans… • … • family history of suicide… • … • has taken an overdose • in the past… • … worriedaboutchild, locked in room, might be suicidal… • time AMIA 2017 | amia.org

8. Workflow AMIA 2017 | amia.org

9. Suicidal behaviour classification • Document 2 • Document 1 • … reportedfleetingsuicidal thoughts… • S = P • … presenting for serious OD withoutsuicidal… • S = N • …strongly deniedacting on suicidal thoughts… • S = N • …risk of suicidal behaviour… • S = P • S = N • Patient S = P • S = P AMIA 2017 | amia.org

10. Results: prevalence and IAA • Suicide-related information prevalence (screening) • <3% on document level • 22% on patient level • Annotator agreement – negated and positive suicide-related information • 100 documents double-annotated • IAA (Cohen’s κ): 0.83 AMIA 2017 | amia.org

11. Results: NLP vs manual annotations AMIA 2017 | amia.org

12. Results: error analysis • Missing negation terms • Nil suicidal • Heuristics and negation scope • Co-reference • Clinically challenging/complex cases • ‘I tried to assessXXX’s suicidal risk – XXX does not knowif XXX wants to killXXXself ’, ‘XXX does not haveanyspecific plan’ AMIA 2017 | amia.org

13. Conclusions • First comprehensive study on detecting suicide risk in ASD adolescents with NLP • Our NLP tool identifies suicide-related information with high precision and recall • Challenging annotation task – document vs patient level • Other contextual attributes important to capture • time information (e.g. past) • subject (e.g. patient, clinician, family member) AMIA 2017 | amia.org

14. Questions • Question 1 Why is longitudinal EHR data likely to be a more reliable resource than self-reported information? • Self-reported information is difficult to analyse • Self-reported information is always inaccurate • Self-reported information is subject to reporting biases • Self-reported information is subject to redundancy AMIA 2017 | amia.org

15. Answer • Self-reported information is difficult to analyse • Self-reported information is always inaccurate • Self-reported information is subject to reporting biases • Self-reported information is subject to redundancy • Explanation:Whilst self-reported information may provide invaluable clues to individuals’ suicidal behaviour, this type of information is subject to reporting biases, in particular in relation to suicide, because many people are motivated to answer in a socially desirable manner, denying suicidal thoughts which they think might be stigmatising. Because EHR data contains routinely collected data from healthcare settings, documented by healthcare professionals, there is less risk of reporting bias, and also a higher likelihood of identifying systematic patterns and trends related to the complexities of underlying risk factors related to suicidal behaviour. AMIA 2017 | amia.org

16. Questions • Question 2 Which of the following options would be an appropriate method to apply to ensure accurate identification of patients at risk of suicidal behaviour? • Statisticalanalysis of diagnosticcodesrelated to suicidal behaviour • Develop a searchengine index of the EHR data and definesearch terms related to suicidal behaviour • Develop a questionnaire to be used at the point of care to screen for suicidal behaviour • Develop a naturallanguageprocessing solution with negation detection that identifies terms related to suicidal behaviour AMIA 2017 | amia.org

17. Answer • Statisticalanalysis of diagnosticcodesrelated to suicidal behaviour • Develop a searchengine index of the EHR data and definesearch terms related to suicidal behaviour • Develop a questionnaire to be used at the point of care to screen for suicidal behaviour • Develop a naturallanguageprocessing solution with negation detection that identifies terms related to suicidal behaviour • Explanation:An appropriate questionnaire might, at least in the long run, capture important information related to suicidal behaviour, but would be very time-consuming, and would not enable retrospective analysis of information already documented in the EHRs. Because most of the information in EHRs is written in free-text, only studying diagnostic codes will lead to very low sensitivity (recall) and lead to a risk of inconclusive results. A method that includes analysis of the free text is therefore essential. Defining appropriate search terms in e.g. a search engine index will, on the other hand, lead to poor positive predictive value/PPV (precision), because absence of suicidal behaviour is often documented in routine care. To ensure a reasonable balance between high sensitivity and high PPV, a natural language processing tool that identifies terms related to suicidal behaviour and deals with detecting negation will provide most accurate identification of patients at risk of suicidal behaviour. AMIA 2017 | amia.org

18. Questions • Question 3 Capturing the prevalence and incidence of suicidal behaviour in young people using NLP approaches from routinely collected health care records has several limitations, which from the following list is not a significant limitation of this approach? • Suicidal behaviourwhichdoesn’t present to mental health or emergency services goesundetected. • Presentations to emergency services with self-harm or suicidal behaviourhave a more benign prognosisthannon-disclosedbehaviours, and are of limitedclinicalrelevance.  • Detectionis non-systematic, and relies on a clinician’sasking and recording the relevant information.  • Wherehealthcare data covers hospital activity, ratherthan a catchment population, incidence and prevalence rates may be inaccurate. AMIA 2017 | amia.org

19. Answer • Suicidal behaviourwhichdoesn’t present to mental health or emergency services goesundetected. • Presentations to emergency services with self-harm or suicidal behaviourhave a more benign prognosisthannon-disclosedbehaviours, and are of limitedclinicalrelevance.  • Detection is non-systematic, and relies on a clinician’sasking and recording the relevant information.  • Wherehealthcare data covers hospital activity, ratherthan a catchment population, incidence and prevalence rates may be inaccurate. • Explanation Cohort studies report between 6-15% of adolescents will describe at least one episode of suicidal behaviour. The majority of adolescents will not seek treatment. Recent longitudinal studies describe the most suicidal and self-harming behaviours seen in adolescence resolves spontaneously in early adulthood. However, presenting with suicidal behaviours to hospital services tends to represent more severe behaviours, and is one of the strongest predictors for later suicide attempts. Most mental health information recorded in the health care record is elicited from a patient during an interview and later summarised. Information is unlikely to be systematically ascertained by semi-structured instruments, using similarly trained raters in a consistent environment. Therefore health care record data will be subject to information bias’, which may underestimate or overestimate (less likely in the latter case) suicidal behaviours. When an acute or mental health hospital accepts a large number of referrals from outside a local catchment area, or where one region is served by a number of different services which don’t have a universal health care record, a health care record from a single source can produce very inaccurate estimates of the true rates within a population. AMIA 2017 | amia.org

20. Thank you! rina.dutta@kcl.ac.uk sumithra.velupillai@kcl.ac.uk