1 / 25

ABML Knowledge Refinement Loop A Case Study

ABML Knowledge Refinement Loop A Case Study. Matej Guid, Martin Možina Vida Groznik, Aleksander Sadikov, Ivan Bratko. Dejan Georgiev Zvezdan Pirtošek. Artificial Intelligence Laboratory Faculty of Computer and Information Science University of Ljubljana, Slovenia.

oro
Download Presentation

ABML Knowledge Refinement Loop A Case Study

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ABML Knowledge Refinement Loop A Case Study Matej Guid, Martin Možina Vida Groznik, Aleksander Sadikov, Ivan Bratko Dejan Georgiev Zvezdan Pirtošek Artificial Intelligence Laboratory Faculty of Computer and Information Science University of Ljubljana, Slovenia Department of Neurology, University Medical Centre Ljubljana, Slovenia • The 20th International Symposium on Methodologies for Intelligent Systems (ISMIS)4-7 December 2012, Macau

  2. About • Knowledge acquisition from data, using Machine Learning • Goal of our approach: acquired knowledge makes sense to a human expert That is: formulated in “human-like” manner, and therefore easy to understand by a human • ML technique: Argument Based Machine Learning, ABML (Možina, Žabkar, Bratko, 2007) • Case study in neurology: diagnosis of tremors

  3. Typical Problem with ML Learning program Inducedknowledge (rules, ...) Examples (patients) May be fine for classification (diagnosis) But: hard to understand by expert, and may not provide sensible explanation

  4. Illustration Learning to diagnose pneumonia A typical rule learner would induce the rule: IF Gender = Male THEN Pneumonia = Yes This will correctly diagnose all patients But it will explain Patient 3 by: Has pneumonia because he is male

  5. How to Fix this with ABML? A possible expert’s explanation for Patient 2: Patient 2 suffers from pneumonia because he has high temperature In ABML we say: Expert annotated this case by positive argument The previous rule is now not consistent with the argument

  6. ABMLtakes expert’s argument into account ABMLinduces the rule consistent with expert’s argument: IF Temperature > Normal THEN Pneumonia = Yes This explains Patient 3 by: Has pneumonia because he has very high temperature

  7. Point of this Illustration • By annotating one chosen example, the expert leads the system to induce a rule compatible with expert’s knowledge • The rule also covers other learning examples • The rule enables sensible explanation of diagnoses

  8. Rest of Talk (by Matej Guid) • How to use ABML in interaction loop between expert and system? • Case study in neurology • Experimental evaluation of this approach

  9. Knowledge Elicitation with ABML critical examples counter examples IF ... THEN ... IF ... THEN ... ... ABML argument-based machine learning arguments experts’ arguments constrain learning obtained models are consistent with expert knowledge experts introduce new concepts (attributes) human-understandable models Možina M. et al. Fighting Knowledge Acquisition Bottleneck with Argument Based Machine Learning. ECAI 2008.

  10. Differentiating Tremors: Domain Description PT MT ET Parkinsonian tremor mixed tremor essential tremor 41 patients resting tremor bradykinesia rigidity Parkinsonian spiral drawings ... 23 patients 50 patients postural tremor kinetic tremor harmonics essential spiral drawings ... • Data set of 114 patients: • learning set: 47 examples • test set: 67 examples • The patients were described by 45 attributes.

  11. ABML Knowledge RefinementLoop Step 1: Learn a hypothesis with ABML Step 2: Find the “most critical” example (if none found, stop) Step 3: Expert explains the example Return to step 1 learn data set Argument ABML critical example

  12. ABML Knowledge Refinement Loop Step 1:Learn a hypothesis with ABML Step 2:Find the “most critical” example (if none found, stop) Step 3:Expert explains the example Return to step 1 Step 3a: Explaining a critical example (in a natural language) Step 3b:Adding arguments to the example Step 3c: Discovering counter examples Step 3d:Improving arguments with counter examples Return to step 3c if counter example found

  13. First Critical Example E.65 MT the initial model could not explain why this example is from class MT Which features are in favor of ETand which features are in favor of PT?

  14. First Critical Example E.65 MT the initial model could not explain why this example is from class MT Which features are in favor of ETand which features are in favor of PT? Bradykinesia = true the expert introduced new attributes Harmonics = true

  15. Counter Examples E.65 MT the critical example Counter examples: ET: Harmonics = true PT: Bradykinesia = true E.67 (PT) E.12 (ET) arguments attached to E.65 What is the most important feature in favor of ET in E.65 that does not apply to E.67? answer: error in data – Harmonics in E.67 was now changed into false Whatis the most important feature in favor of PT in E.65 that does not apply toE.12? answer: misdiagnosis – some arguments in favor of PT in E.12 were overlookedE.12 changed from ET to MT

  16. Another Critical Example E.61 MT the model could not explain why this example is from class MT Which features are in favor of ET? Postural = true Resting = false

  17. Counter Examples E.61 MT the critical example Counter example: ET: Postural = true Resting = false E.32 (PT) argument attached to E.61 What is the most important feature in favor of ET in E.61 that does not apply to E.32? answer: theabsenceofBradykinesia in E.61 The argument in favor of ET in E.61 was thus extended to: Postural = true AND Resting = false AND Bradykinesia = false

  18. ABML Knowledge Refinement Loop Step 1:Learn a hypothesis with ABML Step 2:Find the “most critical” example (if none found, stop) Step 3:Expert explains the example Return to step 1

  19. ABML Knowledge Refinement Loop Step 1:Learn a hypothesis with ABML Step 2:Find the “most critical” example (if none found, stop) Step 3:Expert explains the example Return to step 1

  20. ABML Knowledge Refinement Loop Step 3a: Explaining a critical example (in a natural language) “Presenceofposturaltremorandrestingtremorspeak in favorof ET...” Step 3b:Adding arguments to the example Postural.tremor.up.left = 0Postural.tremor.up.right = 3Resting.tremor.up.left = 0Resting.tremor.up.right = 0 Postural = true Resting = false Step 3c: Discovering counter examples ET: Postural = true Resting = false E.32 (PT) Step 3d:Improving arguments with counter examples Postural = true AND Resting = false AND Bradykinesia = false

  21. The Final Model pure distributions • During the process of knowledge elicitation: • 15 arguments were given by the expert • 14 new attributes were included into the domain • 21 attributes were excluded from the domain

  22. Consistency With the Expert’s Knowledge

  23. The Final Model no counter-intuitive rules pure distributions classification accuracieson the test set: The accuracies of all methods improved by adding new attributes.

  24. ABML Refinement Loop & Knowledge Elicitation critical examples counter examples IF ... THEN ... IF ... THEN ... ... ABML argument-based machine learning arguments easier for experts to articulate knowledge explain single example expert provides only relevant knowledge “critical” examples detect deficiencies in explanations “counter” examples

  25. Questions and Answers ? http://www.ailab.si/matej/ dr. Matej Guid. ArtificialIntelligenceLaboratory,FacultyofComputerandInformationScience, Universityof Ljubljana. Webpage: http://www.ailab.si/matej

More Related