Knowledge Engineering for Bayesian Networks

1. Knowledge Engineering for Bayesian Networks Ann Nicholson School of Computer Science and Software Engineering Monash University

2. Overview • Summary of my BN-related projects • Thoughts on the BN Knowledge Engineering Process • Case Study: Intelligent Tutoring System for decimal misconceptions

3. BN-related projects • DBNS for discrete monitoring • Approximate BN inference algorithms based on a mutual information measure for relevance (with Nathalie Jitnah, ICONIP97, ECSQARU97, PRICAI98,AI99) • Plan recognition: DBNs for predicting users actions and goals in an adventure game (with David Albrecht,Ingrid Zukerman,UM97,UMUAI1999,PRICAI2000) • DBNs for ambulation monitoring and fall diagnosis (with biomedical engineering,PRICAI’96) • Autonomous aircraft monitoring and replanning (withTim Wilkin,PRICAI2000)

4. BN-related projects • Bayesian Poker (with Kevin Korb,UAI’99) • Seabreeze prediction: joint project with Bureau of Meteorology (with Russell Kennett and Kevin Korb,PAKDD’2001) • Comparison of existing simple rule, expert elicited BN, and BNs from 2 automated learners -- Tetrad-II (Spirtes et al. 1993) and CaMML (Wallace and Korb, 1999). • Intelligent tutoring system for decimal misconceptions (UAI2001)

5. Other related research at Monash • Machine learning • Minimum-Message length (Wallace, Dowe) • BN Learning • CaMML (Casual MML) (Wallace, Korb) • Gas for search (Neil, Korb UAI’99) • BNs for Argument Generation (Zukerman,Korb)

6. Elicitation from experts • Variables • important variables? values/states? • Structure • causal relationships? • dependencies/independencies? • Parameters (probabilities) • quantify relationships and interactions? • Preferences (utilities) (for decision networks)

7. BN EXPERT Domain EXPERT BN TOOLS Expert Elicitation Process • These stages are done iteratively • Stops when further expert input is no longer cost effective • Process is difficult and time consuming. • Current BN tools • inference engine • GUI • Next generation of BN tools?

8. Knowledge discovery • There is much interest in automated methods for learning BNs from data • parameters, structure (causal discovery) • Computationally complex problem, so current methods have practical limitations • e.g. limit number of states, require variable ordering constraints, do not specify all arc directions, don’t handle hidden variables • Evaluation methods

9. The knowledge engineering process 1. Building the BN • variables, structure, parameters, preferences • combination of expert elicitation and knowledge discovery 2. Validation/Evaluation • case-based, sensitivity analysis, accuracy testing 3. Field Testing • alpha/beta testing, acceptance testing 4. Industrial Use • collection of statistics 5. Refinement • Updating procedures, regression testing

10. Case Study: Intelligent tutoring • Tutoring domain: primary and secondary school students’ misconceptions about decimals • Based on Decimal Comparison Test (DCT) • student asked to choose the larger of pairs of decimals • different types of pairs reveal different misconceptions • ITS System involves computer games involving decimals • This research also looks at a combination of expert elicitation and automated methods

11. Expert classification of Decimal Comparison Test (DCT) results

12. The ITS architecture Adaptive Bayesian Network Inputs Student Generic BN model of student Decimal comparison test (optional) Item Answers Answer • Diagnose misconception • Predict outcomes • Identify most useful information Information about student e.g. age (optional) Computer Games Hidden number Answer Classroom diagnostic test results (optional) Feedback Answer Flying photographer • Select next item type • Decide to present help • Decide change to new game • Identify when expertise gained System Controller Module Item type Item Decimaliens New game Sequencing tactics Number between Help Help …. Report on student Classroom Teaching Activities Teacher

13. Expert Elicitation • Variables • two classification nodes: fine and coarse (mut. ex.) • item types: (i) H/M/L (ii) 0-N • Structure • arcs from classification to item type • item types independent given classification • Parameters • careless mistake (3 different values) • expert ignorance: - in table (uniform distribution)

14. Expert Elicited BN

15. Evaluation process • Case-based evaluation • experts checked individual cases • sometimes, if prior was low, ‘true’ classification did not have highest posterior (but usually had biggest change in ratio) • Adaptiveness evaluation • priors changes after each set of evidence • Comparison evaluation • Differences in evaluation between BN and expert rule

16. Comparison evaluation • Development of measure: same classification, desirable and undesirable re-classification • Use item type predictions (not yet undertaken) • Investigation of effect of item type granularity and probability of careless mistake

17. Investigation by Automated methods • Classification (using SNOB program, based on MML) • Parameters • Structure (using CaMML)

18. Results

19. Open Research Questions • Methodology for combining expert elicitation and automated methods • expert knowledge used to guide search • automated methods provide alternatives to be presented to experts • Evaluation measures and methods • may be domain depended • Improved BN tools • e.g. visualisation of d-separation