Induction and Decision Trees

1. Induction and Decision Trees

2. Artificial Intelligence • The design and development of computer systems that exhibit intelligent behavior. • What is intelligence? • Turing test: • Developed in 1950 by Alan Turing (pioneer in computer science) • Computer and human in one room • Human “interrogator” in another room • Interrogator asks questions...human OR computer answers • If interrogator cannot tell whether the human or the computer is answering, then the computer is “intelligent”

3. Classification of AI Systems • Knowledge Representation Systems • Capture existing expert knowledge and use it to consult end-users and provide decision support • Main types: Rule-based expert systems, Case-base reasoning systems, Frame-based knowledge systems, Semantic networks • Machine Learning • Algorithms that use mathematical or logical techniques for finding patterns in data and discovering or creating new knowledge • Main types: Artificial neural networks, genetic algorithms, inductive decision trees, Naïve Bayesian algorithms, Clustering and pattern-recognition algorithms Data mining involves primarily a “machine learning” form of AI

4. Data Mining • Textbook definition: • Knowledge discovery in databases • Using statistical, mathematical, AI, and machine learning techniques to extract useful information and subsequent knowledge from large databases • Key point: identifying patterns in large data sets

5. Microsoft SQL Server Data Mining Algorithms • Decision Trees • Naïve Bayesian • Clustering • Sequence Clustering • Association Rules • Neural Network • Time Series

6. Decision Trees for Machine Learning • Based on Inductive Logic • Three types of logical structures commonly used in AI systems: • Deduction • Abduction • Induction

7. Deduction • Premise (rule): if p then q • Fact (axiom, observation): p • Conclude: q • This is classical logic (Modus Ponens). If the rule is correct, and the fact is correct, then you know that the conclusion will be correct. We are given the rule

8. Abduction • Premise (rule): if p then q • Fact (axiom, observation): q • Conclude: p • This form of reasoning is a logical fallacy called “affirming the consequent” (Post hoc ergo propter hoc). The conclusion may be wrong, but it is a plausible explanation of the fact, given the rule. Useful for diagnostic tasks. We are given the rule

9. Induction • Observe p and q together • . • . • . n. Observe p and q together Conclude: if p then q This is stereotypical thinking…highly error prone. We create the rule

10. Example – Baby in the kitchen

11. ID3 Decision Tree Algorithm • “Iterative Dichotomizer” • Developed by Ross Quinlan (1979) • This is the basis for many commercial induction products • The goal of this algorithm is to find rules resulting in YES or NO values. (Therefore, the output of generated rules have 2 possible outcomes) • ID3 generates a tree, where each path of the tree represents a rule. The leaf node is the THEN part of the rule, and the nodes leading to it are the ANDS of attribute-value combinations in the IF part of the rule.

12. ID3 Algorithm • Starting Point: • an empty tree (this tree will eventually represent the final rules created by ID3) • a recordset of data elements (e.g. records from a database) • a set attributes (fields), each with some finite number of possible values • NOTE: one of the attributes is the “decision” field, with a YES or NO value (or some other 2-valued option...GOOD/BAD, HIGH/LOW, WIN/LOSE, etc.) • Output: • a tree, where each path of the tree represents a rule

13. ID3 algorithm • If all records in your recordset are positive (i.e. have YES values for their decision attribute), create a YES node and stop (end recursion) • If all records in your recordset are negative, create a NO node and stop (end recursion) • Select the attribute that best discriminates among the records (using an entropy function) • Create a tree-node representing that attribute, with n branches, where n is the number of values for the selected attribute • Divide the records of the recordset into subsets subrecordset 1, subrecordset 2, ..., subrecordset n corresponding with each value of the selected attribute • Recursively apply the algorithm to each subrecordset i, with reduced attribute set (don’t include already used attributes further down the path)

14. Calculating Entropy • Entropy = mixture, chaos • We want to pick the attribute with the lowest entropy: •  ideally, a particular value for the input attribute leads to ALL yes or ALL no in the outcome attribute…or come as close to this as possible An attribute’s entropy = Where n is the total number of possible values for the attribute and xi is the ith value

15. Baby’s RecordSet of Oven-Touching Experiences

16. ID3 Applied to Baby-in-the-Kitchen • Which attribute to start with? Based on Entropy measure (assuming log base 2), • Touch stove entropy = 0.918 • Mom in kitchen entropy = 1.0 • To see this, note that: • Probability of touching stove leading to ouch is .67, and not leading to ouch is .33 • .67 * .33 = .22 • Probability of mom being in kitchen leading to ouch is .5 and mom being in kitchen not leading to ouch is also .5 • .5 * .5 = .25

17. Applying the Touch Stove Attribute

18. Recurse …apply the Mom in Kitchen attribute where needed

19. Resulting decision rules • If Touch_Oven = No then BOO_BOO = No • If Touch_Oven = Yes and Mom_In_Kitchen = Yes then BOO_BOO = Yes • If Touch_Oven = Yes and Mom_In_Kitchen = No then BOO_BOO = No

20. Now we’ll do this with Microsoft SQL Server