CS 9633 Machine Learning Inductive-Analytical Methods

1. CS 9633 Machine LearningInductive-Analytical Methods

2. Inductive and Analytical Methods • Inductive methods seek general hypotheses that fit observed training data. • Can fail with insufficient data • May be misled by incorrect bias • Analytical methods seek general hypotheses that fit observed training data and prior knowledge. • Can generalize more accurately from less data • Can be misled with insufficient prior data • Combining approaches offers possibility of powerful learning methods.

3. Justifications • Hypotheses output by analytical methods have logical justifications • Hypotheses output by inductive methods have statistical justifications

4. Spectrum of Learning Tasks Inductive Learning Plentiful data No prior knowledge Analytical Learning Perfect prior knowledge Scarce data

5. Desirable Characteristics • Given no domain theory, learn at least as effectively as purely inductive methods. • Given a perfect domain theory, learn at least as effectively as purely analytical methods. • Given imperfect data and imperfect domain theory, combine two operations to outperform either pure method • Accommodate an unknown level of error in the training data • Accommodate an unknown level of error in the domain theory.

6. Learning Problem • Given • A set of training examples, D, possibly containing errors • A domain theory, B, possible containing errors • A space of candidate hypothese H • Determine • A hypothesis that best fits the training examples and domain theory

7. Fitting the Domain Theory and Training Examples • errorD(h) is the proportion of examples from D that are misclassified by H • errorb(h) is the probability that h will disagree with B on the classification of a randomly drawn instance • How do we combine measure of errors?

8. Using Prior Knowledge • Use prior knowledge to derive an initial hypothesis from which to begin the search. • Use prior knowledge to alter the objective of the hypothesis search space. • Use prior knowledge to alter the available search steps.

9. Knowledge Based Artificial Neural Network (KBANN) • General approach • Initialize the hypothesis to perfectly fit the domain theory • Inductively refine the hypothesis as necessary • KBANN • Initialize neural network to predict domain theory • Refine to fit data

10. See Table 12.3 for Example • Includes both training examples and domain theory • Domain theory and training examples are not completely consistent • Examples 2 and 3 are not predicted as positive by the domain theory.

11. Constructing Initial NN • Create a sigmoid unit for each Horn clause in the domain theory. • Sigmoid output > 0.5 is interpreted as true • Sigmoid output < 0.5 is interpreted as false • Input created for each antecedent. • Weights set to compute logical AND of the inputs. • For each input corresponding to a non-negated antecedent, set weight to positive constant W • For each input corresponding to a negated antecedent, set weight to negative W. • Each unit constructed so output will be greater than 0.5 just for cases for its Horn clauses. • Threshold weight of the unit w0 is set to –(n-0.5)W • With 0 and 1 inputs, correct output is guaranteed. • Additional input units added to each threshold unit and weights set approximately to 0.

12. Tuning the neural network • After the initial neural network is constructed, refine the network using inductive learning. • The tuning process learns new dependencies.

13. Summary of KBANN • KBANN generally generalizes more accurately than pure backprop, especially with scarce data. • Methods have been developed for mapping the refined network back to Horn clauses.