Artificial Neural Networks (ANN)

1. CES 514 Lec 11 April 28,2010Neural Network, case study of naïve Bayes and decision tree, text classification

2. Artificial Neural Networks (ANN) Output Y is 1 if at least two of the three inputs are equal to 1.

3. Neural Network with one neuron Rosenblatt 1958(perceptron) (also known as threshold logic unit)

4. Artificial Neural Networks (ANN) • Model is an assembly of inter-connected nodes and weighted links • Output node sums up each of its input value according to the weights of its links • Compare output node against some threshold t Perceptron Model or

5. Training a single neuron Rosenblatt’s algorithm:

6. Linearly separable instances Rosenblatt’s algorithm converges and finds a separating plane when the data set is linearly separable. Simplest example of instance that is not linearly separable: exclusive-OR (parity function)

7. Classifying parity with more neurons A neural network with sufficient number of neurons can classify any data set correctly.

8. General Structure of ANN Training ANN means learning the weights of the neurons

9. Algorithm for learning ANN • Initialize the weights (w0, w1, …, wk) • Adjust the weights in such a way that the output of ANN is consistent with class labels of training examples • Objective function: • Find the weights wi’s that minimize the above objective function • e.g., backpropagation algorithm • details: Nillson’s ML (Chapter 4) PDF

10. WEKA

11. WEKA implementation • WEKA has implementation of all the major data mining algorithms including: • decision trees (CART, C4.5 etc.) • naïve Bayes algorithm and all variants • nearest neighbor classifier • linear classifier • Support Vector Machine • clustering algorithms • boosting algorithms etc.

12. Weka tutorials http://sentimentmining.net/weka/ Contains videos showing how to use weka for various data mining applications.

13. A case study in classification • CES 514 course project from 2007 (Olson) • Consider a board game (e.g checkers, backgammon). Given a position, we want to determine how strong the position of one player (say black) is. • Can we train a classifier to learn this from training set? • As usual, problems are: • choice of attributes • creating labeled samples

14. Peg Solitaire – one player version of checkers • To win, player should remove all except one peg. • A position from which a win can achieved is called a solvable position.

15. Square board and a solvable position Winning move sequence: (3, 4, 5), (5, 13, 21), (25, 26, 27), (27, 28, 29), (21, 29, 37), (37, 45, 53), (83, 62, 61), (61, 53, 45)

16. How to choose attributes? • Number of pegs (pegs). • Number of first moves for any peg on the board (first_moves). • Number of rows having 4 pegs separated by single vacant positions (ideal_row). • Number of columns having 4 pegs separated by single vacant positions (ideal col). • Number of the first two moves for any peg on the board (first_two). • Percentage of the total number of pegs in quadrant one (quad_one). • Percentage of the total number of pegs in quadrant two (quad_two).

17. List of attributes • Percentage of the total number of pegs in quadrant three (quad_three). • Percentage of the total number of pegs in quadrant four (quad_four). • Number of pegs isolated by one vacant position (island_one). • Number of pegs isolated by two vacant positions (island_two). • Number of rows having 3 pegs separated by single vacant positions (ideal_row_three). • Number of columns having 3 pegs separated by single vacant positions (ideal_col_three).

24. Summary of performance

25. Text Classification Text classification has many applications Spam email detection Automated tagging of streams of news articles, e.g., Google News Online advertising: what is this Web page about? Data Representation “Bag of words” most commonly used: either counts or binary Can also use “phrases” (e.g., bigrams) for commonly occurring combinations of words Classification Methods Naïve Bayes widely used (e.g., for spam email) Fast and reasonably accurate Support vector machines (SVMs) Typically the most accurate method in research studies But more complex computationally Logistic Regression (regularized) Not as widely used, but can be competitive with SVMs (e.g., Zhang and Oles, 2002)

26. Ch. 13 Assigning labels to documents or web-pages Labels are most often topics such as Yahoo-categories "finance“,"sports,"news>world>asia>business" Labels may be genres "editorials" "movie-reviews" "news” Labels may be opinion on a person/product “like”, “hate”, “neutral” Labels may be domain-specific "interesting-to-me" : "not-interesting-to-me” “contains adult language” : “doesn’t” language identification: English, French, Chinese, … Types of Labels/Categories/Classes

27. Common Data Sets used for Evaluation Reuters 10700 labeled documents 10% documents with multiple class labels Yahoo! Science Hierarchy 95 disjoint classes with 13,598 pages 20 Newsgroups data 18800 labeled USENET postings 20 leaf classes, 5 root level classes WebKB 8300 documents in 7 categories such as “faculty”, “course”, “student”.

28. Practical Issues Tokenization Convert document to word counts = “bag of words” word token = “any nonempty sequence of characters” for HTML (etc) need to remove formatting Canonical forms, Stopwords, Stemming Remove capitalization Stopwords: remove very frequent words (a, the, and…) – can use standard list Can also remove very rare words, e.g., words that only occur in k or fewer documents, e.g., k = 5 Data representation e.g., sparse 3 column for bag of words: <docid termid count> can use inverted indices, etc

29. challenges of text classification • M.L classification techniques used for structured data • Text: lots of features and lot of noise • No fixed number of columns • No categorical attribute values • Data scarcity • Larger number of class label • Hierarchical relationships between classes less systematic unlike structured data

30. Techniques • Nearest Neighbor Classifier • Lazy learner: remember all training instances • Decision on test document: distribution of labels on the training documents most similar to it • Assigns large weights to rare terms • Feature selection • removes terms in the training documents which are statistically uncorrelated with the class labels • Bayesian classifier • Fit a generative term distribution Pr(d|c) to each class c of documents . • Testing: The distribution most likely to have generated a test document is used to label it.

31. Sec.13.2.1 multiply Stochastic Language Models • Model probability of generating strings (each word in turn) in a language (commonly all strings over alphabet ∑). E.g., a unigram model Model M 0.2 the 0.1 a 0.01 man 0.01 woman 0.03 said 0.02 likes … the man likes the woman 0.2 0.01 0.02 0.2 0.01 P(s | M) = 0.00000008

32. Sec.13.2.1 the class pleaseth yon maiden 0.2 0.01 0.0001 0.0001 0.0005 0.2 0.0001 0.02 0.1 0.01 Stochastic Language Models • Model probability of generating any string Model M1 Model M2 0.2 the 0.0001 class 0.03 sayst 0.02 pleaseth 0.1 yon 0.01 maiden 0.0001 woman 0.2 the 0.01 class 0.0001 sayst 0.0001 pleaseth 0.0001 yon 0.0005 maiden 0.01 woman p(s|M2) > p(s|M1)

33. Sec.13.2 Using Multinomial Naive Bayes Classifiers to Classify Text: Basic method • Attributes are text positions, values are words. • too many possibilities • Assume that classification is independent of the positions of the words • Use same parameters for each position • Result is bag of words model (over tokens)

34. Sec.13.2 Naive Bayes: Learning • From training corpus, extract vocabulary • Calculate required P(cj)and P(xk | cj)terms • For each cj in Cdo • docsjsubset of documents for which the target class is cj • Textj single document containing all docsj • for each word xkin Vocabulary • nk number of occurrences of xkin Textj

35. Sec.13.2 Naive Bayes: Classifying • positions all word positions in current document which contain tokens found in Vocabulary • Return cNB, where

37. Sec.13.2 Naive Bayes: Time Complexity • Training Time: O(|D|Lave + |C||V|)) where Lave is the average length of a document in D. • Assumes all counts arepre-computed in O(|D|Lave) time during one pass through all of the data. • Generally just O(|D|Lave) since usually |C||V| < |D|Lave • Test Time: O(|C| Lt) where Lt is the average length of a test document. • Very efficient overall, linearly proportional to the time needed to just read in all the data.

38. Sec.13.2 Underflow Prevention: using logs • Multiplying lots of probabilities, which are between 0 and 1 by definition, can result in floating-point underflow. • Since log(xy) = log(x) + log(y), it is better to perform all computations by summing logs of probabilities rather than multiplying probabilities. • Class with highest final un-normalized log probability score is still the most probable. • Note that model is now just max of sum of weights…

39. Naive Bayes Classifier • Simple interpretation: Each conditional parameter log P(xi|cj) is a weight that indicates how good an indicator xi is for cj. • The prior log P(cj) is a weight that indicates the relative frequency of cj. • The sum is then a measure of how much evidence there is for the document being in the class. • We select the class with the most evidence for it

40. Two Naive Bayes Models • Model 1: Multivariate Bernoulli • One feature Xw for each word in dictionary • Xw = true in document d if w appears in d • Naive Bayes assumption: • Given the document’s topic, appearance of one word in the document tells us nothing about chances that another word appears • This is the model used in the binary independence model in classic probabilistic relevance feedback on hand-classified data.

41. Two Models • Model 2: Multinomial = Class conditional unigram • One feature Xi for each word pos in document • feature’s values are all words in dictionary • Value of Xi is the word in position i • Naïve Bayes assumption: • Given the document’s topic, word in one position in the document tells us nothing about words in other positions • Second assumption: • Word appearance does not depend on position • Just have one multinomial feature predicting all words for all positions i,j, word w, and class c

42. Parameter estimation • Multivariate Bernoulli model: • Multinomial model: • Can create a mega-document for topic j by concatenating all documents in this topic • Use frequency of w in mega-document fraction of documents of topic cj in which word w appears fraction of times in which word w appears among all words in documents of topic cj

43. Classification • Multinomial vs Multivariate Bernoulli? • Multinomial model is almost always more effective in text applications

48. Sec.13.5 Feature Selection: Why? • Text collections have a large number of features • 10,000 – 1,000,000 unique words … and more • May make using a particular classifier feasible • Some classifiers can’t deal with 100,000 of features • Reduces training time • Training time for some methods is quadratic or worse in the number of features • Can improve generalization (performance) • Eliminates noise features • Avoids overfitting

49. Sec.13.5 Feature selection: how? • Two ideas: • Hypothesis testing statistics: • Are we confident that the value of one categorical variable is associated with the value of another? • Chi-square test (2) • Information theory: • How much information does the value of one categorical variable give you about the value of another? • Mutual information • They’re similar, but 2 measures confidence in association, (based on available statistics), while MI measures extent of association (assuming perfect knowledge of probabilities)

50. Sec.13.5.2 Term = jaguar Term  jaguar Class = auto 2 (0.25) 500 (502) Class auto 3 (4.75) 9500 (9498) 2 statistic (CHI) • 2 is interested in (fo – fe)2/fe summed over all table entries: is the observed number what you’d expect given the marginals? • The null hypothesis is rejected with confidence .999, • since 12.9 > 10.83 (the value for .999 confidence). expected: fe observed: fo