Novel representations and methods in text classification

1. Novel representations and methods in text classification Manuel Montes, Hugo Jair Escalante Instituto Nacional de Astrofísica, Óptica y Electrónica, Mexico.http://ccc.inaoep.mx/~mmontesg/ http://ccc.inaoep.mx/~hugojair/ {mmontesg, hugojair}@inaoep.mx 7th Russian Summer School in Information RetrievalKazan, Russia, September 2013

2. Novel represensations and methods in textclassification Overview of thiscourse

3. Text classification • Text classification is the assignment of free-text documents to one or more predefined categories based on their content • In this course we will review the basics of text classification (document preprocessing and representation and classifier construction) with emphasis on novel or extended document representations

4. Day 1 (today): Introduction to text classification • We will provide a general introduction to the task of automatic text classification: • Text classification: manual vs automatic approach • Machine learning approach to text classification • BOW: Standard representation for documents • Main learning algorithms for text classification • Evaluation of text classification

5. Day 2: Concept-based representations • This session elaborates on concept-based representations of documents. That is document representations extending the standard representation with –latent– semantic information. • Distributional representations • Random indexing • Concise semantic analysis • Multimodal representations • Applications in short-text classification, author profiling and authorship attribution

6. Day 3: Modeling sequential and syntactic information • This session elaborates on different alternatives that extend the BOW approach by including sequential and syntactic information. • N-grams • Maximal frequent sequences • Pattern sequences • Locally weighted bag of words • Syntactic representations • Applications in authorship attribution

7. Day 4: Non-conventional classification methods • This session considers text classification techniques specially suited to work with low quality training sets. • Self-training • PU-learning • Consensus classification • Applications in short-text classification, cross-lingual text classification and opinion spam detection.

8. Day 5: Automatic construction of classification models • This session presents methods for the automated construction of classification models in the context of text classification. • Introduction to full model selection • Related works • PSMS • Applications in authorship attribution

9. Somerelevant material ccc.inaoep.mx/~mmontesg/ • Notes of thiscourse • teaching  Russir 2013 • Notes of ourcourseontextmining • teaching  textmining • Notes of ourcourseoninformationretrieval • teaching  informationretrieval

10. Novel represensations and methods in textclassification Introductiontotextclassification

11. Text classification • Text classification is the assignment of free-text documents to one or more predefined categories based on their content Categories/classes (e.g., sports, religion, economy) Documents (e.g., news articles)

12. Manual classification • Very accuratewhen job is done by experts • Different to classify news in general categories than biomedical papers into subcategories. • But difficult and expensiveto scale • Different to classify thousands than millions • Used by Yahoo!, Looksmart, about.com, ODP, Medline, etc.

13. Hand-coded rule based systems • Main approach in the 80s • Disadvantage  knowledge acquisition bottleneck • too time consuming, too difficult, inconsistency issues Experts Knowledgeengineers Labeleddocuments New document Rule 1, if … then … else Rule N, if … then … Document’s category Classifier

14. Example: filtering spam email • Rule-basedclassifier Hastie et al. The Elements of Statistical Learning, 2007, Springer.

15. Machine learning approach (1) • A general inductive process builds a classifier by learning from a set of preclassified examples. • Determines the characteristics associated with each one of the topics. Ronen Feldman and James Sanger, The Text Mining Handbook

16. Machine learning approach (2) Experts Labeleddocuments(training set) Learning algorithm Newdocument Rules, trees, probabilities, prototypes, etc. Document’scategory Classifier How to represent document’s information?

17. Representation of documents • First step: transform documents, which typically are strings of characters, into a representation suitable for the learning algorithm. • Generation of a document-attribute representation • The most common used document representation is the bag of words. • Documents are represent by the set of different words in all of the documents • Word order is not capture by this representation • There is no attempt for understanding their content

18. Representation of documents Vocabulary from the collection(set of different words) All documents (one vector per document) Weight indicating the contributionof word j in documenti. Each different word is a feature! How to compute their weights?

19. Preprocessing • Eliminate information about style, such as html or xml tags. • For some applications this information may be useful. For instance, only index some document sections. • Remove stop words • Functional words such as articles, prepositions, conjunctions are not useful (do not have an own meaning). • Perform stemmingorlemmatization • The goal is to reduce inflectional forms, and sometimes derivationally related forms. • am, are, is → be car, cars, car‘s → car Have to do them always?

20. Term weighting - two main ideas • The importance of a term increases proportionally to the number of times it appears in the document. • It helps to describe document’s content. • The general importance of a term decreases proportionally to its occurrences in the entire collection. • Common terms are not good to discriminate between different classes

21. Term weighting – main approaches • Binary weights: • wi,j = 1 iff document di contains term tj , otherwise 0. • Term frequency (tf): • wi,j = (no. of occurrences of tj in di) • tf x idf weighting scheme: • wi,j = tf(tj, di) × idf(tj), where: • tf(tj, di) indicatestheocurrences of tj in documentdi • idf(tj) = log [N/df(tj)], wheredf(tj) isthenumber of documetsthatcontainthetermtj. • Normalization?

22. Extended document representations • BOW is simple and tend to produce good results, but it has important limitations • Does not capture word order neither semantic information • New representations attempt to handle these limitations. Some examples are: • Distributional term representations • Locally weighted bag of words • Bag of concepts • Concise semantic analysis • Latent semantic indexing • Topic modeling • … Thetopic of thiscourse

23. Dimensionality reduction • A central problem in text classification is the high dimensionality of the features space. • Exist one dimension for each unique word found in the collection  can reach hundreds of thousands • Processing is extremely costly in computational terms • Most of the words (features) are irrelevant to the categorization task How to select/extract relevant features? How to evaluate the relevance of the features?

24. Two main approaches • Feature selection • Idea: removal of non-informative words according to corpus statistics • Output: subset of original features • Main techniques: document frequency, mutual information and information gain • Re-parameterization • Idea: combine lower level features (words) into higher-level orthogonal dimensions • Output: a new set of features (not words) • Main techniques: word clustering and Latent semantic indexing (LSI) Out of thescope of thiscourse

25. Learning the classification model Categories C1 C2 C3 C4 X2 X1 Howtolearnthis?

26. m1 m3 What is a classifier? • A function: • or: • Given: Terms (d = |V|) Documents (M)

27. m1 m3 What is a classifier? • A function: • or: • Given: Terms (d = |V|) Documents (M)

28. Classification algorithms • Popular classification algorithms for TC are: • K-Nearest Neighbors • Example-based approach • Centroid-based classification • Prototype-based approach • Support Vector Machines • Kernel-based approach • Naïve Bayes • Probabilistic approach

29. KNN: K-nearest neighbors classifier Positive examples Negative examples 1-nearest neighborclassifier

30. KNN: K-nearest neighbors classifier Positive examples Negative examples 3-nearest neighborsclassifier

31. KNN: K-nearest neighbors classifier Positive examples Negative examples 5-nearest neighborsclassifier

32. KNN – the algorithm • Given a new document d: • Find the k most similar documents from the training set. • Common similarity measures are the cosine similarity and the Dice coefficient. • Assign the class to d by considering the classes of its k nearest neighbors • Majority voting scheme • Weighted-sum voting scheme

33. Common similarity measures • Dice coefficient • Cosine measure wki indicates the weight of word k in document i

34. Selection of K How to select a good value for K?

35. Decision surface of KNN http://clopinet.com/CLOP K=1

36. Decision surface of KNN http://clopinet.com/CLOP K=2

37. Decision surface of KNN http://clopinet.com/CLOP K=5

38. Decision surface of KNN http://clopinet.com/CLOP K=10

39. The weighted-sum voting scheme Other alternatives for computing the weights?

40. KNN - comments • One of the best-performing text classifiers. • It is robust in the sense of not requiring the categories to be linearly separated. • The major drawback is the computational effort during classification. • Other limitation is that its performance is primarily determined by the choice of k as well as the distance metric applied.

41. Linear models • Classification of DNA micro-arrays ? x2 Cancer ? No Cancer x1

42. Support vector machines (SVM) • A binary SVM classifier can be seen as a hyperplane in the feature space separating the points that represent the positive from negative instances. • SVMs selects the hyperplanethat maximizes the marginaround it. • Hyperplanes are fullydetermined by a small subsetof the training instances, calledthe support vectors. Support vectors Maximize margin

43. Support vector machines (SVM) Subject to: When data are linearly separable we have:

44. x 0 x2 Non-linear SVMs (on the inputs) • What about classes whose training instances are not linearly separable? • The original input space can always be mapped to some higher-dimensional feature space where the training set is separable. • A kernel function is some function that corresponds to an inner product in some expanded feature space.

45. Decision surface of SVMs http://clopinet.com/CLOP Linear support vector machine

46. Decision surface of SVMs http://clopinet.com/CLOP Non-linear support vector machine

47. SVM – discussion • The support vector machine (SVM) algorithm is very fast and effective for text classification problems. • Flexibility in choosing a similarity function • By means of a kernel function • Sparseness of solution when dealing with large data sets • Only support vectors are used to specify the separating hyperplane • Ability to handle large feature spaces • Complexity does not depend on the dimensionality of the feature space

48. Sec.13.2 Naïve Bayes • It is the simplest probabilistic classifier used to classify documents • Based on the application of the Bayes theorem • Builds a generative model that approximates how data is produced • Uses prior probability of each category given no information about an item • Categorization produces a posterior probability distribution over the possible categories given a description of an item. A. M. Kibriya, E. Frank, B. Pfahringer, G. Holmes. MultinomialNaiveBayesforTextCategorizationRevisited. AustralianConferenceon Artificial Intelligence 2004: 488-499

49. Naïve Bayes • Bayes theorem: • Why? • We know that: • Then • Then

50. Sec.13.2 Naïve Bayes • For a document d and a class cj C . . . . t1 t2 t|V| • Assuming terms are independent of each other given the class (naïve assumption) • Assuming each document is equally probable