Text Classification with Belief Augmented Frames

1. Text Classification withBelief Augmented Frames Colin Tan Department of Computer Science, School of Computing, National University of Singapore.

2. Outline • What are Belief Augmented Frames? • Motivation behind Belief Augmented Frames • Representing Beliefs in BAFs • Some Definitions • Belief Augmented Frame Logic (BAF-Logic) • Applying BAF-Logic to Text Classification • Experiment Protocol and Results • Conclusions

3. What are Belief Augmented Frames? • Belief Augmented Frames (BAF) combine classical AI frames with belief measures. • Frame-based system to structure knowledge and relations between entities. • Belief measures provide uncertain reasoning on existence of entities and the relationships between them.

4. Motivation behind Belief Augmented Frames • Why Belief Measures? • Statistical Measures • Standard tool for modeling uncertainty. • Essentially, if the probability that a proposition E is true is p, then the probability of that E is false is 1-p. • P(E) = p • P(not E) = 1-p • This relationship essentially leaves no room for ignorance. Either the proposition is true with a probability of p, or it is false with a probability of 1-p. • This can be counter-intuitive at times.

5. Motivation behind Belief Augmented Frames • Why Belief Measures? • [Shortliffe75] cites a study in which, given a set of symptoms, doctors were willing to declare with certainty x that a patient was suffering from a disease D, yet were unwilling to declare with certainty 1-x that the patient was not suffering from D.

6. Motivation behind Belief Augmented Frames • Why Belief Measures? • To allow for ignorance our research focuses on belief measures. • The ability to model ignorance is inherent in belief systems. • E.g. in Dempster-Shafer Theory [Dempster67], if our belief in E1 and E2 are 0.1 and 0.3 respectively, then the ignorance is (1 – (0.1 + 0.3)) = 0.6.

7. Motivation behind Belief Augmented Frames • Why Frames? • Frames are a powerful form of representation. • Intuitively represents relationships between objects using slot-filler pairs. • Simple to perform reasoning based on relationships. • Hierarchical • Can perform generalizations to create general models derived from a set of frames.

8. Example BAF

9. Belief Representation in Belief Augmented Frames • Beliefs are represented by two masses: • φT: Belief mass supporting a proposition. • φF: Belief mass refuting a proposition. • In general φT + φF 1 • Room to model ignorance of the facts. • Separate belief masses allow us to: • Draw φTand φFfrom different sources. • Have different chains of reasoning for φT and φF.

10. Belief Representation in Belief Augmented Frames • This ability to derive the refuting masses from different sources and chains of reasoning is unique to BAF. • In Probabilistic Argumentation Systems (the closest competitor to BAF) for example, p(not E) = 1 – p(E).

11. Some Definitions • Degree of Inclination • The Degree of Inclination is defined as: • DI = T - F • DI is in the range of [-1, 1]. • One possible interpretation of DI:

12. Some Definitions • Utility Value • The Degree of Inclination DI can be re-mapped to the range [0, 1] through the Utility function: • U = (DI + 1) / 2 • By normalizing U across all relevant propositions it becomes possible to use U as a statistical measure.

13. Belief Augmented Frame Logic(BAF-Logic) • Belief Augmented Frame Logic, or BAF-Logic, is used for reasoning with BAFs. • Throughout the remainder of this presentation, we will consider two propositions A and B, with supporting and refuting masses TA, FA, TB, and FB.

14. Belief Augmented Frame Logic(BAF-Logic) • A  B: • TA B = min(TA, TB) • FA B = max(FA, FB) • A  B: • TA  B = max(TA, TB) • FA  B = min(FA, FB) •  A: • T A = F A • F A = T A

15. Belief Augmented Frame Logic(BAF-Logic) • BAF-Logic properties that are identical to Propositional Logic: • Associativity, Commutativity, Distributivity, Idempotency, Absorption, De-Morgan’s Theorem, - elimination. • Other properties of Propositional Logic work slightly differently in BAF-Logic. • In particular, some of the properties hold true only if the constituent propositions are at least “probably true” or “probably false” • I.e. |DIP |  0.5

16. Belief Augmented Frame Logic(BAF-Logic) • An Example: • Given the following propositions in your knowledge base: • KB = {(A, 0.7, 0.2), (B, 0.9, 0.1), (C, 0.2, 0.7), (A B R, TONE , FONE,), (A BR, TONE , FONE)} • We want to derive TR, FR.

17. Belief Augmented Frame Logic(BAF-Logic) • Combining our clauses regarding R, we obtain: • R = (A  B)   (A   B) • = A  B  ( A  B) • With De-Morgan’s Theorem we can derive  R: •  R= A   B  (A   B)

18. Belief Augmented Frame Logic(BAF-Logic) • TR = min(TA , TB , max(FA , TB )) = min(0.7, 0.9, max(0.2, 0.9)) = min(0.7, 0.9, 0.9) = 0.7 • FR = max(FA , FB , min(TA , FB )) = max(0.2, 0.1, min(0.7, 0.1)) = max(0.2, 0.1, 0.1) = 0.2

19. Belief Augmented Frame Logic(BAF-Logic) • DIR = TR - FR = 0.7 – 0.2 = 0.5 • UR = (1 + 0.5) / 2.0 = 0.75 • Suppose now it is known that B C R

20. Belief Augmented Frame Logic(BAF-Logic) • Combining our clauses regarding R, we obtain: • R = (A  B)  (B  C)  (A   B) = A  B  C  ( A  B) • With De-Morgan’s Theorem we can derive  R: •  R= A   B   C  (A   B)

21. Belief Augmented Frame Logic(BAF-Logic) •  TR = min(TA , TB , TC , max(FA , TB )) = min(0.7, 0.9, 0.2, max(0.2, 0.9)) = min(0.7, 0.9, 0.2, 0.9) = 0.2 • FR = max(FA , FB , FC , min(TA , FB )) = max(0.2, 0.1, 0.7, min(0.7, 0.1)) = max(0.2, 0.1, 0.7, 0.1) = 0.7

22. Belief Augmented Frame Logic(BAF-Logic) • DIR = TR - FR = 0.2 – 0.7 = -0.5 • UR = (1 - 0.5) / 2.0 = 0.25 • Here the new evidence that B C R fails to support R, because C is not true (DIC = -0.5)

23. Text ClassificationFirst Approach • First Formulation: • Using Individual Word Scores • Assuming that a document dibelongs to a class ck, then for every term tijthe following relation holds: di ck (ti0 ck ti1 ck  ti2 ck …  ti,n-1 ck)

24. Text ClassificationFirst Approach • Likewise, for a document dinot belonging to a class ck, we can derive: di ckm, mk (ti0 cm ti1 cm  ti2 cm …  ti,n-1 cm) • These can be formulated in BAF-Logic: Tdi  ck = min(p(ck | ti0), p(ck | ti1), …, p(ck | ti, n-1)) Fdi  ck = max(min(p(cm | ti0), p(cm | ti1), …, p(cm | ti, n-1)), min(p(cn|ti0), p(cn|ti1),…,p(cn|ti,n-1)), …)), m, n etc  k

25. Text ClassificationFirst Approach • The final score of a document di belong to class cj is given by: • Where:

26. Text ClassificationFirst Approach • Individual term probabilities are derived using Bayesian probabilities:

27. Text ClassificationSecond Approach • We classify the entire document using Naïve Bayes assumption: • Trivial to derive the supporting score that di ck. • It is simply p(ck | di)

28. Text ClassificationSecond Approach • Formulating the Refuting Score is straightforward too: di ck di cm di cn­ di cp…, m, n, p, etc  k • We can formulate both supporting and refuting scores in BAF-Logic:

29. Text ClassificationSecond Approach • We retain the definitions of DI and U from the first approach.

30. Experiment Protocol • Using Andrew McCallum’s “Bag of Words” or BOW library. • Extended “rainbow”, the text-classification front-end, with two BAF classification methods. • Methods are called BAF1 and BAF2 • Also extended with two PAS methods (see paper for more details) • Methods are called PAS1 and PAS2

31. Experiment Protocol • Corpus: • 20 Newsgroups • 80% (16,000) documents used to generate statistics. • 20% (4,000) documents used for testing • Choice of documents for training/testing handled by BOW • Headers removed from all documents

32. Experiment Protocol • Trials • 10 trials were performed using each classification method. • Naïve Bayes, tf.idf, kNN, EMM, Max entropy, Probabilistic Indexing, BAF1, BAF2, PAS1, PAS2 • The average was taken from the 10 trials for each method.

33. Experiment Results

34. Analysis • BAF1 performs poorly. • Using individual word scores appears to be a poor idea. • BAF2 performs very well. • Better than the other methods attempted. • BAF2 Performance slightly better than Naïve Bayers • Appears that considering a document to belong to another class has a positive effect on classification scores.

35. Conclusion • Experiment results show that the use of BAF-Logic to classify documents might be a good idea. • In addition there are features of BAFs (e.g. daemons attached to slots) that might enhance classification performance further. • More work should be done on this. • Understanding better why BAF-Logic works for text classification. • Improving classification performance.