String Similarity Measures and Joins with Synonyms

1. String Similarity Measures and Joins with Synonyms Jiaheng Lu Renmin University of China Joint work with Chunbin Lin, Wei Wang, Chen Li, Haiyong Wang

2. Motivation Example (String Measure) S1=“International Conference on Management of Data NY USA” S2=“SIGMOD 2013 New York United States” no semantic

3. Example (String Measure) S1=“International Conference on Management of Data NY USA” S2=“SIGMOD 2013 New York United States” Synonyms SIGMOD  International Conference on Management of Data NY  New York USA  United States SIGMOD  ACM's Special Interest Group on Management Of Data How to use the existing synonyms?

4. Research Problem 1--(String Measurements) Input Two strings s and t, and a set of synonyms R Output Using R to return the maximal Jaccard similarity Jaccard(s,t,R)

5. Problem 2-- (String Similarity Join) Input Two set of strings S and T, and a set of synonyms R, and a threshold value Output Return all similar pairs , such that Jaccard(s,t,R)>=

6. An example of similarity join Table S1 Table S2 Synonyms SIGMOD International Conference on Management of Data VLDB Very Large Data Bases

7. Existing works on approximate string match with synonyms • Transform based framework (JaccT) [1], compared with our method. • Machine leaning method [2], Hidden Markov Model-based measure. Depend on training data, not efficient • Transform based framework (JaccT) [1], compared with our method. • Machine leaning method [2], Hidden Markov Model-based measure. Depend on training data, not efficient [1] A. Arasu, S. Chaudhuri, and R. Kaushik. Transformation-based framework for record matching. In ICDE, pages 40–49, 2008. [2] M. Bilenko and R. J. Mooney. Adaptive duplicate detection using learnable string similarity measures. In KDD, pages 39–48, 2003. [1] A. Arasu, S. Chaudhuri, and R. Kaushik. Transformation-based framework for record matching. In ICDE, pages 40–49, 2008. [2] M. Bilenko and R. J. Mooney. Adaptive duplicate detection using learnable string similarity measures. In KDD, pages 39–48, 2003.

8. Outline • Motivation & Problem Statement • String Similarity Measures • String Similarity Joins • Experimental Results • Conclusion

9. String Similarity Measures (Full-expansion) S1=“International Conference on Management of Data NY USA” S2=“SIGMOD 2013 New York United States” Synonyms SIGMOD  ACM's Special Interest Group on Management Of Data SIGMOD International Conference on Management of Data NY New York USA United States Expanding using all synonyms S1’=" International Conference on Management of Data NY USASIGMODNew YorkUnited States" S2’=" SIGMOD 2013 New York United StatesInternational Conference on Management of DataNY USAACM's Special Interest Group on Management Of Data Jaccard(S1’,S2’)= 13/18 = 0.72

10. String Similarity Measures (Selective-expansion) S1=“International Conference on Management of Data NY USA” S2=“SIGMOD 2013 New York United States of America” Synonyms SIGMOD  ACM's Special Interest Group on Management Of Data SIGMOD International Conference on Management of Data NY New York USA United States Expanding using only good synonyms S1’=" International Conference on Management of Data NY USASIGMODNew YorkUnited States" S2’=" SIGMOD 2013 New York United StatesInternational Conference on Management of DataNY USA" Jaccard(S1’,S2’)= 13/14 = 0.93

11. String Similarity Measures (Selective) Selective-expansion is: NP-hard : Reduction from 3-SAT Choose synonyms that can increase current similarity by computing the similarity-gain Greedy algorithm Property Optimal, when more than 70% cases in practice.

12. Outline • Motivation & Problem Statement • String Similarity Measures • String Similarity Joins • Experimental Results • Conclusion

13. Similarity Joins (Filtering and Verification) Filtering candidates Verify candidates Generate Signatures with full expansion Similarity Measures Full expansion Prefix method Selective expansion LSH method

14. String Similarity Joins (SN-Join) Prefix method Global ordering: {a b c d e f g h i j k l} S1=“c k, e, a, f” S2=“d, b, f, e, k” Order the strings Threshold=0.8 S1’=“a, c, e, f, k” S2’=“b, d, e, f, k” Get signatures Sig(s1)=“a, c” Sig(s2)=“b, d” No overlap Jacc(s1,s2)<0.8

15. Signatures selection is important How to select signatures to enhance the signature filtering power? It is unrealistic to find a “one-size-fits-all” solution.

16. Estimation-based signatures selection . • Three steps to select signatures: • Generate multiple signatures schemes for each data set. • Given two tables for join, quickly estimate the filtering power of each scheme. • Select the scheme with the best filtering power.

17. An example on estimator Self-join: Filtering results (candidates): (q2,q3) ,(q1,q2) (q1,q3) q1 q2 q3 q1 q2 q3 q1 q1 q4

18. Applying FM sketches on inverted lists Self-join: Filtering results (candidates): (q2,q3) ,(q1,q2), (q1,q3) Using Flajolet-Martin (FM) sketch for each list q1 q2 q3 q1 q2 q3 q1 q1 q4

19. FM sketches （Flajolet and Martin JCSS 1985） Estimates the number of distinct items in a multi-setof values from [0,…, M-1] Assume a hash function h(x) that maps incoming values x in [0,…, M-1] uniformly across [0,…, 2L-1], where L = O(logM) Let lsb(y) denote the position of the least-significant 1 bit in the binary representation of y A value x is mapped to lsb(h(x)) BITMAP 5 4 3 2 1 0 lsb(h(x)) = 2 h(x) = 101100 0 0 0 1 0 0 x = 5 Number of distinct values: 5 3 0 5 3 0 1 7 5 1 0 3 7

20. Estimating the filtering power of a signature scheme Constructing a two-dimensional hash sketch Computing tighter upper and lower bounds of candidates size

21. String Similarity Filtering with Length Filter Filtering candidates Verify candidates Generate Signatures Compute lengths Similarity Measures Full expansion Prefix method Length filter Selective expansion LSH method

22. String Similarity Joins (SI-Join) Length filtering Strings S1=“a b c d e” S2=“x y z” Full-expansion Length range S1’=“a b c d e f g h k” S2’=“x y z s” s1: [5, 9] s2: [3, 4] Synonyms a->f g h x-> s b->k Jacc(s1,s2,R)<0.9

23. String Similarity Joins (SI-Join) Filtering candidates Verify candidates Generate Signatures Compute lengths Similarity Measures Full expansion Prefix/LSH method Length filter Selective expansion

24. String Similarity Joins (SI-tree)

25. Outline • Motivation & Problem Statement • String Similarity Measures • String Similarity Joins • Experimental Results • Conclusion

26. Data sets and algorithms Compared method: JaccT [Arasu et al. ICDE 2008] Three datasets:

27. Effectiveness of different similarity measurements String Similarity Measures Selective-expansion (SE) achieves the best effectiveness.

28. Efficiency of algorithms S: selective expansion F: full expansion String Similarity Joins SI-Join achieve the best performance.

29. Prefix V.s. LSH Prefix scheme VS. LSH schemee Prefix is better LSH is better

30. Estimation effectiveness

31. Outline • Motivation & Problem Statement • String Similarity Measures • String Similarity Joins • Experimental Results • Conclusion

32. Conclusion and future work • String similarity measure with synonyms • Two new measures and a new join algorithm • One estimator for signature selection • Future work: how to deal with synonym ambiguity • E.g. UW = University of Washington UW = University of Waterloo ？ OR

33. String Similarity Measures and Joins with Synonyms