Phonetic String Matching:Lessons from Information Retrieval

1. Phonetic String Matching:Lessons from Information Retrieval Advisor : Dr. Hsu Graduate : Chih-Ling Wang Authors :Justin Zobel Philip Dart 2003 IEEE .

2. Outline • Motivation • Objective • Introduction • Phonetic matching versus information retrieval • Phonetic matching techniques • Performance assessment • Combination of evidence • Conclusions • Personal Opinion

3. Motivation • We explore the accuracy of the phonetic string matching.

4. Objective • In this paper we propose a new phonetic matching techniques and describe the results of a new comparative investigation of phonetic matching.

5. Introduction • Phonetic matching is used to identify strings that may be of similar pronunciation, regardless of their actual spelling.

6. Introduction(cont.) • There are two pragmatic issues that must be addressed in such a phonetic matching system. • One is of speed – answers should be found reasonably quickly. • The other pragmatic issue is accuracy. • The parallels between information retrieval and phonetic matching mean that • They can be measured by the same kinds of techniques. • Methods for improving information retrieval performance may also apply to phonetic matching.

7. Phonetic matching versus information retrieval • In information retrieval, ranking is the process of identifying which of a set of documents are most likely to be similar in content to a given query. • Phonetic matching is the process of identifying which of a set of strings are most likely to be similar in sound to a given query string.

8. Phonetic matching versus information retrieval(cont.) • In both cases the matching process is: fundamentally inexact, since human judgment is required to tell whether the process’s guess is correct • Similarity is relative, unable in isolation to determine whether a query and potential answer are matches. • It is difficult to give an accurate definition of relevance.

9. Phonetic matching versus information retrieval(cont.) • We consider phonetic matching to be the process of identifying strings that, after elimination of possible transmission or cognition errors, may sound the same. • Transmission errors include, sound-alike mistakes in data entry ;mishearing of a spoken name on a imperfect transmission medium. • Cognition errors include, mistaking a pronunciation for an expected word.

10. Phonetic matching techniques Soundex • Soundex uses codes based on the sound of each letter to translate a string into a canonical form of at most four characters, preserving the first letter. • Soundex makes the error of transforming dissimilar-sounding strings to the same code, and of transforming similar-sounding strings to different codes. • There is no ranking of matches: strings are either similar or not similar.

11. Phonetic matching techniques(cont.) Example: reynold(r005043)=>r543 renauld(r050043) =>r543

12. Phonetic matching techniques(cont.) Phonix • Phonix is a Soundex variant. • Letters are mapped to a set of codes using the same algorithm, but a slightly different set of codes is used, and prior to mapping about 160 letter-group transformations are used to standardise the string. • The sequence tjv is mapped to chv if it occurs at the start of a string, and x is transformed to ecs. • These transformations provide context for the phonetic coding and allow c and s to be distinguished.

13. Phonetic matching techniques(cont.) Example: reynold(r005043)=>r543 renauld(r050043) =>r543

14. Phonetic matching techniques(cont.) • In our experiments we consider a variant of Phonix, here called Phonix+, in which truncation is not applied and a minimal edit distance is used to compare the resulting strings.

15. Phonetic matching techniques(cont.) Q-gram methods • A q-gram of string s is any substring of s of some fixed length q. • Simply counting q-grams does not allow for length differences.

16. Phonetic matching techniques(cont.) Example:rhodes;rod We have used this q-gram method with q=2 rhodes= rod=

17. Phonetic matching techniques(cont.) Agrep • Agrep is a utility that embodies a fast algorithm for identifying strings that contain a substring which is identical to a query but for at most k insertions, deletions, or replacements, where k is a predefined constant. • Agrep was not designed for the task of phonetic matching, but rather for fast searching of large files.

18. Phonetic matching techniques(cont.) Edit distances • A simple edit distance, which counts the minimal number of single-character insertions, deletions, and replacements needed to transform one string into another, could be used for phonetic matching since similar-sounding words are often spelled similarly.

19. Phonetic matching techniques(cont.) • For two strings s and t of length i and j respectively, this edit distance can computed with the recurrence relation edit(i,j). • The function returns 0 if and are identical, and 1otherwise.

20. Phonetic matching techniques(cont.) Example: rhodes;rod 4 7 3 2 4 3 1 3 3 3 2 2 3 2 2 4 2 3 2 3 1 2 2 0 2 2 1 1 3 2 1 2 3 2 4 3 2 5 3 3 5 4 3 5 4 5 4 6 4 6 5 5 6 7

21. Phonetic matching techniques(cont.) Editex • Editex is a phonetic distance measure that combines the properties of edit distances with the letter-grouping strategy used by Soundex and Phonix. • Editex also groups letters that can result in similar pronunciations, but doesn't require that the groups be disjoint and can thus reflect the correspondences between letters and possible similar pronunciation more accurately.

22. Phonetic matching techniques(cont.) • Editex is defined by the edit distance recurrence relation with a redefined function returns 0 if and are identical, 1 if and are both occur in the same group, and 2 otherwise. • The function is identical to .If a is h or w and a b then is 1.

23. Phonetic matching techniques(cont.) Phonometric methods • Our algorithms for phonometric matching consist of two stages: • First, the string of letters is converted into a string of phonemes by a string-to-pronunciation conversion algorithm. • The second stage is comparison of strings of phonemes. • The distance between pronunciations as represented by strings of phonemes can be measured more precisely than the distance between strings of letters.

24. Phonetic matching techniques(cont.) Tapering • Tapering is a refinement to the edit distance techniques based on a human-factors property: differences at the start of a pronunciation can be more significant than differences at the end. • A tapered edit distance of particular interest is one in which the maximum penalty for replacement or deletion at start of string just exceeds twice the minimum penalty for replacement or deletion at end of string.

25. Performance assessment • We can now compare the various approaches to phonetic matching.Results are shown in Table 1, which is of 11-point recall-precision. • For many of the techniques tested, only a few distinct ranks are possible, and some techniques only return two ranks, match and not-match.

26. Performance assessment(cont.) • The least effective methods such as Phonix and Soundex only return a small number of answers for most queries. • Phonix and Soundex are not only finding many wrong answers but not finding many right ones. • The “baseline” results are for a trivial phonetic matching method: find all strings with at most one character – an insertion, deletion, or replacement – different from the query.

27. Performance assessment(cont.) • A particular problem of best agrep is the tiny number of correct answers returned – less than one per query – but we stress that agrep was not designed for phonetic matching.

28. Performance assessment(cont.) • An interesting discovery is that even the most successful of the methods fetch rather different sets of answers, sometimes almost without overlap. • As for information retrieval, it seems, two methods can perform well without finding the same answers.

29. Combination of evidence • Phonetic matching has strong parallels with information retrieval. • Matching techniques fetch a ranked list of matches in which each entry has weight attached to it; this weight is the likelihood that the entry is a good match. • Combining the ranked lists produced by different retrieval mechanisms can improve performance.

30. Combination of evidence(cont.) • The “(none)” lines are the results of running the methods individually. • The best performance of all is given by the combination of Phonix+ and the q-gram method, neither of which works particularly well alone.

31. Combination of evidence(cont.) • More sophisticated techniques for combination could be used: • Weighting the ranks from the different techniques. • Combining more than two methods. • That combination of evidence is successful in this context.

32. Conclusions • Two of our proposals – the Ipadist and Editex methods – do indeed lead to improved performance, whereas the third – tapering – was not successful. • We showed that combination of evidence, which has been successfully applied to information retrieval, consistently improves performance. • Our new methods are substantially more effective than existing methods such as edit distances, and that combination of evidence is as valuable in this domain as it is in information retrieval.

33. Personal Opinion • The concept in this paper may use in our research, but I haven’t have a clear idea to implement it.I need more time to think…think…