On Finding Repeats in Strings

1. On Finding Repeats in Strings A Survey of Algorithms 胡吉祥 2004.09.29

2. I．Problem Description • Repeat：Recurrence of a pattern • Aim: 1) Finding occurrence of repeated substrings in a string or a set of strings. 2) Finding a common structure/pattern shared by a family of strings.

3. 1. Basic Definition • A pair of substrings R = (S[i1, j1], S[i2, j2]) is called a repeat: - exact repeat：S[i1,j1]=S[i2,j2] - k-mismatch repeat ： k mismatches between S[i1,j1] and S[i2, j2] - k-differences repeat： k differences (mismatches, insertions, deletions) between S[i1,j1] and S[i2, j2]

4. Definition 2 A repeat in a string x is a tuple Mx,u = (p; i1,i2,….ir), r≥2; Where u = x[i1…i1+p-1] = x[i2…i2+p-1] = ... =x[ir…ir+p-1]. • u is said to be a repeating substring of x and the generator of Mx,u. • p = |u| the period of the repeat and r its exponent.

5. 2.Maximal exact matches • Definition 3: If S[i,i +k]=S[j, j+k], where 1≤k≤ n - 1 and 1≤i, j≤n - k, then we say S[i,i+k]and • S[j, j+k] match. This match is represented by (i,j,k+1 ) or equivalently, ( i,j,k+1 ); i.e., by the • two starting positions of the match and the length of the match.

6. Definition 4 A match (i, j,k) in a string S is left-extensibleif i, j > 1 and Si-1 = Sj-1, and right-extensibleif i+k, j+k<nand Si+k+1 = Sj+k+1. • A match ismaximalif it is neither left-extensible nor right-extensible and it is not the trivial match (1,1,n), where nis the length of the string

7. 3. Approximate Repeats • Exact repeat: identical substrings • Inexact repeats: distance measures between recurrences • Hamming distance: mismatch => k-mismatch repeat • Levenshtein / edit distance: substitution, insertion, deletion => k-differences repeat

8. II. Algorithms for finding repeats in strings • 1. N-gram Iterative / KMR • 2. Martinez’s Sorting Algorithm • 3. Fingerprint • 4. Suffix Tree • 5. GST (Generalized Suffix Tree) • 6. Suffix Array • 7. Grammar Induce / Sequiur • 8. Term Frequency Statistical Approach • 9. Others

9. 1. N-gram Iterative / KMR • [KMR,1972] • A classical algorithm for finding all exact repetition in a string. • A O(Nlogk) algorithm for finding exact repeated k-length words in a N-length sequence.

10. Aim • Given a string S, KMR solves the following problems: • 1. identify the positions of all the words of a fixed length k that appear repeated in s; • 2. find the length kmax of the longest repeated word in s, and solve problem 1 for k= kmax.

11. Idea • Iterative construction of equivalence relation over the positions of s. • Definition: Given a string s=s1s2…sn∈Σn , two position i and j ∈{1,…,n-k+1} of s are said to be k-equivalent, noted i Ek j, if and only if the words of length k staring at these positions in s are identical.

12. Lemma 1: Let s=s1s1…s1∈Σn . Let a,b ∈{1,…,n} with b≤a, and i,j∈{1,…,n-(a+b)+1}. i Ea+b j iff i Ea j and i+b Ea j+b • Lemma 2: Let s=s1s1…s1∈Σn . Let a,b ∈{1,…,n}, and i,j∈{1,…,n-(a+b)+1}. i Ea+b j iff i Ea j and i+a Eb j+a

13. Ek and of its classes is basically an operation of set intersection on the classes of Ek’ for k’<k. • Double Technique => Suffix Sorting • Analysis: Time Complexity: O(nlogk) Space Complexity: O(n)

14. Example for KMR

15. Example for KMR (cont.)

16. Example for KMR (cont.)

17. Example for KMR (cont.)

18. Variation of KMR • [Land,1989]: finding a common structure shared by a family of string • Aim: find a structure common to N strings of characters belonging to a given alphabet. - consider a word is relevant if it appears without any distortion in at least a fixed number q of strings among the N strings, and with some distortion in the N-q remaining strings.

19. Adaptation of the KMR Algorithm • arrange the N strings in a single vector S, take into account the boundaries of the N strings in S, any word must be wholly enclosed in only one string. • memorize the locations in S of the first and last characters of each string. • adapt the KMR method, keep the word which abides by the quorum condition and does not cross any boundary.

20. 2. Matinez’s Sorting Algorithm • [Martinez,1983]: • a sorting algorithm with a time complexity O(nlogn) to display a priori unknown identically repeated patterns in several molecular sequences. • solve the problem of finding repeats in molecular sequences as a sorting problem. • linear in space complexity and NlogN in expected time complexity. • involve no path tracing. The repeats are immediate and are reported during the processing of sorting

21. Idea • (1) First construct a sequence P of pointers such that pointer value P[i] is the location of the ith element in the sequence S. • (2) Sort P so that it constitutes an ordering of S. P[i] <, >,= P[j]  S[P[i]J <, >,= S[P[j]] - all the pointer values which point to the same kind of element in S are grouped together. - at most m groups of pointer values in this first sorting. (m=|Σ|)

22. Idea (cont.) • (3) Sort each of these groups of P so that in the resulting subgroups two pointer values belong to the same one if and only if the elements immediately following the ones they point to are equal. • (4) When no subgroups contain more than one pointer value the process is complete.

23. Example

24. Conclusion: • No more than repeated application of a sorting algorithm. The overall speed is essentially determined by the speed of the sorting algorithm employed. No significant storage space is required beyond that necessary for the sequence S and its pointer sequence P.

25. 3. Fingerprint • Definition： A fingerprint (a.k.a. signature) of an object Ob is a small tag f(Ob) with the following properties: • a) f is a function of Ob. f(A)≠f(B) =>A≠B • b) Pr(f(A)=f(B) | A≠B) = very small

26. Useful Properties of Fingerprints • Fast Calculation • Low collision rate • Cryptographically unbreakable • Updatable • Concatenation of Objects

27. Karp Rabin Style Fingerprints • A=[a0,a1,…an-1] Calculation time linear in N

28. Easy to calculatconsecutive n-grams Easy to calculate signatures of concatenations

29. Discover repetitions using fingerprints • Algorithm FSR: Finds shortest repetition. - Input: A string y∈Σ*Σ, |y|=n - Output: A pair of indices 1≤i<j≤n, s.t. y(i,j-1)=y(j,2j-i-1) and |j-i| is minimal • Time Complexity: O(nlogn)

30. 4. Suffix Tree Based Algorithms • Repeat finding with suffix trees: – Exact repeats – Approxiamte

31. Suffix Trees • A suffix tree is a trie-like data structure representing all suffixes of a string. • A suffix tree of a string S, T(S), is a rooted tree whose edges are labeled with strings such that – all edges leaving a node begin with different characters and – the paths from the root to the leaves represent all the suffixes of S.

32. b x a c x a 6 a c b c x 5 b c x a a 4 c c 3 2 1 Suffix Tree (xabxab) • {xabxac, abxac, bxac, xac, ac, c}

33. (1) Finding Exact Repeats • Folklore: (see e.g. Gusfield, 1997) It is possible to find all pairs of repeated substrings (repeats) in S in linear time. • Idea: • consider string S and its suffix tree T(S). •repeated substrings of S correspond to internal locations in T(S). • leaf numbers tell us positions where substrings occur. • Analysis: O(n + |output|) time, O(n) space

34. Finding maximal exact repeats • Idea: (see e.g. Gusfield, 1997) • For right-maximality (X ≠ Y) – consider only internal nodes of T(S) – report only pairs of leaves from different subtrees (or from different leaf-lists) • For left-maximality (A ≠ B) – keep lists for the different left-characters – report only pairs from different lists

35. Dup: Finding Duplication in Strings and Software. (Baker,1993) • Maximal matching pairs must have both different right context and left context. • build up lists of suffixes grouped by left context; • compare the lists found for its subtrees to identify longest matches. • Analysis: O(n + |output|) time, O(n) space

36. abcbc# # bc c # bc# # bc# Finding Maximal Repeats abcbc left diverse a c

37. (2) Finding degenerate repeats • k-mismatch repeats (Hamming distance) / k-differences repeats (edit distance) • (Kurtz et al. 2000/2001, Adebiyi et al. 2001, Volfovsky et al,2001, Kolapov & Kucherov,2001)

38. Idea: • Minimal length l, up to k errors → filter method (“seed and extend”) - first search for exact repeats of small but appropriate lengths - form maximal approximate repeats by expansion of the exact repeats (called seeds) to the surrounding sequence, allowing k-mismatches or k-differences between recurrences. - two types of exact repeats can be used as seeds: maximal repeats and the super-maximal repeats.

39. Algorithm: 1. Search for local exact repeats (seeds). 2. Extend the seeds while allowing up to k errors. 3. If extension is long enough, output repeat. • Analysis: O(n +ζk3) time with E(ζ) = O (n2/4s), s minimal seed length.

40. 5. Generalized Suffix Tree (GST) • GST: • a Suffix Tree that combines the suffixes of a set {S1, ...., Sn} of strings.

41. $ a ab$ b # b $ ab# # $ ab# # Generalized Suffix Trees - Example T1 = abab T2 = aab # $

42. Generalized Suffix Trees - Applications • Searching for a pattern in a database of strings. • Finding longest common substring.

43. (1) k-common substring Problem (Guesfield,1997) • Problem definition: Longest common substrings of >2 strings: - Input Strings S1, …, SK (total length n) - Output l(j) (and pointers to substrings) for 2 <= j <= K • l(k): the length of the longest substring that appears in at least k distinct strings of S.

44. Solution • - Build a generalized suffix tree for the K strings each string has a unique end character, so each leaf shows up only once - C(v): number of distinct leaf labels in subtree rooted at node v - Given C(v) values and string-depth values, do a simple traversal of tree to find these K-1 values and pointers to locations in substrings • Time complexity: O(Kn)

45. (2) Color Set Size Problem (Lucas Hui, 1992) • CSS(Color Set Size) Problem: given a rooted tree of size n with l leaves colored from 1 to m, m≤l，for each vertex u find the number of different leaf color in the subtree rooted at u. • CSS Linear Theorem: The CSS problem can be solve in O(n) time and space.

46. K-out-of-m • k-out-of-m Problem: find the longest substring that is common to at least k strings for a fixed k between 1 and m. • k-out-of-m Problem  finding an internal vertex in GST such that css(u)>k the path length from root to u is maximum.

47. Algorithm KM • - Build GST for the input strings; - Solve the CSS problem of the GST; - Compute path length for all vertices using pre-order traversal; - Among all vertices, select one with css() value >k and have maximum path length; - Output the path from root to the vertex found above as answer.

48. Multiple common substring Problem • a. Given m input strings, for all k between 1 and m, find the longest pattern which appears in at least k input strings. • b. Given m input strings and integers k and l, find a pattern with length l which appears in exactly k input strings.

49. c. Given m input strings and integers l1<l2,find the pattern with length between l1 and l2 which appears in as many input strings as possible. • All the above problems can be solved in O(n+|output|) time by modified the k-out-of-m solution.

50. 6.Suffix Array • Definition: Given a string D thesuffixarraySA for this string is the sorted list of pointers toall suffixes of D. • (Manber, Myers 1990)