Indexing Text with Approximate q-grams

1. Indexing Text with Approximate q-grams Adriano Galati & Marjolijn Elsinga

2. Overview • Approximate string matching - Neighborhood generation - Reduction to Exact Searching - Intermediate Partitioning • Indexing text using q-grams • Filtration condition • Finding approximate q-grams - Trie data structure - Non-deterministic automaton (NFA) • Parameters

3. Approximate string matching Text Pattern Goal: Retrieve all occurrences of P in T whose edit distance is at most k Edit distance:

4. Solutions All kinds of solutions, most investigated area in computer science In on-line versions of the problem the pattern can be preprocessed, the text cannot Classical solution: using dynamic programming and a matrix is O(mn)

5. Classical solution Fill matrix where ìs the minimum edit distance between P and a suffix of T Initialize the borders with Fill internal cells with

6. Solution (2) If text is large, on-line algorithms are not practical and preprocessing becomes necessary Focus: Sequence retrieving indexes, with no restrictions on the patterns and the occurrences Approaches: • Neighborhood Generation • Reduction to Exact Searching • Intermediate Partitioning

7. Neighborhood Generation Set of strings matching a pattern with k errors is finite ( ) Therefore it can be enumerated Each string can be searched using a data structure This structure is designed for exact matching

8. Neighborhood Generation (2) + O(n) space and construction time • Not optimized for secondary memory • Inefficient in space requirements Is promising for searching short patterns only

9. Reduction to Exact Searching Indexes based on filters Filter checks for simpler condition than the matching condition, discarding large parts of the text Main principle: if two strings A and B match with k errors and k+s non-overlapping samples are extracted from A, then at least s of these must appear without errors in B

10. Reduction to Exact Searching (2) + can be built in linear time and need O(n) space + with some method it is possible to make an index that takes less space then the text itself - Are based on suffix trees or on indexing all the q-grams

11. Intermediate Partitioning • Reduces the search to approximate search instead of exact search • Main principle: if two strings A and B match with at most k errors and j disjoint substrings are taken from A, then at least one of these appears in B with • Split the pattern in j pieces, search each piece in the index allowing errors, extend the approximate matches to complete occurrences

12. Question (Ingmar) I think the main principle is incorrect, because if AAABBBBBB BBBBBBBBB These match with k=3 errors. If we take the disjoint substrings AAA BBB BBB so j=3. Now they say that one of these will appear in the other with errors. However AAA match with 3 errors, BBB with 0 and BBB with 0

13. Answer The pattern is split in j pieces, each piece is searched in the index allowing errors AAA BBB BBB BBB BBB BBB We match BBB with ABB and not with AAA and AAB, because it is not possible to match them with more then errors, with k=3 and j=3, unless we change the parameters

14. Intermediate Partitioning (2) + optimizing point between neighborhood generating (worse with longer pieces) and reduction to exact searching (worse with shorter pieces) Has been used on the patterns but not yet on the text itself

15. Indexing text using q-grams Steps: • Filtering text • Finding approximate q-grams Advantages: • Takes little space • Has an alternative tradeoff • User can decide what is important: saving space or better performance

16. Filtration condition Based on locating approximate matches of pattern q-grams in text Leads to a filtration tolerating higher error levels compared to exact q-gram matching

17. Condition for an approximate match Two strings A and B Now: at least one string Ai appears in B with at most errors Only the q-grams for which this hold, will be used for searching

18. Example: Condition A: CCTC TCTC CCCT B: CCCC CTCT TCTC We see: k=8 We take: j=3 Now e=2, so at least one Aiappears in B with at most 2 errors

19. Question (Peter) “Note that it is possible that , so we are not only ‘distributing’ errors across pieces, but also ‘removing’ some of them” How does this work?

20. Answer k=5 j=3 e=1 A1 x1 A2 x2 A3

21. Q-grams vs. Q-samples Q-grams overlap Q-samples do not overlap String: ABCDEF Q-grams: {ABC, BCD, CDE, DEF} Q-samples: {ABC, DEF} In a q-gram index all the text q-grams are stored in increasing order In a q-sample index only some text q-grams are stored

22. Constructing q-samples We need to extract j pieces from each potential pattern occurrence in the text So: a q-sample every h text-characters We need to guarantee that j q-samples are inside any occurrence of P Minimal length of P = m-k

23. Question (Jacob) Could you please explain how the restriction of h is built up?

24. Answer

25. Next step Best match distance (bed) is calculated for each test sequence of q-samples This is the distance between the q-sample sequence and the involved text (h) The text area h is only examined if its bed is at most k

26. Algorithm Each q-sample sequence has its own counter M M indicates the number of errors produced by the q-sample sequence and is initialized to So: we start by assuming that each q-sample gives enough errors to disallow a match

27. Error-environment After calculating the M for each q-sample sequence, we obtain the e-environment of each q-sample sequence This is the set of possible q-samples that appear inside the q-sample sequence with at most e errors

28. Finishing Now all text areas have its own e-environments connected to it through the q-samples They can be checked with dynamic programming

29. Finding approximate q-grams Finding all the text q-samples that appear inside a given pattern block Note: it is not necessary to generate all since we are interested only in the text q-samples (position)

30. Finding approximate q-grams (2) • Idea: to store all the different text q-samples in a trie data structure • We fill in a matrix such that is the sed between and a suffix of • is relevant for some • In a trie traversal of the q-samples, the characters of are obtained one by one

31. Question (Laurence) • Can you please show me the matrix is build on page section 3 in fig. 4? It is a bit unclear to me how the matrix is initialized and the different cells are being filled.

32. Answer

33. Answer s u r g e r y 0 0 0 0 0 0 0 0 s 1 0 1 1 1 1 1 1 u 2 1 0 1 2 2 2 2 r 3 2 1 0 1 2 2 3 v 4 3 2 1 1 2 3 3 e 5 4 3 2 2 1 2 3 y 6 5 4 3 2 2 2 2

34. Finding approximate q-grams (4) When we reach the leaf nodes (depth q) we check in if there is a cell with value the corresponding text is reported Complexity

35. Finding approximate q-grams (3) Pruning: • All the value of a row to the next are nondecreasing • If all the values of a row are larger than at that point we can abandon that branch of the trie

36. Finding approximate q-grams (5) Alternative way: • To model the search with a non-deterministic automaton (NFA)

37. Finding approximate q-grams (6) Consider the NFA for errors Every row denotes the number of errors seen Every column represents matching a prefix of S Horizontal arrows represent matching a character All the others increment the number of errors

38. Question (Bogdan) I can imagine how the trie can be used together with the matrix in order to benefit from common prefixes of certain q-samples (by reusing the rows of the matrix which are already computed for the common prefix). However, I don't see how this can be done in the case of the NFA. If it can't be done, this would mean that the algorithm has to be run separately for each q-tuple, which probably makes the NFA approach much worse. Am I right to think that or is there a way to run the NFA in a "smarter" way so as to benefit from common prefixes?

39. Bogdam (answer) • Yes, you are right, the algorithm has to run for each q-tuple, but you have to consider the complexity of it, that is linear

40. Parameters of the Problem • Smaller value the search of e-environment will be cheaper • Larger value gives more exact estimates of the actual number of error but with a higher cost to search the e-environment • As grows, longer test sequences with less errors per piece are used the cost to find the relevant q-samples decreases but the amount of text verification increases.

41. Parameters of the Problem (2) • Notice: the index of this approach only stores non-overlapping q-samples, its space requirement is small • Notice: the space consumption of index depends on the interval h

42. Parameters of the Problem (3) • Standard implementation q-gram index stores all the locations of all the q-grams of the text • The number of q-grams • Storing a position takes • space consumption is • Ratio between this method and standard approach

43. Question (Bogdan) • Could you please explain what the "columns" used in the 5th section are? • The table shows how the error level increases the number of processed columns of matrix or NFA

44. Question (Lee/Bram) • The article talks about disjoint non overlapping q-grams. At the end they say that will probably enhance the scheme that they allow overlapping q-grams. Any idea how our current algorithms have to be changed for that and what the advantages are? • http://www.cs.utexas.edu/users/mobios/MoBIoSPapers/2003-IndexingProteinSequences-TR-04-06.pdf

45. Question (Lee) In the second paragraph of section 4 they say “In that particular case we can avoid the use of counters…” Can you explain that ?

46. Answer The error counters M are initialized at a high value After that all pattern-blocks are compared to the corresponding text piece and the counter value is updated to a lower value In this particular case, when e = the error counter can get as low as k+1, which is higher than the initial value

47. Any other questions?