What about the trees of the Mississippi?

1. What about the trees of the Mississippi? Suffix Trees explained in an algorithm for indexing large biological sequences Jacob Kleerekoper & Marjolijn Elsinga

2. Overview • Suffix • Suffix array • Suffix tree • Suffix links tree • Demo

3. Suffix Suffices of mississippi: 1 mississippi 11 i 2 ississippi 8 ippi 3 ssissippi 5 issippi 4 sissippi 2 ississippi 5 issippi  sort alphabetically  1 mississippi 6 ssippi 10 pi 7 sippi 9 ppi 8 ippi 7 sippi 9 ppi 4 sissippi 10 pi 6 ssippi 11 i 3 ssissippi

4. 11 8 5 2 1 10 9 7 4 6 3 0 1 2 3 4 5 6 7 8 9 10 Suffix Index Suffix array m i s s i s s i p p i

5. Search in suffix array Idea: two binary searches- search for leftmost position of X- search for rightmost position of X In between are all suffices that begin with X

6. 11 8 5 2 1 10 9 7 4 6 3 0 1 2 3 4 5 6 7 8 9 10 Suffix Index Search in suffix array Search for leftmost occurrence of is m i s s i s s i p p i more occurrences of is left of this one possible! Found leftmost ippi issippi pi

7. 11 8 5 2 1 10 9 7 4 6 3 0 1 2 3 4 5 6 7 8 9 10 Suffix Index Search in suffix array Search for rightmost occurrence of is m i s s i s s i p p i more occurrences of is right of this one possible! issippi Found rightmost ississippi mississippi pi

8. Result search in suffix array Leftmost occurrence of is: 5 at index 2 Rightmost occurrence of is: 2 at index 3 is can be found at [2..3] in the suffix array

9. Tree & Trie Suffix tree is a compressed digital (suffix) trie

10. Suffix tree definition A suffix tree is a rooted directed tree with m leaves, where m is the length S (the database string) For any leaf i, the concatenation of the edge-labels on the path from the root to leaf i exactly spells out the suffix of S that starts at position i

11. Suffix tree building root p p i m i s s i s s i p p i Suffices of mississippi: • mississippi • ississippi • ssissippi • sissippi • issippi • ssippi • sippi • ippi 9 ppi 10 pi 11 i s s i s s i p p i i i s s i s s i p p i i s s i p p i p p i p p i p p i p p i

12. Result suffix tree building root p m i s s i s s i p p i p i s i i 9 11 10 p p i i ssi si s s i p p i s s i p p i 8 s s i p p i p p i p p i p p i 4 1 5 6 2 3 7

13. Questions Adriano: How is the tree created for ANA$?

14. Answer: Tree creation ANA$ root 1 root 2 ANA$ 3 NA$ 4 A$ 5 $ $ ANA$ A NA$ 2 3 5 NA$ $ 2 4

15. Implicit vs. explicit Trees in which a special end symbol is used are called explicit Searching in this trees can only be stopped at this end symbol, which is always in a leaf A search in a implicit tree can stop at any internal or external node, at the last matching symbol

16. Question Peter: How does this method search for homologous sequences as is done in BLAST and CAFE?

17. Searching in a suffix tree root p m i s s i s s i p p i p i issi 2 ississippi 5 issippi s i i 9 11 10 p p i i ssi si s s i p p i s s i p p i 8 s s i p p i p p i p p i p p i 4 1 5 6 2 3 7

18. Time analysis of suffix tree Building a suffix tree can be done in O(k) where k is the length of the database string Searching a suffix tree can be done in O(n) where n is the length of the query string (Note: only in Ukkonen’s implementation)

19. Question Laurence: Can you explain the suffix links tree?

20. Suffix links A necessary implementation trick to achieve a linear time and space bound during building the tree A suffix link is: a pointer from an internal node xS to another internal node S where x is a arbitrary character and S is a possibly empty substring xS S

21. Suffix linked tree root $ 9 AC 7 $ 8 $ AC AC $ 6 $ 5 AC AC $ $ AC$ 4 AC$ 2 3 1

22. Question Ingmar: Why is the memory bottleneck a problem, and how is it solved with the use of suffix links? Answer: we interpreted the article in such way that the suffix links cause the memory bottleneck and not the other way around

23. Question Lee: How can suffix links cause the memory bottleneck and why is its reliance on virtual memory impractical? Answer: Suffix links are designed to take you from one region of the tree to another. It could be possible, because of the size of the tree, that the region pointed to is not in memory available. The same holds for virtual memory.

24. Question Bram: Why do we need random access of the memory? Answer: a tree is based on pointers, these are not sequentially inserted into the memory, so random access is necessary

25. Question Bogdan: How does this index cope with partial matches, gapped alignments and so forth, or is it just used for exact matches, which usually don’t help a lot? Answer: Your intuition is correct here. Suffix trees as described in the article can only be used for exact (local) matches

26. Question Lee: Can this method be used for protein data as well / can this method also be used for similar matches? Answer: Suffix trees probably can be used for protein data, but it is not possible to implement wildcards or the fact that amino acids are evolutionary related, but do not match exactly in some cases.

27. Question Peter: Why is it a problem that DNA cannot be broken into words, and why doesn’t it use the overlapping intervals as in CAFE? Answer: the begin and end of a base string cannot be determined. Suffices are a special kind of overlapping intervals.

28. Question Bogdan: Why do we have to change the index for each search instead of building the index once and update it when the database is changed? Answer: the index mentioned is the BLAST index and in BLAST the index has to be updated for every search. It has not much to do with suffix trees.

29. Question Adriano: What is the meaning of "cold store" and "warm store"? Answer: We think that cold store means that not the entire database is available in the memory and in the case of warm store the used part of the database is in the physical memory. This can be concluded from the fact that in warm store only short queries are run.

30. Question Bogdan: What is the checkpointing which is done? Answer: “Checkpointing is the process of associating a resource with one or more registry keys so that when the resource is moved to a new node, the required keys are propagated to the local registry on the new node.” We think that the checkpointing is used to first build a portion of the tree in the memory and then put the finished (checkpointed) portion onto the disk

31. Demo Ukkonen’s linear time suffix tree algorithm (on-line available at: http://www.i.kyushu-u.ac.jp/~takeda/PM/SuffixTree/STreeDemo.html)