620 likes | 771 Views
De novo identification of repeat families in large genomes. Alkes L. Price, Neil C. Jones and Pavel A. Pevzner June 28, 2005. What is a repeat family?. A repeat family is a collection of similar sequences which appear many times in a genome.
E N D
De novo identification of repeat families in large genomes Alkes L. Price, Neil C. Jones and Pavel A. Pevzner June 28, 2005
What is a repeat family? A repeat family is a collection of similar sequences which appear many times in a genome. For example, the Alu repeat family has over 1 million approximate occurrences in the human genome: Alu Alu Alu Alu Alu
Identifying repeat families: problem formulation INPUT: Genome containing approximate Alu occurrences Alu Alu Alu Alu Alu OUTPUT: 282bp Alu consensus sequence GGCCGGGCGCGGTGGCTCACG………..GCGAGACTCCGTCTC + consensus sequences of all other repeat families in genome
Identifying repeat families: an easy problem? Alu Alu Alu Alu Alu
Identifying repeat families: an easy problem? Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu
Identifying repeat families: an easy problem? Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu consensus
Identifying repeat families: an easy problem? Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu consensus Difficulties:
Identifying repeat families: an easy problem? Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu consensus • Difficulties: • Regions containing repeat occurrences are not known a priori
Identifying repeat families: an easy problem? Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu consensus • Difficulties: • Regions containing repeat occurrences are not known a priori • Repeat boundaries are not known a priori
Identifying repeat families: an easy problem? Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu Alu consensus • Difficulties: • Regions containing repeat occurrences are not known a priori • Repeat boundaries are not known a priori • Many repeat occurrences appear as partial copies
Identifying repeat families: a difficult problem “The problem of automated repeat sequence family classification is inherently messy and ill-defined and does not appear to be amenable to a clean algorithmic attack.” Bao and Eddy, 2002 In this talk, we present a simple and efficient algorithm for solving this problem.
Why is identifying repeat families important? 1. Repeats are biologically meaningful Repeats are drivers of genome evolution (Kazazian, 2004) which can play a beneficial (rather than parasitic) role (Holmes, 2002). In particular, repeats have been implicated in • Genome rearrangements (Kazazian, 2004) • Drift to new biological function (Kidwell and Lisch, 2001) • Increased rate of evolution under stress (Capy et al, 2000)
Why is identifying repeat families important? 2. Repeat masking • Repeats need to be masked prior to performing most single-species or multi-species analyses. “Every time we compare two species that are closer to each other than either is to humans, we get nearly killed by unmasked repeats.” Webb Miller (personal communication)
Why is identifying repeat families important? • Repeats need to be masked prior to performing most single-species or multi-species analyses. GENE1 GENE2
Why is identifying repeat families important? • If repeat families are known, repeats can be masked using RepeatMasker (http://www.repeatmasker.org). GENE1 GENE2
Why is identifying repeat families important? • If repeat families are known … GENE1 GENE2
Identifying repeat families: manual approaches • For widely studied genomes such as human and mouse, libraries of repeat families have been manually curated: • Repbase Update library (http://www.girinst.org) • RepeatMasker library (http://www.repeatmasker.org)
Identifying repeat families: algorithmic approaches • Many, many new genomes are being assembled. How to identify the repeat families present in these genomes? Clearly, algorithmic approaches are needed.
Identifying repeat families: algorithmic approaches All existing algorithms for de novo identification of repeat families rely on a set of pairwise similarities: • Single-linkage clustering (Agarwal and States, 1994) • REPuter (Kurtz et al., 2000) • RepeatFinder (Volfovsky et al., 2001) • RECON (Bao and Eddy, 2002) • RepeatGluer (Pevzner et al., 2004) • PILER (Edgar and Myers, 2005)
Identifying repeat families: algorithmic approaches Disadvantages of using pairwise similarities: • Computational intractability • human genome: ~106 Alus => ~1012 pairwise alignments • Difficulty defining repeat boundaries • “Local sequence alignments do not usually correspond • to the biological boundaries … Difficulty in defining • element boundaries causes problems in clustering • related elements into families.” • Bao and Eddy, 2002
Identifying repeat families: algorithmic approaches Disadvantages of using pairwise similarities: • Computational intractability • Difficulty defining repeat boundaries Our RepeatScout algorithm uses an efficient method of similarity search which enables a rigorous definition of repeat boundaries.
RepeatScout: the main idea Consider a repeat family with many occurrences in a genome: Equivalently, we have: TAGCACCTTAGGGCGTCTCGCAACGTCTGCCCACGAACGTTAATCAGTAA GATTATCATGAAGCGCTTCGCAACGTCTGCAGCTGTCCAGACCGCTGTCA TATATCCGGTAATCGCCCCGCAACGTCTGCTAACGGGCGTACGGTCGAAT TGACCTGCTCAGGAGCCTTGCAACGTCTGCTCGCGGATGTGTATGCACGC ATCCATGCTCGGTATGAATCCAACGTCTGCTCATGGACATCTCATGACGT CGATCCTCTGAGGCACCTCACAACGTCTGCTCACTGACGCACGGTTGCTG
RepeatScout: the main idea TAGCACCTTAGGGCGTCTCGCAACGTCTGCCCACGAACGTTAATCAGTAA GATTATCATGAAGCGCTTCGCAACGTCTGCAGCTGTCCAGACCGCTGTCA TATATCCGGTAATCGCCCCGCAACGTCTGCTAACGGGCGTACGGTCGAAT TGACCTGCTCAGGAGCCTTGCAACGTCTGCTCGCGGATGTGTATGCACGC ATCCATGCTCGGTATGAATCCAACGTCTGCTCATGGACATCTCATGACGT CGATCCTCTGAGGCACCTCACAACGTCTGCTCACTGACGCACGGTTGCTG Consensus: ?
RepeatScout: the main idea TAGCACCTTAGGGCGTCTCGCAACGTCTGCCCACGAACGTTAATCAGTAA GATTATCATGAAGCGCTTCGCAACGTCTGCAGCTGTCCAGACCGCTGTCA TATATCCGGTAATCGCCCCGCAACGTCTGCTAACGGGCGTACGGTCGAAT TGACCTGCTCAGGAGCCTTGCAACGTCTGCTCGCGGATGTGTATGCACGC ATCCATGCTCGGTATGAATCCAACGTCTGCTCATGGACATCTCATGACGT CGATCCTCTGAGGCACCTCACAACGTCTGCTCACTGACGCACGGTTGCTG Consensus: ?
RepeatScout: the main idea TAGCACCTTAGGGCGTCTCGCAACGTCTGCCCACGAACGTTAATCAGTAA GATTATCATGAAGCGCTTCGCAACGTCTGCAGCTGTCCAGACCGCTGTCA TATATCCGGTAATCGCCCCGCAACGTCTGCTAACGGGCGTACGGTCGAAT TGACCTGCTCAGGAGCCTTGCAACGTCTGCTCGCGGATGTGTATGCACGC ATCCATGCTCGGTATGAATCCAACGTCTGCTCATGGACATCTCATGACGT CGATCCTCTGAGGCACCTCACAACGTCTGCTCACTGACGCACGGTTGCTG Consensus: CAACGTCTGC Idea: greedily extend 1 bp at a time from short l-mer seed
RepeatScout: the main idea TAGCACCTTAGGGCGTCTCGCAACGTCTGCCCACGAACGTTAATCAGTAA GATTATCATGAAGCGCTTCGCAACGTCTGCAGCTGTCCAGACCGCTGTCA TATATCCGGTAATCGCCCCGCAACGTCTGCTAACGGGCGTACGGTCGAAT TGACCTGCTCAGGAGCCTTGCAACGTCTGCTCGCGGATGTGTATGCACGC ATCCATGCTCGGTATGAATCCAACGTCTGCTCATGGACATCTCATGACGT CGATCCTCTGAGGCACCTCACAACGTCTGCTCACTGACGCACGGTTGCTG Consensus: CAACGTCTGCT Idea: greedily extend 1 bp at a time from short l-mer seed
RepeatScout: the main idea TAGCACCTTAGGGCGTCTCGCAACGTCTGCCCACGAACGTTAATCAGTAA GATTATCATGAAGCGCTTCGCAACGTCTGCAGCTGTCCAGACCGCTGTCA TATATCCGGTAATCGCCCCGCAACGTCTGCTAACGGGCGTACGGTCGAAT TGACCTGCTCAGGAGCCTTGCAACGTCTGCTCGCGGATGTGTATGCACGC ATCCATGCTCGGTATGAATCCAACGTCTGCTCATGGACATCTCATGACGT CGATCCTCTGAGGCACCTCACAACGTCTGCTCACTGACGCACGGTTGCTG Consensus: CAACGTCTGCTC Idea: greedily extend 1 bp at a time from short l-mer seed
RepeatScout: the main idea TAGCACCTTAGGGCGTCTCGCAACGTCTGCCCACGAACGTTAATCAGTAA GATTATCATGAAGCGCTTCGCAACGTCTGCAGCTGTCCAGACCGCTGTCA TATATCCGGTAATCGCCCCGCAACGTCTGCTAACGGGCGTACGGTCGAAT TGACCTGCTCAGGAGCCTTGCAACGTCTGCTCGCGGATGTGTATGCACGC ATCCATGCTCGGTATGAATCCAACGTCTGCTCATGGACATCTCATGACGT CGATCCTCTGAGGCACCTCACAACGTCTGCTCACTGACGCACGGTTGCTG Consensus: CAACGTCTGCTCA Idea: greedily extend 1 bp at a time from short l-mer seed Discard a sequence after it stops aligning to consensus
RepeatScout: the main idea TAGCACCTTAGGGCGTCTCGCAACGTCTGCCCACGAACGTTAATCAGTAA GATTATCATGAAGCGCTTCGCAACGTCTGCAGCTGTCCAGACCGCTGTCA TATATCCGGTAATCGCCCCGCAACGTCTGCTAACGGGCGTACGGTCGAAT TGACCTGCTCAGGAGCCTTGCAACGTCTGCTCGCGGATGTGTATGCACGC ATCCATGCTCGGTATGAATCCAACGTCTGCTCATGGACATCTCATGACGT CGATCCTCTGAGGCACCTCACAACGTCTGCTCACTGACGCACGGTTGCTG Consensus: CAACGTCTGCTCAC Idea: greedily extend 1 bp at a time from short l-mer seed Discard a sequence after it stops aligning to consensus
RepeatScout: the main idea TAGCACCTTAGGGCGTCTCGCAACGTCTGCCCACGAACGTTAATCAGTAA GATTATCATGAAGCGCTTCGCAACGTCTGCAGCTGTCCAGACCGCTGTCA TATATCCGGTAATCGCCCCGCAACGTCTGCTAACGGGCGTACGGTCGAAT TGACCTGCTCAGGAGCCTTGCAACGTCTGCTCGCGGATGTGTATGCACGC ATCCATGCTCGGTATGAATCCAACGTCTGCTCATGGACATCTCATGACGT CGATCCTCTGAGGCACCTCACAACGTCTGCTCACTGACGCACGGTTGCTG Consensus: CAACGTCTGCTCACG Idea: greedily extend 1 bp at a time from short l-mer seed Discard a sequence after it stops aligning to consensus
RepeatScout: the main idea TAGCACCTTAGGGCGTCTCGCAACGTCTGCCCACGAACGTTAATCAGTAA GATTATCATGAAGCGCTTCGCAACGTCTGCAGCTGTCCAGACCGCTGTCA TATATCCGGTAATCGCCCCGCAACGTCTGCTAACGGGCGTACGGTCGAAT TGACCTGCTCAGGAGCCTTGCAACGTCTGCTCGCGGATGTGTATGCACGC ATCCATGCTCGGTATGAATCCAACGTCTGCTCATGGACATCTCATGACGT CGATCCTCTGAGGCACCTCACAACGTCTGCTCACTGACGCACGGTTGCTG Consensus: CAACGTCTGCTCACGG Idea: greedily extend 1 bp at a time from short l-mer seed Discard a sequence after it stops aligning to consensus
RepeatScout: the main idea TAGCACCTTAGGGCGTCTCGCAACGTCTGCCCACGAACGTTAATCAGTAA GATTATCATGAAGCGCTTCGCAACGTCTGCAGCTGTCCAGACCGCTGTCA TATATCCGGTAATCGCCCCGCAACGTCTGCTAACGGGCGTACGGTCGAAT TGACCTGCTCAGGAGCCTTGCAACGTCTGCTCGCGGATGTGTATGCACGC ATCCATGCTCGGTATGAATCCAACGTCTGCTCATGGACATCTCATGACGT CGATCCTCTGAGGCACCTCACAACGTCTGCTCACTGACGCACGGTTGCTG Consensus: CAACGTCTGCTCACGGA Idea: greedily extend 1 bp at a time from short l-mer seed Discard a sequence after it stops aligning to consensus
RepeatScout: the main idea TAGCACCTTAGGGCGTCTCGCAACGTCTGCCCACGAACGTTAATCAGTAA GATTATCATGAAGCGCTTCGCAACGTCTGCAGCTGTCCAGACCGCTGTCA TATATCCGGTAATCGCCCCGCAACGTCTGCTAACGGGCGTACGGTCGAAT TGACCTGCTCAGGAGCCTTGCAACGTCTGCTCGCGGATGTGTATGCACGC ATCCATGCTCGGTATGAATCCAACGTCTGCTCATGGACATCTCATGACGT CGATCCTCTGAGGCACCTCACAACGTCTGCTCACTGACGCACGGTTGCTG Consensus: CAACGTCTGCTCACGGAC Idea: greedily extend 1 bp at a time from short l-mer seed Discard a sequence after it stops aligning to consensus
RepeatScout: the main idea TAGCACCTTAGGGCGTCTCGCAACGTCTGCCCACGAACGTTAATCAGTAA GATTATCATGAAGCGCTTCGCAACGTCTGCAGCTGTCCAGACCGCTGTCA TATATCCGGTAATCGCCCCGCAACGTCTGCTAACGGGCGTACGGTCGAAT TGACCTGCTCAGGAGCCTTGCAACGTCTGCTCGCGGATGTGTATGCACGC ATCCATGCTCGGTATGAATCCAACGTCTGCTCATGGACATCTCATGACGT CGATCCTCTGAGGCACCTCACAACGTCTGCTCACTGACGCACGGTTGCTG Consensus: CAACGTCTGCTCACGGACG Idea: greedily extend 1 bp at a time from short l-mer seed Discard a sequence after it stops aligning to consensus
RepeatScout: the main idea TAGCACCTTAGGGCGTCTCGCAACGTCTGCCCACGAACGTTAATCAGTAA GATTATCATGAAGCGCTTCGCAACGTCTGCAGCTGTCCAGACCGCTGTCA TATATCCGGTAATCGCCCCGCAACGTCTGCTAACGGGCGTACGGTCGAAT TGACCTGCTCAGGAGCCTTGCAACGTCTGCTCGCGGATGTGTATGCACGC ATCCATGCTCGGTATGAATCCAACGTCTGCTCATGGACATCTCATGACGT CGATCCTCTGAGGCACCTCACAACGTCTGCTCACTGACGCACGGTTGCTG Consensus: CAACGTCTGCTCACGGACGT Idea: greedily extend 1 bp at a time from short l-mer seed Discard a sequence after it stops aligning to consensus
RepeatScout: the main idea TAGCACCTTAGGGCGTCTCGCAACGTCTGCCCACGAACGTTAATCAGTAA GATTATCATGAAGCGCTTCGCAACGTCTGCAGCTGTCCAGACCGCTGTCA TATATCCGGTAATCGCCCCGCAACGTCTGCTAACGGGCGTACGGTCGAAT TGACCTGCTCAGGAGCCTTGCAACGTCTGCTCGCGGATGTGTATGCACGC ATCCATGCTCGGTATGAATCCAACGTCTGCTCATGGACATCTCATGACGT CGATCCTCTGAGGCACCTCACAACGTCTGCTCACTGACGCACGGTTGCTG Consensus: CAACGTCTGCTCACGGACGT Idea: greedily extend 1 bp at a time from short l-mer seed Discard a sequence after it stops aligning to consensus Stop extending when most sequences no longer align
RepeatScout: the main idea TAGCACCTTAGGGCGTCTCGCAACGTCTGCCCACGAACGTTAATCAGTAA GATTATCATGAAGCGCTTCGCAACGTCTGCAGCTGTCCAGACCGCTGTCA TATATCCGGTAATCGCCCCGCAACGTCTGCTAACGGGCGTACGGTCGAAT TGACCTGCTCAGGAGCCTTGCAACGTCTGCTCGCGGATGTGTATGCACGC ATCCATGCTCGGTATGAATCCAACGTCTGCTCATGGACATCTCATGACGT CGATCCTCTGAGGCACCTCACAACGTCTGCTCACTGACGCACGGTTGCTG Consensus: CAACGTCTGCTCACGGACGTACGGT Idea: greedily extend 1 bp at a time from short l-mer seed Discard a sequence after it stops aligning to consensus Stop extending when most sequences no longer align Note: pairwise alignment is a poor boundary criteria.
RepeatScout: the main idea TAGCACCTTAGGGCGTCTCGCAACGTCTGCCCACGAACGTTAATCAGTAA GATTATCATGAAGCGCTTCGCAACGTCTGCAGCTGTCCAGACCGCTGTCA TATATCCGGTAATCGCCCCGCAACGTCTGCTAACGGGCGTACGGTCGAAT TGACCTGCTCAGGAGCCTTGCAACGTCTGCTCGCGGATGTGTATGCACGC ATCCATGCTCGGTATGAATCCAACGTCTGCTCATGGACATCTCATGACGT CGATCCTCTGAGGCACCTCACAACGTCTGCTCACTGACGCACGGTTGCTG Consensus: AGGCGCCTCGCAACGTCTGCTCACGGACGT Idea: greedily extend 1 bp at a time from short l-mer seed Discard a sequence “after it stops aligning to consensus” Stop extending “when most sequences no longer align” First extend right, then extend left in similar manner
Repeat boundaries: the objective function Let S1, …, Snbe strings containing occurrences of a repeat family which share a short l-mer seed. We define the consensus sequence Q of the repeat family to be the sequence which maximizes A(Q; S1, …, Sn) = ∑ka(Q, Sk) where a(Q, Sk) is a fit-preferred alignment score
Repeat boundaries: the objective function Let S1, …, Snbe strings containing occurrences of a repeat family which share a short l-mer seed. We define the consensus sequence Q of the repeat family to be the sequence which maximizes A(Q; S1, …, Sn) = ∑ka(Q, Sk) – c |Q| where a(Q, Sk) is a fit-preferred alignment score c is a repeat frequency threshold
Repeat boundaries: the objective function • A(Q; S1, …, Sn) = ∑ka(Q, Sk) – c |Q| Optimizing the objective function: • Start with Q = short l-mer seed • Greedily extend Q to the right (left) 1 bp at a time. Stop when + many consecutive iterations fail to improve upon the optimal Q. The optimal Q defines the consensus sequence of the repeat family. This provides a rigorous definition of repeat boundaries.
Repeat boundaries: the objective function TAGCACCTTAGGGCGTCTCGCAACGTCTGCCCACGAACGTTAATCAGTAA GATTATCATGAAGCGCTTCGCAACGTCTGCAGCTGTCCAGACCGCTGTCA TATATCCGGTAATCGCCCCGCAACGTCTGCTAACGGGCGTACGGTCGAAT TGACCTGCTCAGGAGCCTTGCAACGTCTGCTCGCGGATGTGTATGCACGC ATCCATGCTCGGTATGAATCCAACGTCTGCTCATGGACATCTCATGACGT CGATCCTCTGAGGCACCTCACAACGTCTGCTCACTGACGCACGGTTGCTG Consensus: AGGCGCCTCGCAACGTCTGCTCACGGACGT Greedily extend right/left to optimize A(Q, S1, …, Sn)
RepeatScout: finding all repeat families To find all repeat families in a genome, we could apply this procedure to extend all frequent l-mers.
RepeatScout: finding all repeat families To find all repeat families in a genome, we could apply this procedure to extend all frequent l-mers. However, each repeat family spawns a large number of frequent l-mers and could be repeatedly rediscovered.
RepeatScout: finding all repeat families To find all repeat families in a genome, we could apply this procedure to extend all frequent l-mers. However, each repeat family spawns a large number of frequent l-mers and could be repeatedly rediscovered. To address this, we dynamically adjust l-mer frequencies to exclude contributions from repeat families we have already identified.
RepeatScout: postprocessing We discard very short “repeat families” arising from spurious frequent l-mers. We discard repeat families with less than 10 copies. We may further wish to distinguish between • Low-complexity repeat families • Tandem repeat families • Multicopy exon families • Segmental duplication units • Transposon families
Results: the human Alu family Input: Genome containing approximate Alu occurrences Alu Alu Alu Alu Alu Desired Output: 282bp Alu consensus sequence GGCCGGGCGCGGTGGCTCACG………..GCGAGACTCCGTCTC
Results: the human Alu family Input: Genome containing approximate Alu occurrences Alu Alu Alu Alu Alu Desired Output: 282bp Alu consensus sequence GGCCGGGCGCGGTGGCTCACG………..GCGAGACTCCGTCTC RepeatScout Output (on human X chr): 282bp sequence GGCCGGGCGCGGTGGCTCACG………..GCGAGACTCCGTCTC
Results: C. briggsae We benchmarked RepeatScout using the 108Mb C. briggsae genome (Stein et al., 2003), which Stein et al. analyzed using the RECON algorithm (Bao and Eddy, 2002). We ran RepeatMasker (http://www.repeatmasker.org) using either the RECON repeat library or the RepeatScout library as input, and compared the results:
Results: C. briggsae RECONRepeatScout library library 2.0 Mb 23.1 Mb 4.8 Mb