Presentation Transcript

1. BIT 815: Analysis of Deep Sequencing Data Overview: This course will cover methods for analysis of data from Illumina and Roche/454 high-throughput sequencing, with or without a reference genome sequence, using free and open-source software tools with an emphasis on the command-line Linux computing environment* Lecture Topics: Types of samples and analyses Experimental design and analysis Data formats and conversion tools Alignment, de-novo assembly, and other analyses Computing needs and available resources Annotation Summarizing and visualizing results Labs: Lab sessions meet in a computing lab, and will provide students with hands-on experience in managing and analyzing datasets from Illumina and Roche/454 instruments, covering the same set of topics as the lectures. Example datasets will be available from both platforms, for both DNA and RNA samples; students who have their own datasets may contact the instructor prior to the course to discuss opportunities for analysis of their data during the lab sessions. * see http://www.physics.ubc.ca/mbelab/computer/linux-intro/html/ for an overview
2. Introduction to the course and to each other - background in biology, computing, and sequencing - experiments of interest to participants Course structure - 3 two-hour blocks per week, one theme per week * ~ 45 min lecture/discussion * ~ 70 min lab exercises - some assigned reading - participation in classroom discussion is expected - no exams Course Objective - to teach you how to teach yourself
3. The sequencing rate is growing faster than Moore’s Law Stein (2010) Genome Biology 11:207
4. An alternative perspective from an independent source Doubling time 19.8 months Doubling time 2 months Doubling time 7.3 months
5. Sequence data analysis is changing rapidly - relatively few methods are completely static - much of the software is still under active development - new methods and tools are reported every month - staying on the learning curve is essential Why use Linux for sequencing data analysis? - it is well-suited to the task * preferred development platform for most tools * modular design – thousands of independent programs * however … it’s built for speed, not for comfort
6. Modular design in Linux – a ‘toolbox’ approach Individual components of the Linux operating system are written as separate programs Different programs can have similar functions A Linux “distribution” is a collection of programs that work together as an operating system Users have the power to add new programs, or take away existing programs that are not being used, to optimize system performance
7. A map of the software components of the kernel
8. Why is modularity an advantage? - adding new software is relatively straightforward - the operating system can be continually upgraded - adding tools to the toolbox is easy - your analyses are limited only by your imagination; the tools to carry them out are probably already in place There is always more than one way to do it - some sequence analysis tasks have matured to stability - most have not, and are still changing - ‘best practices’ are also changing, and subject to dispute - staying on the learning curve is essential
9. Key principles from the Eric Raymond book chapter Clarity is better than cleverness. Document everything you do, because you won’t remember what you did, or why Programmer time is more expensive than machine time. Don’t worry about optimizing things unless it is necessary Prototype before polishing – get it working before you optimize it. It is often easiest to start with something very simple, then add complexity and capability in steps Design for simplicity; add complexity only where you must. “Make things as simple as possible, but no simpler” – paraphrased from Albert Einstein
10. Computational thinking – four general principles Decompose a complex problem to simple steps. Linux is based on simple tools that do one thing well; these tools require problems to be framed in simple terms. Look for patterns . Recognizing similarities among different types of problems allows re-use of the same tools in new contexts. Generalize patterns to create abstract versions. A tool is most powerful when it can be applied to a variety of problems that all share common features Combine simple tools into more complex pipelines. Repetitive tasks are what computers are good at – our job is to build the algorithms, or sequences of simple steps, that allow the computer to do those repetitive tasks so we don’t have to.
11. File Globbing Exercises Download FileGlobbing.pdf from the course website, and also get smallfiles.zip if you are not using a BIT laptop (http://www4.ncsu.edu/~rosswhet/BIT815/Spring2013/list.html) Note the complexity of the explanation – way more information than you really wanted Start with the simplest forms, and work up from there Right-click on the bit815 (or smallfiles) folder in Documents, choose “Command Prompt Here” from the drop-down menu. Use ls *, ls *.fq, lssmallread[12].fq, ls small*, and other commands to explore what works and what doesn’t
12. File and Directory Commands The Software Carpentry videos introduced several commands related to directories and files – most if not all of those will work in the Gnu On Windows or Mac Terminal command line Start with creating a new directory – mkdir sandbox Change to the new directory, and create some files there:cd sandboxtouch file1 file2 file3 file4 file5 file6 file7 file8 file9 List those files, using the short version or the long one:lsls –la Note that all files are empty (0 bytes in column 5)
13. File and Directory Commands Create another directory within the sandbox directorymkdir bucket Create a symbolic link (equivalent to a Windows shortcut)ln -s ../smallread1.fq mylink Do another long directory listing and look at the output – what information is available there?ls –la
14. File and Directory Commands drwxrwxr-x 2 ross ross 4096 Mar 1 11:05 bucket -rw-rw-r-- 1 ross ross 0 Mar 1 11:05 file1 -rw-rw-r-- 1 ross ross 0 Mar 1 11:05 file2 -rw-rw-r-- 1 ross ross 0 Mar 1 11:05 file3 -rw-rw-r-- 1 ross ross 0 Mar 1 11:05 file4 -rw-rw-r-- 1 ross ross 0 Mar 1 11:05 file5 -rw-rw-r-- 1 ross ross 0 Mar 1 11:05 file6 -rw-rw-r-- 1 ross ross 0 Mar 1 11:05 file7 -rw-rw-r-- 1 ross ross 0 Mar 1 11:05 file8 -rw-rw-r-- 1 ross ross 0 Mar 1 11:05 file9 lrwxrwxrwx 1 ross ross 16 Mar 1 11:06 mylink -> ../smallread1.fq Use pwd to make sure you are still in the sandbox directory Try removing everything in the directory with rm *what happens?
15. File and Directory Commands Remove the bucket directory using rmdir bucket Be very careful when using file globbing with the rm command, because there is no undelete in Linux When something is deleted, it is gone forever, so be careful – always make sure you know what directory you are in, and which files will be affected by any globbing characters you use.
16. Regular Expression Exercises Download RegularExpressions.pdf from the course website (http://www4.ncsu.edu/~rosswhet/BIT815/Spring2013/list.html) Read through the document – note that (as with file globbing) there are different ways to describe the same pattern Open Notepad++ from the Programs menu in Windows, navigate to the Documents/bit815 folder, and open smallread1.fq Use Cntrl-H to open the search and replace dialog box
17. Regular Expression Exercise Enter the following expression in the Find what: box ^@1:([0-9]{1,3}):([0-9]{2,5}):([0-9]{2,5}):([YN]) Enter the following expression in the Replace with: box @InstrID_FlowcellID_lane1_tile\1_xcoord\2_ycoor\3_pass\4 Click the Find Next button to see if the pattern matches anything in the file If not, make sure you entered it correctly. If so, click Replace to see what happens If it works for one example, click Replace All and scroll through the file to see the results
18. Sequencing technology overview - Two different systems on campus: IlluminaGAIIx, 454 - A similar overall strategy for highly-parallel sequencing - Different approaches taken at virtually every step - These different platforms produce data with different characteristics - Other platforms are available off-campus, but are not a focus of the course Similarities - DNA molecules are fragmented and ligated to adaptors - individual DNA molecules are immobilized on a surface - a series of nucleotide addition reactions are carried out - the nucleotide added is detected after each addition - a data file is produced containing the DNA sequences of many fragments
19. Sequencing technology overview - 454 DNA fragmentation – usually sonication Adaptor oligonucleotide addition Images from www.454.com
20. Sequencing technology overview - 454 A single molecule immobilized on a bead PCR amplification in oil-water emulsion creates ~10 million copies per bead Images from www.454.com
21. Sequencing technology overview - 454 DNA-containing beads deposited in wells “Pyrosequencing” produces light when anyof PicoTiterPlate , along with smaller beads nucleotide is incorporated, so only a single with immobilized enzymes for light nucleotide is provided during a cycle, and production light output is recorded during each cycle
22. Sequencing technology overview - 454 A ‘flowgram’ showing light output from each cycle of base addition one flowgram is produced for each of the ~1 million wells in a PicoTiterPlate TACG ‘key’sequence
23. Sequencing technology overview – Illumina Illumina uses a glass ‘flowcell’, about the size of a microscope slide, with 8 separate ‘lanes’. The GAIIx instrument focuses the laser and light detection system only on one of the two surfaces inside the flowcell; the new HiSeq instrument scans both surfaces and therefore doubles the yield of sequence data per lane. Additional improvements in scanning and increases in cluster density make the difference closer to 4x or 5x more data from a HiSeq.
24. Sequencing technology overview – Illumina Fragment DNA, ligate adaptor oligos Single-stranded DNA binds to flowcell surface
25. Sequencing technology overview – Illumina Surface-bound primers are extended by DNA polymerase across annealed ssDNA molecules,the DNA is denatured back to single strands, and the free ends of immobilized strands annealagain to oligos bound on surface of flowcell. This ‘bridge PCR’ continues until a cluster of~ 1000 molecules is produced on the surface of the flowcell, all descended from the singlemolecule that bound at that site. After PCR, the free ends of all DNA strands are blocked.
26. Sequencing technology overview – Illumina Another perspective of the amplification process, showing the clusters of products
27. Sequencing technology overview – Illumina
28. Sequencing technology overview – Illumina
29. Sequencing technology overview – Illumina Although four different colors are used for the fluorescent nucleotides, only two lasers are used to excite the fluorescence. The fluorescent labels are grouped in pairs - labels on A and C are excited by a red laser, and labels on G and T are excited by a green laser. The software assumes that signal from both lasers will be balanced at each cycle. This means that distinguishing between the A signal and the C signal is more difficult for the instrument than A versus G or A versus T. Base substitution errors are the most common type of sequencing error for Illumina instruments. Cycle 1 Cycle 2 Cycle 3 Cycle 4 Cycle 5 AGCCG TAAGT GCTGA CTTAG
30. Understanding FASTQ formator “what do all these symbols mean?” Instrument ID lane tile X Y barcode read# flowcell Header linessequence quality scores Quality scores are numbers that represent the probability that the given base call is an error. These probabilities are always less than 1, so the value is given as 10 times minus log(10) of the probability For example, an error probability of 0.001 (1x10-3) is represented as a quality score of 30. The numbers are converted into text characters so they occupy less space – a single character is as meaningful as 2 numbers plus a space between adjacent values
31. Understanding FASTQ format Illumina v1.8 header version: @HWI-EAS209:06:FC706VJ:5:58:5894:21141 1:N:ATCACG Instrument /flowcell ID lane tile X Y barcode read# Header lines sequence quality scores Unfortunately, at least four different ways of converting numbers to characters have been used, and header line formats have also changed, so one aspect of data analysis is knowing what you have.
32. Illuminaflowcell geometry (GAIIx) 1 2 3 4 5 6 7 8 A flowcell has 8 lanes, which are physically separated. Each lane is imaged during each cycle of sequencing in multiple separate images, called ‘tiles’, which are not physically separated. Tiles within a GAIIx lane are numbered from 1 to 60 down the length of the lane, then from 61 to 120 back up the other side. 1 120 2 119 59 62 61 60
33. Illuminaflowcell geometry (Hiseq) 1 2 3 4 5 6 7 8 Tiles within a Hiseq lane are numbered using a different system . The first digit denotes which surface (1 = lower, 2 = upper), the second denotes a vertical “swath” (1 = left, 2 = middle, 3 = right), and the last two digits denote a tile within that swath (01 means closest to the outflow end of the lane; 08 or 16 means closest to the inflow end of the lane 1101 1201 1301 1102 1202 1302 1107 1207 1307 1108 1208 1308
34. Command-line Exercises Download SAMformatAndCLtools.pdf from the course website (http://www4.ncsu.edu/~rosswhet/BIT815/Spring2013/list.html) Read the first two pages of the document – don’t worry about the “bitwise flag” information; that is for future reference Go back to the command-line window that you used for the file globbing exercise – right-click on the folder that contains the example Fastq and SAM files, and select “Command Prompt Here” from the drop-down menu
35. Command-line Exercises Download SAMformatAndCLtools.pdf from the course website (http://www4.ncsu.edu/~rosswhet/BIT815/Spring2013/list.html) Read the first two pages of the document – don’t worry about the “bitwise flag” information; that is for future reference Go back to the command-line window that you used for the file globbing exercise – right-click on the folder that contains the example Fastq and SAM files, and select “Command Prompt Here” from the drop-down menu