350 likes | 769 Views
Practical Guide to Significantly Improve Peptide Identification Sensitivity and Accuracy. Bin Ma, CTO Bioinformatics Solutions Inc. June 5, 2011. The Sensitivity and Accuracy Dilemma. false. true. score. Publication Guideline.
E N D
Practical Guide to Significantly Improve Peptide IdentificationSensitivity and Accuracy Bin Ma, CTO Bioinformatics Solutions Inc. June 5, 2011.
The Sensitivity and Accuracy Dilemma false true score
Publication Guideline • Earlier experiments paid too much attention on sensitivity and not enough on accuracy. • MCP started the guideline in 2004 to ensure accuracy.
People are generally over-optimistic about how reliable their results are.– ABRF iPRG 2011. “ ” 1% Estimated FDR upper bound Estimated FDR lower bound 30 out of 45 submissions have FDR much higher than the required 1% iPRG/ABRF 2011 Study
PEAKS Achieved both Sensitivity and Accuracy 1% PEAKS PEAKS More peptides in submission
Outline • FDR – pitfalls and solutions • De novo sequencing assisted database search • Three essential examinations to ensure result quality.
FDR Estimation Protein DB Identified Peptides target Search Engine # false target hits # decoy hits decoy
Pitfall 1 – Multiple Round Search more targets than decoys Round 1. Fast Search Round 2. More Sensitive Search # false target hits # decoy hits • Craig and Beavis 2004. Bioinformatics20, 1466–67. FDR underestimation. • Evertt et al. 2010. J Proteome Res. 9, 700-707. • Bern and Kil 2011, J Proteome Res. 10, 2123-27.
Our Solution: Decoy Fusion Equal targets and decoys Decoy sequence append to each target protein. Fast Search More Sensitive Search # false target hits # decoy hits PEAKS DB paper. Submitted.
Pitfall 2 – Mix Protein and Peptide ID A weak hit is “saved” due to the bonus. So is this weak false hit. Idea: Peptides on a multi-hit protein get a bonus on their scores to increase sensitivity. Pitfall target false hit More multi-hit proteins from target DB more false hits are “saved” from target DB FDR underestimation. decoy hit
Our Solution: Decoy Fusion Weak false hits are “saved” with approx. equal probabilities in target and decoy. Get the sensitivity, but still estimate the FDR correctly.
Pitfall 3 – Machine Learning with Decoy Idea: Re-train the coefficients of scoring function for every search after knowing the decoy hits. Pitfall: Risk of over-fit. Machine learning experts only. Adjust scoring function to remove decoy hits after search. Search target false hits decoy hits Fewer target false hits are removed FDR underestimation
Solutions • Don’t use it. • Judges cannot be players. • Only use for very large dataset. • Train coefficients and reuse; don’t re-train for every search. or or
PEAKS 5.3 • PEAKS DB used all these techniques (and many more) to ensure the accuracy while maximizing sensitivity. • Reliable FDR estimation is the top priority in PEAKS DB design.
An Idea to Improve Score Function Idea: If de novo matches a DB peptide, it is likely to be correct. false true score
De Novo Assisted DB Search # matched amino acids between de novo & DB search x+4y best separation line DB Search Score
Including de novo matching as a feature gives the score function a better discriminative power. false after before true score This is just one example of many other new features in PEAKS 5.3 for improving score function.
… far better than what I could ever squeeze out of my data – Stefano Gotta, Siena Biotech “ ”
PEAKS DB Workflow All Spectra De novo both helps to improve DB search, and reports novel peptides. DB search De Novo No Found? Yes DB peptides De novo only
Don’t Trust Software Blindly! • Google “Don’t trust software blindly” returned 5,140,000 results. • As you quality control your experiments, quality control the software’s results too.
Essential Examination 1 #decoy #target in low score region Low #decoy in high score region
Essential Examination 2 Precursor error start to scatter below threshold High scoring peptides should have low precursor error.
Essential Examination 3 • Spectrum annotation around score threshold.
Take Home Message • Another year of dedicated work on PEAKS. • Ensured accuracy; maximized sensitivity. • Do the three essential examinations. • They are simple … at least in PEAKS.
“a big step forward” – Christian Schmelzer, Martin Luther University Enjoy! http://www.bioinfor.com/peaks-download-a-pricing