1 / 35

Improving test standardisation and quality: the role of EQA

Improving test standardisation and quality: the role of EQA. Nicola Wolstenholme DGKL Symposium, IFCC-WorldLab, Istanbul 25 th June 2014. Improving test standardisation and quality: the role of EQA Summary:. What is EMQN? What do we do? Why EQA is important in Molecular Genetics?

cerise
Download Presentation

Improving test standardisation and quality: the role of EQA

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Improving test standardisation and quality: the role of EQA Nicola Wolstenholme DGKL Symposium, IFCC-WorldLab, Istanbul 25th June 2014

  2. Improving test standardisation and quality: the role of EQASummary: • What is EMQN? • What do we do? • Why EQA is important in Molecular Genetics? • Types of Errors • Examples of quality improvement • Keeping up with advances

  3. EMQN – network at a glance • International EQA provider • Started in 1997 • (EU F4 funding) • 1 scheme (HD) • Modelled on UKNEQAS schemes • Accredited to ISO 17043 • Over 1300 member labs world wide 4367

  4. EMQN – network at a glance

  5. Collaborations

  6. EMQN structure • One administrative centre (Manchester) • Director: Dr Simon Patton (1x wte) • Deputy Director: Nicola Wolstenholme (1 x wte) • Admin assistants: Sandra Rogers (0.5 x wte), Richard Rose (0.5 x wte) • Management board • Chairman: Dr David Barton (Dublin). • 12 Representatives of all EU national EQA schemes: • UK, Finland, Italy, Ireland, Germany, France, Belgium, Netherlands, Australia • Scheme organisers and assessors • 120+ from EU and non-EU countries • Participants / contacts • >1300 registered laboratories

  7. EQA Schemes 2014 • 2845 participations by 493 Laboratories • 29 Disease-specific Schemes • Y chromosome micro-deletions, BRCA, CAH, CMT, DFNB1 hereditary deafness, DM, DMD/BMD, FRAX, FRDA, FH, HD, FAP, HNPCC, HFE, HRF, MEN2, Monogenic diabetes, OI, PKU, Porphyria, PWS/AS, RB, SHOX, SCAs, SMA, Wilson disease. • 4 Technique-specific • Microarrays, DNA-Sequencing (Sanger), DNA-Sequencing (NextGen), cffDNA (cell free foetal DNA) • 4 Molecular pathology • Lung Cancer (NSCLC), Melanoma, Colorectal Cancer, Panel testing

  8. EMQN schemes • Our EQA schemes are designed to test the whole analytical process from sample receipt to report: • Full Schemes • test the ability to interpret data in the light of clinical information supplied with a referral, and to produce a clear and accurate report. • Technical schemes • focus on laboratory technical expertise • not all laboratories are responsible for generating a fully interpreted clinical report • Can explore fully the quality of raw data e.g. Sanger Sequencing scheme has run since 2002 • New technologies require new approaches e.g. NGS exomes/targeted/panel tests Benchmarking DNA sequence quality Patton et al., Clinical Chemistry. 2006;52:728-736

  9. Why Participate?The Uniqueness of a Genetic Test • Can be carried out at any stage of life • Can be applied for pre-natal or pre-implantation diagnostics. • Post mortem tissue can also be analysed (e.g. SADS/SIDS) • Relevant to healthy people as well as to those showing symptoms of disease (e.g. carrier test/predictive tests). • Can be highly predictive for the future health e.g. a particular mutation may be associated with a particular phenotype/prognosis • May inform important decisions about which therapy to use • May have important implications for the relatives of the person tested(e.g. BRCA1/2 mutations identified during molecular pathology tests to determine sensitivity to PARP inhibitors including Olaparib). • The genotype established by a single laboratory test is usually not repeated and forms a permanent part of the medical record of the patient. test result and its interpretation must be correct

  10. Why Participate:The Consequences of Mistakes • Test Results are Potentially Life Changing • Errors can result in: • Abortion of a healthy foetus or unexpected birth of an affected child • Inappropriate reproductive decisions • Inappropriate decisions regarding prophylactic surgery (e.g. Br/Ov Ca) or disease monitoring (FAP) • Failure to use any/the most effective treatment e.g. targeted cancer therapy or cardiac therapy • Clinical laboratories aim to provide • The right test result, correctly interpreted, given to the right patient in the right time-frame.

  11. Common types of errors • Sample/data handling (sample swaps) • Wrong genotype • False negative (EGFR - Sanger sequencing sensitivity depends on technical quality and analytical skill) • False positive (AZF markers not present) • Quantitative error (incorrect triplet repeat sizing) • Test design and validation (SNPs under primer binding sites, verification of deletions by 2nd method) • Gross interpretative error

  12. Error rates in EQA • Error rate of ~5% annually • 5 out of 100 reports incorrect?! Figure 1: Error rates 2007-2009 in all EMQN’s EQA schemes Figure 2: Genotyping errors and error rates in Huntington disease scheme from 1997 to 2009. No errors were made in 2002, 2003 and 2006.

  13. Genotyping performance • Poorest in new technologies or new test areas • Molecular Pathology (e.g., EGFR) • 8.4% (2011), 3.12% (2012) • Improves with regular EQA participation (“closed EQAs”) • Mean 1-2% (e.g., CF, UKNEQAS, AZF) • more variable with open EQAs – mean 5% (e.g., EMQN) • Most labs report correct genotypes in all schemes

  14. Genotyping Improvement:AZF testing

  15. Cytogenetic European Quality Assessment Genotyping Improvement:Microarray testing C E Q A

  16. Genotyping Improvement:EGFR testing

  17. Improving Genotyping Performance: Test design and validation • Example 1 • 2003 BRCA scheme showed very high error rate (10 false negatives) • Caused by a deletion under one primer binding site in BRCA1 ex11 • national consortium of labs used the same primers. • 2014 the same case was used and in this instance 14 Laboratories (9.93%) failed to identify this mutation • Sanger sequencing (11 labs - 9/11 labs come from the same country – investigation on-going) • NGS (3 labs - probably a bioinformatics issue –discrepancies between labs using identical technology)

  18. Improving Genotyping Performance: Test design and validation Example 2 • 2007 PWAS scheme results showed variability in MLPA kit performance • Contacted manufacturer about problems • Manufacturer collaborated to improve kits • PWAS scheme participants tested new kits

  19. Improving Genotyping Performance: Test Sensitivity and Validation • Example 3 • 2012 EGFR scheme • 3.12% genotyping error rate • 80.5% (170) labs got all 10 samples correct • Significant concern • 6 labs made > 3 genotyping errors each • 1 x NGS ONLY strategy • 5 x Sanger Sequencing ONLY strategy

  20. Results interpretation

  21. Improving Reporting Performance: Recommendations • Guidelines • CMGS, Swiss, EMQN, OECD have had +ve effect • Report structure • Items missing from reports • Date of sample receipt • Referral reasons • Test limitations (sensitivity & specificity) • reference ranges, reference sequence numbers, kit versions etc. • Avoiding terms such as “negative” and “positive” • Mutation nomenclature

  22. Cytogenetic European Quality Assessment Improving Reporting Performance:Reports should be concise 2009 5 pages! C E Q A 2011 2 pages

  23. Improving Reporting Performance: Reports Components • Feedback to labs via EQA performance and scheme reports Sample details Patient details Test performed Clear result stated at DNA and protein level Clear interpretation of result Test sensitivity stated Test specificity stated Tumour assessment stated

  24. Mutation Nomenclature • EMQN strongly recommends that laboratories use the Human Genome Variation Society (HGVS) nomenclature reporting guidelines (www.hgvs.org) • Genetic test reports may be used when testing other family member • reports can cross borders • Are we testing the same mutation? • e.g. CAHscheme in 2008 • 28 nomenclature descriptions from 38 participants! • Year-on-year improvement • “Applying the HGVS nomenclature has reduced the variations of reporting the genotyping results significantly” 2011 scheme report • c.290-13A/C>G • 655A/C>G • g656C>G • g.656a>g • c.293-13C/A>G • CYP21A2*9 • g.659A/C>G • I2Gnt656 • c.97-13A,C>G • IVS2-13A/C.bp656 • IVS2-13a>g • IVS2-13C>G • c.289-13C>G • c103-13A/C>G

  25. Keeping up with Advances in Molecular Genetics • EMQN is continually developing new EQA schemes to meet the needs of it’s members; • Pilot schemes • Not punitive - no deductions/poor participants • Reduced fees to encourage participation • Collaborative development of schemes • Both with participants other partners e.g. CEQAS, NEQAS • Recent developments include: • Molecular pathology schemes • Cell-free foetal DNA (NIPD) • Microarray schemes • Next Generation Sequencing (NGS)

  26. EQA for NGS • 1st Pilot Jan – Oct 2013 • Big demand (>80 labs) but limited to 24 participants • Selection criteria • Establish current practice (on-line survey) • Offering NGS in a diagnostic setting • Workload (> 20 samples tested /annum) • Coverage of major platforms • Genomic DNA as starting material • Provision of lab contact information! • Single genomic DNA sample • Process using normal NGS procedures • Not specified which genes to test • Guidance • Smallest panel test (by number of genes) • Largest gene (acceptable to test the sample for multiple genes)

  27. Genes tested Frequency is indicated by the scale of the gene name relative to the other genes. For example, BRCA2 was tested by 8 laboratories, KDR by one laboratory.

  28. Variants 980 variants detected for 126 genes & 41 of those genes were tested by >1 laboratory Exonic variants (+/- 5bp) were detected by >1 laboratory in 25 genes (73.5%)

  29. Concordance of Variants 80 variants in 25 genes No concordance (n) x 35 Full concordance (n) x 39 Partial concordance (n)x 6 E (EMQN laboratory); U (UK NEQAS laboratory); V (Validation laboratory or Manufacturer).

  30. Challenges Identified in 1st Pilot • Establishing a “Consensus EQA genome” • Workflow Complete Genomics ✓ Life Tech✓ Illumina ✓ Others Genomic DNA Genome, exome, panel, array Consensus EQA genome Variants from Illumina, Compete Genomics, Life Tech Resolution of discordant variants – Sanger sequencing

  31. EQA for NGS 2014 • 2nd Pilot 2014 • Unrestricted registration • Big demand -159 laboratories registered through EMQN & 30 through UKNEQAS

  32. EQA for NGS 2014 • 2nd Pilot 2014 • Practical “wet lab” component • Single genomic DNA sample • Valid for all testing approaches up to exome • Distribution date: 6th May • Results compared to “consensus EQA genome” • Bioinformatics component • Modified(?) exome fastq data • Pipeline agnostic • Distribution date: Sept 2014 • Results for all labs compared • Consensus made available to all labs to further validate pipelines

  33. How do you know your test results are correct? • EQA participations can give you confidence • Systematic errors – may be detected by large schemes • EQA network and genetic testing community can influence manufacturers • Validation issues with new technologies • Standardisation of nomenclature is important • Best Practice Guidance • FINALLY… • Most of labs reported correct results in all schemes • Regular participation does improve the whole testing and reporting process.

  34. More information • www.emqn.org • EMQN OFFICE (support@emqn.org) • EMQN, c/o Genetic Medicine,6th Floor, St Mary’s Hospital,Oxford Road, Manchester M13 9WL, United Kingdom

  35. Acknowledgements • Sandi Deans • Ros Hasting • Sarah Berwouts • Simon Patton • Scheme organisers and assessors

More Related