1 / 31

Evaluations and Validations post Carter

Evaluations and Validations post Carter . Keith Perry, Head of Evaluations Evaluations & Standards Laboratory. BSMT, November 2006. Today’s presentation. Carter review of NHS Pathology Ways we currently assess Evaluations post-Carter

quennell
Download Presentation

Evaluations and Validations post Carter

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluations and Validations post Carter Keith Perry, Head of Evaluations Evaluations & Standards Laboratory BSMT, November 2006

  2. Today’s presentation • Carter review of NHS Pathology • Ways we currently assess • Evaluations post-Carter • Evaluations & managed introduction of new technology?

  3. 2002 Need for standardised methods Need for co-ordinated system for evaluation managed intro of new technology

  4. 2006 Need for standardised methods Need for co-ordinated system for evaluation managed intro of new technology

  5. What was found ? Carter review of NHS Pathology • Little evidence of a strategic programme of investment for new & replacement technology • Little automated ‘front end’ & sample tracking • NHS too focussed on cost not benefit • Difficult to cost pathology services • Silos • Fragmented pathology services • Relatively small investment can make a difference • Labs can influence expensive treatments

  6. Recommendations Carter review of NHS Pathology • Managed introduction of new technology • - Benefits realisation - Competition • Appropriate investment • Work equipment harder • Healthcare closer to home • More PoC testing – accreditation & links with labs • Common national framework • Standardisation of test results • CEP – PASA role • Evaluations

  7. CE Marking Ways we currently assess • But does not cover: • Ease of Use • Monitoring of internal QCs • External QA • Use of combinations ie kits/equipment • Comparative data • Evaluation and validation

  8. Evaluations Ways we currently assess • Focus on device performance • Comparative • Independent and unbiased • Provide informative reports • Must be accurate and timely • Relevant to standardisation of methods • Encourage development • Should influence managed introduction of new technology

  9. Murex HIV 1/2 VK84/85 Vironostika HIV Uni-form II Ag/Ab Vitros ECi anti-HIV 1/2 Murex HIV Ag/Ab combination Timing of detection of primary HIV following seroconversion Ortho Ab-capt. ELISA BiotestHIV 1/2 recombinant Access HIV 1/2 NEW AxSYM HIV Ag/Ab Combo Innotest HIV-1/-2 GENSCREEN PLUS HIV Ag-Ab Enzygnost HIV 1/2 plus Biotest Anti-HIV TETRA ELISA Biotest Anti-HIV TETRA ELISA 5 0 10 15 20 days Murex HIV 1.2.0 GE94/95 Murex ICE HIV 1+2 VIDAS HIV DUO *Earliest anti-HIV detection Earliest HIV detection Abbott 3rd gen Plus Enzygnost HIV Integral Clonesystems Detect-HIV v1 Vironostika HIV Uniform II plus O IMx HIV1/2 III plus Wellcozyme Anti-HIV Pasteur Genscreen Version 2 AxSYM HIV 1/2 gO = combined antigen-antibody = immunometric = antiglobulin / indirect = Class specific antibody capture Ways we currently assess

  10. Assessments Ways we currently assess Molecular Assessments with CVN in collaboration with ESL to reassure users that their in-house assays give comparable results • In-house real-time PCR assays • Assays assessed to date (DNA viruses: HSV1 and 2; ADV; CMV) • Viral targets chosen via CVN committee & members • Labs submit SOPs • Panel of 25/target, comprising dilutions of strong or tissue-culture grown material extracted using two methods (9), plus clinical specimens (6) sent out for assessment • Next RSV, VZV, Norovirus, & Enterovirus

  11. Quick Look/See Ways we currently assess • Any new equipment or kit • Molecular extraction machines • New media

  12. Clinical Assessments Ways we currently assess • Hospital based study of IDI-MRSA (direct detection of MRSA in nasal swabs by a rapid molecular method) • bioMérieux Vitek

  13. Ways we currently assess Validation • Validation is the confirmation by examination and provision of objective evidence that the particular requirements for a specific intended use are fulfilled (ISO 17025:2005) Validation is evaluation of the whole process& ability to achieve the correct result (not the kit or reagent in isolation)

  14. VALIDATIONFILE Ways we currently assess

  15. Ways we currently assess

  16. National framework ‘Evaluation’ issues post-Carter www.pasa.nhs.uk/evaluation

  17. Prioritisation ‘Evaluation’ issues post-Carter • The CEP-PASA Prioritisation Board meets 3 times per year to consider proposals • Projects that are innovative, related to government priorities or purchasing exercises will gain higher scores • Next closing date for proposals: 31st January 2007 • Next meeting: 14th March 2007 • website www.pasa.nhs,uk.evaluation/propose_project

  18. Extent of NHS use Anyone can submit proposals Adoption/purchase Evaluation Evaluation project proposals Time Prioritisation Board • Chair • Head of CEP • Business Planning Manager • Head of Commissioning and Delivery • Technical Advisor • PASA / Hubs (2 reps) • NICE (1 rep) • NIIII (1 rep) • HTA (1 rep) • NPSA (1 rep) • MHRA (1 rep) National Evaluations Programme for Microbiology Priorities to be aligned to customer requirements& peer-organisations ‘Evaluation’ issues post-Carter

  19. Scope ‘Evaluation’ issues post-Carter • High rate of introduction of new diagnostic tests & automation • Many subject areas are not covered by evaluations

  20. Scope Virology Identification Serology Molecular Automation Bacteriology Parasitology ‘Evaluation’ issues post-Carter HIV EIAsHCV EIAsHBsAg EIAsPoCTs HSV Real-timePCRs Extraction,Processors,Closed systems SyphilisEIAs ChlamydiaNAATs Kiestraplate automation Culture media ToxoplasmaEIAs Products to reduce HAIs

  21. Evidence & processes ‘Evaluation’ issues post-Carter • Well designed studies • Well defined procedures • Challenging specimen panel • Expert input • Technical quality • Comprehensive quality system • Safeguards / checks in place • Project management

  22. ‘Evaluation’ issues post-Carter

  23. Recognised evaluation protocols ‘Evaluation’ issues post-Carter • e.g. for evaluations of lab automation • Developed in consultation with experts / users • Developed with reference to National Standard Methods • Process for review and authorisation

  24. Awareness ‘Evaluation’ issues post-Carter • National Evaluations Register • Study title • Status (proposal, preparation, technical, report in draft, completed) • Expected Output(evaluation report, look/see, literature review, market review) • Lead centre • Contact name, telephone, email

  25. Timing ‘Evaluation’ issues post-Carter • Evaluating relevance & utility is time consuming • Products are often in use prior to evaluation report • Matching evaluations and tender processes

  26. Mechanism ‘Evaluation’ issues post-Carter • The mechanism for where and how evaluations are done is still insufficiently defined • Central front-door through CEP evaluation centres? • Evaluation networks co-ordinated through CEP centres? • - with expert working groups - professional groups - reference laboratories - clinical user groups

  27. ‘Evaluation’ issues post-Carter Cost benefit • How large is the cost? • How large is the benefit? • Is introducing the test/device cost effective? • Resources • Integral to the work of CEP-PASA • CEP has Health Economist in post

  28. Chlamydia NAATs Evaluations & managed introduction of new technology? • Government priority • To tackle a specific issue (Chlamydia screening programme) • Money made available • Implementation of new technology preceded evaluation (used literature reviews) • Multi-site evaluation took >2 years

  29. Evaluations & managed introduction of new technology? Specimen sourcing Routine Transport Evaluation Urine25mls+ Aliquot,Label & Send RoutineScreenSource Labs (3) SDA SDA Portsmouth 2b 2a IdentifydiscordantsHPA-ESL SDA pouch TMA Liverpool TMA 3b 3a Add to TMA tube Report 1 Retest positives PCR UCLH / Kings PCR 4b 4a HPA-STBRL R-timePCR 5 Data collation,analysis, & report HPA-ESL Aim for 4 day turnaround Chlamydia NAATs evaluation

  30. Challenges • Responding to the need for a wide range of microbiological device evaluations • How to prioritise • Joining device performance with procurement • Earlier assessment and usage of innovative technologies • Improved access to available evaluation resultseg National Evaluations Register • Improved sharing of information • Impact on lab autonomy to choose equipment/kits • Funding of evaluations (eg PASA) • Funding for implementation (capital & revenue)

  31. Thanks to…. • All collaborators / advisors • All in Evaluations & Standards Laboratory, including: • - Valerie Bevan, Director ESL • - Standards, Quality Control Reagents & Quality System Units • - The Evaluations Team: • Katrina Barlow Christine Burgess Johanna Curtis Laura Dean Galit Gonen Fu Li

More Related