1 / 82

Verification of Performance Specifications

Verification of Performance Specifications. An Advanced View of Method Validation V ersion 5.0, August 2012. Objectives. Identify test classifications Define what each validation experiment details for testing methods

sheera
Download Presentation

Verification of Performance Specifications

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Verification of Performance Specifications An Advanced View of Method Validation Version 5.0, August 2012

  2. Objectives • Identify test classifications • Define what each validation experiment details for testing methods • Discuss what is recommended to perform each of the validation experiments for testing methods • Recognize how to evaluate data obtained from each of the validation experiments

  3. Pre-Assessment Question #1 A rapid Human Immunodeficiency Virus (HIV) test would likely be classified as a: • High complexity, modified assay • Moderate complexity, unmodified assay • Food and Drug Administration (FDA)-approved, modified assay • Waived, FDA-approved, unmodified assay

  4. Pre-Assessment Question #2 The precision of a test method gives information related to the method’s: • Systematic error • Comparison of results to a reference method • Reproducibility • Likelihood of being affected by hemolysis, lipemia and icterus • Both A and B

  5. Pre-Assessment Question #3 When transferring reference intervals of 20 specimens used, what is the minimum number that must fall within manufacturer’s reference intervals? • 20 • 18 • 16 • 15

  6. Pre-Assessment Question #4 Which linear regression equation component gives information regarding constant bias? • y • x • m (slope) • b (intercept)

  7. Selecting a Method • Evaluate diagnostic tests • Characteristics of testing methods • References: Technical literature and manufacturer’s information • Select method of analysis • Validate method performance • Implement method • Perform tests with appropriate Quality Control (QC) and External Quality Assurance (EQA)

  8. Method Validation What is method validation? Why must we validate? When should we validate? What should we validate?

  9. Method Validation (cont’d) • Why is validation important? • Division of Acquired Immunodeficiency Syndrome (DAIDS) requirement • How important is it that the results produced by the testing method are reliable? • Shouldn’t the laboratory know the level of performance of an adopted test method?

  10. Tests to Validate Waived Non-waived • Unmodified FDA-approved • Modified and/or Non-FDA-approved

  11. FDA Approval Resources • Vendor • Publications • http://www.fda.gov/MedicalDevices/ProductsandMedicalProcedures/InVitroDiagnostics/LabTest/ucm126079.htm

  12. Skill Check What would you consider to be the complexity, per Clinical Laboratory Improvement Amendments (CLIA), of the glucose assay in the workbook? • Waived • Moderate • High

  13. Skill Check What would you consider to be the complexity of a rapid urine pregnancy assay? • Waived • Moderate • High

  14. Skill Check What would you consider to be the complexity of performing a manual white cell differential using a stained whole blood smear? • Waived • Moderate • High

  15. Method Validation • Before you begin: • Be sure you are familiar with the test method before starting • Know what to expect from the method (package insert, discussions with technical assistance, and field service representatives) • Do not include results outside of stated reportable ranges • Predict your findings; establish limits/evaluation criteria

  16. Terms for Discussion Central Tendency Dispersion

  17. Terms for Discussion (cont’d) Values Run

  18. Error in Test Methods • Some error is expected • Examples • Error must be managed • Understanding • Defining specifications of allowable error • Measurement

  19. Total Error of Testing System Total Allowable Error • CLIA Guidelines per analyte • Other Guidelines Systematic Error Random Error Total Error

  20. Error Assessment Random Error (RE) Systematic Error (SE) Total Error (TE) In one direction, cause results to be high or low In either direction, unpredictable Combined effect

  21. Total Error Considerations • Low End Performance Standards • Recommendations derived from upper portion of reportable range are more difficult to achieve at lower concentrations • Maximum Total Error Allowed • Considered to be 30% by David Rhoads, except for amplification methods

  22. Systematic and Random Errors • Systematic Error • Slope/Proportional error • Intercept/Constant error • Bias • Random Error • Mean • Standard deviation (SD) • Coefficient of variation (CV)

  23. Tools for Use Data-Crunching Tools Statistical calculators, graph paper Spreadsheets with calculations Validation Software (Westgard, Analyze-It, EP Evaluator)

  24. How We Will Work Through This Module • One quantitative test taken through the validation process • One qualitative method taken through the validation process

  25. Reportable Range Precision Accuracy Elements of Validation Reference Intervals Sensitivity Specificity

  26. Precision • Definition: Reproducibility • Gives information related to random error Introduction • 20 samples of same material (typically two levels; e.g., Glucose at 50 and 300 mg/dL) • Standard solutions • Control materials • Pools (short term only) What is needed • Repeat testing over short and long term (one day and 20 days, respectively) How we perform the testing

  27. Precision: How We Evaluate the Data • Mean • Standard deviation (SD) • Coefficient of Variation (CV) Calculate the following: • Short term: 0.25 of allowable total error • Long term: 0.33 of allowable total error What amount of random error is allowable, based on CLIA criteria?

  28. Allowable Total Error Database Link for: • Clinical Laboratory Improvement Amendments (CLIA) • College of American Pathologists (CAP) • Royal College of Pathologists of Australasia (RCPA) • Others http://www.dgrhoads.com/db2004/ae2004.php

  29. Precision: Levey-Jennings (LJ) Charts Values Run

  30. Precision: How We Evaluate the Data • Mean • SD • CV: More commonly used, allows for easier comparison How do we compare to manufacturer’s data?

  31. Precision Example 90 mg/dL Mean of Level 1 Glucose CLIA Total Allowable Error 6 mg/dL or ± 10% Total Allowable Error Level 1 Glucose 0.1 x 90 = 9 mg/dL Random error allowed: 0.25 x total allowable 0.33 x total allowable Long-term precision Short-term precision 0.25 x 9 mg/dL 0.33 x 9 mg/dL 2.25 mg/dL 2.97 mg/dL

  32. Activity • Work with Levey-Jennings graph and data • Work with mean and standard deviation to calculate a coefficient of variation, as well as a mean and a coefficient of variation to calculate a standard deviation • Determine if precision data is acceptable

  33. Accuracy • Definition: How close to the true value • Comparison of methods • Gives information related to systematic error • Potential conflicts on interpretation of results (reference values) Introduction • 40 different specimens • Cover reportable range of method • Quality versus quantity What is needed • Duplicate measurements of each specimen on each method • Minimum of five days, prefer over 20 (since replicate testing is same) How we perform the testing

  34. Accuracy: How We Evaluate the Data Graph the Data: Difference plot Real time Comparison plot Calculate y = mx + b Test method on Y-axis b represents constant error m represents proportional error Reference (comparative) method on X-axis Shows analytical range of data, linearity of response over range and relationship between methods

  35. Visual Inspection for Accuracy (x1, y1) Test Method (x2, y2) Slope = (y2- y1) / (x2- x1) Intercept Reference Method

  36. Accuracy: How We Evaluate the Data • Slope: Usually not significantly different from 1 • Intercept: Not significantly different from 0 • Significant difference with Medical Decision Points

  37. Calculate Appropriate Statistics Slope • Measure of proportional bias • m = (y1-y2)/(x1-x2) or “rise/run” • Slope greater than 1 means the Y (Test) values are generally higher than the X (Comparative) values • Slope of 1.11 means the Y (Test) values are on average 11% higher than the X (Comparative) values

  38. Calculate Appropriate Statistics (cont'd) Intercept of the Line • Measure of constant bias between two methods • Y (Test) value at the point where the line crosses the Y axis • If Y intercept is 12, then all Y (Test) values are at least 12 units higher than the X (Comparative) values

  39. Accuracy What type of bias do you see?

  40. Accuracy (cont’d) Constant Bias Proportional Bias

  41. Skill Check Can a linear regression formula offer predictive value in relation to method comparisons? • Yes • No

  42. Activity • Create graph based on sample set • Determine slope from best-fit line • Determine Y-intercept from best-fit line • Explain the relationship between comparative and test results

  43. Reportable Range / Linearity • Definition: Lowest and highest test results that are reliable • Especially important with two point calibrations • Analytical Measurement Range (AMR) and derived Clinical Reportable Range (CRR) Introduction • Series of samples of known concentrations (e.g., standard solutions, EQA linearity sets) • Series of known dilutions of highly elevated specimen or spiked specimens; EQA specimens • At least four levels (five preferred) What is needed How we perform the testing • CLSI recommends four measurements of each specimen; three are sufficient

  44. Reportable Range:How We Evaluate the Data • Measured values on Y-axis versus • Known or assigned values on X-axis Plot mean values of: Visually inspect, draw best-fit line, estimate reportable range Compare with expected values (typically provided by manufacturer)

  45. Reportable Range Activity

  46. Reportable Range Activity (cont'd)

  47. Reportable Range Activity (cont'd)

  48. AMR vs. CRR Analytical Measurement Range (AMR) Linearity Clinically Reportable Range (CRR) Allows for dilution or other preparatory steps beyond routine

  49. Skill Check If you do not have enough specimen to perform a dilution, upon which reportable range component must you rely? • AMR • CRR • Neither A or B • Both A and B

  50. Linearity Materials Utilizing the marketing materials from the two chemistry linearity kits in your handouts: • Determine which kit would be more appropriate for use with the chemistry assay you chose earlier • Explain your reasoning

More Related