Christian C. Matchett, F-ABC Georgia Bureau of Investigation presented at

1. 2007/2008 SWGDRUG Updates Uncertainty Document Christian C. Matchett, F-ABC Georgia Bureau of Investigation presented at 2008 CLIC Technical Training Seminar

2. 2007/2008 SWGDRUG Updates Uncertainty Document sponsored by the Drug Enforcement Administration Office of Forensic Sciences and the National Institute of Standards and Technology

3. Why Address Uncertainty? • Forensic community asking for guidance • Accrediting bodies establishing measures of assessing conformity with ISO • Customer requirements • Jurisdictional requirements • Transparency (nothing to hide) • Potential Exculpatory Information

4. Core Committee Actions • Jan 2008 – Draft Uncertainty document approved and released for public comment • Posted on the website since February 2008 • July 2008 – Comments were addressed and final document adopted on July 22, 2008 • The document should be posted on the website by the end of September 2008

5. SWGDRUG Approach • Tailor the recommendations to drug analysis and answer specific uncertainty questions • Ensure dialog on topic is not dictated to us by those outside of the seized drug community • Offer guidance and direction to laboratories and accrediting bodies • Aware that there is a wealth of information that already exists on Uncertainty • No intentions of repeating existing information

6. Purpose of Document • Provide General Guidance • Principles and themes conveyed, not step-by-step instructions • Raise Awareness and Put In Context • Uncertainty is not doubt, it provides assurance that results and conclusions are fit for purpose • Determine Laboratory Responsibility • Consider customer requirements and address uncertainty through training, procedures and documentation • State Benefits • Enhanced confidence through increased understanding of results • Provides Mechanism to express reliability of results

7. Application of Uncertainty • Qualitative • Quantitative • Purity • Weights

8. Qualitative Analysis • Individual methods have limitations and, consequently, uncertainty • Understanding limitations allows analysts to build an appropriate analytical scheme to correctly ID drugs or chemicals • It is expected that an appropriate analytical scheme will result in, effectively, no uncertainty in reported identifications • Relevant limitations should be documented and may need to be in report

9. Qualitative Examples • Use Part III B Methods of Analysis/Drug Identification • IR and microcrystalline test positive for cocaine – effectively NO uncertainty • Limitations • Marquis test positive for methamphetamine – could be methamphetamine or other amphetamines • GC/MS test positive for ephedrine – could be ephedrine or pseudoephedrine

10. Quantitative Analysis • Uncertainty is defined as an estimate attached to a test result which characterizes the range of values within which the true value is asserted to lie • Precise calculations of measurement uncertainty is not always required

11. Seized Drug Numerical Results • Primary numerical values reported in the analysis of seized drugs are • Weight and Purity • Where a value is critical, an appropriate measurement uncertainty determination shall be applied • Weight close to a statutory threshold • Purity of drug close to level which affects sentencing

12. Weight • Uncertainty of a reported value is dependant on the weighing process. Factors include: • Single item versus multiple items (# of weighing operations) • Tare function as separate weighing operation • Extrapolation of population weight from limited sampling of multiple items • Aggregate weighings • Incomplete recovery of material from packaging • Balance selection (e.g., readability, capacity) • Balance operation (e.g., sample placement, environmental conditions)

13. Purity • Sources of uncertainty for purity determination • Sampling plan (e.g., handling of multiple exhibits) • Sample homogeneity • Analytical method • Sample preparation (e.g., size, matrix effects, solubility) • Analytical technique • Reference material (e.g., purity of standard) • Equipment and instrumentation performance (e.g., glassware, pipetters, balances, chromatographs) • Concentration of analyte • Environmental conditions

14. Purity Approaches • Analytical Error • Address both systematic and random error through method validation and quality assurance • Sampling Error • The sample and sampling procedure are often the greatest contributors to measurement uncertainty • Where appropriate, confidence levels (e.g., 95%, 99.7%) shall be selected based on considerations relevant to the analytical context • Record uncertainty information in validation documents and/or case records

15. Uncertainty Budget • All sources of error are separately identified and tabulated • Assign values to each error source using • Empirical data • Validation process, Historical performance data, Control chart data, proficiency tests • Published data • Combination of empirical and published data • Can exclude insignificant sources • Calculate combined and expanded uncertainty using significant values for procedure

16. Non-Budget Approach • Example 1: Use of data from replicate analyses from a validated method with an appropriate sampling plan • Sources of uncertainty that are separately assessed in the budget method are collectively assessed by experimental measurements • Example 2: Use of two standard deviations (2σ) of the test method results from reproducibility data from the validation studies. • Provides an approximation of the measurement uncertainty for non-critical values

17. Reporting • Uncertainty shall be documented but may not need to be reported • Should be reported when result impacts customer • Even if not reported, analysts shall be cognizant of the uncertainty associated with their results

18. When to Report Uncertainty? • Jurisdictional • Prevailing statutory requirement • Relevant governing body (agency) requirements • Customer requests • Potential exculpatory value • Analytical • Qualitative results where limitations are known (e.g., inability to differentiate isomers) • Quantitative measurements where a value is critical (e.g., weight or purity level close to statutory threshold) • Laboratory accreditation requirements

19. Qualitative Reporting Examples • Contains Ephedrine or Pseudoephedrine Item tested: 5.2 grams net • Visual examination determined that the physical characteristics are consistent with a Schedule IV pharmaceutical preparation containing Diazepam. There was no apparent tampering of the dosage unit and no further tests are being conducted. • Contains cocaine (salt form not determined)

20. Quantitative Reporting Examples • Positive for cocaine in the sample tested Net weight of total sample: 5.23 grams ± 0.03 grams Quantitation: 54.7% ± 2.8% • Sample tested positive for cocaine Net weight: 5.23 grams Purity: 54.7% Confidence Range: ± 2.8%* Calculated net weight of drug: 2.8 grams of cocaine *Confidence range refers to a 95% confidence level

21. Training Recommendations • Individuals responsible for determining, evaluating and documenting uncertainty shall be capable of demonstrating familiarity with foundational concepts and principles of estimating uncertainty • General metrology (terminology, symbols, etc.) • Concepts of random and systematic error, accuracy, precision, propagation of error, etc. • Reporting conventions (sig. figs, truncating, rounding) • Basic statistics (i.e., confidence interval, probability, etc) • All analysts shall be capable of explaining their labs procedures for evaluating uncertainty of qualitative and quantitative analyses

22. Supplemental Documents to Follow • Uncertainty budget • Control chart data applications • Demonstration of balance control using standard weight sets • Summing weights from individual exhibits • Expression of sampling uncertainty based on confidence interval using multiple samplings

23. Core Committee • DEA – Nelson Santos (Chair) • Secretariat – Scott Oulton (non-voting) • FBI - Eileen Waninger • ASCLD – Garth Glassburg • NIST - Susan Ballou • ASTM and NEAFS- Jack Mario

24. Core Committee • CAC & NWAFS - Jerry Massetti • MAFS - Richard Paulas • MAAFS - Linda Jackson • SAFS – Christian Matchett • Toxicology – Dr. Robert Powers • Educator – Dr. Chris Tindall • Educator – Dr. Suzanne Bell

25. Core Committee • Canada - Richard Laing • Japan – Mr. Osamu Ohtsuru • United Kingdom - Dr. Sylvia Burns • Australia - Catherine Quinn • Germany - Dr. Udo Zerell • ENFSI - Dr. Michael Bovens • UNODC - Dr. Iphigenia Naidis

26. THANK YOU • Visit us at • www.swgdrug.org