Estimating error rates of firearm identifications using the CMC method

1. Estimating error rates of firearm identifications using the CMC method T.V. Vorburger (tvtv@nist.gov), J.A. Soons, J. Song, X.A. Zheng, J. Yen, N.F. Zhang, D.B. Ott National Institute of Standards and Technology, Gaithersburg, MD 20899, USA 2018 Impression, Pattern and Trace Evidence Symposium Arlington, VA January 25, 2018

2. Contents • Motivation • Breech Face Impression Topography and the Congruent Matching Cells (CMC) Method • Demonstration of Concept for Matching Breech Face Impressions: • Small Population • 3D Acquisition • CMC Distributions • Model Distributions • Error Rate Calculation for Identifications • Projection: Scaling to Real Case Work and Large Populations

3. Acknowledgements: The funding was provided by the Special Programs Office (SPO) of the National Institute of Standards and Technology (NIST). Thank you to Robert Thompson for guidance and expertise during the course of the work. Certain commercial equipment, instruments, or materials (or suppliers, or software, ...) are identified in this paper to foster understanding. Such identification does not imply recommendation or endorsement by the National Institute of Standards and Technology, nor does it imply that the materials or equipment identified are necessarily the best available for the purpose. Th

4. Motivation SUPREME COURT OF THE UNITED STATES Syllabus DAUBERT et ux., individually and as guardians and litem for DAUBERT, et al. v. MERRELL DOW PHARMACEUTICALS, INC. certiorari to the united states court of appeals for the ninth circuit No. 92-102. Argued March 30, 1993 -- Decided June 28, 1993 ....(c) Faced with a proffer of expert scientific testimony under Rule 702, the trial judge, pursuant to Rule 104(a), must make a preliminary assessment of whether the testimony's underlying reasoning or methodology is scientifically valid and properly can be applied to the facts at issue. Many considerations will bear on the inquiry, including whether the theory or technique in question can be (and has been) tested, whether it has been subjected to peer review and publication, its known or potential error rate, and the existence and maintenance of standards controlling its operation, and whether it has attracted widespread acceptance within a relevant scientific community. The inquiry is a flexible one, and its focus must be solely on principles and methodology, not on the conclusions that they generate… Cornell University Law School, Legal Information Institute, https://www.law.cornell.edu/supct/html/92-102.ZS.html (accessed Dec. 12, 2017)

5. Contents • Motivation • Breech Face Impression Topography and the Congruent Matching Cells (CMC) Method • Demonstration of Concept for Matching Breech Face Impressions: • Small Population • 3D Acquisition • CMC Distributions • Model Distributions • Error Rate Calculation for Identifications • Projection: Scaling to Real Case Work and Large Populations

6. Data for (3D) Topography of Cartridge Case Breech Face Impressions Ejector mark Firing pin impression Cartridge case regions of interest Breech face impression

7. Disk Scanning Confocal Microscopy –Senses surface height by rejecting out-of-focuslight www. nanofocus-ag.com/de/html/3dmicro.html

8. Congruent Matching Cells (CMC) Method(J.F. Song et al.) • A hybrid of feature-based and area-based methods to quantify similarity • Applied here to a pair of topography Images of breech face impressions on cartridge cases • Cells A5 and B5, for example have high similarity (i.e. high CCF), and fit into a congruent pattern of such cells on both surfaces • Altogether there are nine CMCs on the surfaces here.

9. Congruent Matching Cells (CMC): Two Examples of Results

10. Contents • Motivation • Breech Face Impression Topography and the Congruent Matching Cells (CMC) Method • Demonstration of Concept for Matching Breech Face Impressions: • Small Population • 3D Acquisition • CMC Distributions • Model Distributions • Error Rate Calculation for Identifications • Projection: Scaling to Real Case Work and Large Populations

11. Two Demonstrations with Populations of Breech Face Impressions • Fabricated by T.G. Fadul et al., see AFTE J., 45, 376 (2013)-40 cartridge cases from 10 pistol slides-consecutively finished by sandblasting and bead blasting-63 known matching (KM) cartridge case image pairs, 717 known non-matching (KNM) image pairs. • Fabricated by T.J. Weller et al., seeJ. Forensic Sci. 57, 912(2012) -95 cartridge cases from 10 pistol slides -9 of them consecutively finished by sandblasting and bead blasting-370 KM image pairs, 4095 KNM image pairs.

12. CMC result (Weller Set)95 Cartridge Cases, 10 pistol slides, 9 of them consecutively finished by sandblasting and bead blasting -Excellent separation between KNM and KM image pairs. -A second data set from Fadul produced a similar result. Cell size = 0.47 mm x 0.47 mm CCF threshold = 50 % Angle tol. = ± 6 deg. X,Y tol. = ± 20 pixel (62.5 μm) J.F. Song et al. Forensic Science International (in press), http://www.sciencedirect.com/science/article/pii/S0379073817305200

13. Binomial Distribution Model for the CMC KNM Distribution The number of known non-matching cell pairs that qualify as CMC pairs may be modeled as a binomial distribution, similar to modelling coin tosses. Where - N is the total number of evaluated cells in each reference image, ranging from 28 – 42. - is the probability for a non-matching cell pair to qualify as a CMC cell pair. ≈ 0.0011 – for the Weller set.

14. CMC Result (Weller Set) with Numerical Theoretical Model Distributions Binomial Dist. Model for KNM Data Beta Binomial Dist. Model for KM Data

15. Illustration of False Positive Error rate Probability (density) Identification Same-tool (KM) comparisons Different-tool (KNM) comparisons Measured CMC Value (h) Score Cumulative false positive error rate

16. False Positive Error Rate (Weller Set) Identification Binomial Dist. Model for KNM Data Integrated false positive error rate for CMC values > 21 = 1.29x10-56.

17. The Error Rate Applies to a Specific Data Set and Scenario • With 95 cartridge cases from 10 firearms. • Population of image pairs = 370 + 4095 = 4465 • With CMC KNM and KM distributions pre-characterized. • If CMC = 21 for a random selected pair, the probability of non-matching images yielding that result or higher ≈ 10-56. Pretty good odds! But could CMC be applied to a real case work scenario?

18. Contents • Motivation • Breech Face Impression Topography and the Congruent Matching Cells (CMC) Method • Demonstration of Concept for Matching Breech Face Impressions: • Small Population • 3D Acquisition • CMC Distributions • Model Distributions • Error Rate Calculation for Identifications • Projection: Scaling to Real Case Work and Large Populations

19. Samples for Other Data Sets:-No Overlap of KNM Data into the peaks of the KM Data -KNM Distribution is stable 705 Pairs 705 Pairs 738 Pairs 681 Pairs 658 Pairs

20. Conjectures about Possible Application to Real Case Work • Calculate Estimates of False Positive (Identification) Error Rates, • Rather than Likelihood Ratios, • Because the KNM distributions appear stable, and • False Identification Error Rates are especially important to the courts.

21. Hypothetical Example: -A cartridge case is recovered from a crime scene. -A firearm from a suspect is test fired several times. -Topography images are taken, and the CMC method is applied to compare them. -The CMC results are clustered around 32. -Statistical tests indicate that the results are consistent with a single population and consistent with a single firearm.

22. Hypothetical Example (continued): -The topography images of the crime scene and suspect cartridge cases are compared with topography images, housed in a database, from 100 different 9mm firearms manufactured by the same method as the suspect firearm. -The results (red) are consistent with a binomial distribution fitted to a population of CMC values for pairs of non-matching firearms. 180 18 2

23. Hypothetical Example (continued): -One might then estimate the fraction of non-matching image pairs that yield CMC values ≥32 from binomial distributions fitted to the data. -Even with 1,000,000 firearms produced by the same method, which would multiply the result by 106, the calculated probability for a false identification of this type of firearm could still be very small (<10-50). 180 18 2

24. Conclusion – Observations: Applications to practical case work will require: • A thorough analysis of all significant sources of uncertainty • A database with counts of firearms manufactured by different methods with different class characteristics • Data similar to for different types of firearms.  • Significant testing will be required to generalize the results above to other types of manufactured firearms.