1 / 58

Igor Pacheco, CLPE (Miami-Dade Police Department)

RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS: ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS. Igor Pacheco, CLPE (Miami-Dade Police Department) Brian Cerchiai, CTPE (Miami-Dade Police Department)

jparas
Download Presentation

Igor Pacheco, CLPE (Miami-Dade Police Department)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS: ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS Igor Pacheco, CLPE (Miami-Dade Police Department) Brian Cerchiai, CTPE (Miami-Dade Police Department) Stephanie Stoiloff, M.S. (Miami-Dade Police Department) Sneh Gulati, Ph.D. (Florida International University) Presented by: Igor Pacheco, CLPE & Brian Cerchiai, CTPE

  2. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS • This project was supported by Award No. 2010-DN-BX-K268 awarded by the National Institute of Justice, Office of Justice Programs, U. S. Department of Justice. • The opinions, findings, and conclusions or recommendations expressed in this publication/program/exhibition are those of the authors and do not necessarily reflect those of the Department of Justice.

  3. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS Background • The 2009 National Academy of Sciences’ report titled, “Strengthening Forensic Science in the United States: A Path Forward” announced thirteen recommendations to address challenges facing the forensic science community. • This study was conducted specifically to address concerns of their third recommendation, “Research is needed to address issues of accuracy, reliability, and validity in the forensic science disciplines.” • MDPD study was modeled after previous study by Glen Langengurg • Langenburg, G. A Performance Study of the ACE-V Process: A Pilot Study to Measure the Accuracy, Precision, Reproducibility, Repeatability, and Biasability of Conclusions Resulting from the ACE-V process. J. For. Ident., 2009, 59 (2).

  4. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS Background • Evaluated the reliability of Latent Fingerprint Examiners using the Analysis, Comparison, Evaluation, and Verification (ACE-V) methodology in latent fingerprint examinations by • Measured the Accuracy, Precision, Reproducibility, Repeatability, and Bias of four categorical opinions: • Identification • Exclusion • Inconclusive • No value

  5. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS Participation • 109 Experienced Latent Print Examiners Participated • Requirements to participate: • 1 of active latent casework experience • uses the ACE-V methodology • enforcement agency (crime laboratory), or like agency, in the United States. This includes all active, retired or contracted latent examiners.

  6. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS Announcement & Questionnaire Participants from 75 Different Law Enforcement Agencies 53 Local Agencies

  7. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS Announcement & Questionnaire Participants from 75 Different Law Enforcement Agencies 16 State Agencies 53 Local Agencies

  8. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS Announcement & Questionnaire Participants from 75 Different Law Enforcement Agencies Federal Agencies 3 16 State Agencies 53 Local Agencies

  9. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS Announcement & Questionnaire Participants from 75 Different Law Enforcement Agencies Federal Agencies 4 N/A 3 16 State Agencies 53 Local Agencies

  10. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS Announcement & Questionnaire Latent Print Examination Experience

  11. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS Announcement & Questionnaire Latent Print Examination Experience

  12. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS Announcement & Questionnaire Latent Print Examination Experience

  13. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS Announcement & Questionnaire Latent Print Examination Experience

  14. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS Announcement & Questionnaire IAI Latent Print Certification Not Certified Certified

  15. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS Research Questions Will Latent Examiners be able to correctly identify or exclude unknown latent impressions from known standards using the ACE methodology? Will Latent Examiners be able to correctly identify or exclude unknown latent impressions from known standards using the ACE-V methodology? Will Latent Examiners be able to reproduce and repeat conclusions from unknown latent impressions to known standards using the ACE and ACE-V methodology? Will Latent Examiners be able to reproduce and repeat conclusions from unknown latent impression to known standards using the ACE-V methodology under high bias conditions?

  16. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS Research Design and Method • Thirteen confidential volunteer sources were identified • 8 Men & 5 Women chosen due to not having their fingerprints or palmprints in any known AFIS database outside of Miami-Dade County. • 13 sets of Tenprint and Palm Print Standards collected • 2,711 known latent impressions were created (fingers and palms) • Collected from non-porous flat and curved surfaces (plastic, tile, metal, and glass) • Processed using black powder and clear tape on white backing cards (scanned and printed as photos) • The latent impressions included various sizes, clarity, and levels of distortion

  17. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS Research Design and Method • Three (3)Certified Latent Print Examiners independently evaluated and compared 320 latent prints to the known standards using a rating scale that was created for this research. • 80 Latent Prints were chosen for the final test sets • Rating Scales for each latent analysis and comparison: (Analysis) Strength of Value of Latent Print (Comparison) Latent in Agreement with Standard = + 2 Difficulty of Comparison

  18. Latent in Agreement with Standard Strength of Value of Latent Print 4.33 Minutiae: 4.33 Minutiae Formations: 0.00 Clarity:0.67 Total: 5.00 0.00 2.00 6.33 Difficulty of Comparison: 5.67 0 7 14 21 Insufficient to Difficult

  19. Latent in Agreement with Standard Strength of Value of Latent Print 6.00 Minutiae: 10.00 Minutiae Formations: 1.00 Clarity:1.33 Total: 12.33 Difficulty of Comparison: 10.5 0.66 2.00 8.66 0 7 14 21 Difficult to Moderate

  20. Latent in Agreement with Standard Strength of Value of Latent Print 14.00 Minutiae: 14.00 Minutiae Formations: 3.33 Clarity:2.00 Total: 19.33 3.33 2.00 19.33 Difficulty of Comparison: 19.33 Moderate to Easy 0 7 14 21

  21. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS Research Design and Method Distribution of Latent Prints for Testing 30% 24 Latent Prints Source was not Present 70% 56 Latent Prints Source was Present

  22. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS Research Design and Method Source Not Present: Strength of Value of Latent Print (24 Latents) 25% 6 Latent Prints Insufficient to Difficult

  23. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS Research Design and Method Source Not Present: Strength of Value of Latent Print (24 Latents) 25% 6 Latent Prints Insufficient to Difficult 37.5% 9 Latent Prints Moderate to Easy

  24. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS Research Design and Method Source Not Present: Strength of Value of Latent Print (24 Latents) 25% 6 Latent Prints Insufficient to Difficult 37.5% 9 Latent Prints Moderate to Easy 37.5% 9 Latent Prints Difficult to Moderate

  25. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS • Research Design and Method • Study was conducted in 3 Phases over a one year period

  26. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS • Research Design and Method • Phase 1 and 2– 80 Latent ACE Trials • For each latent trial, participants were asked to conduct an analysis, comparison and evaluation • Analysis: Rate clarity, identify anatomical source, certainty of orientation, and whether the impression was of value or of no value. • Comparison: Compare to 3 of the 10 sets of standards that were provided. • Evaluation: For each evaluation, the participants were to indicate whether they had made an identification, an exclusion, or an inconclusive decision.

  27. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS Research Design and Method • Phase 1and 2– 80 Latent ACE Trials • Answer Sheets

  28. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS • Phase 1 and 2 ACE Trials – Analysis • Value and No Value Decisions

  29. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS • Phase 1 and 2 ACE Trials – Results • Value and No Value Decisions (5,963) • 2,457 Correct Identifications • 953 Correct Exclusions • 42 Erroneous Identifications • 235 Erroneous Exclusions • 446 Inconclusives when from the same source • 403 Inconclusives when from different sources

  30. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS • Phase 1 and 2 ACE Trials • Error Rates • False Positive Rate • Percentage of the time the participant made an erroneous identification when given the possibility of making any of the three categorical opinions (Identification, Inconclusive, and Exclusion). The accuracy of ACE and ACE-V examinations are reported as an overall participant error rateafter participants made a sufficiency determination that a latent was of “value” for Identification.

  31. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS • Phase 1 and 2 ACE Trials – Results • 42 Erroneous Identifications • 35 Erroneous IDs – Appear to be Clerical Errors • 19 times correct standards but opposite finger or opposite palm was chosen • 8 times correct standards and not opposite finger was chosen • 8 times incorrect standards but same finger or palm was chosen • 4 Erroneous IDs - Don’t appear to be Clerical Errors • 4 times incorrect standard and not opposite finger was chosen • 3 Erroneous IDs the source was not present • Based on 2 Latent Prints Made by 4 different Examiners

  32. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS Phase 1 and 2 ACE Trials – Results 3 Erroneous IDs the source was not present Latent Print #1 Strength of Value of Latent: 17.00 Correctly Excluded: 31 times Erroneously Identified: 1 time Inconclusive: 8 times

  33. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS Phase 1 and 2 ACE Trials – Results 3 Erroneous IDs the source was not present Latent Print #2 Strength of Value of Latent: 16.00 Correctly Excluded: 33 times Erroneously Identified: 2 times Inconclusive: 5 times

  34. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS • Phase 1 and 2 ACE Trials • Error Rates • False Negative Rate • Percentage of the time the participant made an erroneous exclusion when given the possibility of making any of the three categorical opinions (Identification, Inconclusive, and Exclusion)

  35. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS • Phase 3 ACE-V Trials – Results • (Verification, Bias & Repeatability)

  36. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS • Phase 3 ACE-V Trials – Results • (Verification, Bias & Repeatability) • Phase 3 was comprised of four different groups • Group 1 – ACE-V Trials • Group 2 – ACE-V Trials (High Bias) • Group 3 – ACE-V Trials (Repeatability) • Group 4 – ACE-V Trials (Repeatability, High Bias)

  37. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS Phase 3 ACE-V Trials – Results • ACE-V + Repeatability, Answer Sheet • If your comparison agrees with the identification, please mark the box labeled "Agree".

  38. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS • Phase 3 ACE-V Trials – Results • 487 Correct Identifications (agreed with the correct identification) • 13 Correct Exclusions (disagreed with incorrect identification) • 0 Erroneous Identifications (agreed with incorrect identification) • 15 Erroneous Exclusions (disagreed with the correct identification) • 15 Inconclusives when from the same source • 2 Inconclusives when from different sources

  39. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS • Phase 3 ACE-V Trials • Error Rates • False Positive Rate

  40. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS • Phase 3 ACE-V Trials • Error Rates • False Negative Rate

  41. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS Phase 3 ACE-V Trials – Results • ACE-V + Repeatability, Answer Sheet – High Bias

  42. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS • Phase 3 ACE-V Trials (Biased Conditions)– Results • 178 Correct Identifications (agreed with the correct identification) • 78 Correct Exclusions (disagreed with incorrect identification) • 3 Erroneous Identifications (agreed with incorrect identification) • 15 Erroneous Exclusions (disagreed with the correct identification) • 51 Inconclusives when from the same source • 4 Inconclusives when from different sources

  43. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS • Phase 3 ACE-V Trials (Biased Conditions)– Results

  44. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS • Phase 3 ACE-V Trials – Repeatability • To determine if the participants would repeat their own conclusions from comparisons of unknown latent prints to known standards the results of identification decisions, erroneous exclusions, and inconclusive results (where the source was present) from Phase 1 were sent to the same participants in Phase 3. • 27 latent prints latent prints presented to participants for repeatability • 1,311 participant decisions.

  45. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS • Phase 3 ACE-V Trials – Repeatability

  46. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS • Phase 3 ACE-V Trials – Repeatability

  47. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS • Phase 3 ACE-V Trials – Repeatability

  48. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS • Phase 3 ACE-V Trials – Repeatability

  49. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS • Phase 3 ACE-V Trials – Repeatability (Biased Conditions) • To determine if the participants would repeat their own conclusions from comparisons of unknown latent prints to known standards under biased conditions, the results of identification decisions, erroneous exclusions, and inconclusive results (where the source was present) from Phase 1 were sent to the same participants in Phase 3. • 24 latent prints latent prints presented to participants for repeatability under biased conditions • 333 participant decisions

  50. RESEARCH STUDY FOR THE RELIABILITY OF THE ACE-V PROCESS:ACCURACY, PRECISION, REPRODUCIBILITY, AND REPEATABILITY IN LATENT FINGERPRINT EXAMINATIONS • Phase 3 ACE-V Trials – Repeatability (Biased Conditions)

More Related