Jieun Kim*, Ph.D., Florida State University Bridget Algee -Hewitt, Ph.D., Stanford Univeristy

1. Analysis of Inter- and Intra-Observer Error Associated with the Use of 3D Laser Scan Data of the Pubic Symphysis Jieun Kim*, Ph.D., Florida State University Bridget Algee-Hewitt, Ph.D., Stanford Univeristy *Presenting Author February 20, 2018 NIJ R&D Forensic Symposium

2. Introduction • The three shape-based, computational methods by Slice and Algee-Hewitt (2015) and Stoyanova et al. (2015;2017): • Use 3D laser scans of the pubic symphysis • Target to minimize subjectivity in age estimation by reducing the effects of observer experience in the age-indicator assessment and methodological bias

3. Introduction • Stoyanova et al. 2015 have shown the improved repeatability of the methods through a single observer error test • The computational methods’ performance and reproducibility have not been quantified at the level of multiple observers with different training background and experience levels • Particularly, there is potential for introducing error in the first two steps of data processing: • Initial scanning of the pubic symphysis • Scan editing at different times by different observers.

4. Introduction • Importance of quantifying observer error associated with these steps • An emerging body of research that utilizes computerized, virtual age indicators and high-dimensional image data (e.g. CT/MRI scans): • Villa et al. 2013; 2015; Boyd et al. 2015; Navega et al. 2017; Lottering 2013; 2014; Chiba et al. 2014; Curate et al. 2013, etc.

5. Four Hypotheses • Hypothesis 1. The same observer will edit a set of raw scans inconsistently when editing is done repeatedly overtime • Hypothesis 2. Scan editing skills will vary between observers based on their training background and/or levels of experience • Hypothesis 3. Edited scans with different margin widths left around the pubic symphyseal face will yield different shape measures and age estimates • Hypothesis 4. Edited scans with the ventrally protruding pubic tubercle will yield inaccurate age estimates for the VC method

6. H3: The Effects of Different Margin Widths around the Symphyseal Face • Region of interest • 2mm vs. 4mm vs. 1cm SB Male Cast Lower Phase V 6

7. H4: The Effects of Editing with/without the Pubic Tubercle A pubic symphysis with a tubercle (semi-circled) extending to the face (SB Female Cast Upper Phase VI) Showing two different ventral curvatures depending on presence/absence of the tubercle (SB Male Cast Upper Phase VI)

8. Materials • 3 replicate scans of the 12 Suchey-Brooks’ male casts taken by a single observer (n=36)

9. Methods • Four observers edited each set of the raw SB scans three times • Observers with various experience levels and training background • Observer 1. Biological anthropologist and developer of the scan editing protocol • Observer 2. Scientific computing scholar and developer of the methods’ algorithms and software, forAge, (http://morphlab.sc.fsu.edu/software/forAge/index.html) • Observer 3. Skeletal biologist with <2 years of experience with scan editing • Observer 4. Forensic practitioner new to scan editing

10. Methods (cont.) • From the edited scans, x,y,z coordinates were retrieved and subjected to a series of the shape analyses • 3 sets of shape measures • Slice-Algee-Hewitt (SAH) scores • Bending energy/thin plate splines (BE/TPS) values • Ventral curvature (VC) values • 5 sets of final age estimates obtained from single variable & multivariate regression analyses

11. H1 & H2: Intra- & Inter-Observer Error Test • Intraclass correlation coefficients (ICC) • Two-way random model • Absolute agreement on single measures • ICC guidelines proposed by Cicchetti 1994

12. H3 & H4: The Effects of Different Editing Conditions on final age estimates • Age estimates vs. Known ages • Paired t-test, alpha=0.05

13. Results: Intra-/Inter-observer Error (H1 & H2) • The resulting ICC values were high and mostly fell within the excellent reliability range (0.75-1.0) • This demonstrates that the raw scans were edited consistently within and between observers and the derived shape measures and age estimates were highly reliable among observers.

14. INTRA-observer error: ICC absolute agreement of within-observer shape measures and age estimates derived from first, second, and third time editing (only showing the ICCs of Observer 1), alpha=0.05

15. INTER-observer error: ICC absolute agreement of between-observer shape measures and age estimates, alpha=0.05

16. BE Value* Comparisons between Observers Note the proximity of the BE values generated for each cast by the four observers (ICC= 0.865) *an average of three trial values was used for each observer

17. Comparisons of Final Age Estimates* Generated from the BE values *an average of three trial values was used for each observer All 4 observers yielded consistent age estimates (ICC= 0.829). The “X” symbol indicates the documented age for each of the 12 SB casts.

18. Results: The Effects of Different Margin Widths on Age Estimates (H3) • The edited scans with 2mm and 4 mm margins did not produce age estimates significantly different from the known ages. • However, 1cm margin produced a significantly different mean for all methods (mean diff. b/t estimated & known ages -14-18yrs, p<0.05), except the VC method (mean diff. -9yrs).

19. Results: The Effects of the Ventral Tubercle on Age Estimates (H4) • Unlike the expectation, the inclusion of the pubic tubercle for the shape analysis did not yield inaccurate age estimates for the VC method • However, it did produce statistically significant mean differences for the SAH-score and TPS/BE methods and the two multivariate regression models (p<0.05).

20. Results: The Effects of the Ventral Tubercle on Age Estimates (H4) • A likely reason for this is the fact that the tubercle is located at a different plane from the symphyseal face, and therefore the VC method algorithms are not significantly influenced by the inclusion/exclusion of the feature (mean diff. b/t estimated & known ages -4yrs). • However, the methods that assess the complexity of the symphysial face (e.g.SAH-score & TPS/BE methods) may interpret the tubercle as an extra feature to account for, and therefore produce inaccurate (younger, mean diff. b/t estimated & known ages -16-17yrs) age estimates. • As expected, the scans edited without the tubercle yielded age estimates that were not significantly different from the documented ages.

21. Discussion & Conclusion • The results show high repeatability of the Slice & Algee-Hewitt and Stoyanova et al.’s computational methods regardless of observer’s level of experience or training background. • This further supports using a 3D laser scanner and scanned images to aid in resolving the issue of subjectivity and in standardizing data collection and analysis protocols between observers and institutions. • Despite the simulated improper editing of the scans with various margin widths remaining, the computational methods were robust enough to self-correct and produce consistent and accurate age estimates. However, 1cm margin seems to be a threshold where the methods start generating incorrect age estimates. • Although the inclusion of the pubic tubercle did not necessarily result in inaccurate age estimates for the VC method, we recommend removing the feature as it may “trick” the BE & SAH-score methods and the two multivariate regression models.

22. Acknowledgements • We thank Dr. Judy M. Suchey for providing the documented chronological ages of the SB male casts. • This project is funded by a National Institute of Justice grant (2015-DN- BX-K010) awarded to the senior authors, Slice and Algee-Hewitt.

23. References • Villa C, Hansen MN, Buckberry J, Cattaneo C, Lynnerup N. Forensic age estimation based on the trabecular bone changes of the pelvic bone using post-mortem CT. Forensic Science International. 2013;233(1):393-402. • Villa C, Buckberry J, Cattaneo C, Lynnerup N. Reliability of suchey‐brooks and buckberry‐chamberlain methods on 3D visualizations from CT and laser scans. American journal of physical anthropology. 2013;151(1):158-63. • Villa C, Buckberry J, Cattaneo C, Frohlich B, Lynnerup N. Quantitative analysis of the morphological changes of the pubic symphyseal face and the auricular surface and implications for age at death estimation. Journal of forensic sciences. 2015;60(3):556-65. • Boyd KL, Villa C, Lynnerup N. The use of CT scans in estimating age at death by examining the extent of ectocranial suture closure. Journal of forensic sciences. 2015;60(2):363-9. • Villa C, Buckberry J, Lynnerup N. Evaluating osteological ageing from digital data. Journal of Anatomy. 2016. doi: 10.1111/joa.12544. • Navega D, Coelho JdO, Cunha E, Curate F. DXAGE: A New Method for Age at Death Estimation Based on Femoral Bone Mineral Density and Artificial Neural Networks. Journal of Forensic Sciences. 2017. doi: 10.1111/1556-4029.13582. • Curate F, Albuquerque A, Cunha EM. Age at death estimation using bone densitometry: testing the Fernandez Castillo and Lopez Ruiz method in two documented skeletal samples from Portugal. Forensic Science International. 2013;226(1):296. e1-. e6. • Lottering N, Macgregor DM, Meredith M, Alston CL, Gregory LS. Evaluation of the suchey-brooks method of age estimation in an Australian subpopulation using computed tomography of the pubic symphyseal surface. Am J Phys Anthropol. 2013;150(3):386-99. doi: 10.1002/ajpa.22213. PubMed PMID: 23283754. • Lottering N, Reynolds MS, MacGregor DM, Meredith M, Gregory LS. Morphometric modelling of ageing in the human pubic symphysis: Sexual dimorphism in an Australian population. Forensic science international. 2014;236:195. e1-. e11. • Chiba F, Makino Y, Motomura A, Inokuchi G, Torimitsu S, Ishii N, et al. Age estimation by quantitative features of pubic symphysis using multidetector computed tomography. International journal of legal medicine. 2014;128(4):667-73. • Kimmerle, EH, Prince DA, and Berg GE. Inter‐Observer Variation in Methodologies Involving the Pubic Symphysis, Sternal Ribs, and Teeth. Journal of Forensic Sciences. 2008;53(3): 594-600. • Shirley, NR and Ramirez Montes, PA Age Estimation in Forensic Anthropology: Quantification of Observer Error in Phase Versus Component-Based Methods. Journal of Forensic Sciences. 2015;60(1): 107–111. doi:10.1111/1556-4029.12617. • Slice DE, Algee-Hewitt BF. Modeling Bone Surface Morphology: A Fully Quantitative Method for Age-at‐Death Estimation Using the Pubic Symphysis. Journal of Forensic Sciences. 2015;60(4):835-43. • Stoyanova D, Algee‐Hewitt BF, Slice DE. An Enhanced Computational Method for Age‐at‐Death Estimation Based on the Pubic Symphysis Using 3D Laser Scans and Thin Plate Splines. American Journal of Physical Anthropology. 2015;158(3):431-40. • Stoyanova D, Algee‐Hewitt BF, Kim J, Slice DE. A Fully Computational Framework for Age-at-Death Estimation from the Adult Skeleton: Surface and Outline Analysis of Three-Dimensional Laser Scans of the Pubic Symphysis. Journal of Forensic Sciences.  2017. doi:10.1111/1556-4029.13439. • Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological assessment. 1994;6(4):284.

24. DetelinaStoyanova, PhD Florida State University Scientific Computing detelinastoyanova@gmail.com Bridget Algee-Hewitt, PhD Stanford University Skeletal Variation, Human Genetics, Computational Biology, Forensics bridgeta@stanford.edu Jieun Kim, PhD Florida State University Biological Anthropology jkim17@fsu.edu Dennis Slice, PhD Florida State University Morphometrics, Scientific Computing dslice@fsu.edu Cristina Figueroa-Soto, MA University of Tennessee Waukesha County ME Office Biological Anthropology cfiguer1@vols.utk.edu