1 / 76

Statistical Modeling: Building a Better Mouse Trap, and others

Statistical Modeling: Building a Better Mouse Trap, and others. Dec 10, 2012 at the University of Hong Kong Stephen Sauchi Lee Associate Professor of Statistics Affiliated Professor of Bioinformatics and Computational Biology

aglaia
Download Presentation

Statistical Modeling: Building a Better Mouse Trap, and others

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Statistical Modeling: Building a Better Mouse Trap, and others Dec 10, 2012 at the University of Hong Kong Stephen Sauchi Lee Associate Professor of Statistics Affiliated Professor of Bioinformatics and Computational Biology Department of StatisticsUniversity of Idaho Moscow, Idaho, USA

  2. Statistical Modeling On 3 projects Building a Better Mouse Trap?The Incremental Utility Behind the Methodology of Risk Assessment Predicting Parkinson Disease Status Demographic Impacts on Social Vulnerability in Norway

  3. Building a Better Mouse Trap?The Incremental Utility Behind the Methodology of Risk Assessment Academy of Criminal Justice Sciences NYC 2012-03-16 Zachary Hamilton, PhD Melanie-Angela Neuilly, PhD Robert Barnoski, PhD Washington State University Pullman, Washington Stephen S. Lee, PhD University of Idaho Moscow, Idaho

  4. Emerging technology of risk assessment • Four generations: • 1) clinical judgment • 2) static predictors • 3) dynamic factors • 4) automated • Regression methods utilized for instrument creation • LSI-R: Logistic Regression • COMPAS: Survival Regression • Recent advancements in prediction rarely utilized for criminal risk assessment • Decision trees • Neural networks • Latent class analysis

  5. Non-linear prediction models • Linear approaches assume equal additive quality for all factors (Steadman, et al., 2000) • Typically neglect interaction effects • Machine-learning mirror diagnostic processes more closely (Steadman, et al., 2000) • Machine-learning approaches most commonly used: • Classification Trees (CT) and other recursive partitioning models (CHAID, CART, ICT, Random Forests, etc.) • Neural Networks (NN)

  6. Classification Trees • Hierarchical question-decision tree model (Breiman, 1984) • The final answer is the result of a series of conditioning answers (If this -> then that, etc.) • Used in diagnostic reasoning • No statistical significance • Random Forests • Inductive statistical learning • Aggregation of hundreds of Classification Trees

  7. Hypothetical recidivism tree

  8. Neural Networks Developed in Artificial Intelligence research Data mining technique for pattern recognition Aim at modeling the lower level brain functions Layered nodes of fact-sets instead of rules, used to train the network Based on the training data, the network “learns” to deduce the right answer to any new piece of information Used in psychiatric diagnostic

  9. Schematic Neural Network Recalculation of weights based on predicted and actual outputs

  10. Previous CT and NN research on recidivism prediction Studies using CT-like analyses, as well as NN tend to make use of smaller samples (≤ 1,500) (except Berk et al., 2009; Palocsay et al., 2000; and Silver et al., 2000) Overall, results are mixed, but those finding significant improvement via CT use lack proper validations (Liu, 2010) Studies using NN show very split results (Liu, 2010)

  11. Gaps in the literature • Overall, very few studies have investigated the utility of CT and NN for predicting recidivism • Previous studies have been limited • In power • To violence prediction • The current study remedy such limitations • Close to ½ million cases • General recidivism as well as possibilities for investigating offense-specific recidivism

  12. Washington State Static Risk Assessment • Previously utilized LSI-R • Found laborious by community corrections officers • Evaluated to be strengthened by increase of static items (Barnowski, 2003) • Created current instrument in 2006 • Factors strongly related to recidivism: demographics, juvenile record, commitments to DOC, felonies, misdemeanors, and violations • Removed dynamic items (interview not required) • Instrument scored from logistic regression - logit weights • Comparable Predictive Validity for WA Sample (WSIPP, 2007) • LSI-R AUC = .66 • WA Static Risk AUC = .74

  13. Analysis Plan • 24 variables included from current risk prediction instruement • 3 year follow-up (release from incarceration) • Any felony recidivism • 2 step creation • Construction sample • All offenders released from prison or jail placed on community supervision from 1986 to 2000 (N = 287,417) • Validation sample • All offenders released from prison or jail placed on community supervision from 2001 to 2002 (N = 71,957) • Compare methods of prediction models • Area under the receiver operating characteristic (AUC) • Values of .500s indicate no predictive accuracy • Where .600s are weak, .700s moderate, and above .800 strong predictive accuracy

  14. Descriptive Statistics (N=359,374) Predictor %/Mean(SD) White (not included in model) 79.7 1. Male 18.7 2. Age At Risk 31.7(10.2) 3. Adult Felonies 2.1(1.9) 4. Juvenile Felony Score 32 5. Juvenile Person Score 6 6. Number of DOC Commitments 2.0(1.7) 7. Homicide/Manslaughter 1 8. Felony Sex 7 9. Felony Violent Property 9 10. Felony Non-Dometic Violence Assault 16 11. Felony Dometic Violence Assault 2 12. Felony Weapon 4 13. Felony Property 85 14. Felony Drug 62 15. Felony Escape 8 16. Misdemenor Non-Dometic Violence Assault 23 17. Misdemenor Dometic Violence Assault 21 18. Misdemenor Sex 3 19. Misdemenor Dometic Violence Other 1 20. Misdemenor Weapon 4 21. Misdemenor Property 52 22. Misdemenor Drug 17 23. Misdemenor Escape 1 24. Misdemenor Alcohol 17 NewFelony (Outcome) 44

  15. Logistic regression method • Extended validation sample of original instrument construction • Strongest model predictors (weights) were: 1) Misd. Property, 2) Juvenile Felony, 3) Misd. Dometic Violence Assault, 4) Misd. Drug, 5)Misd. Sex, 6)Male • Findings comparable to original instrument construction

  16. Radom Forest Model Strongest Model Predictors : 1)Felony Adjudications, 2) Misd. Property, 3)Sentence Length 4)Juvenile Felony 5)Age 6) Felony Property

  17. Model Comparisons

  18. Model Comparisons • Significant differences found • Neural network significantly greater predictive validity than random forest • Neural network significantly greater predictive validity than logistic regression but only construction sample

  19. Incremental Utility of Methodological Advancements Neural networks performed best, followed by logistic regression and random forest ROC differences of methods found to be significant but not universally Preliminary nature of findings are stressed

  20. Limitations • Lack of specificity of outcome measure and sample heterogeneity • Any felony within 3 years • Specialization and taxonomic structures not considered • Unit of analysis is incarceration cycle • Violation of independence assumption for repeat incarcerations • Exclusion of dynamic predictors

  21. Future Findings and Policy Implications • Add dynamic predictors to models • Available since 2008 • Prior/preliminary findings indicate only modest improvement • Examine impact of latent variable methods • 4th potential model • Disentangle heterogeneity • Subgroup analyses based on offense specialties • i.e. drug, violent, sex offender

  22. Predicting Parkinson’s disease status with vocal dysphonia measurements Roxana Hickey Bioinformatics & Computational Biology Statistics 519 Multivariate Statistics Term Project Professor Stephen Lee April 27, 2011

  23. Outline • Background • Parkinson’s disease • Vocal dysphonia • Study dataset • Statistical analyses • Conclusions

  24. Parkinson’s disease • Neurological disorder that leads to shaking and difficulty with walking, movement and coordination1 • Affects >1 million people in North America2 • rapidly increased prevalence after age 603 • No cure, but medication available to alleviate symptoms, especially in early stages4 • early detection key to effective treatment strategies http://www.healthtree.com/articles/parkinsons-disease/causes/

  25. Parkinson’s disease & vocal impairment • ~90% of individuals with Parkinson’s disease have some form of vocal impairment5, 6 • characteristics7 • dysphonia(impaired production of vocal sounds) • dysarthria(problems with normal articulation in speech) • may be one of earliest indicators of onset of illness8 • Tests for vocal impairment9,10 • sustained phonations11, 12(focus of this study) • produce single vowel and hold pitch constant • running speech12 • speak standard sentences that contains representative sample of linguistic units

  26. Measures of assessing vocal dysphonia • Traditional methods11, 12 • pitch (F0, fundamental frequency of vocal oscillation) • absolute sound pressure level (loudness) • jitter (variation in F0 from vocal cycle to vocal cycle) • shimmer(variation in amplitude) • noise-to-harmonics ratio • Novel methods13, 14 • nonlinear dynamical systemstheory and nonlinear time series analysis • recurrence period density entropy • detrended fluctuation analysis

  27. Measures of assessing vocal dysphonia • Measurements differ in robustness14 • uncontrolled variation in acoustic environment • physical condition and characteristics of subject • Therefore, chosen measurement methods should be as robust as possible to this variation • Goal of the study: identify an optimal feature set that is both robust to uncontrolled variation and able to classify patients with Parkinson’s disease based on vocal dysphonic symptoms • Additional advantage: possibility of monitoring patients remotely

  28. http://archive.ics.uci.edu/ml/machine-learning-databases/parkinsons/parkinsons.datahttp://archive.ics.uci.edu/ml/machine-learning-databases/parkinsons/parkinsons.data

  29. Subjects & methods • Subjects • 31 individuals • 8 healthy • 23 with Parkinson’s disease (PD) • average of six sustained vowel phonations recorded from each subject • Total n=195 • Calculation of features via software programs • traditional measures • non-standard measures, including new measure proposed by authors: pitch period entropy

  30. Grouping variable: status =0 (healthy) =1 (PD) Variables Measures of variation in amplitude Measures of ratio of noise to tonal components in voice Nonlinear dynamical complexity measures Single fractal scaling exponent Measures of variation in fundamental frequency Nonlinear measures of fundamental frequency variation MDVP = (Kay Pentax) Multi-Dimensional Voice Program

  31. Statistical analyses • EDA • PCA • MANOVA • Hotelling’s T2 • QDA • Classification tree (with random forest)

  32. 0=healthy 1=PD EDA

  33. 0=healthy 1=PD EDA

  34. 0=healthy 1=PD EDA

  35. 0=healthy 1=PD EDA

  36. EDA 0=healthy 1=PD

  37. PCA

  38. MANOVA • template H0: µhealthy = µParkinson’s

  39. Hotelling’s T2 test • H0: µhealthy = µParkinson’s (p=22) • T-square test statistic = 187.48 • df = 48 + 147 – 2 = 193 • critical 20.05, 22, 193 47 (extrapolated) • Conclusion: reject H0 (=0.05)

  40. QDA • park.qda.cv <- qda(park.g[,2:23], park.g$group, CV=T) • table(Actual=park.g$group, Classified=park.qda.cv$class) • CV error rate: • (13+9)/195=11.28% Classified Actual

  41. Classification tree (CART) • table(Actual=park.g$group, Classified=park.cart.pred) Error rate: (10+6)/195= 8.21% Classified Actual

  42. Random forests park.rf <- randomForest(group~., data=park.g, importance=TRUE, proximity=TRUE) park.rf Call: randomForest(formula = group ~ ., data = park.g, importance = TRUE, proximity = TRUE) Type of random forest: classification Number of trees: 500 No. of variables tried at each split: 4 OOB estimate of error rate: 9.23% Confusion matrix: Cumulative error rates overall healthy PD

  43. Random forests varImpPlot(park.rf)

  44. Random forests

  45. Conclusions • Measurements of vocal dysphonia differ between healthy and PD individuals (MANOVA, Hotelling’s T2) • QDA and classification tree able to separate healthy from PD individuals using a reduced “feature set” • Error rates ~9-12% • PPE, shimmer, average and high fundamental frequency measurements • Little et al. (2009) concluded PPE greatly improved classification performance; cautioned against using traditional methods alone and suggested instead using novel methods such as PPE

  46. Regional Demographic Impacts on Drivers of SocialVulnerability:A localview of Norway Patrick Fitzsimons Supported by National Science Foundation Office of Polar Programs Grant ARC-0909191

  47. Special Thanks to: • Committee: Harley Johansen Ph.D. Tim Frazier Ph.D. Stephen Lee Ph.D. • Family and Friends

  48. Outline • Climate Change developments in the Arctic • Migration Patterns relation to Social Vulnerability • Methods and Results • Creation of index for accessibility • Examining recent migration patterns • Multivariate analysis with social vulnerability drivers • Discussion

  49. Barents & Non-Barents Counties Municipalities

More Related