1 / 42

Assessment in Child Welfare: Potential and Limitations of Technologies and Tools

Assessment in Child Welfare: Potential and Limitations of Technologies and Tools. ACWA Sydney, Australia 2010 Aron Shlonsky Associate Professor Factor-Inwentash Chair in Child Welfare University of Toronto Factor-Inwentash Faculty of Social Work. Same World, Two Views.

aysha
Download Presentation

Assessment in Child Welfare: Potential and Limitations of Technologies and Tools

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Assessment in Child Welfare: Potential and Limitations of Technologies and Tools ACWA Sydney, Australia 2010 Aron Shlonsky Associate Professor Factor-Inwentash Chair in Child Welfare University of Toronto Factor-Inwentash Faculty of Social Work

  2. Same World, Two Views

  3. Same World, Inclusive View

  4. Problems in Clinical Decision Making • Large number of “risk factors” • Severe time constraints • Increased information does not improve prediction (Dawes, 1989) • Subjective first impressions • Selected risk factors may not be predictive

  5. Same Person, Two Views?

  6. Decision points in Child Welfare • Report / Screening • investigate • Investigation • Verify • close • Open for ongoing • Ongoing Services (in-home) • Close Case • Out-of-home care • Ongoing Services (out-of-home) • Reunification • LTFC • Guardianship • Adoption

  7. Paths Within the System to Overrepresentation • Reporter • Screening • Entry to Foster Care • Investigation • Ongoing services • Length of Stay in Care • Type of placement • Type of exit

  8. How do you know which way to go?

  9. Safety assessment Safety assessments are intended to accurately identify children who have recently been or are currently being maltreated, or are at risk of imminent harm; and, to determine the nature and type of harm, its severity, and its potential consequences for the child. By definition, these children are already at elevated levels of risk to their health, safety, and well-being Rycus & Hughes, 2003

  10. Risk Assessment “The classic theory of risk assessment is a venerable concept. Regardless of field of application, it always analyzes two factors when attempting to determine potential risk: 1) what is the likelihood that a harmful event will occur, and, 2) if it occurs, what is the potential severity of that harm. Any analysis that asks these two questions with respect to a factor or combination of factors can be called a risk assessment” Rycus & Hughes, 2003

  11. Three Types of Risk Assessment in Child Protective Services • Clinical Judgment • Consensus Risk Assessment • Actuarial Risk Assessment

  12. Actuarial v.Consensus-based:New Investigation Rate Baird & Wagner, 2000

  13. Actuarial v.Consensus-based:New Substantiation Rate Baird & Wagner, 2000

  14. Actuarial v.Consensus-based:New Placement Rate Baird & Wagner, 2000

  15. Two types of risk assessment estimates Prediction:A specific behavioral outcome is declared in advance, e.g., the family will re-abuse their child.Classification:Families are placed in risk classification groups based on their estimated probability of future maltreatment. Human behavior is extremely difficult to predict. Classification recognizes the limitations of risk assessment estimates. Used appropriately, it can help improve worker decisions and help agencies develop more effective case management policies.

  16. Ontario Risk Assessment (ORAM) Tool Properties • Combination Consensus-based (items) and Clinical Decision-making (overall risk) Tool • 22 Items individual items • Five Categories • Each item rated on a 5 point scale • Overall risk rating • Based on clinical judgment • Completed at: • Within 30 days of receipt of the referral • Every 6 months • Case closing

  17. RA Study: Results • Reliability • Poor internal consistency across categories • Caregiver influence minimally consistent (α=0.73) • Four other groupings inconsistent • Poor to moderate Inter-rater reliability • Specific risk items • Overall Risk Score

  18. RA Study: Results • Validity • Predictive validity for Overall Risk • Overall risk score NOT related to future maltreatment • Generally poor predictive capacity for individual items. • Some individual items predicted Overall Risk Score • Caregiver’s Motivation (r=0.44, p<0.01); • Caregivers Expectations of the Child (r=0.40, p<0.01); • Family Identity and Interactions (r=0.37, p<0.01); • Caregiver’s Acceptance of the Child (r=0.35, p<0.01); • Severity of Abuse/Neglect (r=0.34, p<0.01); • Ability to Cope With Stress (r=0.30, p<0.01). • Multivariate modeling of recurrence • Caregiver alcohol or drug use • Child behaviour

  19. California Family Risk Classification by Follow-Up Investigation N = 10.097 18 month post-investigation follow-up (NCCD-CRC)

  20. California Family Risk Classification by Follow-Up Substantiation N = 10.097 18 month post-investigation follow-up (NCCD-CRC)

  21. California Family Risk Classification by Child Placement N = 10.097 18 month post-investigation follow-up (NCCD-CRC)

  22. California Final Family Risk Classification by Follow-Up Injury to a Child N = 2,511 24 month follow-up (NCCD-CRC)

  23. 18 Month Recurrence Performance

  24. The Binary Prediction Conundrum Reclassified California Actuarial Data From Validation Study Follow-up Substantiation Predictive Validity • Overall hit rate Sub No Sub • high or very high 0.35 0.65 • low or moderate 0.12 0.88 • Overall .65 Positive Predictive Value 0.35 • Negative Predictive Value 0.88 • sensitivity 0.69 • specificity 0.64 • Prevalence Rate 0.22

  25. Decision-making in the Context of Uncertainty “…there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns -- the ones we don't know we don't know.” Courtesy of Alan Brookhart

  26. Advantages of Actuarial Risk Assessment • Higher construct validity • Better reliability • Greater reliance of objective scoring criteria • Easier to complete • Use of clinical overrides

  27. What risk assessment does not do • Risk assessment does not assist in case specific clinical decisions • Risk assessment does not inform the worker about anything more than the likelihood of future maltreatment

  28. Decision-making in the Context of Uncertainty "I would not say that the future is necessarily less predictable than the past. I think the past was not predictable when it started."

  29. The role of clinical judgment in risk assessment • Clinical judgment not suspended (Meehl, 1954) • Find the optimal combination of actuarial method and clinical judgment (Holt, 1958). “A risk estimate is like a weather forecast. The forecast may indicate a 60% or higher probability of rain the next day. That forecast may prove to be wrong but since it rained 60% of the time on similar days in the past, it may be a good idea to carry an umbrella” (Gottfredson and Tonry, 1987).

  30. What is Needed at Each Decision Point • Risk Assessment / Prognostic Tool • Diagnostic Tool (if applicable) • Clinical assessment and engagement skills • Effective services • Effective monitoring • Effective feedback to larger administrative data system

  31. Clinical state and circumstances Contextual assessment Contextual assessment Clinical Expertise Client Preferences and actions Research Evidence The Cycle of EBP Appropriate for this client? Actuarial risk assessment Other assessment instruments? Client’s preferred course or at least willing to try? Effective services Haynes, Devereaux, and Guyatt, 2002 Barriers (e.g., cultural conflict)

  32. D NO DETERMINISM Evidence in NOT Deterministic • Evidence-informed • Systematic process • Inclusive • Critical • Flexible

  33. Critical Thinking is KeyEileen Gambrill Eight Critical Thinking Guidelines • Ask Questions: Be willing to wonder. • Define Your Terms: Key to Research • Examine the Evidence • Analyze Assumptions and Biases • Avoid Emotional Reasoning • Don’t Oversimplify • Consider Other Interpretations • Tolerate Uncertainty http://cla.calpoly.edu/~cslem/Invit/1/Chp1c.html

  34. A well Cultivated Critical Thinker • Raises vital questions and problems; • Formulates them clearly and
precisely; • Gathers and assesses relevant information • Uses abstract ideas to
interpret information; • Comes to well-reasoned conclusions and solutions; • Tests them against relevant criteria and standards; • Thinks openmindedly; • Recognizes and assesses assumptions, implications, and practical consequences; • Communicates effectively Elder, 2007

  35. Campbell Collaboration Evolution • Importance of question • Debunk hierarchy • Importance or non-RCT’s • Increasing number of reviews • Equity Group • User Abstracts

  36. What to do? • Teach process of EBP • Invest in good epidemiological research • Fund systematic reviews • Systematic reviews lead to targeted primary studies • Create expertise in EST’s and train to common factors • Learn how to effectively work in teams • Implement and evaluate KTE strategies • Develop and sustain a culture of critical thinking

  37. Discussion

More Related