H. Stephen Leff, Ph.D. Human Services Research Institute

Design of the Seclusion and Restraint Study and The National Registry of Evidence-Based Programs and Practices H. Stephen Leff, Ph.D. Human Services Research Institute Harvard Medical School Department of Psychiatry at the Cambridge Health Alliance Coordinating Center to Reduce and Eliminate the Use of Seclusion and Restraint

LEARNING GOALS • To Review Seclusion and Restraint Study Design To Date • To Increase Understanding of the Origins and Design of The National Registry of Evidence-based Programs and Practices (NREPP) • To Increase Understanding of the Benefits of and Concerns about NREPP • To Increase Our Understanding of NREPP Requirements

Alternatives to Restraint/Seclusion: Evaluation Overview Two key points…. First: COLLABORATION Second: Evaluation of INTERVENTIONS– not sites

Alternatives to Restraint/Seclusion: Evaluation Overview • Four objectives of evaluation: • Insure implementation • Measure effects of best practice components • Develop and test fidelity measures • Support application to NREPP

Alternatives to Restraint/Seclusion: Evaluation Overview • Insuring implementation • Process components (“How closely did implementation match the plan?” etc.) • Outcomes components(“What was the effect of infrastructure development on service capacity and other systemoutcomes?” etc.) • To be captured by fidelity assessment • To assess need for TA

Alternatives to Restraint/Seclusion: Evaluation Overview • Four objectives of evaluation: • Insure implementation • Measure effects of best practice components • Develop and test fidelity measures • Support applications to NREPP

Alternatives to Restraint/Seclusion: Evaluation Overview • Measure effects of best practice components • Multiple interventions with extensive overlap • Lack of standard definitions • Relatively undeveloped evidence base

Alternatives to Restraint/Seclusion: Evaluation Overview Developing and Testing a Fidelity Scale “Fidelity refers to the degree to which a particular program follows a program model… Fidelity measures are tools to assess the adequacy of implementation of program models.” (Bond et al., 2000)

Alternatives to Restraint/Seclusion: Evaluation Overview • Developing and Testing a Fidelity Scale • Effectiveness and fidelity • Defining the intervention • Identifying critical ingredients • Facilitating dissemination • Gary Bond consultation

Alternatives to Restraint/Seclusion: Evaluation Overview • Support application to NREPP • NTAC model • Other models: Support from The Evaluation Center @ HSRI • Importance of comparison sites

National Registry of Effective Programs and Practices (NREPP) A voluntary rating and classification system for substance abuse and mental health prevention and treatment interventions Vision - NREPP becomes a leading national resource for contemporary and reliable information on the scientific basis and practicality of substance abuse and mental health prevention and treatment interventions

NREPP Policy Relevance • Influence SAMHSA discretionary and block grant investments • Serve as a resource for states and communities seeking to implement evidence-based MH and SA prevention and treatment services • Provide an important tool for both public and private purchasers in selection of effective services

NREPP is necessary but not sufficient to promote use of EBPs • Prioritize “Center-driven” technical assistance for implementation of evidence-based services – two key areas: • Implementation of NREPP programs • Science-to-service support to organizations and practitioners • Development of evidence base for new programs • Service to science support to program developers

National Registry of Effective Programs and Practices (NREPP) New web site – Fall 2005 • www.nationalregistry.samhsa.gov • Goals - comprehensive and transparent • Posting of evaluative scores and criteria summaries from scientist reviewers • Posting of utility scores and criteria summaries from stakeholder reviewers • Outcomes searchable database • Self-assessment for candidate programs and links to educational materials on review criteria

National Registry of Effective Programs and Practices (NREPP) Brief history • 1998 – began in Center for Substance Abuse Prevention • 1998-2003 – reviewed and rated over 1,100 substance abuse prevention programs, • information on over 150 Model, Effective, and Promising Programs on web site (www.modelprograms.samhsa.gov) • 2004 –initial expansion of system to include substance abuse treatment, and mental health promotion and treatment programs

The Need for NREPP:Treatments are based on idiosyncratic and ill-informed judgments “A health care delivery system characterized by idiosyncratic and often ill-informed judgments must be restructured according to evidence-based medical practice, regular assessment of the quality of care and accountability. The alternative is a system that makes life and death treatment decisions base on conflicting anecdotes and calculated appeals to emotion.” Millenson, ML. Demanding Medical Excellence: Doctors and Accountability in the Information Age. pg.6 University of Chicago Press, Chicago, 1997

The Need for NREPP:Consumers are confused and concerned “Evidence based practices is a new metaphor to me. I thought evidence was the source of mental health practices. Are you saying it was not? May still not be? On what are practices then based? At no time have I ever heard that people in the mental health profession based their acts upon anything other than evidence. Was that a lie? Is this ‘new’ metaphor also a lie? And how would I tell the difference? “ (Personal Communication, Harold A. Maio, MA)

Medication & Medical Device Development System Behavioral Health Intervention Development System Basic Science: NIH Basic Science: NIMH, NIAAA, NIDA Intervention Development, Testing, Dissemination & Post Dissemination Monitoring: Pharmaceutical & Biotechnology Companies Intervention Development, Testing, Dissemination & Post Dissemination Monitoring:SAMHSA, NIMH, NIDA, NIAAA To a lesser degree, states, counties, local programs, foundation and other funders Evaluation of Evidence: FDA Evaluation of Evidence: SAMHSA, NREPP, to a lesser degree NIMH Consumers, Providers, Purchasers Consumers, Providers, Purchasers

Post Recognition Quality Monitoring Disseminability Studies Multiple & Multi-site Replication Studies Initial Evaluation Studies Pilot Studies: Manuals, Fidelity & Outcome Measures Discovering & Describing Interesting Programs: Basic Research, Clinical Judgment Proposed NREPP Intervention Categories Defined by The Ladder of Evidence in Intervention Science The Evidence Ladder Intervention Science Activity 6. Reliable Intervention 5. Disseminable 4. Effective 3. Conditionally Effective 2. Emerging 1. Program of Interest

Adjustment of Outcome Measures for Alpha Inflation A Priori Identification of Outcome Measures Reliability of Measures Validity of Measures Intervention Fidelity Comparison Fidelity Comparison Condition Assurances to Participants Participant Awareness of Condition Standardized Data Collection Data Collector Bias Selection/Assignment Bias Subject Attrition Missing Data Analysis Meets Data Assumptions A Priori Identification of Analytic Methods Analysis Consistent with Theory Anomalous Findings Replications of Findings NREPP Rating Criteria: Evaluative Criteria

Examples of Psychosocial Interventions with Complex Arrangements and Soft Boundaries that Have Been Shown To Have Substantial Evidence

Alternatives to Restraint/Seclusion: Evaluation Overview . Human Services Research Institute 2336 Massachusetts Ave. Cambridge, MA 02140 www.hsri.org The Evaluation Center @ HSRI www.tecathsri.org

H. Stephen Leff, Ph.D. Human Services Research Institute