1 / 33

Models, Methods, & Measures: Examining the adoption, implementation

Models, Methods, & Measures: Examining the adoption, implementation and sustained use of innovations in the Ohio Mental Health System Phyllis C. Panzano, Ph.D. University of South Florida Decision Support Services, Inc.

acrossen
Download Presentation

Models, Methods, & Measures: Examining the adoption, implementation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Models, Methods, & Measures: Examining the adoption, implementation and sustained use of innovations in the Ohio Mental Health System Phyllis C. Panzano, Ph.D. University of South Florida Decision Support Services, Inc. September 20-21, 2010 Improving Implementation Research Methods for Behavioral and Social Science

  2. IDARP* CLIFFNotes * Innovation Diffusion and Adoption Research Project, Panzano & Roth ODMH; MacArthur Network on Mental Health Policy

  3. ODMH Publications: New Research in Mental Health Volume 15 - 17, IDARP Bulletins 1-7 Journal Articles: Panzano, P.C., and Roth, D. (2006). The decision to adopt evidence-based and other innovative mental health practices: Risky Business? Psychiatric Services. Vol. 57, pp. 1153 – 1161. Seffrin, B, Panzano, PC, & Roth, D (2009). What gets noticed: How barrier and facilitator perceptions relate to the adoption and implementation of innovative mental health practices, Community Mental Health Journal., On-line version currently available through Springer Science and Business Media, LLC 2008. Massatti, RM, Sweeney, HA, Panzano, PC, and Roth, D. (2008). The de-adoption of innovative mental health practices (IMHP): Why organizations choose not to sustain an IMHP; Administration and Policy in Mental Health and Mental Health Services Research, 35:50 – 65. Carstens, C, Panzano, PC, Massatti, RM, Sweeney, HA, and Roth, D. (2007). A naturalistic study of MST dissemination in thirteen Ohio Communities; Journal of Behavioral Health Services and Research. Dissertation: Vaidyanathan, V. (2004). Looking beyond the adoption decision in innovation research: Investigating innovation implementation. The Ohio State University, Columbus, OH. For More Details:

  4. ODMH Research Context • History: Policy (S & R), Strategic Initiative (Hospital Closing), Law (MH Act of 1988) Implementation & Impact Studies • IDARP Catalyst: Funding of CCOEs to facilitate adoption & assimilation of effective & salient practices • IDDT • MST • Family Psycho-education • Cluster-based Planning EvidenceBase • OMAP • MH/Criminal Justice • MH/Schools • Advance Directives Political Salience

  5. ODMH Research Question What factors and processes influence the adoption and assimilation of evidence-based (EBPs) and promising practices by behavioral healthcare organizations*? T1: Adoption T2: Implementation T2: Implementation T3: Implementation T4: Implementation n = 85 n = 50 n = 38 n = 34 2001 2002 2003 2004 2005 2006 2007 2008 2009 ODMH funds 8 CCOEs * User-based model

  6. Extensive Relevant Literature • Core Research Streams: • Innovation development, diffusion, adoption, implementation (e.g., Damanpour; Fixsen et al; Frambach & Schillewaert; Greenhalgh; Hickson et al; Real & Poole; Rogers; Van de Ven; Yin) • Strategic decision making; decision making under risk (e.g., Dutton & Jackson; March & Shapira; Panzano & Billings; Sitkin & Weingart; Staw et al, Tversky & Kahneman) • Health care, innovation & public sector planning (e.g, Meyer & Goes; Nutt; Yin) Prominent Paradigm • Upper Echelon Theory (Hambrick & Mason)

  7. Design, Methods, Measures • Design: Longitudinal (up to 4 rounds), primarily concurrent and prospective; observational field study • Focal entities: 85 Innovation decision processes & 50 implementation efforts involving 4 possible practices • Practices: • All 8 CCOEs volunteered; resources ltd study to 4 • Selection: structured decision process involving OSU faculty, ODMH Director’s and Medical Director’s offices, IDARP researchers • Variability on key innovation-level variables (e.g., evidence, complexity, cost); salience also important

  8. Design, Methods, Measures • Selected 2 team-delivered EBPs (IDDT and MST) as primary; 2 individually-delivered PPs (CBP and OMAP) as secondary • Recruitment of Sites for IDARP • Methods: interviews (structured; process reconstruction), surveys (organization & CCOE), archival data • Key informants: Top decision-makers (CEO, CCO, CFO), implementation managers, primary CCOE liaisons

  9. IDARP: Guiding Models and Key Sources for Measures

  10. 1. Adoption Decision: 1 Model • Card Shark Model • DVs: Adopt/not Adopt; Decision Stage (e.g., Yin) 2. Implementation: 4 Models • Launch; Russian Doll, Dilbert; and Glove Models • DVs: • Continued Use vs De-adoption • Implementation Effectiveness (e.g., fidelity, commitment) • Innovation Effectiveness (e.g., satisfaction, outcomes) • Decision stage: extent assimilated; plans to persist

  11. Model 1: The Adoption Decision The Card Shark Model The decision to adopt depends on calculated risk; the size of your chip stack does matter! • Decision making under risk (e.g., Prospect theory) • Strategic Issue Diagnosis • Climate and Leadership for Innovation

  12. Under Risk A Decision Perceived Risk of Adopting ANTECEDENTS -.50 DECISION STAGE • IMPLEMENTATION • UNDERWAY • JUST DECIDED TO • ADOPT • STILL CONSIDERING • NEVER WILL ADOPT Capacity to Manage or Absorb Risk .40 .28 Risk-taking Propensity Panzano & Roth (2006) Psychiatric Services

  13. Adoption as Decision Under Risk: Some Key Sources for Measures • Survey Scales • Risk & Antecedents: Sitkin and Pablo; Sitkin and Weingart; Panzano & Billings; Bourgeois; Khandwalla • Expectancies: Dutton & Jackson; Thomas & McDaniel • Innovation Attributes: • Moore & Benbaset; Mathieson and Davies; Venkatesh & Davis; Rogers; Tornatsky & Klein • Climate & Culture: Amabile; Bass; Jung et al, Klein & Sorra: Makri et al; Marsick & Watson; Siegel • Attitudes: Aarons; Chatman & O’Reilly; Dunham • Interview Questions • Decision Stage: Nutt, Meyer & Goes; Yin

  14. Model 2: Implementation Success The Launch Model Initial conditions … prior to and at takeoff… have important impacts on the course of events. • Organizational change • Implementation strategy • Planning Process frameworks

  15. Model 2: Factors from earlier stages impact success Decision Success INITIATION IMPLEMENTATION Time 2 Time 1 Time 1 Time

  16. Initiation-Phase Effects SUCCESS Expected Benefits +++ Relative Advantage +++ Results Demonstrability +++ Trust in CCOE (purveyor) +++ * * Assimilation scale; Global positive outcome scale

  17. Decision-Phase Effects SUC C ESS Objective decision +++ Information access +++ Internal influence +++ Commitment +++ * * Assimilation scale; Global positive outcome scale

  18. Model 3: Implementation Success The Russian Doll Model Surrounding conditions and circumstances influence implementation success. • The Meso Paradigm • Levels Issues in Organizational Research • Social Ecology Theory

  19. Level 5: Environment Level 4: Inter-organizational Level 3:Adopting organization Level 2: Project level Level 1: Innovation level • Dependent Variables: • Implementation effectiveness • Innovation effectiveness

  20. Time 1 Environment IOR – Quality of communication (R2 = .13) Org – Learning culture (R2 = .23) Project – Leadership Commitment (R2 = .38) EBP Time 2 Positive outcomes R2 = .38

  21. Model 4: Implementation Success The Dilbert Model Projects can rise and fall depending on how soundly they’re managed. • Climate for implementation • Project management

  22. Climate for Implementation: • Top management support • Goal Clarity • Dedicated resources • Access to training & TA • Rewards/recognition for implementing • Removal of obstacles • Performance monitoring • Freedom to express doubts Holahan et al; Klein, Conn and Sorra; Vaidyanathan, 2004

  23. CLIMATE AND SUCCESS Time 1 Time1 .75 IMPLEMENTATION EFFECTIVENESS Climate for Implementation .45 INNOVATION EFFECTIVENESS Vaidyanathan, 2004

  24. CLIMATE AND SUCCESS Time 1 Time 2 +++ IMPLEMENTATION EFFECTIVENESS Climate for Implementation +++ NS INNOVATION EFFECTIVENESS Panzano et al, 2006

  25. Model 5: Sustained Use If the Glove Still Fits, Keep-wearing-it-model External and Internal Developments Influence Goodness-of-Fit. • Strategic Issue Diagnosis & Management • Project Management

  26. Sustained Use Model Fit Climate Success Such as… Compatibility Capacity Perceived Risk Use History Continued Use vs De-adoption Degree Assimilated Implementation Effectiveness Innovation Effectiveness Developments Panzano and Roth, 2007

  27. Top 2 Reasons for De-adopting1 • Financial resources • Community & network issues • Staffing • Tx Practice Compatibility • Effectiveness • Purveyor (CCOE) Barriers • Technology integration problems 1 12 matched pairs of sustainers vs deadopter sites from organizational surveys and interviews; Massati, Sweeney, Panzano and Roth, 2008

  28. Fit and climate measures differentiate sustainers from de-adopters • Support from external organizations to continue • Degree of ongoing support from top management & organization as a whole • Compatibility of practice with org values • Positive attitudes about practice among staff • Capacity: Know – how and skill at implementing • Access to TA during implementation • Current & projected resource availability

  29. Sustainer Model: CCOE-based Tentative Revision* Fit Implementation Effectiveness (e.g. fidelity) Sustain/ Assimilate Innovation Effectiveness (e.g., outcomes) Climate T3 and T4 CCOE Surveys; n = 34 projects still underway at T4; Panzano & Knudsen et al, 2010

  30. Implementation Models: Some Key Sources of Measures Interview Protocols and Structured Questions • Yin, 1979 • Hickson et al, 1986 • Nutt, 2004 • Van de Ven et al’s survey from the Minnesota Innovation Studies (2000)

  31. Some Key Sources of Measures 1. Survey Scales (organization and CCOE) • Innovation Attributes: see citations for Adoption Model • Attitudes: see citations for Adoption Model plus Dooley et al; Shore et al • Inter-organizational relationships: Oliver; Ring and Van de Ven, Granner and Phillips; El Ansari • Organizational structure, size, resources: Hall; Kimberly; Sutcliffe • Environmental uncertainty: Sutcliffe; Milliken • Politicality: Thomas, Shankster, Mathieu; Dean and Sharfman

  32. Some Key Sources of Measures • Surveys (cont’d) • Leadership: Bass; Makri et al and Championship: Howell and Higgins • Culture and/or climate: Glisson; Fixsen et al; Holahan et al; Klein and Sorra • Fidelity and reinvention: Dusenbury; Rice and Rogers; Van de Ven at al; practice-specific measures • Implementation outcomes: Hickson; Linton; Nutt; Real and Poole, Van de Ven; Yin; • Archival Measures • ODMH databases (e.g., Medicaid) • Agency association database

  33. Concluding Thoughts • Models • Value • Messages • Design, Methods, Measures • Strengths • Weaknesses • Alternatives

More Related