Assessing Intervention Fidelity in RCTs: Models, Methods and Modes of Analysis

1. Assessing Intervention Fidelity in RCTs: Models, Methods and Modes of Analysis David S. Cordray & Chris Hulleman Vanderbilt University Presentation for the IES Research Conference Washington, DC June 9, 2009

2. Overview • Fidelity and Achieved Relative Strength: • Definitions, distinctions and illustrations • Conceptual foundation for assessing fidelity in RCTs • Achieved relative strength, a special case in RCTs • Modes of analysis: • Approaches and challenges • Chris Hulleman -- Assessing implementation fidelity and achieved relative strength indices: The single core component case • Questions and discussion

3. Distinguishing Implementation Assessment from the Assessment of Implementation Fidelity • Two ends on a continuum of intervention implementation/fidelity: • A purelydescriptive model: • Answering the question “What transpired as the intervention was put in place (implemented). • Based on a priori intervention model, with explicit expectations about implementation of program components: • Fidelity is the extent to which the realized intervention (tTx) is faithful to the pre-stated intervention model (TTx ) • Infidelity = TTx – tTx • Most implementation fidelity assessments involve descriptive and model-based approaches.

4. Dimensions Intervention Fidelity • Aside from agreement at the extremes, little consensus on what is meant by the term “intervention fidelity”. • Most frequent definitions: • True Fidelity = Adherence or compliance: • Program components are delivered/used/received, as prescribed • With a stated criteria for success or full adherence • The specification of these criteria is relatively rare • Intervention Exposure: • Amount of program content, processes, activities delivered/received by all participants (aka, receipt, responsiveness) • This notion is most prevalent • Intervention Differentiation: • The unique features of the intervention are distinguishable from other programs, including the control condition • A unique application within RCTs

5. Linking Intervention Fidelity Assessment to Contemporary Models of Causality Rubin’s Causal Model: True causal effect of X is (YiTx – YiC) RCT methodology is the best approximation to this true effect In RCTs, the difference between conditions, on average, is the causal effect Fidelity assessment within RCTs entails examining the difference between causal components in the intervention and control conditions. Differencing causal conditions can be characterized as achieved relative strength of the contrast. Achieved Relative Strength (ARS) = tTx – tC ARS is a default index of fidelity

6. Treatment Strength Outcome .45 .40 .35 .30 .25 .20 .15 .10 .05 .00 TTx 100 90 85 80 75 70 65 60 55 50 Infidelity t tx Achieved Relative Strength =.15 (85)-(70) = 15 tC “Infidelity” TC ExpectedRelative Strength = (0.40-0.15) = 0.25

7. Why is this Important? • Statistical Conclusion validity • Unreliability of Treatment Implementation: Variations across participants in the delivery receipt of the causal variable (e.g., treatment). Increases error and reduces the size of the effect; decreases chances of detecting covariation. • Resulting in a reduction in statistical power or the need for a larger study….

8. The Effects Structural Infidelity on Power .60 Fidelity .80 1.0

9. Influence of Infidelity on Study-size 1.0 .80 Fidelity .60

10. If That Isn’t Enough…. • Construct Validity: • Which is the cause? (TTx - TC) or (tTx – tC) • Poor implementation: essential elements of the treatment are incompletely implemented. • Contamination: The essential elements of the treatment group are found in the control condition (to varying degrees). • Pre-existing similarities between T and C on intervention components. • External validity – generalization is about (tTx - tC) • This difference needs to be known for proper generalization and future specification of the intervention components

11. Infidelity Augmentation of Control PD= Professional Development Asmt=Formative Assessment Diff Inst= Differentiated Instruction So what is the cause? …The achieved relative difference in conditions across components

12. TTX’TTx Infidelity t tx tC “Infidelity” Positive Infidelity .45 .40 .35 .30 .25 .20 .15 .10 .05 .00 Intervention Exposure 100 90 85 80 75 70 65 60 55 50 True Fidelity Intervention Differentiation Achieved Relative Strength =.15 TC Tx Contamination Augmentation of C Intervention Exposure Treatment Strength Outcome Review: Concepts and Definitions

13. Some Sources and Types of Infidelity • If delivery or receipt could be dichotomized (yes or no): • Simple fidelity involves compliers; • Simple infidelity involves “No shows” and cross-overs. • Structural flaws in implementing the intervention: • Missing or incomplete resources, processes • External constraints (e.g. snow days) • Incomplete delivery of core intervention components • Implementer failures or incomplete delivery

14. A Tutoring Program: Variation in Exposure 4-5 tutoring sessions per week, 25 minutes each, 11weeks Expectations: 44-55 sessions Random Assignment of Students Time  Cycle 1 47.7 16-56 Cycle 2 33.1 12-42 Cycle 3 31.6 16-44 Average Sessions Delivered Range

15. Average Number of Tutoring Sessions per Tutor Individual Tutors Variation in Exposure: Tutor Effects The other fidelity question: How faithful to the tutoring model is each tutor?

16. In Practice…. • Identify core components in the intervention group • e.g., via a Model of Change • Establish bench marks (if possible) for TTX and TC • Measure core components to derive tTx and tC • e.g., via a “Logic model” based on Model of Change • Measurement (deriving indicators) • Converted to Achieved Relative Strength and implementation fidelity scales • Incorporated into the analysis of effects

17. What do we measure? What are the options? (1) Essential or core components (activities, processes); (2) Necessary, but not unique, activities, processes and structures (supporting the essential components of T); and (3) Ordinary features of the setting (shared with the control group) • Focus on 1 and 2.

18. Fidelity Assessment Starts With a Model or Framework for the Intervention From: Gamse et al. 2008

19. Core Reading Components for Local Reading First Programs Design and Implementation of Research-Based Reading Programs Use of research-based reading programs, instructional materials, and assessment, as articulated in the LEA/school application 1)Teacher use of instructional strategies and content based on five essential components of reading instruction 2) Use of assessments to diagnose student needs and measure progress 3) Classroom organization and supplemental services and materials that support five essential components Teacher professional development in the use of materials and instructional approaches After Gamse et al. 2008

20. From Major Components to Indicators… Indicators Major Components Sub-components Facets Scheduled block? Block Instructional Time Actual Time Reported time Instructional Material Reading Instruction Instructional Activities/Strategies Support for Struggling Readers Assessment Professional Development

21. Reading First Implementation: Specifying Components and Operationalization Adapted from Moss et al. 2008

22. Reading First Implementation: Some Results Adapted from Moss et al. 2008

23. So What Do I Do With All This Data? • Start with: • Scale construction, aggregation over facets, sub-components, components • Use as: • Descriptive analyses • Explanatory (AKA exploratory) analyses • There are a lot of options • In this section we describe a hierarchy of analyses, higher to lower levels of causal inference • Caveat: Except for descriptive analyses, most approaches are relative new and not fully tested.

24. Hierarchy of Approaches to Analysis: • ITT (Intent-to-treat) estimates (e.g., ES) plus: • an index of true fidelity: • ES=.50 Fidelity = 96% • an index of Achieved Relative Strength (ARS). • Hulleman’s initial analysis: ES=0.45, ARS=0.92. • LATE (Local Average Treatment Effect): • If treatment receipt/delivery can be meaningfully dichotomized and there is experimentally induced receipt or non-receipt of treatment: • adjust ITT estimate by T and C treatment receipt rates. • Simple model can be extended to an Instrumental Variable Analysis (see Bloom’s 2005 book). • ITT retains causal status; LATE can approximate causal statements.

25. More on Fidelity to Outcome Linkages • TOT (Treatment-on-Treated) • Simple: ITT estimate adjusted for compliance rate in Tx, no randomization. • Two-level linear production function, modeling the effects of implementation factors in Tx and modeling factors affecting C in separate Level 2 equations. • Regression-based model, exchanging implementation fidelity scales for treatment exposure variable.

26. Descriptive Analyses • Fidelity is often examined in the intervention group, only. • Dose-response relationship • Partition intervention sites into “high” and “low” implementation fidelity: • My review of some ATOD prevention studies, the ESHIGH =0.13 to 0.18 ESLOW =0.00 to 0.03

27. Some Challenges • Interventions are rarely clear; • Measurement involves novel constructs; • How should components be weighted? If at all. • Fidelity assessment occurs at multiple levels; • Fidelity indicators are used in 2nd and 3rd levels of HLM models, few degrees of freedom; • There is uncertainty about the psychometric properties of fidelity indicators; and • Functional form of fidelity and outcome measures is not always known. • But, despite these challenges, Chris Hulleman has a dandy example…

28. Assessing Implementation Fidelity in the Lab and in Classrooms: The Case of a Motivation Intervention

29. The Theory of Change INTEREST MANIPULATED RELEVANCE PERCEIVED UTILITY VALUE PERFORMANCE Model Adapted from: Eccles et al. (1983); Hulleman et al. (2009)

30. Methods(Hulleman & Cordray, 2009)

31. Motivational Outcome ? g = 0.05 (p = .67)

32. Fidelity Measurement and Achieved Relative Strength • Simple intervention – one core component • Intervention fidelity: • Exposure: “quality of participant responsiveness” • Rated on scale from 0 (none) to 3 (high) • 2 independent raters, 88% agreement

33. Exposure

34. Indexing Fidelity Absolute • Compare observed fidelity (tTx) to absolute or maximum level of fidelity (TTx) Average • Mean levels of observed fidelity (tTx) Binary • Yes/No treatment receipt based on fidelity scores • Requires selection of cut-off value

35. Fidelity Indices

36. Indexing Fidelity as Achieved Relative Strength Intervention Strength = Treatment – Control Achieved Relative Strength (ARS) Index • Standardized difference in fidelity index across Tx and C • Based on Hedges’ g (Hedges, 2007) • Corrected for clustering in the classroom (ICC’s from .01 to .08) • See Hulleman & Cordray (2009)

37. Average ARS Index Group Difference Sample Size Adjustment Clustering Adjustment Where, = mean for group 1 (tTx ) = mean for group 2 (tC) ST = pooled within groups standard deviation nTx = treatment sample size nC = control sample size n = average cluster size p = Intra-class correlation (ICC) N = total sample size

38. Absolute and Binary ARS Indices Group Difference Sample Size Adjustment Clustering Adjustment Where, pTx = proportion for the treatment group (tTx ) pC = proportion for the control group (tC) nTx = treatment sample size nC = control sample size n = average cluster size p = Intra-class correlation (ICC) N = total sample size

39. Average ARS Index Fidelity Achieved Relative Strength = 1.32 Treatment Strength 100 66 33 0 3 2 1 0 TTx Infidelity t tx (0.74)-(0.04) = 0.70 tC Infidelity TC

40. Achieved Relative Strength Indices

41. Linking Achieved Relative Strength to Outcomes

42. Sources of Infidelity in the Classroom Student behaviors were nested within teacher behaviors • Teacher dosage • Frequency of student exposure Student and teacher behaviors were used to predict treatment fidelity (i.e., quality of responsiveness/exposure).

43. Sources of Infidelity: Multi-level Analyses Part I: Baseline Analyses • Identified the amount of residual variability in fidelity due to students and teachers. • Du to missing data, we estimated a 2-level model (153 students, 6 teachers) Student: Yij = b0j + b1j(TREATMENT)ij+ rij, Teacher: b0j = γ00 + u0j, b1j = γ10 + u10j

44. Sources of Infidelity: Multi-level Analyses Part II: Explanatory Analyses • Predicted residual variability in fidelity (quality of responsiveness) with frequency of responsiveness and teacher dosage Student: Yij = b0j + b1(TREATMENT)ij+ b2(RESPONSE FREQUENCY)ij+ rij Teacher: b0j = γ00 + u0j b1j = γ10 + b10(TEACHER DOSAGE)j+ u10j b2j = γ20 + b20(TEACHER DOSAGE)j+ u20j

45. Sources of Infidelity: Multi-level Analyses * p < .001.

46. Case Summary The motivational intervention was more effective in the lab (g = 0.45) than field (g = 0.05). Using 3 indices of fidelity and, in turn, achieved relative treatment strength, revealed that: Classroom fidelity < Lab fidelity Achieved relative strength was about 1 SD less in the classroom than the laboratory Differences in achieved relative strength = differences motivational outcome, especially in the lab. Sources of fidelity: teacher (not student) factors

47. Key Points and Issues • Identifying and measuring, at a minimum, should include model-based core and necessary components • Collaborations among researchers and practitioners (e.g., developers and implementers) is essential for specifying: • Intervention models • Core and essential components • Benchmarks for TTx (e.g., an educationally meaningful dose; what level of X is needed to instigate change) • Tolerable adaptation

48. Key Points and Issues • Fidelity assessment serves two roles: • Average causal difference between conditions; and • Using fidelity measures to assess the effects of variation in implementation on outcomes. • Post-experimental (re)specification of the intervention

49. Thank You Questions and Discussion