Session 3 Design and Analysis of Trials of Therapist Treatments

1. MethodologyResearch Group Session 3Design and Analysis of Trials of Therapist Treatments Chris Roberts University of Manchester Email: Chris.Roberts@manchester.ac.uk Research funded by: MRC Methodology GrantsG0600555 G0900678, G0800606, G0802418 MHRN Methodology Research Group

2. Intervention Trials with clustering • Cluster randomised trials • Health professional activities - therapist trials • Group therapies/ group administered treatments trials In all three types of trial patients can be considered to be in clusters. Clusters may be: • Experimental units define by randomisation. • Observational units related to the delivery of treatment.

3. Overview of presentation • Cluster randomised trials • Design – Analysis –Sample Size – Advantage & Disadvantages • Therapist and Group Therapy Trials • Design – Analysis - In all three types of trial patients can be considered to be in clusters. Clusters may be: • Experimental units define by randomisation. • Observational units related to the delivery of treatment.

4. Cluster Randomised Trials (CRTs)

5. Cluster Randomised Trials Interventions are randomised to groups of patients (clusters) rather than to patients individually. In mental health CRTs are typically used to test: • IMPLEMENTATION of new procedures where the intervention applies to complete clinical units. • PROFESSIONAL BEHAVIOUR CHANGE where the intervention involves training of health professionals. For these types of intervention once the intervention has taken place it is difficult if not impossible for patients to receive the comparator without changing clinical unit or health professional.

6. Mental Health Cluster Randomised Trials Example Systematic assessments of need and care planning in severe mental illness Priebe et al BJP, (2007) 191: 420-426. Aims To test a computer-mediated intervention structuring patient–clinician dialogue (DIALOG). Methods 134 key workers (clusters) were allocated to DIALOG or treatment as usual. 507 people with schizophrenia or related disorders were included. Outcome Measures Primary outcome was subjective quality of life, and secondary outcomes were unmet needs and treatment satisfaction.

7. Illustration of Cluster Randomised Trials Schematic Unit Diagram Classification Diagram

8. A statistical model for a cluster randomised trial Patient variation Treatment effect Variation between clusters Baseline covariatesxand coefficient 0/1 variable indicating treatment cluster(i) indicates the cluster id of the ith patient

9. Methods of statistical analysis Summary measure analysis Use each cluster as the unit of analysis. Obtain an average for each cluster then compare treatments use a t-test applied to the cluster averages. Modelling Methods Maximum Likelihood (ML) REstrict Maximum Likelihood (REML) Generalised Estimating Equation (GEE) Survey Estimators (SVY in Stata)

10. Measuring the effect of clustering Intra-cluster correlation (ICC or ρ) = Proportion of Variance due to cluster Between Cluster Variance Patient Variance ICC=0 implies no variation between clusters. ICC=1 implies all patients in each cluster have the same outcome. Whether the ICC should be allowed to have a value less than one is debated. Typically, the ICC takes values <0.05 for clinical outcomes in large samples.

11. Sample Size and Power for a Cluster Randomised Trial As well as effect size, test size () and s.d., power depends on: • number of clusters (n) • numbers of patients per cluster (m) • intra-cluster correlation coefficient (ρ) Design Effect (DE) = Ratio of the number of subjects required where there is clustering to the number required assuming no clustering. Assuming clusters are of equal size ICC (ρ) Cluster size (m)

12. Power if sample size ignored clustering

13. Increasing the number of Subjects/Cluster

14. Increasing number of clusters/arm

15. Advantages of Cluster Randomised Trial • No contamination between the interventions being compared. • External validity as intervention can be made to correspond more closely to Usual Care as patients not being individually randomised. • Administrative and logistic convenience. • Acceptability to health service staff as less disruptive than individualpatient randomisation. • Potential for greater recruitment of patients and groups. • Patient consent may be simpler.

16. Disadvantages of Cluster Randomised Trial Any two patients within the same cluster are likely to be more similar than two patients from different clusters due to: • Characteristics of patients in a cluster. • Patient interaction within a cluster. • Measured and unmeasured characteristics of health care delivery of a cluster. This effect is called clustering or more formally intra-cluster correlation. Valid statistical methods need to take account of the implications of intra-cluster correlation. This requires: • More complex methods of statistical analysis. • A larger sample size of patients is required to achieve the same power and precision as the corresponding individually randomised trial. Internal validity depends on the selection of patients being independent of randomisation.

17. Summary: Cluster RCTs Use of cluster randomisation may have advantages including external validity but: • Precision is reduced. • Validity depends on patient recruitment being independent of the cluster randomisation.

18. Therapist and Group Administered Treatments Trials

19. Therapist Variation and Clinical Trials Outcome for patients treated by the same therapist may be more similar than outcomes for patients treated by different therapists due to Therapist characteristics such as: • Experience • Training • Competence • Empathy Patients can be considered to be clustered by therapist. This variation has implications for the Precisions and Validity of psycho-therapy trials.

20. Consort Guidance for Non-pharmacological Trials A recently published extension of the Consort Guidelines for Non-pharmacological Treatment Trials has drawn attention to the issue. They recommend that trial of Non-pharmacological Treatment reports how clustering by care-provider has been considered in relation to: • Selection of care providers. • Sample size calculation. • Allocation of care providers to each trial arm. • Statistical analysis of outcome.

21. Type of Comparisons involving Care Providers Techniques Different treatment methods delivered by the same care provider. Face-2-Face and Telephone delivered CBT therapy for patients with OCD. Different surgical methods. Care Provider Characteristics Comparison of nurse practitioners and general practitioners in primary care. Packages Different techniques and different characteristics combined. CBT delivered by a clinical psychologist with Non-directive counselling delivered by a counsellor.

22. Group Administered Treatments • Examples: Group CBT, Substance misuse, Anger management • Outcome may be more similar for subjects in the same class or group as patients may interact and this may be a component of the treatment. • The clustering effect of group deliver only applies to group therapy arm if trial patients are randomised individually. • Groups may be closed, open, or rolling. • Where groups are closed, each therapy group can be considered as a cluster.

23. Care Providers and Trial Designs

24. Trial Designs for Therapist Treatment • Nested (Therapist) Design also called Hierarchical (Therapist) Design. • Partially Nested (Therapist) Design. • Crossed (Therapist) Design also called a Stratified (Therapist) Design.

25. Nested Therapist Design Schematic Unit Diagram Treatment Classification Diagram

26. Partially Nested Therapist Design Schematic Unit Diagram Treatment Classification Diagram

27. Crossed Therapist Design Schematic Unit Diagram Treatment Classification Diagram

28. Crossed Therapist Design Same health professional delivers both interventions. Example: Treatment delivered face-to-face or by phone Similar to: • Multi-centre study: Therapist  Centre/Study • Meta-analysis: Therapist Study • Crossover cluster randomised trial: Therapist  Cluster • Matched pairs cluster randomised: Therapist  Matched pair Treatment effect can be estimated within therapists See Appendix for details of regarding analysis and sample size.

29. Statistical Analyses for Nested and Partially Nested Designs • Statistical analysis of Nested and Partially nested designs are similar to cluster randomised trials. • There is nevertheless added complexity due to the cluster being defined by treatment rather pre-specified for the purpose of randomisation.

30. Comparison of Cluster Randomised, Therapist & Group Therapy Trials

31. Care Providers and Statistical Analysis

32. Models for Nested Therapist Design Patient Variation Therapist (or Group) Variation Treatment Effect Covariance between utherapist(i) and vtherapist(i) not identified. Therapist (or Group)-Treatment Interaction Random Effect

33. Level Heteroscedasticity in a Nested Therapist Design In cluster randomised trial between arm heteroscedasticity generally ignored and not important Treatment Effect Therapist or Group Variation Patient Variation Heteroscedasticity in patient variation between treatments (level 1) may bias estimates of therapist variances (level 2) if cluster size differ.

34. Level Heteroscedasticity in a Nested Therapist Design (Alternative Parameterization) In cluster randomised trial between arm heteroscedasticity generally ignored and not important Treatment Effect Patient Variation Therapist or Group Variation Heteroscedasticity in patient variation between treatments (level 1) may bias estimates of therapist variances (level 2) if cluster size differ.

35. Intra-cluster Correlation for Therapist (ρ) ICC (ρ)= Proportion of Variance due to Therapist Therapist Variance Patient Variance ICC for second treatment where differential clustering is considered is

36. Partially Nested Design Subjects in control arm are treated as clusters of size 1 [ref Roberts & Roberts 2005] Example: No Drug vs. Drug Therapy Comparison of group therapy with individual therapy. Treatment Effect Therapist or Group Variation Patient Variation

37. Heteroscedasticity and the Partially Nested Design Failure to take account of heteroscedasticity can bias estimates of therapist variation and can effect test size [ref Roberts & Roberts 2005] Example: No Drug vs. Drug Therapy Comparison of group therapy with individual therapy. Treatment Effect Therapist or Group Variation Patient Variation

38. Crossed Design How should clustering effect be considered in a crossed design? Loss of precision and design effect depends on variance of vtherapist(i) but not utherapist(i) . A parameter for sample size calculation could be Variance of treatment with therapist interaction (vtherapist(i) ) Pooled Variance of patient within treatment

39. Crossed Design (Alternative parameterization) Three level model with patients nested in treatments nested in therapist qtreat(i)is the variation between treatments within therapist Loss of precision and design effect depends on variance of qtreat(i) but not ptherapist(i) . Variance of treatment with therapist interaction (qtreat(i) ) Pooled Variance of patient within treatment

40. Internal and External Validity

41. Internal Validity and Therapist Variation Randomisations Possible in Trials of Therapeutic Approaches • Random allocation of treatment to therapist to prevent confounding of therapist characteristics with the treatment effect. Example: Schnurr et al. • Random allocation of Therapist to Patient to would prevent confounding of therapist variation with patient characteristics. Example: Schnurr et al.

42. External Validity: Selection of Therapists Where interventions cannot be randomised to therapists (trials of Therapist Characteristics and Psychotherapy Packages) selection of therapists has implications for external validity.

43. External Validity of Therapist Particularly for trials of Therapist Characteristics and Psychotherapy Packages selection of therapist has implications for external validity. Need to consider how representative trial therapists are of non-trial therapists. Just as for patients in any trial inclusion and exclusion criteria are need for therapist. Recruiting therapists directly from clinical practice using broad eligibility criteria might be more representative but this could also increase the intra-cluster correlation.

44. External Validity: Therapists and Patient characteristics Randomisation of treatment to patients balances patients characteristics across arms. Since it is rare to be able to randomise to therapists to patients, therapist variation may be confounded by patient characteristics. Two possibilities for analysis are: • Estimating the ICC by therapist for patient baseline characteristics. • The effect of baseline covariates on the ICC for therapist for the outcome measures.

45. Could ICC for therapist be induced by unobserved confounding? Suppose we have • Outcome Y and unobserved confounder Z • Suppose ICC for Z is Z • Suppose rYZis the correlation between Y and Z • Induced ICC is Z rYZ2 Given that rYZ is unlikely to exceed 0.5 Z would need to be large for an ICC of Y to be induced by unobserved confounding.

46. Internal Validity and Therapist Variation in a Crossed Design. • Patients stratified by therapist. • Treatment effect estimated within therapist. • Random allocation of Therapist to Patient to could in theory prevent confounding of treatment effect by the therapist and patient characteristics. • Randomisation may not be desirable as in usual care certain types of therapist may treat certain type of patient.

47. Treatment Comparisons in Psychotherapy Trials Therapeutic Approaches Either Crossed Design or Nested Design. Therapist Characteristics Nested Design Psychotherapy Packagescombining different approaches and different characteristics Nested Design Whether a trial is of Therapeutic Approaches, Therapist Charactistics or a Package depends on the counter factual treatment. What would/should a patient receive if they did not receive the intervention?

48. An Example Comparison of CBT & NDC for Depression in Primary Care Data from Ward et al BMJ 2000;321:1383-8

49. Trial: CBT v NDC for Depression in Primary Care Participants: 262 patients presenting with depression or mixed anxiety + depression. Interventions: Up to 12 sessions of therapy • cognitive-behaviour therapy (12 psychologist) • non-directive counselling (14 counsellors) Primary outcome measures: Beck Depression Inventory scores at 4 and 12 months. Original published analysis ignored clustering by therapist. Reanalyzed including random effect for therapist.

50. Trial: CBT v NDC for Depression in Primary Care Treatment Classification Diagram