A Multi-Aspect Approach to the Evaluation of Outcome Quality in Psychosomatic Treatment

1. A Multi-Aspect Approach to the Evaluation of Outcome Quality in Psychosomatic Treatment Werner W. WittmannUniversität Mannheim, Lehrstuhl Psychologie II Jürgen SchmidtPrivatinstitut für Evaluation und Qualitätssicherung im Gesundheits- und Sozialwesen (eqs.)mbH, Karlsruhe Workshop: „Quality Management and Outcome Monitoring“Stuttgart, 14.03. - 16.03.2002

2. Jürgen SchmidtPrivatinstitut für Evaluation und Qualitätssicherung im Gesundheits- und Sozialwesen (eqs.)mbH, Karlsruhe

3. A stately mansion of evaluation research and quality control Evaluation research Program evaluation Research design and data analysis Assessment Decision and evaluation aids Donald T Campbell Lee J Cronbach Eleonor Chelimsky Len Bickman

4. Situations, Time Variables Subjects DATA BOX (CATTELL, 1957)

5. Intervention,program ortreatment Post interventionarea of criteriaand goals PRE-Intervention ETR-Box Northwestern Path Area of stakeholderinterests r ETR, CR r PR, ETR EVA-Box PR-Box r PR, CR CR-Box r PR, NTR r NTR, CR Southwestern Path NTR-Box The Five Data-Box Conceptualization

6. Richard Feynman Murray Gell-Mann Egon Brunswik Michael Faraday Success through symmetry: A really magic concept

7. ETR-Box PR-Box CR-Box NTR-Box The beauty of Brunswik symmetry ETR - Experimental Treatment BoxPR - Predictor BoxCR - Criteria BoxNTR - Nonexperimental Treatment Box

8. Full asymmetry, the case of nothing works!All correlations between predictors and criteria are zero! Hierarchy of predictors Hierarchy of criteria

9. Asymmetry due to a broad higher level prediction and a narrower lower level criterion Hierarchy of predictors Hierarchy of criteria

10. Asymmetry due to a narrower lower level predictor and a broad higher level criterion Hierarchy of predictors Hierarchy of criteria

11. The hybrid case of asymmetry, mismatch at the same level of generality! Hierarchy of predictors Hierarchy of criteria

12. Selection effects due to restriction (enhancement) of range 1 Danger to overestimate 1 Danger to underestimate Psychometric reliability of experimental treatment and criterion 2 Dangers to underestimate Construct reliability of experimental treatment and criterion 2 Dangers to underestimate (lack of symmetry) Sampling error 1 Danger to overestimate (possitive error) 1 Danger to underestimate (negative error) The Brunswik-lens-equationfor relating experimental treatment (ETR) to criteria (CR) There are 6 dangers to underestimateagainst 2 dangers to overestimatea true effect size! So do you still wonder??

13. PR-Box CR-Box NTR-Box Southwestern path

14. Treatment(6-8 weeks) -1 +1 +2 +3 years POST 1 POST 2 PRE One year period before intake all patients should base their answers in our assessment instrument on such a one year period One year after discharge f.e. follow-up assessment of MACT_27 Three years after discharge assessments based on one year period second to third year after discharge The structure of the follow-up research design in the Magic-Mountain study

15. Multiple-act criterion

16. Predictor Predictor Treatment Treatment Criterion Criterion Variants of aggregating assessment information Single multiple acts (SMAC)

17. Theory derived indices ? SAC RAC SMAC RMAC Single-act assessment= one specific act in one specific situation Repeated act assessment= repeated observation of one specific act over time and/or situations Single multiple act assessment= one observation of multiple acts, belonging to one behavioral class at one time point or situation Repeated multiple act assessment= repeated observation of multiple acts belonging to one behavioral class over time and/or situations A taxonomy of assessment variants

18. Mact 27 1. How do you feel at follow-up (oneyear after) compared to intake? For indicators 2. to 27.:change in ... 2. quality of life 3. state of health 4. state of mind 5. general condition 6. competitiveness 7. complaints/troubles/disturbances 8. state of health 9. dealing with problems/daily pressure 10. life-style concerning health 11. drug consumption 12. relationship to most closely related people 13. relationship to partner 14. family life 15. ability to work 16. number of visits to physician 17. time of inability/absence from work 18. days in hospital 19. well-being 20. dealing with problems/coping 21. capability for self-help 22. endurance of disappointment 23. coping with work 24. ability to take stress 25. getting along with people 26. getting along with problems/limitations/ impairments 27. balanced mood

19. All 27 components had to be rated according to differences in change comparing the pre year period after treatment in relation to the one year period after treatment. One point was given for improvement or positive reduction, a zero point for no change, reduction or worsening. So the worst case is a MACT_27 score of zero,the best case a MACT_27 score of 27.

20. Distribution of MAC-27-Scores 1 year after treatmentN = 367 (Magic-Mountain II-Study)

21. 1) Quality of alliances to therapist Y 2) Quality of self experiences and realization of insight into own problems .196** .075 Multiple act criterion MACT_27 .044 3) Quality of alliances to co-patients .126 4) Quality of conventional treatment, i.e. nonpsychological .331*** 2 R = .232* **adjusted R2 = .206 y . 12345 5) Amount of demoralization, at discharge (high scores low amount) Significance of Beta-weights: *** p < .001 ** p < .01 N = 151 Regression analysis of MACT_27 on quality of treatment and amount of demoralization

22. Another multiple-act criterion

23. MACT-MO: • Change in ... • Drug consumption • ability to work • number of visits to physician • time of inability to work • days in hospital

24. 1) Quality of alliances to therapist Y .160* 2) Quality of self experiences and realization of insight into own problems -.057 Monetary multiple act criterion MACT_MOconsisting of five monetary aspects .000 3) Quality of alliances to co-patients .071 4) Quality of conventional treatment, i.e. nonpsychological .295** 2 R = .123 **adjusted R2 = .084 y . 12345 5) Amount of demoralization, at discharge (high scores low amount) Significance of Beta-weights: ** p < .01 * p < .05 N = 151 Regression analysis of MACT_MO on quality of treatment and amount of demoralization

25. Quality of alliance to main therapist Quality of realisation of insight into own problems .135* .557*** Quality of alliances to co-patients Amount of demoralizationat discharge(high scores low amount) .170** .200** Quality of conventional treatment .023 .110° KATMISSMissing-data dummy at one year follow-up KATMISS*Quality of conventional treatment 2 R = .418adjusted R2 = .401 Significance of Beta-weights: *** p < .001 ** p < .01 * p < .05 ° p < .10 N = 215 Regression analysis of demoralization on quality of treatment plus the missing-data-dummy

26. NTR-Box (Nonexperimental Treatment Variables) CR-Box PR-Box Quality of relationship to therapist (highscores good therapeutic alliances) MACT_27 Multiple act outcome consisting of 27 aspects (high scores good) one year after discharge .12* Amount of demoralization at discharge (high scores low amount) .41* Quality of self experiences and insight into own problems (high scores good) Amount of demoralization at intake (high scores low amount) .40* .35* -.22* .16* .23* .62* Quality of family relation- ship (high scores good) Amount of live events with negative financial impact (high scores many) Amount of positiv- ely evalu- ated life events (high scores many) -.15* Quality of conventional treatment, i.e. nonpsychological (high scores good) Legend: * Significant standarized path coefficients (N = 151) p < .05 EQS model indizes: CHI-square for the independence model: 319.871 df = 36 R 2MACT27 = .370 (.609) CHI-square for postulated model above: 40.20 df = 26 p = .037 adj. R 2 = .335 Bentler normed fit index: .875 Bentler non normed fit index: .931 Comparative fit index (CFI): .950 .29* These three variables are assessed with respect to the one year time period since discharge, i.e. history effects or extra treatment variables Path-analytic (causal) modelling of treatment effects in the magic mountain study causally investigating the southwestern path of the five-data-box conceptualization

27. Amount of demoralization at intake (high scores, low demoralization r tt = .90 (1.00) -.336°° (-.423) Reduction of demoralization at discharge (post-pretest change scores high scores high reduction) r tt = .70 (1.00) -.383** (-.500) .191° (.225) Hours of therapeutic interventions (higher scores more) r tt = .80 (1.00) .246** (.344) .173° (.231) R2 = .171** adjusted R2 = .159 Significance of Beta-Weights:** p < .01 N = 137 Significance of bivariate correlations:°° p < .001 ° p < .05 (.291) (.280) Regression analysis of demoralization change scores on amount of demoralization at intake and hours of therapeutic interventions demonstrating the validity of change scores Interpretation: The lower the amount of demoralization at intake the lower the amount of reduction of demoralization. The higher the number of hours of interventions the higher the reduction.That means those with high demoralization at intake had a higher amount of reduction than those already low on demoralization at intake! Alhough the bivariate correlations show a small significant effect of selection into treatment, the higher dosage for those already lower on demoralization did not pay off in higher reduction. The values in parentheses correspond to an analysis with corrections for attenuation.

28. COLLECTION INTO TREATMENT PR-Box Amount of demoralization at intake (high scores, low demoralization)r tt = .90 -.336 -.520 NTR-Box F4HSET (2)Insights how to deal with own problemsr tt = .86 F4HSET (3)Copatients experiences a helpfulr tt = .65 F4HSET (4)Classical physical treatment and psychotherapy experienced as helpfulr tt = .65 .247 .486 .153 .130 .246 .335 .051 .182 .111 The pattern of correlation/regression parameters between intake (PR-Box), nonexperimental treatment factors (NTR-Box) and change processes (CR-Box) CR-Box Reduction of demoralization at discharge (Post-pretest change scores, high scores, high reduction). r tt = .70 R 2 = .373adjusted R 2 = .353 all beta-wights are significant N = 137

29. COLLECTION INTO TREATMENT PR-Box Amount of demoralization at intake (high scores, low demoralization)r tt = 1.00 -.423 -.802 NTR-Box F4HSET (2)Insights how to deal with own problemsr tt = 1.00 F4HSET (3)Copatients experiences a helpfulr tt = 1.00 F4HSET (4)Classical physical treatment and psychotherapy experienced as helpfulr tt = 1.00 .291 .740 .295 .170 .457 .165 .076 .248 .448 Attenuation corrected correlation/regression pattern between intake (PR-Box), nonexperimental treatment dimensions (NTR-Box) and change processes (CR-Box)The analysis is based on true scores of all variables only CR-Box Reduction of demoralization at discharge (Post-pretest change scores, high scores, high reduction). r tt = 1.00 R 2 = .768adjusted R 2 = .761 all beta-weights are significant N = 137