1 / 29

Measuring Change in Two-Wave Studies

Measuring Change in Two-Wave Studies. David A. Kenny. The Focus Here in Measuring Change. To understand the causes of change, e.g., intervention analysis. Not: To understand the variable of interest and how it changes. To measure who changes more or less in the sample. A Bit of Notation.

karena
Download Presentation

Measuring Change in Two-Wave Studies

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Measuring Change in Two-Wave Studies David A. Kenny

  2. The Focus Here in Measuring Change • To understand the causes of change, e.g., intervention analysis. • Not: • To understand the variable of interest and how it changes. • To measure who changes more or less in the sample.

  3. A Bit of Notation • Two times • Time 1 measure: Y1 • Time 2 measure: Y2 • X a potential cause of change in Y • easier to think of as a dichotomy (0 and 1) but not required • X may be randomized or not

  4. Change Score Analysis (CSA) • Time 2 minus Time 1 • Seen by some as a naïve model. • Can compute the difference score more indirectly (keep Time 1 in the model).

  5. Control for Baseline (CfB) • Adjust out for the effect of Time 1, i.e. empirically weight Time 1 and not fix its coefficient to one. • Generally preferred method.

  6. Equations • Change Score Analysis (CSA) Y2 – Y1 = a + bX + e • Control for Baseline (CfB) Y2 = a + bY1 + bX + e (will refer to b later)

  7. The Coefficient for Y1 Note that we can rewrite CSA as Y2 = a + 1Y1 + bX + e Thus, CSA fixes the coefficient for Y1 to one, whereas CfB empirically estimates it. For CfB usually, though not always, b is less than 1. In fact, if the variances of Y1 and Y2 are equal, b cannot be bigger than 1 (given that b = 0).

  8. Lord’s Paradox • CSA and CfB yield different results when b is different from 1 and there is correlation between X and Y1.

  9. Why Different? X=1 Y2 X=0 Time 2 Time 1

  10. How Different? X=1 Y2 gap X=0 Time 2 Time 1

  11. CSA: The Gap Persists X=1 Y2 X=0 Time 2 Time 1

  12. CfB: The Gap Narrows (b < 1) X=1 Y2 X=0 Time 2 Time 1

  13. CSA vs. CfB X=1 CSA CfB Y2 X=0 Time 2 Time 1

  14. Randomized Studies • Lord’s paradox happens only when there is some association between X and Y1,i.e. a gap. • If there is randomization, there is no gap in the population. • By chance, there is some gap in the sample.

  15. Estimate of the Treatment Effect (b) • MD1: The mean difference between treatment groups at time 1. • MD2: The mean difference between treatment groups at time 2. • CSA: b = MD2 – MD1 • CfB: b = MD2 - bMD1 • If b < 1, then CfB adjusts less than CSA.

  16. and so if b < 1… • If MD1 is positive (1s start ahead of 0s), more gets subtracted using CSA, resulting in a smaller b than CfB. • If MD1 is negative (1s start ahead of 0s), more gets added using CSA, resulting in a larger b than CfB.

  17. Strategy • In randomized studies and in non-randomized studies in which the gap is small, controlling for the Y1 should be done as it almost always has more power than change score analysis.

  18. What Is Meant by “Small” • Suggestion • If the effect of X on Y1 is not statistically significant and the r between X with Y1 is less than .075 in absolute value or d of less than 0.15 in absolute value, then the gap is small.

  19. But what if there is a gap at Time 1? • The bigger the gap, the greater the divergence between the two methods. • To choose the right analysis, we need to know why there is that gap: What causes that gap and how does it change?

  20. Report the Size of the Gap! • In non-randomized studies, we need to know the size of the gap. • If X is a dichotomy, report d. • If X is continuous, report the correlation of X with Y1.

  21. The Assignment Variable • See Judd & Kenny book (1981) • Z is an unmeasured variable that explains the gap at time 1. • It does so because it determines X (selection), but it also associated with the Y1. • If Z were measured and controlled for, there would be no gap.

  22. Model for CSA • Think of Z as a dichotomy, with levels 1 and 2. • We measure the mean of Y1 and Y2 for different values of Z. • We compute the gap at each time (controlling for X). • The gap is the same at both times.

  23. Continuous Z • Z need not be a dichotomy, but the general points still hold. • The effect of Z on Y1 or g is the same as the effect of Z on Y2, controlling for X.

  24. Example • Z is gender. • Proportionally more men in the treated group than the control group. • Gender differences in Y are same at the two times (controlling for X).

  25. Model for CfB • Z causes or is correlated with causes of Y1. • However, Z has no effect on Y2 once Y1 is controlled. • Y1 serves as a complete mediator of the Z-Y2 relationship.

  26. b 0

  27. Example • Schools try to select low performing students into a the treatment group. • Z is child’s ability at time 1. • As Y1 is a measure of ability, it explains any relationship between Z and Y2 (path from Z to Y2 is zero).

  28. Other Presentations • Change Score Analysis • Standardized Change Score Analysis • Controlling for the Baseline Analysis • Power in the Study of Change

More Related